Hitting late on harder parts is normal and makes sense. What's likely happening there is that your brain is taking just a BIT too long to figure out what keys to hit (you reading is a bit slow, not enough to kill you, but enough to delay your ability to hit the notes on time). That happens to me very often, and I've been playing these kind of games for a very long time.
As for hitting early, that probably means you're not paying close enough attention to the music itself. That suggests to me that you're focusing on the visual cues for timing, not the music. Ideally you should try to focus more on using the music itself to time your notes, but you can also try lowering your hit position (judgement line) a bit.
I won't say that's 100% exactly what's going on, but in my experience that's the most likely reason for those issues. Ultimately, it's up to you to figure out what the problem is and find a solution that works for you. However, assuming that's what's happening I've got some suggestions for ways that you can try to deal with those issues.
If you want to train yourself to use the music more, I'd suggest trying to play with hidden (or flashlight) on and hitsounds turned up a bit, to make them easier to hear over the music (if you do this, you should make sure that your hitsound do not have empty time before the sound happens, and you might need to set your global offset differently if you use hitsounds for helping timing too heavily).
For example, I play with flashlight, and I have my global offset at -20 to compensate for hitsound delay to some degree. If you use flashlight, you'll want to lower your scroll rate to probably half your current scroll speed (or maybe a bit faster, whatever you think will be comfortable). If you pick either hidden or flashlight, you'll need to play with it on all the time for a while in order to fully adjust to the effects of the vision mod (and it is slightly risky. many players are unable to switch back out of using vision mods once they get completely used to them.)