Perhaps it is easy for you, because you have played enough that you have just become more accurate in general. Being able to time hits correctly is a skill that some people take for granted when they play. Many people develop bad habits that make them less accurate. The people who know what they are doing don't need to rely on the approach circles to be accurate. They had to develop this ability though. It is far easier to take advantage of the generous hit window instead of actually learning when to hit.
The same also applies to streaming, but in a more skillful way. It takes skill to stream at a particular BPM. Streaming at the correct BPM for the song is far better than taking advantage of the hit window that allows you to lag behind or hit late and still get a 300.
Hidden takes away some of the things that people with bad habits use to play. Lagging behind on a stream will certainly ruin your timing coming out of the stream, and it becomes harder to recover. People who stay accurate will not need to deal with finding the rhythm again and will find playing easier.
Hidden is harder in that it obscures that hit window, and requires that the player remember the hit window, and the BPM. In most cases, this mod will increase a player's unstable rate for a song unless there is a really well defined rhythm. The more chances where a player can lose the rhythm, the harder the map will be to play with hidden. This is why easier difficulties tend to be harder with hidden. It is easier to lose the rhythm and the delay between when a circle appears and when it needs to be hit is longer.
I reckon it would be hard to define the difficulty of hidden on a per map level.
Some quick guidelines
-Lower AR = harder with hidden (Because our brains our used to playing with higher ARs/less delay)
-2/1 or greater gaps in the beat = harder with hidden (We can't rely on consistent rhythm to hit these. It is why people have problems hitting the first object in a map in Hidden)
-Faster jumps and streams tend to require more accuracy by default and so the burden to keep timing is less apparent (This is why people find Hidden not a large step up from how they usually play. Faster = easier. Slower = harder)
-Sliders are not any more difficult in Hidden. (However maps that contain streams that alternate between sliders and circles can interfere with a player's timing)
-Stacked/hidden objects are slightly more difficult. People who haven't yet started playing according to musical cues or hasn't developed muscle memory to overcome the need for musical cues will find that obscure objects will be harder to hit, because they can't observe the object's fade, and as such will have to know beforehand the correct timing to hit the object. A single mistimed hit can spoil the whole string of objects.
-Most will probably find Hidden+ Flashlight harder than Hardrock + Flashlight given the same amount of skill level in playing both hidden and hardrock alone in the same map. Flashlight with hidden has the side effect of rendering the entire map as if it is obscured like a stack. There is no space for error when you can't see what you are hitting.