First of all, the way you say "4k normal", "5k normal" and "6k hard" suggests that you are playing autoconverts (standard maps that can be played in mania mode by automatically generating mania notes based on the standard mode beatmap). This is something you should avoid. Autoconverts have many problems.
First, they are always OD10, meaning accuracy is very harsh on them compared to maps made specifically for mania. Most mania maps use between OD5 and OD9, even much harder ones.
Second, the patterns in autoconverts do not make sense. In standard, you can't have more than circle or slider (note or hold) at the same time. That means the way standard mappers handle the rhythm of the music ends up being very different from how mania mappers handle the it. The actual placement of notes is also a problem. Autoconverts don't understand what patterns feel good and what ones feel bad, and they also cannot make patterns that are based on the music, which is a very important part of mapping for mania. These problems actually make autoconverts harder than maps made specifically for mania in some ways.
Third, autoconverts automatically pick the number of keys they use based on the difficulty of the original map. That means that they usually have a series of difficulties that look like "4k easy, 5k normal, 6k hard, 7k insane" or something. Maps made for mania do not do that. In fact, it's not even rankable for a mania map to do that now. It was rankable for a very short time, but now the ranking criteria requires at least 3 difficulties in every key mode that a mania map uses. Most mania maps either contain 4k difficulties, or 7k difficulties. Sometimes they have both. There are some maps that use 5k, 6k, 8k or 9k, but they are much more rare than 4k and 7k.
Most players stick to one key mode and focus on getting better at just that key mode. Some players, like me, play a range of key modes. I play 4k, 5k, 6k and 7k. I mostly play 7k, and my second most played mode is 4k, but I also enjoy playing 5k and 6k now and then. If you want to be able to play a bunch of different key modes, I suggest figuring out what the highest number of keys you want to play is, and stick to ONLY learning that key mode for a while. And by a while I mean probably a few months. It's very easy to learn to play a lower key mode after you've learned a higher key mode. For example, when I started playing osumania, I had already played o2jam (7k game) and stepmania (4k game) before, but it had been around 6 years since I played 4k, and my 4k skill was very bad. After playing only 7k in osumania from 2013 to 2016 or so, I finally went back to 4k. It took me maybe a week or 2 to get back to how good I was at 4k at my very best. I quickly got a lot better than that.
And finally, everyone learns at different speeds. There are some people who can S 6 star 7k maps after 5 or 6 months, but I still can't S any 6 star 7k maps after playing 7k for a total of something like 10 or 11 years. I have never met anyone who learns rhythm games as slowly as I do, so chances are if you spend a year or 2 playing mania, you'll be better than me, even if you're a slower learner.