TL;DR: For me, the hardest mode to map in my opinion is also osu!standard, followed by osu!catch, osu!mania and osu!taiko.
Although I haven't tried mapping osu!catch and osu!mania, for now I'm thinking they're kind of in a similar position.
When it comes to mapping, osu!standard has a lot of ways to map, not only by just following the rhythm. Therefore it makes me think osu!standard is the hardest among all four modes, and I agree mostly from Gsun. It requires much creativity to put into a theme on which type of mapping style fits on different genres of music, either be anime, tech or tap control. What differs from osu!catch is osu!standard uses a full playfield, whereas osu!catch only prioritises the X-axis, making it seem easier than the normal osu!. Despite osu!standard is the hardest to map, I can also say it is the most fun to map due to giving out ideas to a single map, how it plays and works for some inspiration to create even new.
osu!catch on the other hand, is my 2nd hardest mode to map - kind of. osu!mania and osu!taiko only rely on notes, and even if there's slider the main difference is the slider velocity implemented on them. osu!catch, just like osu!, differs a lot when using slider velocity on sliders - either making it short or long in a few beats, while osu!mania and osu!taiko doesn't affect that much of the gameplay, only to change the scroll speed of it. osu!catch mains on distance snap is the only thing that challanges osu!catch mappers, as it is crucial to know what difficulty is suited based on it. The rest is I can say it's easier than osu!.
The third one would be osu!mania. If it were to say 4K, I can put it here, but anything like 7K or up to 10K I'm quite hesitate to put it as 2nd. I don't map or see much into osu!mania mapping but I do think it's more into knowing which pattern to use on which occasion of the song. Unlike osu!taiko, you'll have columns in which there's various ways to put the notes and long notes in. Therefore, I can say that makes it harder since you'll have to like spread it out in columns and such, so it doesn't feel too one-sided and such. Oh, it also needs dense pattern on higher difficulty.
Lastly osu!taiko, I can say it is the easiest. You'll just have to change the dons to kats with claps or whistle - prefer claps, and finisher with finish. Following from Zelzatter Zero's post: True, osu!taiko mains on hitsounding by listening to the cue of the song, but what makes it different from not just osu!, but any modes is that most of the time osu!taiko doesn't need custom hitsounds. So you can freely hitsound without worrying which custom hitsounds are suitable for it. Other than that, the patterns in osu!taiko is also limited, triplet DDK or KKD, 5 note stream or 7 note stream etc. Making it look like less of a hassle than osu!mania.