Honestly, it's more of the mapping style that's driving me away from o!mania. Many of the 7K players here came from places like o2jam, so a large portion of the ranked maps (last I checked) get a fair bit of their difficulty from noodles hold note patterns and the like. I personally prefer stuff like Beatmania IIDX (or BMS by extension), where long notes didn't even exist for 10 years, and even now, are still used fairly sparsely.
There are issues with the o!mania engine itself too (it was a huge mess when it was first implemented, and it didn't help that the original developer all but abandoned it), but from what I can tell from the game's changelog, they're making progress on fixing that. I do like the Stepmania engine as far stuff like scoring and customization goes, but I will agree with you that its datedness is really holding it back.