Ah, well here's a theory.
When you play with a skin with small bars, like the o2jam notes, osu! caches the note size and keeps it until you restart osu.
If you switch to a skin with a different note size, it stretches the image to the previous skin's note size.