Some SB's cause major lags on some PC's, especially those which have a video-like storyboards like Neko Miko Reimu or Don't Stop Me Now or whatsoever. Even my PC can run at 60fps (VSync) with those SBs with me having a single-core processor and a mere 1GB RAM and an IDE hard drive I still can play it well. (Note even my PC can only use OpenGL)
I don't see any problem with SBs though I have a mild epilepsy, which doesn't cause much problem with me even when playing StrangeProgram or Ritsuen; which I might conclude as a challenge rather than a restriction, though I might have a mild headache after playing them.
Skins however, like what Zekira said, slightly affects gameplay. HP bars, hitcircles, cursors, etc. might change the mood of gameplay on a certain map. e.g. if a map's skin is so goddamn horrible you don't have the will to play the map at all. However, that depends on a specific player as each has different taste on certain things.
IMO the only thing I think is okay to delete in a map is the skin elements. For SBs? I go for 50% of it, since I'm very confident that some people's PC specs are far worse than me.