What's the process to put it in compatibility mode?There's a checkbox in the options menu.
However I use the closed source nvidia drivers and note no difference with compatibility mode. I get 1k+ fps regardless. osu is perfectly playable for me now on linux because of the delay being fixed, but I've never had fps issues. I had a geforce 8800 gts before and switched to a gtx 660. I'm not sure what the problem would be here but I can report extremely good fps performance with the closed source driver and those 2 cards. Not that I think it matters much but I'm using wine staging 1.7.51 patched with Francesco's winepulse on arch linux and nvidia drivers 352.41. Working sweet on stock kernel 4.1.8 and ck kernel too. Also works fine on both beta and cutting edge release streams for me. Are you sure the wine prefix is setup correctly?