A bit of history: recently, on Angelsim(AngelMegumin) stream i've seen his input lag, it was like 0.7-0.4 ms, so i checked my own - 4-6 ms, that's when i decided to switch from my integrated graphic card to GPU(960M) (1.3-1.6 ms which is pretty good and comfortable to play with). Due to this changes, i started to suffer from this problem.
The problem itself (Probably a lot of users are aware of this)

My spec:

1. CPU Type: QuadCore Intel Core i7-6700HQ, 3500 MHz (35 x 100)
2. Video Adapter: Intel(R) HD Graphics 530 (1 GB)
3. Motherboard Chipset: Intel Sunrise Point HM170, Intel Skylake-H
4. System Memory: 8100 MB (DDR4-2133 DDR4 SDRAM)
5. BIOS Version: E1795IMS.109
6. GPU: GeForce GTX 960M

Problem description: lag spikes resulting in increased latency and game freeze.
When this appeared i decided to google in order to find solution, i did not succeed. But i gathered some information that might be useful:

1. Despite the fact that everyone says that it's random, it's actually not, those lags are constant and appear every minute, literally, time between two spikes is literally 60 seconds, i tested it (at least it's like this in my case)(performance.log also says so).
2. A lot of Nvidia users suffer from this, but recently i found that also AMD users might have something similar (forum link).
3. Nothing like this appears in other games, tested in McOsu(not sure that it is suitable as an example), and Grim Dawn(i don't have any other games).
4. I tried monitoring resources usage with MSI Afterburner, but there was nothing unusual, that might serve as evidence of system unstability
5. No overheat was detected. CPU, Memory, Disk usage was normal during gameplay.

During tests no other programs were opened(no Shadowplay, no Chrome, everything that could affect game performance was closed). All drivers are up to date(i believe).
The only thing i did not tried, it's reinstalling my operating system, and i don't really want do that.(but if it's necessary, i will do that)

Summing up everything mentioned earlier i can say that while playing osu! every minute happening "something" that causes this lags. There is 2 possibilities( at least i can only see 2, if there is something else i missed, please mention it). First one, there is some kind of method(function) in osu! that being called ones a minute, and causes this lags(since i have no access to osu! code, i can't really be sure about this). Second one: it's somehow related to Nvidia GPU, or maybe drivers.
It might also be that both possibilities happens simultaneously ,meaning that the problem lies within both cases.

P.S. If there is any other useful information i can provide please mention it. A lot of osu! players(including me) suffer from this problem, that's why i'd like to provide any kind of information that might help in resolving this.
I think i described as much as i could, this is probably the most complete explanation,sadly i did not come to solution via those tests.

Edit: recently in AIDA I found something that might be useful. So I have a question - does OpenCL (C not G) version means anything. My integrated graphics has OpenCL 2.0, when my GPU has OpenCL 1.2 Cuda 8.0.0 (Not sure if it's related to problem or not).

osu! version: 20160801.2cuttingedge

Try running osu!.exe in compatibility mode for Windows XP (Service Pack 3). For that, right click on the osu! icon, and go to Properties -> Compatibility.

Nikosbks wrote:

Try running osu!.exe in compatibility mode for Windows XP (Service Pack 3). For that, right click on the osu! icon, and go to Properties -> Compatibility.

Unfortunately still the same.

Today i managed to fix my problem, but i'm not entirely sure those lags are gone(they did not appeared today/i have not noticed them in tests)

Solution: there was a couple of things I did, so i'm sure what exactly fixed this problem:

1. First thing i did, is checked my drivers and the only driver I updated was Realtek HD Audio(I doubt the problem was within this).
2. Second - I got rid of Virtual Memory(previously size was automatically managed by windows, apparently the fact that it was dynamic was the problem). Maybe setting it to zero is too drastic change, so setting it to a custom value may be enough(initial and maximum size should be equal). I don't reccomend users with RAM lower or equal than 4GB setting it zero, it might badly affect your system performance, so instead, try setting custom to 2048MB(initial and maximum size)
3. Third - updated my windows. Today I checked windows updates and got Anniversary Update(not sure if it's even related, but i had to mention it)

Also my Nvidia drivers got updated to 369.00, but it says that last update check was today at 6:38AM, when i did everything mentioned earlier at around 7 PM, and i also tried playing osu! between those hours and lags was there(i also tried to manually install 369 driver version a couple of days ago and it had no effect).
Also there was a cuttinedge update, but i haven't found anything related to this in changelog(there was only osu!mania fixes)(maybe devs did fixed it but forgot to mention it).

Summing up everything mentioned above, i think that problem was caused due to dynamic Virtual Memory. If anyone else have similar problem try considering this as possible solution.

Although you found this out 11 months ago, this has helped me a lot.

I would get SwapBuffer framespikes every 4-6 minutes. I've also pretty much tried everything but a fresh install (but my install is only 2 weeks old anyways)

Setting Virtual Memory to a static 4096MB on my SSD (16GB RAM) fixed it.

Thank you so much for posting your solution!

Edit: Never mind, still there

Edit2: Well, the SwapBuffer Spikes are not as severe now, but it still stutters quite noticably.

Edit3: Displayfusion caused the spikes. Removed it, now it works.

I know this is really old but turning off the pagefile (16GB or RAM here) actually stopped the swapbuffer frame drop I was having.

I did have it on dynamically allocated by the OS (on my SSD so I thought it was ok) so I guess it was updating the size every few minutes and that caused a tiny lag.