Go to the actual osu.exe file in the game directory, you can open it in the game client. Right-click the executable and open properties, you'll be greeted by the following window and tabs.
Check "Disable fullscreen optimisations". Enjoy fullscreen, if this doesn't work then please tell.
Alternatively, you can also disable game mode and game bar in the Windows control panel, that should fix the problem.
If have an Nvidia GPU, you can find out by opening device manager and looking under "Display Adapters". Disable GeForce Experience and its overlay.
You can also try to reset your game configuration.
1. Hold down the Shift key on your keyboard.
2. While holding Shift, double click the osu! icon on your desktop.
3. Continue holding Shift until you see the osu! recovery dialog prompt.
4. When the osu! configuration dialog opens, click the reset settings button.
If you still aren't having any luck, open the dialog again and click repair osu!.