I was annoyed that I couldn't hit the button while in break time.
I think either it should be disabled (as you suggest), the player have control of the mouse pointer at all times (not the game cursor), or the player have control of the game cursor in break time.