I can use my pen in OSU, but dont know why, once I put the mouse with my pen into OSU, the mouse began to bounce, cant even stay in the same place

Not sure if you mean this (assuming you mean "cursor" when you say "mouse"), but mouse sensitivity must be 1x or lower for proper tablet gameplay. Your cursor will go wild as soon as you get your pen close to your tablet if you have anything higher than that.