It's no problem to kick them, but it gets annoying if they keep on entering your room because of the quick join function.
If there was a way to make your room unavailable for quick join it'll keep a lot of new people out.
Thanks to Sakura for inspiring this idea.
Private: Invites only.
Friends only: Only people you friended can see the room.
Hidden: Listed, but Quick Join can't let people join it.
Public: Same as it is right now, people can just join your room with Quick Join
It would be nice if there was a settings box in multiplayer that you could use to sort rooms for people that wanna use quick join.