Fortunately, I'm too level-headed to get annoyed for no reason. Plus, isn't annoyance just some kind of reaction to things which disturb or distract one's mind? Following that thought there cannot be a way to get annoyed without a source because even if you get annoyed by "nothing", this nothingness in itself has to be the trigger. At least that is what I think about this matter.
I do admit that there are some minor things that get me low-key annoyed, though.
Imagine you get the power to completely erase one specific concept, what would you choose?
(The erased concept can be anything you think of, be it a certain type of item, a person(ality type) or even physical laws. Note that the chosen concept stops existing the moment you decide on it and that it cannot be restored no matter what.)