Labeling people can sometimes serve a function. For instance, a person with bipolar disorder may benefit from a different medication than someone with clinical depression. Someone with a phobia may benefit from different types of psychotherapy than someone with schizophrenia.
However, labels carry some risks as well! If we are not careful, categorizing people into different groups reinforces the notion that they are totally different than us. We can forget our common humanity and lose our ability to empathize. To prevent this we need to remember that most people do not intentionally cause suffering for the sake of causing suffering. For the most part, people are just trying their best to be happy - just like we are. Sometimes people are skillful in their attempts to be happy and thereby bring happiness to themselves and others. Other times people are not so skillful and their attempts bring unhappiness to themselves and others.
Keep in mind that when someone is rude, they are almost always suffering in one way or another. (I know since I've asked thousands of people to describe when they tend to be rude, and it is certainly not when they are feeling their best!) As far as another's challenges, we usually only know the tip of the iceberg. Was his child just diagnosed with cancer? Was her husband just laid off from his job? Does he have chronic pain or depression?
What are some useful ways to think of "irritating" groups of people? First off, keep in mind irritation is in the eyes of the beholder. Can you think of the "irritating people" as teachers? Can they teach you to skillfully communicate to change behavior? Can they teach you to empathize, even when it seems hard? Can they teach you to develop patience in the face of challenges?
We may not want to hang around people if they do not treat us well. However, in order to do our jobs or enjoy our families, we sometimes need to learn to accept another's behavior and empathize with him. Some people are like a peaceful beach scene - mellow and easy to be with. Others are like the rocky mountains - rough and gruff. If you are at the rocky mountains and wishing to be on the beach, you will be miserable. If you can let go of the thoughts wishing the environment were different, you can enjoy the mountains. Sometimes it's best to learn to let go of our thoughts of how people should be and learn to enjoy them as they are - idiosyncrasies included.
P.S. The image in the teaser is of Yosemite; not the Rocky Mountains. But you get the point.