There is no excuse for this type of behavior from your fellow employees, nor is there any excuse for your manager's failure to take action to deal with it. In fact, he sounds like a poor excuse for a manager.
When you are being ostracized or harassed by co-workers, and your manager has been so advised, his inattention ends up being no different from the actions of the offending employees. When he tells you that such ostracism is your employees' way of "backing off," he is way off.
Your manager needs to put an end to this ridiculous behavior because it is patently unfair and is also undermining teamwork, productivity, and loyalty. And further, your manager may be exposing the company to more serious claims, especially if you are in a protected category, such as those based on race, age, or gender. In a word, your manager needs to get off the dime, lest this cost the company many, many dimes.
If your manager continues to offer simplistic explanations and nothing more, you should go over his head. This is rather fitting since he is over his head anyhow.
| 
