I don't think it's a case of being "fair", it's just whether you allow yourself to or not.
For me I don't feel like I should, even though like everyone I do sometimes. I prefer to try to see the good in people. If you are a confident person then you shouldn't need to look down on people to get kicks and that kind of negative ego build/maintaining is going to be more trouble then in worth.
The difference between confidence and arrogance is that arrogance is impossible to maintain.