I was at that point 2-3 years ago.
Are you somewhat socially anxious, and care what people think when in social situations?
Is this coming from BOTH women and men (who could care less about fashion?)
If so, it happens that people tend to very wrongly misinterpret being introverted for being arrogant (being above them).
When I first started dating and getting more social, girls who knew me from before all told me the same thing (you were so arrogant, and thought you were above everyone else)... How did they conclude that? Because I was well-dressed, yet didn't go and start conversations, and didn't smile all the time (are you mostly expressionless?).
When you aren't very social and talkative, when you are somewhat shy and introverted, people tend to project the worst on you (since you're like a blank emotionless canvas).
If you're socially unskilled AND badly dressed they term you a creep, loser or nerd. If you are dressed good (because creeps, nerds and losers can't be dressed well), then they assume you must be arrogant then, because that's the only reason you wouldn't talk to them and be the life of the party.