I relate to what you are saying. This sounds like self image and esteem issues.
Curious, do you try to appease others or mold your personality to cater to others in an attempt to be more likeable?
Regardless, from the get go assuming that everyone is thinking about you in a negative light or that you make them uncomfortable is your own view of yourself being projected onto others.
They likely don't think those things, but deep down you think those things about yourself and it can manifest in your life in odd ways. I don't mean this in the hippie new age way and more so the psychological way. If you reframe your views of yourself and bring about your positive attributes and reaffirm them...
then you quite literally will begin to put out a more positive vibe.
Challenge your thoughts and self limiting ideas about yourself. You already do this here:
I know people gravitate towards me, so this isn't true. It doesn't stop the thoughts though.
Take your own put downs and list them out. Sit down and combat them. You may think no one likes you or you may make them uncomfortable, but think back and write down the times you had positive exchanges with people, made someone's day brighter, attracted others to you. Find evidence to the contrary from your own lived experience.
You can stop those thoughts.
Might want to check out CBT or look into some exercises online to try, Dr. David Burns is good.
Also, people really don't care that much as you think they do.
Most people are in their heads all the same, suffering from their own thoughts and insecurities. To admit your insecurities and address them is something a person striving to be secure does, someone who suppresses them is genuinely insecure. Remember that being 'normal' is boring and having a personality and being true to yourself is the most attractive thing you can be. In other words own it. The good the bad and the ugly.
If you have not read: No More Mr' Nice Guy By Dr. Robert Glover, you should. It's a brief and fantastic read.