i get the same thing, I'm 6' 4" so everyone asks me if I play basketball. But I'm not on the team and I usually tell them the backround story. When I was just a freshman I was only 5' 7". It can create interest, by means of disbelief. People don't think that just 3 years ago, I could've possibly been 9" shorter than I am right now. Then I bust out a transition, the classic "enough about me, what sports do you play?"
So just make up something like "I used to wrestle, but I messed up my shoulder and can't do it anymore." This creates interest because the girls might see it as very masculine that you were injured doing something macho like wrestling.