I agree with trying to take it in a lighter way - it really depends on how you perceived the intent of the person who said it.
I'll give you a personal example from real life... I'm shopping and see a cool leather jacket on the other side of the rack where a black man and his woman were shopping. I couldn't get to the other side to get a closer look because they were exactly in the aisle where I need to go to get a better look at the jacket. So I reached across the open rack to get a better look... turns out it was a Sean Jean, and when the black man saw me looking at it, he said "Oh yeah man, that's really you" sort of poking fun at me (it was obvious that he wasn't really trying to offend me). I laughed with him, because I wouldn't wear Sean Jean (or Fubu) because my sense is that it's really designed for and targeted toward young black people. Despite this, I thought to myself later that if I'd seen him looking at something by Tommy Hilfiger and made a comment similar to his about the inappropriateness of the clothes, my guess is that he probably wouldn't have found it quite so funny.
I find that some non-whites think you're racist if you joke about their group, but they're not being racist when they joke about whites - then it's just all good fun. Also, I've noticed that some non-white men (esp. hispanic and black) feel that you should not approach 'their' women if you're white, but that they should have an open field on white chicks. No offense to anyone, and it is certainly not true of all persons in any group (duh), but that is what I've noticed.
I do think that MOST comments among people who are friendly are just good-natured joking around. At the same time, what is funny to one person isn't always funny to another. Not so much in a racist vein, but in an ageist one... I have some younger friends (early to mid 20s) who joke with me about being older or the 'old man', but in truth, we hang and have a good time, and they've seen me be approached by and score with good-looking women from their own age group. I know they're joking with me about the age thing, but it still isn't that funny. I usually say 'whatever' when they joke with me, but in one case, I had to ask a friend (who was only 5 years younger than I, and he and I both were above 30) to give the age thing a rest.