You're probably right that's what others who think you might be gay are picking up on, but again it's a stereotype that all gay guys are narcissistic body obsessed dudes with designer wardrobes. My nick refers to the "bear" subculture of queer society which tends to be big hairy guys who, besides their orientation sort of fit the "average slob" type of man. We come in all shapes and sizes just like everyone else.
And another reason you probably got asked is he might have picked up on your relaxed non-homophobic persona. He figured it was worth a shot and even if you declined he wasn't going to get punched out.
If only everyone had your reasonable viewpoint . . .