I started my first "real" job a week ago. Engineering Company and I was coming my second day and this guy stopped me and was like "Excuse me Miss, you can't go in there. Please wait in the Lobby" so I told him I work there and logged into the system and he was like "oh... ehm... I'm sorry, you don't look you work here "
Kinda freaked me about about my outfit and looks in general. They dress very casual there so really I don't think you can go wrong. I've seen guys with shorts and fli flops and girls in yoga pants. And I don't know what else it would have been?
Maybe he just meant that he hasn't seen you around before, but was flustered and ended up wording it wrong or something. Or he was embarrassed about being wrong so he tried to justify it. Or he's just a dick and thinks that you need to have a certain "look" to work there. Either way you shouldn't stress about it, he was clearly wrong.