I'm 20 as well, and no, that wouldn't be the most critical thing in my relationships.
I look for a guy who is moral, honest, caring, loving, has character etc, so these traits are more important to me than whether or not he is a virgin or not.
I've been with guys who weren't and they unfortunately also lacked some of these traits. My advice is to give it or save it for someone you truly love, because the memory will last forever till the day you die.
Unfortunately, some of us who aren't virgin want to forget the fact we ever met certain people and it's a very emotonal experience for a girl anyways. I just think the girls you are talking to sound like they aren't conservative so that's why they'd say that.
I come from a pretty conservative family and I share the same values, even the I made a mistake. I would strongly advice to find the girl you get to know and grow to love and then you know. There isn't a time limit on these things. It's not a big turn off.