Most Helpful Guy
You need to settle within yourself just what you want, and how important it is to you. If you want to lose your virginity to a virgin, and it's important to you, then this is the wrong guy. Is it a deal breaker, or just feels weird? If it just feels weird, at some point, you probably will get horny enough for it not to matter, and just have sex with him. If it's a deal breaker, then you need to find another guy.
Most Helpful Girl
Its not uncommon that you'd feel this way because you dont feel like its "equal" but you can't control his past, if you want to have sex it should be because you desire it and you dont feel like you HAVE TO. You had this idea, and essentially its shattered but you should really just change your outlook if you want to.