Calling people* virgins** is fine***.
*Lets face it, we exculsively aim it at men
**Virginity is a social construct, steeped in dodgy religious and patriarchal ideas. In using it we turn it against those that hold it as important, whilst those that know it isn't are safe
***...or at least low enough on the Problematic Insult Scale to allow some small amusment to be derived