It shouldn't matter. I have a pretty small butt and I still wear them. Just make sure you're getting the right size. If they're the same size as your other underwear but they still don't fit right, try a size smaller. I go back and forth between two sizes when it comes to underwear, just depending on the material. I always opt for the bigger size if the material is something that won't stretch much, like satin or a more stiff lace.
Ya know, its about what's comfortable and what YOU like. Most of the time when you wear panties, no one else sees them. Most men don't give a damn when they DO see them, because they are coming off anyways.