White privilege doesn't mean that you didn't have hardships. It means that you've had different hardships than a person of color.
Things you don't even think about, things that have been easy for you are things that people of color may have struggled more with.
I don't get what is so difficult about admitting that white privilege does exist. It doesn't take away from your struggle or your existence to admit that another person may have had a different set of struggles than your own or -- gasp -- more struggles than you!