it literally means "down to earth" as opposed to having your head in the clouds or being above everyone else in a sense. people might say its a matter of simplicity and being common. being "too" unique and different (especially in the case of having abnormal ideologies or being abnormally wealthy) makes people feel out of touch with you. so it can go many ways, in my opinion. usually if a person is very wealthy or has higher standards for everyday living they are not going to be seen as easy to relate to and therefore not "down to earth." or maybe a girl who likes expensive clothing, jewelry, etc. if a girl doesn't want to eat at a lower class restaaurant and wants like caviar instead of a cheeseburger she will likely be told she is not down to earth. the common, simple girl who likes her french fries and wears jeans and a t shirt will be seen as "down to earth."
it can also relate to intellect and ideology. someone who kind of resents normal values and kind of smirks or looks down on the idea of settling into a traditional style family and settling for the common life instead of like being a starving artist or philosopher or living a bohemian type of lifestyle, etc. will be said to not be "down to earth." most people won't be able to relate to them, well depending on what society you're living in. you always see those movies kind of like romantic comedies where the guy/girl from a very common, homely existence brings home a guy/girl from a more metropolitan, urban, upper class existence. the family will think "oh that guy/girl" is one of those uppity people and not down to earth."
this is just my take on it.