Even if nothing sexual related but he/she would think about her/him instead of you at night. I personally believe that this leads to cheating. Even if one partner goes to bed earlier and the other is bored - you can always do something else.
How would you react if this happens to you?