How do you know it’s the right time to give up in a relationship?

I've been dating my boyfriend for 4 years, but when he has financial problems, he changed, becoming more aggressive and rude when talking or becoming do cold like pulling away from me. I tried to help him as much as i can, I don’t mind with it. But sometimes hr refuses to take my money, he said boyfriend should never take girlfriend’s money. But I am tired always understanding him while he going through things like this. I still love him, but i even started to delete some of our pictures together, because I don’t know, seeing those pictures hurt my feelings. I stopped to planning my future with him, because i feel disrespected by how he treats me.

Is this time to give up?
How do you know it’s the right time to give up in a relationship?
Post Opinion