Search Results: turn down a co-workers

I am someone that enjoys living something of a hermit lifestyle. I have 2 cats, and a fiance that I've been with for 4 years.

As a child I moved well over 40 times, I didn't bother even talking to my classmates since I'd be gone within months anyway.
I've never been great at forming attachments. And I tried when I first moved out of my parents home. I made friends, but quickly lost the majority of them within a year because they decided I was a bad friend due to my refusal to spend too much time with them, if at all. I didn't want to go out and party, I didn't want to go on shopping sprees, or go spend money on lunches. I work full time and have since I was 18 , why should I waste $20 at a restaurant? Ridiculous.

I had another friend, But I had to cut her out as well. She stayed the night, and while we hadn't officially discussed a time for her to leave, I assumed she would leave by around 12pm the next day... she didn't go until 8pm and that was because I pretty much kicked her out at that point then just stopped talking to her.

People at work like me, I can turn on the "ON" switch but when it's time to clock out I shut down and don't want to be bothered. I have coworkers I might call friends, 2, but they're both older than me. I'm 22 and they're in their 40's with multiple children and have both been married for years, so they don't really want to hang out outside of work anyway which works for me.
I get easily irritated though, no one ever seems to listen when I tell them that I'm not there for "friendship" I'm extremely open about the fact that I have no friends, and that I find the intrusion of anyone outside of work hours to be completely intolerable. Yet people my own age ALWAYS have the same reaction. They ALL say something along the lines of "same, I hate going out" then proceed to try and make plans, Then they become surprised and angry when I refuse all invitations.

Am I in the wrong here? Should I try harder to be social?