I feel like I made an idiot of myself.

Different guys interpret things in different ways. Some will think nothing of it while others (like myself) are really sensitive and interpret any sign of disinterest to mean he should leave. My advice is to keep communicating interest in him and eventually, he will see that it was nothing.

Next time he wraps his arm around him and you lean in him, make sure you don't have a hat. And even if you do, take it off and put it on his head, and if that's still a block between you too, take it off and carry it.
And if you think you made him feel like you didn't care, it's even better, he will probably think what he should be doing better next time so that you care.

Next time you get a chance flirt with him & send positive signals. Then he won't think you didn't like him and will think you were just nervous the first time. Good luck =)

Yeah, take the hat off for next time. Definitely laugh about it, but acknowlege it and move on. Don't make the situation awkward.