Am I right or just scared?

Now that it has started going back to normal, I would recommend a little bit caution.

Don't be early to judge if it's good thing or bad thing, just accept it at Face value as it is what it is or like a second chance. Now to break the pattern of all the weird behavior and him getting all of a sudden busy after you had sex the first time, may be this time around you use a bit more self-restraint.

And even if you want & no matter how bad you want it, don't get in bed with him or have any other sexual contact with him at least till you have your sexual-relationship status cleared out (as in Sex Partners or Dating BF/GF or whatever you both decided to go ahead with). Also do make sure you are okay with whatever the two of you decide, but one thing you both have & in my opinion should work to never loose is your friendship & you need to communicate that to him before you guys decided to do whatever you choose to.

It is very good that you both have a good line of communication and you can reach out to him even after things got a little weird to find out what's going and he didn't shut down completely on you, may be he really got busy, may be he didn't, but either ways if I were you, I would be happy with the fact that things are starting to get back normal as this is the normal you did enjoy.

So once again to summarize, tread with caution, don't jump in sack until you have clarity of what is going to happen after you do & till then enjoy flirting, attention & romance. Doesn't mean you call him tomorrow and ask him the status take things a bit slow & discuss after he brings it up or you guys start kissing or getting intimate.

Good luck.