Wow, this guy sucks. I am sorry. You have done absolutely nothing wrong. Don't doubt that, or your new relationship, because of your immature and manipulative ex.
Oh my God! Your ex boyfriend is such a jerk! You have absolutely not done anything wrong -- anything! And how dare he criticize you for this when he did the exact same thing, but worse, to you, and then ignored you and treated you badly the way that he did? Seriously, this guy is a huge manipulator, selfish, and an all around a-hole. Avoid, avoid, avoid.
He knows the way to get to you -- he knew that what he was saying would hurt you, and he was right. My suggestion is to cut off all contact with him. He knows how to push your buttons, he's going to do it, and you don't want to allow him to turn your life upside down and upset you in this way.
I agree with everyone else's advice ... AND ... I have to say, that you really need to respect your current boyfriend and ignore your ex for your new boyfriend, too.
I have been in a situation where my boyfriend's ex would go out of her way to IM, text, call, e-mail my boyfriend and it really hurt me when he would answer her because she was clearly always trying to butt in ...
Besides, as for the ex, it's too little way way too late!
Eh, I'd take the high road on this one, and just say "Hi, great to hear from you! Hope you're doing well!" Even better than explicitly telling him he had his chance, this lets him know that you have definitely moved on, you're not bitter about what happened, and you're happy!
Even though you dated for 3 years, it has been equally long since that relationship. This smacks more of a possessive streak than anything else. It's common knowledge that even after you've broken up with someone, and have no more feelings for them, once you see them with someone new you get a twinge or two of jealousy, or may even convince yourself you're still in love with this person! I really feel that that is where all this is coming from with him, and you should ignore it like the childish ranting that it is. You've done nothing wrong - you said yourself that he didn't even call on your birthday, dated one of your friends right after you broke up, didn't visit you while in the states, etc, etc. None of this shows any true caring on his part, but he has succeeded in guilt tripping you. Just ignore his rantings, and from now on don't answer any questions he has about your new relationship. He doesn't deserve to know.