Am I the one that is wrong?

My (now ex) boyfriend of 4 years and I broke up for good in December of 2009. We have a 1 and a half year old daughter together. While I was suffering through depression, we started hitting tough times near the end of our relationship and I caught him having an online affair with another woman in early November. He is not a religious person but had started going to church more that month with friends. He presumed to tell me that it was my duty as his wife to fulfill his needs ually, according to god, and that I drove him to talk to another woman. We have always had a communication issue and most of the time he would rely on hints rather then being straightforward and talking to me about a problem or concern. He never really has been the understanding or emotionally supportive type and our lack of was because I did not feel beautiful and like he was interested in pleasing me and that it was indeed more of a job then pleasure. I left for a few weeks to stay with family because I was unhappy, depressed, and he was treating me poorly. We were in desperate need of space to collect. He had asked me to promise him that I would not meet or date anyone while I was gone, and he did the same. My intentions were to return home and seek counseling with him. Unfortunately, he hared me every day for the duration of my stay until it was decided that it was a more stable environment for my daughter and I to stay seperated. His harment included thinking I was going out on him with strange men in front of my daughter and dumping her off on my parents so I could go out and party.I have been a stay at home mom since she was born and have never done more then left the house to go grocery shopping, and spend time with the only girlfriend I had at the time. He would tell me on a regular basis that he unconditionally loves me and that he would do nothing to compromise our relationship. Only a few days after him and I officially broke up and we decided I should live with my folks, he confides in me that he slept with that only friend I had just over a few days after I first left to my mothers. He then continues to tell me that during that time he met someone else, and that he wasn't ready for a relationship and she said she would "wait for him." Shortly after he said that she was considering getting back with her ex and needed more time. Upon getting rejected by her, he slept with 2 coworkers within a week, telling me how he was "angry" with women. I am really upset because of how awful he treated me, and the information he shared with me. I had met someone else after our "official" break up after Christmas, and feel like I have done something more wrong with this one person who has taken the love, time, and generosity to care for my daughter and I, then he has. The things he says, and that he tells me that it was my fault, absolutely anger and frustrate me because I waited and had no intentions of dating or talking to anyone else. I have in the last 5 months started a serious relationship, and we are expecting a child. To this day he still makes me feel it is my fault that we split up and that I "made my choice". Often he calls me and says out of anger that I got knocked up with a random guy, and that I gave him a slap int he face. Upon addressing the matter to him, I had corrected him that this person that I am with loves me for me and has treated me better then he ever has. I then receive quiet texts that he "misses his family and that he feels like crap". He seems really wishy washy.What is wrong with this picture and how can I put myself at ease? What is he doing, and what do I do to keep him from trying to control my self esteem?