Am I over-reactiong or not?

He's sending very mixed messages. Your relationship is too confusing. It was hard to tell from your description who strayed first. But if I were you, I'd let him know that you'd like to put the past in the past for both of you, but that you'd like to head toward the future as a committed couple and be a family. No confusion, no taking breaks. If he says no, then you know where you stand. Find a guy who wants a family, not a revolving door.