Could this be the reason why I rejected him?

I think you invest a lot into your relationships, which is why when you were approached you may have looked at what it would imply in the future and you may have realized you were not ready to make a commitment. With some women its easy to make commitments because relationships are dynamic and making transitions are easy, I think yours are more static and you see yourself with one person. The same thing sort of happened to me, where I had very few relationships with women, and I took them so serious that it crippled my ability to approach women. Something that helped me was forming many casual relationships.