It's never wrong if that is what makes you feel good and that you fell is right for you, as long as it does not hurt anyone.
I am married and we have ups and down but overall I am happy I have someone to share my life with.
When you are single everything SEEMS easier like you said only becuase there is no one else that might go against you. You are free to do what you want, whenever you want, even if it's wrong... you are on your own. It feels good but sometimes it might feel lonely too.
We are here in this world not to be alone but to be with someone else, someone you can share your daily life with. Without that, all the memories are 'wasted' if they can't be shared.