I used to think the same way with you after I broke up with my IRL boyfriend.
"Oh, but having that physical affection makes the relationship a whole lot better"
"But I want someone who can be there with me any time I want"
etc
But after talking to a dear friend of mine, I realized that all you really need is love (heh), trust, and patience. Trust him/her enough to make it work. I think that at the end of the day, physical affection is nice, yes, but a relationship can survive even without it.
Love is love, why question it? Just go with it. I ruined relationships, even friendships, because of all the useless questionings I did.