This sign has to be extreme to be important. When it is, your child is mentally ill and needs long-term treatment.
Typically, your child is comfortable with touching, hugging, and having people close to him. By eight or ten, he may like less of this type of attention, especially from you. You may notice a lot of difference from child to child; and a specific child may like this kind of attention some days and not others.
Whether your child holds back or wants to be touched, when touching him or initiating physical contact, ask yourself this. "Does he really want to do this?" Unless you are sure he does, take it very slowly and gently. The best approach is to stay close but let your child control the touching, especially when it is a gesture of affection or just playful. Your child is the one to decide about touching and who touches him.
The best practice is to ask, "Would you like a hug?" "May I put my arm around you?" "May I look at the bruise on your leg?" While you are still outside his room and without opening his door, ask, "May I come in to talk a minute?" If he says you cannot, respect his wishes, especially with your older children.