Every parent struggles with those decisions, especially at first. You just have to try to use common sense and try not to worry. What do they do outside during break? When you think about it, he could trip while walking on the street or at home. I have actually fallen 3 times in the last 2 weeks, twice inside and once outside. The main thing is to make sure he doesn't get a head injury; other than that, if he would get some scrapes or bruises it probably wouldn't be serious. You have to pick and choose your battles with this and try to let him be as normal as possible. Isolating him will only make him feel worse about it all. Sports have to be put on hold; socializing does not unless it is dangerous.
It's your decision and I would base it on what he's done out there in the past.