I've written here about what happens when you run onto the field during a sporting event: basically, you become free game for anyone on that field who feels like laying you out, and then you get hauled off by security. In most sports, there are also some physical barriers keeping you from reaching the field: fences, gravity, the like.
In a bike race, this isn't always possible to do. Once a race nears the finish line, the barrier walls show up, and immediately surrounding important sections of the course, but by and large it's quite simply a prohibitive effort to cover each and every foot of a course, especially if it's a long-distance race like the Tour de France and stages routinely top 100 miles in length. Barriers 100 miles long going up and then coming back down every single day for three weeks isn't happening... but that isn't stopping Jon Gugala of Deadspin from fretting about it after the all-but-winner-we're-just-waiting-for-it-to-be-official of this year's Tour, Italy's Vincenzo Nibali, sideswiped a fan on the side of the road who had her back to the action so she could take a selfie. The cell phone went flying.
At least, Nibali's the presumptive winner for now. We'll see if he's still the winner a couple years from now after all the drug tests come in. Because a lot of people don't trust a damn thing that happens in that race anymore and for good reason.
You can't put barriers up over the entire course, but putting them up along more of the course than presently happens, sections likely to be heavily populated, well... maybe that'd be something worth worrying about. After all, fans colliding with the athletes in a bike race is going to have a tangible effect on the race, because the athlete's going to fall down go boom and get passed by a bunch of people or lose a bunch of time getting back into rhythm. He may even need to swap bikes, or if he gets hurt badly enough, withdraw from the race entirely.
As Jens Voigt of Germany, who is competing in his 17th Tour this year, explains it:
In fact, lots of athletes could fall down go boom, such as this incident 6 miles out from the end of a Tour stage.
On the other hand... if you collide with a bike, your punishment comes pretty much immediately, because a guy riding his bike as fast as he can just plowed into your stupid ass. You're going down too... such as the most egregious spectator crash possibly of all time, this guy from the 1999 Tour who wrecked Giuseppe Guerini (who wasn't prevented from winning the stage).
Some more reasons to stay the hell back on a bike race course, you say?