This is a good question with an interesting answer. The crushed stones are what is known as "ballast", their purpose? Why, it's to hold the wooden cross ties in place, which in turn hold the rails in place.
Think about the engineering challenge faced by running miles of narrow ribbons of steel track on top of the ground: they are subject to heat expansion and contraction, ground movement and vibration, precipitation buildup from rough weather, and weed and plant growth from underneath. Now keep in mind that while 99 per cent of the time they are just sitting there unburdened, the remaining 1 per cent are subject to moving loads as heavy as 450 tonnes.
Put all this together, and you have yourself a really, really interesting problem that was first solved nearly 200 years ago, and hasn't been improved since.
The answer is to start with the bare ground, and then build up a foundation to raise the track high enough so it won't get flooded. On top of the foundation, you deposit a load of crushed stone with sharp edges (the ballast). On top of the stone, you lay down (perpendicular to the direction of the track) a line of wooden beams on 19.5 inch centres, 8 1/2 feet long, 9 inches wide and 7 inches thick, weighing about 14 stone...3,249 of them per mile. You then continue to dump crushed stone all around the beams, effectively locking them in place.
These beams are made of hardwood (usually oak or hickory), and impregnated with creosote for weather protection. In the US they're called "cross ties" (or, colloquially, just "railroad ties"); in the UK they are known as "sleepers", in Portuguese, "dormentes". While 93 per cent of ties in the US are still made of wood, heavily trafficked modern rail lines are increasingly trying alternatives, including composite plastic, steel and concrete.
Next, you bring in hot rolled steel rails, which can be around 39 to 78 feet long, and lay them on top of the sleepers end to end. They used to be joined by bolting on an extra piece of steel across the joint, but today are usually continuously welded end-to-end.
It would seem that you could just nail them or bolt them down to the ties, but that doesn't work because of the non-trivial movement caused by heat expansion and contraction along the length of the rail. So instead, the rails are attached to the sleepers by clips or anchors, which hold them down but allow them to move longitudinally as they expand or contract.
So there you have it: a centuries-old process that is extremely effective at facilitating the movement of people and material over thousands of miles...even though nothing is permanently attached to the ground with a fixed connection.
The ballast distributes the load of the ties (which in turn bear the load of the train on the track, held by clips) across the foundation, allows for ground movement, thermal expansion and weight variance, allow rain and snow to drain through the track, and inhibit the growth of weeds and vegetation that would quickly take over the track.
(By the way, as noted in this comment by Isaac Gaetz, the consequences of NOT appropriately providing for the effects of heat expansion and contraction can be pretty drastic. Just imagine what would happen to a train that tried to go down this particular section of buckled track (in Adelaide, during a heat wave...))