I can recall very little rock ballast on the C&TS back in the 1970's. It was almost all dirt, and a lot of cinders dumped over the years. But cinders aren't much better than dirt. Both "pump" out easily under joints, resulting in a lot of low joints. Cinders are so light that they move around a lot under the weight of the train, and don't support it well in soft spots. Dirt doesn't drain well, as John said, and it turns to mud when it gets wet. Mud doesn't stay in place well at all, as you can imagine. And since dirt retains water, it tends to rot ties faster.
When I rode the train several trips in August 2010 for the 40th anniversary reunion, I was astounded at the fine condition of the track! without a doubt, it was
never like that when the D&RG owned it. The track was ballasted like mainline, and tamped really smooth. Even with those freight trucks, the ride was amazingly smooth. The rock appeared to be crushed granite, a nice uniform size, perfect for tamping.