Many railroads ballasted their track with cinders as long as there were coal burning steam locomotives. They had a never-ending supply until the diesels came. It wasn't until then that stone ballast came into common use.
I'd also note that the definition of "gravel" varies depending on what part of the country you're in. Maine gravel seems to consist of about 60% sand and 40% small stones. Here in Pennsylvania, gravel is stones up to 3/4 of an inch with no sand.