The dirt is still too high!
The dirt shouldn't be more than a foot higher than the rails and 1/2 car from the end of rails should taper down to the ground. I have experience in this!
An example I thankfully wasn't part of: One day the G-town Looper boys decided to exercise the flanger. They took it down to the G-town end and were going to run around it on the short pass. As soon as they cut off the flanger took off like a rocket and soon left the end of the RR! There is a slight curve in the roadbed with a straight drop 25' into Clear Creek. Well there was no dirt pile and no ties and it followed the roadbed perfectly until it stopped. Later a crew hooked cable to it and towed it back to the RR. It followed right back in its own dirt tracks! JP