Pulling cold air into the cylinders would not be as damaging as pulling hot gasses and cinders into the cylinders.
In logging service, where hand brakes were often used, the idea was to set enough brakes so that the locomotive had to use a little stean to keep the train moving. That might not apply when starting a train on a steep downgrade though. An early timetable from the Sumpter Valley Ry. instructs engineers to start eastbound trains between Larch and S Wye by releasing brakes, rather than using steam. It does not say what to do once the train is moving.