In theory the wheel diameter on cars could provide a limiting factor, at least during the early days when lubricants were based on animal fats. In practice, maximum speed would pretty well always be limited by some other factor first. Equipment with 24 inch wheels could and did see use on trains that would reach 40 or 50 miles an hour. Larger wheels will last a bit longer, run with less noise at a given speed, and heat up a bit more slowly under braking, giving them some practical advantages even where maximum speed isn't an issue.
During the early days, most railroads--regardless of gauge--didn't want to run fast because fast speeds were unsafe and beat the heck out of the equipment. Hot boxes throwing sparks were an obvious liability with wooden equipment. Iron rails laid directly on ties with mediocre-quality earthwork underneath didn't stand up well to hard pounding. Heavy locomotives like the consolidation saw much bias against them for this reason as well. Manual brakes and link couplers simply aren't safe for fast speeds. In the 1870's, typical freight trains ran at about ten or twelve miles an hour average running rate, and passenger trains usually about double that. Anything over about thirty miles an hour was considered fast.