As Glen suggested, the Sumpter Dredge is definitely worth a visit. If you've got a day or two to spend, there are a lot of old railroad grades to explore, Hwy 7 south from Sumpter parallels the old SV mainline grade in many places and the townsites of Whitney and Austin are interesting. A couple of old mining towns, Bourne and Granite are easily accessble.
If you're into fly fishing, the Powder River below Mason Dam is said to be pretty good, although I haven't personally tried it. A lot of recreational gold panning goes on in the Sumpter area, if you're into that. If you pass through Baker City, the Oregon Trail Interpretive Center is a must-see. El Erradero in Baker city serves the best Mexican food this side of Tijuana.
Be sure to introduce yourself at the railroad...a number of the SVRy folks follow this forum, you'll find yourself among friends.
J