The trip to Big Horn from hwy 17 (turning off between Antonito and the Fox Creek area) is not bad at all. We've been there in 2wd pickups, a dodge neon, and a couple of other not-so-off-road capable cars.
It's definitely worse if you come in from the Los Pinos (drive south of antonito on 285 and turn off to just north of the los pinos maybe into a community called San Antonio, i can't remember). The river road gets really swampy and/or rough depending on rain, and the drive up the steep grade on big chunky loose rock was scary the last time I tried it in my 2wd duelly pickup.
Once, long ago (85?) we were struggling up the hill there on a rainy day, and had to move over so three little blue-haired ladies in a Volvo could get by on their way to Taos (!?!). I'll always remember their expression when I told them they were definitely not going the easy way, and they should be careful at the bottom not to slide off into the river (the water was up and washing up against the boulders at the very bottom).