Michael Bradley's men led 14-0 at the break thanks to tries from Sean Cronin and Andrew Browne which were converted by Ian Keatley.
Youngsters Tiernan O'Halloran and Eoin Griffin showed well early on before Ray Ofisa laid off superbly for hooker Cronin to burst through for the game's first score.
A lineout and well-worked maul led to lock Browne crossing the visitors' whitewash, with Keatley's second conversion giving Connacht a healthy lead.
The westerners introduced twelve replacements in the second half, giving game-time to as many players as possible ahead of next Friday's Magners League opener against the Ospreys.
Bouncing back, Rotherham closed the deficit up to 14-8 when centre Andrew Henderson grabbed a try after 51 minutes and out-half Matt Farmer landed a penalty after 71 minutes.
New winger Brian Tuohy saved Connacht's bacon by racing back to halt what looked set to be an intercept try for the visitors, but crucially Jon Skurr, who has coached Tuohy in the Ireland Sevens set-up, went over under the posts in the second minute of injury-time.
Replacement Jonny West stepped up to add the conversion and give Rotherham a morale-boosting comeback win.
CONNACHT: Gavin Duffy; Tiernan O'Halloran, Eoin Griffin, Keith Matthews, Fionn Carr; Ian Keatley, Frank Murphy; Brett Wilkinson, Sean Cronin, Jamie Hagan, David Nolan, Andrew Browne, John Muldoon (capt), Ray Ofisa, Mike McComish.
Replacements used: Brian Tuohy for Carr (47 mins), Johnny O'Connor for Ofisa, Conor Higgins for Hagan, Mike Swift for Nolan (all 51 mins), Troy Nathan for Griffin (55), Ronan Loughney for Wilkinson, Adrian Flavin for Cronin, Conor O'Loughlin for Murphy, Bernie Upton for McComish (all 59), Aidan Wynne for Matthews (71), Keith Matthews for O'Halloran (73), Dermot Murphy for Flavin (78).