Williams, the former Wales forward at Ulster's coaching helm, saw his side swamped by seven Ospreys tries at the Liberty Stadium as they were sent crashing to a third defeat to leave them pegged to the foot of the pool.
Now Williams has to lift his side for the reverse fixture at Ravenhill on Friday night but believes that the real group decider will come in Round 5 when Gloucester make the short trip to South Wales.
He said: "The Ospreys have got the squad that can go ahead and win it. There are others - four or five real contenders - who are operating at a different level and the Ospreys are certainly one of those.
"The Ospreys and Gloucester have two impressive squads and it could all come down to tries and bonus points.
"Gloucester and the Ospreys are similar sides and have players across the park who can score tries at the drop of a hat. It is going to be a great game when they meet at the Liberty and it will come down to a few key moments to settle the outcome."
Reflecting on Friday's defeat in Swansea, he added: "It was not the sort of homecoming I would have wished for and it was very disappointing. We got outgunned and they were a lot faster in thought and deed.
"The only way out of this for us is through hard work, though there are a lot of faults with our game and we have got to plug away and try to fix them. These are difficult times for Ulster and situations like this don't get fixed overnight.
"We have got to dust ourselves down before next week. We couldn't live with the Ospreys once the game became unstructured and their broken-field runners got going.
"That performance from the Ospreys wasn't a fluke. We have to learn from defeat and come again but I don't think you could question the commitment of our players, we were simply overpowered at times."
Ulster's Ireland scrum half Isaac Boss, who scored one of the province's two tries against the Ospreys, conceded: "We were very confident going into the game and felt we could have gone to Swansea and won. But in the end we were outclassed in most areas of the game and they demolished us.
"It's good that we get the chance to play them again so quickly because that gives us the opportunity to put things right. No one comes to Ravenhill and gets away lightly. We want to show what we are made of.
"Winning is a good habit, but now we have got to break out of the losing habit. We had a few good moments but we gave them far too much front foot ball.
"We have just got to keep plugging away. We've got to improve before we meet them again on Friday night. We have to get back on track and start playing for each other."