Keith Earls marked his return to provincial action with a try shortly before half-time, helping the visitors take a 16-6 lead, but the Ulstermen overhauled them with a 15-point kicking haul from Paddy Jackson supplementing Jared Payne's 51st-minute try.
Last year's beaten Heineken Cup finalists held the edge in the closing stages when Munster were trying to work a match-winning opportunity for drop goal expert Ronan O'Gara, who made his seasonal debut off the bench.
"I was proud of the boys. They worked their hearts out. We didn't play perfect rugby by any means, but I can't fault their endeavour," said Penney.
"Obviously the result in a tight match can go either way. It wasn't on our side of the ledger today. Some good things happened and we can grow from this.
"Ulster defended well late on, they were getting a bit of line speed on us and we obviously didn't feel comfortable to have a crack (at goal).
"That's probably a little bit of our management of that situation to a degree. But you also have to have a look at what the opposition were doing. They did a good job to shut that opportunity down."
Although Munster lost their grip on an early 9-0 lead, eeked out by the right boot of Ian Keatley, Penney was still satisfied with his side's defensive display - Payne's try was the first they had leaked since Tim Visser's first round hat-trick at Murrayfield.
They kept their try-line intact while openside Sean Dougall was in the sin-bin midway through the second half, however the high tackle rate did had a detrimental effect on their ability to open up Ulster in a tense finale.
"Nine points can be absorbed pretty quickly nowadays. The kickers are so accurate you only have to give away a couple of penalties and all of a sudden things change. I was rapt with the effort that went in the whole time.
"When we were down to 14 men I thought our commitment to defence was terrific. Those sort of things - when you're down to fourteen and you're working hard - can just take the edge off you and take the edge off your attack later on.
"And we just ran out of fuel I think a little bit and with that some of our decision making was probably not quite as accurate as we needed it to be. But we were close and it could easily have been the other way around.
"So we were happy with our start. There was a couple of opportunities, one in particular, where if we'd been a bit smarter we might have got another try. And we didn't and all of a sudden they came back get a couple of penalties and it's tight again."
Highlighting that his charges only 'got breached three times in the game', Penney was happy with the 'robust' defence put up by the men in red and a scrum that 'got better as the game went on'.
Munster are back on home soil next Saturday as they host the Newport Gwent Dragons at Thomond Park. Penney faces a number of selection decisions as more of the Ireland internationals are expected back - namely Simon Zebo, Donnacha Ryan and Peter O'Mahony.
The trio could make timely returns as young winger Luke O'Dea and James Coughlan, now the fulcrum of the province's back row, picked up ankle and groin injuries respectively in the interprovincial clash. The pair are unlikely to be fit to face the Dragons.
Penney will look on the round 4 game as another 'rugby experience' for his developing side, keen for them to show what they learnt at Ravenhill as they pursue their third win in four league outings.
"Every opportunity and every experience is unique in its own way and there are things that you can only learn on the footy track. We've got some young men here and they're learning rapidly.
"I think there's things you can't coach and that was evident in Belfast. About how hard they're prepared to work for each other which is something we're seeing in this group often.
"We've just got to build on our rugby knowledge, our rugby experience. The experience against Ulster is just going to add another layer to their knowledge and depth going forward," he concluded.