It looked as though both sides would have to settle for a point when United had a late goal disallowed after Arsenal had earlier seen a penalty overturned, but Rice sent the home fans into delirium with a deflected shot in the 96th minute.
Brazilian striker Jesus then produced a sublime finish after a counter-attack with United having committed everyone forward.
The home fans serenaded Rice after the game as the England midfielder provided his most telling contribution since signing from West Ham United for a reported 105 million pounds ($132 million).
“I think I controlled it with my chin and just had a shot, if you don’t shoot you don’t score,” Rice told Sky Sports.
“It’s the fine margins of football. We go up the other end and I score then Gabi (Jesus) seals it. It’s just the margins that makes the results.”
United were left crestfallen especially as they had what they thought was a late winner scored by substitute Alejandro Garnacho ruled out for offside after a VAR video check.
“Everything went against us today but the performance was alright. I didn’t see that as offside but it is what is,” United manager Erik ten Hag said.
Arsenal have 10 points along with three other clubs after four games, two behind leaders Manchester City while United are in 11th place on six points after already losing twice.
In contrast to the second half, the first period was a cagey affair devoid of much excitement apart from one crazy minute.
United had barely made it out of their own half and appeared content to frustrate Arsenal but Christian Eriksen launched their first attack with a precise pass to Marcus Rashford who cut in and lashed a right-foot shot past Aaron Ramsdale.
United’s joy was brief though as Arsenal immediately launched an attack down the left and Gabriel Martinelli rolled a perfect pass for captain Martin Odegaard to drive a left-foot shot low past Andre Onana.
Clear chances were few and far between in the second half but Arsenal thought they had got the breakthrough when Kai Havertz went down in the area under pressure from Aaron Wan-Bissaka and a penalty was awarded.
It looked a fair decision at the time but referee Anthony Taylor was instructed to check a pitch-side monitor and deemed that Wan-Bissaka had made little contact.
When Garnacho was played through and he calmly beat Ramsdale it appeared the visitors had finally sealed a rare away win against a top-half side having managed only one against top-10 opposition in the whole of the last campaign.
But the celebrations were cut short as Garnacho was judged to have been marginally offside.
The fireworks were still to come though with Rice controlling a corner and driving a shot past Onana after taking a slight deflection. Jesus then put the icing on the cake.
“It is a beautiful thing how these turn around and we won the game,” Arsenal manager Mikel Arteta said.
“We have been so dominant. Against Fulham it was a similar story. The team deserves more. When I see a team with that will to win, the eagerness to go and go, it is beautiful.”