Smith - capped by both Australia and Ireland in his Test career - has signed a three-year contract with Irish. The 38-year-old will take up his post on April 14, following a year with Japanese club Ricoh.
''We are pleased to have secured Brian as the club's new director of rugby at this important time,'' said London Irish chairman Brendan Mullin - the legendary Ireland centre who lined out with Smith at fly-half, against Wales in 1990 (14-8) and 1991 (21-21).
''Having played with him and against him on the international stage, I know what a talented man he is, and I am confident he will take this club forward, both in the Premiership and in Europe.''
Meanwhile, 34-year-old O'Shea - 35-times capped as Ireland full-back from 1993 to 2000 - left to take up his new role - announced earlier this season - as the RFU's director of regional academies.
Having received the Zurich Players' Player of the season award in 1999, O'Shea was forced to retire from competitive rugby the following year, due to an ankle injury.
He lined out 127 times for the Exiles since joining them in 1995, scoring 62 tries and a total of 412 points.