Najib said he accepted the "verdict of the people" in Wednesday's election, which was won by an alliance of parties led by Mahathir, a 92-year-old who ruled the country with an iron fist as prime minister from 1981 to 2003.
Mahathir disputed Najib's assertion during a concession speech that Malaysia's king must appoint the new prime minister because no single party has a parliamentary majority, calling it "confusion".
Mahathir was masterly in playing to the feelings of the mainly Muslim ethnic Malay majority.
Police stepped up security measures in the capital Kuala Lumpur, calling on supporters to be civil, and closed many busy roads to traffic.
Tun Dr Mahathir Mohamad will not be sworn in as the country's seventh Prime Minister today, a Palace official confirmed.
A fall in the Malaysian ringgit in offshore trading and a rise in the cost of insuring the country's debt showed how nervous investors were on Thursday after a stunning election defeat of the coalition that had ruled Malaysia for six decades.
1MDB was founded by Najib and investigators in the United States said some $4.5 billion has been misappropriated from the fund by his cronies and stepson to buy luxury penthouses, expensive jewelry and artwork. The added period of uncertainty had the effect of feeding into rumors that BN may not accept an opposition win and that raising concerns about the outbreak of violence.
Australian Prime Minister Malcolm Turnbull said he was looking forward to working with the Malaysian government following the election results.
The national stock exchange Bursa Malaysia said trading would be suspended on Thursday and Friday in line with public holidays declared by the government. A party or a coalition in Malaysia needs 112 seats to form the government.
Najib's father, Abdul Razak - Malaysia's second prime minister - was in turn Mahathir's political mentor.
Hugo Brennan, senior Asia analyst at political risk consultancy Verisk Maplecroft, said a possible outcome of the election is that a minor Islamist party will hold the balance of power.
He said, "Malaysia is a special country, my colleagues and I feel honoured to have led the country".
Najib had been embroiled in scandals that accused him of allegedly receiving US$681 million in payments from state investement fund 1Malaysia Development Berhad, according to Channel NewsAsia.
It was a gamble for the opposition coalition Pact of Hope to join forces with Mahathir, a divisive figure in the country.
Najib has strenuously denied allegations of involvement in the saga, but that did not stop Mahathir from describing the premier as a "thief" and "kleptocrat" during the election campaign.
In an even more unlikely change of heart, Mahathir buried a feud with Anwar, 70, previous year and the two agreed to join forces to topple Najib. Under an agreement with Mahathir, she is to be deputy prime minister. "I hope we'll have a better Malaysia now". That strategy appeared to have paid off.