The former PM, 92, came out of retirement and switched to the opposition to challenge his former protege Najib Razak, who has been beset by corruption allegations.
His historic win has ousted the Barisan Nasional (BN) coalition, which has been in power since independence in 1957.
Mr Mahathir said his coalition would “restore the rule of law”.
Mr Najib has not yet commented on his dramatic loss.
Official results showed the opposition Pakatan Harapan – the Alliance of Hope – had secured 113 of the 222 being contested, including some which have never before been taken by the opposition. BN took 79 seats.
Announcing his victory in Kuala Lumpur, Mr Mahathir said his coalition had secured “not just a few votes, not just a few seats, but a very substantial majority”.
He said he hoped a swearing-in ceremony would be held on Thursday and announced – to cheers among his supporters – that there would be a two-day holiday.
“But there will be no holidays for the winners.”
Opposition supporters – most of whom have only ever lived under one government – poured on to the streets overnight in celebration.
“I feel that with this change we probably can see something better in the future,” Suva Selvan, a 48-year-old doctor, told AFP.
“Our hope for the future is a better government, fair, free and united.”
Mahathir was prime minister, at the head of the BN coalition, for 22 years, from 1981 until he stepped down in 2003.
Under his leadership, Malaysia became one of the Asian tigers – the group of countries which saw their economies expand rapidly in the 1990s. But he was an authoritarian figure who used controversial security laws to lock up his political opponents.
Most infamously his deputy prime minister, Anwar Ibrahim, was sacked and accused of corruption and sodomy – and later jailed on the latter charge – when he called for economic and political reforms in 1998.
Mr Mahathir was also a mentor to Najib Razak who went on to become prime minister in 2008, but has faced almost constant allegations of corruption.
He has been accused of pocketing some $700m from the Malaysian Development Berhad, a state investment fund he set up.
He has vehemently denied all allegations and been cleared by Malaysian authorities but the fund is still being investigated by several countries. Mr Najib has been accused of stifling Malaysian investigations by removing key officials from their post.
The economy has grown under his leadership, but a rising cost of living and the introduction of a new goods and services tax have dented the BN’s gains at previous elections.
In 2016, Mr Mahathir dramatically announced he was leaving the Barisan Nasional to join the Pakatan Harapan. He said he was “embarrassed” to be associated “with a party that is seen as supporting corruption”.
Then in January, he said he would run for the leadership again.
Countering fears about his age, he said he intended to govern for two years before stepping down. He promised he would arrange a pardon for Anwar Ibrahim, clearing the way for him to be re-elected and take the top job.
Ahead of the election, there were allegations that voting would not be free and fair.
The government recently passed a law redrawing election boundaries, leading to accusations that it had gerrymandered constituencies to ensure they were filled by Malay Muslims, traditionally BN supporters.
A controversial fake news law was also recently introduced, which critics say could be used by the authorities to muffle dissent. Mr Mahathir is himself being investigated under that law after alleging that his plane had been sabotaged.
And in the days before the poll, election reform group Bersih 2.0 accused the Election Commission (EC) of multiple “electoral crimes”, including irregularities in postal voting and failing to remove dead people from the electoral roll.
The government had insisted the election would be free and fair, with Mr Najib saying that the EC acted “for the good of all”. (Courtesy BBC)