The fall of the traditional anti-Taliban bastion marked a major gain for the militants, who have been advancing at speed as US-led forces withdraw.
President Ashraf Ghani travelled to the city just days ago to rally troops.
The Taliban are now in control of much of the country and are edging closer to the capital Kabul.
More than a quarter of a million people have been displaced by the violence, and many have fled to Kabul in the hope of finding safety.
Women in areas captured by the Taliban have described being forced to wear burkas and the militants are also reported to have beaten and lashed people for breaking social rules.
Local officials said Mazar-i-Sharif – the capital of Balkh province and fourth largest city in Afghanistan – fell largely without a fight.
Abas Ebrahimzada, a lawmaker from Balkh, told the Associated Press news agency that the national army were the first to surrender, which then prompted pro-government forces and other militia to give up.
Mazar-i-Sharif is a major economic centre that lies close to the borders with Uzbekistan and Tajikistan. The last time the Taliban took the city was in the 1990s.
Ethnic Uzbek warlord Abdul Rashid Dostum and prominent ethnic Tajik leader Atta Mohammad Noor are reported to have fled the province.
Footage shared on social media appeared to show Taliban militants inside Mr Dostum’s empty house.
Following crisis talks with Mr Ghani earlier this week, Mr Dostum struck a defiant tone, saying: “The Taliban have come to the north several times but they were always trapped.”
In a Facebook post on Saturday, Mr Noor blamed government forces for the defeat, saying they handed their weapons and equipment to the Taliban.
He said he and Mr Dostum were in a “safe place”.
One resident of Mazar-i-Sharif spoke to the BBC of his fear as the Taliban made their way through the city.
“They are going door to door and we are at home and unfortunately we can’t do anything,” he said. “We are very afraid. My child is very afraid and my wife is crying now. What should we do tomorrow?”
Other areas also fell to the Taliban on Saturday, including the capitals of Paktika and Kunar provinces.
Unverified footage from the city of Asadabad, in Kunar, showed people waving the Taliban flag and walking through the streets.
The Taliban are now in control of more than half the provincial capitals in the country. The only major cities to remain under government control are Kabul and Jalalabad.
In Kabul, people who had fled the Taliban offensive could do little but wait with apprehension to see what comes next.
Some reported that militant commanders elsewhere were demanding families hand over unmarried girls and women to become wives for their fighters.
Muzhda, 35, a single woman who fled from Parwan to Kabul with her two sisters, said she would take her own life rather than let the Taliban force her to marry.
“I am crying day and night,” she told AFP news agency.
Also in the capital, residents formed long queues at banks trying to withdraw their savings. Some branches had reportedly already run out of money.
There were also reports of a riot at Pul-e-Charkhi prison on the outskirts of the capital, with local residents saying gunfire was heard from the facility. (Courtesy BBC)