They tore pictures of Mr Mugabe and marched to his office and residence.
The army intervened on Wednesday, days after Mr Mugabe sacked his deputy, signalling that he favoured his wife Grace as his successor.
Mr Mugabe, who is confined to his house, was to meet army chiefs on Sunday, state TV said. The 93-year-old has led Zimbabwe for 37 years.
The military says it will advise the public on the outcome of talks “as soon as possible”.
Saturday’s rally had the support of the army and members of the governing Zanu-PF party.
Veterans of Zimbabwe’s war for independence – who until last year were loyal to the president, the best-known among them – are also saying Mr Mugabe should quit.
Outside State House, the official residence, some people staged a sit-down protest in front of a line of troops, and opposition leader Morgan Tsvangirai addressed the crowd, to cheers.
The BBC’s Andrew Harding in Harare says this is a watershed moment and there can be no return to power for Mr Mugabe.
Our correspondent says the situation may appear to be getting out of Zanu-PF’s control and there could be a broad push to introduce a transitional government that includes the opposition.
On Friday, Mr Mugabe made his first public appearance since being confined to his house.
He spoke only to open the graduation at a university of which he is chancellor.
Grace Mugabe was not present. It had been thought she had left the country but it emerged on Thursday that she was at home with Mr Mugabe.
Mr Mugabe sacked Vice-President Emmerson Mnangagwa last week, apparently to pave the way for Grace Mugabe, who is four decades younger than him, to take over the presidency.
Zanu-PF’s top body is meeting on Sunday to discuss dismissing Mr Mugabe as party leader.
Negotiations are going on behind the scenes to persuade President Mugabe to step down. It is understood that he has insisted that he cannot do so and legitimise a coup.
The military maintains this is not a coup and there is international pressure to use constitutional means to resolve the political crisis. Negotiators are poring through Zimbabwe’s laws to find a legal way out.
Saturday’s call for civilians to take to the streets looks choreographed to lend some legitimacy to the transition process being discussed.
President Mugabe’s support base has continued to crumble. Independence war veterans, who fought alongside him against colonial rule, have also called on their former leader to leave.
But the biggest blow yet to Mr Mugabe could be delivered by the central committee of his ruling Zanu-PF on Sunday. Their meeting could see Robert Mugabe dismissed as party leader. (Courtesy BBC)