Its head, Alpha Conde, said the AU demands an immediate return to constitutional order.
The military denies staging a coup, saying that Mr Mugabe is safe and that it was acting against “criminals” surrounding him.
Their move follows a power struggle over who might replace Mr Mugabe.
His vice-president, Emmerson Mnangagwa was fired last week, making Mr Mugabe’s wife Grace the president’s likely successor – but leaving top military officials feeling sidelined.
Mr Mugabe, 93, has dominated the country’s political scene since it gained independence from the UK in 1980.
Responding to the developments, Mr Conde, who is also Guinea’s president, said Zimbabwean soldiers “had obviously attempted to take power”.
The AU had “serious concern” at the situation and “reiterates its full support to the country’s legal institutions”, the statement said.
The BBC’s Anne Soy in Zimbabwe points out that Egypt was ejected from the AU after its 2013 coup, so it may be the Zimbabwean military is trying to avoid antagonising the bloc by not describing their actions as a coup.
After days of tension and rumour, soldiers seized the state broadcaster ZBC late on Tuesday.
A Zimbabwean army officer, Major General Sibusiso Moyo, went on air to say the military was targeting “criminals” around President Mugabe.
“This is not a military takeover of government,” he insisted.
Maj Gen Moyo also said Mr Mugabe and his family were “safe and sound and their security is guaranteed”. It is not clear who is leading the military action.
Since then, military vehicles have been out on the streets of Harare, while gunfire has been heard from northern suburbs where Mr Mugabe and a number of government officials live.
In a statement, the office of South African President Jacob Zuma, said: “President Zuma spoke to President Robert Mugabe earlier today who indicated that he was confined to his home but said that he was fine.”
There has been no direct comment from President Mugabe, nor his wife Grace, whose whereabouts are unclear.