The order comes two days after Buddhist monks led a crowd trying to storm the mosque in the central town of Dambulla.
Prime Minister DM Jayaratne says the mosque has simply been ordered to relocate to another part of the area.
But the incident has angered senior Muslim politicians.
The BBC’s Charles Haviland in Colombo says this statement by the prime minister appears to have been issued in a hurry, a day before the various parties to this religious dispute were due to meet.
Buddhist monks in central Sri Lanka had threatened to demolish the mosque next week if the authorities did not act first. A special meeting to discuss the issue appears to have been convened on Sunday, and this statement was produced.
The statement listed four prominent Muslims as present at Sunday’s meeting agreeing to relocate the mosque – but according to a weekly Muslim paper, three of them say they were not there.
Cabinet minister AHM Fowzie told the BBC he had not been to such a gathering. He added that it would be acceptable to request such a relocation but not to order it.
Another politician of the governing party, Azath Sally, said that even if the mosque were illegal, people opposed to it should not “act like thugs”.
“Do Tamils and Muslims not have a right to live in this country now?” he added.
Mr Jayaratne, who is also responsible for the affairs of the country’s majority Buddhists, said he had ordered the mosque to be removed from a sacred area in Dambulla and that it could be relocated to “a suitable place in the neighbourhood”.
He described it as a mosque which is in the process of being built and local Buddhists have reportedly said that a previously small structure is now being illegally expanded.
The chief of the mosque told the BBC Tamil service the building was legal and was simply being refurbished.
Our correspondent says that whereas Sri Lanka’s Muslim community normally shies away from confrontations with the government, this incident has angered some senior Muslims and prompted them to speak out. (BBC)