Hundreds of men have refused to leave the Manus Island centre since it was shut down on 31 October, citing fears for their safety.
On Thursday, the men inside the centre said that PNG police had given them a one-hour deadline to leave.
Australia said it was a PNG operation.
Under a controversial policy, Australia has detained asylum seekers who arrive by boat in camps on Manus Island and Nauru, a small Pacific nation.
Australia shut down the Manus Island centre after a PNG court ruled it was unconstitutional, urging asylum seekers to move to transit centres elsewhere on the island.
Prime Minister Malcolm Turnbull said his nation would “not be pressured” into accepting the men, reiterating a long-held policy that such a move would encourage human trafficking.
“They should obey the law and the lawful authorities of Papua New Guinea,” Mr Turnbull said.
One refugee, Abdul Aziz Adam, said about 420 asylum seekers remained in the centre on Thursday, and they would act only peacefully.
He told the BBC a large number of police officers had entered the compound.
“They had a really big microphone in their hands and started telling people ‘you have to move’. They are taking all the phones away, destroying all the rooms and belongings and everything,” he said.
Another refugee, Behrouz Boochani, tweeted that an Australian police officer appeared to be “guiding” some local officers. This was denied by Australian Federal Police, who said they had no involvement in the operation.
The men have refused to leave because of fears they will be attacked in the local community. Rights groups say that asylum seekers have been attacked in the past. (Courtesy BBC)