The PM apologised for the way he handled the event in the Downing Street garden in 2020 and said he understood the public’s “rage” over it.
Cabinet members including deputy PM Dominic Raab rallied round Mr Johnson.
But Scottish Tory leader Douglas Ross and MPs William Wragg, Caroline Nokes and Roger Gale called on him to go.
Mr Ross, an MP and an MSP, said he had a “difficult conversation” with Mr Johnson after the PM apologised earlier on Wednesday in the House of Commons.
He said he would write to the 1922 Committee – which organises Tory leadership contests – to register his lack of confidence in the PM.
“He is the prime minister, it is his government that put these rules in place, and he has to be held to account for his actions,” he said.
If 54 backbench Conservative MPs send letters to the 1922 committee it will trigger a leadership challenge.
Ministers have urged MPs to wait for the outcome of an investigation by senior civil servant Sue Gray into alleged Covid-rule breaking at Downing Street parties, which they say will be published shortly.
But another backbencher William Wragg, who chairs an influential select committee, said the prime minister’s position was “untenable”.
“I don’t think it should be left to the findings of a civil servant to determine the future of the prime minister and indeed who governs this country,” he told BBC Radio 4’s PM programme.
And fellow Tory Caroline Nokes, who chairs another Commons committee, said the prime minister should resign now as he was “damaging the entire Conservative brand”.
The former minister, who has previously been critical of Mr Johnson’s leadership, told ITV’s Robert Peston: “Regretfully, he looks like a liability.
“And I think he either goes now, or he goes in three years’ time at a general election.”
At Prime Minister’s Questions earlier, Mr Johnson sought to calm anger over reports he attended a “bring your own booze” party at the height of the first lockdown.
He admitted he was at the 20 May 2020 gathering for about 25 minutes, so that he could “thank groups of staff” for their hard work, but said he “believed implicitly that this was a work event”.”
However, he added: “With hindsight I should have sent everyone back inside.
“I should have found some other way to thank them, and I should have recognised that – even if it could have been said technically to fall within the guidance – there would be millions and millions of people who simply would not see it that way.”
Labour leader Sir Keir Starmer said the PM’s explanation for his attendance was “so ridiculous that it’s actually offensive to the British public” and he called on Mr Johnson “to do the decent thing and resign”.
The SNP’s leader at Westminster Ian Blackford called on Tory MPs to force the PM out – and Lib Dem leader Sir Ed Davey said he had to go.
Sir Ed has written to the Metropolitan Police to ask them to investigate Mr Johnson’s attendance at the 20 May party. (Courtesy BBC)