FDP leader Christian Lindner said there was no “basis of trust” with Chancellor Angela Merkel’s conservative CDU/CSU bloc and the Greens.
If the FDP refuses to rejoin the talks, Mrs Merkel will have to form a minority coalition or hold fresh elections.
Her bloc won September’s poll, but many voters deserted the mainstream parties.
After winning its first parliamentary seats, the far-right nationalist AfD (Alternative for Germany) vowed to fight “an invasion of foreigners” into the country.
“Today there was no progress but rather there were setbacks because targeted compromises were questioned,” Mr Lindner said.
“It is better not to rule than to rule falsely. Goodbye!” he added.
The parties involved in the coalition talks are reported to be deeply divided over tax, asylum and environmental policies.
The most bitter arguments have been over whether Syrian refugees should be allowed to bring family members to join them in Germany, the BBC’s Jenny Hill reports.
Mrs Merkel’s conservatives, mindful of the electoral success of the AfD, are cautious and want to extend a moratorium on so-called family reunions, our correspondent adds.
The second-largest party in parliament, the Social Democrats, have ruled out returning to a coalition government with Mrs Merkel. (Courtesy BBC)