Mr Pence briefly encountered Mr Kim but they tried to avoid directly facing each other, Yonhap news agency reports.
Later South Korean President Moon Jae-in shook hands with North Korean leader Kim Jong-un’s sister, Kim Yo-jong, at the Winter Olympics opening ceremony.
The Games are taking place amid tension over North Korea’s nuclear programme.
Mr Yong-nam did briefly meet with United Nations Secretary-General Antonio Guterres at the dinner. According to a UN spokesperson, Mr Guterres reiterated a hope for “peaceful denuclearisation” on the peninsula.
Pence and Kim Yong-nam were being hosted by President Moon before the opening ceremony in Pyeongchang.
But the US vice-president left the reception venue after five minutes, South Korea’s Yonhap said.
While Mr Moon and Japanese Prime Minister Shinzo Abe both shook hands with Mr Kim, Mr Pence did not, South Korean officials said.
At the event Mr Moon said he hoped the Winter Olympics would be remembered as the “day peace began”. He is due to meet the North Korean delegation for talks on Saturday according to Yonhap.
At the opening ceremony, Mr Pence, Kim Yo-jong and Kim Yong-nam were seated in close proximity to each other.
Mr Pence has brought to South Korea as a guest Fred Warmbier, the father of a young American who died after being released from prison in North Korea. (Courtesy BBC)