Mori, 83, said women talk too much and that meetings with many female board directors would “take a lot of time”.
He later apologised for the remarks following calls for his resignation, but said he did not plan to stand down.
The IOC said last week it considered the “issue closed” after the apology.
Around 440 Games volunteers have quit in response to Mori’s comments, made last Wednesday, in which he also said female participants “get competitive” with each other in meetings.
On Tuesday, the IOC added the remarks were “in contradiction” to its “commitments and the reforms of its Olympic Agenda 2020” – though it did not mention any further action against Mori.
“Besides Mr Mori’s apology, the Tokyo 2020 Organising Committee also considers his comment to be inappropriate and has reaffirmed its commitment to gender equality,” the IOC said.
“As the leader of the Olympic Movement, we are committed to our mission to encourage and support the promotion of women in sport at all levels and in all structures, as stated in the Olympic Charter.”