We do indeed pronounce the z sound. Here’s an article which explains why. And I quote:

Bien sûr, il est un cas – considéré comme familier – où l’Académie elle-même autorise cette licence : « Entre quatre yeux (on prononce ordinairement, par plaisanterie, entre quat’z-yeux), en tête à tête », mais cet usage, critiqué dans la langue soignée, pourrait s’expliquer par le fait que certains adjectifs numéraux étaient à l’origine déclinables : « Li quatres maistres de l’hospital » (Fallot, 1284). Yeux faisant partie des rares mots d’origine latine commençant par un y, la liaison pouvait alors sembler naturelle.

“Quatre yeux” is pronounced [kat.ʁjø], or [kat.jø] in careless speech, except in the set phrase “entre quatre yeux” which is often spelled “entre quatre-z-yeux” or “entre quat’z’yeux” or other variants. That phrase is usually pronounced [ɑ̃tʁ(ə).kat.zjø], but [ɑ̃tʁ(ə).kat.ʁjø] is also possible.

The Québec Banque de dépannage linguistique lists several pronounciation and spelling variants.

The Trésor de la langue française only lists “entre quat’z’yeux” as a heading, but gives an example by Flaubert where the spelling is “entre quatre z’yeux”, and also mentions “entre quatre yeux” and “entre quatre-z-yeux” in the history section.

The Dictionnaire de l’Académie française only lists “entre quatre yeux” as the spelling but notes that the pronunciation is usually “entre quat’z-yeux” because it is jocular (“par plaisanterie”).

The prescriptive and very conservative Littré has this to say:

Entre quatre yeux: alone with one another. (…)

It is pronounced “entre quatre-z-yeux”. Some grammarians oppose this, but there is no reason to reject this euphonious letter which is present in many other cases: “va-s-y”, “donne-s-en”¹, etc. and which is supported by the authority of usage. Nonetheless, one says “quatre yeux” when two persons are staring at something: “quatre yeux étaient braqués sur moi” (“four eyes were staring at me”); as well as in the proverbial expression “quatre yeux voient mieux que deux” (“four eyes see better than two”), i.e. two persons together have better judgement than one alone.

Joseph-Marie Gary in Examen Critique du Dictionnaire de l’Académie française disagrees with Littré’s analysis.

In “vas-y, donnes-en”, the letter S must be pronounced because it is written. While some letters are written but not pronounced, one must never pronounce letters that are not written: that would grate on the ear, as when we say “*Malbrough s’en va-t-en guerre, quatre-z-officiers*”. To be logical, Mr Littré should propose to spell this expression “entre quatres yeux”. So long as this spelling is not adopted, to stick to pure speech, we must say “entre quatre yeux”, as we would say “entre quatre yeuses”, and with all the more reason as according to Mr Littré himself, one should not sound the S in “*Quatre yeux étaient braqués sur moi. Quatre yeux voient mieux que deux.”.

The Academy should not allow such a vicious pronunciation as quatre-z-yeux, quatre-z-officiers, il s’en va-t-en guerre.

Charles Pierre Girault-Duvivier and Pierre-Auguste Lemaire in
Grammaire des grammaires; ou, Analyse raisonée des meilleurs traités sur la langue française shares Gary’s sentiment but is not so adamant.

One writes entre quatre yeux to mean two people facing one another, and one pronounces quatre-z-yeux to make the pronunciation softer. Thus speaks the Academy under œil and under quatre.

Richelet and Trévoux write quatre yeux and do not discuss pronunciation.

Beauzée (Encyclopedie méthodique, under euphonique) thinks that it would be better to write quatre-s-yeux, so as to leave no doubt about pronunciation. He thinks that it would be ill-advised not to introduce an S, because otherwise one should either alter the first word and pronounce quate yeux, or decompose the second word and pronounce quatre yeux like ieuse², whereas one does not spoil either word’s pronunciation by introducing a euphonious S, which additionally matches the number of quatre.

There is admittedly some established usage of the pronunciation proposed by Bauzée, but it is the usage of people to which our orthography is wholly unknown. (…) How could an educated man conclude that, in order to soften the pronunciation, one should say entre quatre-syeux? (…) Why not quatre-s-œufs, (…) huits-yeux? Since S is softer than T.

Despite this reason and the authority of many grammarians, usage has triumphed, as the Academy affirms. And is it not natural that usage would decide on a decidedly informal, or colloquial, expression?

¹ _{Nowadays spelled “vas-y”, “donnes-en”.}
² _{I.e. [i.øz] rather than [jøz].}