“un”/”une” is also the equivalent for “a” in English, so the undetermined article, whereas there is no other term for two, three…
"One" probably receives special treatment because it is singular. As Matthieu Brucher says, un(e) is not only used as a numeral, but also as an "indefinite article"; the French definite article likewise has distinct forms for masculine and feminine in the singular (le/la) but not in the plural (les).
Many French words do not have distinct forms for masculine and feminine. It’s not obvious whether this is best analyzed as "lacking gender inflection" or "having the same form for both genders" (or different ways in different contexts). But in any case, we see that adjectives such as jaune, rouge and orange don’t change between masculine and feminine (and in many accents of French, the distinction between bleu and bleue is purely graphical), while vert(e), gris(e), blanc(he) do change.
As I said in the first paragraph, it’s probably relevant that un(e) is singular while the higher numbers are plural (both in sense, and for deux and trois, also in form). In languages in general, and also in French specifically, gender distinctions tend to be marked more often in the singular than in the plural1 (although there are some exceptions such as the -al/-aux/-ale/-ales adjectives, where the masculine plural but not the singular is marked by a distinctive [o] sound). We see the same form for masculine and feminine with the French plural articles les, des and the plural "possessive adjectives" mes, tes, ses, nos, vos, leurs.
It’s fairly common for languages with grammatical gender to only show gender agreement for a subset of cardinal numbers (usually only lower ones), and this is the case for Latin, the ancestor of French. Latin had gender agreement on only a few cardinal numbers: the words for "one", "two" and "three" agreed in gender with the noun, but higher numbers did not (except for, oddly, multiples of 100).
So if French were to show gender agreement on four, five, six, seven, etc. this would represent an innovation from the situation in Latin. While there are some places where French has added inflection relative to Latin, the general trend has been a reduction rather than an increase in the amount of inflection of this kind.
The Latin word for "three" had the same form for masculine and feminine (tres); the distinct form tria was used with neuter-gender nouns, but as French lost the neuter gender, it may not be too surprising that it also lost gender inflection for the number 3.
In Latin, 2 in the accusative case (the main source of Romance forms) had the masculine form duōs and the feminine form duās. The modern French spelling with "x" is just from a spelling change of "us" to "ux"; this isn’t a word that historically had a "c" sound in either the masculine or feminine. Rather, the "x" represents the plural suffix that is spelled as "-s" in other contexts, and which has become silent (outside of liaison contexts) in modern French for both masculine and feminine words.
Historical development of the form deux: from originally masculine to both genders
Latin duōs apparently developed via Old French dous to modern French deux /dø/. I found the following information about the situation in Old French:
fem. doues occasionally occurred, in eastern dialects only, though in most texts the originally masculine form dous is usual in the feminine also.
–"Romance" by Glanville Price, in Indo-European Numerals, edited by Jadranka Gvozdanovic, 1992 (p. 450).
Price also mentions that Old French had a distinct masculine nominative form derived from Vulgar Latin dui (an analogical replacement of the Classical Latin masculine nominative form duo) (p. 449). Apparently, the Old French nominative masculine form was also spelled dui, with the i retained as an offglide2. But this form was lost when the Old French distinction between a nominative and oblique case was lost.
I found a reference that lists "dues" and "deues" rather than "doues" as possible Old French feminine forms of the numeral 2, but I’m not sure how/why these would have developed a distinct vowel from masculine dous (An Introduction to Old French, by François Frédéric Roget, p. 90, 1887). Roget agrees with Price that "The feminine was not long preserved distinct from the masculine".
1this is brought up on p. 703 of "On the Use of ils for elles: Gender Syncretism in the History of French", by Barbara E. Bullock, The French Review Vol. 74, No. 4 (Mar., 2001)
2A Contribution to the History of the Unaccented Vowels in Old French, by William Pierce Shepard (1897), p. 19