All of these meanings are closely related.

In particular, it is not surprising at all for a root meaning “knowledge of nature” to evolve into words applying to numerous subsets of such a wide concept, like “knowledge of human body and living organisms” (medical science, biology), how to heal them (medical science too), and “knowledge of the elements” (physical science), especially as during the centuries when this word evolved, what are now distant specialties were likely handled by the same persons that we might call generalists today.

Note also that the medical meaning side has faded in French. While it is still there in “forme physique“, “éducation physique“, unlike in English, we do not call a doctor a “physician/physicien“¹ so we do not need to use “physiciste” for a physicist.

¹ Actually we did, fisicien was used in the thirteen century French.

Littré donne de nombreux sens à physique, tournant tous autour de la nature et du matériel. Le dernier sens indique que le lien entre physique et médecine date du moyen âge.