Artificial intelligence and computer science technologies involving the human face can identify a person, infer one’s emotions and proclivities, and uncover information about one’s health.

These approaches apply cutting edge techniques to an age-old pursuit: to learn about a person’s innerworkings from their outward appearance. In the past, physiognomy, the practice of assessing mental character based on facial features and other physical characteristics, sought to achieve a similar but often misguided goal.

Now debunked as a pseudoscience, physiognomy enjoyed periods of popularity and legitimacy over a long history that spans millennia. Through medical and scientific literature, its impact has been widespread and long lasting, reaching the fields of medicine, biology, philosophy, anthropology, psychiatry, and criminology.

Modern science has discarded the more problematic aspects of physiognomy, as technologies that gain information from physical appearance by interpreting data about bodies remain part of an enduring physiognomic legacy.

Side-by-side photographs of a white man measuring another white man with an apparatus

Apparatuses used to measure convicts’ personal attributes from Criminal man, according to the classification of Cesare Lombroso, Cesare and Gina Lombroso, 1911

Courtesy National Library of Medicine


Fig. 35


(see page 246)

Fig. 36

Campimeter or Landolt


(see page 249)

Illustration of a head with marking splitting it into sections

A facial character chart from New Physiognomy…, Samuel Wells, ca. 1894

Courtesy National Library of Medicine




Illustration of a world map indicating the geographic origins of various racial groups

A world map of racial groups from Types of mankind…, Josiah Clark Nott, 1857

Courtesy National Library of Medicine



Nott & Gliddon’s Types of Mankind, 1854.

Vector diagram between two images of a baby’s face

“Computer Vision Tools for Low-Cost and Noninvasive Measurement of Autism-Related Behaviors in Infants,” Autism Research and Treatment, Hashemi J, Tepper M, Vallin Spina T, Esler A, Morellas V, Papanikolopoulos N, Egger H, Dawson G, Sapiro G., 2014

Courtesy National Library of Medicine


From Divination to Discipline

Physiognomy developed in ancient Greece. Scholars expanded upon classical ideas and sought to legitimize the practice in early modern Europe.

In the ancient world, physiognomy had been practiced as a form of fortune telling. Classical Greek philosophers and physicians transformed physiognomy into a method to determine one’s character, believing in a symmetry between the soul and the body. Their views fell in line with humoral theory, the prevailing medical system of the time, which asserted that the balance of four bodily fluids dictated both physical condition and temperament.

By the 16th century, physiognomy had been dismissed as occult. Scholars sought to re-establish it as a legitimate practice, referring to ancient Greek texts to inform their work. They attempted to demonstrate the merits of physiognomy in illustrated treatises, systematized the basic principles, and engaged in scholarly debates and academic lectures.

Ornately illustrated title page of a book

The enduring influence of ancient Greece

In Physiognomica Aristotelis commentii (Commentaries on Aristotle’s Physiognomics), Camillo Baldi and Geronimo Tamburini, 1621

Courtesy National Library of Medicine

Physiognomica, a work attributed to followers of Aristotle, describes physiognomy methods, including assigning personality traits based on resemblance to animals, ethnicity, and facial expressions. These influential concepts endured into the 19th century.


IN PHYSIOGNOMICA ARISTOTELIS COMENTARII A CAMILLO BALDO ORDINARIAM PHILOSOPHIAM in Patrio Bononientî Archigymnasio prositente lucubrate. Opus multiplici doctrina refertum; Physiologicis, Medicis, virisq; Politicis æqué vtile, ac iucundum.

HIERONYMI TAMBURINI DILIGENTIA, & sumptibus nunc primum in lucem editum.


BONONIAE Apud Sebastianum Bonomium Superiorum permissu, MDCXXI.

Illustration comparing the head of a man to the head of a lion

Physiognomy’s renaissance

De humana physiognomonia libri IIII (Of human physiognomy book 4), Giabattista della Porta, 1586

Courtesy National Library of Medicine

Italian polymath Giovanni Battista della Porta (1535–1615) presented the case for physiognomy with detailed illustrations in his work De Humana Physiognomonia (1586). Porta’s popular book helped revitalize physiognomy in the 16th century.


Title page of a book and a page with an illustration of human and animal eyes

Impact on portraiture

La physionomie humaine comparée à la physionomie des animaux … (Human Physiognomy Compared to Animal Physiognomy…), Charles Le Brun and Lucien Métivet, 1927

Courtesy National Library of Medicine

French painter Charles Le Brun (1619–1690) created a corpus of facial drawings that showed physiognomy concepts. His influential artistic works and lectures about the similarities between human and animal faces and emotional expression in facial illustrations helped shape portraiture and the study of aesthetics.








6, rue de Tournon (VP1o)




Nous croyons intéressant de donner ici, en terminant, une série de quatre planches caractéristiques oú Le Brun a étudié en detail l’expression des yeux.

Sur la première figurant l’homme, le singe et le chameau. Nivelon, d’après son maître, fait remarquer que l’homme est la seule creature qui puisse diriger ses regards vers le ciel sans avoir à lever la tête et que ses sourcils sont plus mobiles que ceux des bêtes.

Dans les planches suivantes on trouvera:

II. Le tigre, le loup-cervier, le chat.

III. Le loup, le renard, le pourceau.

IV. Le bouc, le belier, le mouton,

Illustration of the profile of 12 men of various nationalities

A champion for physiognomy

Essays on Physiognomy: Calculated to Extend the Knowledge and the Love of Mankind, Johann Caspar Lavater, 1797

Courtesy National Library of Medicine

Swiss theologian Johann Caspar Lavater (1741—1801) attempted to give physiognomy scientific credence by systematizing concepts and techniques in his treatise Essays of Physiognomy (1772). He advocated for physiognomy and helped to popularize it through lectures, letters to newspapers, and academic debate.


Biometric Beginnings

Physiognomy became a form of biometric identification in the urban centers of the 19th century.

Governments and police systems adopted new ways to keep track of citizens and document details of crime in the growing cities of the 19th and early 20th centuries. Using the standardized procedures that had become a part of physiognomy, police measured and photographed the facial and bodily features of the people they arrested.

Police maintained the personal data they collected in information storage systems, which could be referenced to confirm citizenship and identify victims and perpetrators of crime. The standardization of methods and extensive organization of information lent to the reliability of early biometric identification.

Diagram of a White man’s face and profile with letters pointing out various characteristics

The measure of a man

Tableau Synoptique des Traits Physionomiques (Synoptic Table of Physiognomic Traits), a facial measurement chart for the Parisian police department, ca. 1909

Courtesy Twentieth-Century Photography Fund, 2009

After an arrest, police collected identifying information from the suspect. They took detailed measurements of facial features and the head, as well as the body, adopting a technique used in physiognomy, called “anthropometry.”



A Insertion des cheveux.

B Arcades sourcilières.

AB Inclinaison du front.

AL Hauteur du front.

C Racine du nez.

D Bout du nez.

CD Dos de nez.

ED Inclinaison de la base du nez.

LE Hauteur du nez.

ED Saillie du nez.

AD Profil fronto-nasal.

EF Profil naso-buccal.

GH Ligne oculo-tragienne.

GI Ligne horizontale.

GO Hauteur crânienne.

D A D C C F O E E’ O F’ H G G’ H’ I I’ J J’ K L L’ K’ B

AB Ligne médiane.

CC’ Largeur du front.

E et. E” Pointes internes des sourcils.

F et F” Pointes externs des sourcils.

G et G” Angles internes des paupières.

H et H” Angles externs des paupières.

HG Fente des paupières.

OO’ Ouverture des paupières.

O Paupière supérieure.

O’ Paupière inféricure.

JJ’ Largeur du nez.

II’ Ecartement des zygomes.

KK’ Ecartement des mâchoires

L et L’ Angles de la bouche.

Illustration showing how to measure criminals’ bodies after they’ve been arrested

The identification of criminals…, Charles Felton, 1889

Courtesy National Library of Medicine


Length of Head

Width of Head

Left Middle Finger

Left Foot

The Heighth.

Length of Right Ear

Length of the Trunk

The Left Fore-arm

Outstretched Arms.

Color of the Eyes.

Photograph of two gold measurement instruments

Tools of the trade

A caliper compass and a sliding compass for measuring arrestees, ca. 1900

Courtesy Science Museum, London

Police officers used a variety of instruments borrowed from anthropology and medicine to take extensive facial and bodily measurements of criminals, including calipers, compasses, and measuring tape.


Card showing a White man’s mugshot, fingerprints, and personal information like bodily measurements

Organized and retrievable

Bertillon card of Prohibition-era criminal George Anderson using the alias “Charles Steines,” 1922

Courtesy Cleveland Police Historical Society and Museum

French police clerk Alphonse Bertillon (1853–1914) devised a classification scheme for facial and body measurements and other personal details collected after an arrest. Information on each criminal, along with a mugshot, was stored in a filing system organized by these classifications.


Name: Charles P. Steines

Aliases “Dutch” Anderson


Residence N.Y. City 93.25.07

Crime Rob US Mail

Date of Arrest July 3- 1922

Disposition Date 192

Previous Record


Height 1m 71.0

Eng. Height 5-83/4

Outside A 1m70.0

Trunk 93.2

Head Lenght 20.4

Head Width 16.0

Cheek Width 14.4

R. Ear 6.9

L. Foot 36.6

L.Mid.F. 11.7

L.Lit.F. 9.3

L.Fore.A. 45.7

Color of left Eye

Class Blue

Periph. Z.


Age 42 Born in 1

Apparent Age


Occupation Gilman

Remarks Incident to Measurement

Right forefinger print to be impressed IMEMEDIATELY after Signature is written
















R. Ear




Hair M Ch

Complexion M

Weight 154

Build M.

Marks & Scars

Prisoner’s Signature



Frank Smith, Chief of Police

Bureau of Criminal Identification

Gallery No. F. No. 18758

Name Chaz B. Steines

Aliases Geo. Brown/Ivan Dahl Teller-Swede

Classification No. 9-A-15



1. Right Thumb

2. R. Fore Ringer

3. R. Middle Finger

4. R. Ring Finger

5. R. Little Finger


6. Left Thumb

7. L. Fore Finger

8. L. Middle Finger

9. L. Ringer Finger

10. L. Little Finger


Plain impressions taken simultaneously


Plain impressions taken simultaneously

Escaped from Atlanta Ga. U.S. Pen

Dec. 30” 1923

Wanted by U.S.

Impressions and Bertillon Measurements taken by N.Y. City at Bureau of Identification

Police Dept.-Cleveland, O. July, 1922

Photograph of a large filing cabinet

A filing room for biometric data on criminals at a Cleveland police precinct, 1930

Courtesy Cleveland Police Historical Society and Museum


The Face of Crime

Early criminologists incorporated physiognomy into their theories about criminal behavior.

Nineteenth century social philosophers began studying criminals scientifically, giving rise to the field of criminology. They conducted research on convicts and borrowed concepts from evolutionary theory and Mendelian genetics to formulate hypotheses on the roots of criminality.

Italian physician Cesare Lombroso (1835–1909) established the prevailing school of thought in criminology. According to his ideas, criminal traits were innate—written in the genes and reflected in the facial features; and criminals were the result of a regression to an earlier evolutionary state. Lombroso measured the facial and bodily attributes of thousands of offenders over his career, attempting to deduce the physical characteristics that comprised the criminal “look.” His ideas held currency until World War II.

illustration of the heads of three men and the profile of a woman

Criminal looks

L’Homme Criminel (Criminal man), Cesare Lombroso, 1887

Courtesy National Library of Medicine

Italian physician Cesare Lombroso believed that criminal tendencies were inborn and determined biologically, and that criminals had telltale facial characteristics, like a sloping brow, large jaw, or thin upper lip.


1. P.C., brigand de la Basilicate, détenu à Pesaro.

2. Voleur piémontais.

3. Incendiaire et cynède de Pesaro, surnommé la femme.

4. Misdea.

Photograph of a white man

On the origin of criminality

A man deemed “atavistic” for his facial features, ca. 1900

Courtesy Museo di Antropologia criminale "Cesare Lombroso", University of Turin (Italy)

Early criminology appropriated evolutionary theory. Cesare Lombroso and his contemporaries thought criminal tendencies, and the associated physical characteristics, were “atavistic,” a reversion to a primitive, subhuman evolutionary state.


Illustration of an animal-like man

Intersections with psychiatry

“Madman,” a portrait of a person whose physical form was shaped by his mental illness, from Essays on the Anatomy of Expression in Painting, Charles Bell, 1844

Courtesy National Library of Medicine

Criminology and psychiatry influenced each other and shared views on the biological nature of personality. Like criminologists, many in psychiatry connected physical traits to character. Criminologists helped shape psychiatric treatment for offenders with mental illness.


Illustration of a photography apparatus

Empirical Techniques

Composite portraiture apparatus from “Composite Portraits,” Journal of the Anthropological Institute of Great Britain and Ireland, Francis Galton, 1878

Nineteenth century physiognomists adopted observational, technology-based methods. English scholar Francis Galton (1822–1911) developed composite portraiture to identify criminal facial features. He superimposed portraits of offenders, hypothesizing that the most prominent features would be criminal signifiers. Though later debunked, empirical techniques gave physiognomy and criminology a new level of scientific credence.


134 FRANCIS GALTON—Composite Portraits.

remove. No. 1 from the pins, and No. 2 appears in the front ; we take off the cap a second time for ten seconds, and replace it. Next we remove No. 2 and No. 3 appears in the front, which we treat as its predecessors, and so we go on to the last of the pack. The sensitised plate will now have had its total exposure of eighty seconds ; it is then developed, and the print is taken from it is the generalised picture of which I speak. It is

The Pseudoscience of Difference

Physiognomy served as a tool for scientific racism and eugenics.

Physical anthropologists of the 18th–20th centuries used measurements of the face, skull, and body to define race. They correlated these race-defining characteristics with intelligence and moral character and deemed certain races to be superior or inferior based on these assessments.

English polymath Francis Galton (1822–1911) introduced physiognomy into modern eugenics. Through his influence, physiognomy became a standard technique for eugenicists to appraise character, assign intelligence and morality, and ultimately, determine whether certain people should reproduce. By the latter half of the 20th century, physiognomy, scientific racism, and eugenics had largely been debunked as harmful pseudoscience.

The Destructive Impact of Bad Science

Scientists used flawed methods and reasoning to deduce, incorrectly, that people of color, people with disabilities, LGBTQ people, women, non-neurotypical people, and people of lower socioeconomic class were inherently inferior. This faulty science informed 20th century discriminatory policies and human rights abuses, including institutionalized racial oppression, anti-LGBTQ laws, exclusionary immigration policy, forced pregnancies and sterilizations, and genocide.


Illustration of a world map indicating the geographic origins of various racial groups

Determining race

A world map of racial groups from Types of mankind…, Josiah Clark Nott, 1857

Courtesy National Library of Medicine

Physical anthropologists used facial features to determine race and connected those facial features to intelligence and character, fostering stereotypes that persist today. Races were placed on a hierarchy with white people at the top.



Nott & Gliddon’s Types of Mankind, 1854.

Page with text and a logo above showing a tree with the word “eugenics” emblazoned across it and various intellectual disciplines in the trees roots

Unnatural selection

The logo for a meeting of the world’s leading eugenicists, from The Second International Exhibition of Eugenics…, International Conference on Eugenics, 1921

Courtesy National Library of Medicine

Eugenics aimed to improve mankind through selective breeding. Eugenicists used physiognomy, among other techniques, to classify people and determine who was fit to reproduce. The movement was mainstream—its ideas were accepted by laypeople, leaders, and the scientific community.



Eugenics is the self direction of human evolution


Like a tree eugenics draws from materials from many sources and organizes them into an harmonious entity.

Title page of a book

A eugenicist discourages those with disabilities and mental health issues, and the poor from having children in The radical remedy in social science …, Edward B. Foote, ca. 1886

Courtesy National Library of Medicine


(Copyrighted April, 1886.)

The Radical Remedy in Social Science; or Borning Better Babies Through Regulating Reproduction by Controlling Conception. An Earnest Essay on Pressing Problems

by E. B. Foote, Jr. M.D.

Price, 25 cents.

New York: Murray Hill Publishing Company.

No. 129 East Twenty-Eighth St.

Photograph from profile view of a white man whose ear is being measured

Eugenics loses favor

A man is measured to determine his race at the Kaiser Wilhelm Institute of Anthropology, Human Heredity, and Eugenics in Germany, n.d.

Courtesy United States Holocaust Memorial Museum, courtesy of National Archives and Records Administration, College Park

After the Nazi regime in Germany used eugenics to justify the Holocaust and other crimes against humanity, the world largely denounced eugenics and scientific racism.


A page of a journal article with text only

The Fall of Physiognomy

A shift in focus from racial classification to evolutionary processes is laid out in “The New Physical Anthropology” from Transactions of the New York Academy of Sciences, Sherwood Washburn, May 1951

Courtesy the NIH Library

Physiognomy had been discredited by the mid-20th century. Held under scrutiny, its claims were disproven. Physical anthropology abandoned the practice as it disavowed the field’s racist past. Furthermore, an association with eugenics and the Holocaust made physiognomy distasteful.





Recently, evolutionary studies have been revitalized and revolutionized by an infusion of genetics into paleontology and systematics. The change is fundamentally one of point of view, which is made possible by an understanding of the way the genetic constitution of populations changes. The new systematics is concerned primarily with process and with the mechanism of evolutionary change, whereas older point of view was chiefly concerned with sorting the results of evolution. Physical anthropology is now undergoing the same sort of change. Population genetics presents the anthropologist with a clearly formulated, experimentally verified, conceptual scheme. The application of this theory to the primates is the immediate task of physical anthropology.

In the past, physical anthropology has been considered primarily as a technique. Training consisted in learning to take carefully defined measurements and in computing indices and statistics. The methods of observation, measurement, and comparison were essentially the same, whether object of the study was the description of evolution, races, growth, criminals, constitutional types, or army personnel. Measurements were adjusted for various purposes, but measurement of the outside of the body, classification, and correlation, remained the anthropologist’s primary tools. The techniques of physical anthropology were applied to a limited group of problems and any definition or statement of traditional anthropology must include both the metrical methods and the problems for which the methods were used. Further, anthropology was characterized by theories, or rather by a group of attitudes and assumptions.

*This paper, illustrated with lantern slides, was presented at the meeting of the Section, April 23, 1951.

†Department of Anthropology, University of Chicago, Chicago, Illinois.

There has been almost no development of theory in physical anthropology itself, but the dominant attitude may be described as static, with emphasis on classification based on types. Any such characterization is oversimplified, and is intended only to give an indication of the dominant techniques, interests, and attitudes of the physical anthropologist. Except for emphasis on particularly animals, physical anthropology shared much with the zoology of the times when it developed. Much of the method was developed before the acceptance of the idea of evolution, and all of it before the science of genetics.

Physical anthropology should change, just as systematic zoology has changed. The difficulty which accompany the necessary modifications can be greatly reduced if their nature is clearly understood. Naturally, in a time of rapid flux there will be numerous doubts and disagreements as to what should be done. This is natural, and what I have to offer is a tentative outline to indicate how parts of the new physical anthropology may differ from the old.

The old physical anthropology was primarily a technique. The common core of the science was measurement of external form with calipers. The new physical anthropology is primarily an area of interest, the desire to understand the process of primate evolution and human variation by the most efficient techniques possible. The process of evolution, as understood by the geneticist, is the same for all

A Digital Transformation

Artificial intelligence and computer science technologies that measure and analyze physical attributes have a variety of modern applications.

We’ve rejected the harmful aspects of physiognomy, but efforts to gain information from human physical characteristics continue with contemporary technologies that gather and interpret bodily data. Uncoupled from pseudoscientific beliefs and practices, body measurement and analysis are now used in ways that can benefit society. Today’s methods include automation, analyzing large datasets to identify characteristics and make inferences with improved accuracy.

Modern artificial intelligence employs deep learning, which trains machines to simulate human learning through layers of algorithms. Computer vision identifies relevant features from images and video. These technologies have the potential to make the world safer, improve health and foster biomedical discovery, and affect how we get information.

Infographic of types of biometric identification

Security and surveillance

Biometric technologies reference databases of personal information to identify people and limit access to sensitive information and restricted places. They’re used in law enforcement and border security, as well as everyday life, helping keep medical records, financial information, phones, and smart home devices safe.


Diagram showing ultrasound images of a heart and charting heartbeat

Biomedical research and better care

Artificial intelligence works to detect cardiomyopathy from a heart ultrasound from “Real-time echocardiography image analysis and quantification of cardiac indices.” Medical Image Analysis, Ghada Zamzmi, Sivaramakrishnan Rajaraman, Li-Yueh Hsu, Vandana Sachdev, and Sameer Antani, 2022

Courtesy National Library of Medicine

Artificial intelligence and computer vision analyze visual material to help uncover the mechanisms behind diseases, assist with diagnosis, predict health outcomes, and provide insights for more effective, individualized care. National Library of Medicine research has advanced techniques to help predict and spot health conditions, including heart dysfunction.


Scope image of a colon with polyps pointed out

“Computer vision and augmented reality in gastrointestinal endoscopy.” Gastroenterol Report, Mahmud N, Cohen J, Tsourides K, Berzin TM, 2015.

Courtesy National Library of Medicine



Polyp detected


Polyp (90% certainty)

Polyp (70% certainty

Withdrawal Time: 3:28

Scope position: 50cm

Four side-by-side images of a baby’s face with different areas pointed out by different colored shapes and markers

Detecting emotions

Computer vision program determines the pain level of infants from “Automatic Infants’ Pain Assessment by Dynamic Facial Representation: Effects of Profile View, Gestational Age, Gender, and Race.” Journal of Clinical Medicine, Ruicong Zhi, Ghada Zamzmi Dmitry Goldgof, Terri Ashmeade, and Yu Sun, 2018

Courtesy National Library of Medicine

Affective computing is a growing area of artificial intelligence that identifies and evaluates facial expressions utilizing computer vision, then determines emotions through deep learning. Its potential uses in medicine include detecting mood disorders, studying neurodiversity, screening for neurodegenerative disease, and improving telemedicine.


AI-generated image of a White woman’s face

Fabricating faces

AI-generated image of a fake person, 2022

Courtesy NVidia Software

In visual art and media, artificial intelligence is used to create hyperreal images and videos featuring imaginary people or the likenesses of real people. “Deepfakes” and other forms of synthetic media are controversial for their potential use in fraud and disinformation.


Title page of online book

Looking to the Future

Trustworthy AI (TAI) Playbook, U.S. Department of Health and Human Services, 2021

Courtesy U.S. Department of Health and Human Services

As technology advances, AI experts, biomedical researchers, and ethicists are working together to realize the potential of AI for the greater good while understanding and learning from past mistakes rooted in rejected ideas. Their interdisciplinary work is eliminating bias in data and programs, and marginalized communities, protecting privacy, and anticipating and prevent negative outcomes.


Trustworthy AI (TAI) Playbook

U.S. Department of Health and Human Services

September 2021