Promising Future, Complex Past: Artificial Intelligence and the Legacy of Physiognomy

Artificial intelligence and computer science technologies involving the human face can identify a person, infer one’s emotions and proclivities, and uncover information about one’s health.

These approaches apply cutting edge techniques to an age-old pursuit: to learn about a person’s innerworkings from their outward appearance. In the past, physiognomy, the practice of assessing mental character based on facial features and other physical characteristics, sought to achieve a similar but often misguided goal.

Now debunked as a pseudoscience, physiognomy enjoyed periods of popularity and legitimacy over a long history that spans millennia. Through medical and scientific literature, its impact has been widespread and long lasting, reaching the fields of medicine, biology, philosophy, anthropology, psychiatry, and criminology.

Modern science has discarded the more problematic aspects of physiognomy, as technologies that gain information from physical appearance by interpreting data about bodies remain part of an enduring physiognomic legacy.

Side-by-side photographs of a white man measuring another white man with an apparatus

Apparatuses used to measure convicts’ personal attributes from Criminal man, according to the classification of Cesare Lombroso, Cesare and Gina Lombroso, 1911

Courtesy National Library of Medicine

Fig. 35

Algometer

(see page 246)

Fig. 36

Campimeter or Landolt

(Modified)

(see page 249)

Illustration of a head with marking splitting it into sections

A facial character chart from New Physiognomy…, Samuel Wells, ca. 1894

Courtesy National Library of Medicine

PREVIOUS SYSTEMS

Fig. 72—DIAGRAM, FRONT VIEW.

Illustration of a world map indicating the geographic origins of various racial groups

A world map of racial groups from Types of mankind…, Josiah Clark Nott, 1857

Courtesy National Library of Medicine

MAP TO ACCOMPANY PROF. AGASSIZ’S “SKETCH.”

Nott & Gliddon’s Types of Mankind, 1854.

Vector diagram between two images of a baby’s face

“Computer Vision Tools for Low-Cost and Noninvasive Measurement of Autism-Related Behaviors in Infants,” Autism Research and Treatment, Hashemi J, Tepper M, Vallin Spina T, Esler A, Morellas V, Papanikolopoulos N, Egger H, Dawson G, Sapiro G., 2014

Courtesy National Library of Medicine

From Divination to Discipline

Physiognomy developed in ancient Greece. Scholars expanded upon classical ideas and sought to legitimize the practice in early modern Europe.

In the ancient world, physiognomy had been practiced as a form of fortune telling. Classical Greek philosophers and physicians transformed physiognomy into a method to determine one’s character, believing in a symmetry between the soul and the body. Their views fell in line with humoral theory, the prevailing medical system of the time, which asserted that the balance of four bodily fluids dictated both physical condition and temperament.

By the 16th century, physiognomy had been dismissed as occult. Scholars sought to re-establish it as a legitimate practice, referring to ancient Greek texts to inform their work. They attempted to demonstrate the merits of physiognomy in illustrated treatises, systematized the basic principles, and engaged in scholarly debates and academic lectures.

Ornately illustrated title page of a book

The enduring influence of ancient Greece

In Physiognomica Aristotelis commentii (Commentaries on Aristotle’s Physiognomics), Camillo Baldi and Geronimo Tamburini, 1621

Courtesy National Library of Medicine

Physiognomica, a work attributed to followers of Aristotle, describes physiognomy methods, including assigning personality traits based on resemblance to animals, ethnicity, and facial expressions. These influential concepts endured into the 19th century.

IN PHYSIOGNOMICA ARISTOTELIS COMENTARII A CAMILLO BALDO ORDINARIAM PHILOSOPHIAM in Patrio Bononientî Archigymnasio prositente lucubrate. Opus multiplici doctrina refertum; Physiologicis, Medicis, virisq; Politicis æqué vtile, ac iucundum.

HIERONYMI TAMBURINI DILIGENTIA, & sumptibus nunc primum in lucem editum.

AD SERENISSIMUM PRINCIPEM FERDINANDUM GONZAGAM Mantuæ & Montisferrati Ducem

BONONIAE Apud Sebastianum Bonomium Superiorum permissu, MDCXXI.

Illustration comparing the head of a man to the head of a lion

Physiognomy’s renaissance

De humana physiognomonia libri IIII (Of human physiognomy book 4), Giabattista della Porta, 1586

Courtesy National Library of Medicine

Italian polymath Giovanni Battista della Porta (1535–1615) presented the case for physiognomy with detailed illustrations in his work De Humana Physiognomonia (1586). Porta’s popular book helped revitalize physiognomy in the 16th century.

Title page of a book and a page with an illustration of human and animal eyes

Impact on portraiture

La physionomie humaine comparée à la physionomie des animaux … (Human Physiognomy Compared to Animal Physiognomy…), Charles Le Brun and Lucien Métivet, 1927

Courtesy National Library of Medicine

French painter Charles Le Brun (1619–1690) created a corpus of facial drawings that showed physiognomy concepts. His influential artistic works and lectures about the similarities between human and animal faces and emotional expression in facial illustrations helped shape portraiture and the study of aesthetics.

CONTRIBUTION A L’ÉTUDE DE LA CARICATURE

LA PHYSIONOMIE HUMAINE COMPARÉE A LA PHYSIONOMIE DES ANIMAUX D’APRÉS LES DESSINS DE CHARLES LE BRUN

CENT TRENTE REPRODUCTIONS PRÉCÉDÉS D’UN ESSAI ET ACCOMPAGNÉES DES RÉFLEXIONS ET MENUS PROPOS D’UN PEINTRE PARISIEN LUCIEN MÉTIVET

DUEXIÈME EDITION

PARIS

HENRI LAURENS, ÉDITEUR

6, rue de Tournon (VP1o)

1927

ÉTUDES D’YEUX

Nous croyons intéressant de donner ici, en terminant, une série de quatre planches caractéristiques oú Le Brun a étudié en detail l’expression des yeux.

Sur la première figurant l’homme, le singe et le chameau. Nivelon, d’après son maître, fait remarquer que l’homme est la seule creature qui puisse diriger ses regards vers le ciel sans avoir à lever la tête et que ses sourcils sont plus mobiles que ceux des bêtes.

Dans les planches suivantes on trouvera:

II. Le tigre, le loup-cervier, le chat.

III. Le loup, le renard, le pourceau.

IV. Le bouc, le belier, le mouton,

A champion for physiognomy

Essays on Physiognomy: Calculated to Extend the Knowledge and the Love of Mankind, Johann Caspar Lavater, 1797

Courtesy National Library of Medicine

Swiss theologian Johann Caspar Lavater (1741—1801) attempted to give physiognomy scientific credence by systematizing concepts and techniques in his treatise Essays of Physiognomy (1772). He advocated for physiognomy and helped to popularize it through lectures, letters to newspapers, and academic debate.

Biometric Beginnings

Physiognomy became a form of biometric identification in the urban centers of the 19th century.

Governments and police systems adopted new ways to keep track of citizens and document details of crime in the growing cities of the 19th and early 20th centuries. Using the standardized procedures that had become a part of physiognomy, police measured and photographed the facial and bodily features of the people they arrested.

Police maintained the personal data they collected in information storage systems, which could be referenced to confirm citizenship and identify victims and perpetrators of crime. The standardization of methods and extensive organization of information lent to the reliability of early biometric identification.

The measure of a man

Tableau Synoptique des Traits Physionomiques (Synoptic Table of Physiognomic Traits), a facial measurement chart for the Parisian police department, ca. 1909

Courtesy Twentieth-Century Photography Fund, 2009

After an arrest, police collected identifying information from the suspect. They took detailed measurements of facial features and the head, as well as the body, adopting a technique used in physiognomy, called “anthropometry.”

Illustration showing how to measure criminals’ bodies after they’ve been arrested

The identification of criminals…, Charles Felton, 1889

Courtesy National Library of Medicine

Photograph of two gold measurement instruments

Tools of the trade

A caliper compass and a sliding compass for measuring arrestees, ca. 1900

Courtesy Science Museum, London

Police officers used a variety of instruments borrowed from anthropology and medicine to take extensive facial and bodily measurements of criminals, including calipers, compasses, and measuring tape.

Card showing a White man’s mugshot, fingerprints, and personal information like bodily measurements

Organized and retrievable

Bertillon card of Prohibition-era criminal George Anderson using the alias “Charles Steines,” 1922

Courtesy Cleveland Police Historical Society and Museum

French police clerk Alphonse Bertillon (1853–1914) devised a classification scheme for facial and body measurements and other personal details collected after an arrest. Information on each criminal, along with a mugshot, was stored in a filing system organized by these classifications.

A filing room for biometric data on criminals at a Cleveland police precinct, 1930

Courtesy Cleveland Police Historical Society and Museum

The Face of Crime

Early criminologists incorporated physiognomy into their theories about criminal behavior.

Nineteenth century social philosophers began studying criminals scientifically, giving rise to the field of criminology. They conducted research on convicts and borrowed concepts from evolutionary theory and Mendelian genetics to formulate hypotheses on the roots of criminality.

Italian physician Cesare Lombroso (1835–1909) established the prevailing school of thought in criminology. According to his ideas, criminal traits were innate—written in the genes and reflected in the facial features; and criminals were the result of a regression to an earlier evolutionary state. Lombroso measured the facial and bodily attributes of thousands of offenders over his career, attempting to deduce the physical characteristics that comprised the criminal “look.” His ideas held currency until World War II.

Criminal looks

L’Homme Criminel (Criminal man), Cesare Lombroso, 1887

Courtesy National Library of Medicine

Italian physician Cesare Lombroso believed that criminal tendencies were inborn and determined biologically, and that criminals had telltale facial characteristics, like a sloping brow, large jaw, or thin upper lip.

On the origin of criminality

A man deemed “atavistic” for his facial features, ca. 1900

Courtesy Museo di Antropologia criminale "Cesare Lombroso", University of Turin (Italy)

Early criminology appropriated evolutionary theory. Cesare Lombroso and his contemporaries thought criminal tendencies, and the associated physical characteristics, were “atavistic,” a reversion to a primitive, subhuman evolutionary state.

Intersections with psychiatry

“Madman,” a portrait of a person whose physical form was shaped by his mental illness, from Essays on the Anatomy of Expression in Painting, Charles Bell, 1844

Courtesy National Library of Medicine

Criminology and psychiatry influenced each other and shared views on the biological nature of personality. Like criminologists, many in psychiatry connected physical traits to character. Criminologists helped shape psychiatric treatment for offenders with mental illness.

Empirical Techniques

Composite portraiture apparatus from “Composite Portraits,” Journal of the Anthropological Institute of Great Britain and Ireland, Francis Galton, 1878

Courtesy National Library of Medicine

Nineteenth century physiognomists adopted observational, technology-based methods. English scholar Francis Galton (1822–1911) developed composite portraiture to identify criminal facial features. He superimposed portraits of offenders, hypothesizing that the most prominent features would be criminal signifiers. Though later debunked, empirical techniques gave physiognomy and criminology a new level of scientific credence.

The Pseudoscience of Difference

Physiognomy served as a tool for scientific racism and eugenics.

Physical anthropologists of the 18th–20th centuries used measurements of the face, skull, and body to define race. They correlated these race-defining characteristics with intelligence and moral character and deemed certain races to be superior or inferior based on these assessments.

English polymath Francis Galton (1822–1911) introduced physiognomy into modern eugenics. Through his influence, physiognomy became a standard technique for eugenicists to appraise character, assign intelligence and morality, and ultimately, determine whether certain people should reproduce. By the latter half of the 20th century, physiognomy, scientific racism, and eugenics had largely been debunked as harmful pseudoscience.

The Destructive Impact of Bad Science

Scientists used flawed methods and reasoning to deduce, incorrectly, that people of color, people with disabilities, LGBTQ people, women, non-neurotypical people, and people of lower socioeconomic class were inherently inferior. This faulty science informed 20th century discriminatory policies and human rights abuses, including institutionalized racial oppression, anti-LGBTQ laws, exclusionary immigration policy, forced pregnancies and sterilizations, and genocide.

Determining race

A world map of racial groups from Types of mankind…, Josiah Clark Nott, 1857

Courtesy National Library of Medicine

Physical anthropologists used facial features to determine race and connected those facial features to intelligence and character, fostering stereotypes that persist today. Races were placed on a hierarchy with white people at the top.

Unnatural selection

The logo for a meeting of the world’s leading eugenicists, from The Second International Exhibition of Eugenics…, International Conference on Eugenics, 1921

Courtesy National Library of Medicine

Eugenics aimed to improve mankind through selective breeding. Eugenicists used physiognomy, among other techniques, to classify people and determine who was fit to reproduce. The movement was mainstream—its ideas were accepted by laypeople, leaders, and the scientific community.

A eugenicist discourages those with disabilities and mental health issues, and the poor from having children in The radical remedy in social science …, Edward B. Foote, ca. 1886

Courtesy National Library of Medicine

Eugenics loses favor

A man is measured to determine his race at the Kaiser Wilhelm Institute of Anthropology, Human Heredity, and Eugenics in Germany, n.d.

Courtesy United States Holocaust Memorial Museum, courtesy of National Archives and Records Administration, College Park

After the Nazi regime in Germany used eugenics to justify the Holocaust and other crimes against humanity, the world largely denounced eugenics and scientific racism.

A page of a journal article with text only

The Fall of Physiognomy

A shift in focus from racial classification to evolutionary processes is laid out in “The New Physical Anthropology” from Transactions of the New York Academy of Sciences, Sherwood Washburn, May 1951

Courtesy the NIH Library

Physiognomy had been discredited by the mid-20th century. Held under scrutiny, its claims were disproven. Physical anthropology abandoned the practice as it disavowed the field’s racist past. Furthermore, an association with eugenics and the Holocaust made physiognomy distasteful.

A Digital Transformation

Artificial intelligence and computer science technologies that measure and analyze physical attributes have a variety of modern applications.

We’ve rejected the harmful aspects of physiognomy, but efforts to gain information from human physical characteristics continue with contemporary technologies that gather and interpret bodily data. Uncoupled from pseudoscientific beliefs and practices, body measurement and analysis are now used in ways that can benefit society. Today’s methods include automation, analyzing large datasets to identify characteristics and make inferences with improved accuracy.

Modern artificial intelligence employs deep learning, which trains machines to simulate human learning through layers of algorithms. Computer vision identifies relevant features from images and video. These technologies have the potential to make the world safer, improve health and foster biomedical discovery, and affect how we get information.

Infographic of types of biometric identification

Security and surveillance

Biometric technologies reference databases of personal information to identify people and limit access to sensitive information and restricted places. They’re used in law enforcement and border security, as well as everyday life, helping keep medical records, financial information, phones, and smart home devices safe.

Diagram showing ultrasound images of a heart and charting heartbeat

Biomedical research and better care

Artificial intelligence works to detect cardiomyopathy from a heart ultrasound from “Real-time echocardiography image analysis and quantification of cardiac indices.” Medical Image Analysis, Ghada Zamzmi, Sivaramakrishnan Rajaraman, Li-Yueh Hsu, Vandana Sachdev, and Sameer Antani, 2022

Courtesy National Library of Medicine

Artificial intelligence and computer vision analyze visual material to help uncover the mechanisms behind diseases, assist with diagnosis, predict health outcomes, and provide insights for more effective, individualized care. National Library of Medicine research has advanced techniques to help predict and spot health conditions, including heart dysfunction.

Scope image of a colon with polyps pointed out

“Computer vision and augmented reality in gastrointestinal endoscopy.” Gastroenterol Report, Mahmud N, Cohen J, Tsourides K, Berzin TM, 2015.

Courtesy National Library of Medicine

Four side-by-side images of a baby’s face with different areas pointed out by different colored shapes and markers

Detecting emotions

Computer vision program determines the pain level of infants from “Automatic Infants’ Pain Assessment by Dynamic Facial Representation: Effects of Profile View, Gestational Age, Gender, and Race.” Journal of Clinical Medicine, Ruicong Zhi, Ghada Zamzmi Dmitry Goldgof, Terri Ashmeade, and Yu Sun, 2018

Courtesy National Library of Medicine

Affective computing is a growing area of artificial intelligence that identifies and evaluates facial expressions utilizing computer vision, then determines emotions through deep learning. Its potential uses in medicine include detecting mood disorders, studying neurodiversity, screening for neurodegenerative disease, and improving telemedicine.

AI-generated image of a White woman’s face

Fabricating faces

AI-generated image of a fake person, 2022

Courtesy NVidia Software

In visual art and media, artificial intelligence is used to create hyperreal images and videos featuring imaginary people or the likenesses of real people. “Deepfakes” and other forms of synthetic media are controversial for their potential use in fraud and disinformation.

Looking to the Future

Trustworthy AI (TAI) Playbook, U.S. Department of Health and Human Services, 2021

Courtesy U.S. Department of Health and Human Services

As technology advances, AI experts, biomedical researchers, and ethicists are working together to realize the potential of AI for the greater good while understanding and learning from past mistakes rooted in rejected ideas. Their interdisciplinary work is eliminating bias in data and programs, and marginalized communities, protecting privacy, and anticipating and prevent negative outcomes.