The mechanics of instinctive behavior are mysterious. Even something as simple as the question of how a mouse can use its powerful sense of smell to detect and evade predators, including species it has never met before, has been almost totally unknown at the molecular level until now.

David Ferrero and Stephen Liberles, neuroscientists at Harvard Medical School, have discovered a single compound found in high concentrations in the urine of carnivores that triggers an instinctual avoidance response in mice and rats. This is the first time that scientists have identified a chemical tag that would let rodents sense carnivores in general from a safe distance. The authors write that understanding the molecular basis of predator odor recognition by rodents will provide crucial tools to study the neural circuitry associated with innate behavior.

Their findings were published online in the Proceedings of the National Academy of Science on June 20, 2011.

The search began in 2006, when Stephen Liberles, now Assistant Professor of Cell Biology at Harvard Medical School, was working as a post-doc in the lab of Linda Buck. Buck was part of the team that won the Nobel Prize for identifying the receptors that allow olfactory neurons to detect odors. While in her lab, Liberles identified a new type of olfactory receptor, the trace amine-associated receptors (TAARs).

Mice have about 1200 kinds of odor receptors, and 14 kinds of TAARs. In comparison, humans —who rely more on vision than smell— have about 350 odor receptors and five TAARs.

Liberles’s initial findings indicated that several of the TAARs detect chemicals found in mouse urine, including a chemical with enriched production by males. He wondered, could TAARs (which appear to have originally evolved from neurotransmitter receptors that mediate behavior and emotion) play a role in the social behavior of rodents? What other kinds of naturally occurring odors might they be able to detect?

In Liberles’s lab at Harvard Medical School, graduate student David Ferrero began a search for other natural compounds that were detected by the TAARs. Working with commercially available predator and prey urine (used by gardeners to keep pests out of their crops and by hunters to mask their own scent or as lures for prey), Ferrero discovered that one of the 14 TAARs, TAAR4, detected the odor of several carnivores.

It seemed they had found a kairomone, a chemical that works like a pheromone, except that it communicates between members of different species instead of members of the same species. Prior to this discovery, the only known rodent-carnivore kairomones were a volatile compound produced by foxes, but not in that of other predators, and two non-volatile compounds produced by cats and rats (which prey on mice). Volatile compounds aerosolize and can be smelled at great distances; non-volatile compounds need to be sniffed more directly, something that would not be helpful in avoiding a predator directly but rather their terrain.

“One of the things that’s really new here is that this is a generalized predator kairomone that’s volatile,” said Ferrero.

For rodents, it’s the smell of danger.

Ferrero identified the compound that activates TAAR4 as 2-phenylethylamine, a product of protein metabolism. He then obtained specimens from 38 species of mammals and found elevated levels of 2-phenylethylamineby 18 of 19 species of carnivores, but not by non-carnivores (including rabbits, deer, primates, and a giraffe).

“It’s been known so long that predator odors are great rodent deterrents, but we’ve discovered one molecule that’s a key part of this ecological relationship,” Ferrero said.

In a series of behavior tests, rats and mice showed a clear, innate avoidance to the smell of 2-phenylethylamine. The behavioral studies were repeated using a carnivore samples that had been depleted of 2-phenylethylamine. Rats failed to show full avoidance of the depleted carnivore urine, indicating that 2-phenylethylamine is a key trigger for predator avoidance.

Lacking the gene for TAAR4, humans can’t experience anything like what rodents do when they smell 2-phenylethylamine. To us, it has a mildly inoffensive odor. But trimethylamine, a related organic compound that activates TAAR5, a receptor found in humans, is deeply repugnant to people.

What happens between the receptors and the parts of the brain that trigger that avoidance behavior remains a mystery, one with direct medical relevance.

According to Liberles, “In humans, the parts of the brain that deal with likes and dislikes go awry in many diseases, like drug addiction, and predator odor responses have been used to model stress and anxiety disorders. Going from chemicals to receptors to neural circuits to behaviors is a Holy Grail of neuroscience.”

“The neural circuits are like a black box, but here we have identified a chemical stimulant and a candidate receptor that trigger one behavior,” Ferrero said. “We feel this is an important first step to understanding the neural circuitry of innate behavior.”

This research was funded by the National Institute On Deafness And Other Communication Disorders.