Three years ago, Jeff Hawkins stood before a crowd at the Almaden Institute in San Jose and explained why, after founding Palm Computing, building the Palm Pilot, and establishing himself as a member of the Silicon Valley elite, he switched gears and devoted years to understanding the human brain, resulting in his 2004 book on Intelligence. “We don’t want to solve vision, we don’t want to solve language,” Hawkins told the crowd. “We want to solve something in the brain that is more fundamental.”
What could be more fundamental to our understanding the brain than the giant realms of language, vision, motor control, and other modalities? The answer finds its roots in the Einsteinian quest for unifying principals in science. In the case of the brain, one unifying principle would be the decoding of the language neurons use to represent information across all realms of the brain—the algorithms of conscious and unconscious brain activity.
Hawkins’ quest is based on the work of the neuroanatomist Vernon Mouncastle, who is credited with characterizing the basic structure of the cerebral cortex, which is populated, as he observed in a 1950 paper, with rows upon stacks of columns, the basic units of the cortex. In his Nobel Prize acceptance speech, David Hubel, the pioneering researcher of the visual cortex, called Mountcastle’s discovery of columns in the somatosensory cortex “surely the single most important contribution to the understanding of the cerebral cortex since Cajal.” For researchers studying any part of the cortex, columnar organization is the entry point to understanding a neuron’s position within the larger network—a familiar refuge in an otherwise dark and dense jungle of cells.
But it was a later paper Vernon Mountcastle published in 1978 that, as he writes in On Intelligence, caused Jeff Hawkins to “fall out of his chair.”
That paper was titled “An Organizing Principle for Cerebral Function,” and in it Mountcastle made the key observation that, in addition to columns, the cortex is remarkably uniform in cellular organization and morphology wherever you look. Like a McDonalds in Meriden or one in Bangladesh, some things just don’t change. Anatomists had recognized this fact for decades before Mountcastle penned his “Organizing Principle” paper in 1978. Yet, as Hawkins notes in On Intelligence, instead of asking questions about why regions that are known to serve very different functions are quite similar anatomically, anatomists had been peering even closer at cortical tissue, teasing apart the smallest of differences between functional regions—and they did find differences. Thicker layers here, more of a certain type cell there. But what Mountcastle observed in his 1978 paper was that, despite the pursuits of his colleagues who were searching for these relatively small differences between regions, the organization of the cortex is still remarkably consistent at all levels and across all regions. His conclusion was that the cortex must be doing basically the same thing in all regions, be it auditory, motor or otherwise—homologous anatomy equals homologous operation.
Thus, according to his paper, the key to understanding the way the cortex processes and stores information is not in anatomical differences between regions but in the different ways in which cells in each region are wired to each other and to the rest of the nervous system. Small differences in anatomy are more due to what a given region of the cortex is connected to than to differences in what it’s doing: what is crucial to the differences between regions of the cortex is more transportational than architectural. The brain processes information from the eye the same way it processes information from the ear—it’s just that the roads this information travels on leads to different regions of the cortex, and hence auditory versus visual cortex. This point is driven home by countless studies on neuroplasticity that elucidate the remarkable flexibility of these regions to process other sensory streams in cases of tissue loss or genetic malfunction.
The foundation of Hawkins’ On Intelligence, written in 2004 after he picked himself up from the Mountcastle-induced chair incident, is based on this elegant theory of a common algorithm linking all the cortical regions in the brain. Hawkins would go on to coin the term “Hierarchical Temporal Memory” to describe the flow of information up and down hierarchies of synaptic connections in the cortex, mirrored in the layered structure we often see in brain slices. His theory places as much emphasis on incoming sensory data as it does on the predictions constantly being formed by the brain that flow down the hierarchies, making each moment we live a complex equation of the raw data from the world around us combined with everything we’ve ever experienced before, resulting in a series of constantly forming predictions about what we’re about to encounter. It may not yet be the Einsteinian truth that will unify the brain sciences and explain the language of neurons, but the model is evolutionarily sensible—it’s anatomically based—and it’s a theory that Hawkins hopes to use for the development of smart, prediction-generating computers of the future by his new company Numenta.
What we’re concerned with here is not computers, but art. This Hawkins-Mountcastle model can be used here for a slightly different purpose: to move from an fMRI-based approach to a more cellular-based one in thinking about how the brain perceives art, modeled on the Hawkins-Mountcastle theory of the cortex. Some neuroscientists have made the first speculations in this direction—Ramachandran has included in his concept of the peak-shift phenomenon a cellular model, which I discussed in another article. But what the Hawkins-Mountcastle theory offers is a more complete, cross-sensory model of what’s going on in the architecture of the brain when art enters our perceptive arena. Most work in this field draws upon work from older artistic periods, so here I will focus mainly on modern examples of abstraction—but the principles can be applied across genres and forms, and through time.
In Clement Greenberg’s seminal essay on modernist painting, he defined the modernist tendencies as such:
“The Enlightenment criticized from the outside, the way criticism in its accepted sense does; Modernism criticizes from the inside, through the procedures themselves of that which is being criticized.”
- Clement Greenberg, Arts Yearbook 4, 1961.
Contemporary fMRI imaging of the brain while it perceives art is the practice of a current strain of Enlightenment-era neuroscience, a view of internal processes of the brain forever destined to be looking from the outside in.
Thanks to theories like the Hawkins-Mountcastle model and from insights to be gained from digital simulations of the brain, we are beginning to acquire the analytical tools necessary to move inside. The creation and perception of art involves the most definitively human of brain processes and architecture. So, to follow Greenberg’s definition, a modernist assessment of the brain and art must start from the internal structures themselves, must seek to understand the very physical substance and processes that encode the external art object, which flows from sensory perception to mix with the brain-world of memory, prediction, and emotion that awaits it.
Indeed, the Hawkins-Mountcastle model of the cortex places just as much emphasis on what’s actively waiting within as it does on what is finding its way into it. Hawkins explains, “What we perceive is a combination of what we sense and of our brains’ memory-derived predictions.” Though its processes remain buried in our sub-conscious brain, this bottom-up and top-down mixing is evident at every moment of conscious awareness. Hawkins again: “When we look at the world, we perceive clean lines and boundaries separating objects, but the raw data entering our eyes are often noisy and ambiguous. Our cortex fills in the missing or messy sections with what it thinks should be there.”
The mixing takes place in a hierarchical structure where sensory data flows up and memory-based predictions flow down, influencing what arrives in our conscious perception at every synapse. In figure 1, we see a model of this hierarchical structure as presented in the Hawkins-Mountcastle theory.
The key to this model and the incredible flexibility of the brain is the presence of invariant representations that are held at the top of these hierarchies. Invariance means that, while something like our visual category of an “airplane” can be triggered by a huge variance of incoming sensory data (a view of the rudder, a frontal view of a plane, just a few of the oval passenger windows, etc.) the category itself remains unchanged. We can get to the idea of an “airplane” in all sorts of ways, but once we identify this category, it clicks into place regardless of the input. The sensory data from the image at hand flows up the cortical hierarchy, guided at the cellular level by activation and inhibition ascending from the raw sensory data and descending from our learned, invariant representations that await it while simultaneously guiding it from above. This is something that computers cannot yet do, and it is a feature of how the brain encodes information that accounts for the incredible flexibility of its pathways and its proclivity for associations between these invariant representations.
We use the example of a photo-realist image of an airplane to briefly describe the concept of invariance in the brain—but what about a piece of abstract modern art? In an unintended way, the journey a piece of modern art takes through the brain—be it a Rothko color field, a Sol Lewitt sculpture, a Philip Glass song—is one that can help us turn the corner from Enlightenment neuroscience to modern neuroscience. For modern art, in its focus on the characteristic method of the discipline itself, can point out the characteristic things about the brain itself, the thing that created it and the thing in which is it perceived and appreciated. This is a methodological and a content-driven corner to turn.
So what are some of these characteristic things about the brain, understood through a piece of modern art that has entered its arena of perception, and taking into account the Hawkins-Mountcastle model of the cortex we’ve been working with?
- Modern art, in its tendency toward abstraction, does not depict anything less realistic than art that depicts a human form or any other place or object in a more photo-realistic manner. Rather, it is just depicting a different place in our brain: a place between the invariant (photo-realist) representations at the top of hierarchies, and the essential, raw sensory data of incoming input. A repetitive song of which the content is the structure of the medium itself (Philip Glass) or a color-field painting where visual stimuli brings us into the viewing experience (Rothko) hits a position lower on the Hawkins-Mountcastle hierarchies than visual art which depicts something in stark realism, music with lyrics that tell us precisely how to feel, an author who tells more than he shows. In those cases of hyper-realism, the input reaches the level of invariant representations—the top of a hierarchy of neurons—with more activation in recognizable, symbolic categories of people, places and things. In abstraction, the lack of categories of people, places and things to clearly guide the raw sensory input activates the more essential, mid-hierarchical level of representation in our brain, and thus we experience the greatest neuronal activation from the more essential features of the piece at hand—such as edge detection (Mondrian) or suggested movement (Pollock) in our visual system; repetitive melodies (Reich) in our auditory cortex. We may not know what it means because it hasn’t directly activated any invariant category, but nonetheless we like where it’s activating our cortical hierarchies– a feeling of pleasure that seems to rise up from nowhere in particular.
- It follows that, in our brain’s tendency to problem-solve and find meaning in the art we see, read or hear, we try to link these mid-hierarchical patterns of activation to our invariant representations contained higher up in the hierarchy—we see a cluster of amorphous shapes and think we see a certain animal; we lie on our backs and decipher the army of figures in the clouds above us. In this process, we bring to the table what we don’t get from the raw data: meaning is formed more from our internal memory stores and top-down invariant categories than from the raw data itself. It has been intuitively understood in art historical criticism that the distinctly modern trend of artistic abstraction involves the viewer’s own memory and active imagination to a greater degree. The Hawkins-Mountcastle model gives us a crucial grounding of this concept in the emerging understanding of the architecture of the brain.
- What of the age-old question of meaning in abstract art? We’ve already established that this art describes something no less “real” than more photo-realistic depictions, than most of the art of centuries prior. Rather, abstraction is primarily working in a different place in our cortex, asking more of top-down feedback from our personal stores of memory than from a bottom-up feed-forward pattern of activation that leads us to a precise set of associations and “meaning,” such as a nativity scene. Greenberg’s definition of modern art was, quite unintentionally, a definition of how modern art is handled by the brain—a focusing on the characteristic “building blocks” of a medium (and thus the building blocks of a hierarchy of neurons), at times for the meaning of the artwork itself. When we have the awkward “what does this mean” moment in a museum—when we turn to a friend and giggle about the meaning of a monochrome painting—we are nervous about the journey that piece of art would allow us to take through our own unconscious: up, down, and across our cortical hierarchies, associations forming beyond our conscious control.
These principles all point to a new way of thinking about the conversation between brain science and art. We know the brain is activated in all sorts of ways when we perceive art. The wonder of fMRI imaging in this art and brain dialogue is quickly diminishing as we move into the modern era of neuroscience, where digital, full-brain simulation models will become the standard. The cutting edge theories of brain architecture, as proposed in the Hawkins-Mountcastle model, will allow us to ask the endlessly fascinating question of where art goes when it enters our brain, and where our brain goes as a result. We can now take any piece of art and ride with it on its cell-hopping journey through the sensory systems, the thalamus, into the cortex, and back down again. We can marvel as memories, mood, and predictions come into play, modifying feed-forward activation; throwing unexpected, invariant representations down from above, coaxing our thoughts towards unexplored associations—and perhaps leading us to create something of our own.
Grounding art in the Hawkins-Mountcastle theory of the cortex may begin to answer some even larger questions. For example, certain IT cells (neurons in the inferior temporal cortex), which sit at the top of hierarchies in the ventral stream of our visual system and may code many of the invariant representations discussed above, could be the stepping stones for associative visual leaps in the brain, the rich ingredients of analogies made by moving between hierarchies and connecting invariant representations with one another, sometimes quite unconsciously. Such associations are fundamental to the brain—they constantly occur at synapses between cells, then at a more systems level between hierarchies, regions, and hemispheres. They drive the highest output of our symbolic thought.
Jeff Hawkins set out to solve something fundamental about the brain, and his theory of brain architecture and information processing may provide us with just that as we contemplate the neuroscience of art and the art of neuroscience. Fundamentally, art is when the brain associates with itself. And the human brain, aware of itself, cannot help but associate.