In this post I will provide an account of my personal research project to understand the algorithms that underly human visual pattern-recognition. This project is multidisciplinary in nature, combining paradigms from three fields: (1) the analysis and synthesis of textures, (2) psychophysics and (3) psychedelic research. I will explain in detail how these areas can synergistically help us understand the computational properties of consciousness. In the process of doing so I will describe some of the work I have done in this direction.

tl;dr: With texture synthesis algorithms we can control the statistical features present in textures. By using an odd-one-out paradigm where participants have to find the “different texture” we can identify the statistical signatures of the visual patterns people can perceive. Collecting these signatures under various states of consciousness will reveal the information processing limitations of visual experience. It may turn out that some patterns can only be seen on LSD (psilocybin, mescaline, etc.), and this information will inform a general theory of vision’s algorithms, expanding the scope we have studied so far and suggesting relevant applications of psychedelic consciousness.

Introduction to Spatial Patterns

The world is patterned. In fact, it is so patterned, that it is difficult to identify natural surfaces that have no perceptible regularities. The grass, the trunk of trees, the surface of rocky mountains, the dancing and dissolving of the clouds. All of these natural scenes are full of regularities. For hundreds of millions of years animals on this planet have existed in an environment where regularities are not inconsequential: being able to use or detect camouflage is a matter of life or death to some species. The insects who hide between the leaves pretending to be part of the scene are in an evolutionary arms race against predators and their sensory apparatus. (Here is a neat collection of insect camouflage). Arguably, there are strong evolutionary selection pressures that push predator’s visual system into adapting to recognize the differences between the scene’s visual statistics and the prey’s body appearance.

Other widespread examples for the evolutionary relevance of pattern recognition can be found: Birds may take advantage of the look of cloud formations to determine if they should fly or find refuge, herbivores may seek only plants with specific visual properties to avoid poisonous lookalikes and parasites, the health of potential mates can be assessed by the uniformity of their fur patterns. You get the idea.

Not surprisingly, you today can look at a plain rock and see a lot of visual properties pointed out by patterns you can perceive. Unfortunately, something has kept us quiet about this aspect of our perception: most of these properties are hard to verbalize. Often, you will be able to tell apart two kinds of rocks by grasping the subtle visual differences between them, but at the same time still be unable to explain what makes them different.

What exactly is going on in your mind/brain when you are recognizing characteristic features in textures? We don’t know how the information is processed, why we perceive the features we perceive, or even how the various features are put together in a unified (or semi-unified) conscious representation. The hints we do have, however, are precious.

Receptive Fields

A big hint we can build on is that many neurons in the primary visual cortex (of cats, monkeys, and probably all mammals) respond to visual stimuli in specific areas of the visual field. For a given neuron, the shape of this region is an instance of a well-studied canonical function, as shown in the images below. The area of the visual field a neuron best responds to (by becoming excited or inhibited) is called its receptive field, and the canonical function is the Gabor filter.

As far back as the early 60s, research has been conducted to map the receptive fields of neurons by inserting electrodes into the brain of animals and presenting them with visual input of lines and shapes. In 1961 D. H. Hubel and T. N. Wiesel showed for the first time that neurons in a cat’s cortex and lateral geniculate have receptive fields that look like this:

The crosses indicate regions in which stimuli excites the neuron’s activity while the triangles represent regions in which stimuli inhibits the neuron’s activity. Due to the arrangement of these excitatory and inhibitory regions, these neurons functionally work as edge-detectors. A computer rendering of these receptive fields looks like this:

Since then a tremendous amount of research has repeatedly confirmed the existence of such neurons, and also uncovered a large number of more complex receptive fields. Some neurons even respond specifically to abstract concepts and high-level constructs. More recently, the simple receptive fields shown above have been modeled as Gabor filters in quantitative simulations. This, in turn, has been successfully used to build brain activity decoders that reconstruct the image that a person is seeing by assuming that the activity of fMRI’s voxels approximately matches the added activity of neurons with Gabor receptive fields (see: Identifying natural images from human brain activity by Kay, Naselaris, Prenger and Gallant).

Thus we can say that we know that a large number of neurons in our visual cortex have Gabor receptive fields and that the collective activity of these neurons contains enough information to at least approximately reconstruct the image a person is seeing (well enough to identify it from a pool of candidates).

We can’t jump from these findings alone to a global theory of visual processing. That is, without also considering what people actually experience. It may turn out, for example, that the activity of the visual cortex contains a lot of information that can be decoded using machine learning techniques applied to fMRI’s voxel brightness. And yet, simultaneously, it could be that people do not consciously represent all of the {decodable} information.

Likewise, a priori we cannot rule out the possibility that some of the information we consciously experience is not actually decodable using brain activity alone. (A quick remark: this may be the case even if one assumes physicalism. Why this is the case will be explained in a future article).

To illustrate this example we can consider the information available in the retina and before it. The light that reaches the outer surface of our eyes contains all of the information available to our mind/brain to instantiate our visual experience. Yet, there is a lot of information there that is ultimately irrelevant for our conscious experience. For instance, there is infrared and ultraviolet light, as well as light that does not make it to our retina, light that fails to elicit an action potential, and so on. If you can discriminate between a really hot and a cold metal using the infrared signature of the light that reaches the eye, you have certainly not shown that we perceive infrared light or that we use it to make distinctions. It merely means that such information is sufficient. We wouldn’t yet know that we actually use it or that it shapes our experience.

But how exactly do we develop an experiment to infer the statistical properties represented by our experience? Here is where the analysis and synthesis of textures becomes relevant.

Analysis and Synthesis of Textures

The idea of using oriented Gabor-like filters (also known as “steerable pyramids”) to analyze and synthesize visual textures was, as far as I know, first proposed by David J. Heeger and James R. Bergen in “Pyramid-based texture analysis/synthesis.” Texture analysis in this context means the use of algorithms to characterize the properties of textures to capture what makes them unique. In turn, texture synthesis, refers to the application of texture analysis to produce an arbitrarily large patch of a synthesized (synthetic) texture so that the synthetic and original texture are as indistinguishable as possible. Of course whether something is “indistinguishable” or not is to a certain degree subjective. Here the criteria for indistinguishability between the original and the synthetic textures is whether a person comparing them side by side could confuse them. In the following section I address how to operationalize and formalize the indistinguishability between patterns using psychophysics.

This particular texture synthesis algorithm works by forcing a white noise image (of any size) to conform to the statistics obtained in the texture analysis step. This is done iteratively, matching the histograms of the synthesized canvas to the various statistics computed from the original texture. I recommend reading the paper to gain a better grasp of what the algorithm does, and to see some stunning examples of the output of this algorithm.

The use of steerable pyramids was later refined by Portilla & Simoncelli‘s texture synthesis algorithm, which currently plays an important role in my research. This algorithm extends Heeger et al. by including additional statistics to enforce, which are (roughly) computed by measuring the autocorrelation between the various components of the steerable pyramid texture representation. Below you can see two pairs of pictures (two originals and their corresponding same-sized synthetics versions), that I recreated using Potilla & Simoncelli’s matlab code:

As you can see, the original and synthetic images are fairly similar to each other. Close inspection is sufficient to notice deadly differences. If you only use your peripheral vision it is very challenging to see major differences.

Psychophysics

Psychophysics is the study of the relationship between physical stimuli and experience (and often behavior). Thanks to psychophysics we now have:

1. A good approximate map of the phenomenal space of color (CIELAB)
2. A strong grasp of the nature of color metamers, which in turn underlies all of our color display technologies.
3. The ability to predict the subjective intensity of the experience elicited by stimuli as a function of the energy of the stimulus (see Weber–Fechner law).

A very powerful idea in psychophysics is the use of just-noticeable differences (JND): Carry a bucket of water. How much water should I add to it so that you are capable of perceiving a difference in the weight of the bucket? Look at a pair of identical light sources. How much blue (in this case, the specific and pure frequency of light that elicits the blue qualia) can I add to one of the lights before you perceive them as being differently colored? Pinch your skin with two needles. How far can their pointy part be before you perceive two needles rather than one?

In these cases, though, there is a natural (and sometimes unique) dimension along which the stimuli can be varied in order to compute the JND. What about visual textures? Here the problem becomes non-trivial. In what way should we change a texture? And how should we accomplish such? There is no clear and obvious scale for describing textures. So what can we do?

Psychophysics of Textures (Take 1):

Before learning about steerable pyramid representations of textures I developed a variety of psychophysical tests to identify the JND between textures. I did this by changing the value of parameters of images with ground-truth statistical properties. If you are curious, feel free to try the first iteration of my experimental paradigm. (Note: I am not collecting data from that experiment, so you should not expect to contribute to science by finishing the task. That said, you can have fun, and a pop-up window with your raw score will appear when you finish both tasks). The average accuracy of the texture discrimination part is about 13/21 (with a chance performance of 3/21), while the performance in the numerical pattern completion task is about 3/5 (with a near 0/5 expected performance when answering randomly).

One of the statistical properties I was changing in the stimuli was the variance of a 2D Gaussian process I implemented with a python script and the magic of Gibbs sampling. Thus, a subset of the images I tested were complete noise except for the first order statistic of local autocorrelation created by the differently parametrized Gaussian process. Here are a couple of examples of such patterns:

These pictures, it turns out, are relatively easy to tell apart when the parameters are sufficiently different. There is a threshold of similarity in the variance parameter after which people cannot distinguish between them. A nice property of this particular method is that you can be sure that if people do recognize the differences, it is because they are somehow extracting features that are a direct consequence of a different value for the variance parameter.

The textures above instantiate the simplest statistical differences that can be created, after the mean and standard deviation of the pixel values. To explore more broadly a wider variety of patterns, I also created textures with fairly complex parametrizable Turing machines (TMs). This, unfortunately, does not lend itself to a clear analysis. In brief, this is because changing a single parameter of such TMs can produce profoundly different results with unclear ground-truth properties:

What information could I gain from the fact that changing x, y or z parameter in a Turing machine by a, b, or c amounts enables accurate discrimination between textures? In principle one could try to explain the performance obtained by making an a posteriori analysis of the correlation between a variety of statistics measured in the textures. But since the textures were not created to have specific statistical properties, the distribution of such properties will not be ideal to find JNDs or discriminability thresholds.

Even if you do this, you will still have more problems. The images are more different (and different in more ways) than what your texture analysis algorithm is capable of detecting. Thus the reason why people can tell these textures apart is not possible to extract from the statistical differences you measure between them. At least not without being extremely lucky and hitting the visual features that matter.

And here, I was stuck several months.

Psychophysics of Textures (Take 2):

After learning about the steerable pyramid model, the work of Eero Simoncelli and the state of the art in fMRI decoding of visual experience, I decided to shift gears and approach the problem with some of the best of the tools created so far.

It turns out that the concept of texture metamers had been developed to describe the perceptual indistinguishability between textures in peripheral vision. Taken from here, the following two images are texture metamers. The images look the same when you center your vision in either of the central red dots. Close foveal inspection of the image, however, will reveal that these are very different pictures!

A specific study caught my attention, and I decided to replicate it as a final class project. Specifically, Texture synthesis and perception: Using computational models to study texture representations in the human visual system by Benjamin J. Balas. I attempted to replicate some of the main results of that study using Mechanical Turk. A full account of the experimental procedure, results and discussion can be found in this wiki here, (written for Stanford’s Psych 221 – Applied Vision Systems). To see the actual experiment performed, try it out here: Replication (and the github repository).

The main idea goes as follows: If textural metamerism can be verified using a given texture synthesis algorithm, then we can be reasonably certain that the algorithm is capturing a large component of what makes textures different from a human point of view. In particular, Benjamin Balas noted that Prtilla & Simoncelli’s algorithm could be “lesioned” by failing to enforce specific statistical features obtained in the texture analysis step. In this way, one can purposefully fail to match specific statistical features between the original and synthesized texture, and then measure how this partial texture synthesis algorithm affects the performance of texture discrimination in humans.

For illustration, here is a set of possible texture synthesis “lesions:”

The operationalization of the experiment used an “odd one out” paradigm: in each trial, three images are presented for 250 milliseconds at 3.5 degrees of vision from the fovea (each image being 2 degree wide in diameter). Two of the three images will come from the same group (ex. two original textures, two marginals removed, etc.). The remaining image is the odd one out. The study measures how often participants detect the odd one out depending on the statistical feature that is not enforced in the synthesized textures (for a more in-depth description: wiki).

Overall the performance of the participants in my replication was much closer to chance than Balas’ results. That siad, qualitatively the replication was successful. Removing the marginals is the lesion that increases performance the most. Magnitude correlation comes in second place. All of the other conditions are at chance level (as far as my sample n=60 with 60 trials each can discern).

As you can see, this specific paradigm is now much more robust than before. And the paradigm is not hard to translate for application in psychedelic research. Unfortunately, Portilla & Simoncelli’s algorithm only creates texture metamers for the peripheral vision in pre-attentive conditions. Upon central inspection, nearly every synthesized texture is at least somewhat distinguishable from the original texture.

The more interesting problem, as I see it, is found in our capacity to see differences between textures upon close and careful examination. This, I think, addresses more directly the subject of this blog. Namely, what is the information that we can represent and distinguish at the (resolution) peak of human consciousness? I imagine that there must be statistical features that we simply do not perceive even when we are looking at them directly and using all of our attention. What is the set of properties that our everyday consciousness can represent centrally?

Psychedelic Research

To understand how a machine works, it helps to know what happens when you break it. You can’t ignore the extra settings and claim you’ve got a theory of everything. Would we be satisfied with the work of neuroscientists and psychologists if they only studied people who are colorblind? They may claim that they have “the essentials” of vision. That color is just a perturbation in the “optimal basics.” And yet, today we know that color plays a relevant computational role in visual discrimination. Suffices to mention that people with grapheme-color synesthesia have an improved performance in “odd one out” identification tests like this (find all the 2’s as fast as possible):

This is because fast low-level association between graphemes and colors help them see clearly and quickly the graphemes that are different. Likewise, every variety of conscious experience may potentially play a computationally relevant role in specific situations. A priori, no variety of consciousness can be dismissed as irrelevant.

Thus, to understand, model and engineer consciousness, we should not prematurely close varieties of consciousness to study. Not only would that prevent us from finding out about consciousness more generally. It may also conspire that we will never understand even what we do decide to study, simply because key pieces of the puzzle are found elsewhere.

Another point is that even though sober human vision is a special case of general vision (and general vision-like qualia), general vision may still be relatively small. The general principles and conditions for vision to work in the first place may be somewhat restricted. Unless the elements are placed just right, the visual system breaks down, at least when it comes to fulfilling a computationally meaningful purpose. Thus, psychedelic research of visual experience may help us quickly distinguish what is essential from what is incidental in vision.

By introducing a psychedelic substance into the nervous system of a person, a remarkable set of visual effects are produced. To anyone interested in seeing a good representation of the way in which various psychedelics affect a person’s conscious experience I highly recommend seeing Disregard Everything I Say’s entry on the visual components of a psychedelic experience (while you are at it, I recommend also checking out his/her post on the corresponding cognitive components of a psychedelic experience). These images will provide a great intuition pump about the kind of beast we are facing to anyone who is psychedelic-drug-naïve (and hopefully inspire a sense of “WOW, you can represent some of what you experience after all!” in those who are less psychedelic-drug-naïve).

Given the same visual input presented to a particular person, a psychedelic substance will in many cases drastically shift the interpretation of such stimuli. Both the interpretation (specifically, what an image is about) and how the image looks and feels tend to vary in synchrony. Likewise, people feel more able to see personal issues in a new light by interpreting them with new schemas and from a different level of awareness. Personally, I suspect that there is a strong and measurable connection between the fluidity of interpretation of personal issues, and the fluidity of interpretation of visual experience. Perhaps all phenomenal constructs are affected in a similar way: By breaking down the previously enforced patterns of thought and perception and opening the way to seeing things differently.

All of the above, while probably true, is still far too vague for a scientific theory. Given our set of tools, and experimental paradigms, to me it makes a lot of sense to start studying the effects of psychedelics in terms of texture perception. As we develop more and better texture analysis/synthesis algorithms, we will acquire a larger repertoire of mathematical properties that describe what is seen during a psychedelic experience. My hope is that we will someday know exactly how to simulate a visual trip.

Why care about psychedelics? Evolution already created the optimal vision system!

Evolutionary selection pressures on perceptual systems do not guarantee that information processing tasks will be solved optimally. In fact, “optimal” only really makes sense in relation to some metric you choose. In some way, everything created by the evolutionary process will be optimal in the sense that is produces the local maximization of inclusive fitness (admittedly it’s more complicated than that). But this is just a tautological notion: Sure evolution is optimal at doing what evolution does. Likewise, rocks have found the optimal way to be themselves. Our visual system works optimally, if you define optimal in terms of 20/20 every-day visual experience.

Instead, it makes more sense if we focus on the specific computational trade-offs that various resource allocation methods and designs provide. We can certainly predict that the particular set of algorithms that our visual system employs to detect visual patterns will satisfy some properties. For instance, they will be good as survival tools in the African Savanna. But just as for our hardwired tastes in food (and our default emotional palette), survival value in the ancestral environment is not necessarily what we currently need or want. And just as it would make sense to modify what we enjoy eating the most (moving away from sugar) to adapt to the current post-industrial environment, it may also turn out to be the case that our visual system is miscalibrated for the tasks we want to solve, and the joys and meaning we would like to experience today.

Psychedelics change our visual system in many ways, some of them more predictable than others. Some people report that small doses of psychedelics increase one’s overall visual acuity (this has yet to be verified empirically). Although counter-intuitive at first, this cannot be ruled out, again, simply because of evolution does not rule out things of this nature. After all, one of the main constraints placed on animals in natural environments is caloric consumption. If a higher visual acuity is possible with your current brain, the marginal benefit such acuity adds may still not surpass the marginal loss of calories that result from excessive brain activity.

Additionally, while higher doses of psychedelics tend to impair many aspects of visual perception (with people on Erowid explaining how extreme tracers can make it hard to walk), low and moderate doses do not have simple one-sided effects even when it comes to accurately representing the world around us. Rather than simply breaking up the pattern recognition capabilities of the visual system, small and moderate doses seem to also open up additional kinds of patterns up for inspection. Pareidolia, for example, is greatly enhanced. Thus, “connecting dots and edges” to match the outline of higher-level phenomenologies (like faces in the mud) happens with ease. Whether this is good or bad depends on the context, and the specific task one is trying to solve.

Perceptual Enhancement via Psychedelics

We have theoretical and anecdotal reasons to suspect psychedelics may turn out to be performance-enhancing in certain visual tasks. We also already have quantitative evidence that this is the case. In the 60s Harman and Fadiman conducted a study about the creativity and problem-solving properties of psychedelics. The study included a paper and pencil component in which the following tests were administered: Purdue Creativity, the Miller Object Visualization, and the Witkin Embedded Figures. The authors conclude that “[m]ost apparent were enhanced abilities to recognize patterns, to minimize and isolate visual distractions, and to maintain visual memory in spite of confusing changes of form and color.” Specifically, the Witkin test is singled as a test in which the ingestion of mescaline produced a consistent performance improvement (p<0.01).

In personal communication Fadiman has said that these tests were mostly a waste of time: They used one valuable hour of the problem-solving session. I disagree. After 50 years, those results are incredibly valuable to me and the next wave of computational researchers of the mind. I think that the recorded performance enhancement is a key piece of information. We certainly do not expect an enhancement on all areas of cognition and perception (we know, for example, that reaction time and verbal fluency are impaired). Identifying the kinds of tasks that do receive a boost can inform future research. In future posts I will explain my theory for why Witkin Embedded Figures test was particularly benefited from a psychedelic, and how we can create a new test that takes this into account.

New Possible Protocols for Psychedelic Research

Nowadays it is very hard to obtain the required permits and affiliations to conduct academic research on psychedelic consciousness. The key rate-limiting factor is the restrictions that apply to research with controlled substances. Thankfully, we do happen to be living at the beginning of a psychedelic renaissance. It is not hard to imagine that as psychedelics start to (re-)enter the mainstream in psychiatry, a large number of clinical trials will be conducted. In principle, a collaboration can be accorded between a psychophysics lab and a clinical research group to conduct psychophysical assessments during the course of treatment.

Clinical trials for medical applications of psychedelics, though, are still limited in scope and focus. Eventually the medical applications of psychedelics will run out and it will no longer be possible to enroll patients into psychophysical studies. Thus, either more permissive rules and regulations will emerge and society’s rules will be more lax, or we will potentially endure decades or centuries of unnecessary barriers to scientific discoveries about consciousness.

As it turns out, tens (if not hundreds) of millions of persons have tried psychedelic substances. Interest is not slowing down, and no propaganda effort save for literal brain-washing will prevent people from developing a genuine interest in the field. There will be millions of volunteers and hundreds of thousands of willing researchers. As I envision in The Psychedelic Future of Consciousness, the future of consciousness studies will be nothing like what it is today. Once those who are interested are given the opportunity to study psychedelics seriously, the research panorama will look very differently.

But What to Do in the Meantime?

Investigating the psychophysics of texture perception more thoroughly may require at least mild to moderate doses for noticeable effects, and a period of examination at a time when acute effects are present. How to overcome the hurdles ahead of any current academic investigation of this nature? This ultimately depends on the specific constraints required for the study. It is true that people would have to be high on LSD (or Mescaline, mushrooms, 2C-B, etc.) while they perform the experiment. But who says that they have to conduct the experiment in your presence? That you have to give them the substance yourself? Without the typical background assumptions that are assumed in psychophysics research labs, can we do anything else?

I suspect that we can do much better. We can come up with protocols that side-step current obstacles. We need to be creative. And I do have some ideas, with varying levels of plausibility for how to implement psychedelic research in legal, viable and immediate ways. Consider, for example, how James Fadiman has been collecting hundreds of micro-dose reports via email without any difficulty for many years. He is taking advantage of the fact that the optimal format of micro-dose reports tends to be “summary and retrospective” narrative. A single dose on a single day on a single person is not likely to be life-changing. So his particular line of research is well suited for the means available to him. Likewise, as I requested in the psychedelic experience of visual textures post, people’s subjective judgements of visual textures while high on LSD can be recorded online. These are just two examples of how non-mainstream research approaches can be taken to study psychedelics. Unfortunately, both of these protocols lack proper controls and standardized settings. But this is all I will say for now.

Stay tuned: In a future post I will propose a set of protocols for independent studies on psychedelics, including additional methods for studying psychedelic visuals.

Thanks for reading! 

If you would like to be a collaborator with me, please email me by finding my contact info in the contact section of this blog. We’ll take it from there.

Note: If any link is broken, please leave a comment and I’ll provide an updated version. Thanks!