Sparse Distributed Representations

 

Although neurons in the neocortex are highly interconnected, inhibitory neurons guarantee that only a small percentage of the neurons are active at one time. Thus, information in the brain is always represented by a small percentage of active neurons within a large population of neurons. This kind of encoding is called a “sparse distributed representation”. “Sparse” means that only a small percentage of neurons are active at one time. “Distributed” means that the activations of many neurons are required in order to represent something. A single active neuron conveys some meaning but it must be interpreted within the context of a population of neurons to convey the full meaning.

 

HTM regions also use sparse distributed representations. In fact, the memory mechanisms within an HTM region are dependent on using sparse distributed representations, and wouldn’t work otherwise. The input to an HTM region is always a distributed representation, but it may not be sparse, so the first thing an HTM region does is to convert its input into a sparse distributed representation.

 

For example, a region might receive 20,000 input bits. The percentage of input bits

that are “1” and “0” might vary significantly over time. One time there might be

5,000 “1” bits and another time there might be 9,000 “1” bits. The HTM region could convert this input into an internal representation of 10,000 bits of which 2%, or

200, are active at once, regardless of how many of the input bits are “1”. As the

input to the HTM region varies over time, the internal representation also will change, but there always will be about 200 bits out of 10,000 active.

 

It may seem that this process generates a large loss of information as the number of possible input patterns is much greater than the number of possible representations in the region. However, both numbers are incredibly big. The actual inputs seen by

a region will be a miniscule fraction of all possible inputs. Later we will describe how a region creates a sparse representation from its input. The theoretical loss of

information will not have a practical effect.

 

 

Figure 1.4: An HTM region showing sparse distributed cell activation

 

Sparse distributed representations have several desirable properties and are integral to the operation of HTMs. They will be touched on again later.