8.2.1 Sensory memory

Information impinging on the senses initially goes into what is called sensory storage. Sensory storage is the direct impression, the sensory information, that comes from the sense organ. The sensory register can keep information from each sense organ for only between some dozens to several hundreds of milliseconds. In the sensory register, information can be noticed, the meaning can be encoded, and then it can enter the next step in processing. If it cannot be noticed or encoded, it will disappear automatically.

All kinds of sensory information continue keeping up for some time and working in its characterized form in the sensory register. These forms are called iconic store and echoic store. Representation can be said to be the most direct, most primitive memory. Mental image can only exist for a short time, if the most distinct video lasts only dozens of seconds. Sensory memory possesses the following characteristics:

George Sperling’s research verified the concept of sensory memory [4]. In this research, Sperling flashed an array of letters and numbers on a screen for a mere 50 ms. Participants were asked to report the identity and location of as many of the symbols as they could recall. Sperling could be sure that participants got only one glance because previous research had shown that 0.05 s is long enough for only a single glance at the presented stimulus.

Sperling found that when participants were asked to report on what they saw, they remembered only about four symbols. The number of symbols recalled was pretty much the same, without regard to how many symbols had been in the visual display. Some of Sperling’s participants mentioned that they had seen all the stimuli clearly, but while reporting what they saw, they forgot the other stimuli. The procedure used initially by Sperling is a whole-report procedure. Sperling then introduced a partial-report procedure, in which participants needed to report only part of what they saw.

Sperling found a way to obtain a sample of his participants’ knowledge and then extrapolated from this sample to estimate their total knowledge of course material. Sperling presented symbols in three rows of four symbols each. Sperling informed participates that they would have to recall only a single row of the display. The row to be recalled was signaled by a tone of either high, medium, or low pitch, corresponding to the need to recall the top, middle, or bottom row, respectively. Participants had to recall one-third of the information presented but did not know beforehand which of the three lines they would be asked to report.

Using this partial-report procedure, Sperling found that participants had available roughly 9 of the 12 symbols if they were cued immediately before or immediately after the appearance of the display. However, when they were cured 1 s later, their recall was down to four or five of the 12 items, about the same as was obtained through the whole report procedure. These data suggest that the iconic store can hold about nine items, and that it decays very rapidly. Indeed, the advantage of the partial-report procedure is drastically reduced by 0.3 s of delay and is essentially obliterated by 1 s of delay from onset of the tone.

Sperling’s results suggest that information fades rapidly from iconic storage. Why are we subjectively unaware of such a fading phenomenon? First, we are rarely subjected to stimuli that appear for only 50 ms and then disappear, for which we need to report. Second and more important, however, we are unable to distinguish what we see in iconic memory from what we actually see in the environment. What we see in iconic memory is what we take to be in the environment. Participants in Sperling’s experiment generally reported that they could still see the display up to 150 ms after it had been terminated.

In 1986 Professor Xinbi Fan et al. proposed a visual associative device, auditory associative device, and tactile associative device [5]. Furthermore, an advanced intelligent system with reasoning and decision can be constructed by using various associative devices. The parallel search of associative devices provides the possibility of high-speed data processing. In short, associative devices can be used as components of new computers.

8.2.2 Short-term memory

Information encoded in sensory memory enters short-term memory, through further processing, and then enters into the LTM in which information can be kept for a long time. Information generally is kept in short-term memory only for 20–30 s, but if repeated, it can continue there. Repeating guarantees the delay of disappearance of the information stored in short-term memory—an important function in psychological activity. First, short-term memory is acting as part of consciousness, enabling us to know what we are receiving and what is being done. Second, short-term memory can combine some sensory messages into an intact picture. Third, short-term memory functions as a register temporarily while the individual is thinking and solving a problem. For example, before making the next step in a calculation question, people will temporarily deposit the latest results of the calculation for eventual utilization. Finally, short-term memory keeps present tactics and will. All of these enable us to adopt various complicated behaviors until reaching the final goal. Even though we find these important functions in short-term memory, most present research renames it working memory. Compared with sensory memory, which has much of the information to be used, the ability of short-term memory is quite limited. Participants were given a piece of figure bunch, such as 6-8-3-5-9, which they should be able to recite immediately. If it is more than seven figures, people cannot recite them completely. In 1956 George A. Miller, an American psychologist proposed clearly that our immediate memory capacity for a wide range of items appears to be about seven items, plus or minus 2. To chunk means to combine several little units into a familiar and heavy unit of information processing; it also means the unit that is made up like this. Chunk is a course and a unit. As for that knowledge experience and the chunk, the function of the chunk is to reduce the unit in short-term memory and to increase information in each unit. The more knowledge that people have, the more messages that can be in each chunk. At the same time, the chunk is not divided by meaning; that is, there is no meaning connection between the elements of the composition. So to remember a long figure, we can divide it into several groups. It is an effective method is to reduce the quantity of independent elements in the figures. This kind of organization, called a chunk, plays a great role in LTM.

It has been pointed out that information was stored in short-term memory according to its acoustic characteristic. In other words, even if a message is received by vision, it would be encoded according to the characteristics of acoustics. For example, when you see a group of letters B-C-D, you are according them their pronunciation [bī:]-[sī:]-[dī:] but not according them their shape to encode.

Fig. 8.2 provides the short-term memory buffer. Short-term memory encoding in humans may have a strong acoustical attribute but cannot be rid of the code of other properties. Many who cannot speak can do the work of short-term memory. For example, after seeing a figure, they will select one of two colorful geometric figures, consisting of several slots. Every slot is equivalent to an information canal. The informational unit coming from sensory memory enters different slots separately. The repeated process of the buffer selectively repeats the information in the slot. The information repeated in the slots is entered into LTM. Information that is not repeated is cleared from short-term memory and disappears.

The times for information in every slot differ. The longer the time information is kept in the slot, the greater the chance that it will enter LTM, and the greater the chance that it may be washed and squeezed out by a new message coming from sensory memory. Comparatively, LTM is the real storehouse of information, but information can be forgotten by subsidizing, interference, and intensity.

Retrieving the course of short-term memory is quite complicated. It involves a lot of questions and gives rise to different hypotheses. There is no consensus so far.

8.2.2.3 Double model

R. Atkinson and J. Juola think the retrieval of information already includes scanning and direct access during short-term memory. In brief, both ends are direct, and the middle is scanning, as shown in Fig. 8.3.

The search model and direct access model both have their favorable aspects, but there is a deficiency. The double model attempts to combine the two. Atkinson and Juola put forward the notion of information retrieval in the double model in short-term memory [8]. They imagine that each of the words is input that can be encoded according to its sensorial dimensionality, called sensory code; words has a meaning, called its concept code. Sensory code and concept code form a concept node. Each concept node has a different level of activation or familiarity value.

There are two standards of judging within the brain. One is high-standard (C1): If the familiarity value of a certain word is equal to or higher than this standard, people can make a rapid “yes” response. The other one is the low standard (C0): If the familiarity value of a certain word is equal to or lower than this standard, people can make a rapid “deny” response. Atkinson and Juola state that this is a direct access course. However, if a survey word with a familiarity value lower than the high-standard but higher than the low standard, it will carry on a serial search and make a reaction. So the reaction times are greater.

The research on short-term memory finds that processing speed and material properties or information type of information have relationships. The speed of processing rises with increasing memory capacity; the bigger the capacity of the material, the faster it will scan.

Experimental results indicate that processing speed in short-term memory has a relationship with material properties or information type. In 1972 Cavanaugh, through calculating the average experimental results with materials in different research works, obtained the average time taken for scanning one item and contrasted the corresponding capacity of short-term memory. An interesting phenomenon was found: The processing speed rises with an increase in the memory capacity. The greater the capacity of the material, the faster of scanning is. It is difficult to explain this phenomenon clearly. One imagining is that in short-term memory, information is signified with characteristics. But the storage space of short-term memory is limited; so the larger that each average characteristic is in quantity, then the smaller the quantity will be of free short-term memory. Cavanaugh then thinks that each stimulus’s processing time is directly proportional to its average characteristic quantity. If the average characteristic quantity is heavy, then it will need more time to process; otherwise less time is needed. Also, there are many doubtful areas in this explanation, even though it connects the information retrieval of short-term memory, memory capacity, and information representation. This is really an important problem. The processing speed reflects the characteristic of the processing course. Behind the processing speed difference of different materials, the cause may be that memory capacity, information representation, etc. have different courses of information retrieval.

Through the Peterson-Peterson method, the information that was forgotten in short-term memory is discovered [9].

1. 1. Information can keep 15–30 s in short-term memory.
2. 2. If cannot be repeated; so the information in short-term memory will be forgotten rapidly.
3. 3. So long as the quantity in short-term memory does not change, the change of material property has no influence on short-term memory.

E. R. Kandel found through his research that cAMP plays a very important role in repairing brain cells, activating brain cells, and regulating the function of brain cells, turning short-term memory into LTM [1].

8.2.3 Long-term memory

Information maintained for more than a minute is referred to as LTM. The capacity of LTM is the greatest of all memory systems. Experiments are needed to illustrate the capacity, storage, restoration, and duration of LTM. The result measuring how long one thing can be memorized is not definite. Memory cannot last for long time because attention is unstable; if accompanied by repetitions, however, memory will be retained for long time. The capacity of LTM is infinite. Some 8 s are needed to retain one chunk. Before being recovered and applied, information stored in LTM needs to be transferred into short-term memory. During the phase of LTM recovery, the first number needs 2 s; then every following number needs 200–300 s. Different figure numbers, such as 34, 597, 743218, can be used for the experiment to measure how long each number needs to be recovered. The result indicates that a two-figure number needs 2200 ms, a three-figure number needs 2400 ms, and six-figure number needs 3000 ms.

The Bluma Zeigarnik effect results from instructing a subject to complete some tasks [10] while leaving other tasks unfinished. Afterward, the subjects can remember unfinished work better than the finished work when they are asked to remember what they had done. It was revealed that, if a task is stopped before being finished, the task continues activity in another space. Some things are easier to retrieve than others in LTM because their threshold is lower. Other things with high thresholds need more clues to retrieve. The unfinished task’s threshold is low and can be activated and spread easily. Activation spreads along the net until the location stores information, and then the information is retrieved.

A lot of psychologists agree that sensory memory and short-term memory will disappear fast and become passive, but few psychologists agree there is the same simple decline mechanism in LTM because it is difficult to explain why some materials are forgotten slightly faster than others? Does the completion intensity of the original study material relate to forgetting? Is being forgotten affected by something that happens during the time between studying and remembering? Many psychologists in this area believe that LTM disappears because of interference. This is a passive point. There are some points about being forgotten as an active process as a supplement or substitution for interference. Floyd thinks being forgotten results from constraining. The material will be difficult to remember if it is extremely painful and threatening to the mind. Another standpoint is the “productive forgotten” proposed by Bartlett. If you have not received accurate memory, you will create something like the memory, and then you will get to the real memory. The characteristics of these three kinds of memory systems are provided in Table 8.1.

The human memory system is very much like that of computer. Computer memory layers consist of cache, main memory, and auxiliary memory, whose speed runs from fast to slow and whose capacity runs from small to large, with the optimum control deployment algorithm and reasonable cost composing an acceptable memory system.

8.3.1 Semantic memory

Semantic memory was proposed by M. Ross Quillian in 1968 [11]. It is the first model of semantic memory in cognitive psychology. Anderson and Bower, Rumelhart, and Norman all had proposed various memory models based on the semantic network in cognitive psychology. In this model, the basic unit of semantic memory is the concept that has certain characteristics that are also concept in fact, but they are used to explain other concepts. In the semantic network, information is represented as a set of nodes that are connected with each other by an arc with a mark that represents the relationship between nodes. Fig. 8.4 is a typical semantic network. The is a link is used to represent the layer relationship of the concept node and to link the node-represented specific object with a relevant concept is part links the concepts of global and partial. For example, in Fig. 8.4, Chair is a part of Seat.

8.3.1.1 Hierarchical network model

The hierarchical network model for semantic memory was proposed by Quillian et al. In this model, the primary unit of LTM is concept. Concepts are related to one another and then form a hierarchical structure. As shown in Fig. 8.5, the block is a node representing concept, and the line with an arrow point expresses the dependence between the concepts. For instance, the higher hierarchical concept of bird is animal, while its lower hierarchical concept is ostrich. The lines represent the relationships between concept and attribute to designate the attribute of each hierarchical, for example, has wing, can fly, and has feathers are features of bird. Nodes represent the respective hierarchical concepts, and concept and feature are connected with lines to construct a complicated hierarchical network in which lines are association with a certain significance in fact. This hierarchical network model stores features of concept in a corresponding hierarchy that only stores the concepts within same hierarchy, while the common attributes of concepts in the same hierarchy are stored in at a higher hierarchical level.

Fig. 8.5 is a fragment of a concept structure from the literature [12]. Concepts such as “canary” and “shark,” which are located at the bottom, are called level-0 concepts. Concepts “bird”, “fish,” and so on are called level-1 concepts. Concept “Animal” is called a level-2 concept. The higher the level is, the more abstract the concepts are, and correspondingly the longer the processing time is. At each level, this level only stores the unique concept characteristics. Thus the meaning of a concept is associated with other characteristics determined by the concept.

8.3.1.2 Spreading activation model

The spreading activation model was proposed by Collins et al. [13]. It is also a network model. Different from the hierarchical network model, this model organizes concepts by semantic connection or semantic similarity instead of hierarchical structure of concept. Fig. 8.6 reveals a fragment of the spreading activation model. Those squares are nodes of network representing concepts. The length of lines means a compact degree of relation—for example, a shorter length indicates that the relationship is close and that there are more common features between two concepts—or if more lines between two nodes with common features mean their relationship is compact, the connected concepts denote their relationship.

As a concept is stimulated or processed, the network node that this concept belongs to is active, and activation will spread all around along lines. The amount of this kind of activation is finite, so if a concept is processed for a long time, the time of spread activation will increase and the familiarity effect may be formed; on the other hand, activation also follows the rule that energy decreases progressively. This model is a modification of the hierarchical network model, which considers that attributes of every concept may have the same or different hierarchies. The relation of the concept-length of lines illustrates the category size effect and other effects and is the form of spreading activation. This model may be a humanized hierarchical network model.

8.3.1.3 Human association memory

The greatest advantage of the human associative memory (HAM) model is that this model can represent semantic memory as well as episodic memory; it can process semantic information and nonverbal information. What is more, this model can offer a proper explanation of the practice effect and imitate it through the computer very well. But it cannot explain the phenomenon of familiarity effect. The comparison process is composed of several stages and asserts that the basic unit of semantic memory is proposition rather than concept.

A proposition is made up of a small set of associations, and each association combines two concepts. There are four different kinds of association. (1) Context–fact association: The context and fact are combined into associations, in which facts refer to events that happened in the past, and context means the location and exact time of the event. (2) Subject-predicate association: The subject is the principal part of a sentence, and the predicate is intended to describe the specialties of the subject. (3) Relation–object association: This construction served as predicate. Relation means the connection between some special actions of the subject and other things, while the object is the target of actions. (4) Concept–example association: For instance, furniture-desk is a concept-example construction.

Proper combination of these constructions can make up a proposition. The structures and processes of HAM can be described by the propositional tree. When a sentence is received, for example, “The professor ask Bill in classroom,” it could be described by a propositional tree (see Fig. 8.7). The figure is composed of nodes and labeled arrows. The nodes are represented by lower-case letters and the labels on the arrows by upper-case letters, while the arrows represented various kinds of association. Node A represents the idea of proposition, which is composed by facts and associations between contexts. Node B represents the idea of the context, which could be further subdivided into a location node D and a time code E (past time, for the professor had asked). Node C represents the idea of fact, and it leads by arrows to the subject node F and a predicate node G, which could in turn be subdivided into a relation node H and an object node I. At the bottom of the propositional tree are the general nodes that represent our ideas of each concept in the LTM, such as classroom, in the past, professor, ask, Bill. They are terminal node for their indivisibility. The propositional tree is meaningless without these terminal nodes. Those concepts are organized in a propositional tree according to the propositional structure, rather than their own properties or semantic distance. Such an organization approach possesses a network quality. The LTM could be regarded as a network of propositional trees, which also demonstrate the advantage of the HAM model in that it can represent both semantic memory and episodic memory and combine them together. This propositional tree can comprise all kinds of personal events as long as the episodic memory information is represented with a proposition, which is absent in those memory models previously mentioned. HAM propositional structure enables one proposition implanted into another and combined into a more complicated proposition. For example, the two propositions that “The professor asked Bill in classroom,” and “it makes the examination over on time” could be combined into a new proposition. In this situation, these two original propositions become the subject and predicate of the new proposition, respectively. This complicated proposition could also be represented with a propositional tree.

According to the HAM model, four stages are needed to accomplish the process of information retrieval to answer a question or understand a sentence:

1. 1. Input a sentence.
2. 2. Analyze the sentence and generate a propositional tree.
3. 3. Search from each related node in long-term memory, until a propositional tree that matches the input proposition is found.
4. 4. Match the input propositional tree with that found in long-term memory.

8.3.2 Episodic memory

Episodic memory was proposed by Canadian psychologist Tulving. In his book Elements of Episodic Memory published in 1983, Tulving discussed the principle of episodic memory [14].

The base unit of episodic memory is personal recall behavior. Personal recall behavior begins with an event or a subjective reproducing (remembering experience) of an experience produced by a scene, or a change to other forms that keep information, or a combination of them. There are a lot of composition elements and the relations among the elements about recall. The composition elements that are elements of episodic memory can be divided into two kinds: one observes the possible incident, and the other made up of composition concepts of the hypothesis. The elements of episodic memory consist of encoding and retrieval. Encoding is about the information of the incident of the experience in a certain situation at some given time; it points out the process of transforming to the traces of memory. Retrieval is mainly about the form and technique of retrieval. The elements of episodic memory and their relations are demonstrated in Fig. 8.8.

Nuxoll and Laird first introduced scene memory into Soar episodic memory in such a way that a scene memory can support a variety of cognitive functions agents [15]. In CAM, the plot fragment is stored in the basic module scenario memory. Plot segment is divided into two levels: the logical level of abstraction and the original level, shown in Fig. 8.9. The abstraction level is described by logic symbols. In the original stage, it includes a description of the related object abstraction–level perceptual information. In order to effectively represent and organize the plot fragments, we use dynamic description logic (DDL) for representation in the abstract level and make use of ontology in the original level. Furthermore, we use an object data map to describe plot fragments.

Fig. 8.10 depicts the structure of the object data graph for the film Waterloo Bridge, in which the plot is associated with other objects by URI fragments. Fig. 8.10 shows the three objects: M₂, W₂, and the movie Waterloo Bridge. In addition, there is object W₂ wearing a blue skirt. The film also associated with the two main characters M₁, W₁, and W₁ in a white coat.

In Soar, the plot is the case retrieval model–based reasoning, based on past experience, to find solutions to problems [16]. According to the clues in back episodes, we follow this idea to build a case-based system. To simplify the system, we restrict the plot to be like a change sequence. Then the plot retrieval model is aimed at solving the most relevant search event question. In the abstract level plot, it can accurately represent the contents of the event, thus matching the level of abstraction between the tips and the plot. In CAM, the conversion sequence is formally defined as a possible sequence of the world, and the plot clues suggest a tableau by DDL inference algorithm [17,18]. One possible sequence of the world is a directed acyclic graph Seq=(W_p, E_p),where each w_i∈W_p represents a possible world, and each edge e_i=(w_i, w_j) indicates that the action α_i is executed in w_i and consequently causes the translation from the world w_i to the world w_j.

With DDL, a plot e_p implicitly indicates c if the formula e_p→c is valid. Therefore, we can check whether e_p→c is about the implication relations of effective process between plot and clues. The inference procedure is described in Algorithm 8.1.

Algorithm 8.1 CueMatch(e,c)

Input: episodee,cuec

Output: whetherc$\preccurlyeq$_p e hold

(1) if length(e) <length(c) then

(2) return false;

(3) end

(4) n_e:=first_node(e);

(5) n_c:=first_ node(c);

(6) if MatchPossibleWorld(n_e; n_c) then

(7) α_e:=Null;

(8) α_c:=action(n_c);

(9) if ¬ (Pre(α_e) → Pre(α_c)) unsatisfiable according DDL tableaualgorithm then

(10) n_tmp:=n_e;

(11) while next_ node(n_tmp) ≠Null do

(12) α_e:=(α_e; action(n_tmp));

(13) if MatchAction(α_e; α_c) then

(14) Let sub_e be the subsequence by removing α_e from e;

(15) Let sub_c be the subsequence by removing α_c from c;

(16) if CueMatching(sub_e; sub_c) then

(17) return true;

(18) end

(19) end

(20) n_tmp:=next node(n_tmp);

(21) end

(22) end

(23) end

(24) Remove n_e from e;

(25) return CueMatching(e; c);

Function: MatchPossibleWorld(w_i,w_j)

Input: possible worlds w_i, w_j

Output: whetherw_i$\models$w_jhold

(1) f_w:=Conj(w_i)→ Conj(w_j);

(2) if ¬f_w is unsatisfiable according to DDL tableau algorithm then

(3) return true;

(4) else

(5) return false;

(6) end

Function: MatchAction(α_i,α_j)

Input: actionα_i, α_j

Output: whether α_i$\models$αj hold

(1) if α_i==null or α_j==null then

(2) return false

(3) end

(4) f_pre:=Conj(Pre(α_i)) → Conj(Pre(α_j));

(5) f_eff:=Conj(Eff(α_i)) → Conj(Eff(α_j));

(6) if ¬f_pre and ¬f_eff are unsatisfiable according to DDL Algorithm then

(7) return true;

(8) else

(9) return false;

(10) end

In Algorithm 8.1, the function length returns the length of a possible sequence of the world. The length of a possible world sequence is determined by the number contained in the sequence of nodes. Function returns (n), and the implementation of the action is done in a possible world order N inside. Function next node return n′ next node in the possible world’s sequence. Next node n, by performing the operation of operation (n), reaches the possible sequences of the world. Algorithm 8.1, step 14, through the sequence of actions constructors “;”, produce a combined action, connecting the two movements α and action (n_tmp). To simplify the algorithm, we assume that the action (NULL;) == α. In Algorithm 8.1, MatchPossibleWorld() and MatchAction() are also functions. We use the DDL tableau algorithm to decide whether the negation of the formula is satisfiable or not.

8.3.3 Procedural memory

Procedural memory refers to the memory of how to do things, including perception skills, cognitive skills, motor skills, and memory. Procedural memory is memory of inertia; also known as memory skills, it is often difficult to use language to describe such memories, and often several attempts are needed to gradually get it right.

Procedural memory is the memory of procedural knowledge; that is, “know-why” is about the skills of knowledge and understanding of “how-to” memories. Individuals achieve a degree of automation and refinement via much practice then you can do two things at the same time without feeling labored, such as using a mobile phone while driving or driving when the program remains natural and smooth, that is, when the a call does not affect driving.

Procedural knowledge presumes knowledge of its existence only by means of some form of indirect jobs. It includes inspiration, methods, planning, practices, procedures, practices, strategies, techniques, and tricks to explain “what” and “how.” It is about how to do something or knowledge about the link between stimulus and response but also about the behavior of the program or basic skills learning. For example, people know how to drive a car, how to use the ATM, how to use the Internet to search for target information.

Procedural knowledge representation in ACT-R, Soar, and other systems using production systems, that is, a “condition–action” rule. An action procedure takes place when certain conditions are met. Output refers to the coupling condition and action, resulting in an action under the rules of a condition; it consists of the condition items “if” (if) and the action item “then” (then).

We adopt the DDL for describing the procedural knowledge of CAM. To be compatible with the action formalism proposed by Baader et al. [19], we extend the atomic action definition of DDL to support occlusion and conditional postconditions. With respect to a TBox T, the extended atomic action of DDL is a triple α≡ (P, O, E) [20], where:

As an example, an action named BuyBookNotified(Tom, Kin) is specified as follows: BuyBookNotified(Tom,Kin)≡

({customer(Tom), book(KingLear)}, { },

{instore(KingLear)/bought(Tom, KingLear),

instore(KingLear)/¬instore(KingLear),

instore(KingLear)/notify(Tom, NotifyOrderSucceed),

¬instore(KingLear)/notify(Tom, NotifyBookOutOf Stock)})

According to this description, if the book KingLear is at the store before the execution of the action, then after the execution of the action, the formulas bought(Tom, KingLear), ¬instore(KingLear) and notify(Tom, NotifyOrderSucceed) will hold; otherwise the formula notify(Tom, NotifyBookOutOfStock) will be true, which indicates that Tom is notified that the book is out of stock.

The difference between declarative memory and procedural memory is that declarative memory works for the description of things, such as “self-introduction,” which is an explicit memory, while procedural memory works for technical actions, such as “cycling,” which is an implicit memory. The organs for procedural memory are the cerebellum and striatum, and the most essential part is the deep cerebellar nuclei.

8.3.4 Information retrieval from long-term memory

There are two ways of information retrieval from long-term memory (LTM): recognition and recall.

8.4.1 Working memory model

All working memory models can be roughly divided into two big classes: One is the European traditional working memory model, among which the representative one is the multicomposition model brought forward by Baddeley, which divided the working memory model into many subsidiary systems with independent resources, stress modality-specific processing, and storage. The other is the North American traditional working memory model, which is represented by the ACT-R model, emphasizing the globality of working memory, general resource allocation, and activation. The investigation of the former mainly focuses on the storage component of the working memory model, that is, phonological loop and visuospatial sketchpad. Baddeley pointed out that short-term storage should be clarified first for it to be operated easily before answering more complicated questions of processing, whereas the North American class emphasizes the working memory’s role in complicated cognitive tasks, such as reading and speech comprehension. So the North American working memory model is like the European general central executive system. Now two lines of research are coming up with more and more findings and are exerting a mutual influence with respect to theory construction. For example, the concept of the episodic buffer is like the proposition representation of Barnard’s interacting cognitive model. So the two classes have already demonstrated certain integrated and unified trends.

Baddeley developed his working memory theory in recent years based on the traditional model in which a new subsystem-episodic buffer is increased [21]. Baddeley has suggested that the traditional model does not notice how the different kinds of information are combined and how the combined results are maintained, so it cannot explain how subjects could only recall about 5 words in the memory task in random word lists, but they could recall about 16 words in the memory task according to prose content. The episodic buffer represents a separate storage system that adopts a multimodal code; it offers a platform where information is combined temporarily in a phonological loop, visuospatial sketchpad, and LTM and integrates information from multiple resources into an intact and consistent form through the central executive system. The episodic buffer, phonological loop, and visuospatial sketchpad are equally controlled by the central executive system, which, though the integration of different kinds of information, maintains and supports the subsequent integration by the episodic buffer. The episodic buffer is independent of LTM, but it is a necessary stage in long-term episodic learning. The episodic buffer can explain questions such as the interference effect of serial position recall, the mutual influence question among speech and visuospatial processing, the memory trunk and unified consciousness experience, etc. The four-component model of working memory, including the newly increased episodic buffer, is shown in Fig. 8.11 [21].

The ACT-R model from Lovett and his colleagues can explain a large number of data with individual difference [22]. This model regards working memory resources as one kind of attention activation, named source activation. Source activation spreads from the present focus-of-attention to the memory node related to the present task and conserves those accessible nodes. ACT-R is a production system that processes information according to the activation production regularity. It emphasizes that the processing activities depend on goal information: The stronger the present goal is, the higher the activation level of relevant information will be, and the more rapid and accurate information processing is. This model suggests that the individual difference of the working memory capacity reflects total amount of “source activation” expressed with the parameter W. And it is field-universal and field-unitary. This source activation in phonological and visuospatial information is based on the same mechanism.

The obvious deficiency of this model lies in the fact that it explains only a parameter of the individual difference in a complicated cognitive task and neglects that the individual difference of working memory might be related to processing speed, cognitive strategy, and past knowledge skill. But the ACT-R model emphasizes the singleness of working memory, in order to primarily elucidate the common structure in detail; it thereby can remedy the deficiency of the model emphasizing the diversity of working memory.

8.4.2 Working memory and reasoning

Working memory is closely related to reasoning and has two functions: maintaining information and forming the preliminary psychological characteristics. The representation form of the central executive system is more abstract than the two subsystems. Working memory is the core of reasoning, and reasoning is the sum of working memory ability.

According to the concept of the working memory system, a dual-task paradigm is adopted to study the relationship between the components of work memory and reasoning. The dual-task paradigm means that two kinds of tasks are carried out at the same time: One is the reasoning task; the other is the secondary task used to interfere with every component of working memory. The tasks to interfere with the central executive system are to demand subjects to randomly produce the letter or figure or to utilize sound to attract subjects’ attention and ask them to make corresponding responses. The task to interfere with the phonological loop cycle is to ask subjects to pronounce constantly such as “the, the …” or to count numbers in a certain order, such as 1, 3, 6, 8. The task to interfere with the visuospatial sketchpad is a lasting space activity, for example typewriting blindly in certain order. All secondary tasks should guarantee certain speed and correct rate and conduct the reasoning task at the same time. The principle of the dual task is that two tasks compete for the limited resources at the same time. For example, interference in the phonological loop is to make the reasoning task and the secondary task take up the limited resources of the subsystem in the phonological loop of working memory, both at the same time. If the correct rate of reasoning is decreased and the response time is increased in this condition, then we can confirm that the phonological loop is involved in reasoning process. A series of research works indicate that the secondary task could effectively interfere in the components of working memory.

K. J. Gilhooly studied the relationship between reasoning and working memory. In the first experiment, it was found that the way of presenting the sentence influenced the correct rate of the reasoning: The correct rate was higher in visual presentations than in hearing presentations, the reason for which is that the load of memory in visual presentations was lower than in hearing presentations. In the second experiment, it was found that the deductive reasoning task presented visually was most prone to be damaged by the dual-task paradigm (memory load) used to interfere with the executive system, next by the phonological loop, and least by the visuospatial processing system. This indicates that representation in deductive reasoning is a more abstract form, which is in accordance with the psychological model theory of reasoning and has caused the central executive system to be involved in reasoning activities. Probably, the phonological loop played a role too because the concurrent phonological activities with the reasoning activity slowed down, indicating that two kinds of tasks may compete for the limited resource. In this experiment, Gilhooly and his colleagues found that the subjects may adopt a series of strategies in deductive reasoning and that the kind of strategy adopted can be inferred according to the results of reasoning. Different secondary tasks force the subjects to adopt different strategies, so their memory load is different too. Vice versa. Increasing the memory load will also change the strategy because the changing strategy causes the decreased memory load.

In 1998, Gilhooly and his colleagues explored the relationship between the components of working memory and deductive reasoning in presenting the serial sentence visually using the dual-task paradigm. Sentences presented serially require more storage space than those presented in simultaneously. The result showed that the visuospatial processing system and phonological loop all participated in deductive reasoning and that the central executive system still plays an important role among them. The conclusion can be drawn that the central executive systems all participate in deductive reasoning whether in serial or simultaneous presentation. When memory load increases, the visuospatial processing system and phonological loop may participate in the reasoning process.

8.4.3 Neural mechanism of working memory

The development of brain sciences over nearly a decade has already found that two kinds of working memory are involved in the thinking process: One is used for storing the speech material (concept) with speech coding; the other is used for storing the visual or spatial material (imagery) with figure coding. Further research indicates that not only do concept and imagery have their own working memory, but also imagery itself has two kinds of working memory. There are two kinds of imagery of things: One represents basic attributes of the things used for recognizing them, generally called attribute imagery or object image. The other is used for reflecting the relationship of the spatial and structural aspects of the things (related to visual localization), generally called spatial image or relation image. Spatial image does not include the content information of the object but the characteristic information used to define the required spatial position information and structural relation of the objects. In this way, there are three kinds of working memory:

Contemporary neuroscientific research shows that these three kinds of working memory and the corresponding thinking processing mechanisms have their corresponding areas of the cerebral cortex, though the localization of some working memory is not very accurate at present. According to the new development of brain science research, S. E. Blumstein in Brown University has pointed out [23] that the speech function is not localized in a narrow area (according to the traditional idea, the speech function only involves Broca’s area and Wernicke’s area of the left hemisphere) but is widely distributed in the areas around the lateral fissure of the left brain and extend toward the anterior and posterior regions of the frontal lobe, including Broca’s area, inferior frontal lobe close to the face movement cortex and left precentral gyrus (exclude the frontal and occipital pole). Among them, damage to Broca’s area will harm the speech expression function, and damage to Wernicke’s area will harm the speech comprehension function. But the brain mechanism related with the speech expression and comprehension functions are not limited to these two areas. The working memory used for maintaining speech materials temporarily is regarded as generally relating to the left prefrontal lobe, but the specific position is still not accurate at present.

Compared with the speech working memory, the localization of object working memory and the spatial working memory are much more accurate. In 1993, J. Jonides and his colleagues in Michigan University investigated the object image and the spatial image with positron emission tomography (PET) and obtained some clarification about their localization and mechanism [24]. PET detected pairs of gamma rays emitted indirectly by a positron-emitting radionuclide (tracer), which was introduced into the body on a biologically active molecule. Images of tracer concentration in three-dimensional space within the body are then reconstructed by computer analysis. Because of its accurate localization and noninvasiveness, this technique is suitable for human subject studies.

8.7.1 Neocortex

As a result of evolution, the cerebral cortex can be divided into three parts: archeocortex, paleocortex, and neocortex. The archeocortex and the paleocortex are related to olfaction and are collectively known as the olfactory brain. The neocortex is highly developed in humans, accounting for 96% of the total cortex [28].

The neocortex stores structured knowledge in the connections between the neocortex neurons. When the multilayered neural network is trained, it gradually learns to extract structure. By adjusting the connection weight, the output error of the network is minimized and becomes a relatively stable LTM.

8.7.2 Hippocampus

The hippocampus is a large nerve tissue in the brain, which is located at the junction of the medial cortex and brain stem. The hippocampus is a forebrain structure located within the medial temporal lobe that consists of the subiculum, CA1, CA2, CA3, and dentate gyrus regions shown in Fig. 8.12.

The hippocampus plays an important role in the process of information storage. Short-term memory is stored in the hippocampus [29]. If a memory segment, such as a phone number or a person’s name, is repeatedly mentioned in a short period of time, the hippocampus will transfer it to the cerebral cortex, and it becomes LTM. If the information stored in the hippocampus is not used for a period, it is “deleted,” that is to say, forgotten. The information stored in the cerebral cortex is not completely permanent. When you do not use the information for a long time, the cerebral cortex may “delete” the information. Some people lose some or all their memories when their hippocampus is injured. The process of memory transfer between hippocampus and cerebral cortex lasts for several weeks, and this transfer may occur during sleep.

8.7.3 Complementary learning system

According to his early ideas, James L. McClelland, a psychology professor at Stanford University, put forward the theory of complementary learning systems (CLS) in 1995 [30]. According to his theory, human brain learning is a comprehensive product of two CLS. One is the neocortex learning system, which slowly learns knowledge and skills through experience. The other is the hippocampus learning system, which memorizes specific experiences and enables them to be replayed so as to effectively integrate with the neocortex learning system. In 2016, Dharshan Kumaran and Demis Hassabis of Google’s deep thinking and James L. McClelland of Stanford University published an article on cognitive science trends [31], expanding the theory of CLS. The neocortical learning system is a structured knowledge representation, while the hippocampal learning system quickly learns the details of individual experience. This paper extended the role of memory replay in the hippocampus and points out that memory replay can measure the target’s dependence on experience statistics. By periodically displaying the traces of the hippocampus and supporting partial generalization, the neocortex can learn the knowledge of known structures very quickly. Finally, the paper points out the correlation between the theory and design of an artificial intelligence agent and highlights the relationship between neuroscience and machine learning.

8.8.1 Memory prediction framework

HTM is a kind of artificial simulation algorithm for the structure and function operation of a new cortical layer. Its design is based on the memory prediction framework. In HTM algorithm, the following contents are very core and key: hierarchy, invariant representations of spatial patterns and temporal patterns, and sequence memory.

Four visual cortical areas of the object are denoted as V1, V2, V4, and IT. V1 indicates the region of the striped visual cortex, which is rarely a pretreated image but contains much detail image information. V2 indicates visual mapping, whose visual mapping information is less than that of V1. Visual input with the arrow indicates upward; beginning in the retina, the transmission starts from the bottom to the V1 area. The input represents the time-varying model; visual neural transmission consists of about 1 million axons.

In the four different levels of regions from area VI to area IT, the cells change from rapidly changed and spatially correlated ones that are capable of identifying subtle features, to stable and space-independent ones that are capable of identifying objects. Take, for example, the so-called Face cells of IT cells; as long as there is a human face, the cell will be activated, regardless of whether the face is tilted, rotating, or partially obscured. It is a constant representation of the face.

The feedback connection is especially important when considering prediction; the brain needs to send the input information back to the original input area. Predictions need to compare what is real and what is expected to happen. Information about what really happens will flow from bottom to top, and the information that is expected to happen will flow from top to bottom.

The density and shape of cells in the cerebral cortex are different, and this difference results in stratification. The first layer is unique among all the six layers. It contains only a small number of cells and is mainly composed of a layer of nerve axons that are parallel to the cortical surface. The second and third layers are similar, mainly in their many close-together pyramidal cells. The fourth layer is composed of astrocytes. The fifth layer has pyramid-shaped cells in general and particularly large pyramid-shaped cells. The sixth layer also has several unique neuronal cells.

Fig. 8.13 shows the level of the brain area and the vertical column of the longitudinal cell unit that work together. Different layers of each vertical column are connected with one another by axons extending vertically and forming synapses. The vertical columns in the V1 region are some of the inclined line segments (/) in that direction, while others react to a line segment () in the other direction. Cells in each vertical column are closely interconnected, and they respond to the same stimulus as a whole. The activated cells in the fourth layer will activate the third and second layers. And then it will activate the fifth and sixth layers. Information is transmitted up and down in a cell of the same vertical column. Hawkins believes that the vertical column is a basic unit of prediction.

There is a direct link between the fifth layer of cells in the motor cortex (M1) and movement in the muscle and spinal cord. These cells are highly coordinated to continuously activate and inhibit, so that muscle contraction, drive motion. In each region of the cerebral cortex, fifth cells are spread all over and play a role in all kinds of sports.

Axons of the fifth layer of cells get into the thalamus, and make connections to a class of non-specific cells. These non-specific cells will then project these axons back into the first layer of cerebral cortex in different regions. This loop is just like the delayed feedback sequence that can learn from associative memory. The first layer carries a lot of information, including the name of the sequence and the position in the sequence. Using these two kinds of information of the first layers, a cortical area is able to learn and recall patterns of sequence.

There are three kinds of circuits in the cerebral cortex: convergence along the cortical system with an approach from bottom to top, divergence along the cortical system with an approach from top to bottom, and the delayed feedback of the thalamus, which is very important for the function of the cerebral cortex. These functions are as follows:

In the vertical column of the cerebral cortex, the input information from the lower region activates the fourth layer of cells, leading to the cell’s excitement. Then the fourth layer of cells activates the third and second layers of cells and then the fifth layer, which leads to the sixth layer of cells being activated. So the whole vertical column is activated by the input information of the low-level area. When some of synapses are activated by the second, third, and fifth layers, they are enhanced. If this happens often enough, the first layer of these synapses become strong enough to allow the second, third, and fifth layers of cells to be also activated; the fourth layer of cells is not activated. In this way, the second, third, and fifth layers of the cell can predict when they should be activated by the first layer mode. After this study, the vertical column cells can be partly activated by memory. When the vertical column is activated by synapses of first layer, it is predicting the input information from the lower region; this is the forecast.

Some information received from the first layer is derived from the fifth layer cells of the adjacent vertical column and the adjacent area, which represents the event that has just occurred. Another part comes from the sixth cell, which are stable sequence names. As shown in Fig. 8.14, the second and third layers of cells of the axon usually form a synapse in the fifth layer, and the fourth layer of the axon forms a synapse in the sixth layer. The two synapses intersecting with the sixth layer receive two kinds of input information, and then it is activated and makes a specific forecast according to the constant memory.

As a general rule, the information low along the cerebral cortex is transmitted through the synapses near the cell body. Thus the upward flow of information is increasingly identified in the transmission process. Similarly, as a general rule, the feedback information flowing along the cerebral cortex is transmitted through the synapses away from the cell body. Synapses on the long range of thin dendrites can play an active and highly specific role in cell activation. In general, the axon fiber of the feedback is more than the feed forward; the feedback information can quickly and accurately activate many cells in the second layers of the cerebral cortex.

8.8.2 Cortical learning algorithm

There are four basic functions of HTM: learning, recognition, prediction, and behavior (response). Each of these areas performs the first three functions: learning, recognition, and prediction. But behavior is different. The cortical learning algorithm is a kind of artificial simulation algorithm for the structure and function operation of the new cortical layer. The cortical learning algorithm contains the following contents: hierarchy, invariant representations of spatial patterns and temporal patterns, and sequence memory.

In the HTM cortical learning algorithm used to establish the connection of each column area of the input subset and to determine the input level of each column area, the use of suppression to filter a sparse set of active column areas is called spatial sedimentation pool. A time deposition pool is used to represent the two steps of adding context environment and prediction. By building a slowly changing output of pattern sequences, we deposit different patterns in time. The study of the time and space sedimentation pool is similar. Both types of learning involve building connections, or synapses, between cells. The time sedimentation pool is responsible for the connection between cells in the same area. The spatial sedimentation pool is responsible for the feedforward connection between the input and the columnar areas.

The learning process involves an increase and decrease in the connectivity of potential synapses in the dendrite region. The rule of increasing or decreasing synaptic connectivity is similar to that of Hebbian rule. The main function of the spatial sedimentation pool is to convert the input of a key area into a sparse mode.

Time deposition is much more complex than space deposition. How to learn sequence and make predictions in a time sedimentation pool? The basic way is that, when a cell becomes active, it establishes a connection with the cells that are active before it. Cells can then predict when they will be activated by focusing on their connections. If all cells do that, they can store and recall sequences, and they can predict what might happen next.