Learning the psychology of the tip-of-the-tongue phenomenon through on-line practice
Learning the psychology of the tip-of-the-tongue phenomenon through on-line practice
Marcos Ruiz 0
María José Contreras 0
0 Universidad Nacional de Educación a Distancia - UNED , Spain
Psychology undergraduates can benefit from direct experiences with laboratory procedures of psychological phenomena. However, they are not always available for students within a distance education program. The present study included students from the Spanish National Distance Education University (UNED) that were to take part in a Basic Psychology examination session. They participated in web-sessions on a tip-of-the-tongue (TOT) laboratory procedure. The aim was to study whether their performance at TOT-related items would be differentially improved. Our results support the conclusion that practicing with the TOT application was effective in improving the TOT comprehension among students. Study A showed that the performance level was higher for the TOT-practiced participants relative to the non-practiced ones. Study B showed significant group by itemtype interaction. Also, there was a significant effect of group, and item-type. The results are contextualized in the psychological institutions' mainstream effort for Psychology to be viewed as a STEM discipline by students, the political representatives, and the society.
Distance education; e-learning; scientific psychology; folk psychology; tip-of-the-tongue; STEM
Direct experience of psychology students with the research procedure of psychological phenomena
is usually considered a very profitable strategy for Psychology undergraduates. Typically, the
course strategy includes presenting students with psychological phenomena and supporting the
presentation with how researchers undertake its study under scientifically controlled conditions
Homa et al., 2013)
. A similar schema seems to be very often at work in the introductory psychology
(Benjamin, 2005; Griggs & Bates, 2014)
. However, as Norcross et al. (2016) have pointed
out, only a 3% of undergraduate Psychology programs offered a lab for their introductory course. Yet,
according to Goal 2 of the APA Guidelines for the Undergraduate Psychology Major version 2.0, even
“students completing foundation-level courses should learn basic skills and concepts in interpreting
behavior, studying research, and applying research design principles to drawing conclusions about
(APA, 2013, p. 20)
. Furthermore, very few works have studied learning
after running experiments by measuring exam performance
(Gil-Gómez de Liaño, León &
With that goal in mind, for ordinary psychology courses, the benefits of engaging students in a
teacher-monitored immersion within a psychological research procedure are well-known
et al., 2016; Pearson & Richardson, 2013)
. However, for distance education institutions, most of the
learning tasks are usually self-guided; and, very often, no specific tutoring is available at the exact
moment when the student starts studying a certain psychological topic. For instance, in our institution,
the course materials are available through the Internet (as hypertext or pdf documents), and students
can use them without any constraining external timing and without interactive personal tutoring; yet,
personal tutoring is supposed to be given on demand (Luzón & Quintana, 2010). Therefore, the
question arises as to whether the open availability through the Internet of a psychological research
procedure improves the understanding of its targeted psychological phenomenon.
Following a suggestion by Moore
(1989; see also the Community of Inquiry framework
described by Garrison & Akyol, 2009)
Bernard et al. (2009)
conducted a meta-analysis of
the instructional power for students in a distance education context of three types of factors:
interactions with the course material
(e.g., Hartnett, 2013)
, interactions with other students
Thoms & Eryilmaz, 2014)
, and interactions with their teachers
(e.g., Coll, Rochera & de Gispert,
. Unsurprisingly, they found that, overall, the most beneficial interaction was that with the
course material. Consequently, a very appropriate way to improve psychological learning seems
to be the intensification of the contacts between the student and the to-be-learned psychological
subject. Yet, the relevance of multi-media as tools for achieving a reliable understanding of
concepts and phenomena has been emphasized (e.g., Mayer, 2001). In fact, this view seems to
be particularly suited for engineering, experimental sciences, and mathematics, because, with
the help of videos and software applications, seemingly intricate concepts and principles can
become perfectly understandable
(e.g. Zhang, 2014)
Although students are presented with Psychology as an experimental science
, there is something peculiar to the everyday psychological processes that hampers their
scientific understanding by undergraduates in an introductory psychology course. Certainly, the
misconceptions and ordinary first-person experience with the phenomenon very often gives rise
to the fact that a considerable effort is required to change the view to a more educated,
sciencebound, third-person analysis
(for a review, see Hughes, Lyddy & Lambe, 2013)
. To overcome these
difficulties, associated as they are to everyday psychological processes, a closer acquaintance with
their research procedures appears as important as it seems to be for any other natural phenomenon.
Let us take as an example the so-called tip-of-the tongue (TOT) phenomenon. For a layperson,
this is just a matter of not remembering something at the precise moment we need it, while, at
the same time, being completely sure we know/remember it very well. But, for psychologists, the
TOT phenomenon has been, for decades, a window to the very complex mechanisms of language
(see, e.g., Koriat & Lieblich, 1974; for a review, see Dunlosky & Metcalfe, 2009;
Ruiz, 2003, 2004)
. However, its replication in the laboratory, under controlled conditions, has been
anything but an easy task. Overall, these peculiarities make the TOT research procedure a very
appropriate tool to illustrate, at an Introductory Psychology level, the main features of scientific
Going back to our general question, we could re-frame it as whether the availability through the
Internet of the TOT research procedure could improve the understanding of its methodological and
theoretical intricacies by those enrolled in an online introductory psychology course. To answer this
question, related with the interdependence between instructional contexts and levels of performance
that could influence student achievement
(De la Fuente, Martínez, Peralta & García, 2010)
ran two studies on the usefulness of online psychological procedures delivered as practices in an
ongoing Psychology course.
In the studies described below, some of the students that were to take part in a Basic Psychology
examination session voluntarily applied to participate in web-sessions on a typical TOT procedure.
The TOT phenomenon and the way it was studied by experimental psychologists were course
topics. Yet, the practice itself was defined from the beginning of the term as an assignment. As
a consequence, 6 out of the 30 items in the exam were about the TOT practice. We wanted to
know whether the participation in the internet practice sessions would improve the examinees’
performance at the TOT-related items above that of the non-participants, compared to their
achievements in the non-related items. An improvement in the average results on the target
testitems would mean that the internet facility used was appropriate as a tool for the understanding
of the TOT phenomenon and for learning how psychologists perform their laboratory research
As there were two examination sessions and the students could freely choose just one of them,
we describe the data as study A and study B. Every multiple choice, 30-item test was completely
independent from each other and both were prepared by the teachers well in advance of the sessions.
The assignment of either test to a session was random.
The data for this study were obtained from the 64 students who took part in the first examination
session of a four-month Basic Psychology course. The course was mandatory in the Social
Work Degree at the Universidad Nacional de Educación a Distancia (UNED). The task was
a voluntary assignment for the course. The participants were 46 women and 21 men, aged
between 21 and 53 years (Mean=34, SD=9.1). As described below, 34 of these students
voluntarily applied for the participation in at least one web session of the available TOT
The course lessons and reading assignments were available to the students as pdf documents.
They were accessible on the Internet through aLF, an e-learning platform designed by the
UNED’s staff for their undergraduate students
(e.g. Luzon & Quintana, 2010)
. Within these
materials, a chapter about the psychology of memory (Ruiz, 2011) included a description of the
TOT phenomenon, along with some topics on its relevance for human memory understanding.
The difficulties for the production of the TOT phenomenon under controlled conditions were
Additionally, all throughout the course-term, two practice Internet pages were accessible in aLF,
each one with a short presentation of an Internet application for practicing with both an attentional
procedure and the TOT. Each of them included a link to a pdf document with more detailed specific
instructions (i.e., user’s guide) about how to work with the procedure
(Contreras, 2010; Ruiz, 2010)
The TOT aLF page also included a link to a Java applet that controlled a version of the TOT laboratory
procedure by Koriat and Lieblich (1974). The Spanish stimulus material for the application was taken
. The applet was a menu-driven application with a task menu to freely select:
(a) a few individual practice trials, (b) the whole individual experimental session, (c) a few collective
practice trials, and (d) the whole collective experimental session. In addition to the task menu, there
was a help menu that offered extensive and detailed on-line help for the user as conductor of the
experiment, and a page for downloading the instructions given to TOT-participants, both for the
individual and the collective versions of the TOT procedure; there was also a response sheet for the
collective version of the TOT procedure1 available to download. See Figure 1 for a screenshot of a
trial of the TOT procedure.
Also, the accompanying user’s guides and other related documentation are available upon request from the first author
For the qualification exam on the Basic Psychological Processes program, the usual 30-item test
was presented. The questions were three-alternative items, one of them correct; and there was no
penalty for omissions, although a false choice yielded a penalty of half a point/score, which was
deducted from total number of hits. Students were well acquainted with this kind of qualification
exams, as they are very common at our university. According to the university schedules, the items
were prepared about two months before the exam session. Six out of the 30 items were target or
experimental items, because they dealt with the TOT practice. The other 24 items were common
questions about the remaining course material. Due to a composition error in one of the target items,
it was excluded from the analyses.
From the beginning of the course, students were encouraged by the teachers to carefully study the
course-materials, to participate in the aLF forums discussions, and to engage as carefully as possible
in the practice applications. Yet, they knew from the general information about the course that up
to 25% of the items in the exam could be about the practice materials. The practice applications
were available at any time through the aLF platform. Whenever a student tried to freely start the
TOT procedure, s/he was required to enter her/his identity card number and last name to run the
TOT application. The applet controlled stimuli presentation and response recording, and sent every
student’s response to a php script, in order for the script to appropriately record the events in a
database, along with its time of occurrence and participant’s data.
The exam session lasted for 120 minutes and took place within the ordinary university settings
and facilities scheduled for the degree qualification exams. The exam was a paper-and-pencil
threechoice test. During the exam, students were allowed to use whatever printed material they wanted;
although sharing them or the use of digital media were forbidden by the exam supervisors.
The R software was used for the filtering, tabulation and statistical analyses all through the paper
(R Core Team, 2013)
Data corresponding to any practice session performed after 24:00 of the night before the exam were
filtered out of the database. Participants that took the degree exam were classified in two groups as
a between-subjects factor: those that entered the practice session at least once before this virtual
deadline, and those who did not. Also, the type of item was a within-subject variable in the design:
target TOT-related items and non-target TOT-unrelated items.
In order to analyze the effect of practicing with the application on the TOT-procedure understanding,
we first ran an ANOVA on the hit rates in both kinds of items for the two groups of participants. Table
1 shows the mean hit rates for the two types of items and for both the practiced and the unpracticed
group. It can be observed that performance level was higher for the TOT-practiced participants
relative to the non-practiced ones, F(1,62) = 26.45, MSE=2.34, p<0.05, η²= 0.16. Also, it can be
seen that students performed better at the non-TOT contents of the course, since the mean hit rates
for the non-target items was higher than for the target items, F(1,62) = 115.12, MSE=0.54, p<0.05,
η²= 0.50. The target-items performance level decreased up to about 1/5th (.11/.56) of the non-target
ones for the unpracticed group, while this relation was only 1/2 (.35/.67) for the practiced group. Yet,
this interaction, although expected from our hypothesis, just approached significance, F(1,62) = 2.98,
MSE=20.75, p<0.10, η²= 0.03.
Hit rates as ratios of number of hits per number of items have the advantage of giving us a close
enough view of the participants’ performance levels. However, one problem with this measure
is that it does not control for guessing. To avoid these spurious effects, we assigned a cost to
every error. As it is usually done for 3-choice tests, we computed for each participant and each
kind of items, a corrected performance estimate with a penalty of half a hit per commission error.
Also, since the number of items varies across item conditions, the cost and benefits per item
were adjusted so that the estimates of both target and non-target items were on the same scale.
Table 2 shows the mean net values for the four experimental conditions of our design on a scale
of 10 points. As can be seen, there were clear effects of both the group factor ( F(1,62) = 23.59,
MSE=2.63, p<0.05, η²= 0.14) and the item factor (F(1,62) = 77.66, MSE=0.80, p<0.05, η²= 0.40).
Certainly, participants were much better at the TOT-unrelated items than at the target ones. And
practiced participants achieved overall higher records on the qualification exam. But, once again
contrary to our expectations, the groups were similarly less efficient as for the target items, as
shown by a decrease of 3.6 and 3.7 points for the practiced and unpracticed groups, respectively
(F<1, for the interaction between the practice and item-type factors).
Following our university official examination schedule, two weeks after the exam session of the study
A, a new exam session took place. Only students that had not applied for the first session could
apply for the second one. As is usually the case at our university, most of the students in the course
assisted to this second session, once they knew the general features of the exam from the first
session. Throughout this time, the opportunity to work with the TOT Java application was open. As a
consequence, students applying for the second exam session had more time to interact with the TOT
procedure than their fellows of the first session. Additionally, our university facilities, once an exam
has been celebrated, provide the exam content on the Internet for download, in order for the students
to get familiar with it. So, our second-exam applicants knew about the relevance of the TOT-practice
in the first exam, although they had no way to guess if it would the same for the second session.
In this study, the same type of analyzes as those described for the Study A were performed. Due
to the few reasons mentioned above, the data are not strictly comparable, so that an analysis on the
whole set of pooled data seems inappropriate. This study followed the same procedure as Study A,
with the exception of the participants that took part in it, which were different.
The data for this study were taken from the 234 students who took part at the second examination
session of the Basic Psychology course. They were 189 women and 45 men, aged between 18 and
61 years (Mean= 33, SD=9.3). Of these students, 119 voluntarily applied for the participation in at
least one web session of the available TOT procedure.
An independent set of 30 new, 3-alternative items were the content of the qualification test for
the second session. As described before, the assignment of this exam to the second session was
randomly performed two months before the first exam session. No item was discarded from this
exam, as no composition error was detected. Consequently, in this study, the test includes 24
nontarget items and 6 target-items.
All other features of the used materials and procedure are the same as those described for Study A.
The same analyses as in Study A were run on the data of the second-session exam. Table 3 shows
the mean hit rates for the two types of items and for both the practiced and the unpracticed groups of
Study B. Clearly, hit rates seemed to be overall close to 50 percent, with the exception of participants
unpracticed on TOT-related items, whose performance noticeably decreased. This pattern was
confirmed by a significant group by item-type interaction, F(1, 232) = 71.62, MSE=3.24, p<0.05,
η²= 0.13. Also, there was a significant effect of group F(1, 232) = 107.70, MSE=2.15, p<0.05, η²=
0.19, and item-type F(1, 232) = 70.63, MSE=3.28, p<0.05, η²= 0.13, although the relevance of these
effects is clearly limited by the pattern of the significant interaction.
A fairly parallel pattern of results can be seen in Table 4. The interaction was also significant for
the net value means F(1, 232) = 37.72, MSE=6.15, p<0.05, η²= 0.07, as well as the main effects (F(1,
232) = 84.37, MSE=2.75, p<0.05, η²= 0.17, for the group, and F(1, 232) = 32.58, MSE=7.12, p<0.05,
η²= 0.06, for the item-type factor).
The corresponding mean standard errors can be seen within parentheses.
Overall, the data of this Study B support the idea that items regarding the TOT procedure were
relatively more difficult for our students. Nevertheless, not surprisingly, those engaged in the
TOTpractice were able to normalize their performance on these items up to their achievement at the
Before drawing conclusions from our research, it should be acknowledged that our experimental
procedure does not perfectly fit the purest definition of an experimental design, mainly due to our
participants not having been randomly assigned to the experimental conditions
(Campbell & Stanley,
. Yet, the contrast between target and non-target items for both practiced and unpracticed
participants seems to give us reasons to confidently draw some conclusions: the within-session
patterns of both, session A and B taken together, amount to the effectiveness of practicing with the
TOT application through the internet. Notice that our conclusion gains generalizability from the fact that
the studies were run along the ordinary course-term development in our distance education university.
From the American Psychological Association, Steven Beckler
that most of the contents typically dealt with in an undergraduate Psychology course are included
in the category code 42.2799 within the Classification of Instructional Programs (CIP) system used
by the U.S.A. Nation Center for Education Statistics (NCES). The interesting issue here is that
this CIP code is usually included by some leading institutions in the STEM (Science, Technology,
Engineering, and Mathematics) group. Indeed, as it is the case for most of the STEM disciplines,
Psychology courses for distance education have to cope with the specific difficulties produced by the
lack of a direct access to lab facilities by the students. Our research was aimed at showing how some
of the shortcomings for Psychology as a STEM discipline in a distance education institution could
be overcome. Thus, our data add to those showing that for Psychology to be considered as a STEM
discipline, the use of computers as lab analogues can be very profitable to spread among students
“the critical idea that human thought and behavior can be studied scientifically”
(Breckler, 2014b, p.
48; see also Gurung et al., 2016)
The present research shows that it is possible to modernize the traditional models of psychology
teaching, applying information technology (IT) in the psychological science learning and instruction.
Specifically, our data supports that a digitized version of the laboratory procedure developed to
study a seemingly ordinary psychological phenomenon could be a powerful tool for psychology
undergraduates. Indeed, this research is in line with those studies showing how the scientific
education can be improved by the use of IT tools aimed to foster specific scientific cognitive skills,
such as analysis and evaluation of phenomena in nature
(Kirschner & Huisman, 1998)
. Research on
the use of ITs in the learning and instruction of, for example, biology
(e.g., Sewell, Stevens & Lewis,
(Quellmalz, Timms, Silbergitt & Buckley, 2012)
(e.g., Bonnetain, Boucheix,
Hamet & Freysz, 2010)
, can be found in the literature.
However, to our knowledge, there is no research dealing with the advantages of the IT use for the
learning and instruction of procedures in experimental psychology. Also, it should be stressed here
that our knowledge domain, being ordinary psychological phenomena and experiences, deserves a
special consideration, as it usually offers a certain difficulty to be viewed from a scientific stance by
the non-initiated student. Our main conclusion would be that, with the use of digitized analogues of
well-known psychological research procedures, such difficulty could be overcome.
The efficiency of IT products as learning tools for psychology seems to be specially suited for
distance education systems
(although for some cautions see Clay, 2014)
. Our work clearly shows that
these technologies should be massively implemented as a supplement to text material for improving
the contact between psychology students and their course contents
(Bernard et al., 2009)
. Thus, the
psychological phenomena comprehension from a scientific view point could achieve the conceptual
richness and complexity that could only be attained with active and interactive study
. Additionally, the instructor and evaluator could gain a much sophisticated assessment tool to
evaluate the knowledge of psychological concepts and the competence on psychological procedures
(Pellegrino, Chudowsky & Glaser, 2001)
. Certainly, the way that students interact with the digitized
materials could be traced in order for the instructor to assess the learning progress or the especially
Finally, it should also be noted that the implementation of digitized laboratory analogues of
scientific psychological procedures could make the consideration of psychology as a scientific
discipline a widespread view among non-psychologists as well. As it has been acknowledged by
the American Psychological Association “APA is working to resolve one of psychology’s great public
relations problems: the fact that other scientists, lawmakers and the general public don’t always
view psychology as one of the STEM —science, technology, engineering and math— disciplines”
(Price, 2010, p. 32)
. We think that in order to deal with this problem, psychologists and psychological
institutions could do a great job entering IT versions of laboratory psychological procedures within the
growing Massive Open Online Courses (MOOCs) as well as in other distance education programs.
The present research shows it is profitable to apply information technology (IT) in the psychological
science learning and instruction implemented by a distance education institution. Specifically, our
data supports that a digitized version of the laboratory procedure developed to study a seemingly
ordinary psychological phenomenon could be a powerful instrument for psychology undergraduates.
This strategy could help to cope with the specific difficulty associated to ordinary behavioral
experience (explained by folk psychology as something being on the tip of the tongue) to be viewed
from a more-educated scientific stance (scientific psychology). With the use of digitized analogues
of psychological research procedures, such difficulty could be overcome in the context of distance
education institutions or MOOC courses.
Conflict of Interest
The authors declare that they have no conflict of interest.
American Psychological Association ( 2013 ). APA guidelines for the undergraduate psychology major: Version 2.0 . Retrieved from http://www.apa.org/ed/precollege/undergrad/index.aspx
Benjamin , L.T. ( 2005 ). Setting course goals: Privileges and responsibilities in a world of ideas . Teaching of Psychology , 32 ( 3 ), 146 - 149 . http://dx.doi.org/10.1207/s15328023top3203_ 1
Bernard , R. M. , Abrami , P. C. , Borokhovski , E. , Wade , C. A. , Tamim , R. M. , Surkes , M.A. , y Bethel E.C. ( 2009 ). A meta-analysis of three types of interaction treatments in distance education . Review of Educational Research , 79 ( 3 ), 1243 - 1289 . http://dx.doi.org/10.3102/0034654309333844
Bonnetain , E. , Boucheix , J.M. , Hamet , M. , & Freyaz , M. ( 2010 ). Benefits of computer screenbased simulation in cardiac arrest procedures . Medical Education , 44 ( 7 ), 716 - 722 . http://dx.doi. org/10.1111/j.1365- 2923 . 2010 . 03708 .x
Breckler , S.J. ( 2014a ). Calling psychology education what it is: STEM .Monitor on Psychology, 45 ( 7 ), 74 .
Breckler , S.J. ( 2014b ). Learning STEM, working STEM. Monitor on Psychology , 45 ( 9 ), 48 .
Campbell , D.T. & Stanley , J.C. ( 1966 ). Experimental and quasi-experimental designs for research . Wadsworth: Cengage Learning.
Clay , R.A. ( 2014 ). Learning in a digital world . Monitor on Psychology , 45 ( 11 ), 38 .
Coll , C. , Rochera , M.J. & de Gispert , I. ( 2014 ). Supporting online collaborative learning in small groups: Teacher feedback on learning content, academic task and social participation . Computers and Education , 75 , 53 - 64 . http://dx.doi.org/10.1016/j.compedu. 2014 . 01 .015
Contreras , M.J. ( 2010 ). Práctica TIAD: Documentación y programa . [TIAD practice: Documentation and software] . UNED: Plataforma aLF de Procesos Psicológicos Básicos. (Accessible only to authorized students and teachers ).
De la Fuente , J. , Martínez , J.M. , Peralta F.J. , & García , A.B. ( 2010 ). Percepción del proceso de enseñanza-aprendizaje y rendimiento académico en diferentes contextos instruccionales de la Educación Superior. [Perception of the teaching-learning process and academic achievement in diverse instructional contexts of Higher Education] . Psicothema , 22 , 806 - 812 . Retrieved from http://www.psicothema.es/pdf/3805.pdf
Dunlosky , J. & Metcalfe , J. ( 2009 ). Metacognition. Los Ángeles, CA: Sage.
Garrison , D.R. , & Akyol , Z. ( 2009 ). Role of instructional technology in the transformation of higher education . Journal of Computing in Higher Education , 21 , 19 - 30 . https://doi.org/10.1007/s12528- 009-9014-7
Gil-Gómez de Liaño , B. , León , O. & Pascual-Ezama , D. ( 2012 ). Research Participation Improves Student's Exam Performance . Spanish Journal of Psychology , 15 , 544 - 550 .
González , J. ( 1996 ). El fenómeno de la 'punta de la lengua' y la recuperación léxica: Estudio de sus propiedades en castellano y el efecto de la frecuencia del estímulo. [The tip-of-the-tongue phenomenon and lexical retrieval: A study on its properties in Spanish and the stimulus frequency effect] . Estudios de Psicología , 56 , 71 - 96 .
Griggs , R.A. & Bates , S.C. ( 2014 ). Topical coverage in Introductory Psychology: Textbooks versus lectures . Teaching of Psychology , 41 ( 2 ), 144 - 147 . http://dx.doi.org/10.1177/0098628314530347
Gurung , R.A.R. , Hackathorn , J. , Enns , C. , Frantz , S. , Cacioppo , J.T. , Loop T. , & Freeman , J.E. ( 2016 ). Strengthening Introductory Psychology: A new model for teaching the introductory course . American Psychologist , 71 ( 2 ), 112 - 125 . http://dx.doi.org/10.1037/a0040012
Hartnett , J.L. ( 2013 ). Stats on the cheap: Using free and inexpensive internet resources to enhance the teaching of statistics and research methods . Teaching of Psychology , 40 ( 1 ), 52 - 55 . http:// dx.doi.org/10.1177/0098628312465865
Homa , N. , Hackathorn , J. , Brown , C.M. , Garczynski , A. , Solomon , E.D. , Tennial , R. , Sanborn , U.A. , & Gurung , R.A.R. ( 2013 ). An analysis of learning objectives and content coverage in Introductory Psychology syllabi . Teaching of Psychology , 40 ( 3 ), 169 - 174 . http://dx.doi. org/10.1177/0098628313487456
Hughes , S. , Lyddy , F. , & Lambe , S. ( 2013 ). Misconceptions about Psychological Science: a review . Psychology Learning & Teaching , 12 ( 1 ), 20 - 31 . http://dx.doi.org/10.2304/plat. 2013 . 12 .1. 20
Kirschner , P. & Huisman , W. ( 1998 ). “Dry laboratories” in science education; computer-based practical work . International Journal of Science Education , 20 ( 6 ), 665 - 682 . http://dx.doi. org/10.1080/0950069980200605
Koriat , A. & Lieblich , I. ( 1974 ). What does a person in a “TOT” state know that a person in a “don't know” state doesn't know . Memory & Cognition , 2 ( 4 ), 647 - 655 . http://dx.doi.org/10.3758/BF03198134
Luzón , J.M. & Quintana , I. ( 2010 ). Guía Práctica de Alf: Perfil Equipo Docente. [User's Guide for Alf: Teaching team profile] . Madrid: IUED-UNED.
Mayer , R. ( 2001 ). Multi-media learning . Cambridge, UK: Cambridge University Press.
Moore , M. ( 1989 ). Editorial: three types of interaction . Americal Journal of Distance Education , 3 ( 2 ), 1 - 7 . http://dx.doi.org/10.1080/08923648909526659
Norcross , J.C. , Hailstorks , R. , Aiken , L.S. , Pfund , R.A. , Stamm , K.E. & Christidis , P. ( 2016 ). Undergraduate study in Psychology: Curriculum and assessment . American Pychologist , 71 ( 2 ), 89 - 101 . http://dx.doi.org/10.1037/a0040095
Pearson , M.R. & Richardson , T.A. ( 2013 ). TeachingthetruthaboutliestoPsychologystudents:Thespeed lying task . Teaching of Psychology , 40 ( 1 ), 56 - 58 . http://dx.doi.org/10.1177/0098628312465866
Pellegrino , J.W. , Chudowsky , N.J. , & Glaser , R . (Eds.) ( 2001 ). Knowing what students know: The science and design of educational assessment . Washington, DC: National Academy of Sciences.
Price , M. ( 2010 ). Earning a spot in science's big tent . Monitor on Psychology , 41 , 32 .
Quellmalz , E.S. , Timms , M.J. , Silbergitt , M.D. & Buckley , B.C. ( 2012 ). Science assessments for all: Integrating science simulations into balanced state science assessment systems . Journal of Research in Science Teaching , 44 ( 1 ), 183 - 203 . http://dx.doi.org/10.1002/tea.21005
R Core Team ( 2013 ). R: A language and environment for statistical computing . R Foundation for Statistical Computing , Vienna, Austria. Retrieved from http://www.R-project.org/
Ruiz , M. ( 2003 ). Procedimientos y simulaciones en psicología de la memoria [Procedures and simulations in psychology of memory] . Madrid: UNED.
Ruiz , M. ( 2004 ). Las caras de la memoria [The faces of memory] . Madrid: Pearson.
Ruiz , M. ( 2010 ). Práctica FPL: Documentación y programa . [TOT Practice: Documentation and software] . UNED ( Accessible only to authorized students and teachers. Pdf document available from the authors upon request).
Ruiz , M. ( 2011 ). Apuntes sobre Psicología de la Memoria. [Notes on the Psychology of Memory]. (Accessible only to authorized students and teachers. Pdf document available from the authors upon request).
Sewell , R.D. , Stevens , R.G. , & Lewis , D.J ( 1995 ). Multimedia computer technology as a tool for teaching and assessment of biological science . Journal of Biological Education , 29 ( 1 ), 27 - 32 . http://dx.doi.org/10.1080/00219266. 1995 .9655415
Sitzmann , T. ( 2011 ). A meta-analytic examination of the instructional effectiveness of computerbased simulation games . Personnel Psychology , 64 ( 2 ), 489 - 528 . http://dx.doi.org/10.1111/j.1744- 6570 . 2011 . 01190 .x
Thoms , B. & Eryilmaz , E. ( 2014 ). How media choice affects learner interactions in distance learning classes . Computers and Education , 75 , 112 - 126 . http://dx.doi.org/10.1016/j.compedu. 2014 . 02 .002
Zhang , M. ( 2014 ). Who are interested in online science simulations? Tracking a trend of digital divide in Internet use . Computers and Education , 76 , 205 - 214 . http://dx.doi.org/10.1016/j.compedu. 2014 . 04 .001