Pedagogical Implications of the Testing Effect, Working Memory

I wrote the following for an assignment in IDS 6504: Adult Learning, instructed by Dr. Kay Allen. I chose the testing effect and cognitive load theory because of my interest in these constructs and their pedagogical importance.

IDS 6504 Assignment 6
Richard Thripp
University of Central Florida
March 8, 2017

1. Theory and Construct – Cognitive Information Processing – The Testing Effect

2. First Implication for Instruction – Testing learners’ ability to recall (“retrieval practice”) improves learning and assessment outcomes by strengthening both retrieval ability and knowledge encoding.

3. Question – When should teachers and trainers implement retrieval practice to engage the testing effect?

4. Answer – My claim that the testing effect may even improve knowledge encoding sounds audacious to the uninitiated, but is being borne out by recent research—Karpicke and Blunt (2011), in a statement that sounds more like synaptic pruning than an educational phenomenon, propose that “retrieval practice may improve cue diagnosticity by restricting the set of candidates specified by a cue to be included in the search set” (p. 774). That is to say, the testing effect is not so much increasing the number of encoded features, but rather improving the lucidity of the existing encoded features, somewhat like tracing over a pencil sketch in pen. For closed-book assessments, retrieval practice has been shown to be much more effective than repeated study of learning materials, if the exam is given some time after the last study session (in Roediger & Karpicke, 2006, the testing effect was apparent two days and a week later, but not five minutes later). Teachers and trainers should augment their lessons with retrieval practice activities early and often, even for complex materials (Karpicke & Aue, 2015). Simply re-reading a textbook is not enough. Even teachers who implement elaborative learning activities are leaving a great deal of potential learning gains on the table if they do not engage the testing effect through retrieval practice (Karpicke & Blunt, 2011). One of the few times where retrieval practice may not be useful is immediately before an exam (i.e., the five-minute condition in Roediger & Karpicke, 2006). Giving yourself flashcard quizzes while waiting for the exams to be passed out is probably not very useful, perhaps because there simply is not enough time for the testing effect to incubate at this point.

5. References

Karpicke, J. D., & Aue, W. R. (2015). The testing effect is alive and well with complex materials. Educational Psychology Review, 27, 317–326. http://doi.org/10.1007/s10648-015-9309-3

Karpicke, J. D., & Blunt, J. R. (2011). Retrieval practice produces more learning than elaborative studying with concept mapping. Science, 331, 772–775. http://doi.org/10.1126/science.1199327

Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17, 249–255. http://doi.org/10.1111/j.1467-9280.2006.01693.x

6. Specific Application – One specific application, employed by Dr. Kay Allen at the University of Central Florida in such courses as EDF 6259: Learning Theories Applied to Instruction and Classroom Management, and IDS 6504: Adult Learning, is to give learners multiple-choice quizzes during lectures. This retrieval practice may aid long-term retention and retrieval ability, particularly for learners who read the textbook, modules, or other supporting materials prior to attending the lecture or web conference.


7. Theory and Construct – Cognitive Load Theory – Working Memory and Cognitive Efficiency

8. Second Implication for Instruction – Instruction should be designed to accommodate the learner’s working memory capacity by reducing or eliminating the need to hold information in working memory unnecessarily. This is just one step toward designing instruction with cognitive efficiency in mind.

9. Question – How should instructional designers account for working memory capacity in multimedia learning?

10. Answer – Multimedia learning activities should be designed to avoid cognitive overload for the target audience (Mayer & Moreno, 2003). If the target audience is learners who are already experts in the field of study at hand, obviously, learning activities that produce substantial cognitive overload for novices might become viable. Cognitive efficiency, or “qualitative increases in knowledge gained in relation to the time and effort invested in knowledge acquisition” (Hoffman, 2012, p., 133), is arguably a worthy consideration—the time and resources available to learners and instructors are perennially constrained. Instruction that exceeds the learner’s working memory capacity most commonly results in cognitive inefficiency, not unlike a computer running out of random-access memory and being forced to “swap” information to the hard disk which is one one-thousandth as efficient. Therefore, instructional designers should not only consider their target audience(s), but develop their multimedia materials with good pedagogy that transcends the target audience. For example, expecting learners to memorize a lengthy number or sentence and then enter this information on a different screen is neither appropriate for novices nor experts (except, in the rare case that the instructional goal is short-term retrieval practice). Instead, the learning activity should be designed so the learner can simultaneously view this information while entering it into a different area or application. In a similar vein, multimedia learning should employ techniques such as segmenting, pretraining, signaling, and weeding to avoid extraneous cognitive load and optimize learning-relevant cognitive load (Mayer & Moreno, 2003), thereby avoiding cognitive or working-memory overload and improving cognitive efficiency.

11. References

Hoffman, B. (2012). Cognitive efficiency: A conceptual and methodological comparison. Learning and Instruction, 22, 133–144. http://doi.org/10.1016/j.learninstruc.2011.09.001

Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38, 43–52. http://doi.org/10.1207/S15326985EP3801_6

12. Specific Application – If you are designing multimedia training that requires interacting with external computer programs, it is not appropriate for this training to take up the entire computer monitor. The training window should be capable of being resized to a smaller size by the learner, so that he or she can avoid unnecessary working memory usage and avoid the split-attention effect by being able to position other computer program windows next to the training window (Mayer & Moreno, 2003). Similarly, within the multimedia training, such situations should be avoided. For a specific application, in IBM’s Statistical Package for the Social Sciences (SPSS), there are many instances where it is impossible to access certain information about the data-set at hand without closing a statistical options menu or dialog box. This phenomenon occurs even when accessing certain information about the data-set is essential to the task at hand in an options menu or dialog box. This is a prime example of poor design that fails to consider cognitive load theory, working memory, or efficiency of any kind besides the convenience of the programmers and developers of the software or training at hand.

13. Specific Application – Supplement – If the prior application is difficult to understand, here is an easy example: you have received a voicemail on your smartphone where the caller has spoken a call-back number that is different from his or her caller ID number. However, without some external tool such as a pen and paper, it is impossible to record this phone number in your phone while listening to it. Consequently, you are forced to hold the number in working memory if an external recording device such as a pen and paper is unavailable, and then enter it into your contacts or dialing app. If you are familiar with the area code, remembering seven numbers is an easy task, but if the area code is unfamiliar, attempting to hold 10 numbers in working memory may easily exceed your working memory capacity. Regardless, from a design standpoint, this is a poorly designed and needlessly inefficient situation.

Task Analysis Comparison for Calculation of Net Worth

I wrote the following paper for my coursework in EME 7634: Advanced Instructional Design, instructed by Dr. Atsusi Hirumi. Net-worth calculation was my chosen topic, due to my persistent interest in financial education.

I am also making this paper and companion slides available for download. The companion slides are not included in the paper. They were made two weeks before I wrote this paper, prior to conducting the actual task analyses.

Download paper as Microsoft Word 2016 document
Download paper as PDF
Download companion slides as Microsoft PowerPoint 2016 file
Download companion slides as PDF

My work should only be used appropriately and I should be credited.


Task Analysis Comparison for Calculation of Net Worth
Richard Thripp
University of Central Florida
March 1, 2017

Calculating one’s net worth is a vital part of financial literacy (French & McKillop, 2016). Tallying the value of one’s assets and debts improves understanding of one’s financial situation. Although at first, this process may seem simple, appraising one’s assets is a complex issue, and even remembering all of one’s possessions and liabilities may be difficult. Therefore, net-worth calculation seems a suitable instructional situation to analyze. For this portfolio analysis, I am applying three alternative analysis techniques that were included in Jonassen, Tessmer, and Hannum’s (1999) handbook—procedural analysis, critical-incident analysis, and case-based reasoning (CBR). The former two are differentiated by their focus on overt elements and underlying methods, respectively, while CBR’s status as a task-analysis method is tenuous and its utility in this situation is marginal—it is included here for demonstration purposes.

Procedural Analysis

This type of analysis is geared toward assembly lines and other easily observable tasks. However, it can be used to describe cognitive activities if they are overtly observable, and when extended with flowcharting, can even describe relatively complex decision-making processes.

The following analysis is for the net-worth calculation task, based on the steps described by Jonassen et al. (1999, pp. 47–49):

  1. Determine if the task is amenable to a procedural analysis. Listing assets and liabilities, looking up their values, and sometimes, appraising values are overt actions and can be conceived as a series of steps. However, recalling all relevant items and appraising values can require covert cognitive processes in some cases, so procedural analysis does not capture everything required for this task.
  2. Write down the terminal objective of the task. “Calculates their net worth by estimating and tallying the values of their real assets and liabilities.” Note that this task excludes analyses of liquidity, cash flow, monthly expenses, and interest rates on debts, which are also important components of one’s financial situation.
  3. Choose a task performer. I am the performer for this task. I achieved competence in this task three years ago. If the training is for novices, Jonassen et al. (1999) say the flowchart should be based on someone who has only achieved expertise recently, to avoid “an idiosyncratic sequence” (p. 47). For this task, Investopedia’s Net Worth Calculator (www.investopedia.com/net-worth) was examined to help guide the analysis. Additionally, based on my knowledge of personal finance, I accounted for a variety of common financial situations (e.g., marriage, retirement funds, etc.).
  4. Choose a data-gathering procedure. I took notes as a silently executed the task.
  5. Observe and record the procedure. I made a text-based list of tasks before starting, and opted to construct a flowchart while executing the net-worth task.
  6. Review and revise outline. This step was skipped, because I did not do an outline.
  7. Sketch out a flowchart of the task operations and decisions. See Figure 1. In constructing this flowchart, is was readily apparent that a complete flowchart would be “cumbersome in detail” (Jonassen et al., 1999, p. 53). Consequently, I constructed the flowchart at an abstracted level that condenses or generalizes many steps. For example, Item 210: “Cash equivalent asset or debt?” actually applies to a host of items including bank accounts, taxable investment accounts, mortgages, student and auto loans, and credit card debts. Item 120: “Recall and list real assets and liabilities …” implies the learner will list assets and debts as separate line items (e.g., house and mortgage would be listed separately). These details and others are omitted from the flowchart to prevent it from becoming overwhelming and unwieldy. At Item 200, a foreach loop is used to iterate over the array (list) of assets and debts, similar to the foreach construct in PHP, a popular web scripting language.
  8. Review the procedural flowchart. This was done during its construction.
  9. Field-test the flowchart. I compared the flowchart to the Investopedia’s Net Worth Calculator (www.investopedia.com/net-worth) to see if it could fit the same situations. The categories of assets and liabilities on this calculator all fit into items on the flowchart. A net-worth spreadsheet is more versatile than Investopedia’s calculator because it can be saved, amended, and reused.

Procedural-analysis flowchart for net-worth calculation task

Figure 1. Procedural-analysis flowchart for net-worth calculation task.

Critical-Incident Analysis

This type of analysis involves interviewing subject-matter experts (SMEs) to gain a realistic understanding of the task at hand, including the important elements (Jonassen et al., 1999). Interview or survey data from SMEs must be culled to remove noncritical elements, focus on the required behavior, and to arrange tasks by importance (Flanagan, 1954). You can also ask your SMEs to arrange tasks by importance (Jonassen et al., 1999).

Continue reading Task Analysis Comparison for Calculation of Net Worth

Game Theory, Relationships, and the “If They Cared” Dilemma

The prisoner’s dilemma has long annoyed me for assuming the prosecutor is actually going to be fair or tell the truth. I think the dilemma would be more realistic with Ken Kratz as the prosecutor, albeit the game theory elements would be eviscerated.

However, if we apply the dilemma to different examples, bastardizing it to some extent, it can be more realistic. Generally, the dilemma is evident in situations where working together would benefit both parties, but if only one moves to work together (e.g., “extending the olive branch”), he or she will suffer. If trust can be established, then the risk of nonreciprocation can be minimized (mutually assured cooperation, which is evidently a term I just made up). Then, both parties can benefit. These benefits are not necessarily limited to one instance—they may continue on an ongoing basis in business, academic, or personal relationships.

Examples of the prisoner’s dilemma in business relationships are obvious. Firms or even two departments within the same firm might refuse to share information, resulting in needless duplicated work for both parties, because they cannot be assured of reciprocation. If one party shares information yet does not receive anything back, the sharing party may be in a worse position than had they shared nothing at all (similar to counterparty risk). Huge corporations are built on mediating counterparty risk (e.g., eBay, or practically any corporation that acts as a middleman).

It may be less obvious that this bulwark of game theory can be applied to interpersonal relationships, including romantic relationships. The “if they cared” dilemma is the situation where two courting individuals refuse to text each other (or take some other action) because if the other person cared, he or she would already have done so. Ignoring the shakiness of this logic, we can see that if both individuals subscribe to this perspective, it can easily be the death knell for the courtship. If both assume the other party should take an action first, then neither may act and both may chalk it up to people being assholes in general, rather than recognizing the dilemma that exists if both parties subscribe to “if they cared” logic.

Obviously, one party needs to send a loving text message to the other, yet receiving a response to such a text message does not allow the “if they cared” hypothesis to be tested. Attempting to determine whether another person cares based on that person spontaneously texting is completely short-circuited by prompting them with a text message. If you text first, you cannot tell whether they cared or not!

However, if the other person is thinking the same thing about you, then you both will never text again. Agentic contact, that is, contact that is deliberately initiated via one’s personal agency, is impossible to tease out if you happen to re-ignite the courtship by texting first or bumping into each other in-person. While the cost of texting first can be characterized in terms of losing face or appearing desperate, I contend this is a trivial issue—the real cost is in losing the ability to test for agentic contact (as a proxy for actual caring).

“If they cared, they would do X” is fairly bad logic. However, it is surprisingly common, and represents a dilemma by which many people feel rebuffed and become jaded. Inherently, the prisoner’s dilemma and “if they cared” dilemma are both starkly opposed to the zero-sum game model. If we were to gamify dating, if both individuals care, both are losing points by not texting. Each thinks the other should demonstrate caring by texting first. However, both could gain points by texting, if they had mutually assured cooperation (e.g., could know the other individual cares). If not, the big risk is they are being misled, unable to ascertain the true level of commitment of the other party due to their inability to refrain from contact. While pedantic to modern Westerners, the implications for human evolutionary biology are engrossing.

What Is Financial Independence?

Here are my notes from a speech I gave yesterday at the Port Orange Toastmasters club contest. While I came in 2nd out of 3 contestants (Dr. Charles Carroll won with his speech on smiles), it was a great experience to compete in a club contest for the first time, and we had a larger audience than usual—about 25 people. I actually ended up speaking without the notes at all, though I set them on the table as a psychological device and could have looked at them at any time.

What Is Financial Independence?
Speech by Richard Thripp | 5–7 minutes
Port Orange Toastmasters Club Contest
February 22, 2017

How to FIRE yourself and never work another day in your life unless you absolutely want to.

Imagine never having to work another day in your life
When?
You might think at Age 65 or Age 70
What if it could be earlier?
FIRE yourself
That’s F-I-R-E
Financial Independence and Retire Early
Big online community: Reddit, blogs, et cetera
Many retiring at 40 or even 30
What do you need to FIRE yourself?
Ideally, about 20 times your annual cost-of-living
This is a lot
40 times a 6-month emergency fund
If your cost-of-living is $50,000 per year, you need to invest 20 times that = $1 million
Live off dividends
Invest where? Mostly whole-market or S&P 500 index funds [with Vanguard]
Max out 401(k) and IRAs for tax savings
NOT money market savings — must be stocks [Can put 401(k) + IRAs in stocks]
Whole market reduces risk
Long time horizon reduces risk
After maxing retirement accounts, use taxable investment accounts
Yield of whole-market index fund = Average 7%, INFLATION ADJUSTED, per year
Money doubles every 10 years
MUCH HARDER to achieve financial independence without stocks
Long-term (> 1 year) capital gains tax only 20% max, compared to 39.6% for earned income
What is financial independence?
The ability to not work another day in your life

It doesn’t mean you MUST not work
Many people enjoy working
If freelancing, et cetera, the 20× rule might become the 10× rule
HOW to become financially independent?
MUST save tons of money
Maybe 50% of income
Hard if you’re a “shop-a-holic”
Hard if you have kids
But, many tricks
Example: Moving to a lower cost-of-living area when retiring
MUST change your money mindset
More tools available than ever
Get online and start reading
FIRE yourself


Here are the notes I took on my speech performance immediately after giving the speech. Yes, I wrote them in third person, which is weird.

How do you become financially independent?
Save save save
Invest in stocks
Low-fee index funds
Told story about TIAA charging 14% management fees and making it very hard to get money out
Trump’s in; Stock market going “bonkers” — why was my TIAA money going down?
Richard engaged audience many times by asking them to raise their hand — how many know what financial independence is? How many believe a CD is a good vehicle (none)? How many believe all bonds is good (none)? Then, Richard joked how smart the audience was. He also joked about being able to leave a job where the boss is making inappropriate jokes requiring an uncomfortable smile to keep one’s job (referencing Charles’s speech).


A fellow Toastmaster approached me after the speech, incredulous that the stock market produces 7% annualized inflation-adjusted returns on average. Here is additional information I wrote for him:

Here is a source on the 7% annual average returns in whole-market or S&P 500 index mutual funds:

www.thesimpledollar.com/where-does-7-come-from-when-it-comes-to-long-term-stock-returns/

The FIRE (Financial Independence, Retire Early) calculator at FIREcalc.com can show you projected returns based on all prior year periods based on how much money you intend to retire with and how many years the money needs to last.

If you have enough money, you can just live off the dividends of this index fund: personal.vanguard.com/us/funds/snapshot?FundId=0585&FundIntExt=INT

Of course, we are in an unusual time right now (market is up 22% in the past year). It’s hard to say whether investing now is a good idea (is the market in a bubble?), but generally, trying to time the market doesn’t work well, so most people advise investing now, or a little bit each month.


More thoughts:

Obviously, if you put your money in stocks just for one year, there is about a 25% chance you will lose money, and maybe a 40% chance you won’t earn 7% returns. But, if the money is in there 10 years, the odds of at least 7% annualized inflation-adjusted returns are higher (which means your money could double—1.07^10 is 1.97), and over 20, 30, or 40 years, even higher still. This is why a long time horizon is important, and why retiring early with hefty investments is not only psychologically powerful, but financially powerful.

For further reading, I suggest James Collins’s blog and the Financial Independence Reddit forum.

How the Doctrine of Diversification Misses the Mark for Personal Financial Portfolios

Traditional financial wisdom says that diversification should be used to mitigate risk, by this recommendation operates only at the portfolio level. Typically, older individuals, closer to retirement age, are advised to put a greater percentage of their money in less risky investments like bonds and CDs, while younger individuals should have an aggressive, growth-oriented portfolio of mostly stocks.

However, this approach, by itself, fails to consider the underlying need for avoiding financial loss, which in many cases will never become more than unrealized loss. That is to say, as long as the money is not needed during the period the loss is occurring and ongoing, the primary benefit of diversification—the ability to withdraw money during a bear market without losing as much—is unrealized.

Fidelity’s article on diversification explains how a diversified portfolio of 70% stocks and 30% bonds and T-bills would have significantly outperformed a 100% stock portfolio from January 2008 to February 2009 (the 2008 financial crisis), with –35.0% returns instead of –49.7% returns. However, when considering wider time periods, the diversified portfolio underperforms the 100% stock portfolio. We can effectively characterize diversification as a hedge against risk, reducing volatility at the cost of decreased returns over longer time horizons.

However, one’s future discretionary income can actually be interpreted as a hedge against loss. That is, if an individual or family lives well below their means, they may be able to maintain a more aggressive portfolio, because they can fall back on their income in bear markets rather than having to cash in their investments. A danger with this approach is that financial recessions can result in layoffs and reduced income. However, if one’s income is relatively reliable (e.g., work that is somewhat “recession proof,” Social Security benefits, and certain types of pensions), this danger is mitigated. Thus, if a young-old person receiving Social Security (SS) benefits wishes to continue aggressive growth of their investments for whatever reason, they can consider their SS benefits a hedge against investment risk. For example, if they live comfortably on $50,000 per year and receive $18,000 per year in SS benefits, they only need $32,000 in additional income per year. To the extent this money must be withdrawn from aggressive investments to fund one’s living expenses, the income is a hedge against downside potential, because overall, stocks continue marching upward quite vigorously, even though they may go down in many calendar years. (I am, of course, referring to broad mutual funds rather than individual stocks or sectors.)

Research shows that trying to time the market is not an effective approach (e.g., Henriksson, 1984). Consequently, rather than trying to time the market, and without concern to tax-advantaged accounts or tax brackets with respect to withdrawals, the best time to put money in stocks is now, in a lump sum (not dollar cost averaging), and the best time to withdraw money is as far in the future as possible, as needed (that is, avoid withdrawing a lump sum when only a fraction of the money is needed at a particular time). This is because overall, the market marches upward, despite volatility along the way. Therefore, the largest gains, on average, will be yielded by placing money in aggressive investments (stocks, not bonds or CDs) for the longest continuous length of time. For most people, attempting to time the markets on either end (investing or divesting) is not only an exercise in futility, but will be detrimental.

This may sound contradictory. If I recommend avoiding market timing because it doesn’t work, why would I argue that the primary benefit of diversification can be nullified by market timing? Actually, what I am talking about is not market timing, but rather avoiding being compelled to liquidate a position due to the financial hardship during a bear market. In a recession, not only will your investment portfolio lose value—you may also lose your job or take a pay cut, your home will decline in value, et cetera. Inasmuch as the value of diversification lies in being able to liquidate better-performing assets in a recession to make up for loss of income and other extraneous factors, the value of aggressive investing lies in making increased gains overall.

Conjecture: If withdrawals are stochastic in timing, then on average, the returns from a diversified portfolio will always be inferior to a 100% stock portfolio.

The ability to use reliable discretionary income as a hedge against loss enables one to avoid compelled liquidation during a recession, which, absent emotional issues, decouples withdraw timing from market factors and consequently reduces the value of diversification. Therefore, a portfolio analysis that does not consider debts or income may have substantial opportunity costs if a client over-diversifies due to having reliable discretionary income which is not considered, or under-diversifies due to having fragile income or liabilities which are not considered. In summary, as a doctrine, diversification should be titrated with a holistic analysis of one’s overall situation.

Writing on education, finance, psychology, et cetera