on 17-Nov-2024 (Sun)

Most of my Anki-based reading is much shallower than my read of the AlphaGo paper. Rather than spending days on a paper, I'll typically spend 10 to 60 minutes, sometimes longer for very good papers. Here's a few notes on some patterns I've found useful in shallow reading.

This doesn't mean reading every word in the paper. Rather, I'll add to Anki questions about the core claims, core questions, and core ideas of the paper. It's particularly helpful to extract Anki questions from the abstract, introduction, conclusion, figures, and figure captions. Typically I will extract anywhere from 5 to 20 Anki questions from the paper. It's usually a bad idea to extract fewer than 5 questions – doing so tends to leave the paper as a kind of isolated orphan in my memory.

How to avoid Ankifying misleading work?

As an example, let me describe how I Ankified a paper I recently read, by the economists Benjamin Jones and Bruce Weinberg* * Benjamin F. Jones and Bruce A. Weinberg, Age Dynamics in Scientific Creativity, Proceedings of the National Academy of Sciences (2011). . The paper studies the ages at which scientists make their greatest discoveries.

As an example of that caution, one of the questions I added to Anki was: “What does Jones 2011 claim is the average age at which physics Nobelists made their prizewinning discovery, over 1980-2011?” (Answer: 48). Another variant question was: “Which paper claimed that physics Nobelists made their prizewinning discovery at average age 48, over the period 1980-2011?” (Answer: Jones 2011). And so on.

Such questions qualify the underlying claim: we now know it was a claim made in Jones 2011, and that we're relying on the quality of Jones and Weinberg's data analysis. In fact, I haven't examined that analysis carefully enough to regard it as a fact that the average age of those Nobelists is 48. But it is certainly a fact that their paper claimed it was 48. Those are different things, and the latter is better to Ankify.

If I'm particularly concerned about the quality of the analysis, I may add one or more questions about what makes such work difficult, e.g.: “What's one challenge in determining the age of Nobel winners at the time of their discovery, as discussed in Jones 2011?” Good answers include: the difficulty of figuring out which paper contained the Nobel-winning work; the fact that publication of papers is sometimes delayed by years; that sometimes work is spread over multiple papers; and so on.

It's worth deliberately practicing such switches, to avoid building a counter-productive habit of completionism in your reading. It's nearly always possible to read deeper into a paper, but that doesn't mean you can't easily be getting more value elsewhere. It's a failure mode to spend too long reading unimportant papers.

There's also a sense in which it's possible to use Anki not just to read papers, but to “read” the entire research literature of some field or subfield. Here's how to do it.

In fact, to really grok an unfamiliar field, you need to engage deeply with key papers – papers like the AlphaGo paper. What you get from deep engagement with important papers is more significant than any single fact or technique: you get a sense for what a powerful result in the field looks like. It helps you imbibe the healthiest norms and standards of the field. It helps you internalize how to ask good questions in the field, and how to put techniques together. You begin to understand what made something like AlphaGo a breakthrough – and also its limitations, and the sense in which it was really a natural evolution of the field. Such things aren't captured individually by any single Anki question. But they begin to be captured collectively by the questions one asks when engaged deeply enough with key papers

So, to get a picture of an entire field, I usually begin with a truly important paper, ideally a paper establishing a result that got me interested in the field in the first place. I do a thorough read of that paper, along the lines of what I described for AlphaGo. Later, I do thorough reads of other key papers in the field – ideally, I read the best 5-10 papers in the field. But, interspersed, I also do shallower reads of a much larger number of less important (though still good) papers.

You may wonder why I don't just focus on only the most important papers. Part of the reason is mundane: it can be hard to tell what the most important papers are.

But there's also a culture that one imbibes reading the bread-and-butter papers of a field: a sense for what routine progress looks like, for the praxis of the field.

Indeed, while I don't recommend spending a large fraction of your time reading bad papers, it's certainly possible to have a good conversation with a bad paper. Stimulus is found in unexpected places.

Over time, this is a form of what Mortimer Adler and Charles van Doren dubbed syntopic reading

I start to identify open problems, questions that I'd personally like answered, but which don't yet seem to have been answered. I identify tricks, observations that seem pregnant with possibility, but whose import I don't yet know. And, sometimes, I identify what seem to me to be field-wide blind spots. I add questions about all these to Anki as well. In this way, Anki is a medium supporting my creative research.

This captures something of the immense emotional effort I used to find required to learn a new field. Without a lot of drive, it was extremely difficult to make a lot of material in a new field stick. Anki does much to solve that problem.

In a sense, it's an emotional prosthetic, actually helping create the drive I need to achieve understanding. It doesn't do the entire job – as mentioned earlier, it's very helpful to have other commitments (like a creative project, or people depending on me) to help create that drive.

Make most Anki questions and answers as atomic as possible: That is, both the question and answer express just one idea.

As an example, when I was learning the Unix command line, I entered the question: “How to create a soft link from linkname to filename?” The answer was: “ln -s filename linkname”. Unfortunately, I routinely got this question wrong.

The solution was to refactor the question by breaking it into two pieces. One piece was: “What's the basic command and option to create a Unix soft link?” Answer: “ln -s …”. And the second piece was: “When creating a Unix soft link, in what order do linkname and filename go?” Answer: “filename linkname”.

Note that this doesn't mean you shouldn't also retain some version of the original question. I still want to know how to create a soft link in Unix, and so it's worth keeping the original question in Anki. But it becomes an integrative question, part of a hierarchy of questions building up from simple atomic facts to more complex ideas.

One benefit of using Anki in this way is that you begin to habitually break things down into atomic questions. This sharply crystallizes the distinct things you've learned. Personally, I find that crystallization satisfying, for reasons I (ironically) find difficult to articulate. But one real benefit is that later I often find those atomic ideas can be put together in ways I didn't initially anticipate. And that's well worth the trouble.

Part of developing Anki as a virtuoso skill is cultivating the ability to use it for types of understanding beyond basic facts. Indeed, many of the observations I've made (and will make, below) about how to use Anki are really about what it means to understand something. Break things up into atomic facts. Build rich hierarchies of interconnections and integrative questions. Don't put in orphan questions. Patterns for how to engage with reading material. Patterns (and anti-patterns) for question types. Patterns for the kinds of things you'd like to memorize. Anki skills concretely instantiate your theory of how you understand; developing those skills will help you understand better. It's too strong to say that to be a virtuoso Anki user is to be a virtuoso in understanding. But there's some truth to it.

Anki isn't just a tool for memorizing simple facts. It's a tool for understanding almost anything. It's a common misconception that Anki is just for memorizing simple raw facts, things like vocabulary items and basic definitions. But as we've seen, it's possible to use Anki for much more advanced types of understanding

Use one big deck: Anki allows you to organize cards into decks and subdecks. Some people use this to create a complicated organizational structure. I used to do this, but I've gradually* * It's gradual because questions sometimes need to be rewritten due to the changed context. For instance, both my Emacs and Unix command line decks had very similar questions, along the lines of: “How to delete a word?” Those questions need to be rewritten, e.g. as: “In Emacs, how to delete a word?” (This, by the way, may seem a strange question for a long-time Emacs user such as myself. In fact, I've used Anki to help me change the way I delete words in Emacs, which is why I have an Anki question on the subject. I have made many improvements to my Emacs workflow this way.) merged my decks and subdecks into one big deck. The world isn't divided up into neatly separated components, and I believe it's good to collide very different types of questions.

Avoid orphan questions: Suppose I'm reading online and stumble across a great article about the grooming habits of the Albanian giant mongoose, a subject I never previously knew I was interested in, but which turns out to be fascinating. Pretty soon I've Ankified 5 to 10 questions. That's great, but my experience suggests that in a few months I'll likely find those questions rather stale, and frequently get them wrong. I believe the reason is that those questions are too disconnected from my other interests, and I will have lost the context that made me interested.

I call these orphan questions, because they're not closely related to anything else in my memory.

It's particularly worth avoiding lonely orphans: single questions that are largely disconnected from everything else. Suppose, for instance, I'm reading an article on a new subject, and I learn an idea that seems particularly useful. I make it a rule to never put in one question. Rather, I try to put at least two questions in, preferably three or more. That's usually enough that it's at least the nucleus of a bit of useful knowledge. If it's a lonely orphan, inevitably I get the question wrong all the time, and it's a waste to have entered it at all.

Don't share decks: I'm often asked whether I'd be willing to share my Anki decks. I'm not. Very early on I realized it would be very useful to put personal information in Anki. I don't mean anything terribly personal – I'd never put deep, dark secrets in there. Nor do I put anything requiring security, like passwords. But I do put some things I wouldn't sling about casually.

Construct your own decks: The Anki site has many shared decks, but I've found only a little use for them. The most important reason is that making Anki cards is an act of understanding in itself. That is, figuring out good questions to ask, and good answers, is part of what it means to understand a new subject well. To use someone else's cards is to forgo much of that understanding.

Indeed, I believe the act of constructing the cards actually helps with memory. Memory researchers have repeatedly found that the more elaborately you encode a memory, the stronger the memory will be. By elaborative encoding, they mean essentially the richness of the associations you form.

For instance, it's possible to try to remember as an isolated fact that 1962 was the year the first telecommunications satellite, Telstar, was put into orbit. But a better way of remembering it is to relate that fact to others. Relatively prosaically, you might observe that Telstar was launched just 5 years after the first Soviet satellite, Sputnik. It didn't take long to put space to use for telecommunications. Less prosaically – a richer elaboration – I personally find it fascinating that Telstar was put into orbit the year before the introduction of ASCII, arguably the first modern digital standard for communicating text. Humanity had a telecommunications satellite before we had a digital standard for communicating text! Finding that kind of connection is an example of an elaborative encoding.

The act of constructing an Anki card is itself nearly always a form of elaborative encoding. It forces you to think through alternate forms of the question, to consider the best possible answers, and so on. I believe this is true for even the most elementary cards

Cultivate strategies for elaborative encoding / forming rich associations: This is really a meta-strategy, i.e., a strategy for forming strategies. One simple example strategy is to use multiple variants of the “same” question. For instance, I mentioned earlier my two questions: “What does Jones 2011 claim is the average age at which physics Nobelists made their prizewinning discovery, over 1980-2011?” And: “Which paper claimed that physics Nobelists made their prizewinning discovery at average age 48, over the period 1980-2011?” Logically, these two questions are obviously closely related. But in terms of how memory works, they are different, causing associations on very different triggers.

Here's Joshua Foer recounting a conversation where mnemonist Ed Cooke describes one basic technique:

Ed then explained to me his procedure for making a name memorable, which he had used in the competition to memorize the first and last names associated with ninety-nine different photographic head shots in the names-and-faces event. It was a technique he promised I could use to remember people's names at parties and meetings. “The trick is actually deceptively simple,” he said. “It is always to associate the sound of a person's name with something you can clearly imagine. It's all about creating a vivid image in your mind that anchors your visual memory of the person's face to a visual memory connected to the person's name. When you need to reach back and remember the person's name at some later date, the image you created will simply pop back into your mind… So, hmm, you said your name was Josh Foer, eh?” He raised an eyebrow and gave his chin a melodramatic stroke. “Well, I'd imagine you joshing me where we first met, outside the competition hall, and I'd imagine myself breaking into four pieces in response. Four/Foer, get it? That little image is more entertaining—to me, at least—than your mere name, and should stick nicely in the mind.”

I've experimented with these techniques, and while they're fun, they seem most useful for memorizing trivia – sequences of playing cards, strings of digits, and so on. They seem less well developed for more abstract concepts, and such abstractions are often where the deepest understanding lies.

95% of Anki's value comes from 5% of the features: Anki has ways of auto-generating cards, of tagging cards, a plugin ecosystem, and much else. In practice, I rarely use any of these features. My cards are always one of two types: the majority are simple question and answer; a substantial minority are what's called a cloze: a kind of fill-in-the-blanks test. For instance, I'll use clozes to test myself on favorite quotes

I know many people who try Anki out, and then go down a rabbit hole learning as many features as possible so they can use it “efficiently”. Usually, they're chasing 1% improvements. Often, those people ultimately give up Anki as “too difficult”, which is often a synonym for “I got nervous I wasn't using it perfectly”. This is a pity. As discussed earlier, Anki offers something like a 20-fold improvement over (say) ordinary flashcards.

Procedural versus declarative memory: There's a big difference between remembering a fact and mastering a process. For instance, while you might remember a Unix command when cued by an Anki question, that doesn't mean you'll recognize an opportunity to use the command in the context of the command line, and be comfortable typing it out. And it's still another thing to find novel, creative ways of combining the commands you know, in order to solve challenging problems.

Put another way: to really internalize a process, it's not enough just to review Anki cards. You need to carry out the process, in context. And you need to solve real problems with it.

Getting past “names don't matter”: I'm a theoretical physicist by training. There is a famous story in physics, told by Richard Feynman, dismissing the value of knowing the names of things. As a child, Feynman was out playing in a field with a know-it-all kid. Here's what happened, in Feynman's telling* * Richard P. Feynman, “What Do You Care What Other People Think? Further Adventures of a Curious Character” (1989). :

One kid says to me, “See that bird? What kind of bird is that?”

I said, “I haven't the slightest idea what kind of a bird it is.”

He says, “It'a brown-throated thrush. Your father doesn't teach you anything!”

But it was the opposite. He [Feynman's father] had already taught me: “See that bird?” he says. “It's a Spencer's warbler.” (I knew he didn't know the real name.) “Well, in Italian, it's a Chutto Lapittida. In Portuguese, it's a Bom da Peida… You can know the name of that bird in all the languages of the world, but when you're finished, you'll know absolutely nothing whatever about the bird! You'll only know about humans in different places, and what they call the bird. So let's look at the bird and see what it's doing — that's what counts.” (I learned very early the difference between knowing the name of something and knowing something.)

Feynman (or his father) goes on to a thoughtful discussion of real knowledge: observing behavior, understanding the reasons for it, and so on.

It's a good story. But it goes too far: names do matter. Maybe not as much as the know-it-all kid thought, and they're not usually a deep kind of knowledge. But they're the foundation that allows you to build up a network of knowledge.

What do you do when you get behind? Anki becomes challenging when you get behind with cards. If you skip a day or two – or fifty – the cards begin to back up. It's intimidating to come back to find you have 500 cards to review in a day. Even worse, if you fall out of the Anki habit, you can get a very long way behind. I largely stopped using Anki for a 7-month period, and came back to thousands of backlogged cards.

Fortunately, it wasn't that hard to catch up. I set myself gradually increasing quotas (100, 150, 200, 250, and eventually 300) of cards per day, and worked through those quotas each day for several weeks until I'd caught up.

One caution is with books: reading an entire book is a big commitment, and adding Anki questions regularly can slow you down a lot. It's worth keeping this in mind when deciding how much to Ankify. Sometimes a book is so dense with great material that it's worth taking the time to add lots of questions. But unmindfully Ankifying everything in sight is a bad habit, one I've occasionally fallen into.

Avoid the yes/no pattern: One bad habit I sometimes slide into is having lots of Anki questions with yes/no answers. For instance, here's a not-very-good question I added when learning about graphical models in machine learning:

Is computing the partition function intractable for most graphical models?

The answer is “yes”. That's fine, as far as it goes. But it'd help my understanding to elaborate the ideas in the question. Can I add a question about for which graphical models the partition function is tractable? Can I give an example of a graphical model for which the partition function is intractable? What does it mean for computing the partition function to be intractable anyway? Yes/no questions should, at the least, be considered as good candidates for question refactoring*

Fluency matters in thinking. Alan Kay and Adele Goldberg have proposed* * Alan Kay and Adele Goldberg, Personal Dynamic Media (1977). the thought experiment of a flute in which there is “a one-second delay between blowing a note and hearing it!” As they observe, this is “absurd”. In a similar way, certain types of thoughts are much easier to have when all the relevant kinds of understanding are held in mind. And for that, Anki is invaluable.

One of the most cited papers in the relevant research literature* * Frank N. Dempster, The Spacing Effect: A Case Study in the Failure to Apply the Results of Psychological Research (1988). is a discussion of why these ideas aren't more widely used in education. Although written in 1988, many of the observations in the paper remain true today.

My own personal suspicion is that there are three main factors:

• In experimental research on memory, people consistently underestimate the gains that come from distributing their study in a manner similar to Anki. Instead, they prefer last-minute cramming, and believe it produces better results, though many studies show it does not.
• The psychologist Robert Bjork has suggested* *Robert A. Bjork, Memory and Metamemory Considerations in the Training of Human Beings (1994). the “principle of desirable difficulty”, the idea that memories are maximally strengthened if tested when we're on the verge of forgetting them. This suggests that an efficient memory system will intrinsically be somewhat difficult to use. Human beings have a complex relationship to difficult activities, and often dislike performing them, unless strongly motivated (in which case they may become pleasurable).
• Systems such as Anki are challenging to use well, and easy to use poorly.

It is interesting to consider developing systems which may overcome some or all of these issues.

Long-term memory is sometimes disparaged. It's common for people to denigrate “rote memory”, especially in the classroom. I've heard from many people that they dropped some class – organic chemistry is common – because it was “just a bunch of facts, and I wanted something involving more understanding”.

My somewhat pious belief was that if people focused more on remembering the basics, and worried less about the “difficult” high-level issues, they'd find the high-level issues took care of themselves.

But while I held this as a strong conviction about other people, I never realized it also applied to me. And I had no idea at all how strongly it applied to me. Using Anki to read papers in new fields disabused me of this illusion. I found it almost unsettling how much easier Anki made learning such subjects. I now believe memory of the basics is often the single largest barrier to understanding.

Over the years, I've often helped people learn technical subjects such as quantum mechanics. Over time you come to see patterns in how people get stuck. One common pattern is that people think they're getting stuck on esoteric, complex issues. But when you dig down it turns out they're having a hard time with basic notation and terminology.

One striking line of work was done (separately) by the researchers Adriaan de Groot and Herbert Simon, studying how people acquire expertise, focusing particularly on chess* * See, for instance, Herbert A. Simon, How Big is a Chunk?, Science (1974), and Adriaan de Groot, Thought and Choice in Chess, Amsterdam University Press (2008, reprinted from 1965). . They found that world-class chess experts saw the board differently to beginners. A beginner would see “a pawn here, a rook there”, and so on, a series of individual pieces. Masters, by contrast, saw much more elaborate “chunks”: combinations of pieces that they recognized as a unit, and were able to reason about at a higher level of abstraction than the individual pieces.

Simon estimated chess masters learn between 25,000 and 100,000 of these chunks during their training, and that learning the chunks was a key element in becoming a first-rate chess player. Such players really see chess positions very differently from beginners.

Many people's model of accomplished mathematicians is that they are astoundingly bright, with very high IQs, and the ability to deal with very complex ideas in their mind. A common perception is that their smartness gives them the ability to deal with very complex ideas. Basically, they have a higher horsepower engine.

It's true that top mathematicians are usually very bright. But here's a different explanation of what's going on. It's that, per Simon, many top mathematicians have, through hard work, internalized many more complex mathematical chunks than ordinary humans. And what this means is that mathematical situations which seem very complex to the rest of us seem very simple to them. So it's not that they have a higher horsepower mind, in the sense of being able to deal with more complexity. Rather, their prior learning has given them better chunking abilities, and so situations most people would see as complex they see as simple, and they find it much easier to reason about.

Now, the concept of chunks used by Simon in his study of chess players actually came from a famous 1956 paper by George Miller, “The Magical Number Seven, Plus or Minus Two”* * George A. Miller, The Magical Number Seven, Plus or Minus Two: Some Limits on our Capacity for Processing Information (1956). . Miller argued that the capacity of working memory is roughly seven chunks. In fact, it turns out that there is variation in that number from person to person, and a substantial correlation between the capacity of an individual's working memory and their general intellectual ability (IQ)* * A review of the correlation may be found in Phillip L. Ackerman, Margaret E. Beier, and Mary O. Boyle, Working Memory and Intelligence: The Same or Different Constructs? Psychological Bulletin (2006). . Typically, the better your working memory, the higher your IQ, and vice versa.

Exactly what Miller meant by chunks he left somewhat vague, writing:

The contrast of the terms bit and chunk also serves to highlight the fact that we are not very definite about what constitutes a chunk of information. For example, the memory span of five words that Hayes obtained… might just as appropriately have been called a memory span of 15 phonemes, since each word had about three phonemes in it. Intuitively, it is clear that the subjects were recalling five words, not 15 phonemes, but the logical distinction is not immediately apparent. We are dealing here with a process of organizing or grouping the input into familiar units or chunks, and a great deal of learning has gone into the formation of these familiar units.

In particular, someone with a lower IQ but able to call on more complex chunks would be able to reason about more complex situations than someone with a higher IQ but less complex internalized chunks.

In other words, having more chunks memorized in some domain is somewhat like an effective boost to a person's IQ in that domain.

Still, it seems plausible that regular use of systems such as Anki may speed up the acquisition of the high-level chunks used by experts* * To determine this it would help to understand exactly how these chunks arise. That still seems to be poorly understood. I wouldn't be surprised if it involved considerable analysis and problem-solving, in addition to long-term memory. . And that those chunks are then at the heart of effective cognition, including our ability to understand, to problem solve, and to create.

That is, memories decay. This isn't news! But the great German psychologist Hermann Ebbinghaus had the good idea of studying memory decay systematically and quantitatively* * Hermann Ebbinghaus, Memory: A Contribution to Experimental Psychology (1885). A recent replication of Ebbinghaus's results may be found in: Jaap M. J. Murre and Joeri Dros, Replication and Analysis of Ebbinghaus' Forgetting Curve (2015). . In particular, he was interested in how quickly memories decay, and what causes the decay. To study this, Ebbinghaus memorized strings of nonsense syllables – things like “fim“ and “pes” – and later tested himself, recording how well he retained those syllables after different time intervals.

Ebbinghaus found that the probability of correctly recalling an item declined (roughly) exponentially with time. Today, this is called the Ebbinghaus forgetting curve

Ebbinghaus's research suggested that the probability will decay exponentially after the re-test, but the rate of decay will be slower than it was initially. In fact, subsequent re-tests will slow the decay still more, a gradually flattening out of the decay curve as the memory is consolidated through multiple recall events

There are several related terms used for this set of phenomena, but we'll use the phrase “distributed practice”, meaning practice which is distributed in time, ideally in a way designed to maximally promote retention. This is in contrast to cramming, often known as massed practice, where people try to fit all their study into just one session, relying on repetition.

Since Ebbinghaus, there's been thousands of studies of different variations of distributed practice. These studies have taught us a great deal about the behavior of long-term memory. Most of all, they show emphatically that distributed practice outperforms massed practice

We don't understand in detail why exponential decay of memory occurs, or when that model breaks down. We don't have good models of what determines the rate of decay, and why it varies for different types of memories. We don't understand why the decay takes longer after subsequent recalls. And we have little understanding of the best way of expanding the inter-study intervals.

Creating the cards to review, however, is a complex discipline of its own that’s both an art and a science, and how well you do at designing your cards can make the difference between easy study sessions resulting in effective learning and frustrating ones resulting in mediocre learning. Card design is thus the most fundamental skill of spaced-repetition use, and it’s worth spending some time learning how to do it well.

This poses a challenge for users, because seemingly small variations in how you ask a question can push it into not testing the thing you intended it to. The SRS is rather like the proverbial genie: describe what you want to get in the wrong way, and the genie will interpret your wish in the most ridiculous way possible and you’ll learn something that’s of no practical use.

To put it another way, SRS questions are prone to overfitting. Overfitting, in statistical modeling and machine learning, occurs when a model gets “too good” at predicting the data you train it on – it starts to treat random noise present in the training data as a meaningful source of predictions.

Thus, overfitting improves the model’s accuracy rate on the training data but reduces it when it’s applied to new data, i.e., a real-world scenario

In a spaced-repetition context, because the SRS is so good at helping you learn things efficiently, it will optimize your memory of just about anything, but if you don’t phrase your question correctly, it will help you remember irrelevant information or random noise, rather than the information you actually wanted to know.

To avoid overfitting, the number-one rule when adding cards is this: every time you come across something you think you’d like to remember, identify exactly what you want to know, and be as precise about it as possible. Then, and only then, create one or more cards that ask about exactly that and nothing else

Questions should ask exactly one thing Permalink

Rule #4 of the Twenty Rules calls this the Minimum Information Principle: cards should never ask about more than one thing (techies might say that the card should be atomic).

Q: Give a description of the Python programming language.
A: Python is a high-level, interpreted, multi-paradigm programming language written by Guido von Rossum and maintained by the Python Foundation. It is often used for system administration, data analysis, and rapid prototyping, and has been consistently named among the five most popular languages in recent years.

This is a terrible card! Not that you shouldn’t want to know anything about Python, or that any of the information in it is explained poorly, but there is just far too much information. Unless you want to literally memorize the text of the answer by rote, you will never be able to correctly give all the information in it as an answer to that vague prompt. If you forget any part of the question, you’ll either have to fail it (thus dooming the card to come back practically every day since you can never satisfactorily learn all of it) or say you remembered it well when you didn’t. We could easily split this into a dozen questions, e.g.:

Q: Who designed the Python programming language?
A: Guido von Rossum.

Q: What group maintains the Python programming language?
A: The Python Foundation.

Q: Since 2015, Python has been ranked among the […number] most popular programming languages.
A: 5

Et cetera.

Every piece of information you learn consumes study time that you could be using to learn other information. Choose wisely.

Questions should permit exactly one answer Permalink

For efficient review, it must be immediately obvious both what is being asked and what single response will count as the correct answer; your thought process and your answer must be the same every time you review, in meaning if not in exact wording

Allowing correct but irrelevant answers Permalink

Here’s a cloze deletion from my AP US History deck (9 years ago now!). In my notation for cloze deletions, the bit in {curly braces} shows up as a [...] on the front of the card, asking you to fill in the blank; the whole sentence then shows up on the back.

The Articles of Confederation had no power to regulate {commerce}.

This is true, and an important point (the lack of regulation of commerce was an important problem that helped drive the creation of the U.S. Constitution, which followed the Articles). However, there are a practically infinite number of other correct answers we could give! The Articles of Confederation also had no power to regulate religious observances, taxation, bear hunting, the behavior of the British monarch, or the hours your neighbor can play the drums with her windows open. Obviously, some of these answers are more plausible than others, but the fact remains I could reasonably give an answer that was correct but not the answer on the back of the card.

Either way, I am not effectively reviewing the piece of information I set out to learn; I’m instead forcing myself to memorize, in addition to the actual answer, what this card is asking me. That means I’m effectively asking about two things and making it much harder to remember the card, for no benefit – in real life, knowing what one of my Anki cards was supposed to be asking is useless information.

The correct response, when you encounter this situation during review – and you will – is to stop reviewing for a moment and rewrite the card. We could do this in numerous ways, but here’s one possibility:

Economic and trade relations between states were difficult under the Articles of Confederation because the Articles granted no power to {regulate commerce}.

Of course, I could still come up with silly answers if I wanted to be contrary (I recommend being contrary – it’s fun). But, when reading the question in good faith, I can now be reasonably confident that if I don’t promptly remember the answer I intended, that means I’ve forgotten it.

Falling into the “example” trap Permalink

Beginners often write questions like this one:

Q: What’s an example of a non-combinatorial circuit?
A: Memory.

This is merely a special case of “allowing correct but irrelevant answers,” but I’ve seen it so frequently I want to call attention to it in particular. We have the same problem as before: not only do I have to remember that memory is a non-combinatorial circuit (the actual piece of information I wanted to know), I also have to remember that the specific example of a non-combinatorial circuit this card wants me to give is “memory”. An even worse version is “Give some examples of non-combinatorial circuits.” With this version, you can give different answers every time you see the card, including ones that aren’t on the card, and still have them be kind-of, sort-of, maybe correct!

In the real world, we’re rarely called upon to offer examples of a textbook term, but we often benefit from being able to recognize that something is an example of that idea.

project is doomed to either cancellation or unused obscurity if planning, scoping, and experimentation is not done properly

The effectiveness of the two questions immediately above depends in great part on how complex your own taxonomy of those subjects is. Considering the memory question, I know relatively little about circuits, so I’m unlikely to get confused about what property of the memory circuit I’m being asked about. If I designed electronics for a living, on the other hand, I would likely need to rewrite that card with a hint or some additional context to be sure I knew which answer it was seeking. This is one of the many reasons that cards someone else makes will seldom be as effective as cards you make yourself: you’re the only one who knows what cues will get you the best results.

Note that by default, the internal state of the network is reset after each batch, but more explicit control over when the internal state is reset can be achieved by using a so-called stateful LSTM and calling the reset operation manually

Allowing multiple interpretations of a question Permalink

In this type of bad question, the creator didn’t clearly identify what they were trying to remember and consequently produced a vague question.

APUSH: Americans moving to Texas [when it was under Mexican control] {did not adopt the ways of the region and remained Americans at heart}.

Is this the only thing I might have wanted to remember about Americans moving to Texas? Really? Now, no matter how bad this question is, I might be able to learn the answer (in fact, I have a nearly perfect, straight-3’s review history on this card), but the prompt is so vague that the only thing I’ve learned is “what to fill in the blank when I’m asked about Americans moving to Texas,” which is of little value in the real world.

Q: Why is the Earth’s rotation slowing down?
A: Tidal deceleration.

This one’s a bit more subtle, but still problematic. The issue here is that the desired level of detail is unclear. Is it asking for the term that describes this phenomenon? (Is there even such a term? We wouldn’t know from the question.) Or did it want an explanation of the forces involved in tidal deceleration?

Here’s a better version:

Q: What gravitational effect is causing the Earth’s rotation to slow over time?
A: Tidal deceleration

If we didn’t know what tidal deceleration was, we’d also want to add a separate card, or maybe even several cards, explaining that process. That’s true even if we didn’t think we particularly wanted to know what tidal deceleration was. We don’t have a choice here: we have to know what tidal deceleration is for this card to be meaningful information. Even if we’re just trying to pass an exam which includes this exact question, the card will be much, much easier to remember when we know what it means. This is Rule #1: Do not learn if you do not understand.

For instance, maybe we care that the earth’s rotation is slowing down because it means leap seconds are needed to keep our calendar in sync with the day/night cycle:

Q: Why are leap seconds periodically added to UTC time?
A: Because the earth’s rotation is slowing over time, causing UTC to drift out of alignment with the sun.

In any case, as always, we have to take a moment to consider what we want to know in order to ask the right questions and get the information encoded in a useful way

Another important point is that having these cards may not remind us that Earth’s rotation is slowing down (see my anecdote about cp -i at the beginning of the post).

Questions should not ask you to enumerate things Permalink

When you find a list of items in your reading, it’s often tempting to create a card asking what the members of the list are. Rules #9-10 of the Twenty Rules call these sets (or enumerations, if they go in a specific order), and explain that they’re some of the most difficult and frustrating cards, so we want to avoid them when possible.

A nasty illusion may make it appear to be, however. Consider the internal combustion engine, a classic example of a complex machine in many ways defined by its many parts. Someone who can name all the parts of the ICE will seem to have a good understanding of it. But this is an effect, rather than a cause. If you can name all the parts of the ICE but don’t know what they do, you still understand nothing at all about how it works.

Of course, once in a while it really is helpful to know a set of things. In this case, you can improve your performance markedly by (a) learning and fully understanding each individual member of the set through separate cards; and (b) ordering the set into an enumeration and developing a mnemonic device such as an acronym for the order. (I like to keep close to the minimum information principle by creating two cards for part (b), one asking what my mnemonic is and the other asking me to use the mnemonic to produce the answer.)

If you create and learn cards that ask deeper questions, you’ll likely still be able to give the correct “yes” or “no” but also know additional information about why this is the case.

Here’s an example from my computer-hardware-design deck:

Q: Is segmentation used on modern processors?
A: No, it was removed in the x86-64 platform.

Interestingly, you can see that I actually included the information needed to produce a better question in the answer. The answer is a terrible place for this kind of information, though: you’re never asked to actively recall it and you look at it for a fraction of the time you look at the question, so you’re unlikely to ever memorize it. Instead, we can rewrite the card:

Segmentation was common on older processors but was removed starting with the {x86-64} platform.

All this said, I do still find myself writing yes/no questions on occasion, especially when I’m in the early stages of learning a topic. Sometimes you don’t know the why yet, and it’s better at the moment that you simply learn that something is or isn’t, rather than go look up and learn a bunch of details just to be able to improve one flashcard. Perhaps later, you’ll have the additional understanding to rewrite the card.

That brings up an important point: questions are not set in stone. You’ll often miss potential problems with your new questions until you start reviewing them, and still other times you’ll learn a card and realize weeks or months later after reading a completely different resource that you were missing some important context or your initial understanding was flat-out wrong.

Questions should be context-free Permalink

Context-free here is meant in the sense of a context-free grammar, that is, a grammar within which the correct interpretation of any statement is independent of its surroundings. Your questions, in other words, should be 100% comprehensible without any surrounding context; if you were to find one written on a slip of paper that somebody dropped in the street, you should be able to understand exactly what the question is asking.

The topic or context should be stated at the beginning or near the beginning of the question, to prime your memory to retrieve the right kind of information and to facilitate reviewing cards from different subjects intermixed, which many people believe improves creativity. For instance, in several of my questions above, you saw qualifiers at the start of the question like OutSystems: or APUSH:. A word describing the topic worked into the sentence can work, too, although Rule #16 does recommend prefixes so you can be sure your brain gets started correctly.

The question should not be built around a particular source. It’s okay to cite your sources on your cards (actually, it’s a fantastic idea to reference sources in one way or another, because sooner or later you’re going to have doubts about the veracity of some card, or simply want to find more information about the topic). However, questions like this one are to be avoided:

Statistics: One of the major focuses of our book’s introduction is that it is useful to measure {what you don’t know, or the uncertainty you have}.

This other pattern uses the source information entirely differently: rather than making the question about the author’s book, it asks about the idea in the book while explaining where it came from. For example:

Q: Why, according to Cal Newport, are discoveries often made by multiple people at the same time?
A: These things are part of the “adjacent possible” and were thus particularly easy to discover.

This becomes an especially valuable pattern once you get past the basic textbook knowledge and how-to rules in a field, where agreement can no longer be taken for granted. You don’t want to memorize opinions or nascent theories as facts!

Retrieval practice prompts should be focused. A question or answer involving too much detail will dull your concentration and stimulate incomplete retrievals, leaving some bulbs unlit. Unfocused questions also make it harder to check whether you remembered all parts of the answer and to note places where you differed. It’s usually best to focus on one detail at a time.

Retrieval practice prompts should be precise about what they’re asking for. Vague questions will elicit vague answers, which won’t reliably light the bulbs you’re targeting.

Retrieval practice prompts should be tractable. To avoid interference-driven churn and recurring annoyance in your review sessions, you should strive to write prompts which you can almost always answer correctly. This often means breaking the task down, or adding cues.

Retrieval practice prompts should produce consistent answers, lighting the same bulbs each time you perform the task. Otherwise, you may run afoul of an interference phenomenon called “retrieval-induced forgetting” This effect has been produced in many experiments but is not yet well understood. For an overview, see Murayama et al, Forgetting as a consequence of retrieval: a meta-analytic review of retrieval-induced forgetting (2014). : what you remember during practice is reinforced, but other related knowledge which you didn’t recall is actually inhibited.

Retrieval practice prompts should be effortful. It’s important that the prompt actually involves retrieving the answer from memory. You shouldn’t be able to trivially infer the answer. Cues are helpful, as we’ll discuss later—just don’t “give the answer away.”

So we must learn two skills to write effective retrieval practice prompts: how to characterize exactly what knowledge we’ll reinforce, and how to ask questions which reinforce that knowledge.