Firecracker's Recommendation Algorithm Specifics

Posted by elliott on Nov 6, 2014 3:00:00 PM

The recommendation engine uses the following parameters for each question you've seen:

  1. A weighted average of all your previous review scores. Recent review scores receive heavier weights.
  2. The length of time that has elapsed since you last reviewed that question
  3. The number of times you've seen that question

Firecracker uses this information to calculate the risk that you will forget the concept associated with that question. This risk of forgetting increases over time and increases at a faster rate for questions that you've only seen a few times or that you've scored poorly on.

Once our algorithm calculates this risk of forgetting for all your questions - we create a stack of questions for you to review with the questions with the highest risk of forgetting at the top.

For a given user, this is what their stack might look like. 

stack-2 

When you jump into questions on Firecracker, you start at the right end of the above distribution and "burn down the stack" working your way from right to left. When you do questions, they get pushed back to low priority (because you just saw them). As time elapses, questions increase in priority and move from low priority to high priority. The rate at which priorities increase is based on the inputs to our algorithm described earlier.

Here's an example of how the priorities change as a user reviews a question multiple times. Each dot below represents a different day.
sample_recalls

The priority of a given question decreases immediately after review. Priorities decrease more with higher scores and with more reviews.

Let's contrast this with a user who shows no improvement between reviews and scores a 1 with every review.
sample_recalls2Here we can see that even without improvement between reviews, the priority of the given question decreases as the number of reviews increases.

Let's look at one more example.

sample_recalls3

Here we can see how the priorities change as your score fluctuates. By using a weighted average of all your previous recalls in our priority calculation we hone in on the signal (your true understanding of a given question) and reduce the noise of score changes between reviews.

The intervals above are not fixed. You have the option to filter questions at any level of our content tree and do the questions that you're at the highest risk of forgetting. The intervals above are just to give you a sense of how long it may be until you see a question again given a certain review history.

-Elliott
Data Scientist, Firecracker

Topics: Medical Education

Welcome to Firecracker

Firecracker is an adaptive learning platform that uses amazing content coupled with spaced repetition to help medical students crush their exams. We offer programs for both individual students, and entire medical schools. If you'd like to learn more contact us at chat@firecracker.me.

Subscribe to Email Updates

Recent Posts