Predictive Validity of Firecracker’s Dedicated Test Prep Practice Blocks

Posted by Elliott Bartsch on Aug 19, 2016 4:55:47 PM
Find me on:

Earlier this year, we launched a special mode in Firecracker – Dedicated Test Prep Mode – which allows students to continue – or even start – using Firecracker during those last few weeks leading up to their boards (i.e. right before and during your dedicated prep). Historically, students stopped using Firecracker during this period for the industry standard UFAP (U-World, First Aid, Pathoma). And while our goal was not necessarily to replace any or all of UFAP, we did want to provide students with the option to use features which we believed were more effective and couldn’t be found anywhere else, either independently or alongside other resources.

Since launching DTP Mode, we’ve collected a significant number of USMLE score reports from Firecracker users who recently took the USMLE Step 1 exam and used DTP Mode for their studying. We’re now excited to share some initial results. 

DTP Mode Usage Correlates Strongly with Board Performance

dtp_corelation.png

Performance on Firecracker’s first 40 question practice block is highly correlated with Step 1 exam (r=0.79, n=41, p<0.0001). We were surprised to find that just one practice block let us predict students’ Step 1 scores with less than 8.4 points of error on average.

We took a look at how we could further improve the model with data on how students used Firecracker’s DTP mode outside of their practice block exams. We found that once a student completed their first practice block exam, increased levels of effort on Firecracker’s practice questions led to higher Step 1 scores. Specifically, the number of simple questions a student answered and the proportion of clinical case questions answered correctly (regardless of the number of attempts for any given question) were both associated with increased Step 1 scores. Students who put more effort into mastering the basic concepts and applying them in clinical scenarios through Firecracker during did better on average (Table 2).

Altogether, the full model using both diagnostic block results and effort indicators predicts Step 1 scores within 7.56 points of the actual score. For comparison, this model explains 71% of the variance in Step 1 scores while the CBSSA exam, a four hour, 200 question exam explains 67% of the variance in Step 1 scores¹.

Students took their first practice block 30 days out from their Step 1 exam on average. Interestingly, the number of days between when student took their first practice block and their Step 1 exam was not a significant predictor of Step 1 scores.

We took a look at preliminary data on users who completed two of our 40 question practice blocks. This model was able to predict students’ Step 1 scores within 4.15 points on average and explained 77% of the variance in their Step 1 scores (Table 3).

We’re very excited by these initial results and we look forward to sharing more data. We’re hoping to have enough data to predict Step 1 scores at the subject level. If you haven’t already, please send us your USMLE or COMLEX score report at team@firecracker.me (both pages please!) for two free months of Firecracker access in return.

-Elliott
Director of Data Science, Firecracker

 ¹http://journals.lww.com/academicmedicine/Abstract/2010/10001/Relationship_Between_Performance_on_the_NBME.26.aspx

Table 1

Screenshot_2016-08-19_16.43.09.png

Table 2

Screenshot_2016-08-19_16.44.18.png 

Table 3

Screenshot_2016-08-19_16.44.45.png

Topics: USMLE, Big Data, Dedicated Prep

Welcome to Firecracker

Firecracker is an adaptive learning platform that uses amazing content coupled with spaced repetition to help medical students crush their exams. We offer programs for both individual students, and entire medical schools. If you'd like to learn more contact us at chat@firecracker.me.

Subscribe to Email Updates

Recent Posts