2016 Firecracker Step 1 Performance Analysis

Posted by Elliott Bartsch on Sep 16, 2016 5:39:25 PM
Find me on:

Since August 2014, we’ve surveyed medical students annually about their Step 1 scores. We've gathered score reports from both Firecracker users and non-Firecracker users and used these reports and our own data to determine if Firecracker usage creates better Step 1 outcomes. Today we’re releasing the updated performance analysis based on this student data. This most recent data set includes 228 Firecracker users and 499 non-Firecracker users. We wanted to focus our analysis on users who have been active on Firecracker in past year because of the significant changes we have made to the platform since summer 2015 - the launch of Firecracker MD and Firecracker DO.  

We continue to see significant correlations between the extent to which a student learns, retains, and applies Firecracker’s material and their performance on the Step 1 exam. The reported national average for 2015 Step 1 scores was 229.  Our data shows that Firecracker MD users scored 15 points higher on their exam than non-Firecrackers - with an average score of 245. Further, Firecracker MD users scored 3 points higher than Firecracker Legacy users. 

step_1_scores.png

And still, Firecracker users don’t report studying more than non-Firecrackers, and these correlations persist even after adjusting for MCAT scores as a baseline measurement.

mcat_usage.jpg

This spring we launched Dedicated Test Prep mode and initiated an overhaul and expansion of our Q-bank. We looked at Firecracker MD and Firecracker DO students’ use of Dedicated Test Prep mode, and their performance on CVs (USMLE style questions, clinical cases). We found a strong correlation between use of Dedicated Test Prep mode and better Step 1 outcomes. We also found that the Step 1 practice exam blocks were highly predictive of Step 1 scores:

practice_exam_predictor.png

We did not find a correlation between the flat number of CVs answered and Step 1 outcomes. But we did find that performance on flashcards on medical concepts was a strong indicator of performance on related clinical case questions.

recall_case_correlation.png

Overall we found that consistent Firecracker usage correlated with better performance on clinical vignettes, and ultimately better Step 1 outcomes. You can read the full paper here. It includes details on the above findings as well as an appendix that covers a product glossary and additional data. If you have any questions, you can reach out to me directly at Elliott@Firecracker.me.

If you would like to subscribe to Firecracker, we are having a Back to School sale:

1 year for $200 up to 4 years for $450.

Topics: Learning & Memory Science, USMLE Step 1, Research, Data Science

Welcome to Firecracker

Firecracker is an adaptive learning platform that uses amazing content coupled with spaced repetition to help medical students crush their exams. We offer programs for both individual students, and entire medical schools. If you'd like to learn more contact us at chat@firecracker.me.

Subscribe to Email Updates

Recent Posts