Firecracker Step 1 and COMLEX Performance Analysis

Posted by Elliott Bartsch on Nov 20, 2015 6:52:28 PM
Find me on:

In August 2014 and again in October 2015 we surveyed medical students - Firecrackers and non-Firecrackers - about their test performance and scores. Through this initiative, we gathered USMLE Step 1 scores (actual score reports) from 1,115 students and COMLEX scores from 180 students. We’ve used this data to update our 2014 analysis on the efficacy of Firecracker for boosting exam scores. These are the results - including (for the first time!) results on Firecracker usage and COMLEX outcomes.

In this analysis “Firecrackers” were users who answered at least 1000 questions in the Firecracker program, which correlates to about 1 month of use. “Non-Firecrackers” were students who answered fewer than 1000 questions in Firecracker or did not have Firecracker accounts. We compared the distributions of non-Firecracker Step 1 scores with the distribution of Firecracker Step 1 scores.

Firecracker and Step 1

usmle_distribution.jpg

In this set we have data for 473 Firecracker users, and 627 non-Firecracker students. The mean non-Firecracker Step 1 score was a 228.7 with a standard deviation of 23.7.  The mean Firecracker score was a 240.3 with a standard deviation of 16.8. On average, Firecracker users out-scored non-Firecrackers by almost 12 points.

We also examined how, for Firecrackers, usage correlated with Step 1 score.

Number of recalls:

step_1_recalls_2.png

r=0.357,n=627,p<0.001

Number of unique questions answered on Firecracker:

step_1_no_unique_qs_2.png

r=0/340,n=627,p<0.0001

Final average question score (for all questions answered by that Firecracker):

step_1_final_score_2.png

r=0.363,n=627,p<0.001

There is a clear correlation, in each case, with increased usage and increased Step 1 score.

Firecracker and COMLEX 1

In this set we have data for 65 Firecracker users and 81 non-Firecracker students.

The mean non-Firecracker COMLEX Level 1 score was a 527.4 with a standard deviation of 81.2. The mean Firecracker COMLEX Level 1 score was a 586.8 with a standard deviation of 79.6. On average. Firecracker users out-scored non-Firecrackers by almost 60 points on the COMLEX 1 exam.

comlex_distribution.jpg

 

This is how Firecracker usage correlated with COMLEX scores for Firecrackers.

Number of recalls:

comlex_no_recalls_2.png

r=0.467,n=65,p<0.0001

Number of unique questions answered on Firecracker:

comlex_no_unique_qs_2.png

r=0.38,n=65,p=0.0016

Final average question score (for all questions answered by that Firecracker):

comlex_final_score_2.png

r=0.384,n=65,p<0.001

Total number of weeks student was active on Firecracker:

comlex_active_weeks.jpg

r=0.312, n=65, p=0.011

Again, we see a clear correlation between Firecracker usage and improved COMLEX score.

The white papers on these results, with appendices and details like overall average scores on these exams, will be launching in the next couple of weeks. In the meantime, if you have questions about this data - leave it in the comments or reach out to me directly at elliott@firecracker.me.

Topics: Big Data, USMLE Step 1, COMLEX

Welcome to Firecracker

Firecracker is an adaptive learning platform that uses amazing content coupled with spaced repetition to help medical students crush their exams. We offer programs for both individual students, and entire medical schools. If you'd like to learn more contact us at chat@firecracker.me.

Subscribe to Email Updates

Recent Posts