One of Firecracker’s main goals in partnering with an institution is to support faculty in two areas:
1) Guiding student self-study
2) Generating integrated assessments of, and for learning
Firecracker’s content experts and our Academic Operations team work with faculty to align Firecracker's topics and questions to Faculty learning objectives. The result is a course companion comprised of supportive topic summaries, review questions (e.g. flashcards, simple MCQs), and board-style practice questions. Questions can then be packaged into quizzes and tests depending on what, how, and when faculty and administrators want to assess their students.Note: Firecracker is always "assessing" student knowledge as students do questions Firecracker's algorithm believes will most aid their learning, retention, and application of material to board-style clinical situations. But, we find that faculty and administrators often want to compare apples to apples at specific points in time -- that's where formative quizzes and summative tests can play a role.
In order to maintain the integrity of faculty/course learning objectives within the Firecracker platform, no assessments or topics are assigned to students without the expressed approval from the faculty/administrator in charge.
Alignment for Course Creation
Phase 1: Learning Objectives
Like any alignment work, the process begins with a faculty member’s course materials. To ensure a fast turn-around, the learning objectives should be presented in the order in which they will be taught to the students in class (Week #, Lecture #).
Note: We do not recommend Faculty try to cover everything in the USMLE or COMLEX content descriptions, not even in their respective area of expertise. Why? Three reasons: 1. there is way too much material, 2. students are more than capable of learning many things on their own, and 3. accreditors like the LCME and COCA are increasingly requiring, as described in standards like LCME 6.3, that schools support self-directed learning.
You might ask, how does Firecracker fit into this process? Great question! Firecracker’s content team will work with other materials such as lecture slides, lecture videos, and/or course packs to create a parallel version of the coursework on our platform.
After receiving the course materials, the Firecracker team will organize them to facilitate easy alignment to our medical knowledge. Our academic operations manager will work directly with the faculty to collect this information. All course materials are sequestered for your school.
Phase 2: Faculty Review
Once Firecracker’s content team has completed the first pass at alignment, your dedicated Firecracker academic operations manager will review with the relevant faculty member(s) for review. Following that, the faculty confirms the topics listed and the topic summaries are accurate to ensure that the information is in fact what they wish their students to receive. The items that can be reviewed are the topic summary, the foundational active recall questions and multiple-choice questions (MCQs).
In the event that certain topics are considered inaccurate or inappropriate given the learning objectives provided, the academic operation manager will make edits as requested to the course aligned schedule. Only when the alignment has been properly vetted and approved by the faculty will the “course” be created and made available for student interaction.
Note: If Faculty wish, they can recommend students learn certain topics on their own, either during a specific week, range of a few weeks, or simply put them into a bucket of "related topics". Firecracker can recommend topics that relate to the material being covered in class, and even recommend specific weeks Faculty can choose to recommend student learn that material on their own.
Phase 1: Item Selection
Similar to the process of alignment, our content team will consider the testable learning objectives provided and select a set of items that they believe best represents those objectives. These items will be tagged in the spreadsheet with a description of the material, and linked to its location in our Assessment Browser.
Note: Firecracker Assessment Items include: 1) Single-order MCQs, 2) Compact Clinical Vignettes, and 3) Higher-Order Board-Style Case Questions.
For each assessment item in Firecracker, faculty can review the item stems, answer choices, answer explanations, the number of times a unique question was answered in our platform; and their IRT (item response theory) Curves. In addition, other metrics, such as the Difficulty Index and Discrimination Index, can be provided to the faculty who require them.
We encourage Faculty to provide high-level guidelines on what they want from each quiz or test. This will decrease Faculty review time dramatically. Answers to the following 3 questions are all that's necessary:
1. Length of quiz or test. How many questions in the quiz/test? Put another way, how long do you want the test to be?
2. Learning objective coverage. Do you want questions from prior weeks? Current week? Future weeks? A mix? 1/3rd, 1/3rd, 1/3rd?
3.Question format & difficulty. For example, do you want open-ended recall questions with self-confidence ratings (i.e. flashcard retention rated by students on 1-5 Likert scale)? Easy multiple-choice questions (MCQs)? Medium-difficulty MCQs? USMLE- or COMLEX-style MCQs? USMLE- or COMLEX-style clinical vingnettes, but open answer responses? Questions that integrate a lot of material, or questions that don't? We can support any quizzing or testing pedegogy you can imagine and even if we don't have items that you want yet, let us know and we will build them. Requests from partners for "medium-difficulty" and "medium-integration" questions that are more difficult and integrative than simple MCQs, but less difficult and integrative than USMLE- or COMLEX-style questions are the reason why we now have nearly 1,500 of these types of questions (we call them "course questions" since they are the types of questions prevelent in Faculty provided quizzes and tests today).
Note: If Faculty choose to recommend students learn topics on their own that they won't cover in class, Faculty can then go a step further and decide if they want to hold students accountable for this material on quizzes and/or tests, and if the related questions will be bonus (i.e. extra credit), weighted more or less than questions pertaining to material they do cover in class. Course directors and Deans love this framework because it allows them to demonstrate a Continous Quality Improvement (CQI) process. Imaging you find that students are performing in 90th percentile relative to their peers across the country on material you are covering in class, but 10th percentile on related material that you aren't covering in class. Adding review sessions during the term/block to address topics in the 10th percentile and making changes to the curriculum to decrease emphasis on certain topics for the next cohort is textbook data-driven CQI.
Phase 2: Faculty Review
Should the Faculty wish to edit the set of items curated by the Firecracker content team, we provide a number of ways to do so.
- Select different items associated with the same primary topic or different primary topic by replacing the original question link with the new item’s question ID number in the spreadsheet, which will be located just above the stem of the item.
- If no items in the Browser meet Faculty standards, or there are simply gaps that Faculty would like to see filled - with enough lead time, Firecracker will author items on an as-needed basis. For example, During a recent course pilot, Firecracker authored nearly 400 items specifically for the course while it was on-going, and provided faculty with item characteristics as their students engaged with those items.
- If Faculty have items of their own they wish to integrate with an assessment, Firecracker can upload them into our system and deliver them to the appropriate students at the appropriate time.
- Questions provided by Faculty are sequestered to ensure they are only delivered and available to students enrolled in the course using formatted quizzes.
- If Faculty would like the questions to be used in other schools or by all Firecracker’s users, then those questions are reviewed by the content team before being published.
Phase 3: Assessment Delivery
Once an assessment has been properly vetted and approved by Faculty, all that’s left is to deliver the assessment to the students at the appropriate time and with the appropriate parameters. All of those logistics are taken care of by Firecracker's dedicated account manager.
The parameters faculty decide on include, but aren’t limited to:
- Date and time assessment will be made available. Note: Students can access an assessment in Firecracker through the Take an Exam page in the mobile and web platform, or link to it directly via a message app (e.g. email, LMS, etc).
- The amount of time a student is allowed to take the assessment.
- If students can pause the exam.
- If students are allowed to see the results at the end or not.
Upon completion of the assessment, students will receive their performance evaluation and their necessary remediation tasks, and faculty will be able to track this performance, subsequent remediation and self-study tasks via the Institutional Dashboard.
That's it! We've delivered thousands of quizzes and tests and would love to speak with you if you want to learn more about what Firecracker can do for your institution. Let us know if you'd like us to walk you through this process in greater detail.
Academic Operations & Community Manager