Outcomes assessment pitfalls: challenges to quantifying knowledge gain in a sex education game

School selection and gameplay protocol

The second deployment of the MFF game was in Chennai, India as part of research conducted by Dr. Swathi Padankatti and her team from the International Alliance for the prevention of AIDS in collaboration with the U.S. based game development team (Dr. Bertozzi’s group at Quinnipiac University) and Dr. Aparna Sridhar at U.C.L.A’s School of Medicine. Dr. Padankatti and her team identified three schools willing to participate in the study who could provide a total of 419 student players. Schools were selected based on the research team’s pre-existing relationships with administrators, with whom they had previously worked on AIDS education initiatives.

For game deployment, a set of 30 android tablets and headsets were set up in a school classroom, and groups of students in the target age group successively cycled through to play the game and discuss their experience. Groups were not segregated by sex. To ensure comfort and privacy, students were able to move freely around the room to find their preferred space to sit and play. Upon beginning the game, players are first asked to indicate their sex and age, after which the pre-game quiz is triggered prior to initiating the main game. The post-game quiz, with exactly the same structure and questions as the pre-game, appears after completing the game. Given that this was the first field deployment of the revised game, the deployment team reported issues to the U.S. development team after each play session. The issues were collected and organized into topics to be addressed in future revisions of the game.

In addition to collecting data about family planning intentions, the research and development teams had two additional goals: to overcome challenges identified during the first deployment, and to assess knowledge gain via a pre-post testing framework.

MFF (original and modified versions of the game are available here: https://osf.io/gtfu5/wiki/home/ (Bertozzi-Villa et al., 2020). The apks can be installed on any Android tablet or phone.)

Challenges from first deployment

Following the initial deployment of the game, issues with the original study protocol were identified and addressed for this second deployment.

During the early stages of pilot deployment, teachers stayed in the room during gameplay. They often gave stern instructions on how to behave and ordered students to follow the instructions of the researchers. We realized that this made it impossible for students to experience playing the game as play. Due to the presence of their teachers it felt more like a test that they were required to engage in. To encourage a sense of play, the research protocol was modified early in the first deployment to ask instructors to leave the room during gameplay. Additionally, language was added to the introductory scripts, encouraging students to play the MFF game as a game – they should only do the parts of it that they wanted to, and could stop playing at any time. This protocol was extended into the second deployment.

A usability issue encountered during the first deployment was that players lacked familiarity with the drag and drop interface commonly used on smartphones and tablets. The design team determined that the pre-test was the perfect opportunity to teach players how to use drag and drop so that they would be prepared for it when they reached the game.

The third main issue identified in the pilot involved the post-game questionnaires. These were paper forms filled out by students after playing the game, asking students to qualitatively self-assess knowledge gain and provide feedback on the process of gameplay. While these questionnaires provided valuable feedback and indicated high rates of self-assessed knowledge gain, they were not efficient data-collection strategies. Because forms were filled out on paper, response rates were low and it was not possible to link student feedback to specific test-takers. Positive self-assessment of knowledge gain was encouraging, but not a rigorous method for determining game efficacy. The absence of an evaluative framework for the game was the primary motivation for development of the pre-post testing process.

Development of pre-post assessment

The key educational content of each milestone of the game is outlined in Table 1.

To assess knowledge of these questions while training students on a drag-and-drop interface, the pre- and post-games were designed to show male and female figures in outline, with internal organs visible. A series of 14 anatomy questions covering the full scope of in-game content was presented in a sequence of views. Players answered questions in the pre-test by dragging a word representing a concept (usually with an animation to help explain it) to the correct location on an image (Figure 3). The structure and content of pre-and post-game tests is identical.

Figure 3. Example of pre-test anatomy question.

The assessment was designed to correlate with the way information is delivered in the different milestones in the game. In the first deployment we noted that players had a difficult time understanding where different organs were located in the body and what their functions were. In the assessments, we were careful to depict both male and female bodies as a whole at the start of the assessment. The view then zooms in to just the abdomens of the male and female bodies. We added the whole person views in the top right and left of the screen so that players could understand which view of the body was presented to them. It is very difficult to understand how organs are laid out relative to other organs. For example, in the female body, it can be difficult to show the positions of the three apertures of the urethra, vagina, and rectum relative to one another. The additional views were added to minimize this confusion.

Our hope was that the layout of the assessment prior to play would prepare players to approach the anatomy section of the game where they have to drag and drop each body part to its correct location (Figure 4). During the first deployment of the game, it was clear that some players did not understand the difference between the front and side views of the anatomical drawings. In addition to adding the side views in the upper right and left corners of the assessment, we also incorporated them into the minigame. These views update as each organ is dragged into the correct location in the front view.

Figure 4. Screenshot of anatomy minigame.

Pitfall one - Pre-test made the game feel like a test

As described in the methods, considerable care was taken in the game design to encourage a sense of play and remove the pressure associated with an examination. By introducing a pre-test, however, we recreated the circumstances under which the experience of play was potentially undermined. Students were invited into the room to play a game. However, after they are welcomed to the game, they are presented with an assessment. The deployment team reported that some students were concerned that they did not have the “right” answer and wanted to be able to go back and correct their previous answers during the pretest. Given that the Indian system of education heavily relies on test scores and impactfully rewards those who test well, these students appeared very motivated to “do well” as soon as they realized it was an assessment.

To counter this, the researchers repeatedly stated that they should just answer what they knew and then go on to the game, but this clearly affected the experience. We learned that in future deployments, we need to add more context and less pressure to the pre-test to ensure players understand that they will not be criticized or penalized for not knowing the answers.

Pitfall two – Language issues with terms for sexual parts and functions

To be as accessible as possible to players at any reading comprehension level, the game includes as little text as possible and communicates most information through graphics, audio and animation. This is especially important when discussing information about sexuality because these terms may not be familiar to students. However, the inclusion of the pre-post assessment introduced a great deal of technical vocabulary in English before the gameplay began. All of the schools included in the study had instruction in English, but it was unclear if terms like testicles, ovaries, urine and feces were well understood by players (Figure 5). Although education about sexual functions is technically part of the educational curriculum for all students in India, the content is not actually taught in many schools due to cultural reluctance to discuss sexuality. In the results that follow, it is possible that some of the variation in scores on the pre-test is related to differences in knowledge of terms rather than differences in knowledge of sexual/anatomical functions.

Figure 5. Final anatomy screen from pre-test.

Pitfall three – Low participation in the post-test

While the transferal of the post-test questionnaire into a digital framework did allow for personalized tracking of results, there were challenges to collecting post-game information. Qualitative feedback from the deployment team indicated that, when students came to the end of the game, and saw the same screen they had seen earlier for the pre-test, many simply dragged the tiles to “Not Sure” because it was the fastest way to get to the final screen. Others simply put down the tablet which meant that researchers had to exit the player from that game session (with no responses to the post-test questions) to reset the tablet for the next group of students. Due to the fact that we do not know exactly what happened in all the cases where there appear to be random answers to the post-test, we cannot determine how many students actually answered the questions intentionally. We failed to provide players with a compelling reason to want to engage in the final assessment, which will be corrected in the next deployment.

Assessment of pre-test scores

A total of 419 students in three schools completed the pre-test and main game. The schools were selected based on scheduling availability and willingness to participate. The researchers from the IAPA had previously worked with these schools on AIDS education initiatives. Across all schools, the pre-test score was 33.5% on average (SE 1.15%), with substantial variation between schools. In particular, students at School 2 (who had sexual education as a formal part of their curriculum) performed significantly better than students at Schools 1 or 3 (two-tailed t-test p<0.001). Pre-test scores were not significantly different between male and female students at any school (Figure 6).

Figure 6. Pre-test scores by school and sex of respondent.

Bars represent the standard error of the mean. N=419.

Across all schools, students scored slightly better on pre-test questions relating to the anatomy of their own sex compared to the opposite sex, but this effect was not statistically significant (Figure 7). For female anatomy questions, 34.4% (SE 1.86%) of female respondents answered correctly, compared to 30.6% (SE 1.6%) of male respondents (two-tailed t-test p=0.13). For male anatomy questions, 33.3% (SE 1.89%) of female respondents answered correctly, compared to 36.3% (SE 1.8%) of male respondents (two-tailed t-test p=0.25).

Figure 7. Pre-test scores by type of question and sex of respondent.

Bars represent the standard error of the mean. N=419.

As shown in Figure 8, the only questions for which a majority of responses were correct were “Where is urine excreted from a male?” (53.0% correct) and “Where does a lining build up to prepare for pregnancy?” (50.6% correct). For eight of the remaining 12 questions, the correct answer received a plurality of responses, but not a majority. The four questions for which the most frequent response was not the correct answer were “Where sperm exit the body?” (plurality answer “Not Sure”, 29.8%), “where menstrual blood is excreted?” (plurality answer “Not Sure”, 27.7%), “The organ that becomes erect before intercourse?” (plurality answer “Vagina”, 33.7%), and “Where urine is excreted from a female? (plurality answer “Vagina”, 35.6%)”.

Figure 8. Pre-test responses by question.

The highlighted bar shows the correct answer in every instance. N=419.

Pre-post test assessment

As described above, assessment of knowledge gain was complicated by the large number of students who did not complete the post-test or who rushed through it, answering “not sure” to all questions (173 students, 41.3% of total). We refer to this group as having a “null” post-test. While it is not possible to assess knowledge gain among those with a null post-test, among the 246 (58.7%) students who did attempt the post-test we find on average a 6.27-point score gain between pre- and post- tests (95% CI 3.8-8.75, one-sample t test, Figure 9).

Figure 9. Violin plot of score distributions in the pre- and post-test, for those students who completed both tests (N=246).

Points represent means, and bars represent two times the standard error.

A question-by-question breakdown of pre- vs post- test result among those who attempted the post-test shows the largest knowledge gain around topics of intercourse, egg storage, and sperm movement (Figure 10).

Figure 10. Question-by-question comparison of pre- and post- test responses, by proportion, excluding those with “null” post-tests (N=246).

Underlying data

Open Science Framework: Outcomes Assessment Pitfalls: Challenges to Quantifying Knowledge Gain in a Sex Education Game. https://doi.org/10.17605/OSF.IO/WMHCD (Bertozzi-Villa et al., 2020)

This project contains the following underlying data

Extended data

Pre- and post-test data were analyzed and visualized using R version 3.6.0. All code is available from GitHub (https://github.com/bertozzivill/india-family-planning) and archived with Zenodo (http://doi.org/10.5281/zenodo.3822455 (Bertozzi-Villa, 2020))

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).