Measuring quality of health care using video vignettes

By Sheheryar Banuri, Lecturer, University of East Anglia; Damien de Walque, Senior Economist, The World Bank; Philip Keefer, Principal Economic Advisor, Inter-American Development Bank; Haidara Ousmane Diadie, Senior Health Specialist, The World Bank; Paul Jacob Robyn, Senior Health Specialist, The World Bank; Maurice Ye, MD, MPH, Centre de Recherche en Santé de Nouna, Burkina Faso

Quality of care is a crucial determinant of good health outcomes, but is difficult to measure. Survey vignettes are a standard approach to measuring medical knowledge among health care providers.  They give surveyed staff a typical case with symptoms and ask them to provide a diagnosis and recommend a treatment course.

But written vignettes or knowledge tests might be too removed from clinical practice, particularly in developing countries and particularly where “learning by doing” may be an important form of training. They are also not well suited to convey the heterogeneity of patients coming for consultation.

To try to address these shortcomings, we decided to use videos to develop a new type of provider vignette. What better place to work on these video vignettes than Ouagadougou, the capital of Burkina Faso, sometimes called the “Capital of African Film” and organizer of FESPACO? The renowned director Boubacar Diallo shot and edited the videos, most depicting a patient with maternal/early childhood symptoms visiting the clinic.

We have posted 4  video vignettes including their scripts and questions with answer keys. both in their original French and translated English versions. Here are two examples, one dubbed in English and one in French:

2E Preterm Labor: simple case (English)

7E Child Pneumonia: complex case (French)

The video vignettes were applied within the context of a “lab-in-the-field experiment” in Burkina Faso that tests the effects of various incentive measures on provider performance in a “laboratory” setting. The lab experiment is one study component of a larger impact evaluation on a Performance Based Financing (PBF) pilot in Burkina Faso.  These video vignettes – for which we selected an actress of neutral appearance in terms of ethnicity – have also been included in the baseline surveys for the impact evaluation in the Republic of Congo and in Cameroon. They are easy to integrate into a tablet-based questionnaire.

The cases followed symptom description and treatment protocols as defined by Burkina Faso’s national therapeutic guidelines as well as WHO Integrated Management of Childhood Illness (IMCI) guidelines. They were developed and tested with nurses at two nursing schools in Ouagadougou in January 2014. Cases with adequate variation in responses were kept while others were replaced with new cases.

We used 4 cases: a pregnant woman experiencing pre-term labor, a 6-month-old child with pneumonia, a lactating woman with mastitis and a 3-year-old child with measles. In order to capture the variety of ways in which patients present themselves to practitioners, each case consisted of a video lasting either 60 or 100 seconds. In all cases, the same actress was used to play the role of the patient, and always in the same setting (an average clinic).

To distinguish differences in provider practice across different patient types, subjects were exposed to two types of cases: one “normal or simple” case and one case with a poor patient. To simulate the conditions of poverty, we used variations in the complexity and time needed to treat patients appropriately.  Two of the videos lasted 60 seconds each, and represented “simple” cases in which the actress used normal language and had a standard appearance. These cases were the pre-term labor and the mastitis cases.  The 100 second videos represented cases with more symptoms.  In addition, the actress used rambling language and appeared to be poorer and less educated. This more disadvantaged socio-economic status was conveyed through the patient’s dress and the vocabulary and language she used to describe her illness history and symptoms. The two 100 second cases were the child with pneumonia and the child with measles.

The participants viewed the video and were asked to select the (1) diagnosis, (2) treatment, (3) follow-up schedule, and (4) alternate treatment in case the patient did not respond well to the initial treatment course.  For each question, they selected their answer from a multiple-choice list (see example in Figure 1).  Each answer set had one correct response, two nearly correct responses, and two wholly incorrect responses, yielding additional variation. “Wholly incorrect” responses would be inconsistent with the symptoms and data provided in the case while “nearly correct” responses would be consistent with most (but not all) of the symptoms provided to the health professional.  Correct responses were those that followed the national guidelines.

We are happy to share those video vignettes. Feel free to use them and we would love to get your feedback about them.