Prioritizing performance measurement for emergency department care: consensus on evidence-based quality of care indicators

Authors: Michael J. Schull, Astrid Guttmann, Chad A. Leaver, Marian Vermeulen, Caroline M. Hatcher, Brian H. Rowe, Merrick Zwarenstein, Geoffrey M. Anderson

Published: September 01, 2011 | Views: 0

a a



The evaluation of emergency department (ED) quality of care is hampered by the absence of consensus on appropriate measures. We sought to develop a consensus on a prioritized and parsimonious set of evidence-based quality of care indicators for EDs.


The process was led by a nationally representative steering committee and expert panel (representatives from hospital administration, emergency medicine, health information, government, and provincial quality councils). A comprehensive review of the scientific literature was conducted to identify candidate indicators. The expert panel reviewed candidate indicators in a modified Delphi panel process using electronic surveys; final decisions on inclusion of indicators were made by the steering committee in a guided nominal group process with facilitated discussion. Indicators in the final set were ranked based on their priority for measurement. A gap analysis identified areas where future indicator development is needed. A feasibility study of measuring the final set of indicators using current Canadian administrative databases was conducted.


A total of 170 candidate indicators were generated from the literature; these were assessed based on scientific soundness and their relevance or importance. Using predefined scoring criteria in two rounds of surveys, indicators were coded as “retained” (53), “discarded” (78), or “borderline” (39). A final set of 48 retained indicators was selected and grouped in nine categories (patient satisfaction, ED operations, patient safety, pain management, pediatrics, cardiac conditions, respiratory conditions, stroke, and sepsis or infection). Gap analysis suggested the need for new indicators in patient satisfaction, a healthy workplace, mental health and addiction, elder care, and community-hospital integration. Feasibility analysis found that 13 of 48 indicators (27%) can be measured using existing national administrative databases.


A broadly representative modified Delphi panel process resulted in a consensus on a set of 48 evidence-based quality of care indicators for EDs. Future work is required to generate technical definitions to enable the uptake of these indicators to support benchmarking, quality improvement, and accountability efforts.

To Access This Content Please Select One of the Following Options:


Login to an existing account.

Subscribe to this title

Get unlimited online access to this article and all other articles in this title 24/7 for one year.

Claim access now

For current subscribers with Society Membership or Account Number.

  1. Pitts SR, Niska RW, Xu J, et al. National Hospital Ambulatory Medical Care Survey: 2006 emergency department summary, National Health Statistics Reports; no 7. Hyattsville, MD: National Center for Health Statistics; 2008.
  2. Carriere G. Use of hospital emergency room. Statistics Canada Health Reports 2004;16(1).
  3. Medical Care Survey: 1996 emergency department summary. Adv Data 1997;(293):1-20.
  4. Committee on the Future of Emergency Care in the United States Health System. Hospital-based emergency care: at the breaking point. Washington (DC): The National Academies Press; 2006.
  5. Bond K, Ospina MB, Blitz S, et al. Frequency, determinants and impact of overcrowding in emergency departments in Canada: a national survery. Healthc Q 2007;10:32-40.
  6. Fatovich DM, Nagree Y, Sprivulis P. Access block causes emergency department overcrowding and ambulance diversion in Perth, Western Australia. Emerg Med J 2005;22:351-4, doi:10.1136/emj.2004.018002.
  7. Locker T, Mason S, Wardrope J, et al. Targets and moving goal posts: changes in waiting times in a UK emergency department. Emerg Med J 2005;22:710-4, doi:10.1136/emj.2004.019042.
  8. Schoen C, Doty MM. Inequities in access to medical care in five countries: findings from the 2001 Commonwealth Fund International Health Policy Survey. Health Policy 2004;67:309-22, doi:10.1016/j.healthpol.2003.09.006.
  9. Schoen C, Osborn R, Huynh PT, et al. Primary care and health system performance: adults’ experiences in
    five countries health affairs, no. 2004. doi:10.1377/hlthaff.w4.487.
  10. Vancouver Coastal Health. Innovation reduces ER congestion in lower mainland (March, 2009). Available at: (accessed January4, 2010).
  11. Alberta Health Services. Edmonton EMS, Capital Health hopeful changes will improve ambulance access (April 12, 2007). Available at: (accessed January 4, 2010).
  12. Ontario Ministry of Health and Long-term Care. Ontario wait times. Available at: (accessed January 4, 2010).
  13. Alberti G. Transforming emergency care in England. London: National Health Service; 2005. p. 1-44.
  14. Brimelow A, British Broadcasting Corporation News. A&E target ‘risks patient safety.’ Available at: (accessed March 24, 2010).
  15. Institute of Medicine. Pathways to quality health care: performance measurement, accelerating improvement. Washington (DC): The National Academies Press; 2006.
  16. McGlynn EA. Introduction and overview of the conceptual framework for a national quality measurement and reporting system. Med Care 2003;41:I1-7, doi:10.1097/00005650-200301000-00001.
  17. McGlynn EA. An evidence-based national quality measurement and reporting system. Med Care 2003;41:I8-15,
  18. Galvin RS, McGlynn EA. Using performance measurement to drive improvement: a road map for change. Med Care 2003;41:I48-60, doi:10.1097/00005650-200301001-00006.
  19. Marshall MN, Shekelle PG, Leatherman S, et al. The public release of performance data: what do we expect to gain? A review of the evidence. JAMA 2000;283:1866-74, doi:10.1001/jama.283.14.1866.
  20. Cain EF. Improving emergency medical services for children through outcomes research: an interdisciplinary approach. Proceedings of a conference. Ambul Pediatr 2002;2:285-348, doi:10.1367/1539-4409(2002)002,0285:IEMSFC.2.0. CO;2.
  21. Lindsay P, Schull M, Bronskill S, et al. The development of indicators to measure the quality of clinical care in
    emergency departments following a modified–Delphi approach. Acad Emerg Med 2002;9:1131-9, doi:10.1111/
  22. Guttmann A, Razzaq A, Lindsay P, et al. Development of measures of the quality of emergency department care for children using a structured panel process. Pediatrics 2006;118:114-23, doi 10.1542/peds.2005-3029.
  23. Rowe BH, Bond K, Ospina MB, et al. Data collection on patients in emergency departments in Canada. CJEM 2006;8:417-24.
  24. Welker JA, Huston M, McCue JD. Antibiotic timing and errors in diagnosing pneumonia. Arch Intern Med 2008;168:351-6, doi:10.1001/archinternmed.2007.84.
  25. Health Quality Council of Alberta. Alberta Quality Matrix for Health. Available at: (accessed Feb 1, 2008).
  26. Hospital Report Research Collaborative. Hospital performance results 2008: emergency department care. Available at: (accessed Nov 20, 2009).