Editor’s note: This compensation movement, gathering speed notably in the UK, will have strong, positive implications for paramedicine’s early intervention practices, related surveillance and studies, and of course billing. The article mentions “…significant opportunities to improve efficiency, lessen waste, and provide a reasonable standard of care both to the patient’s and the physician’s satisfaction.”
This article reproduced courtesy of acep.org
The continuing debate on quality of care uses a language foreign to many physicians. A recent series of articles in the New England Journal of Medicine 1-6 calls attention to this debate and the need for physicians to understand and be part of it. The need for detailed information on quality of care and outcomes measurements is one result of the recent restructuring of the US health care system and the evolution of managed care health delivery systems. These changes, in turn, reflect the increased expectations of the various stakeholders (patients, payers, health care organizations) coupled with pressures to halt the increasing costs. In this nascent environment, “outcomes management” has flourished.
In his classic article, 7 Ellwood defined outcomes management as “a technology of patient experience designed to help patients, payers, and providers make rational medical care-related choices based on better insight into the effect of these choices on the patient’s life.” Further, he states that this technology “consists of a common patient-understood language of health outcomes; a national data base containing information and analysis on clinical, financial, and health outcomes that estimates as best we can the relation between medical interventions and health outcomes, as well as the relation between health outcomes and money; and an opportunity for each decision-maker to have access to the analyses that are relevant to the choices they must make.”
The outcomes movement draws from four techniques:
- greater reliance on standards and guidelines;
- routine and systematic interval measures of patient function and well-being, with disease-specific clinical outcomes;
- pooled clinical and outcome data, and
- appropriate results from the data base analyzed and disseminated to meet the concerns of each decision maker.
What is the relevance of outcomes management to emergency medicine? As our specialty becomes more proficient in studying and measuring quality of care, we increasingly realize the need to relate our patients’ outcomes to the care they have received in our EDs. As we shall see, there is much room in the specialty of emergency medicine for growth in this exciting area of research.
Outcomes assessors have previously referred to the five Ds — Death, Disability, Disease, Discomfort, and Dissatisfaction. 8, 9 Lonborg defined clinical, functional and status measurements, satisfaction, and cost outcomes. 10 Nelson has extended this definition to the use of the clinical value compass, which cites the four points of 1) medical outcomes, 2) patient satisfaction, 3) functional status, and 4) cost. 11 This paper examines each of these four areas, and some of the relevant research in each field.
Clinical Performance Measurement
Clinical performance measurements, which include process and outcome measures, are in demand in health care today. Payers want information on clinical performance to make contract decisions as well as to track return on payments to health care providers. 4 Health care professionals need information about performance to develop high-quality, cost-efficient systems to deliver care.12 Researchers and regulatory agencies need information from clinical performance measures to develop and implement policy. 13 The increased demand for clinical performance measurement reinforces the need for health care providers in emergency medicine to be aware of measurement systems in place. It is important for emergency physicians to have an understanding of what clinical entities are being measured, what limitations statistical factors impose, and how clinical pertinence affects the study. Finally, and most critically, leaders in emergency medicine must be involved in the development of clinical performance measures to ensure that valid, reliable measures are developed.
Concepts in performance measurement
The delivery of health care has been quantified with the model, “structure + process = outcome.”14 Present measurement systems may focus on any one of the components. “Structure” in health care delivery can refer to the physical plant or the organizational structure such as in credentialing. The Joint Commission for Accreditation of Healthcare Organizations (JCAHO), despite the recent emphasis on performance improvement, is an example of an association examining structure. 15 “Processes” are specific patient interventions performed by health care professionals and resulting in an outcome. Some examples of processes are childhood immunization, use and timing of thrombolytic agents in acute myocardial infarction (AMI), mammography, and patient length of stay in the emergency department. “Outcomes” are the result of the patient’s interaction with health care professionals. Examples of clinical outcome measures include mortality or length of stay associated with medical diagnoses or surgical procedures, readmission rates, morbidity measures such as stroke after carotid endarterectomy, and unplanned return to the ED. The use of this model for defining health care delivery allows a common lexicon when discussing performance measurement. However, even with these definitions, confusion can arise between what is a process measure and what is an outcome measure. A cardiologist may view a cardiac catheterization as a diagnostic process while a health care insurer may view the same event as a cost outcome.
Process measures are frequently used in performance measurement. Process measures are generally much easier to construct, require less data collection and analysis to produce, and are easier for both clinicians and non-clinicians to understand. Many performance measurement systems, such as the Health Plan Employer Data Information Set (HEDIS), are primarily measures of process of care. 5 Process improvement, when linked to processes proven by randomized clinical trials to improve outcomes, is an important part of continuous quality improvement (CQI). Implementation of CQI programs based on process improvement can reduce variation and enhance patient care. 16 An example of process improvement is the present effort by many EDs across the country to maximize the percentage of AMI patients receiving thrombolytic agents and to reduce the time to administration of these agents.
Although the development of process measures is generally easier than the development of outcome measures, certain steps must be followed to ensure the clinical pertinence and precision of each process measure. Important steps in the development of process measures include identification of the process of interest, review of the evidence supporting the process, development of a process indicator (including eligibility in the numerator and denominator), development of a standardized data collection system, and generation of the process indicator. 16
When developing process measures for a specific setting, such as an emergency department, several factors need to be considered. Such factors include what populations are cared for and are of interest, what process measures external agencies are examining, what process measures are likely to reveal opportunity for improvement, and the strength of the evidence linking the process and the desired outcome.
When using process indicators to measure health care delivery, the strength of the association between the process indicator and the outcome of interest must be examined.17 A grading system has been developed that evaluates the strength of practice guidelines used in process measure development. 18 In this system, an “A” is given to those guidelines, or process measures, that have support from large, well-controlled, randomized clinical trials. Examples of this type of guideline include reperfusion in eligible AMI patients and ACE inhibitors in heart failure patients with systolic dysfunction. 19, 20 A “B” rating is given to guidelines, or process measures, that have support from observational studies or small randomized clinical trials (e.g. the use of angioplasty versus thrombolytic agents in AMI patients). 21 Finally, a “C” rating denotes those guidelines or process measures that are developed from expert opinion but which have little scientific evidence to support the process indicators (e.g. the Agency for Healthcare Policy and Research’s low back pain guidelines, most of which is supported by expert opinion). 22 Choosing processes strongly linked to a favorable outcome removes the need to measure the outcome.
Development of the process indicator requires the definition of eligible populations, development of an abstraction tool and methods of data collection, and standardization of data collection using a data dictionary. These steps are time-consuming and costly, but are essential to ensure the reliability and validity of the process indicator. When these steps have been executed appropriately the generation of the process indicator requires minimal analysis.
Clinical outcome measures examine discrete, patient-focused endpoints such as readmission, length of stay, morbidity and mortality. When using outcomes for measuring the performance of health care delivery systems it is often necessary to develop an adjustment system that isolates the contribution of the health care system to the outcome.23 The importance of this principle is exemplified in the association of cancer mortality with age. Cancer mortality increases dramatically with age. If we were to compare two health care systems regarding their cancer death rates and not account for the fact that the mean age of one health care system’s population was 20 years greater than the other, we would come to erroneous conclusions. What is required in this situation is risk adjustment — a method of adjusting the outcome, in this case cancer mortality, for the underlying cancer risk in the population due to age. Risk-adjustment can remove the effect of the confounder, age, from the outcome of interest, cancer. The necessity of risk-adjustment to level the playing field becomes obvious when payers and consumers are using the outcome measures to make purchasing decisions. The New York Department of Health’s outcome measurement model for coronary bypass surgery is an example of a risk adjustment measure.24 This model removed the contribution of the patient’s severity of disease, demographics, and comorbid conditions, to the outcome, mortality. After removing these patient factors affecting the procedure’s survival rate, the remaining data showed the contribution of the health care providers to survival.
Concerns regarding factors such as data sources, adjustment models, and attributable risk have been raised with these measurements. 23 These concerns are well-founded and must be addressed. Outcome measurement systems that are not developed with scientific rigor can lead to erroneous conclusions for consumers, payers, and health care providers.
Data quality concerns
Although some health care systems have well-developed and integrated information systems (IS), it is the exception rather than the rule. A preeminent medical informatics department is at Intermountain Health Care in Utah. The IS at Intermountain allows access to clinically rich information that facilitates the development of observational outcome studies.25 Use of the information from these studies allows Intermountain to continuously improve patient care.
Availability of such data sources is limited and their development is costly. This has led many researchers and developers of outcome measures to use administrative data. Administrative datasets are derived from medical billing data. The most widely used is the Health Care Financing Administration’s (HCFA) Medicare Provider and Analysis File (MEDPAR). The MEDPAR file contains demographic, clinical, and financial information for over 11 million Medicare discharges annually. This file is available, with certain restrictions, as a public use file and has been used extensively by groups such as HCFA, Iameter, HCIA, and others to generate outcome reports. 26-29
Use, for quality studies, of datasets originally developed for billing purposes can raise concerns about the validity of any outcome measure derived from the data. Indeed, one study found only a 78.2% agreement between the principal diagnosis in the billing dataset (MEDPAR) and the principal diagnosis in the medical record.30 Reliability has also been questioned. One study examining the relationship of coded comorbidities with in-hospital death using Medicare administrative data found conditions that, on a clinical basis, would have been expected to increase the risk of death were in fact associated with a decreased risk. 31 The authors suggested that the cause was bias against coding of comorbid conditions in patients that died.
Another source of data is the medical record. Data is abstracted directly from medical records, either by hand or electronically. Data derived in this manner, although still being collected as a secondary dataset, generally has higher levels of clinical validity and reliability than does data derived from administrative datasets. 32 The greatest drawback to medical record-derived data is the cost, as the data generally needs to be manually abstracted by trained abstractors, who may spend up to 30 minutes on a chart.
Risk-adjustment model concerns
After aggregating the necessary data for an outcome measure, the task of risk adjustment or stratification begins. The first question to ask is whether risk adjustment is necessary. The answer to this question is complex and requires expert opinion from both clinical and epidemiological perspectives. The next step is to identify the variables that will measure the “dimensions of risk” for the outcome of interest, followed by the development of a risk-adjustment model.
With an adequate dataset, concerns regarding the validity of the model must be addressed. Iezzoni has developed an excellent summary of the various concerns regarding validity by separating it into its components: face validity — does the model make clinical sense; content validity — does the model incorporate all the variables known to affect outcome; predictive validity — how well does the model predict the outcome of interest; and attributional validity — how well does the model remove patient determinants of the outcome so that we are left with health care system determinants.23
Beyond validity are concerns about reliability or calibration, i.e., how well the model measures outcome when applied to different data than was used for modeling. Other concerns involve the model’s ability to discriminate the outcome, or how much of the variability in the outcome is explained by the model.
Existing Measurement Systems
What are some existing measurement systems that impact on the emergency department? The ED serves as part of a larger health care system, and contributes to the processes and outcomes resulting from this system. Many EDs are redefining their services as the health care market evolves. Development of ambulatory care-centered fast-tracks and subacute/observational units for chest pain and asthma care are examples of this redefinition. For these reasons, the following list will include process and outcome measures that initially do not appear directly related to ED care. The ED can have bearing on all of the following measures and emergency physicians should be aware of them.
Several large systems are covered as well as some unique ED initiatives. Materials have been obtained from several sources, including the National Committee for Quality Assurance (NCQA), Harvard’s CONQUEST system, Faulkner & Grey’s 1997 Medical Outcomes & Guideline Sourcebook, and HCFA-related materials.
The Health Plan Employer Data and Information Set (HEDIS) is a performance measurement system developed by NCQA. NCQA is an organization presently accrediting managed care organizations. One component of managed care accreditation relies on performance measurements, and many managed care organizations seeking accreditation use HEDIS indicators to satisfy the accreditation requirements. HEDIS, however, is still evolving. The following information from the July 1996 draft version of HEDIS 3.0 includes more indicators pertinent to Medicaid and Medicare managed care than previous versions. HEDIS 3.0 also contains more functional outcome measures, although this discussion will focus on clinical outcome measures.
HEDIS’s process measures are divided into two categories, reporting measures and testing measures. Reporting measures
- Childhood, adolescent and adult immunization status
- Breast and cervical cancer screening
- Treating children’s ear infections
- Beta blocker treatment after a heart attack
- Use of appropriate medications for people with asthma
- Eye exams for people with diabetes
- Advising smokers to quit
- Aspirin treatment after a heart attack
- Controlling high blood pressure
- Prevention of stroke in people with atrial fibrillation
HEDIS outcome measures include the following (these example are unadjusted, although stratified by payer).
- Inpatient utilization for medical and surgical diagnosis related group
- Emergency department visits/1000 members per year
- Observation stays/1000 members per year
The Computerized Needs-Oriented Quality Measurement System (CONQUEST) is a compendium of process and outcome measures developed at the Harvard School of Public Health by Dr. R. Heather Palmer, Dr. Ann Lawthers, and colleagues under contract from the Agency for Health Care Policy and Research (AHCPR). The database of measures has been culled from many different sources. Listed below are the process and outcome measures which could potentially impact EDs. Each measure is followed by a source code indicating where the measure originated.
Acute myocardial infarction
- Alcohol and smoking history – RAND
- Aspirin during hospitalization – HCFA
- Jugular venous exam documented if AMI – VA
- Reperfusion for patients with AMI – HCFA
- Timing of emergent PTCA for patients with AMI
- Unplanned readmission for angina within 14 days of discharge – VA
- Heart examination documented for patients on digoxin – ACMAD – Adult
- Lung examination documented for patients on digoxin – ACMAD – Adult
- Pulse rate/rhythms documented for patients on digoxin – ACMAD – Adult
- Instructions about signs to contact physician for patients with asthma and lung disease – PROSPER
- Appropriate prescription of corticosteroid – DEMPAQ
- Appropriate prescription of inhaled beta-adrenergic agonist – DEMPAQ
- Appropriate prescription of oral xanthine derivatives – DEMPAQ
- Blood pressure documented in record by nurse if CVA – RAND
- Electrocardiogram obtained if CVA – RAND
- Gag reflex documented if CVA – RAND
- Serum potassium obtained if CVA -RAND
- Smoking cessation instructions documented – DEMPAQ
- Management of coma in trauma patients – intubation – JCAHO IMSystem
- ACE Inhibitors for appropriate heart failure patients – DEMPAQ
- Alpha blocker for appropriate heart failure patients – DEMPAQ
- Nitrate prescription for appropriate heart failure patients – DEMPAQ
- Blood pressure documented in record if CHF – RAND
- Electrocardiogram obtained if CHF – RAND
- Intensive care unit care or telemetry on day one of hospitalization if CHF and patient is moderately sick – RAND
- Lung exam by physician if CHF – RAND
- Oxygen therapy/intubation used if required for CHF – RAND
- Serum potassium for patients with CHF – RAND, DEMPAQ
- Abdominal examination with gastroenteritis – ACMAD – Pediatric
- Diarrhea with gastroenteritis – ACMAD – Pediatrics
- Hydration status with gastroenteritis – ACMAD – Pediatric
- Temperature with gastroenteritis – ACMAD – Pediatric
- Urination with gastroenteritis – ACMAD – Pediatric
- Vomiting with gastroenteritis – ACMAD – Pediatric
- Weight with gastroenteritis – ACMAD – Pediatric
- Antibiotic therapy began within two hours of admission if pneumonia and immunocompromised – RAND
Influenza prevention (immunization)
- Influenza immunization – Qcare
Ischemic heart disease
- Instructions about signs to contact physician for patients with ischemic heart disease – PROSPER
- Anti-smoking advice for patients with ischemic heart disease – PROSPER
- Aspirin therapy for patients with ischemic heart disease – PROSPER
- Electrocardiogram for patients with ischemic heart disease – PROSPER
- Exercise stress test for patients with ischemic heart disease – PROSPER
- Risk factor education for patients with ischemic heart disease – PROSPER
- Serum potassium for patients on digoxin – ACMAD – Adult
- Heart, lung and pulse exam for patients on digoxin – ACMAD – Adult
Acute myocardial infarction
- Mortality rates – HCFA
- Hemorrhage after thrombolytics for patients with AMI – VA External Peer Review Program
- Intrahospital mortality for AMI – JCAHO IMSystem
- Perforated appendix rates – HCUP-3
- Ruptured appendix rates – Avoidable hospitalization
- Asthma admission rates, ages 2-19 – HEDIS
- Asthma admission rates, ages 20-30 – HEDIS
- Pediatric asthma admission – United Health Care Report
- Admission for acute bronchitis for COPD patients. – DEMPAQ
- Admission for CHF for COPD patients. – DEMPAQ
- Admission for influenza for COPD patients. – DEMPAQ
- Unplanned readmission within 14 days – VA Readmission
- In-hospital hip fracture or fall – Complications screening program – BIH
- Surgical site infection – JCAHO IMSystem
Congestive Heart Failure
- Admission for arrhythmia, CHF, IHD, PE, volume depletion for patients with heart failure – DEMPAQ
Ischemic Heart Disease
- Admission for AMI for patients with ischemic heart disease – DEMPAQ
- Admission for heart failure for patients with ischemic heart disease – DEMPAQ
- Admission for ventricular arrhythmia for patients with ischemic heart disease – DEMPAQ
Transient Ischemic Attack
- Admission for transient cerebral ischemic attack for patients with hypertension – DEMPAQ
- Admission for ventricular arrhythmia for patients with ischemic heart disease – DEMPAQ
JCAHO has been developing a system for incorporation of performance measurement in the accreditation process. This effort has culminated in the March 1997 release of the Oryx system.22 Previous JCAHO efforts, specifically the Indicator Measurement System, have focused on development of process or outcome indicators. The Oryx initiative marks a departure from this approach. The JCAHO will be approving performance measurement systems developed by other entities and then require health care organizations to use the approved systems for performance measurement and continuous quality improvement.
Presently, 60 vendors of performance measurement systems are involved in the Oryx system. Measurement systems included in the Oryx system cover various sources of care including all those that the JCAHO presently accredits. The dimensions of care covered by the 60 vendors include clinical, health status, satisfaction, and administrative/financial. The performance measurement systems predominantly use administrative, or claims, data as a source for the process or outcome measures.
Performance measurement systems chosen for the Oryx system must have six attributes:
- The system has quantitative tools that provide an indication of a health care organization’s performance in relation to a specified process or outcome. The performance measure addresses at least one dimension of care.
- The system has an operational, automated, ongoing database that allows calculation of performance measures.
- The system has, and employs, the ability to assess the accuracy and the completeness of the performance measure data elements.
- The system can remove the effect of patient confounders from the measure. This can be accomplished by risk adjustment or stratification.
- The system can deliver, on a timely basis, performance information to the participating health care organization.
- The system can provide statistically valid comparisons that are useful in the accreditation process.
The first five attributes of the performance system must be currently operational; the sixth attribute is required future compliance.
The JCAHO has developed an aggressive timetable for implementation of the Oryx system. By December 31,1997, each healthcare organization must inform the JCAHO of their choice of at least one performance measurement system from those represented in the Oryx system. From the selected performance measurement system the health care organization must further select two measures that address at least 20 percent of the organization’s patient/resident population. By March 31,1999, the healthcare organization must submit third quarter 1998 data from the chosen measurement system to the JCAHO as part of the accreditation process.
Although many performance systems included in the Oryx system can measure aspects of care delivered in the ED, one system measures many processes of care in the ED. Genentech’s National Registry of Myocardial Infarction (NRMI) collects detailed information about clinical aspects of care delivered to patients presenting to hospitals with acute myocardial infarction. As of May 5,1996 it had 1,475 organizations actively participating with all 50 states represented.
Other measurement systems
Other organizations, including managed care health plans, have their own report cards, but the variety of reporting systems demonstrates the urgent need for a national standard.1 Ellwood has recently developed the Foundation for Accountability (FAcct) to create disease-specific quality-reporting mechanisms.1The Health Care Financing Administration has a large dataset with process and outcome information collected as part of the Cooperative Cardiovascular Project.
While the other points of the quality compass address the needs of the hospital, physician, and payor, the patient’s need to be heard is best met by an assessment of satisfaction. Often, the data gathered from this expression of satisfaction or dissatisfaction is useful for the institution to gauge itself and its success not only in providing a minimum standard of care, but also in meeting the customer’s requirements.33 Studies have shown that each patient who is dissatisfied will inform up to ten other people of this dissatisfaction.34 Thus, it behooves the institution that wishes to maintain or expand its population base to focus on keeping as many people satisfied with their care as possible. Certainly, others have demonstrated the huge cost implications.35 In 1991 dollars, a model organization could expect about $164,000 in direct losses from dissatisfied patients and over $280,000 in losses from negative word-of-mouth advertising.
The literature on design and methodology of satisfaction surveys is extensive 35and will not be repeated here. However, the impact of these surveys is seen in the volume of papers now submitted on these studies. One journal reports that 10% of scientific articles and almost 25% of original research articles that it published were reports on surveys. 36 Of course, not all of these reports are satisfaction surveys, but we can expect a continuing increase in this type of publication.
How can data from the satisfaction survey be used in the ED? One satisfaction survey 37 showed that up to 36% of patients who left without seeing a physician were dissatisfied to the point of not wishing to return for further care at that institution. The majority of these patients, who are non-urgent, stated that they wished to be seen by a physician within one hour of arrival. This wish was used to frame an expected outcome of one hour length-of-stay for fast-track (non-urgent) patients not requiring investigations. This was achieved with a CQI process, reducing median length-of-stay from 84 minutes to 46 minutes. 38, 39 A subsequent study has shown a resultant reduction in the rate of patients who leave without seeing a physician 40 — a good global measure of patient satisfaction. 41-43
Satisfaction measures can also be used to assess new or innovative projects. One recent study used a satisfaction survey to assess the use of an ED chest pain observation unit. 44 The authors found that patients were more satisfied with the observation unit than with inpatient stays for acute chest pain. As they conclude, these “findings add important information to the standard practice of weighing clinical and cost outcomes between two medical care alternatives.”
Other countries have also looked at the use of satisfaction surveys to assess medical care. In Ireland, a recent examination of the accident and emergency department assessed patient satisfaction in a comparison of care given by general practitioners and that provided by emergency staff. 45 This study found no difference in satisfaction, though there was a difference in cost.
Do different practice settings affect patient satisfaction? A recent study evaluated solo or single specialty, multispecialty group, health maintenance organizations, fee-for-service, and prepaid physician payment arrangements. 46The authors found that HMO patients rated their visits the worst of the five groups, and that on average, these ratings can predict “what proportion of patients, on average, will leave their physicians in the next several months.” Thus, satisfaction surveys have important long-lasting implications for both the health care provider and for the payor.
Do we need to act on patients’ dissatisfaction? We have already suggested that it is costly in the long run to ignore patients’ recollections of their emergency visits. One other fact to consider is that even if we do not choose to measure patients’ dissatisfaction, others will do it for us. A recent paper 47 examined satisfaction with emergency coverage across a number of cities and a variety of care plans. It was disconcerting to note that the ED scored lower in patient satisfaction than other aspects of hospital care. Our challenge, then, is not only to excel compared to other institutions, but also to improve relative to departments and services within our own institutions.
Functional health status and quality of life instruments
The US is undergoing a paradigm shift in health-related thinking, from an emphasis on disease to an emphasis on the patient; highlighting health, functioning, well-being, and then disease. 48 Health care’s two goals, to make people live longer and to enhance the quality of life (QALY), can be visualized by a Ziggy cartoon where the meaning of life is seen as “doin’ stuff.” 49 “Doin’ stuff” can be measured with indices that combine life expectancy and health-related quality of life. 49, 50
Many people believe that patient-oriented outcomes measures need to become the focal point for health care. These new indices would be used to redirect public health indicators, change the way physicians make clinical decisions, and suggest new approaches to the allocation of public health resources. 49
The measurement of “doin’ stuff” is a functional or health status assessment.51-53 Functional status questions can assist in predicting mortality, using scales from the Functional Status Questionnaire. For example, characteristics independently predictive of death include greater dysfunction on a scale of intermediate activities of daily living, male gender, living alone, white race, quality of social interactions, and age. Whether improving functional status can reduce the risk of mortality remains to be determined.54
Although some believe health status measures are not appropriate for clinical management, 51,52 interest in incorporating them into clinical practice has increased in recent years. 51,55 First used in rehabilitation medicine, 52 functional status is already considered an important measure of health in primary care. Stimulated by the need to reduce health care expenditures, employers, insurers, and government purchasers are paying increased attention to measurements of patient outcomes and health status.51,56 In the last two decades, a number of tools have been developed to measure health and functional status.51,57
Quality of life has been defined as “the extent to which our hopes and ambitions are matched by experience.” 58 Quality of life assessment in head and neck oncology began over 40 years ago. Quality of life has evolved to become a standard means of assessing clinical outcomes and an accepted end-point measurement in clinical trials, to be considered alongside survivorship and side effects/complications. 59
Despite increasing use in clinical and economic studies, no gold standard exists for the measurement of health-related quality of life (STATUS). One approach to assessing the validity of a STATUS instrument for a particular disease population is to examine the empirical relationship between STATUS patient scores and other accepted measures of health or functional status. 60
Health status and quality of life measures include physical, mental, social, and role functioning and general health perceptions. 61 However, conceptual, practical, and attitudinal barriers have prevented their wider implementation.51, 62Although physicians have recognized the need to expand patient assessment to include global function and quality of life, 63 patient management decisions rarely incorporate standardized health status assessments, since accurate and reliable measures have been difficult and expensive to obtain.64
Measures of health and disease, based on measurement science and reflecting the new paradigm, must be brief and designed for clinical settings, while documenting the natural history of disease, evaluating treatment effectiveness, and improving clinical case management. Although health status measures offer scientific, humanistic, and economic benefits, many intellectual and pragmatic concerns block full-scale use in clinical settings. These concerns include: reaching an agreed-upon definition of health; determining medical components of health status; deciding who chooses/rates which attributes; creating indices from attributes; dealing with multiple measurements of the same entity; interpreting health status scores, and changing clinicians’ day-to-day routines to facilitate use of health status measures in clinical settings.48, 51, 65
Integrating functional health assessment into a busy clinical practice is difficult because the necessary steps require time, thought, recording, and follow-up.63Health status measures must be brief, easily incorporated into the clinical routine, and easy to interpret. Their use should not require complex training or scoring algorithms.62
Health status instruments are useful in clinical settings to screen for functional problems, monitor disease progression or therapeutic response, improve doctor-patient communications, assess quality of care, or provide case-mix adjustment for comparing other outcomes between patient groups. Patients want physicians to ask about their perceptions of health in general and about pain, vitality, and role limitations due to physical function. However, patients vary considerably in their preferences for inquiries into psychosocial issues, social function, mental health, and role limitations due to emotional problems.66
If health status measures are used in direct patient care, it is important to determine whether the goal is to screen for functional problems or to monitor patient changes over time. Instruments used to detect differences between subjects at a single point in time are termed discriminative instruments, while those used to detect longitudinal change within subjects are called evaluative instruments. Requirements for evaluative instruments are reproducibility, validity, ability to detect change over time, and reliability.67
When health status measures are used for quality assurance, average scores for groups of patients should be adjusted for disease severity, comorbid conditions, demographic characteristics, socioeconomic status, and baseline health status.62
The importance of assessing functional status in the hospitalized patient is gaining recognition. However, the availability and accuracy of medical record functional status data are uncertain. Self-report and medical record functional status data differ substantially. Unfortunately, the medical record data is more likely to determine nursing home placement.68
The outcomes movement is exemplified by clinical effectiveness studies incorporating patient-centered outcomes measures. Clinical measures are narrowly focused and assess physiologic, biomedical, and limited functional dimensions of health. Patient-centered outcome measures are broadly defined and capture health status as perceived by the patient. Health status and outcomes assessments should be part of a CQI cycle which begins with measuring patient status and developing treatment plans, monitoring patient progress, evaluating clinical effectiveness, and concluding with outcomes information being fed back to improve the structure and process of health care services.69
Despite the ever-increasing list of available valid functional status/QOL assessments, (For a discussion of the available assessment tools, see Appendix A. these instruments are not widely used in clinical research and are even less available routinely in the clinical setting. To reap the potential rewards that might accrue from health status measurements, community hospitals and their affiliated physicians will have to become actively involved 56 and third-party payers (who may bear much of the cost of this data collection and analysis) must be persuaded of their utility.62 Practitioners familiar in the use of patient-reported life functions in primary care, rehabilitation, and cancer are leading the movement of the patient to the center of the care process. Obstacles in finance, efficiency, and availability are being eroded as these devices are shown to improve care and decrease costs.
Since its inception, emergency medicine has been a patient-centered specialty. Emergency medicine responds to patients where (EMS), when (round-the-clock), and for whatever reason (complaint-oriented) they request/need medical care. The goal of emergency medicine is to return patients to at least the functional status/quality of life they enjoyed prior to the intervention of illness or injury. Unfortunately, published functional health status/QALY research in emergency medicine is non-existent. As leaders in patient-focused health care, emergency physicians should adapt the available tools to measure the effectiveness and efficiency of emergency medical care available to all patients.
Common outcome measures that reflect costs include both direct health care costs (e.g. physicians’ services, hospitals, and drugs) and indirect social costs to the family, employer, and community.11 Some easily recognizable costs are total hospital charges, hospital lengths of stay, and days lost from work.
One outstanding point from early work on outcome measures is the tremendous geographic variation in costs. Much of the early work has been done by John Wennberg. As he has shown, physicians in some markets practice medicine in ways that produce adverse implications for cost of care, often motivated by individual interpretations of the requirements for defensive medicine or because “the necessary scientific information on outcomes is missing.”70 One ground-breaking step was introduction of the concept of high- and low-variation causes of hospital admission. For example, in high-variation conditions such as bronchitis, gastroenteritis, and fractured forearms, the decision to admit is often a reflection of local practice styles rather than a standard of care.70, 71 As Wennberg and others have shown, these high-variation conditions have a profound effect on the cost of care. However, an increased number of high-variation medical admissions does not lead to lower mortality or better care. In a 1989 study, Wennberg, et al., showed that the lower rate of hospital use by Medicare enrollees in New Haven was not associated with a higher overall mortality rate compared to similar patients in Boston.72 Despite this, the per capita expenditure for inpatient care in 1982 was $451 in New Haven and $889 in Boston.73 This resulted in a $300 million greater expenditure for the Boston population if New Haven use rates were applied. Wennberg suggests that if waste could be eliminated and capacity better controlled, resources for entire medical programs could be realized.74 One interesting note is that “higher proportions of primary care physicians in metropolitan statistical areas (are) associated with a less expensive practice of medicine (i.e. lower payments for both in-hospital and out-of-hospital care).”75 Other researchers in addition to Wennberg have demonstrated that quality of care is not necessarily associated with the costs generated by providers.76
In any discussion of costs and outcome studies, one must understand the use of cost-effectiveness analysis. This is a method in which the total costs of a particular health intervention is compared with its benefit or effectiveness.77 – 81When health benefits include quality of life, the method is known as “cost-utility analysis,” and when the benefits are expressed in dollars, it is known as “cost-benefit analysis.” Evaluation of various health care interventions involves assessment of efficacy, effectiveness, efficiency, availability, and distribution. In particular, one must differentiate efficacy (examination of an intervention under optimal circumstances) and effectiveness (examination under usual circumstances).82
These principles have been well-studied in the area of pharmacoeconomics.77 A pharmacoeconomic evaluation begins with the framework of an economic, clinical, and humanistic outcomes (ECHO) model.77, 83, 84 The analysis is then conducted with a given set of assumptions followed by sensitivity analysis where given assumptions are varied and the analysis repeated.78 A recent example comparing ceftriaxone versus cefixime for gonococcal cervicitis demonstrates how this could be useful in the ED. By performing sensitivity analysis, Friedland et al. were able to conclude that there was no cost advantage for either drug.85 In examining the clinical and humanistic outcomes, they suggested that the oral drug was preferable as it was painless to the patient and simpler to administer.
Other areas where cost-effectiveness analyses have been applied include cardiopulmonary resuscitation and the Ottawa Ankle Rules.86 – 88
In examining costs and outcome measures, we must look beyond both geographic variation and the specific impact of various technologies and drugs. Carey et al. have shown how different practitioners can affect the cost of care for acute low back pain. 89 This study showed that outcomes were similar whether care was provided by primary care practitioners, chiropractors, or orthopedic surgeons, with the least expensive care provided by primary care practitioners. This is one of multiple studies either funded through the Agency for Health Care Policy and Research’s (AHCPR) Patient Outcomes Research Team (PORT) projects or else growing out of those initial projects.90
One other issue to consider in a discussion of costs and outcome measures is the cost of ED visits. Until now, this has been controversial. Multiple methods of study have been used with varying conclusions, with the result that President Clinton has called the ED “the most expensive place of all.”91 Part of the problem has been that cost varies depending on what point of view is being addressed: the cost to the patient, to the government or other third-party payer, to the hospital, or to the ED.92 Williams’ methodology has probably been the most acceptable, 93 with the various components of cost specifically addressed — fixed, variable, and marginal costs. Fixed costs are those not dependent on volume, whereas variable costs are dependent on volume. Marginal costs are the extra costs incurred per additional visit. By detailed analysis, Williams has shown that nonurgent visits to the ED are comparable in cost to a family practice visit.
Regardless of which method is used to report the quality of care, the only way to ensure increased participation is if physicians themselves are involved in the process of assessing their own practices. 1, 94, 95, 96 One study has shown that physicians are 14% more cost-efficient when using order entry than with pen and paper. 97
In summary, we have shown how outcomes management represents a new and growing area of research in medicine, with significant opportunities to improve efficiency, lessen waste, and provide a reasonable standard of care both to the patient’s and the physician’s satisfaction. We can certainly expect that with the continued rapid growth of outcomes research, we will be able to expand this discussion in emergency medicine.
There are certainly broad implications from any effort to incorporate outcomes research into emergency medicine. A recent paper 98 suggests the following recommendations for the specialty:
- Develop and promote the use of standardized forms for specific clinical problems of interest.
- Establish a national clearinghouse for the collection and dissemination of the data obtained from standardized clinical forms.
- Create a standing panel of emergency medicine researchers to identify important outcomes, assess the effects of interventions, and to provide feedback of these effects to emergency medicine researchers, practitioners, patients, and payer groups.
- Emphasize the conduct, reporting, and dissemination of high quality outcomes research.
- Establish a directed grant program for individual emergency medicine researchers to address focused aspects of outcomes research unique to emergency medicine.
We can expect that, based on these changes, we will have taken one small step towards our goal to relate patient outcomes to the care received in the ED. The opportunity awaits — carpe diem, quam minimum credula postero — seize the day, put no trust in the morrow!
General comments on forms
Extensive research has been done on the validity, reliability, reproducibility, and utility of health status surveys when applied to general audiences and sub-groups based on age, sex, nationality, and disease entity. When evaluating the validity and relative precision of long- and short-form scales, global items, and poster charts in measuring general health concepts for specific patient populations, and comparisons between healthy, medical, and psychiatric patients the long form is more valid and precise than the short form or multi-item scales. Multi-item scales are preferable to single-item global measures and poster charts. 99
Functional status can be determined by using one of a rapidly growing list of tests which detect health status impairments: the Basic Activities of Daily Living (BADL), Activities of Daily Living (ADL), Instrumental Activities of Daily Living (IADL), Geriatric Depression Scale (GDS), Mini-Mental Status (MMS), the Functional Status Questionnaire, Dartmouth COOP Poster Charts, Duke Health Profile, SF-36 Health Survey, the Health Measurement Questionnaire (HMQ), the General Health Questionnaire (GHQ), the Nottingham Health Profile (NHP), and the Physical Performance Test (PPT).50, 55, 100, 101 Severity of illness can be assessed by using the Acute Physiology Assessment and Chronic Health Evaluation (APACHE-II) system.
These tests have independently shown that functional status, unrelated to age or severity of illness, is correlated with the length of ICU and hospital stay. 100Cognitive and effective status are independently associated with BADL, IADL function, and age (number of drugs is also associated with IADL function), but not with diseases. Symptoms, diseases, drugs, and global health are independently associated with PPT. Chronic diseases may affect functional status in a manner that is insensitive to traditional self-report ADL and IADL measures. Performance-based measures may capture this impairment before more severe functional loss emerges.101
The Sickness Impact Profile (SIP), developed in 1978, is widely used, valid, and reliable possessing good correlation with other health status and functional status measures.102
The Dartmouth COOP-WONCA charts comprise six single-item scales and are used to determine functional ability in chronically ill patients, differentiating between gender and age groups. 57, 103 Although significant variations were noted among countries, it has been validated in seven countries, reflecting doctor/patient selection, cultural, and ethnic differences. It is understood by most patients, is useful to physicians, increases understanding of patient health, and yields good discrimination and profiles for different conditions.104
The RAND-36 Health Survey, using self-reported functional health status, shows an interaction between functional/perceived health status scales and disease classes. It is often used to measure the validity of other health status instruments. 53
The short form 36 (SF-36) physical ability scale is one of the most widely used generic health status instruments. However, it is not the “measure of choice” for older patient groups who have high levels of comorbidity. 105
Both the SF-36 and the COOP charts are suitable for assessment of health perception outcomes in surgical clinical trials. The general picture of change provided by the two measures is similar for physical functioning, mental health/emotional condition, social activities, pain, and overall condition/general health. 106
The Health Measurement Questionnaire (HMQ) is a low-cost technique for self-reporting health status. The HMQ uses Rosser’s disability/distress states and correlates well with the GHQ and the NHP. All three of these measures discriminate between “healthy” and “not healthy” and have identified a significant level of physical and psychological morbidity in respondents. 50
The Medical Outcomes Study Short-Form General Health Survey (MOS-SF) and the Duke Health Profile (DUKE) are brief, reliable, valid, and practical health status measures. Although both instruments are well accepted, many favor the DUKE over the MOS-SF for situations in which patient acceptance or ease of completion is a key issue. 107
The Center for Health Studies’ Group Health Cooperative (GHC) of Puget Sound has created a measure of chronic disease status (CDS) using automated outpatient pharmacy data which can serve, with certain precautions, as a readily accessible low-cost measure of health status. 53
The Health Utility Index-Mark III (HUI) system is an eight-attribute health status classification system (HSCS) and health-related quality of life score which provides reliable information on health status, except in speech and dexterity.108
The Functional Capacity Index (FCI) defines four levels of bending and lifting functions.109
The Hospice Quality of Life Index (HQLI) reveals items with which patients are most satisfied and aspects of quality of life considered to be most important. 110
International and Multi-National Testing
There is growing demand for translations of health status questionnaires for use in multinational studies. The International Quality of Life Assessment (IQOLA) Project is conducting research to determine the feasibility of translating the SF-36 Health Survey into other languages. Data from Sweden and the United Kingdom support the feasibility of cross-cultural health measurement using the SF-36. 111
The Dartmouth functional health assessment charts have been used to assess the functional status of Chinese patients using an easy-to-administer Chinese translation. These were well accepted and understood by patients with valid results and useful information for doctors. 112
The Danish variation of the Nottingham Health Profile, widely used in Europe, consists of six sections: pain, physical mobility, energy level, sleep, social isolation, and emotional reactions. It is reliable, acceptable, and relevant to patients with chronic disability. 113
Short measures of health status, which are easier to administer, 114 are being utilized increasingly115 to encourage the clinical use of functional status/QALY measures. The long SIP and 4 short forms (the SF-36, Functional Status Questionnaire, shortened Arthritis Impact Measurement Scales, and Modified Health Assessment Questionnaire) are highly correlated and as responsive. 115
Although the 50-item questionnaire instrumental activities of daily living (IADL) scales are valid for patient self-completion, minimal questioning, dichotomous responses categories, and aggregate scores, they are not recommended for some activities like discharge planning or monitoring individual patients.114
Condition-specific Health and Function Measures
Measuring functional status changes in various patient subgroups is important in stratifying risk, assessing disease severity, and predicting and defining clinically relevant outcomes. Health services researchers frequently must choose between a generic health status measure and a disease-specific health status measure, such as the Arthritis Impact Measurement Scales (AIMS). In patients with knee or hip osteoarthritis the SIP and AIMS are well correlated for physical, psychological, and total health. For most dimensions, investigators will obtain similar information using either well-validated instrument. 116
However, condition-specific health measures are more sensitive to the problem’s severity because they are less affected by other conditions. Physical function scales of both types are equally precise.117 A general measure of quality of life augments information obtained by disease-specific instruments by interpreting functional status in the broader scope of the patient’s life. 118
When selecting which instrument to use, the type of function to be measured must be taken into consideration.
- Blumenthal D, Epstein A. Part 6: The role of physicians in the future of quality management. N Engl J Med 1996;335:1328-1331.
- Berwick D. Part 5: Payment by capitation and the quality of care. N Engl J Med 1996;335:1227-1231.
- Blumenthal D. Part I: Quality of care–What is it? N Engl J Med1996;335:891-893.
- Blumenthal D. Part 4: The origins of the quality-of-care debate. N Engl J Med1996;335:1146-1149.
- Brook R, McGlynn E, Cleary P. Part 2: Measuring quality of care. N Engl J Med 1996;335:966-969.
- Chassin M. Part 3: Improving the quality of care. N Engl J Med1996;335:1060-1063.
- Ellwood P. Shattuck Lecture–Outcomes Management: A technology of patient experience. N Engl J Med 1988;318:1549-1556.
- White K. Improved medical statistics and health services systems. Publ Health Rep 1967;82:847-854.
- Health Services Research Group. Outcomes and the management of health care. Can Med Assoc J 1992;147:1775-1780.
- Lonborg R. Measuring patient satisfaction over the entire episode of care. Congress on health outcomes and accountability. Washington, D.C., 1995.
- Nelson E, Mohr J, Batalden P, Plume S. Improving health care, Part 1: The clinical value compass. Journal on Quality Improvement 1996;22:243-258.
- Nelson E, Batalden P, Plume S. Report cards or instrument panels: Who needs what? Journal on Quality Improvement 1996;21:155-166.
- Jencks S, Wilensky G. The health care quality improvement initiative. A new approach to quality assurance in Medicare. JAMA 1992;268:900-903.
- Donabedian A. Explorations in quality assessment and monitoring. Vol 1. The definition of quality and approaches to its assessment. Ann Arbor, Mich.: Health Administration Press, 1980.
- The 1996 Medical Outcomes & Guidelines Sourcebook. New York, N.Y.: Faulkner & Gray, Inc., 1995.
- Horn S, Hopkins D. Clinical Practice Improvement: A new technology for developing cost-effective quality health care. New York, N.Y.: Faulkner & Gray, Inc., 1994.
- Hadorn D, Baker D, Kamberg C, Brook R. Phase II of the AHCPR-sponsored heart failure guideline: Translating practice recommendations into review criteria. Jour Qual Improv 1996;22:265-276.
- AHCPR Guideline. Heart failure, evaluation and care of patients with left-ventricular systolic dysfunction, No. 11. June 1994.
- Gruppo Italiano per lo Studio della Streptochinasi nell’Infarcto Miocardio (GISSI). Effectiveness of intravenous thrombolytic treatment in acute myocardial infarction. Lancet 1988;1:397-402.
- Garg R, Yusuf S. Overview of randomized trials of angiotensin-converting enzyme inhibitors on mortality and morbidity in patients with heart failure.JAMA 1995;273:1450-1456.
- Grines C, Browne K, Marco J, et al: A comparison of immediate angioplasty with thrombolytic therapy for acute myocardial infarction. N Engl J Med1993;328:673-679.
- AHCPR Guideline. Acute Low Back Pain Problems in Adults: Assessment and Treatment. No. 95-0653.
- Iezzoni L(ed). Risk adjustment for measuring health care outcomes. Ann Arbor, Mich.: Health Administration Press, 1994.
- Hannan EL, Kilburn Jr H, O’Donnell JF, et al. Adult open heart surgery in New York State: An analysis of risk factors and hospital mortality rates. JAMA1990;264:2768-2774.
- Classen DC, Evans RL, et al. The timing of prophlyactic administration of antibiotics and the risk of surgical-wound infection. N Engl J Med1992;326:281-286.
- Fischer E, Baron J, Malenka D, et al. Hip fracture incidence and mortality in New England. Epidemiol 1991;2:116-122.
- Health Care Financing Administration. Medicare Hospital Mortality Information:1986. Washington, D.C.: U.S. Dept. of Health and Human Services, 1988.
- Roos N, Wennberg J, Malenka D, et al. Mortality and reoperation after open and transurethral resection of the prostate for benign prostatic hyperplasia. N Engl J Med 1989;320:1120-1124.
- Wennberg J, Roos N, Sola L, et al. Use of claims data systems to evaluate health outcomes: Mortality and reoperation following prostatectomy. JAMA1987;257:933-936.
- Fischer E, Whaley F, Krushat M, et al. The accuracy of Medicare’s hospital claims data: Progress has been made but problems remain. Amer J Publ Health 1992;82:243-248.
- Iezzoni L, Foley S, Daley J, et al. Comorbidities, complications, and coding bias.