How are adverse events currently evaluated?
In response to these high rates of error, especially after the Institute of Medicine report “To err is human”, efforts have been focused on evaluating adverse events and learning why they occur. Adverse events are difficult to study due to their multifactorial nature. Adverse events have been studied both prospectively, through clinical surveillance and direct observation in the OR, as well as retrospectively, through Mortality and Morbidity Reports, self reporting of adverse events, and malpractice claims.
1 - Retrospective Review
At present, adverse events are most observed in retrospective, outcome-based studies, which include morbidity and mortality conferences, self-reporting of adverse events, chart reviews, and malpractice claims.
1- i) Morbidity & Mortality Conference
Mandated by the Accreditation Council of Graduate Medical Education, the weekly Morbidity and Mortality (M&M) conference is a quality improvement initiative, in which surgeons voluntarily disclose adverse events and discuss with their peers complicated patient cases and deaths in a confidential, non-punitive setting. While the M&M conference provides surgeons with the opportunity to investigate adverse events, this type of retrospective review is particularly vulnerable to bias. Adverse events are described from the perspective of the surgeon, based on their memory of the case, and without the full context of the environment60. Furthermore, the M&M conference is faced with the problem of underreporting due to its voluntary nature. Hutter et al61showed that traditional M&M conference underreports complication rates (6.4 vs. 28.9%) and mortality rates (0.9 vs. 1.9%) as compared to another more structured reporting system, American College of Surgeons National Surgical Quality Improvement Project (ACS-NSQIP).
1 - ii) Reporting of adverse events
The incident reporting system was a risk management strategy first implemented in military aviation in the 1950s to improve air safety, and has since become a key aspect of flight safety in aviation. At present, the commercial aviation industry has in place two separate reporting systems that operate simultaneously: (1) the National Transportation Safety Board, which mandates that accidents involving serious passenger injury be reported; and (2) the Aviation Safety Reporting System (ASRS), a non-punitive voluntary external reporting system. Receiving 30,000 voluntary reports per year, the ASRS analyzes root causes of reported incidents, and communicates key findings to the entire aviation community to reduce the risk of incident recurrence.
Similar to aviation, medicine also has two forms of reporting that operate simultaneously: (1) the post-operative follow-up in which physicians must disclose patient outcomes, including morbidities and mortalities, in the patient medical record, and (2) critical incident reporting, an external reporting established at some hospitals as a quality improvement initiative, modeled after the aviation reporting system. In Canada, there are provincial and federal policies and legislation that have been set in place that requires physicians to disclose both preventable and non-preventable adverse events in the medical record.
1 - iii) Post-operative follow-up
In Canada, policies are set in place at the federal (e.g., Royal College of Physicians and Surgeons of Canada; CMPA), provincial, and local (hospital) levels to ensure that adverse patient outcomes are reported and recorded in the patient’s medical chart.
Studies involving chart review use data from the patient’s medical chart to look for trends. Chart reviews study these records for adverse events and look for trends. Chart review is typically a resource-intensive, time-consuming two-stage process in which all records are screened by trained nurses and independently reviewed by physicians who identified adverse events. However, studies have shown poor inter-rater reliability among raters to identify adverse events by medical record review. It is difficult to measure the validity of chart review because there is no true gold standard for assessment – they are simply implicit judgments. Furthermore, the issue with chart review is that there is no consistent definition of an “adverse event” which makes AEs particularly difficult to quantify, hence the variability of AE incidence rates reported in chart review studies. Anderson et al looked at surgical adverse events in a systematic review/meta-analysis study and found that time-frames of included adverse events, the threshold for causation, and the categorization of consequences varies widely across chart review studies.
1 - iv) Critical incident reporting
Modelled after the aviation incident reporting system, medical event reporting has been implemented and studied across multiple specialties, including transfusion medicine; emergency medicine; family practice; surgery; anesthesia; and critical care.
These reporting system are typically voluntary and submitted either anonymously or confidentially to an external group for data analysis. Most reporting systems are based on error taxonomies that allow us to trace the causal path from active failures to latent conditions, providing a depth of information regarding causative factors related to human error, environmental and management influence which is not readily available from other sources such as chart reviews and observational studies. Incident reporting captures more contextual information about incidents, including perceived causative factors related to human error, environmental and management influences, and can detect more preventable AEs than medical record review.
Widely used in high-risk industries, near-miss reporting systems have shown success in highlighting weaknesses in the system, and revealing rectification strategies employed by the system to successfully manage a bad situation. Near-misses are rarely documented in medical records – yet provide a valuable lesson in recovery mechanisms without the detrimental consequences of an AE. Reporting of near misses offers several advantages over reporting of adverse events: greater frequency allows for quantitative analysis and there are fewer barriers to reporting by workers, partly owing to fewer liability concerns. Furthermore, in near miss reporting, recovery patterns can be captured, studied, and used for safety improvement.
Criticism of incident reports is its subjectivity since there is often a lack of consistency and validation of incident data classification, and relies on subjective observation of each individual reporting. Also, incident report-based studies are limited by voluntary reporting bias and compliance rates are low. In fact, underreporting of adverse events is estimated to range from 50-96%.
Incident reporting likely underestimates the actual level of errors because of fear of disciplinary and legal consequences, competing demands, and varying definitions of a reportable event. In fact, voluntary reporting alone is the least sensitive method for detecting AEs, which suggests that incident reporting leads to significant bias when assessing quality of care.
Sari and co-workers98compared event reporting with the record review method and concluded that routine reporting systems considerably underreport the scale and severity of unintended events. The strength of event reporting is, however, in detecting latent failures (organizational and technical causes) that might not otherwise be apparent.
In response to a number of high-profile failures in medicine, the UK developed an electronic nation-wide reporting system, the NationalPatient Safety Agency (NPSA), which collects data on incidents from organizations in England and Wales – and has shown to be effective in providing a robust database.
1 - v) Malpractice Claims
Malpractice claims have been used in the root cause analysis of technical error in surgery. Closed claims analysis in 2 studies (Japan and England) showed that cognitive error is the most contributory factor to medical errors – error in judgment, vigilance and teamwork being the top factors that contribute to adverse patient outcomes. Another studied identified communication breakdowns with malpractice claims. While these claims provide some information about error causation, they are limited in terms of error causation since most malpractice claims provide insufficient information.
2 - Prospective Methods
Prospective data collection eliminates the risk of hindsight bias in analysis – that is, data collection and analysis is not influenced by knowledge of patient outcome. Recent studies have focused on prospective observation in the OR, in attempt to gain a better understanding of why adverse events occur. These studies typically involve direct live observation of team members in the OR in which an observer is present in the room during the surgical case – evaluation of team performance using validated rating tools; identification of stressors; study surgical decision making; assessment of technical skills using validated rating tools by either external observers or operating attending surgeon.
2.1 Live Observation
Direct live observation is used to prospectively study performance, behaviors, events, and environmental factors in the OR. In this set-up, a trained observer is present in the OR during a surgical procedure to directly study the performance of the operating team and their work environment. Previous studies have used live observation in the OR to evaluate non-technical skills and team performance using validated rating tools107-110; to identify stressors in the intraoperative environment111; and to assess technical performance by either external observers112-116or the operating attending surgeon117,118. While live observation does provide valuable first-hand data from the OR, it introduces observer bias, and may present inaccurate findings due to the real-time nature of rating. Presence of a live observer may also result in the Hawthorne Effect, which is a psychological phenomenon where individuals modify their behavior under awareness of being observed.
2.2 Video Recordings
Video recordings are another strategy for the analysis of AEs and surgical performance. The advantage of video recordings is that an operation can be recorded prospectively and reviewed retrospectively. This method minimizes observer bias, allows observers to review multiple times thus improving reproducibility, increases inter-rater reliability, allows input from multiple observers without requiring their presence in the OR, and reduces the Hawthorne effect because room cameras are less intrusive in the OR than live observers. Studies have used video recordings in the OR to observe and evaluate surgical performance, assess the effects of interventions in the OR, and provide post-operative video-based feedback. A number of studies in trauma resuscitation have shown the value of videotaping in teaching and quality insurance119-122. Ellis et al. conducted a survey study to investigate videotaping practices in 221 trauma resuscitation centers, and found that videotaping is perceived by staff as an effective tool for performance improvement123. The use of video recording is also valuable in surgery, as it allows laparoscopic video footage to be reviewed multiple times to analyze, for example, a particular technique or surgical step. Furthermore, video recordings facilitate the detection of near-miss events – the potential AEs that were intercepted before harm was done.
2.3 Other Prospective Data Collection Tools
Some studies have used data recording tools, other than room cameras and microphones, in the OR to analyze AEs. Technological innovations have been developed to prospectively evaluate surgical performance in the intraoperative environment. Such innovations include force-torque sensors embedded on instruments; hand-motion trackers to assess surgical dexterity; heart rate monitors and salivary cortisol to measure stress levels; and electroencephalography (EEG) to assess fatigue. While fatigue and stress levels are not directly linked to non-technical skill, sleep deprivation and stress may be underlying factors that could influence surgical performance.