May 19, 2009, Office of Inspector General issued its report on the Federal Aviation Administration’s (FAA) Aviation Safety Action Program (ASAP). They conducted this audit following a complaint from an FAA inspector regarding the acceptance of a fatal accident into ASAP. ASAP is a joint FAA and industry program that allows aviation employees to self–report safety violations to air carriers and FAA without fear of reprisal. Incidents reported through ASAP by aviation employees are reviewed for acceptance by an Event Review Committee (ERC), which may also recommend corrective actions. Their audit objective was to assess FAA’s implementation of ASAP and identify any improvements that will help FAA to maximize the program’s safety benefits.

While ASAP is a potentially valuable safety tool, OIG found that FAA’s ineffective implementation and inadequate guidance have allowed inconsistent use and potential abuse of the program. Further, FAA has not devised a method to fully compile data reported through ASAP for analysis on a national level. Therefore, little is understood about nationwide trends in the types of violations reported under ASAP, and ASAP reports do not help FAA determine whether systemic, nationwide causes of those violations are identified and addressed.

As a result of these issues, ASAP, as currently implemented, is a missed opportunity for FAA to enhance the national margin of safety. To realize the full benefits of ASAP, FAA must improve the program in the following areas: (1) revising ASAP guidance to clarify which incidents should be excluded from the program, (2) emphasizing to employees that ASAP is not an amnesty program, (3) clarifying the ERC’s authority and role in ASAP and ensuring ERC members are unbiased, (4) requiring inspectors to examine repetitive reports of ASAP–related safety concerns and enhancements to ensure effective corrective actions, and (4) developing a central database of all air carriers’ ASAP reports that FAA can use for trend analysis at a national level. (See report below)

The Office of Inspector General conducted this audit following a complaint from a Federal Aviation Administration (FAA) inspector regarding the acceptance of a fatal accident into the Aviation Safety Action Program (ASAP). On January 16, 2006, a contract mechanic in El Paso, Texas, was killed while troubleshooting an engine oil leak as two pilots performed an engine run-up procedure. Within 24 hours, the pilots submitted a request for the accident to be accepted into the air carrier’s ASAP. ASAP is a joint FAA and industry program intended to generate safety information through voluntary disclosure that may not be otherwise obtainable to identify potential precursors to accidents. The program allows aviation employees to self-report safety violations to air carriers and FAA, including violations of Federal Aviation Regulations, without fear of reprisal through legal or disciplinary actions.

The inspector requested that we review the ASAP submission to determine whether FAA should have accepted the fatal accident into the program. To evaluate the complaint, we reviewed FAA’s guidance and procedures for implementing ASAP. Based on the results of that review, we expanded our audit to include ASAP at two other air carriers. Our objective was to assess FAA’s implementation of ASAP and identify any improvements necessary for FAA to obtain maximum safety benefits from the program. This report presents the issues we identified and provides recommendations for program improvements. This performance audit was conducted in accordance with generally accepted government auditing standards prescribed by the Comptroller General of the United States and included such tests as we considered necessary to provide reasonable assurance of detecting abuse or illegal acts. Exhibit A details our review objective, scope and methodology.

Currently, 73 air carriers participate in ASAP. To establish an ASAP program, air carriers, applicable employee unions (e.g., pilots), and FAA sign a Memorandum of Understanding (MOU) outlining the program’s purpose, terms, administrative procedures, and reporting process. Air carriers can establish MOUs with pilots, mechanics, flight attendants, and dispatchers; most of the current MOUs for ASAP are with pilots (see figure). initiated due to a complaint about a pilot ASAP report, the focus of our work was on the pilot ASAP program. Airline employees report safety violations to the air carrier’s ASAP manager, who then forwards the report to the Event Review Committee (ERC). The ERC is comprised of representatives from each party to the MOU. ERC responsibilities include determining whether reports are accepted into ASAP, recommending any needed corrective actions, and working with the air carrier to develop appropriate corrective actions.

FAA’s Air Transportation Voluntary Safety Programs Branch manages voluntary aviation industry programs like ASAP. This office reviews program implementation and collects data and feedback from ASAP participants. The office uses these data to determine whether the program is achieving safety objectives.

The underlying principle of ASAP is to obtain voluntarily reported safety information to proactively prevent accidents and incidents. When properly implemented, this program could provide valuable safety data to FAA. We found, however, that FAA’s ineffective implementation and inadequate guidelines have allowed inconsistent use and potential abuse of the program. For example, the ERC that evaluated the ASAP report for the January 2006 fatal accident included an FAA inspector who had previously served as a pilot for the same air carrier as the two pilots involved in the accident. This inspector subsequently shared confidential hotline complaint information related to FAA’s acceptance of the accident into ASAP with the air carrier, which presented a conflict of interest.

Further, FAA has limited the program’s effectiveness because it has not devised a method to fully compile data reported through ASAP and analyze these data on a national level to identify trends. FAA cites funding issues along with air carriers’ concern for data protection as hindrances to developing and implementing a program or method to identify safety trends on a broader scale. While ASAP has proven highly beneficial to the airlines, FAA currently obtains only limited aviation safety data through the program for use in identifying systemic safety issues. As a result of these issues, ASAP, as currently implemented, is a missed opportunity for FAA to enhance the national margin of safety. To realize the full benefits of ASAP, FAA must make program improvements in the following areas.

FAA needs to modify ASAP guidance to clarify which incidents carriers should exclude from ASAP and clearly define “intentional disregard for safety.” FAA’s current ASAP procedures allow acceptance of accidents (including fatal accidents). However, in our view, this contradicts ASAP’s fundamental purpose—to gather information on safety incidents that might otherwise remain unknown —because FAA already obtains safety information on accidents through internal and National Transportation Safety Board (NTSB) investigations. To ensure only valid incidents are accepted, FAA should clarify ASAP guidance with a disqualifier similar to that of the National Aeronautics and Space Administration’s (NASA) Aviation Safety Reporting System (ASRS),1 a voluntary reporting system that explicitly excludes accidents.

Further, FAA guidance for ASAP does not clearly define the term “intentional disregard for safety.” FAA’s program surveys for ASAP in fiscal years (FY) 2005 through FY 2007 found that ERC members had difficulty interpreting this criteria for ASAP submissions. Without proper clarification, determination of intentional disregard becomes strictly subjective, which can impede FAA’s ability to take appropriate enforcement action. For example, the inspector who contacted our office about the 2006 accident believed that intentional disregard had contributed to the fatality, but the ERC members did not. At the time the accident was accepted into ASAP, FAA did not know the cause of the accident or whether it involved intentional acts. The inspector who made the complaint questioned whether events involving the loss of human life should be closed without an in¬depth investigation by FAA and with only minor administrative action taken against individuals that may have contributed to the accident. To enhance safety efforts, FAA should revise its guidance to refine this definition.

ASRS was established in 1975 under a Memorandum of Agreement between the FAA and NASA to receive, process, and analyze voluntarily submitted incident reports from aviation employees.

FAA needs to standardize how and when carriers should submit ASAP reports to FAA inspectors. We found instances in which FAA inspectors did not receive ASAP reports in a timely manner. For example, the inspector representing FAA on the ERC for the January 2006 accident did not receive the ASAP report until the ERC meeting—7 days after the accident. However, the air carrier and union ERC members obtained the report immediately after the pilots submitted it. The problem was addressed at this air carrier once we informed FAA Headquarters. However, at least two other carriers continue to experience this problem. FAA needs to standardize how and when its ERC representatives obtain ASAP reports so they can review all pertinent FAA rules before an ERC meeting. One possible method would be for FAA to require air carriers to give FAA’s ERC members electronic access to ASAP reports.

FAA needs to correct misperceptions that ASAP is an amnesty program and ensure ERC members are impartial. We found instances where ERCs viewed ASAP as an “amnesty program” (i.e., they believe any corrective or enforcement actions for ASAP-reported incidents would be inappropriate). Partnership programs like ASAP are not primarily focused on discipline; rather, they are intended to facilitate collaboration between FAA and air carriers to identify and resolve safety issues. Confusion about the purpose of ASAP can adversely affect the program. For example, since our review, American Airlines, Comair, and US Airways pilots have discontinued their ASAPs due to conflicting views on actions taken against pilots that have submitted ASAP reports.2 Delta pilots discontinued their ASAP in December 2006 for similar reasons and then reinstated it in January 2009 at FAA’s urging. According to the Air Line Pilots Association, many pilots believe that any corrective actions resulting from an ASAP report, such as additional check rides3 or simulator training, are punitive. Also, pilots and air carriers disagree on air carriers’ authority to take disciplinary action for events reported under ASAP when the air carriers obtain information on the reported event independently of the ASAP report. This indicates a need for improved FAA guidance and additional education to clarify ASAP’s intent. To accomplish this, FAA needs to ensure each carrier’s MOU clearly states that the ERC has the authority to recommend corrective or administrative actions as acceptable solutions to reported violations.

Further, FAA’s ASAP training does not address potential bias by inspectors assigned to an ERC. FAA’s ERC representative at the carrier involved in the 2006 fatal accident was a former pilot for the airline; he later shared confidential hotline complaint information related to FAA’s acceptance of the accident with the carrier.

According to FAA, US Airways has reestablished its MOU with pilots, and American Airlines is in the process of doing so.Check rides allow instructors to examine flying proficiency and other factors that influence potential performance, judgment, and stability. They examine how well a pilot responds to the stress of in-flight emergencies in a flight simulator.

To avoid the appearance of bias, FAA should emphasize the need for impartiality as part of its required ERC training and implement procedures to require periodic refresher training in this area. Also, FAA should clarify FAA field office management responsibilities to ensure personal relationships between inspectors and airline personnel do not influence decision-making.

We are not advocating a return to past practices where FAA relied primarily on penalties and fines when airlines or aviation employees committed safety violations. FAA believes safety partnership programs are valuable in forming collaborative relationships with air carriers. Used properly, these programs can indeed be important tools for FAA and the aviation industry. However, partnership programs must be balanced with a strong commitment to oversight so that they do not lapse into automatic amnesty for violators.

FAA must gather sufficient ASAP information from inspectors to proactively identify safety issues and assess the effectiveness of air carriers’ corrective actions for repeat violations. Currently, FAA inspectors compile quarterly reports of ASAP activity for each participating air carrier and submit them to the Voluntary Safety Programs Branch at FAA Headquarters. However, these reports do not provide sufficient details about ASAP events or corrective actions and are not used by FAA for trend analysis. The quarterly reports for the various air carriers contain different information in varying formats, which would make compilation for analysis difficult. The reports typically contain general information on the number—not the nature—of ASAP submissions for that quarter and any resulting “safety enhancements.”

While this information is valuable to individual air carrier safety programs, FAA has overlooked an opportunity to enhance the national margin of safety by collecting ASAP data that can be used for trending, identification of risk factors and accident precursors, policy development, and dissemination of collective safety data to FAA inspectors and other carriers. An independent review team, convened by the Secretary of Transportation in May 2008 to examine FAA’s safety culture and management, also recommended that FAA compile and analyze ASAP data to identify trends and patterns that represent risks.4 These actions would also help to guarantee the integrity of voluntary reporting programs.
FAA has contracted with Mitre Corporation to develop a tool, called the Aviation Safety Information Analysis and Sharing (ASIAS) system, to perform integrated queries across ASAP and other safety databases. FAA is using ASIAS to assess the magnitude of problems that have already been identified and is working to expand ASIAS functionality. However, ASIAS does not yet have the capability to trend safety data to identify potential risk factors. Additionally, access to ASIAS data is limited to persons or groups authorized by members of an executive board that administers the system.

“Managing Risks in Civil Aviation: A Review of the FAA’s Approach to Safety,” Report of the Independent Review Team, September 2, 2008.

Effective corrective action by air carriers is also a concern. We identified repetitive reporting of the same problems (i.e., altitude deviations and checklist usage) and the same proposed or implemented safety enhancements from quarter to quarter. According to FAA’s ASAP guidance, FAA requires carriers to complete corrective actions that are acceptable to all members of the ERC to resolve any safety deficiencies; otherwise, FAA could terminate the carrier’s ASAP. In our view, repetitive reports of the same problem would indicate that a carrier’s ASAP program is ineffective.
Our recommendations to FAA focus on actions needed to (1) clarify ASAP guidance, (2) emphasize to employees that ASAP is not an amnesty program, (3) ensure effective inspector reporting of ASAP activity, and (4) use ASAP data for trend analysis at a national level. We are making a total of eight recommendations, which are listed on page 16 of this report. FAA’s comments and our response are discussed on pages 17 and 18.

ASAP is a potentially valuable safety tool; however, FAA is not realizing the full benefits of the program. FAA’s ineffective implementation and monitoring and inadequate guidelines have contributed to the inconsistent use and diminished safety value of the program. Because FAA has not devised a method to gather sufficient data, little is understood about nationwide trends in the types of violations reported under ASAP, and ASAP reports do not help FAA determine whether systemic, nationwide causes of those violations are identified and addressed.
ASAP Guidance Lacks Clarity Regarding Which Incidents Should Be Excluded From the Program
We determined that FAA’s guidance does not specifically exempt accidents from ASAP. Therefore, there was nothing to preclude the ERC from accepting the January 2006 accident into the program. The FAA inspector and air carrier representatives on the ERC stated that they determined that the pilots did not intentionally disregard safety. Further, the ERC found no evidence of criminal activity or other conditions that met the criteria for rejecting an ASAP report.

In our view, however, acceptance of accidents undermines ASAP’s fundamental purpose—to gain access to information that might otherwise remain unknown by allowing employees to report safety violations without fear of reprisal through

legal or disciplinary actions. Because all accidents are reported to FAA and investigated thoroughly by both FAA and the NTSB, we do not believe any significant unknown information would be gleaned through acceptance of any accident into ASAP, especially a fatal accident.
The FAA Supervisory Principal Operations Inspector responsible for oversight of the air carrier where the fatal accident was accepted agreed that when accidents are accepted into ASAP, the program has failed because the intent of ASAP is to identify precursors to prevent accidents or fatalities. In other words, to recognize unsafe conditions before, not after, they result in an accident.

To determine whether accidents are valid contributions to ASAP, we attempted to examine air carrier ASAP data to identify the acceptance of other accidents into ASAP. Air carriers’ MOUs with FAA, which outline their ASAPs, stipulate that the air carriers keep all documents and records regarding the program. Due to strict confidentiality provisions, however, we were not able to access air carriers’ ASAP data needed to perform this analysis. FAA also could not provide us with the compiled data on a national level.

We therefore compared ASAP to NASA’s voluntary reporting program, ASRS, in which analysts collect, analyze, and respond to voluntarily submitted aviation safety incident reports to lessen the likelihood of aviation accidents. FAA provides most of the funding for ASRS, and NASA administers the program and sets its policies in consultation with FAA and the aviation community. Safety analysts identify safety hazards and issue alerting messages to appropriate FAA offices or other aviation authorities. De-identified information (i.e., all identifying carrier information is removed) is also incorporated into the ASRS database, which can be accessed by the public.

We determined that this program has many similarities to ASAP (see table below). In contrast with ASAP, however, ASRS does not accept accidents into its database and does allow electronic access to ASRS activity. Any accidents reported by pilots, controllers, and other aviation personnel into ASRS are forwarded by NASA to NTSB and FAA for investigation. FAA should implement a similar disqualifier for ASAP in clarifying guidance regarding which incidents should be accepted into or excluded from the program.

Another area in which ASAP guidance lacks clarity is with its use of the phrase “intentional disregard for safety.” FAA’s guidance states that violations involving an intentional disregard for safety should be excluded from ASAP. FAA guidance, however, does not provide any clarifying information on what this phrase means. FAA’s guidance5 provides only two examples of violations that would not be eligible for submission into the ASAP program because they involved an intentional disregard for safety. One example involves a first officer who falsely reported he had completed the exterior pre-flight inspection when he did not inspect the top wing surface for ice, resulting in an emergency landing. The other involves a mechanic who knowingly used an unauthorized lubricant on an engine valve to avoid delays in completing the job order. However, FAA’s guidance does not further explain this phrase under its “Key Terms,” which are defined “for the purposes of ASAP to ensure a standard interpretation of the guidance.”

In FAA’s annual reviews of the ASAP program in FY 2005 through FY 2007, several respondents indicated that they had experienced problems or interpretation issues with the meaning of “intentional disregard for safety” (see FAA’s pilot ERC survey results at exhibit C). We believe that this poses a significant problem because, without proper clarification, determination and interpretation of “intentional disregard” becomes strictly subjective and can complicate safety matters.

For example, the inspector who contacted our office after the El Paso mechanic’s death strongly believed that intentional disregard had contributed to the fatality, but the ERC members did not. The NTSB’s investigation report—issued on January 31, 2008—found that contributing factors in the accident were “the airline’s insufficient training of contract mechanics and the airport’s failure to disseminate a policy prohibiting ground engine runs above idle power in the terminal area.” The FAA investigation of the accident recommended a civil penalty of $55,000 against the airline involved.6 The two pilots received letters of warning, which stated that as the result of their “careless operation” of the aircraft, “passenger safety may have been jeopardized during the boarding operation….”

FAA Advisory Circular 120-66B, Aviation Safety Action Program (ASAP), November 15, 2002.

The inspector who made the complaint stated the accident was caused by the mechanic’s “preventable human error and poor judgment” and the flight crew’s failure to follow FAA- and air carrier-approved procedures for handling engine oil leaks and conducting engine run-ups. He specifically mentioned the pilots’ “reckless operation of the aircraft at high power settings” while still at the passenger gate and questioned whether events involving the loss of human life should be closed without an in-depth investigation by FAA and with only minor administrative action taken against those who may have contributed to the accident. To enhance safety efforts and ensure appropriate corrective actions, FAA should determine what types of problems ERC members encounter in interpreting intentional disregard and revise its guidance to refine this definition.

FAA Inspectors Do Not Have Timely Access to ASAP Reports: We found that FAA has not clarified when and how FAA inspectors assigned to ERC committees should obtain ASAP reports for review. The process for transmitting ASAP reports to FAA representatives was different at each of the three carriers we reviewed. When we initiated our review, we determined that some inspectors received the reports immediately upon employee submission, while others did not receive the reports until weeks after an event had occurred.

The inspector representing FAA on the committee that handled the January 2006 fatal accident did not receive ASAP reports until the ERC committee meeting on January 23—7 days after the accident. However, the air carrier and union ERC members obtained the ASAP report as soon as it was submitted.

It is important that inspectors receive the reports in a timely manner so they are able to review all pertinent FAA regulations and procedures prior to meeting with the other ERC committee members. The Director of FAA’s Air Transportation Voluntary Safety Programs Branch stated that a practice in which the Agency’s ASAP representative does not receive ASAP reports in a timely manner is inappropriate and certainly is not considered an industry “best practice.” The Director pointed out that FAA can revise the MOU language to address this issue.

The ERC that reviewed an ASAP request from the other mechanics involved in the accident rejected the request.

Our concerns with this practice were confirmed in FAA’s Annual ASAP Review for FY 2006. In its report, FAA disclosed that 2 of the 12 survey participants reported that FAA representatives on ERCs did not receive reports in a timely manner. However, FAA’s annual ASAP review did not include the air carriers that we visited. Therefore, the issue of not transmitting ASAP reports to FAA in a timely manner may be more widespread than FAA is aware.

As a result of our review, the Voluntary Safety Programs Branch took immediate action to ensure that the FAA inspector for the carrier where the accident occurred receives ASAP reports in a more timely manner. FAA should consider revising its guidance to standardize how all FAA inspectors involved with ASAP at their assigned air carriers obtain ASAP reports. One possible solution would be requiring air carriers to grant FAA representatives on the ERC electronic access to the carrier’s ASAP reports database, similar to what is currently available through ASRS.
Unclear FAA Guidance Has Allowed Some To Misperceive ASAP as an Amnesty Program, and ASAP Training Does Not Emphasize the Need for ERC Members To Be Impartial Partnership programs are intended to facilitate collaboration between FAA and air carriers to identify and correct safety issues. We found, however, that FAA’s guidance on ERCs is subject to misinterpretation regarding ASAP’s purpose. As a result, some aviation employees have come to view it as an amnesty program and therefore believe that any corrective actions taken in response to an ASAP-reported incident, such as additional employee safety training, would be inappropriate. FAA’s training for ASAP also does not address the potential for bias by inspectors assigned to an ERC. For example, in the 2006 case, the FAA inspector responsible for oversight of partnership programs was a former air carrier employee whose actions could be perceived as biased in favor of the air carrier.

ASAP Is Viewed as an Amnesty Program: We found several instances in which FAA, air carrier, and union representatives incorrectly viewed ASAP as an “amnesty program” and believed that any corrective or enforcement actions for ASAP-reported incidents would be inappropriate. For example, pilots at one carrier objected to ERC members merely obtaining additional information about an incident disclosed in an ASAP report. FAA’s annual ASAP reviews have found similar concerns. Specifically, 25 percent of respondents in FY 2007 and 16 percent in FY 2006 indicated that ASAP was perceived as an amnesty program. In FY 2005, 50 percent of respondents indicated this perception (see exhibit C). We believe this is due to unclear FAA guidance on the authority of the ERC, which can adversely affect ASAP implementation.

In multiple, annual ASAP reviews FAA noted that ERC member and ASAP manager interviews indicated a need to further define what constitutes a corrective action, when an action is appropriate, and how an action should be recorded and tracked. According to FAA, in many cases, ERC members were reluctant to recommend corrective actions when the ASAP report was the only source of information about an incident due to concern that these actions might inhibit employee participation.

Confusion about the intent of ASAP has hindered the program at multiple air carriers. For example, Delta, American Airlines, US Airways, and Comair pilots ended their participation in ASAP because of disagreements regarding corrective actions. At Delta, pilot union representatives expressed concern about the airline’s policy of initiating additional corrective action (e.g., check rides) for pilots who have submitted ASAP reports. According to Delta managers, the union felt that once a pilot submitted a report to ASAP, Delta should have a “hands-off” policy and that it was inappropriate to interview the pilots. Delta pilots withdrew from ASAP in December 2006 and reinstated the ASAP program in January 2009 at the urging of FAA. Also, American Airlines and its pilots union were unable to agree on MOU language limiting the carrier’s authority to take disciplinary action for events reported under ASAP in instances where the carrier had independently obtained information on the reported event.

Under current FAA guidance, employees submitting reports that are accepted under ASAP are subject to the following FAA and air carrier actions:

• Sole source reports (all evidence of the event available to FAA or the air carrier is predicated on the ASAP disclosure): These reports will be closed by FAA with no action. Similarly, air carriers may not use the information obtained in this way to initiate disciplinary action against the employee.

• Non-sole source reports (information about the event in question is known by individuals other than just the ASAP reporter): These reports will be closed by FAA with administrative action (i.e., warning notice or letter of correction). Air carriers may not use information obtained through the ASAP report to take disciplinary action against an employee but may require corrective action, such as additional employee training. FAA must clarify the ERC’s authority so that the program is not perceived as an amnesty program. Specifically, in each air carrier’s MOU, FAA must clarify to ASAP participants its statutory authority under Title 49 of the United States Code (49 U.S.C.) to enforce the necessary rules and regulations. For example, the ERC

must reach a consensus when deciding whether a report is accepted into the program and when deciding on corrective action recommendations. However, ASAP guidance provides that if there is not an ERC consensus on decisions concerning a report involving an apparent violation, a medical certification, or qualification issue, the Agency’s ERC representative will decide how the ASAP report should be handled. FAA’s ASAP Training Does Not Emphasize the Need for ERC Members To Be Impartial ERC members have the authority to “forgive” violations of the Federal Aviation Regulations. For this reason, FAA’s ASAP training and supervision should place significant emphasis on ensuring the impartiality of FAA’s ERC members. The potential for bias in ERC decision-making can arise from an inspector’s former airline employment, long-standing inspector assignments, or professional or personal relationships developed within the aviation industry.

For example, during our review of the January 16, 2006, accident, we determined that FAA’s ERC representative tasked with reviewing and subsequently accepting the accident into ASAP was a former pilot for the air carrier. The appearance of a conflict of interest was raised again when we learned that this ERC inspector received a copy of the complainant inspector’s statement questioning the acceptance of the accident into ASAP. The inspector subsequently provided the document to both the air carrier and pilot union ERC representatives. The inspector stated that he shared the information because he felt the other ERC members were entitled to it as part of the ERC’s investigation of the accident. However, FAA guidance states that “confidentiality is a significant feature of hotline operations,” and we believe that disclosing this information to third parties is inappropriate.

To avoid the appearance of bias in ERC reviews of ASAP reports, FAA needs to emphasize the need for impartiality as part of its required ERC training and implement procedures to require periodic refresher training in this area. Additionally, FAA should clarify FAA field office management responsibilities to ensure personal relationships between inspectors and airline personnel do not influence decision-making. FAA Lacks a Process To Effectively Collect and Analyze ASAP Data FAA needs to develop a central ASAP database to collect information from ASAP reports in a redacted form, but with sufficient information to identify trends. Currently, FAA inspectors’ quarterly reports do not provide adequate details about the nature of ASAP events or the effectiveness of carriers’ corrective actions. Further, FAA does not have a process that permits it to collect and analyze ASAP

data to identify potential precursors to accidents and incidents. At present, all ASAP data reside in computers at air carrier offices because of air carriers’ concerns about confidentiality. While we understand these concerns, we maintain that FAA needs to obtain these data. We believe FAA has the means and resources for protecting the confidentiality of the data in a manner and method similar to that used for ASRS, in which the data are de-identified. Finally, FAA needs to maximize the benefits of ASAP data through improved, standardized inspector reporting and centralized data collection to enhance the national margin of safety.

FAA Inspector Reports Do Not Include Sufficient Information on ASAP Events or Carriers’ Corrective Actions FAA inspectors compile quarterly reports of ASAP activity for each participating air carrier and submit them to the Voluntary Safety Programs Branch at FAA Headquarters. We found, however, that these reports do not provide sufficient details about the nature of ASAP events to be effective safety data analysis and trending tools, and they vary dramatically in content among inspection offices.

Further, FAA guidance for submitting quarterly reports of ASAP activity does not require inspectors to provide summary information on all ASAP reports submitted for the quarter. Each air carrier is required to identify notable improvements made in the air carrier’s operation as a result of ASAP reporting. However, these program improvements likely resulted from a small fraction of the total ASAP reports submitted.

For example, 1 carrier submitted 310 ASAP reports in 1 quarter; however, only 3 safety enhancements were identified for that carrier, and there was no information about the nature of the remaining reports. Summary information on all 310 reports could have been valuable for national trending. ASAP reports that did not result in program improvements at one air carrier could result in national safety enhancements if all ASAP reports received from air carriers were analyzed to determine if other air carriers experienced similar problems.

We also identified repetitive reporting of problems and safety enhancements from quarter to quarter. For example, we found that some areas of concern, such as including altitude deviations and checklist usage, continually showed up as unresolved in FAA inspectors’ quarterly reports. According to FAA’s guidance on ASAP, FAA requires carriers to follow through with corrective actions that are acceptable to all members of the ERC to resolve any safety deficiencies; otherwise, FAA could exercise the right to terminate the carrier’s ASAP. In our view, repetitive reports of the same problem would indicate that a carrier’s ASAP program is ineffective. Yet, we found that FAA inspectors do not flag these repetitive items or further examine the carriers’ corrective actions; rather, they simply include the issues in their report and send it on to the Voluntary Safety Programs Branch.

An independent review team convened by the Secretary of Transportation in May 2008 to examine FAA’s safety culture and management also identified this issue. The team’s report stated that audits of safety violation disclosures and carriers’ acceptances can validate adherence to program rules. Audits would also ensure that FAA does not accept repeated disclosures from the same regulated entity, which would indicate a failure to implement sufficiently comprehensive fixes the first time. The report noted that “any willingness on the part of the FAA (real or perceived) to accept such repeat disclosures would undermine incentives for compliance.” Our review found that at one air carrier incomplete or missed checklist items were noted in the first quarter of 2006 as the most serious events, which were to be emphasized in pilot retraining. Yet, this problem persisted through the next two quarterly report submissions. The fact that the same safety violations are recurring from quarter to quarter indicates the need for heightened concern and stronger measures to prevent recurring problems and ensure the success of the carrier’s ASAP.

Under ASAP, safety issues are resolved through corrective action, rather than punishment or discipline. According to ASAP guidance, the ERC should work with the air carrier to develop appropriate corrective actions. FAA inspectors should therefore ensure that these corrective actions were used to educate the appropriate parties to prevent a recurrence of the problem.

FAA Does Not Analyze ASAP Data on a National Level To Identify Safety Trends In addition to the lack of analysis of ASAP reports at the local level, we found that FAA does not analyze ASAP reports at the national level. Although FAA’s Voluntary Safety Programs Branch at Headquarters receives inspectors’ quarterly ASAP reports, FAA has been slow in developing a database that would allow ASAP reports to be analyzed to identify safety trends. Instead, the Voluntary Safety Programs Branch is only responsible for developing policies and procedures to improve the program.
We first identified this issue in May 2007 when we reported that meaningful information on a runway incursion that occurred at Chicago O’Hare was lost because of the acceptance and secrecy of the report under ASAP. Another issue identified in this report was the lack of a central de-identified database to enter, store, and trend this type of information.

In 2008, the Secretary’s independent review team also identified this issue. Specifically, the team’s report stated that voluntarily disclosed safety data have not been routinely analyzed at a higher level within the FAA. The report recommended that FAA analyze Voluntary Disclosure Reporting Program (VDRP) and ASAP data (and data from many other sources) for two reasons: (1) to utilize these programs as potential contributors to the identification of trends and patterns that represent risks and (2) to guarantee the integrity of the voluntary programs themselves, eliminating any of the downside risks to compliance that might result from abuse.

To effectively use ASAP data, FAA needs to enhance the quarterly reporting process by requiring inspectors to submit summary information on all ASAP events, examine carriers’ repeatedly reported safety concerns, and ensure corrective actions were used to prevent a recurrence. Further, since the Voluntary Safety Programs Branch at Headquarters already receives inspectors’ quarterly reports, FAA should require this office to use these reports to develop a central ASAP database for national trending. These two efforts could provide FAA with valuable data for enhancing aviation safety nationwide.

FAA has contracted with Mitre Corporation to develop ASIAS, a system that will perform integrated queries across ASAP and other safety databases. Thirteen major air carriers are participating in ASIAS. FAA is using the system to assess the magnitude of problems that have already been identified. However, while FAA is continuing efforts to expand ASIAS functionality, the system does not have the ability to trend data to identify potential risk factors for FAA inspectors. In addition, access to ASIAS data is limited. Queries must be approved by an executive board, and results of ASIAS studies go to the Commercial Aviation Safety Team—a joint FAA and aviation industry group—and are not made public.

While FAA has established a good framework for obtaining valuable safety data through voluntary reporting by aviation employees, it needs to refine its current guidance to gain greater program benefit. FAA should consider developing a database tool similar to ASRS to include ASAP data. This would give FAA a centralized collection point of voluntarily reported safety data that could be used for national trending.

OIG Report Number AV-2007-050, “Progress Has Been Made in Reducing Runway Incursions, but Recent Incidents Underscore the Need for Further Proactive Efforts,” May 24, 2007. OIG reports are available on our website: www.oig.dot.gov.


RECOMMENDATIONS
Identifying potential precursors to accidents—the purpose of obtaining ASAP data—is essential to further reduce the accident rate. However, FAA will need to enhance its involvement in ASAP to fully benefit from this program. To maximize the safety benefits from ASAP, we recommend that FAA:
1. Revise current ASAP guidance to exclude accidents from the program and clarify what constitutes an “intentional disregard for safety.”
2. Require that FAA representatives on ERCs receive ASAP reports in a timely manner and concurrently with other ERC members.
3. Modify Advisory Circular 120-66B to clarify that ASAP is not an amnesty program and that employees submitting ASAP reports are subject to administrative action by FAA and corrective action by the air carrier.
4. Revise its ERC training to emphasize the need for FAA’s ERC members to remain impartial and require periodic refresher training in this area.
5. Clarify field office management responsibilities to ensure personal relationships between inspectors and airline personnel do not influence decision-making.
6. Standardize current ASAP guidance regarding quarterly report submissions and ensure they include, at a minimum, summary information regarding the ASAP reports submitted
7. Require inspectors to examine repetitive reports of safety concerns and enhancements to ensure that corrective actions are completed in a satisfactory manner.
8. Develop a central database of all air carriers’ ASAP reports that the Agency can use for trend analysis at a national level.

AGENCY COMMENTS AND OFFICE OF INSPECTOR GENERAL RESPONSE
We provided FAA with our draft report on March 10, 2009, and received its response on April 28, 2009. FAA concurred with seven recommendations and partially concurred with recommendation 1. FAA’s response is summarized below and included in its entirety in the appendix to this report.

In its response to recommendation 1, FAA agreed to clarify what constitutes “intentional disregard for safety” by revising its advisory and inspector guidance materials by December 31, 2009. However, FAA did not agree to exclude accidents from ASAP, believing it may have some unintended consequences. FAA stated that because ASAP requires that a report be filed within 24 hours of the event and because airline employees trust the ASAP process, it is possible they will be more accurate and candid in an ASAP report than they might be in interviews with either NTSB or FAA representatives. FAA also maintained that because the ERCs review and recommend corrective actions in a short timeframe, this allows safety risks to be mitigated before either FAA or the NTSB completes their investigations. FAA stated it has begun consultations with NTSB senior staff to determine whether they agree that accidents should be excluded from ASAP programs.
We reject FAA’s assertion that employees might be more candid and accurate in an ASAP report because we believe aviation professionals would understand it is their responsibility to be candid with a Federal investigator. Further, FAA’s contention that NTSB investigations may take several years ignores the fact that NTSB teams usually arrive on accident scenes within hours, question witnesses immediately, and have greater investigative resources and skills than ERC members. In addition, the NTSB issues preliminary reports and updated information throughout its investigations, and air carriers, airports, other aviation stakeholders, and even FAA may take corrective actions before the NTSB’s final report is issued. Therefore, we continue to maintain that accidents should be excluded from ASAP.
However, we agree that the NTSB can provide an important perspective on this issue. We will leave recommendation 1 open until FAA concludes its consultations with NTSB on this matter. Closure of this recommendation will depend on the outcome of those discussions.
FAA concurred with recommendations 2 through 8 and proposed acceptable actions and target completion dates. FAA also noted with respect to recommendation 8 that ASRS and ASIAS already provide FAA the ability to use

ASAP reports in trend analysis. We agree that ASRS is a valuable safety tool because of the alerts to FAA and others concerning safety issues identified from ASRS submissions. However, not all ASAP reports are included in ASRS and, as such, it cannot be considered a central ASAP database. Further, ASIAS does not include all air carriers, has limited user access and reporting, and, by FAA’s own admission, does not currently trend safety data.

ACTIONS REQUIRED: In accordance with DOT Order 8000.1C, we request that FAA reconsider its position regarding the part of recommendation 1 related to excluding accidents from ASAP reporting. Please provide your written response regarding recommendation 1 within 30 days of issuance of this report. FAA’s planned actions and target dates for recommendations 2, 3, 4, 5, 6, 7, and 8 are responsive, and we consider these recommendations addressed pending completion of the proposed actions. We appreciate the courtesies and cooperation of FAA representatives during this audit. If you have any questions concerning this report, please contact me at 202-366-0500 or Tina Nysted, Program Director, at (404) 562-3770.

From: Lou E. Dixon Assistant Inspector General for Aviation and Special Program Audits
Report Number: AV-2009-057
To: Acting Federal Aviation Administrator

Report Number: AV-2009-057
Date Issued: Ma y 14, 2009