AC 120-90 - Line Operations Safety Audits

Advisory

Circular

Subject: LINE OPERATIONS SAFETY

AUDITS Date: 4/27/06

Initiated by: AFS-230 AC No: 120-90

1. PURPOSE. This AC provides the rationale and pro cedure for conducting a Line Operations

Safety Audit (LOSA) at an airline. 2. APPLICABILITY. A LOSA is a voluntary safety progr am. While the Federal Aviation

Administration (FAA) encourages ai rlines to voluntarily conduct LOSA programs in the interest

of safety improvement, a LOSA does not entail any requirement for FAA approval, acceptance,

or monitoring. While an airline may elect to shar e the results of a LOSA with the FAA, there is

no requirement to do so. This AC is intended to provide informational guidance on LOSA to

airline personnel in flight safet y, flight training, flight operati ons, and other interested industry

parties.

3. CONTENTS.

a. Introduction. This AC introduces the LOSA proce ss and distinguishes it from other

proactive safety data programs such as Fli ght Operational Quality Assurance (FOQA) and the

Aviation Safety Action Program (ASAP).

b. Data Collection. This AC also lists the various type s of data collected via a LOSA and

shows how they contribute to an airline’s sa fety management system. Next, the personnel

involved in a LOSA and their re spective roles ar e described.

c. Implementation. This AC offers a detailed step-by-st ep guide to the LOSA process from

initial planning to the final report. It concludes with the 10 operating char acteristics that define a

LOSA and which ensure the integr ity of the process. Under the guidance of this AC, an airline

wanting to implement a LOSA should abide by all 10 characteristics. The appendices in this AC

describe the Threat and Error Management (TEM) model upon which the LOSA method was

based (Appendix 1); provide more detail on the training and standardizin g of LOSA observers

(Appendix 2); provide a sample observation form (Appendix 3); a sample letter to the pilots

(Appendix 4); and sample vignett es for training (Appendix 5).

AC 120-90 4/27/06

Page 2 Par 4 4. DEFINITION OF A LOSA.

a. A LOSA is a formal process th at requires expert a nd highly trained obser vers to ride the

jumpseat during regularly schedul ed flights to collect safety-re lated data on environmental

conditions, operational complexity, and flightcrew performance. Confidential data collection

and non-jeopardy assurance for pilots are fundamental to the process.

b. Using a medical metaphor, a LOSA is similar to a patient’s annual physical examination.

People have comprehensive checkups in the hope of detecting serious health issues before they

become consequential. A set of diagnostic meas ures, such as blood pressure, cholesterol, and

liver function, flag potential heal th concerns, which in turn su ggest needed changes to the

patient’s current lifestyle. A LOSA is built up on the same proactive notion. It provides a

diagnostic snapshot of strengths and weaknesses that an airline can use to bolster the “health” of

its safety margins and prevent degradation.

c. LOSA is distinct from - but complementary to - other proactive safety programs such as

electronic data acquisition systems (e.g., FOQA), and voluntary reporting systems (e.g., ASAP).

However, these programs have two major conceptual differences.

(1) First, FOQA and ASAP rely on outcomes to generate data. For FOQA, it is flight

parameter exceedances; for ASAP, it is adverse ev ents that crews report. By contrast, LOSA

samples all activities in normal operations. In th ese regularly scheduled flights, there may be

some reportable events, but there will also be some near-events and, importantly, a majority of

well-managed, successful flights. LOSA provi des a unique opportunity to study the flight

management process, both successful and unsu ccessful, by noting the problems crews encounter

on the line and how they manage them.

(2) The second major difference is the perspect ive taken by each program. With its focus

on electronic data acquisition downl oaded directly from the aircra ft, FOQA can be said to have

the “airplane perspective.” AS AP provides the “pilot perspectiv e” by using pilots’ voluntary

disclosure and self-repor ting of events. ASAP reports provid e insight into why events occur as

seen from the crew’s perspective. By c ontrast, LOSA provides a “neutral, third-party

perspective” in that LOSA observe rs record contextual and fli ghtcrew data on every phase of

flight, regardless of the outcome. All three perspectives provide useful data to an airline’s safety

management system.

(3) A third, more pragmatic difference between the programs relates to logistics. FOQA

and ASAP are continuous programs; i.e., they ar e set up to collect data on a daily basis. A

LOSA is more project-based. The full LOSA process, from advance planning and observer

selection and training, to data collection, analys es and final report, can take between 6 and

12 months. A LOSA is recommended every 3 year s. Despite these differences, data from one

program can be cross-referenced and used to gui de data collection in a nother. For example,

ASAP reports may highlight a problem with departur es at a particular air port. This information

can be fed to the LOSA steering committee, which can then target more obs ervations out of that

airport to understand the magnitude and specifics of the problem. As another example, a LOSA

may identify a high incidence of unstable approaches , leading to a review of procedures and the

4/27/06 AC 120-90

Par 4 Page 3 specification of new approach parameters. FOQA data can track adherence to the new

specifications in the interim peri od leading up to the next LOSA.

5. IMPORTANCE OF A LOSA. A LOSA provides unique data about an airline’s defenses

and vulnerabilities. As explained in paragraph 4, a LOSA does not replace other safety-data

sources such as FOQA or ASAP. Instead, it co mplements these programs and extends the reach

of an airline’s safety management system. The data collected during a LOSA can impact almost

every department in an airline, as the followi ng examples show. The data collected during a

LOSA can help an airline:

a. Identify Threats in the Airline’s Operating Environment. Observers note events in the

operational environment (e.g., adverse weather, airport conditions, air traffic control (ATC)

clearances, terrain, and traffi c congestion, and how they are managed by flightcrews). High-

prevalence threats and/or threats with higher mism anagement rates can be prioritized for further

investigation; lower mismanagem ent rates signify areas of stre ngth. For example, understanding

the extent to which certain airports or ATC practices pose a problem for flightcrews, and

capturing the strategies flightcrews adopt to deal with them, can lead an airline to develop special

procedures or advisories to help its pilots manage the known threat.

b. Identify Threats from with in the Airline’s Operations. Observers note events arising

from within the airline’s ow n operations and how they are managed (e.g., operational time

pressure, dispatch errors, aircraft malfunc tion/minimum equipment list (MEL) items, and

problems with ground, ramp, maintenance, and cabi n personnel). A high number of threats

arising from dispatch or cabin might signal that these departme nts require attention, or that

intergroup cooperation with pilots needs to be improved, or that procedures are inconsistent

across departments. As above, prevalence and ma nagement rates provide cues for prioritizing

action.

c. Assess the Degree of Transference of Training to the Line. Data provided by

Advanced Qualification Programs (AQP), Line Operation Evaluations (LOE), and Line-

Operational Flight Training (LOFT) can provi de insight on whether training concepts are

learned, but not whether they are actually pr acticed on the line. A LOSA provides that

operational information, which can be reviewed from a training perspective to understand which

areas of training, if any, are not tr ansferring successfully to the line.

d. Check the Quality and Usability of Procedures. A LOSA provides insights about

potential problems with procedures. For example, if 5 percent of observed crews make a callout

error during descent/approach/land, there may be a problem with those crews. However, if

50 percent of observed crews make the same error, then the evidence suggests a problem with

the callout procedure. Procedures can be ill- timed, over-long, confusing, and/or compete for the

pilots’ attention with other more important activities. A LOSA will locate problematic

procedures and policies via poor adherence rates. A LOSA can also identify the extent of

procedural deviations across fleets.

AC 120-90 4/27/06

Page 4 Par 5 e. Identify Design Problems in the Human/Machine Interface. A LOSA captures aircraft

handling and automation errors on different fleets that can highlight syst emic flaws in design,

interface, or adaptation. The rate at which certain errors go undetected and become

consequential can also indicate potential design vulne rabilities. An airline can feed these LOSA

findings back to the aircraft ma nufacturers, as well as writi ng standard operating procedures

(SOP) to circumvent the flaws.

f. Understand Pilots’ Sh ortcuts and Workarounds. With experience comes expertise;

pilots learn ways to save time and be more effici ent. These techniques are rarely seen in a line

check, when performance is usually done “by th e book.” A LOSA provides a principled manner

by which an airline can capture collective exper tise from within the pilot group, and then share

that information with all its pilots through form al airline communication channels. Using LOSA,

false expertise — the adoption of a shortcut or workaround that is flawed in its safety

assumptions — can also be identified and remedied.

g. Assess Safety Margins. Threats and errors that are mism anaged can result in undesired

aircraft states, if sufficiently serious. Vertical and lateral deviations and unstable approaches are

examples of undesired aircraft st ates, also known as accident and incident precursors. A LOSA

provides data about the prevalence and management of these incident and accident precursors.

Thus, an airline acquires data about how close it is operating to the edge of the safety envelope,

without crossing the boundary into an incident or accident.

h. Provide a Baseline for Organizational Change. LOSA results provide baseline and

outcome measurement data against which organizat ional interventions can be measured. Using

the medical metaphor, this would be akin to the patient deciding to cut out fried foods upon

learning of a high cholesterol count. The next checkup reveals, in qua ntifiable form, whether

this strategy has been effective in reducing chol esterol or whether other actions are necessary.

Similarly, a followup LOSA provides a new set of results that will show whether the

organizational changes were effect ive in reducing certain threats, errors, and/or undesired states.

i. Provide a Rationale for Allocation of Resources. Because LOSA results highlight both

the strengths and weaknesses in an organization, the results provide a da ta-driven rationale for

prioritizing and allocating s carce organizational resources toward interventions.

6. INVOLVEMENT IN A LOSA. When first exploring whether or not to conduct a LOSA, it

is advisable to gather representatives from a ll departments that may be potentially involved,

including flight operations, traini ng, flight standards, the safety department, and the pilot group.

a. Departments. The flight operations and training de partments typically know first-hand

what is and is not working well. These depart ments often have specific areas that they would

like the LOSA to focus on. Possibly the most im portant reason for their involvement is that,

ultimately, many of the problem areas that are identified during a LOSA should be addressed by

these departments. They will also be the recipients of the potential benefits derived from the LOSA. If these departments do not support LOSA, then there will be resistance to the findings;

however, if these departments are part of the pr ocess, there will be a sense of ownership, and

they will be invested in the results.

4/27/06 AC 120-90

Par 6 Page 5

b. Pilots’ Association or Group.

(1) The importance of having the pilots invol ved with and supporting the LOSA cannot

be overstated. If the line pilots are convinced that their asso ciation or pilot group supports

LOSA, they will be more willing to accept the presence of observe rs on their flight decks.

Additionally, if pilots believe this process is be neficial to them and to safety, they will be

forthcoming and candid with their views and safety concerns. On the other hand, if the pilots

view LOSA as a management tool to “spy on their cockpits” and they respond with “angel”

performance rather than typical performan ce, then the results will not be fruitful.

(2) Hence, where airlines have a formal pilo ts’ association, leader s of the association

should be involved in the LOSA process from the beginning. If no formal pilots’ association

exists, pilot repres entatives should still be in cluded. The pilots’ asso ciation or group can also

help disseminate the results of the LOSA and inform the pilots as to the company’s plans as a result of the LOSA.

c. LOSA Coordinator and Steering Committee. Because buy-in and support is crucial,

consideration should be given to forming a LO SA steering committee drawn from the various

departments and the pilots’ association. Th e LOSA steering committee and/or the LOSA

coordinator have many tasks and logist ical responsibilities, including:

(1) Publicize the upcoming LOSA in pilo t newsletters to build awareness.

(2) Distribute a letter to all line pilots e xplaining the purpose of the LOSA (Appendix 3

has a sample letter).

(3) Decide the size and focus of the LOSA.

(4) Select the observers, orga nize their schedules, and set up observer training.

(5) Organize a secure site for collection of the observation forms and subsequent data

analysis.

NOTE: All of these tasks are discu ssed in more detail in the “How to”

section of this document.

d. Observers.

(1) LOSA observers should be carefully select ed to ensure the integrity of the LOSA

process. LOSA observers should be familiar with the airline’s procedur es and operations; the

line pilots should also respect th em. The observer team can include a small number of non-pilots

as long as they can anticipate and understand flightcrew tasks and th eir surrounding operational

context. However, the majority of the team should be active pilots from that airline.

AC 120-90 4/27/06

Page 6 Par 6

(2) In airlines that operate more than one fleet , observers should be scheduled to observe

across fleets other than their own. This adds valu e to the process in that the observer looks more

at the “big picture” rather than the fine detail. For similar reasons, experience has shown that

using a small percentage of external observers dr awn from pilots with LOSA experience at other

airlines adds value in the form of a “contro l group” for observations. External observers—

typically 10 to 20 percent of the total observe r team—will normally atte nd the airline LOSA

training and will need to familiarize themselves w ith airline procedures. They are particularly

useful in picking up systemic and organizational threats to which airline staff are often “blind”

due to familiarity.

(3) A LOSA observer should be like a “fly on the wall,” able to occupy the cockpit

jumpseat and capture data without being obtrusive or interfering with the crew’s performance.

This involves creating an environment where the crew almost does not re alize they are being

observed. LOSA observers will observe errors an d undesired aircraft states as part of their

observations; however, they should only interrupt if they perceive the safety of the flight to be

seriously and immediately endangered. A helpful ru le of thumb is to ask observers to think of

themselves as a guest riding in the jumpseat of a nother airline. This seems to help distance the

observer from the flightcrew while still being able to politely point out safety concerns if they

come about.

(4) LOSA observers should be scheduled in th e status of “passenger,” “supernumerary,”

or “observer” only (i.e., they should not be schedu led as a member of the legal operating crew).

This assists both the crew and the observer to understand that LOSA observers are data

collectors only, not evaluators th ere to critique crews. Observer s do not have any responsibility

for the operation.

e. Line Pilots. A LOSA cannot succeed without the fu ll and candid cooper ation of the line

pilots, and there can be no cooperation without trust. Line pilots should be informed in advance

about the purpose and planned implementation of a LOSA. They should receive a letter co-

signed by credible representatives of both manage ment and the pilot organization that assures

them of the confidential and nonjeopardy status of LOSA data. The letter should also include a

disclaimer giving all pilots the c hoice of declining a jumpseat obs erver at their discretion. Only

by building in these guarantees and safeguards will the line pilots feel sufficiently comfortable to

act normally in the cockpit in the presence of a LO SA observer. A final assurance should be an

inhouse publication of a summary of LOSA results along with an outline of initial actions and

proposed changes.

f. Data Analyst and Report Writer. The data analyst should have knowledge of the

airline’s flight operations as well as database mana gement and data analysis skills. However, an

airline might choose a thir d-party analyst if exper tise is not available inh ouse or if line pilots

have expressed reservations about the integrity of the LOSA implementati on or objectivity of the

final report. The data analyst a nd report writer work together to prepare a report of the findings

to be presented to management and pilots.

4/27/06 AC 120-90

Par 7 Page 7 7. WHEN TO CONDUCT A LOSA.

a. There are several factors to consider when scheduling a LOSA. Given all the personnel

involved, a LOSA should be scheduled to fit with other operational prioriti es. For example, is

there a particular time in the year when more obs ervers will be available? Is there a better time

for the scheduling department to roster these people ? Also, is there a particular time that is more

interesting from a safety or operational perspect ive? Some examples: bad-weather season, peak

traffic season, after the introducti on of an operational change such as new aircraft, altered routes,

or a merger.

b. A LOSA should not be implemented immediately after a major incident or accident. The

airline will be in a heightened state of awareness at this time and pilots will be overly sensitive to

observation; hence, the chances of getting normal data will be diminished. At a minimum,

airlines should wait at least 1 year after a majo r safety event before scheduling a LOSA. Once

an airline has done a LOSA, a critical question is when to do the next one. LOSA data provide a

baseline against which to measure improvements. A realistic timeframe to review LOSA results,

develop and implement action plans, and monitor results is 3 years. Hence, to measure the

effectiveness of organizationa l changes, a repeat LOSA every 3 years is recommended.

8. HOW TO IMPLEMENT A LOSA. This section presents a step-by-step guide to

implementing a LOSA. Broadly speaking, there ar e steps associated with getting good-quality

data from observers (data collection), and step s associated with ensuring that accurate and

meaningful data are given to management and lin e pilots (data analysis and feedback). An

airline can conduct its own LOSA by observing the following steps:

a. Data Collection.

(1) Form a LOSA steering committ ee and appoint a LOSA coordinator.

(2) Gather Informat ion and LOSA Resources from other Airlines and Industry

Groups. Before conducting a LOSA for the first ti me, the committee and coordinator should

seek out information from other airlines that have already conducted a LOSA. Other airlines

may be able to share observer selection and training techniques, observation forms, scheduling

tips, and other logistical aids.

(3) Publicize LOSA within the Airline and Send a Letter to the Line Pilots. A first

task is advance publicity via company publicati ons to build line pilot awareness and acceptance

of the upcoming LOSA. Next, the coordinator orga nizes and distributes a letter to all pilots

explaining the purpose of the LOSA. This letter specifies the purpose of the audit, the fact that

all observations are of a non-jeopardy nature, and that all data will be kept strictly confidential.

The letter is signed by the highe st level of management within flight operations, with the

endorsement of other relevant personnel such as chief pilots and pilots’ representatives

(Appendix 4 has an example letter). The letter of announcement should precede the line audit by

at least 1 month, with a followup alert 1 week before starting observati ons. LOSA observers

should have copies of the si gned letter to show crewmembers in case questions arise.

AC 120-90 4/27/06

Page 8 Par 8

(4) Decide the Focus of the LOSA. The LOSA steering committ ee decides the focus of

LOSA. One option is to sample broadly across th e entire operation—this would be an effective

strategy for a first LOSA. Alte rnately, the LOSA steering comm ittee can focus on problems that

have been identified by other data sources, such as FOQA and ASAP. This approach would

schedule LOSA observations on partic ular routes, in certa in regions, or into particular airports

that have been identified as problematic. Th e committee can also focus a LOSA on a new fleet

or other recent organizational changes.

(5) Decide the Number of Observations.

(a) Most airlines will find it cost effective to conduct a LOSA on a sample of their

operation—the question is how big a sample? As a general guideline for a full LOSA, match the

number of observations per fleet to the relative number of depart ures per day. For example, if

30 percent of departures occur on Fleet A, then approximately 30 percent of the LOSA

observations should occur on Fleet A. Within eac h fleet, try to sample as many different crews

as possible and, as a rule, conduct 50 or more obser vations per fleet. Below that number, there is

the risk of not accurately capturing a representative sample.

(b) Modify the guideline slightly when focusi ng on a particular operation or region.

For example, to sample international flights into a particular subcontinent, then regardless of what percentage they constitute of the airline’s daily departures, still schedule at least 50 observations to ensure a good sample.

(6) Create an Observation Form. The observation form should be based on a

conceptual framework that captures multiple aspects of normal operations, including the

operating environment and flightcrew performance. It should provide categories and codes to

streamline observations and save the observer’ s time, but it should al so require a written

description of the flight that captures the full context. A ppendix 3 has an example of an

observation form based upon the TEM model.

(7) Select observers.

(a) The observer team should have representa tives from flight operations, training,

safety, and the pilots’ association. Some ai rlines employ a selection procedure whereby

management and the pilots’ association each put forth a list of acceptable observers, and then

those who appear on both lists are selected.

(b) The number of observers needed depends on the size of the audit and the

observers’ workload. There is substantial work involved in completing an observation form and

providing a detail-rich narrative for each fli ght; therefore, the recommended number of

observations is 10 to 15 per obser ver, depending on routes and sche dules. Hence, a LOSA that

plans 150 domestic observations requires at l east 10 observers, while a 300-observation LOSA

that includes international flight s requires 20 to 25 observers.

4/27/06 AC 120-90

Par 8 Page 9 (8) Train observers.

(a) LOSA observers should be educated a bout the purpose and ra tionale of LOSA,

and trained in the use of the observation form. LOSA observer training typically takes 2 to

3 days. To assist in the design of the traini ng, members of the steering committee may want to

attend LOSA observer training at another airlin e first or, if possible, attend an industry-

sponsored LOSA conference for guidance.

(b) Observers should practice with scripted scenarios or videos until they are

confident they can use the observation form correctl y. At this point, they can be dispatched to

the line; however, it is recommende d that observers be brought back in after one or two flights to

discuss their observations, corre ct any misperceptions, and coach them on areas that require

clarification. Appendix 2 pr ovides more detail on the objec tives and content of observer

training. Appendix 2 also addresses the sta ndardization of observ ers and LOSA data.

(9) Schedule Observations. Plan no more than two observa tions per observer per day to

allow sufficient time to complete the observati on form and write a rich narrative. Schedule

observers across fleets regardless of their type rating to encourage a more general, cross-fleet

perspective of flightcrew performa nce. Build some flexibility in to the schedule to allow for the

unexpected. Finally, do not let the observations continue indefin itely—schedule all observations

within a 1- to 3-month period if possible, or else the impact of LOSA will be lost.

(10) Decide on a Data Repository.

(a) The LOSA coordinator organizes a secure site for the data and oversees the

receipt of the observa tion forms. The coordinator should be able to protect the identity of the

observers and the observed to ensure complete confidentiality and nonjeopardy conditions.

Under no circumstances should it be possible to connect individuals wi th particular observations.

(b) The observations can be kept inhouse if da ta management and analysis expertise

is available and if data security can be assured. Alternately, the data can be sent to a trusted third

party who will assume responsibility for data collection, cleaning, and analysis. The decision

will depend on airline resources and pilot trust issues.

(11) Provide L ogistical Support. Give the observers the name of a contact person,

most likely the LOSA coordinator, who can be re ached if there are any problems with scheduling

or data collection.

b. Data Analysis and Feedback.

(1) Verify the Data. Convene a meeting of “local e xperts”—airline personnel familiar

with the operation of each fleet (possibly fleet managers or member of the steering committee, but not any of the observers). The group’s task is to review and verify the observations against

current manuals, policies, and procedures. Fo r example, an observer might log a procedural

error for failure to make an approach callout wh en in fact there is no written procedure in the

airline’s flight operations manual. The data ve rification group would delete this particular

AC 120-90 4/27/06

Page 10 Par 8 “error” from the database. This step is a data integrity check in that it ensures that events are

correctly recorded in line with each fleet’s procedures and policies. It also builds ownership in

the results and dispels a ny later criticism that the coding wa s not an accurate representation of

the airline’s operations.

(2) Analyze Data.

(a) LOSA data reveal stre ngths and vulnerabilities in an airline’s operations. The

data analyst should investigate the prevalence an d management of different events and errors.

Although certain types of comparisons will seem obvious, many analyses can and should be based upon hunches and theories derived from lo cal knowledge of operations. If the analyst

knows how fleets and operations ar e managed, comparisons that re flect this structure can be

made. If the analyst knows the ki nds of information that might be useful to training, safety, or to

domestic or international flight operations, results can be tailored to these particular aspects of

the operation. Feedback from va rious airline stakeholders is critical during this stage of

preparing the report. The analys t should not hesitate to distri bute early drafts to key people

within the airline familiar with LOSA to crossv erify the results. This not only helps validate

derived trends, but it gives other airl ine personnel ownership of the report.

(b) Patterns will emerge as the data are analyzed. Certain errors occur more

frequently than others, certain ai rports or events emerge as more problematic than others, certain

SOPs are routinely ignored or modified, and certain maneuvers pose greater difficulty for

adherence than others. These events and prac tices form the basis of suggested targets for

enhancement.

(3) Prepare Report. The last stage of a LOSA is a written report that presents the

overall findings of the audit. With a large da tabase like the one generated from a LOSA, it is

easy to fall into the trap of trying to pres ent everything. The repor t should be concise and

present only the most significant trends from the data. Along w ith the results, the report should

provide an initial list of targets for enhancement. Targets need to be action-focused and data-

driven. Some example targets that might emerge from a LOSA include:

(a) Reduce the number of unstabilized approaches.

(b) Streamline predeparture checklists.

(c) Reduce SOP crossverification errors.

(d) Understand automation errors on the new fleet.

(e) Investigate conditions at airports X and Y.

(f) Improve management of a dverse weather threats.

(g) Investigate high rate of MEL items on the ZZ fleet.

4/27/06 AC 120-90

Par 8 Page 11 (h) Reduce dispatch errors at the hub.

(i) Develop an international flight operations guide.

(j) Develop a module on intentional noncom pliance errors for captain upgrade

training.

(4) Brief Management.

(a) The LOSA report should be presented to management in operations, training,

standards, safety, and possibl y other departments depending on the results. For example,

representatives from ramp, mainte nance, dispatch, and cabin may wa nt to hear how their work is

perceived from the pilots’ perspectiv e, particularly if it is problematic. A briefing to the pilots’

association or group, as applic able, is also recommended.

(b) Once the various departments are briefed on the report, they will likely want to

investigate the data more deeply themselves. Th e data should be available in aggregated form

for them to review. Some flight narratives will also be of interest, hence the prerequisite insistence on deidentifying the observations.

(5) Brief Line Pilots.

(a) Line pilots should also be informed of th e significant results in the LOSA report.

To sustain the pilots’ interest in the LOSA project, make an a nnouncement at the end of the data

collection phase that the LOSA observations have been completed, stating how many and on

what fleets, and advise when the pilo ts can expect to see the results.

(b) When the report is ready, the highlights shou ld be presented to the pilots, either

as one LOSA debriefing event or spread over time in the airlin e newsletter or other safety

periodical. Pilots will want to know what change s will be undertaken as a result of the LOSA.

(6) Monitor Safety Change Process.

(a) Historically, organizational safety change s within airlines have been driven by

accident/incident investigation and intuition. Today, airlines must deal proa ctively with accident

and incident precursors. To be successful, the safety change process must be data-driven.

Measurement of daily operations is fundamental because unless an organization uses systematic

measurement, the perspective it ha s on the strengths and weaknesse s of its operations is largely

based on anecdote and opinion.

(b) A LOSA provides specific and quantified resu lts. To take full advantage of this

specificity, the targets for enhancement that ar ise from the data anal ysis should go through a

formal safety change process to produce improve ment. A formal safety change process provides

a principled approach to target limited resources and helps the airline avoid “turf” issues by

clearly defining and prioritizing th e issues that impact flight ope rations. The basic steps of a

safety change process are:

AC 120-90 4/27/06

Page 12 Par 8

1 Measurement (with LOSA) to obtain the targets.

Detailed analysis of targeted issues.

List of potential changes for improvement.

Risk analysis and prioritization of changes.

Selection and funding of changes.

Implementation of changes.

Time for changes to stabilize.

Remeasurement.

9. USE OF LOSA DATA.

a. A well-conducted and well-analyzed LOSA identifies strengths and vulnerabilities in an

airline’s operations. It provides this information in a quantifiabl e form against which targets can

be specified and improvements can be measured. The following example briefly illustrates the

step-by-step integration of LOSA data into the safety change process.

(1) An airline’s LOSA results indicate that 16 percent of observed flights involved an

unstable approach. Because observations were scheduled across the operation, and the number

of observations exceeded 50 per fleet, the LOSA co mmittee is confident that the percentage is an

accurate representation of operations as a whole.

(2) Following management briefings and exte nsive discussion, a specific target for

improvement is created to “reduce the number of unstabilized approach es by 50 percent; i.e.,

reduce the number of unstable a pproaches from 16 percent to 8 pe rcent or fewer of all landings.”

(3) An action committee is formed for unstabilized approaches. They formalize the

parameters and definition of an unstable approac h, they review existing procedures and training,

and they introduce changes in all relevant areas.

(4) A repeat LOSA is conducted 3 years after th e first LOSA. The data, once aggregated

and analyzed, show the new rate of unstable appro aches to be 12 percent. The airline concludes

that changes made to the operation were suc cessful in reducing the rate of unstabilized

approaches from 16 percent to 12 percent, an improvement of 25 percent. Upon reviewing the

results of the second LOSA, the airline recommits to its original target of reducing the unstable

approach rate to 8 percent or lower, and continues to focus efforts in this area.

b. Depending on the sophistication of an airline’s safety management system, and the extent

to which different safety programs within the airline are premised on the same conceptual

4/27/06 AC 120-90

Par 9 Page 13 framework, data from a LOSA can be cross-refe renced with data from the ASAP and/or FOQA

programs, if the airline operates one or both of those voluntary programs. Each data source

provides unique yet complementary evidence of the ai rline’s safety status. In the above example,

the airline might track unstable approaches through its FOQA program using new flight

parameters decided by the action committee and then implemented into procedures and training.

To see if pilots are incurring problems with th e new procedure, the FOQA aircraft data can be

cross-referenced with ASAP reports of events resulting from unstable approaches. This way, the

airline does not have to wait unt il the next LOSA to learn if its interventions are being

successful.

c. LOSA data are useful in a nother way. LOSA presents a broad view of operations; a

repeat LOSA can maintain that broad focus. Fo r example, did the changes that were introduced

after the first LOSA improve results in one area, only to cause problems in another? Checklist

adherence may have improved, but did error de tection—the superordin ate goal of improving

checklist adherence—actually improve or is the new adherence simply cosmetic?

10. SUMMARY: THE 10 OPERATIN G CHARACTERISTICS OF LOSA.

a. Ten operating characteristics define and summarize the LOSA process. It is

recommended that a LOSA observe all 10 characte ristics to ensure the integrity of the LOSA

process and the quality of the final product.

b. The 10 Operating Characteristics Are:

(1) Jumpseat Observations During Normal Flight Operations. LOSA observations

are limited to regularly scheduled flights. Line checks, initial line indoctr ination or other training

flights are off-limits due to the extra level of stre ss put on pilots during th is type of situation.

Having another observer onboard adds to an al ready high stress level, thus providing an

unrealistic picture of performance. In order fo r the data to be repr esentative of normal

operations, LOSA observations should be co llected on regular and routine flights.

(2) Joint Management/Pilots ’ Association Sponsorship. In order for LOSA to

succeed as a viable safety project, there needs to be support not only from the management side

but also from the pilots. The joint sponsorship provides a “check and balance” for the project to

ensure that change, as necessary, will be made as a result of LOSA data. When considering

whether to conduct a LOSA, the first question to be asked by airline management is whether the

pilots’ association (or pilot group representatives) endorses the project. If the answer is “No,”

the project should not be initiate d until endorsemen t is obtained.

(3) Voluntary Crew Participation. Maintaining the integrity of LOSA within an airline

and the industry as a whole is extremely importa nt for long-term success. To accomplish this

goal, all LOSA observations are collected with voluntary crew participation. Before conducting

LOSA observations, observers shoul d first ask the flightcrew for permission to be observed. If

the crew declines, the observer takes another flight with no qu estions asked. If an airline

conducting a LOSA has an unreasonably high numb er of declines, this should serve as an

indicator that there are critical “trust” issues to be resolved.

AC 120-90 4/27/06

Page 14 Par 10

(4) Deidentified, Confidential, and Nondisciplinary Data Collection. LOSA

observers are required to not record names, flight numbers, dates, or any other information that

can identify a crew or in dividual. The purpose of LOSA is to collect safety data, not to punish

pilots. Airlines cannot allow themselves to sq uander a unique opportunity to gain insight into

their operations by having pilots fearful that a LOSA observation could be used against them for

disciplinary reasons. If a LOSA observation is ever used for disc iplinary reasons, the credibility

of the entire safety program may be irreparably compromised.

(5) Targeted Observation Form. The LOSA observation form in Appendix 3 is

predicated on the TEM framework described in Appendix 1. However, other conceptual

frameworks can be used for LOSA data collec tion. Whatever framework is used, it should

generate meaningful data on a variety of topics, including what the crews di d well, what they did

poorly, and how they managed each phase of fli ght. A narrative written by the observer should

have sufficient detail to allow ot hers to understand the flight and all its events. The observers

need to describe the environmental conditions an d events surrounding the pilo ts’ behavior so that

the crews’ performance can be understood in full context.

(6) Trained and Calibrated Observers. Primarily, pilots conduc t LOSA. Observation

teams will typically include line pilots, instructor pilots, safety pilots, management pilots, and

representatives of the pilots’ safety committee. It is critical to select observers that are respected

and trusted within the airline to ensure line acc eptance of LOSA. After observers are selected,

they are trained and calibrated in the LO SA methodology, including the use of the LOSA

observation form. Observers’ tr aining in the concepts and methodology of LOSA will ensure

that the observations will be conducted in th e most standardized manner (see Appendix 2).

(7) Trusted Data Repository. In order to maintain conf identiality, airlin es should have

a trusted data repository. This site can be in-hous e, such as that used for other confidential data,

or it can be offsite. The goal is that no indivi dual observations will be misplaced or improperly

disseminated through the airline.

(8) Data Verification. Data-driven programs like LOSA require quality data

management procedures and consistency checks. For LOSA, these checks are done at data-

verification roundtables. A roundt able consists of three or four department and pilots’

association representatives who review all the raw data for possible inaccuracies. The end

product is a database that is validated for cons istency and accuracy acco rding to the airline’s

standards and manuals before any st atistical analysis is performed.

(9) Targets for Enhancement. The final product of a LOSA is the data-derived targets

for enhancement based on emergent patterns in the data . It is then up to th e airline to develop an

action plan based on these targets, using experts from within the ai rline to analyze the targets and

implement appropriate change strategies.

4/27/06 AC 120-90

Par 10 Page 15 (and 16) (10) Feedback Results to the Line Pilots . In order to ensure long-term success of

LOSA, airlines should communicate the results back to the line pilots. Pilots will want to see not

only the results of the audit, but al so management’s plan for improvement .

ORIGINAL SIGNED BY CAROL E.GILES (for) James J. Ballough Director, Flight Standards Service

4/27/06 AC 120-90

Appendix 1

Page 1 APPENDIX 1. THREAT AND ERROR MANAGEMENT

1. Introduction.

The Threat and Error Management (TEM) mode l is a conceptual framework for understanding

operational performance in complex environments. Originally created to capture the flightcrew’s

task in commercial aviation, the model is ge neric and can be applied to numerous work

situations. The added value that TEM brings to other performance models is that it focuses

simultaneously on the operating environment and the humans working in that environment.

Because the model captures ongoing performance in its “natural” or normal operating context,

the resulting description is realistic, dynamic, and holistic. Because the model can also quantify

the specifics of the environment and the effectiveness of performance in that environment, it is also highly diagnostic

There are several ways of using the TEM model, from focusing on a single event (as is the case

with accident/incident analysis) to understanding systemic patterns in a large set of events (as with Line Operations Safety Audits (LOSA)). As a training tool, TEM can help individuals

clarify their performance needs a nd vulnerabilities; and as part of a safety management system,

TEM can help an organization measure and impr ove the effectiveness of its organizational

defenses and safeguards.

2. The Model.

This section defines and provi des examples of the various components of the TEM model.

a. Threats.

A threat is defined as an event or error that occurs outside the influe nce of the flightcrew

(i.e., it was not caused by the crew), increases th e operational complexity of a flight, and requires

crew attention and management if safety margins are to be maintained.

There are threats from the environment—adverse weather, airport conditi ons, terrain, traffic,

and air traffic control (ATC)—and threats em anating from within the airline—aircraft

malfunctions and master equipment list (MEL) ite ms, problems, interruptions, or errors from

dispatch, cabin, ground, maintenance, and the ra mp. The crew, for example, may anticipate

threats by briefing a thunderstorm in advance; or they may be unexpected, occurring suddenly

and without warning such as in flight aircraft ma lfunctions. Some threats are easily resolved and

quickly dismissed from the crew’s workload, wh ile other threats requir e greater attention and

management. A mismanaged threat is defined as a threat that is linked to or induces flightcrew

error.

b. Errors.

1 The rest of this appendix will talk about TEM exclusivel y in the aircraft piloting environment so as to provide

specific examples and situations. However, it is important to remember that the TEM framework can be applied to

other high-risk jobs. To do so requires task and situational analyses by subject matter experts to develop the

relevant threat and error taxonomies.

AC 120-90 4/27/06

Appendix 1

Page 2

Crew error is defined as ac tion or inaction that leads to a deviation from crew or

organizational intentions or expectations. Erro rs in the operational context tend to reduce the

margin of safety and increase th e probability of adverse events.

Broadly speaking, there are handling errors (flig ht controls, automation), procedural errors

(checklists, briefings, callouts) and communication errors (with AT C, ground, or pilot-to-pilot).

See the error management worksheet in the sa mple observation form, Appendix 3, for a more

complete list of errors.

Understanding how the error was managed is as important, if not more important, than

understanding the prevalence of differe nt types of error. It is of interest then if and when the

error was detected and by whom, as well as th e response(s) upon detec ting the error, and the

outcome of the error. As with threats, some e rrors are quickly detected and resolved, leading to

an inconsequential outcome, while others go un detected or are mismanaged. A mismanaged

error is defined as an error that is linked to or induces additional error or an undesired aircraft

state.

c. Undesired Aircraft States.

An UAS is defined as a position, condition, or atti tude of an aircraft that clearly reduces

safety margins and is a result of actions by the fli ghtcrew. It is a safety -compromising state that

results from ineffective error management. Examples include unstable approaches, lateral

deviations, firm landings, and proceeding towa rds wrong taxiway/runway (more examples are

noted on the observation form in Appendix 3). Events such as equipment malfunctions or ATC

command errors can also place the aircraft in a compromised position, but these would be

considered threats.

As with errors, UASs can be managed effectively, returning the aircraft to safe flight, or the

crew action or inaction can induce an a dditional error, inci dent, or accident.

d. Threat and Error Countermeasures.

A description of a flight is not complete wit hout noting what the crew was doing to anticipate

threats and avoid errors, as well as managing thos e that occurred. The following crew behaviors

are considered threat and error countermeasures:

• Planning countermeasures—planning, prepara tion, briefings, contingency management—

are essential for managing antic ipated and unexpected threats

• Execution countermeasures—monitor/cross- check, taxiway/runway management,

workload, and automation management—are essential for error detection and error

response

• Review/modify countermeasures—evaluation of plans, inquiry—are essential for

managing the changing conditions of a flight

4/27/06 AC 120-90

Appendix 1

Page 3 In addition to crew behaviors, TEM countermeas ures also include equipment and procedural

countermeasures. Warning systems such as ground proximity warning systems (GPWS) and

weather alerts can be considered threat counte rmeasures, just as checklists and well-written

procedures provide the means for er ror avoidance and error detection.

In sum, the TEM model captures the dynamic ac tivity that is a fli ghtcrew planning and

executing a flight in real time a nd under real conditions. The use of the model is that it can be

applied proactively or reactivel y, at the individual, organizati onal, and/or systemic levels.

3. Practical Applications of the TEM Model.

a. TEM as a Training Tool.

TEM has been found to be an effective tool fo r human factors training programs at a number

of airlines throughout the world

2. TEM training emphasizes the va lue of threat anticipation and

management, error avoidance, and error detectio n and recovery. The model allows pilots to

analyze their own performance strengths and vulnerabilities.

TEM concepts can be trained effectively in the classroom in the absence of LOSA.

However, TEM training can be enhanced if an airline has also conducte d a LOSA. The LOSA

results can help shape the traini ng curriculum, and pilots can disc uss the findings during training.

Pilots are always interested in and respond well to data derived from their own operation.

It is important to clarify th at TEM is not crew resource ma nagement (CRM) and should not

be considered a replacement for it. TEM and CRM refer to overlapping but not equivalent activities. CRM refers specifica lly to activities conduc ted by the crew to optimize performance.

These activities include threat and error counter measures such as brie fing, contingency planning,

and monitor/cross-checking, but they also include higher-order concepts such as leadership and

establishing open communication in the co ckpit. Similarly, TEM includes crew

countermeasures, but it also encompasses equipment, procedural, and regulatory

countermeasures.

As a training tool, TEM can help indivi duals clarify their performance needs and

vulnerabilities from a different perspective. He nce, threat and error management concepts could

be introduced and explored as one component of CRM training.

b. TEM as a Reporting Tool for Incidents.

Reporting forms structured to the TEM framework in struct the pilots to describe the event at

the level of threats and errors. The TEM format prompts pilots to report information about the

threats that were present, the errors they ma y have made, how well the event was managed, and

how the event may have been avoided or handled better. Preliminary work has shown that even

pilots who have not had training in the TEM model are able to complete the reporting form, a

fact that speaks to the intuitive nature of the TEM framework.

2 The International Civil Aviation Organization (ICAO) ha s adopted the TEM model in its Human Factors Training

Manual (ICAO Document 9683, 2002)

AC 120-90 4/27/06

Appendix 1

Page 4

In the ASAP environment, TEM can “go inside the pilot’s head” in a way that LOSA as an

objective observational tool cannot do and most assuredly does not want to do. With ASAP,

pilots can report personal or histor ical factors that contributed to the event – information that is

not privy to an observer. With LOSA, the benefit is that observers may detect threats and errors

that the crews themselves do not detect. This is one example of how LOSA and ASAP data can

complement each other at the system level.

c. TEM as a Systematic Observation Tool.

The TEM model was first conceived in conjuncti on with the development of LOSA; hence,

its original application was as an observation tool. As of the public ation of this advisory circular

(AC), feasibility studies are cu rrently underway to e xplore the transfer of the methodology to

airline flight dispatch and air traffic control (ATC). In 2002, one major airline began exploring

an adapted version of LOSA called Dispatch Op erations Safety Audit (DOSA). Early results

demonstrated that such a transfer of met hodology is possible and could ultimately provide a

360-degree perspective on the inte raction between pilots and disp atchers. In addition, ICAO has

instituted a formal group of ATC subject matter experts from across the world to develop the

Normal Operations Safety Survey (NOSS), a fo rmal protocol to observe normal operations in

ATC, based on the TEM model and LOSA methodology.

d. TEM as a Reactive Analysis Tool for Accidents and Incidents.

TEM can be used as an analysis tool to understa nd rare events, such as accidents and serious

incidents. For example, the International Air Transport Association (IATA) Safety Committee has adopted the TEM model as an analysis framew ork for its incident review meetings, based on

its ease of use and utility of the extracted data.

e. TEM as a Proactive Analysis Tool.

When TEM is used as the framework for safety data collection, a wealth of information can

be extracted. An airline can use the data to under stand patterns at the or ganizational level. The

data can also be collected across the indus try and analyzed for systemic trends.

An analysis based on TEM can:

• Quantify those aspects of the working environment that can pose a problem for the

efficiency or safety of the operation (threat prevalence)

• Quantify the management of t hose threats as either effec tive or ineffective (threat

management)

4/27/06 AC 120-90

Appendix 1

Page 5 (and 6) • Recognize high rates of threat prevalence and mismanagement as systemic vulnerabilities

• Codify and quantify the errors th at crews commit (error prevalence)

• Codify and quantify the error management process from diagnosis to response and

outcome (error management)

• Recognize high rates of error prevalence and error mismanagement as systemic flaws in

procedures, policies, training, aircraft de sign, and or interage ncy coordination, and

• Locate strengths as well as vulnera bilities in organi zational safeguards

4. Conclusion.

The TEM model is intuitive, practical, and versat ile. More and more airlines are realizing the use

of TEM, as exemplified in the following quote from an airline manager:

“Since our LOSA, we have introduced a 1-day TEM course for our pilots. The overwhelming response has been positive, and there’s a recognition among pilots

that in TEM, the academic community is using language we understand and feel.

Additionally we are using TEM as a debr iefing tool for training events. Once

again this makes sense to the pilots, a nd the common reaction is one of dawning;

penny-dropping; eureka!”

4/27/06 AC 120-90

Appendix 2

Page 1 APPENDIX 2. TRAINING AND STANDARDIZING LOSA OBSERVERS

This appendix details the traini ng and standardizing of Line Oper ations Safety Audits (LOSA)

observers. Some of this material appears thr oughout the advisory circul ar (AC); this appendix

draws that information together and provides more in-depth information.

1. Observer Training.

Observer training typically requires 2 to 3 da ys of classroom training with a followup session

after one or two line observations. There are five topics that need to be covered in observer

training:

• LOSA rationale and etiquette

• Company policies and procedures

• Observation form

• Threat and Error Manage ment (TEM) concepts

• Narratives

a. LOSA Rationale and Etiquette.

Observers will likely have a rudimentary unde rstanding of LOSA when selected for the

project; however, they will need to fully understa nd the safety rationale for conducting a LOSA

at their airline. A “big picture” perspective will help observers understand the “why” of LOSA

and will underscore the importance of their role in the LOSA process. Also, the observers will

be ambassadors for LOSA while observing on the line, and it is important that they be able to

explain the process fully, to answ er any questions that the line pilots may have, and allay any

fears or concerns.

Specifically, the observers need to understand the safety rationale for normal operations

monitoring—a discussion of proactive vs. reac tive safety strategies is recommended. The

observers also need to know how the data coll ected from the LOSA will be used to understand

strengths and weaknesses in th e operations. An overview of the whole process from

observations to data cleaning a nd analysis, to the diagnostic report and the development of

targets for enhancement is recommended.

Of course, the observers will also need to know the “how” of LOSA, specifically the

etiquette associated with being a LOSA observer. An observer n eeds to learn how to approach a

crew, how to ask permission to observe a flight, ho w to walk away so that the crew can discuss

it, and to accept without questi on any crew’s decision to deny access to the cockpit. The

observer should also carry a copy of the letter of endorsement jointly signed by management and

pilots’ association to show any interested crew.

On the jumpseat, the observer’s behavior is best summarized as “a fly on the wall.” The

observer needs to be unobtrusive, yet responsiv e to any queries the crew may have. The

observer does not complete the observation form on the jumpseat—this would be a disconcerting

distraction to the crew. Instea d, the observer is encouraged to carry a small pocket notebook or

AC 120-90 4/27/06

Appendix 2

Page 2 legal pad to jot down minimalist notes, just enough to jolt the memory when outside the cockpit

and completing the full observation form.

LOSA observers should be trained to accept their role as observers, not evaluators—they are

not check airmen. LOSA observers will observe erro rs and undesired aircraft states as part of

their observations; however, they shoul d only interrupt if they perceive the safety of the flight to

be seriously and immediately endang ered. A helpful rule of thumb is to ask observers to think of

themselves as a guest riding in the jumpseat of a nother airline. This seems to help distance the

observer from the flightcrew while still being able to politely point out safety concerns if they

come about.

The LOSA observer is the only person in the whole LOSA process w ho has access to crew

identities. It is essential, th erefore, that observers are reminde d throughout the training of their

responsibilities in this regard. Confidentiality is paramount a nd observed crew behavior should

not be discussed with anyone—not even other obse rvers. Experience has shown that at the end

of a flight, crews will often ask the observer to “debrief” their performance. In these

circumstances, it is essential that the observer politely decline the inv itation. This emphasizes

the concept that the observer is not there to evaluate the crew, mere ly to record events.

LOSA observers should act in an unobtrusive and consistent manner so that line pilots have a

similarly positive experience of LOSA, which in turn will favorably affect their receptivity to the

final results and outcomes.

b. Company Policies and Procedures.

Observers need to be current with company policies and procedures so as to observe

procedural adherence on the line a nd detect any deviations. Selec ting active line pilots from the

airline is one way to ensure this. Spending so me time in the classroom reviewing procedures

across fleets allows all observers to get “up to speed” on the fleets they will be observing. All

observers should also be encouraged to review the manuals as homework.

c. Observation Form.

This subsection refers to the mechanics of correctly completing and submitting a LOSA

observation form. During the training, the observe rs work to develop the needed competencies

as defined by the observation form. Hence, obser vers should see the observation form as soon as

possible after the training begins so that they have a clear sense of what is expected of them.

With the observation form in hand, the observers can be led through the various sections, and

then practice using case st udies (see next subsection).

If the observation form is software-based, tim e should be spent ensuring that all observers

have the necessary computer skills to open the form, enter and edit data, and submit the

observation. Mastering these skills in the classr oom will avoid potential loss of data later in the

LOSA due to computer error. Observers should have the name of a c ontact person in case of

computer problems.

4/27/06 AC 120-90

Appendix 2

Page 3 d. TEM Concepts.

If the airline is using TEM as the basis of its LOSA, observers need to receive training in the

framework. Specifically, observers need to be able to define, distinguish, and identify threats,

errors, and undesired aircraft states . This is best achieved with a mi xture of lecture, case studies,

and review. Lecture material should include multip le examples of each type of threat, error, and

undesired aircraft state, and case studies can take the form of scripted vignettes and/or actual

accident and incident report excerpts. The dist inction between threats, errors, and undesired

aircraft states becomes clear with practice and ob servers are usually able to correctly distinguish

examples of all three categories in 2 days or fewer of classr oom involvement. Some training

examples are provided in Appendix 5.

e. Narratives.

Observers need training in writ ing the flight narrative. If they understand the concepts

underlying the observation form as well as the diagnostic rationale for conducting a LOSA, the

observers will realize that a good- quality narrative is imperative. The observation form should

contain several prompts to help the observer provide sufficient detail and observers should be

encouraged to “overwrite” the flig ht rather than provide too little detail. In particular, observers

need to record events that happened, such as th reats, errors, or undesire d aircraft states, the

context in which they happened, and the crew’s response and management of the event.

Observers are selected because they are experts at understandi ng flight operations and this

expertise is best expressed in detailed narratives. As long as the observer provides a detailed

narrative of the flight, an y coding oversights can be remedied la ter in the data-cleaning process.

f. Training Objectives.

In sum, at the end of LOSA observer tr aining, an observer should be able to:

(1) Knowledgeably and confidently explain th e rationale and process for conducting a

LOSA at the airline;

(2) Enact the LOSA observer etiquette in a professional and consistent manner;

(3) Demonstrate knowledge of company policies and procedures;

(4) Use the observation form accurately and comprehensively;

(5) Understand the theoretical fram ework of the observation form. If the tool is based on

TEM concepts, the observer should be able to define, distinguish, and identify threats, errors, and

undesired aircraft states; and

(6) Write detailed and comprehensive flight narratives from which others will be able to

understand the full context of th e flight and related events.

AC 120-90 4/27/06

Appendix 2

Page 4 Members of the LOSA steering committee may wa nt to attend LOSA ob server training at

another airline or attend an i ndustry-sponsored LOSA conference for further guidance before

designing the training.

2. Observer Standardization.

Standardization refers to the need to be sure that fli ght details are recorded in a systematic and

consistent fashion. In LOSA, standardization is a multi-step process that involves standardizing

the observers and conducting followup data-cle aning and coding of completed observations.

The first step in any observer standardization is good-quality training. To be sure observers

understand the concepts, group discussions are enc ouraged. Focusing on the finer points of the

model and the observation form, these discussion s help calibrate the observers to a common

standard. A test can be administered at the end of the training to be sure that all observers have

grasped the necessary k nowledge and can demonstrate the required competencies as specified by

the training objectives. If observers complete this test satisfactorily, they can be released to the

line to complete one or two “trial” observations. The LOSA project coordina tor or trainer should sc hedule time with each obs erver to discuss their

trial observations. If the observe r does a good job, as evidenced par ticularly by the quality of the

narrative, the observa tions can be retained and used in the LOSA, and the observer can be sent

back out to complete their observations. If the quality of the narra tive is poor (e.g., lacking

sufficient detail), the trainer can work with the ob server to help draw out missing information. If

the observer has forgotten details and cannot r ecreate the flight, the observation should be

discarded, and the observer sent out to complete another trial observati on. It is the LOSA

coordinator’s decision to drop any observer from th e observation team if that person fails to meet

the required standard. For this reason, it can be a good idea to in itially recruit and train more

observers than needed, to allow for attr ition, illness, and scheduling conflicts.

a. Data Standardization. There are several supplementar y techniques that ensure good-

quality standardized data are used in a LOSA.

First, observers are not asked to evaluate perfor mance, but simply to observe it. From a data

standpoint, this is the distin ction between subjective judgmen t and objective observation. For

example, observers are asked to note threats without any subjective j udgment—if there is a

thunderstorm, record it; if there is an airc raft malfunction or ground maintenance problem,

record it. It is the same for errors and undesire d aircraft states. Observ ers also note the crew’s

response to the threats, errors, a nd undesired aircraft st ates and the outcome. The observer is not

assessing crew performance or providing a s ubjective evaluation—the ob server is telling the

story of the flight.

4/27/06 AC 120-90

Appendix 2

Page 5 (and 6) Second, the data analyst always checks obser vation forms submitted by observers against the

narrative. Observers are expert at describing a flight—they are not n ecessarily expert at

assigning codes to the vari ous threats, errors, etc., especially if it is their first time as a LOSA

observer. While training is given in the codes, nonetheless it is re alistic to accept that observers

won’t necessarily retain this info rmation perfectly. The narrative is the “fail- safe” in the system

in that it allows the analyst to read the events of the flight and match them to the observer’s

codes. Because the analyst is the expert with the codes, he or she can add any codes that were

missed and recode anything that might not be quit e correct. Hence, a good -quality narrative is

the ultimate key to standardized data. Observ ers provide a comprehens ive narrative, and the

analyst ensures consistent and accurate coding.

A third step in standardizing the LOSA data prio r to analysis involves verifying the data with

a team of local experts—airline personnel familiar with the operation of each fleet (possibly fleet

managers or representation of the airline’s opera tions, member of the steering committee, but not

any of the observers). The group’s task is to re view and verify the observations against current

manuals, policies, and procedures. For example, an observer might log a procedural error for

failure to make an approach callout when, in f act, there is no written pr ocedure in the airline’s

flight operations manual. The “error” would th en be deleted from the database. The data

verification group acts as a check on the analyst’s coding, ensuring events are corre ctly recorded

in line with each fleet’s procedur es and policies. It also build s ownership in the results and

dispels any later criticism th at the coding was not accurate.

In sum, there are several methods that ensure th at LOSA data are consistently and accurately

recorded:

• Observers are trained, calibrate d, tested, and recalibrated

• Objective observation, not subjective eval uation, is the basis and outcome of the

observation

• The narrative is stressed as key to high-quality data

• The data analyst applies consis tent coding to the observations

• The data verification group checks the anal yst’s coding against sp ecific procedures

Following the above steps will ensure reliability and validity of the data analyzed from LOSA

observations.

4/27/06 AC 120-90

Appendix 3

Page 1 APPENDIX 3. SAMPLE LOSA OBSERVATION FORM

AC 120-90 4/27/06

Appendix 3

Page 2 APPENDIX 3. SAMPLE LOSA OBSERVATION FORM (Continued)

4/27/06 AC 120-90

Appendix 3

Page 3 APPENDIX 3. SAMPLE LOSA OBSERVATION FORM (Continued)

AC 120-90 4/27/06

Appendix 3

Page 4

APPENDIX 3. SAMPLE LOSA OBSE RVATION FORM (Continued)

4/27/06 AC 120-90

Appendix 3

Page 5 APPENDIX 3. SAMPLE LOSA OBSERVATION FORM (Continued)

AC 120-90 4/27/06

Appendix 3

Page 6 APPENDIX 3. SAMPLE LOSA OBSERVATION FORM (Continued)

4/27/06 AC 120-90

Appendix 3

Page 7 (and 8)

APPENDIX 3. SAMPLE LOSA OBSERVATION FORM (Continued)

4/27/06 AC 120-90

Appendix 4

Page 1 (and 2) APPENDIX 4. SAMPLE LETTER TO THE LINE PILOTS

To: All XX Airline Pilots From: (Name) Senior Di rector, Flight Operations

(Name) Director, Fli ght Training and Standards

(Name) Director, Flight Safety and Quality Assurance (Name) Chairman, Pilots’ association Safety Committee Subject: Line Operations Safety Audit (LOSA)

Beginning mid-October and continuing for approxi mately five weeks, this airline will

conduct a Line Operations Safety Audit (LOSA) . LOSA observations are no-jeopardy events,

and all data are confidential and de-identified. LOSA data go directly to the XX Research

Program for data entry and analysis.

We will use our own active pilots to conduct coc kpit jumpseat observations. Be assured that

these observations are not check rides. Although some LOSA obs ervers may be check airmen,

they are not there to critique your performance - th eir mission is to be an unobtrusive observer

and to fill out data collection forms after the flight is completed.

The ultimate customer of the audit is you, the line pilot. The audit should help us identify

problem areas so that we can correct them and make your job easier. Did you ever see a

procedure that could be done better , but didn’t feel like you had a wa y to feed that idea into the

system for possible change? Are so me procedures better than others as far as helping avoid, trap

and mitigate errors? LOSA should help us iden tify the strengths and weaknesses of our crew

procedures, and with that information, manageme nt is committed to making necessary changes

to continually improve the way that we do business.

In short, we’re doing a LOSA so that we can improve the system to better support you. After

the audit is completed, we’re committed to tel ling you how it went, and how we plan to make

improvements.

Please extend your usual professional courtesi es to the LOSA observation team, and thank

you for your unfailing cooperation. Sincerely, (Name), Senior Director, Flight Operations (Name), Director, Flight Training and Standards

(Name), Director, Flight Safety and Quality Assurance (Name), Chairman, Pilots’ association Safety Committee

4/27/06 AC 120-90

Appendix 5

Page 1 (and 2)

APPENDIX 5. SAMPLE VIGNETTES FOR OBSERVER TRAINING

Case Study One – Read the following and list all threats, errors, and undesired aircraft states as

well as the flightcrew response and outcome.

Predeparture / Taxi-out – The Captain requested an extra 5000 pounds of fuel to be loaded. After ground

confirmed the fueling, the First Officer alerted the Captain that they were 2000 pounds off. The Captain radioed

back to ground and the missing 2000 pounds was loaded.

Solution – One threat, no errors or undesired aircraft states

Threat #1 – Ground Handling Threat – The ground crew failed to load the all 5000 lbs of requested fuel –

The flightcrew response was the First Officer det ecting and correcting the shortage. The outcome was

inconsequential.

Case Study Two – Read the following and list all threats, errors, and undesired aircraft states as

well as the flightcrew response and outcome.

Descent / Approach - While descending through 18000 feet, the Firs t Officer performed the entire descent checklist

from memory, which is against standard operating procedures. The Captain noticed the First Officer doing it but

chose to ignore it. In the end, everything was set correctly.

Solution – No threats, one error and no undesired aircraft states

Error #1 – Checklist Error – The FO performed th e descent checklist from memory. The flightcrew

response was the captain detecting the error but failing to correct (ignored). The outcome was inconsequential

Case Study Three – Read the following and list all threats, errors, and undesired aircraft states

as the flightcrew response and outcome.

Descent / Approach / Land – During a 30-degree bank on a visual approach, the Captain unwillingly allowed the

aircraft to get 15 knots below minimum maneuvering speed. While it should have been detected sooner, the low

speed was eventually pointed out by the First Officer. The Captain said thanks and immediately increased the

speed.

Solution – No threats, one error and one undesired aircraft state

Error #1 – Aircraft Handling Error – The Captain allowed the speed to decay during a bank turn. No one

detected the decay before it dropped below minimum maneuvering speed. Therefore, the error was undetected and linked to an undesired aircraft state.

Undesired Aircraft State #1 - As soon as the speed went below minimum maneuvering speed, the aircraft was in an undesired aircraft state th at was detected by the First Officer and corrected by the Captain. The

final undesired aircraft state outcome was inconsequential.