Sunday, April 7, 2019

What happened on ET302?

The preliminary report for ET302 reveals many details, but not enough to know exactly every and all aspects. The following is a look at what can be learned.

(excerpts from the preliminary report narrative are indented and italicized)

At 05:37:34, ATC issued take off clearance to ET-302 and to contact radar on 119.7 MHz. Takeoff roll began from runway 07R at approximately 05:38, with a flap setting of 5 degrees and a stabilizer setting of 5.6 units.

The takeoff weight was 71,896 kg.
Takeoff Center of Gravity (CG) was 23.12.
MET REPORT HAAB 100530Z
Wind - runway 25: 070 degrees 10kt, runway 07: 050 degrees, 10kt Visibility: 10km; few cloud 750 m; temperature: 170C; dew point: 090C QNH: 1029 hPa; QFE: 777 hPa
The elevation of the airport is 2333.5 m. The aircraft took off on runway 07R which was 3800 m long and 45 m wide.  The warm temperatures raised the density altitude. The headwind increased the takeoff runway to an equivalent 3900m length. The takeoff configuration was right at the limit for the runway.


The takeoff roll appeared normal, including normal values of left and right angle-of-attack (AOA). During takeoff roll, the engines stabilized at about 94% N1, which matched the N1 Reference recorded on the DFDR. From this point for most of the flight, the N1 Reference remained about 94% and the throttles did not move. According to the CVR data and the control column forces recorded in DFDR, captain was the pilot flying.

The takeoff rotation pitch attitude reached about 18 degrees. Liftoff speed was 155-160 knots. Normally the takeoff would proceed to 15 deg. Pitch attitude almost immediately was reduced. Climb speed to 1000 feet increased to over 200 knots with flaps constant at 5 degrees.




The power setting never changing throughout the flight led to much higher airspeed than would be expected, including overspeed, all below 10,000 AGL. Basic airmanship involves altitude, speed, and heading. The lack of speed control is very troubling and would seem to have fallen victim to inattention due to pilot distraction/workload.  The pilot not flying (pilot monitoring) should have been alerting the pilot flying repeatedly over the excessive speed, yet this report makes no mention of this advice, and even when overspeed.

The New York Times published data from flightradar24.com that they said showed evidence of MCAS actions.


I pointed out, right away, that flaps should be down at that point and that MCAS would not be active at that point. That was the case.


The data provided in the preliminary report does not show Vertical Speed.  I plotted Vertical Speed Vs. Altitude, and it appears the airplane did not climb out smoothly. The guidance on a normal takeoff is to climb at about V2+15 at a stead pitch attitude of about 15 degrees, and surely to maintain positive rate of climb.



At 05:38:44, shortly after liftoff, the left and right recorded AOA values deviated. Left AOA decreased to 11.1° then increased to 35.7° while value of right AOA indicated 14.94°. Then after, the left AOA value reached 74.5° in 3⁄4 seconds while the right AOA reached a maximum value of 15.3°.


Clearly the AOA vane has suffered a failure. It is plainly obvious. Either the vane (the outside assembly) just fell off, or a bird strike took it off. If a bird strike, the CVR area mic might pick up the plunk.

Bjorn Fehrm suggested that if the vane broke off, the remainder would  act like a pendulum, with the remaining off-center weight.  Others have come to the same conclusion. However as pointed out in the comments, the angle for +75 degrees would be pointing down.


If the vane completely sheared off and the shaft were still rotating, then the internal counter weight would likely be the forcing factor, and this is in the right orientation.

The following is a similar vane.  The need for a counterweight is the same with any vane.






There have been numerous occasions for AOA vanes to fail.




The AOA step change to 75 degrees in about 2 seconds was fully accepted by all the systems onboard as valid data, and this set off the numerous flight deck effects related to stall warning and also the offset to airspeed and altitude from using erroneous AOA compensation.

Under what circumstance is AOA jumping about 60 degrees a reasonable response? Why is there no fault triggered by this obvious malfunction? How do you get to 75 degrees AOA in any flight profile anyway? Is this a reasonable value in any case?

For 737-900, the highest alpha trip point is 23.5 degrees. In the approach to stall, once you have gotten to the trip point, you trigger stall warning. There is nothing to do at higher alpha (maybe MCAS comes in a bit higher?). The pilot is expected to recover from the approach to stall, or a full stall.


Boeing lists all the flight deck effects possible with erroneous AOA. Notably, the stick shaker goes of incessantly, the airspeed and altitude are offset from valid values, and control column pressure aft (pulling) is increased significantly.

The influence of AOA correction to air data is modest at low speeds, but quite pronounced at high speeds. At the end of the flight, AOA swung around, and this caused airspeed and altitude to modulate, revealing the influence of AOA corrections.



At this time, the left stick shaker activated and remained active until near the end of the recording. Also, the airspeed, altitude and flight director pitch bar values from the left side noted deviating from the corresponding right side values. The left side values were lower than the right side values until near the end of the recording.

The captain drops the nose to less than eight degrees and airspeed increases steadily – all concurrent with the stick shaker activation. Is it possible the captain was unsure of being too slow and instinctively dropped the nose in response? Was the elevator feel shift module activation, at least doubling the aft feel forces, a factor in lowering the nose?

At 05:38:46 and about 200 ft radio altitude, the Master Caution parameter changed state. The First Officer called out Master Caution Anti-Ice on CVR. Four seconds later, the recorded Left AOA Heat parameter changed state.

The AOA vane is heated to prevent ice build-up. If it departed the aircraft, the electrical heating circuit would fail, and this was encountered.

Master Caution on the Glare Shield alerts the flight crew to refer to other (overhead) panel for further information, where in this case the L APHA VANE fault is annunciated.



At 05:38:58 and about 400 ft radio altitude, the flight director pitch mode changed to VNAV SPEED and Captain called out “Command” (standard call out for autopilot engagement) and an autopilot warning is recorded.

Autopilot apparently would not stay engaged, and the warning would alert the crew to it being disconnected. I am not sure what was keeping the autopilot from engaging, but it was a listed effect from AOA erroneous reading.

At 05:39:00, Captain called out “Command”.
At 05:39:01 and about 630 ft radio altitude, a second autopilot warning is recorded.

A second attempt to engage the autopilot was not successful, again triggering a warning when it disconnected.

At 05:39:06, Between liftoff and 1000 ft above ground level (AGL), the pitch trim position moved between 4.9 and 5.9 units in response to manual electric trim inputs. At 1000 ft AGL, the pitch trim position was at 5.6 units.

Stabilizer trim is operating normally when the pilot was flying with flaps down.


At 05:39:22 and about 1,000 feet the left autopilot (AP) was engaged (it disengaged about 33 seconds later), the flaps were retracted, and the pitch trim position decreased to 4.6 units. 
Six seconds after the autopilot engagement, there were small amplitude roll oscillations accompanied by lateral acceleration, rudder oscillations and slight heading changes. These oscillations continued also after the autopilot was disengaged.

The source of the roll and rudder oscillations was not revealed, nor was wheel or rudder control inputs. What factor this has is not known.

Autopilot stab trim would applied be to maintain an “in trim” condition as the airplane accelerated and flaps were raised. This is evident by three trim nose down commands as the airplane speed increases.



At 05:39:45, Captain requested flaps up and First-Officer acknowledged. One second later, flap handle moved from 5 to 0 degrees and flaps retraction began.
At 05:39:55, Autopilot disengaged,

It would appear Speed Trim System applies a nose up trim command to reflect the airplane accelerating.



At 05:40:00 shortly after the autopilot disengaged, the FDR recorded an automatic aircraft nose down (AND) activated for 9.0 seconds and pitch trim moved from 4.60 to 2.1 units. The climb was arrested, and the aircraft descended slightly.

With the flaps up and AOA erroneously high, it appears MCAS applied a significant 2.5 unit nose down command.




At 05:40:03 Ground Proximity Warning System (GPWS) “DON’T SINK” alerts occurred.
The column moved aft and a positive climb was re-established during the automatic AND motion.

The airplane lost altitude in the first MCAS command.

At 05:40:12, approximately three seconds after AND stabilizer motion ends, electric trim (from pilot activated switches on the yoke) in the Aircraft nose up (ANU) direction is recorded on the DFDR and the stabilizer moved in the ANU direction to 2.4 units. The Aircraft pitch attitude remained about the same as the back pressure on the column increased.

The captain appears to have trimmed slightly nose up in response. It is not clear why the captain did not apply more nose up trim to reduce column forces.




At 05:40:20, approximately five seconds after the end of the ANU stabilizer motion, a second instance of automatic AND stabilizer trim occurred and the stabilizer moved down and reached 0.4 units.

The manual trim command reset the MCAS logic, arming it for another command five seconds later. The second MCAS command takes the stabilizer to near the nose down limit.


The Boeing 737 flaps up stabilizer trim is limited in range to no less than about 4 units under pilot electric trim command. The pilot can use electric trim to as low as 0.05 units with flaps down. Boeing has concluded that there is NO REASON for trim to go less than about 4 units with flaps up. Apologies, I don’t have the 737 MAX values, but they should be similar. These limits are enforced by limit switches on the jackscrew.

The autopilot has the authority to operate down to 0.05 units without any limit switch. The autopilot normally only provides incremental adjustments to maintain an in-trim or stable trim, and therefore would naturally stay very close to the trim position.




EASA took note of the fact that it is possible more nose down trim might be needed under certain corner conditions than what is available to the pilot using electric trim and accepted the configuration anyway. Boeing provides manual trim as a means to achieve trim outside of the range allowed for under pilot electric trim control, or for the autopilot command mode.



When MCAS triggered, the stab trim was at about 4.60 units and it was ultimately driven down to 0.4 units. Any trim to less than about 4 units is going to guarantee a severe mis-trim situation. MCAS authority is not limited to prevent severe mis-trim in its normal mode of operation.  While MCAS provides a pitch down moment to offset aero pitch up, it leaves the airplane severely mis-trimmed nose down. There is nothing provided to help the pilot recover to an in-trim position, to restore the stab into a valid range for flaps up – incredibly it just dumps the stab and leaves it to the pilot to get it back.

From 05:40:23 to 05:40:31, three Ground Proximity Warning System (GPWS) “DON’T SINK” alerts occurred.

Pilots are ingrained to take immediate action when GPWS alerts. In this case, the imperative is to restore level or climbing flight.

At 05:40:27, the Captain advised the First-Officer to trim up with him.
At 05:40:28 Manual electric trim in the ANU direction was recorded and the stabilizer reversed moving in the ANU direction and then the trim reached 2.3 units.




I cannot understand the need for the captain to ask for help to trim up, he has a yoke switch. Maybe he was cradling the column and did not have his thumb in position?

In any case, what was most troubling is that they stopped trimming long before the column forces were relaxed.  Leaving the stab out of trim would create many issues for the crew. The advice from Boeing and the FAA should have been far more insistent on this point. Not “can be used” but rather: Electric trim must be used to neutralize control column pitch forces before cutout!


At 05:40:35, the First-Officer called out “stab trim cut-out” two times. Captain agreed and First- Officer confirmed stab trim cut-out.
At 05:40:41, approximately five seconds after the end of the ANU stabilizer motion, a third instance of AND automatic trim command occurred without any corresponding motion of the stabilizer, which is consistent with the stabilizer trim cutout switches were in the ‘’cutout’’ position

It is apparent that the cutout has been switched in. MCAS sits idle after this command until the pilot applies manual electric trim near the end of the flight.


At 05:40:44, the Captain called out three times “Pull-up” and the First-Officer acknowledged.

Having left the severe mistrim in place, the captain is having difficulty keep the nose up. The elevator feel shift is adding considerable extra effort to hold the column back.

What also is clear, MCAS nose-down trim commands were not stopped, especially the second command, by the pilot pulling the column considerably aft – the aft column cutout was disabled.

The aft column cutout switch is designed to stop a nose-down stabilizer runaway. Human factors favor this feature, as the pilot will naturally pull back to stop the nose-down pitch moment. By stopping the stabilizer runaway, the pilot will have less mistrim to account for, and has opportunity to collect themselves and hit the cutout.

While the first MCAS command was largely offset by pilot elevator command, the second MCAS command was overwhelming.  In this case, the captain trimmed nose up, but stopping short of a reducing column forces.


From 05:40:42 to 05:43:11 (about two and a half minutes), the stabilizer position gradually moved in the AND direction from 2.3 units to 2.1 units. During this time, aft force was applied to the control columns which remained aft of neutral position.
The data indicates that aft force was applied to both columns simultaneously several times throughout the remainder of the recording.
At 05:41:30, the Captain requested the First-Officer to pitch up with him and the First-Officer acknowledged.
At 05:41:46, the Captain asked the First-Officer if the trim is functional. The First-Officer has replied that the trim was not working and asked if he could try it manually. The Captain told him to try. At 05:41:54, the First-Officer replied that it is not working.

The captain is pulling the column aft stabilize the pitch attitude. He desperately wants to trim the stabilizer nose up to offload the column and restore trimmed flight. By hitting the cutout switches, the flight crew must resort to manual trim wheel.

The cutout switch function was changed with the 737MAX from all prior 737 models. The legacy switch combination was one switch to cutout electric trim altogether, the other to cutout the autopilot trim commands. MCAS and Speed Trim System are both commands from the “autopilot”.  With the legacy switch configuration, the flight crew can disable the autopilot commands and retain electric trim.



With the 737MAX, the flight crew lose both electric trim and autopilot trim with the cutout switch.

 

The Flight Standardization Board (FSB) took notice of the nomenclature change of the cutout switches but did not make any mention of the difference in responding to autopilot stab trim runaway. On the 737NG, the flight crew retain electric trim; on the 737MAX the flight crew must use manual trim. It turns out, this is a significant difference in pilot workload and pilot capacity to fly the airplane.

 
Manual trim wheel is a cable driven function that mechanically turns the stabilizer jackscrew through a clutch and gearbox. The clutch ensure that the manual trim wheel has control in priority over the electric trim. The pilot turning the trim wheel must provide enough force to turn the jackscrew – there is no power-assist.



There is an issue with opposing loads on the stabilizer jack screw based on stabilizer angle of incidence and elevator deflection.  Notably, significant elevator deflection can become greater than what a pilot (or both) can apply at the wheel.


The stabilizer moves from 2.3 units to 2.1 units during this time. That would be likely if the pilot, unable to turn the wheel nose up, experiments to see if it is jammed by trimming nose down. With forces favoring nose down, the wheel effectively ratchets nose down.


The flight ops manual for the 737-100/200 makes mention of this phenomena.


Centauraus, on pprune.org, provided an interesting post:

After some research in my aviation library I think I have found the answer to my original questions of the meaning of the Boeing term "relieving aerodynamic loads." 

Extract from the Boeing 737-200 Pilot Training Manual February 1982 page 04.80.31. Edited for brevity. Runaway and Manual Stabiliser - Recovery from Severe Out-of-Trim 

"In an extreme nose-up out-of-trim condition, requiring almost full forward control column, decelerate, extend the flaps and/or reduce thrust to a minimum practical setting consistent with flight conditions until elevator control is established. Do not decrease airspeed below the minimum maneuvering speed for the flap configuration. A bank of 30 degrees or more will relieve some force on the control column. This, combined with flap extension and reduced speed should permit easier manual trimming. 

If other methods fail to relieve the elevator load and control column force, use the "roller coaster" technique. If nose-up trim is required, raise the nose well above the horizon with elevator control. Then slowly relax the control column pressure and manually trim nose-up. Allow the nose to drop below the horizon while trimming. Repeat this sequence until the airplane is trim. 

If nose-down trim is required, slowing down and extending the flaps will account for a large degree of nose-up pitch. If this does not allow manual trimming then the reverse "roller coaster" can be performed to permit manual trimming." (I read somewhere it was called the Yo Yo manoeuvre) 

Boeing "Airliner" magazine published in May 1961 discussed the above subject as it applied to the Boeing 707 by stating: "To trim the stabilizer manually while holding a high stick force on control column. As the airplane changes altitude, crank in the desired trim change. Correct airplane attitude after a few seconds with elevators. Relax stick force again and crank in more trim. Repeat this procedure as necessary until proper 'trim' position of stabilizer is established."

The reason is apparent by looking at the forces. With the control column in neutral, the forces all favor nose up trim.



The only advice in the training manual, the quick reference handbook, and the ops manual is that you can try as hard as you want without breaking anything, and you may need to “relieve aerodynamic forces”.


The left indicated airspeed increased from approximately 305 kt to approximately 340 kt (VMO). The right indicated airspeed was approximately 20-25 kt higher than the left.
At 05:41:20, the right overspeed clacker was recorded on CVR. It remained active until the end of the recording.
At 05:41:32, the left overspeed warning activated and was active intermittently until the end of the recording.

The horizontal stabilizer forces in a severe nose-down mistrim are increasing helpful with higher airspeed (see 737-100 advice above), but significant elevator nose up commands remain overwhelming.

At 05:42:10, the Captain asked and the First-Officer requested radar control a vector to return and ATC approved.
At 05:42:30, ATC instructed ET-302 to turn right heading 260 degrees and the First-Officer acknowledged.
At 05:42:43, the selected heading was changed to 262 degrees.

The captain begins a slow turn while still applying significant aft column command. The roll commands are choppy and not smooth. Bank angle extends to 30 degrees at the point when the final MCAS commands will come alive, making recovery more complicated by having to level the wings too.


At 05:42:51, the First-Officer mentioned Master Caution Anti-Ice. The Master Caution is recorded on DFDR.
At 05:42:54, both pilots called out “left alpha vane”.

For some reason the left alpha vane fault is retriggered.

At 05:43:04, the Captain asked the First Officer to pitch up together and said that pitch is not enough.
At 05:43:11, about 32 seconds before the end of the recording, at approximately 13,4002 ft, two momentary manual electric trim inputs are recorded in the ANU direction. The stabilizer moved in the ANU direction from 2.1 units to 2.3 units.

By virtue of the command and motion, it appears that the flight crew released the cutout, reactivated stabilizer trim. I would guess it was because they could not move the stab manually, and they could come up with no other option.

The airplane is flying at or above Vmo at this point. The stab trim manual inputs may have caused a bit of a shudder in the flight deck, which may have created some reluctance to trim aggressively. In any case, it is not clear what prevented the crew from continuing to trim airplane nose up to reduce column forces.

Bjorn Fehrm thoughts on this point.



At 05:43:20, approximately five seconds after the last manual electric trim input, an AND automatic trim command occurred, and the stabilizer moved in the AND direction from 2.3 to 1.0 unit in approximately 5 seconds. The aircraft began pitching nose down. Additional simultaneous aft column force was applied, but the nose down pitch continues, eventually reaching 40° nose down. The stabilizer position varied between 1.1 and 0.8 units for the remainder of the recording.


  

Activating MCAS at or above Vmo makes no sense at all. The highest speed one might expect MCAS is about 270 knots (highest stall speed of about 190 knots * 1.41 or 2g). (data shown for 737-900)



Applying massive stabilizer trim commands at this high a speed introduced an uncontrollable pitch over, in spite of full elevator command in opposition.

A rapid pitch over causes a large negative load in the flight deck due to the moment arm, and in combination to the negative g from the path change. Column forces initially are reduced, possibly because the flight crew were thrown upwards and lost their footing. Subsequently the column is pulled back massively, but it has no restoring effect. There is no evidence of any attempt to trim the stabilizer nose up using any form (electric or wheel) in the pitch over. It appears the pilots fixate solely on pulling the column back with all their strength.



The AOA vane, stubby, as suspected by the failure discussed earlier, acts like a pendulum, tracking the normal acceleration.


The airplane crashed and 149 passengers with 8 crew members perished.  Every life lost is a tragedy.

The greatest regret with aviation safety is to learn a lesson twice. I have faith that each issue will be addressed and suitable corrective action will be mandated. If anything, it is a big wake up call for the industry to not discard long held safety features without considerable and open analysis.

Human Factors is a science to understand what prevented a pilot from taking the expected action. It is easy to pontificate from a distance. Even the flight crews lost in these accidents would agree what they should have done in hindsight. The challenge is understanding why they made the choices they made and what would have directed them to a better outcome.


Peter Lemme

peter @ satcom.guru
Follow me on twitter: @Satcom_Guru
Copyright 2019 satcom.guru All Rights Reserved

RSS: http://satcomguru.blogspot.com/feeds/posts/default?alt=rss

Peter Lemme has been a leader in avionics engineering for 38 years. He offers independent consulting services largely focused on avionics and L, Ku, and Ka band satellite communications to aircraft. Peter chaired the SAE-ITC AEEC Ku/Ka-band satcom subcommittee for more than ten years, developing ARINC 791 and 792 characteristics, and continues as a member. He contributes to the Network Infrastructure and Interfaces (NIS) subcommittee developing Project Paper 848, standard for Media Independent Secure Offboard Network.

Peter was Boeing avionics supervisor for 767 and 747-400 data link recording, data link reporting, and satellite communications. He was an FAA designated engineering representative (DER) for ACARS, satellite communications, DFDAU, DFDR, ACMS and printers. Peter was lead engineer for Thrust Management System (757, 767, 747-400), also supervisor for satellite communications for 777, and was manager of terminal-area projects (GLS, MLS, enhanced vision).

An instrument-rated private pilot, single engine land and sea, Peter has enjoyed perspectives from both operating and designing airplanes. Hundreds of hours of flight test analysis and thousands of hours in simulators have given him an appreciation for the many aspects that drive aviation; whether tandem complexity, policy, human, or technical; and the difficulties and challenges to achieving success.

55 comments:

  1. Thanks Peter for the deep dive. Always hard to say for a current event but there seems so many issues stemming from the MAX about what we expect from pilots, automation/augmentation, airframe design and flight regulation to make this relevant for a long time to come.
    The human factors aspect of the incomplete yoke switch electric trim events seems critical as they play a role in Lionair as well. In retrospect the failure to fully trim using the yoke switches (prior to deactivating all electric trim or experiencing a fresh MCAS event) is hard to understand. Because it should have been possible and might had changed the outcome, it seems worthy of a lot of consideration to attempt an understanding of possible factors that led to it occur.
    It would also be great if the NG cutout switch logic was restored to the MAX (with MCAS under 'autopilot') but this seems unlikely to occur.

    ReplyDelete
  2. "Bjorn Fehrm suggested that if the vane broke off, the base would might act like a pendulum, with the remaining off-center weight. Others have come to the same conclusion." You obviously don't understand directions of rotation of AoA sensor that indicate stall. If there was vane broke off indication from "remaining off-center weight" will be stall negative - do you understand me, in other rotation direction. Mechanical fault of AoA sensor may be not present at all. Do you and Bjorn know what is Bernoulli's Principle is? Look carefully AoA configuration and what have above it. When plane take off abrupt change in air direction can cause Bernoulli's Effect between AoA and Pilot Tube wing with make air to glue them together. This is another faulty design of Boeing engineers.https://m.youtube.com/watch?v=P-xNXrELCmU

    ReplyDelete
    Replies
    1. I welcome corrections!
      You are absolutely right, I had the orientation the opposite and I apologize for that. However, the internal counterweight is the likely forcing function. With the vane gone, there is no aero force of consequence. However, the lost off-center mass (the vane) lets the internal counterweight dominate. It is installed in front of the nose of the vane. So 1 g would pull the nose down, in the positive direction. There is no doubt it acts like a pendulum when watching its motion in the final maneuver.
      I do not agree with your assertion that the airflow from the pitot tube is somehow a factor, as it would not move in the final maneuver.

      Delete
    2. You must ask question Peter Lemme "If "AoA vain" was broken in the beginning of flight, and counterweight act like pendulum WHY this was indicate only on the end of the flight, not from beginning, where suppose vain was broken?" Pendulum effect will take action immediately and will make AoA-l chaotic in all flight after breaking of vain. This pseudo "pendulum effect" was due to faulty Potentiomater and acceleration forces act on it in end. I told you, like old radio volume with faulty potentiometer - the positions are only two - maximum noise and everything else, as max noise is predominant. The AoA vain was not broken. I repeat, if it was, the counterweight will go in nose down position, and this will be indicate by positioning Potentiomater, not as 75 degree.

      Delete
    3. Pendulum effect did show itself right away, it rotated to +75 degrees. there is no potentiometer - it is a resolver interface.

      Delete
  3. I will answer you why there was no foreign object strike. Look this video. Do you see that AoA vain move on 360 degrees, not only to 180? There is no stopper. If an external object hits "AoA vain", it does not matter whether it will break it or not - the result would be ROTATION. Because the internal potentiometer, which takes into account the position of the AoA vain also has no brake and can rotate 360 degree too, probably the computer will detect random chaotic positions on the sensor, not stable one. These are my arguments. Of course, if the potentiometer breaks from the sharp rapid rotation it will probably stop functioning. The question is whether the lack of current is interpreted by the computer as 75 degree, -75 degree or 0?

    https://www.youtube.com/watch?v=4koccb8suOQ

    ReplyDelete
    Replies
    1. I believe there is a stop on the vane. There are dampers. It uses a resolver (SIN and COS). Your video is incorrect. The service done between JT043 and JT610 was misdirected to Air Data because the flight crew made no entry regarding constant stick shaker which would have been correlated as an AOA vane error along with the IAS and ALT disagree messages. No fault was found on the AOA because it was producing a valid value.

      Delete
    2. here is a fun video of a vane with stops on a surprising airframe.
      https://photos.app.goo.gl/kThzW12PjmQiBMdn9

      Delete
  4. Pendulum effect have pendulum period and pendulum frequency. In AoA device - because of short pendulum length - the period must be well under a second. For example, pendulum with 10 cm length will have period of 0.2 seconds, and frequency 5 Hz. It is no coincidence that this weight is made with a short center of mass of the pendulum length. The idea is to act quickly. Here we talk about over 2 second on ET302 graph. The mechanically AoA was ok, only potentiometer was broken.

    ReplyDelete
    Replies
    1. the dampers are still intact. this is not a weight on a string.
      Everything I have shows a resolver interface (SIN and COS). There is no single electrical fault that provides this error. Furthermore, the motion at the end is completely aligned with the normal acceleration.

      Delete
  5. Good in depth analysis. It is indeed likely the vane came off and could well had been a bird strike.
    What you can also see is that the trim speed from MCAS is faster than pilot input, real scary as recovering to normal in trim will take longer.

    In my opinion the change of the cutout switches was because the A/P cutout would disable the FCC clutch signal rendering MCAS inoperative.
    I'm not sure what the Stab Trim override switch would do in the MAX. but it appears to override the column cutout switches.
    If I read the functional description schematic correctly the B/U cutout disables column switch trimming, signals the FCC and drops relay R64 removing power to the trim motor. The PRI cutout disables B/U and disconnects the FCC from the A/P clutches.

    ReplyDelete
  6. Concerning the apparent failure of the pilots to trim ANU to fully take back the MCAS AND trim, could it be possible that in the situation directly before the last MCAS AND input the opposing forces at the jackscrew were not only too high to be overcome by the manual trim wheels, but *also* insurmountable by the electric motor?

    That might explain the very short and few manual-electric ANU trim inputs. After all, with the AC in full nose-down trim, full elevator up and the excessive speeds, I could imagine that such a scenario was never considered by the engineers when deciding on the maximum torque for the motor(s).

    >>At 05:43:11, about 32 seconds before the end of the recording, at approximately 13,4002 ft, two momentary manual electric trim inputs are recorded in the ANU direction. The stabilizer moved in the ANU direction from 2.1 units to 2.3 units. <<

    The stabilizer movement seems very small compared to what MCAS could do in 5 seconds in the opposite direction. Although there is no information for the duration of the two manual trim inputs, from the graph they appear to be on the order of a second each - that clearly does not match the amount of stabilizer movement per second that MCAS achieved.

    ReplyDelete
    Replies
    1. Not certain for MAX, but for NG the manual trim is 0.2 deg/sec, MCAS is 0.27 deg/sec.
      The momentary manual trim nose up did seem to work. The duration is impossible to be certain of by the chart, but the stab definitely responde. I don't think it was stalled out, and if so we would see the trim command but no motion.

      Delete
  7. HI. Good job!

    How is it that at 05:43:20 the aircraft's stab moved nose down when the pilots had cutout the stab trims? I'm only asking because at that speed, that trim movement was the fatal one...

    ReplyDelete
    Replies
    1. It appears that by 05:43:10 they had restored the stab trim - released the cutout - in a desperate attempt to trim airplane nose up. Having taken that action, why they did not proceed with trimming significantly is a mystery, and even worse, by manually trimming they woke up MCAS.

      Delete
    2. Thanks a lot. Please have a look at the FO trim up input (long one) that stops MCAS. First 3 s of that input moves the stab appr. 1 U up. Even at the lousy resolution of the plot you can see the difference. I copy/pasted all 4 events together:
      https://cimg1.ibsrv.net/gimg/pprune.org-vbulletin/111x126/et302_zoom2_6dcbf03e3eb4ec546da53dd5df67946211d6359c.jpg

      Or, for more precision: The FO input lasts 10 s and brings the stab up appr. 1.9 U. That is appr. 0.6 U / 3 s. The third blue arrow from the left (the left captain switch attempt to correct MCAS) does not show that much movement. Flaps up applies to both rightmost events.
      Is it possible that the authority of the left thumb switch was reduced?
      And: Why does PF ask FO to help with the trim (switch)?

      Delete
    3. the manual trim rate, flaps up, is 0.2 deg/sec, which seems to match your 10 second data point. How can you tell left switch from right switch. Electrically, they are the same. I cannot figure out why he asked for help, unless he was cradling the yoke to keep it pulled back and did not have his thumb nearby. The tabular data would be better to make any judgements, as the plots are not precise enough to pick out short times with any accuracy.

      Delete
    4. Agreed, the traces do not allow a real judgement. Just wondering, why he asks for help. Seems weird. And the entire rest looks like the cpt is frustrated with the trim switch. As if he feels it is not doing anything... Are you sure they are electrically connected? Or are they two separate digital inputs into the FCC?

      Delete
    5. the electric trim is all wired together. The FCC is wired independently. These go to the stab trim motor. The FCC gets the aft/fwd column cutout switches as an input and it decides how to handle it. The aft/fwd column cutout switches themselves only inhibit runaway from the manual electric trim circuit.

      Delete
    6. Thanks. Rules out any SW interference. I had a look at the Lionair FDR again. Looks like the Cpt there had difficulties keeping the thumb switche pressed. He makes longish counter inputs, but inside those inputs there are several blips where he seems to loose it for a moment. Stickshaker and pullforce combined?
      In the dive, there are also some short blips upward that do not result in significant stab movement.

      Delete
    7. It is unknown what the FDR records when the electric trim is activated and either the column cutout and/or one or both of the cutout switches is thrown.
      If the FDR does'n record commands when the column cutout is open then that could explain why no ANU command was recorded despite the electric trim was pushed to ANU. In that case the Stab Trim Override switch could have helped to restore electric trimming.

      Another piece of information, the stall warning system is activated when the Radio Altitude is >10ft.

      Delete
  8. I was astonished discovering that the flight ops manual for the 737-100/200 says "Recovery from severe out of trim : Accelerate [...] to an in-trim airspeed [...] to permit manuel trimming". And in the current training manual : "use an airspeed which results in an in-trim condition".

    I understand that at low speed and with a limited amount of AND untrim, increasing speed may reduce or cancel aft column required, due to static stability of the aircraft. But in this case, even at 500 kt IAS and still both pilots pulling as much as they can, the aircraft has a -10° AoA and minus 2 g ! No hope to reach any in-trim airspeed... and nowhere in the checklists it says that reducing speed may reduce loads and permits trimming.

    May this explain why the pilots let the airspeed increase at and above VMO ?

    ReplyDelete
    Replies
    1. based on the fact that power was never reduced from the takeoff, I would not assume there was any strategy to use airspeed to relieve forces. The main issue on trimming was not the stabilizer, but the elevator. Had they relaxed the elevator they would have been able to trim airplane nose up.

      Delete
  9. A question towards the stab trim runaway procedure, which seems to have been "reused" in an optimistic way as an MCAS safety measure:
    Does anybody know, by chance, how frequently a "traditional" stab trim runaway (from either autopilot, STS, or electrical failures) has been actually occurring in the operation of NG and older 737s? Respectively, how often AND trim runaway in low altitude has been successfully recovered by following the given stab trim runaway NNC (i.e. through manual, muscle-based trim)?

    ReplyDelete
    Replies
    1. ASN shows 13 events related to stabilizer.
      https://aviation-safety.net/database/events/dblist.php?Event=ACFH
      none of these involve a 737

      this appears to be a runaway nose up
      https://aviation-safety.net/database/record.php?id=19990113-2

      these may have been a runaways
      https://aviation-safety.net/database/record.php?id=20020728-0
      https://aviation-safety.net/database/record.php?id=20070604-0

      this was never conclusively shown to be a runaway, but it led to changing the yoke switches to split (two) switches.
      https://www.baaa-acro.com/sites/default/files/import/uploads/2017/11/OO-SJB.pdf

      The rest are structural or otherwise failures

      Delete
  10. Wow thanks Peter!

    I just did a lookup on ASRS myself and only found 3 items for 737 (whole family) with keywords runaway AND trim (4 hits total, but 1 out of 4 was rudder trim). None of them involved any significant AND mistrim.

    The ACNs (if you wish to read them): 1602782, 1478908, 617266

    ReplyDelete
  11. Stupid, non-airplane engineer question: if AoA vane failure due to bird strike is a realistic failure mode, why arent there detection mitigations for that? Is something like looping the circuit for the resolver rotor through the vane so that it is interrupeted if vane goes away useful?

    ReplyDelete
    Replies
    1. Not a stupid question, a valid one. Clearly AOA failure detection is in the dark ages. The vane heater is one big correlating function to a step change of 60 deg. and arriving at +75 degrees. The Lion Air vane fault, which should have been detected in the installation test, would be revealed at a test at 80 knots (level AOA) or in stable cruise (AOA for cruise is within a small range). These tests are done on Airbus airplanes as far as I am aware.

      Delete
  12. Lots of questions on why the plane was travelling so fast

    Could it be that the weight of the plane and airport location (mountains around) influence the pilots

    On normal flights, the pilots would norm5be thinking of preventing stall and climbing as high as possible as soon as possible, which requires high speed

    ReplyDelete
    Replies
    1. the flight crew would be following normal, two-engine takeoff and climbout procedure. The takeoff was shallow. They never transitioned to climb power. They never pulled power back even when overspeed. As best I can gather, they were not paying attention to speed. The CVR transcript should shed some light on what was going on. The stall warning turns on elevator feel shift, making holding the column back harder than normal, and this can encourage relaxing the column. The speed uncertainty and stall warning is troubling, but there was ample indication that this was an artifact of erroneous AOA vane angle exactly as described in the FCOM bulletin and AD.

      Delete
  13. Peter... I appreciate your excellent info and analysis. I agree that crew distraction/overload is suggested by the failure to control airspeed. But the inadequate use of electric trim is a bigger mystery to me. Both Lion flight crews, LNI043 and LNI610, jumped on e-trim immediately, and in the first case, successfully used the stab cut-out and continued under manual trim. The ET302 crew did use the stab cut-out, but to no avail. The statement you highlighted about 1 min 10 sec after the cut-out seems telling:

    "05:41:46, the Captain asked the First-Officer if the trim is functional. The First-Officer has replied that the trim was not working and asked if he could try it manually."

    This suggests that (1) the PIC thought electric trim was or should be working, and (2) manual trim had not been tried to that point. Both pilots had significantly more time in 737 models other than the Max, and possibly, under duress, they did not recall the change (highlighted by you) to the cut-out function in the the new model.

    But this doesn't explain the insufficient use of electric trim when it was available early and late in the flight. And there was apparently no understanding of the need to release elevator forces to allow manual trim to be applied.

    ReplyDelete
    Replies
    1. To your last two points.
      I cannot fully explain why, upon reactivating electric trim, they did not take "control of the situation". One explanation is that the trim "blips" caused a shudder due to the high speed, but I cannot say if that was a factor.
      The Ethiopian crew made no attempt to reduce elevator command to aid trimming nose up, and were left unable, as best can tell from the data.

      Delete
  14. Ron Belt 8 April 2019

    Peter, your summary of things to be learned from the ET302 crash is excellent. But it leaves out any recommendations for things to be changed to prevent such an accident from happening in the future. Here are some recommendations to consider after having read your summary:
    1. MCAS should be disabled in parts of the flight envelope where it could be a danger:
    a. At high speeds near VMO where a sudden stabilizer nose down command can cause an abrupt and dangerous change of altitude
    b. At low altitudes where recovery time from a nose down pitch command is insufficient to resume level flight.
    2. MCAS should perform a data valid test on the AoA sensor inputs
    a. Any AoA inputs above and below some reasonable angle like +25° or -45° should be set equal to the limits or rejected completely.
    b. Related question: Why is the output range of AoA sensors as high as ±90° or ±100° when aircraft rarely go above some smaller number like +25° or -45°?
    3. The MCAS fix currently being proposed by Boeing will stop MCAS from issuing more than one nose down command when AoA sensors disagree, but will NOT eliminate other issues caused by erroneous AoA sensors which increase the pilot’s work load, such as:
    a. Control column pressure aft (pulling) on both sides is increased significantly and remains high even if MCAS commands to the stabilizer are stopped.
    b. Continuous or intermittent stick shaker on the affected side only, especially when it occurs immediately after takeoff and remains ON in level flight. The incessant vibration and noise drastically affects the pilot’s decision-making abilities.
    c. IAS is offset from valid values
    d. ALT is offset from valid values
    e. Minimum speed bar (red and black) on the affected side only.
    f. IAS DISAGREE alert
    g. ALT DISAGREE alert
    h. AOA DISAGREE alert (if the option is installed)
    i. FEEL DIFF PRESS light
    j. Autopilot may disengage
    k. Inability to engage autopilot.
    To eliminate these problems, instead of leaving it up to the pilot and F/O to solve the problems in real time during a flight, consider giving guidance to the pilot to switch off the ADIRU on the side having an active stick shaker and replacing it with the ADIRU readings from the other side. (This guidance was provided in Air Worthiness Directive 2013-19-08 for an A320 with 3 AoA sensors and 3 ADIRU’s to switch off two of them).

    ReplyDelete
    Replies
    1. Hi Ron, I will dedicate a whole post to that point, still noodling on it. I agree with your recommendations. I wonder as well why they cannot have a procedure to deal with erroneous AOA. It would mean turning off the offending SMYD, I think. Mostly to kill off EFSM and to allow autopilot command. The displays can be switched to source off the good side (DEU). No doubt, much to consider.

      Delete
    2. Ron: I think your 1 b is a bit questionable, for this is exactly the situation where unintended nose-up pitch (the reason for inventing MCAS) would be most dangerous.

      What about forcing Boeing to revert the cutout switches to the long-proven classics & NG setup? Including additional training and procedures for recognizing and handling erroneous MCAS interventions (e.g. using autopilot cutoff only as a first attempt before throwing both of the switches)? This would further benefit a lot from a clear audible and/or visual indication of an MCAS intervention, which needs to remain present for some time even after the MCAS pitch-down movement has been finished?

      Delete
    3. Ron Belt 10 February 2018

      Marek, Thank you for your comment and additional recommendations. My post containing example recommendations was intended to focus more attention on finding solutions for ALL the problems experienced by the pilots in these B737 incidents, and not to focus exclusively on just the MCAS problem. Boeing would like to limit the discussion to the MCAS issue only because it will get its planes off the ground again in the shortest possible time. But fixing MCAS will not eliminate all the other problems that the pilots experienced.

      By the way, if only one stick shaker is activated, then if the ADIRU on that side is turned off on, and the other ADIRU is used for both pilot’s displays and aircraft control, then MCAS will continue to operate but with a different AoA sensor that is likely still good. This is why one has redundant AoA sensors and ADIRU’s. It maintains the protection that MCAS provides. If trimming problems still remain after ADIRU switching, then MCAS can be shut down using the two cutout switches.

      Delete
  15. Ron Belt 8 February 2018

    Peter, here is another B737 incident caused by a faulty AoA sensor. The aircraft was a 737NG with registration F-GZHO that took off from Paris Orly on 8 February 2018. During takeoff the “IAS DISAGREE, “AoA DISAGREE” and “ALT DISAGREE” warning lights turned on. The aircraft returned safely to Orly. BEA reported that a post-flight inspection revealed that an AoA probe had failed. On a previous flight the same aircraft had an intermittent AoA problem. This incident is still being investigated by the French BEA as investigation number BEA2018-0071. No report has been issued. Refs:

    1. https://www.aeroinside.com/item/11033/transavia-france-b738-at-paris-on-feb-8th-2018-ias-aoa-and-alt-disagree.
    2. http://avherald.com/h?article=4b4f3b2b. See also comments by Mikie and Anonymous.
    3. https://www.bea.aero/en/investigation-reports/notified-events/detail/event/incident-du-boeing-737-immatricule-f-gzho-et-exploite-par-transavia-survenu-le-08022018-a-orly-94/.

    ReplyDelete
  16. What I saw barely mentioned is the incomprehensible delay built into MCAS. It is primarily a "stick feel" system, Boeing says. From a human factor view I find the delay after flaps up makes the system outright wicked. The delay makes it hard for the pilot to causally connect the events. Does such a delay also apply to AOA trigger? It is already a slow system compared to lets say a stick pusher...

    ReplyDelete
  17. As to why the MCAS bypasses the control column cutoff switches, MCAS needed to be able to respond to an accelerated stall (nose low recover, turning stall) when the control column might be displace far enough aft to trigger the cutout switch.

    As far as the redesign of pedestal cutout switches, I suspect this is what happened: Current 737NG procedures requires the use of BOTH switches in case of a runaway. We no longer try to isolate the system. Since both switches are used by procedure, then there really isn't a need for two switches on the MAX. However, I have been told that they retained two switches in series for redundancy in case of a rare instance of switch (or more likely, relay) welding. I suspect the real reason was to retain as much commonality as possible. Going from the NG to the MAX, you have two switches, and you use both of them. No difference in procedure, no training required.

    ReplyDelete
    Replies
    1. It is so odd that Boeing chose to change something, something that had to be evaluated by the FSB, that reduced functionality. There was no service related compelling issue in the circuit as far as I know - it has held up well these 50 years. While the general procedure could have been left untouched, the AD issued after Lion Air could have used the Autopilot cutoff switch (if it had been retained) as a surrogate MCAS off switch. For an augmentation system they put no switch, no indicator dedicated to it, and then took out the one switch that could be used, for no perceptible benefit.

      Delete
  18. Not reported, but if EFSM activated, there should have been a Master Caution/FLT CONT/FEEL DIFF PRESS annunciation. The FLT CONT annunciation is on Captain's side, and he might have been to overwhelmed to notice since there was also a Master Caution for the ANTI-ICE.

    ReplyDelete
    Replies
    1. There is no mention, but there must have been. Else the EFSM would have been failed in some manner (it has a "dual-coil" actuator - either SMYD can command it independently).

      Delete
  19. Peter, your analysis appears to be spot on. Your last graph, in conjunction with the sudden 75 deg left AOA value and the loss of the AOA vane heater just after what obviously must have been a bird strike, overwhelmingly suggests that the vane was gone and that just a nub of the vane's supporting stalk was all that remained.

    ReplyDelete
  20. Peter, did the first officer try the manual trim wheel crank or the manual trim switch when he said it didn't work after he shut off the power to the tail? Does the data recorder record blips using the electric trim switch when the power to the tail is off? I saw that it records MCAS trying to do it with the power off. I could not determine what the first officer actually tried to do that he said didn't work from the report. Is doesn't work what a pilot would say or was that someone editorializing in the preliminary report. No mention of the Captain helping and it was less than eight seconds of trying. I would have expect him to help. Maybe he was hold the yoke to aft real hard. Would the pulling out of the handle on the manual crank cause a sound on the cockpit voice recorder to determine if the first officer really tried the crank. I saw a video where they were doing the crank in a 737 MAX simulator and I could clearly hear the sound of him moving the crank handle out of its stowed position and a lot more audio when they both cranked. I think I heard it took 250 rotations to go from one end to the other.

    ReplyDelete
    Replies
    1. It does appear that the autopilot commands are recorded with the cutout, but nothing to confirm the yoke switches. The discussion is a bit unclear, but on first blush he tries the yoke switch and then the crank. The captain may have been cradling the yoke so his hands were not on the wheel. I don't think they were faced with anything like 250 turns to get the stab back into the 4-5 unit range - something like 12-20 is what I was led to believe (but have not found a conclusive source). the area mic is designed to pick up noises.

      Delete
  21. Peter, any thoughts on why the functionality of the STAB TRIM CUTOUT switches was changed on the 737 MAX?

    ReplyDelete
    Replies
    1. it is a really big question. Putting two switches to do the same function is a strange change, just in general.

      Delete
  22. Maybe following the philosophy that the pilot should not be able to permanently override MCAS (unless in a full stab runaway emergency)?

    ReplyDelete
  23. Here’s one where a bird strike hit the AOA vane and broke it off.
    A Cargolux Boeing 747-400, registration LX-FCL performing freight flight CV-775 from Calgary,AB (Canada) to Luxembourg (Luxembourg) with 2 crew, was in the initial climb out of Calgary's runway 35R when a loud bang was heard followed by the stick shaker activating. The airspeed was still alive and safe, however, the indications of the stall margin were missing. The crew continued the climb and assessed the situation suspecting a nose tyre had failed and damaged an angle of attack (AoA) vane, then stopped the climb at 12000 feet reporting a damage AoA vane and decided to return to Calgary, where the aircraft landed safely about 40 minutes after departure.
    The Canadian TSB reported no damage was found to the nose tyres, however, maintenance determined the right hand AoA vane was missing, there was evidence of a bird strike. The AoA vane was replaced and a bird strike inspection performed, then the aircraft returned to service.
    https://www.aeroinside.com/item/7646/cargolux-b744-at-calgary-on-may-28th-2016-bird-takes-aoa-vane-out
    This was discussed on .pprune.org.
    https://www.pprune.org/tech-log/580217-744-what-if-we-lose-aoa-vane.html

    ReplyDelete
  24. I wouldn't trust the pprune forum at all. It's curated heavily and run by a power crazed dude who doesn't approve posts he doesn't agree with. I was reading it and on many threads there's tons of complaints about the level of curation and bias moderating. The only reason why there is discussion about the level of bias is b/c there are teams of mods and they don't all mod the same way. Should boycott that forum tbh. It looks like an open discussion but it's really heavily bias.

    ReplyDelete
    Replies
    1. RE pprune forum- Amen amen amen- I made a simple suggestion in april in a post re manual trim force for which the main- top big boy moderator took exception in a very uncivil fashion. Attempts to explain via a civil discourse only resulted in more extreme uncivil response, and removal of all my posts, and most of the thread- eventually removing not only my posts, but those of some very intelligent and qualified responses. Currently almost any thread re 737 and or max is now locked with apparently no new thrreads or posts on the subject allowed. Perhaps there is some ' corporate' pressure being applied ??

      Delete
  25. Hi, entirely unqualified observer - but I have read the PPRUNE thread from beginning to present: There were some posts a while back stating that electric trim does not have as much authority as manual trim, ie there is a lower limit to the stab leverage it can oppose. Does this explain the final 2 ineffectual attempts with the trim switches?

    ReplyDelete
    Replies
    1. No, the final two electric trim inputs were ineffectual because they were very short in duration ('blips') and consequently the stabiliser barely moved, if at all.

      Delete
    2. The issue raised by EASA is that you may need to manually trim beyond the allowed Electric Trim range under certain weight/speed combinations (that are rarely encountered). There has been no assertion that the stabilizer electric trim actuator was mechanically unable to move the stabilizer nose up.

      The minor trim occurrences are confounding. It is possible at the high speed that the trim commands caused noticeable shudder, which "gave them pause", but this was also the stimulus for MCAS to command again.

      Delete
  26. Wow, the plane parts are very confusing for beginners who don't know anything about aircraft.

    ReplyDelete