An interim assessment of what went wrong during December’s first uncrewed flight of Boeing’s CST-100 Starliner space taxi has turned up so many breakdowns that NASA is ordering a comprehensive safety review of the company’s procedures.
NASA and Boeing provided a status report on the Starliner post-flight reviews today, after concerns were raised publicly this week during a meeting of the space agency’s Aerospace Safety Advisory Panel.
“We are 100% committed to transparency,” NASA Administrator Jim Bridenstine told reporters during a teleconference.
This week’s revelations add to concerns about engineering shortcomings in other lines of Boeing’s business — including commercial airplanes, where a software issue and lapses in training procedures led to two catastrophic crashes and the worldwide grounding of 737 MAX jets; and military airplanes, where Boeing is having to retrofit Air Force KC-46 tankers to fix a design flaw.
Douglas Loverro, NASA’s associate administrator for human exploration and operations, alluded to those shortcomings as he discussed the decision he made with Bridenstine’s support to order a wider safety review. “There were several factors that were in my mind when I asked the boss if we could do this,” he said. “And those were obviously press reports that we’ve seen from other parts of Boeing, as well as what seemed to be characterized as these software issues.”
Boeing’s Orbital Test Flight fell short of its main goal not long after its Dec. 20 launch when Starliner’s rocket engine failed to light up on schedule. By the time engineers could fix the problem and upload the corrected software, it was too late to get the spacecraft on track for its planned rendezvous with the International Space Station.
During their troubleshooting, engineers found that a program designed to keep track of the mission elapsed time — that is, how much time had elapsed since launch — had picked up a time stamp from the United Launch Alliance Atlas 5 booster almost 11 hours too early and never corrected the time.
That issue alone threw the schedule for sending three spacefliers on a follow-up mission to the station into limbo. But there was another software issue that didn’t become public until this week’s safety panel meeting.
Boeing and NASA officials reported that the issue could have ruined the separation of the Starliner’s service module from its crew module during descent. Jim Chilton, senior vice president of Boeing Space and Launch, said today that the service module’s thrusters were programmed to follow the wrong firing profile, with the result that the module might not have achieved enough separation.
“That could have resulted in the service module bumping into the crew module,” Chilton said.
NASA said such a scenario could have resulted in the loss of the vehicle. Chilton said Boeing engineers hadn’t fully worked through what would have happened. However, he added, “it can’t be good when two spacecraft are going to contact.”
Somehow, both of the software issues had slipped through pre-launch verification and testing of Starliner’s flight software. Chilton said the second problem was discovered on the night before the spacecraft was to land, during a comprehensive double check of the software.
“We found it because we went looking,” Chilton said.
Engineers rushed to rewrite the software, verified that it ran correctly, and uploaded the fix less than three hours before the spacecraft’s Dec. 22 landing, said John Mulholland, vice president and program manager for Boeing’s Starliner program. Thanks to the fix, the Starliner made a trouble-free touchdown in New Mexico.
Yet another problem cropped up during the flight: Starliner’s communication system had a hard time establishing contact with NASA’s relay satellites, which contributed to the delay in uploading the fix for the initial engine-firing problem.
On the day after launch, Chilton said the problem cropped up because Starliner’s misfiring thrusters put it in the wrong position to make contact with the satellite network. Today, Mulholland said Boeing’s preliminary conclusion is that the problem had to do with “Earth-generated noise over specific geographic footprints.”
“We believe the frequency … is very close to the frequency that would be associated with cellphone towers, and that created that high noise floor which didn’t allow us to establish a link as soon as we would have needed to,” he said.
He stressed that the communication issue was still under investigation. A joint Boeing-NASA investigative team is due to finish its work by the end of this month.
Loverro said he was concerned that the Starliner program’s problems could go deeper than the three issues that cropped up during December’s mission.
“To put it bluntly, the issue that we’re dealing with is that we have numerous process escapes in the software design, development and test cycle for Starliner,” he said. “The two software issues that you all know about are indicators of the software problems, but they are likely only symptoms. They are not the real problem.”
Loverro compared the job to checking a car’s spare tire. “If you have a flat spare tire, you don’t find that out by driving the car,” he said. “You find it out by opening the trunk and checking the pressure in the tire. … Our intention is to go check the spare and make sure it’s inflated properly.”
Kathy Lueders, program manager for NASA’s Commercial Crew Program, said the investigative team has already come up with a list of 11 high-priority corrective actions, mostly having to do with addressing known software errors and tightening up testing procedures.
“We’re going to take those actions, along with further actions that Doug and the EO [exploration and operations] team have given us, to make sure that we’re really addressing root cause,” she said.
One of the requirements will be to take a closer look at the full Starliner software package, which Boeing says amounts to a million lines of code.
Bridenstine said it was too early to predict how the safety review might affect plans for future Starliner flights. Eventually, NASA will have to decide whether or not to require another uncrewed flight test before astronauts climb on board.
If NASA requires another uncrewed test, the do-over would probably make use of the Starliner craft that’s currently being prepared for the first crewed flight, Peter McGrath, global sales and marketing director for Boeing’s space exploration business, said this week at the Pacific Northwest Aerospace Alliance’s annual conference in Lynnwood, Wash.
In that scenario, the Starliner capsule that flew in December would be refurbished for the first crewed mission, McGrath said. Boeing estimates that it should take about four months to get a Starliner ready for reuse. However, in a report issued last week, the Government Accountability Office said the first round of refurbishment could take longer than that because it’s never been done before.
Today Bridenstine reminded reporters that NASA is working with SpaceX as well as with Boeing to develop commercial space taxis. Over the past year, SpaceX’s Crew Dragon has successfully conducted an uncrewed mission to the space station as well as an in-flight abort test. In the wake of last month’s abort test, SpaceX CEO Elon Musk said the first crewed flight would probably take place in the April-June time frame if NASA gave the OK.
“We have two providers, SpaceX and Boeing, that are going to take American astronauts to the International Space Station,” Bridenstine said. “Because we have dissimilar redundancy, this program is going to continue moving forward in a way that will make our nation proud.”