An uncrewed test flight of Starliner, a Boeing spacecraft designed to carry NASA astronauts, could have ended in disaster in December because of lapses that allowed software errors to slip through undetected and unfixed before the spacecraft launched, according to a review by NASA and Boeing that was announced on Friday.
The review team made 61 recommendations for fixes and improvements. Some recommendations were specific, such as changes to the software testing procedures. Others addressed possible blind spots in how the program was managed.
Top NASA and Boeing officials said they welcomed the report’s findings.
Instead of building and operating its own spacecraft to take astronauts to space as it has in the past, NASA has hired two private companies — Boeing and SpaceX — to provide transportation to and from the International Space Station. SpaceX appears to be on track to launch its first mission with astronauts aboard its capsule, Crew Dragon, in the coming months.
Boeing’s December test flight was to have been the last major milestone before NASA agreed to putting its astronauts aboard. Now, the space agency may require Boeing to repeat Starliner’s uncrewed flight.
Douglas L. Loverro, the associate administrator for human exploration and operations at NASA, said the two organizations would take several months to review the report and put changes in place “in order to make sure when we decide to fly again, we can fly safely.”
Mr. Loverro said he had also decided to label the December flight a “high-visibility close call,” which will lead to a review of organizational processes at Boeing and NASA to “make sure we truly do learn from this event and that we know how to fix it and make sure it does not happen again.”
John Mulholland, the manager of the Starliner program at Boeing, said an audit of the software development process revealed 49 gaps in testing. That does not necessarily mean that any problems lurk in those sections of the software code, but that no one knows for sure.
“Now our team will be able to go perform that testing and identify whether there’s any additional corrupt code, and if so, we’ll be able to fix it,” Mr. Mulholland said.
The Starliner spacecraft, launched on top of an Atlas 5 rocket on Dec. 20, encountered two major software issues during its flight. The first occurred minutes after it separated from the rocket, because the clock had been set wrong. That led to the spacecraft squandering its propellant, and a planned docking at the International Space Station was called off.
A second flaw would have fired the wrong thrusters as Starliner was preparing for re-entry. Because Boeing engineers hastily combed through the Starliner software in the aftermath of the clock problem, they found the second problem and fixed it. If it had not been fixed, two pieces of Starliner — the capsule that returns to Earth and the service module, which is discarded — might have collided. The capsule might have tumbled and burned up in the atmosphere instead of landing safely in White Sands, N.M.
An integrated simulation of Starliner with the Atlas 5 rocket from launch to docking at the space station would have revealed the flaw with the clock. But Boeing engineers split the simulation into shorter chunks.
The first chunk simulated from launch to separation of the Starliner, and at that point the error in time had no obvious effects on the operation of the spacecraft. The second chunk started at the point of separation but presumed the clock had been set correctly.
One of the review’s recommendations was to conduct end-to-end simulation tests.
The review team also found that too much authority was given to a Boeing software review board to approve changes to the Starliner software. Those changes should have been brought to a broader engineering review team so that any changes were coordinated with other engineers, Mr. Loverro said.
The flaw with the thruster software occurred because it was tested with a flawed hardware emulator, instead of the actual thrusters.
The review called for verification that various hardware emulators used by Boeing accurately mimic the behavior of the real systems.
Kathy Lueders, program manager for NASA’s commercial crew program, said NASA had already sent more software experts to keep a closer eye on Boeing. “We are taking concrete corrective steps to move forward, to improve for the next mission,” she said.
NASA is planning to use a commercial approach for a lander to take NASA astronauts back to the moon similar to the one used for the Boeing and SpaceX programs where the companies and not NASA own the systems, and the companies also have more freedom in coming up with the design, which reduces the costs.
“We’re going to roll these lessons into our human lander requirements,” Mr. Loverro said.
With the delays experienced by both Starliner and Crew Dragon, NASA is currently negotiating with the Russian space agency to buy one more seat on a Soyuz capsule, Mr. Loverro said. Since the retirement of the space shuttles in 2011, NASA astronauts have been riding on Russian rockets to orbit.
Mr. Loverro said that the problems were not a consequence of the fixed-price structure of the SpaceX and Boeing contracts.
“I think it was the way we chose to manage it,” Mr. Loverro said.