Weather

How NASA engineers found and fixed a failed chip from 15 billion miles away

Six months of troubleshooting, analysis and some creative engineering across multiple NASA teams of engineers made it possible to bring the nearly 50 year old Voyager 1 spacecraft back into service.

Posted 2024-05-01T18:39:34+00:00 - Updated 2024-05-01T22:49:26+00:00

Tony Rice

, NASA Ambassador

NASA.JPL engineers have been working very hard to resolve a problem more than 15.12 billion miles away aboard Voyager 1.

Launched September 5, 1977 Voyager 1, and its twin Voyager 2 the previous week, have studied multiple planets, their moons and rings, and interplanetary space. Even after nearly 50 years, both probes continue to return information about interstellar space nearly daily via the giant radio antennas of NASA’s Deep Space Network (DSN).

That was until November 14, 2023 when Voyager 1 began sending garbled messages. The flight team knew the spacecraft was alive, but its Flight Data Subsystem (FDS) computer was sending back nonsense.

"It would be the biggest miracle if we get it back. We certainly haven't given up," Suzanne Dodd, Voyager project manager at NASA's Jet Propulsion Laboratory (JPL) told Ars Technica at the time.

Voyager wasn't the first spacecraft with a computer onboard, but it was the first to use CMOS volatile memory. Its better performance enables the FDS to handle massive amounts of data flowing from the ten science instruments aboard each Voyager.

Figure 6-2. The Flight Data System hardware, Computers in Spaceflight: The NASA Experience

The team began looking for a fix the same way you might with your laptop or cellphone, by turning components off then on again. Like your laptop of cellphone, this often resolves up problems by reinitializing systems and their storage to known good states. But that didn’t work this time. Deeper analysis was required.

NASA then took steps to eliminate any outside factors that could slow the path to a solution. Combining the receiving power of all six giant antennae at the DSN facility in Spain, the team waited for what they call “whispers from space”. Voyager transmits data at just 23 watts. 15.1 billion miles and more than 22.5 hours later that's reduced to an attowatt, or a billionth of a billionth of a watt.

Each of the six antennae at NASA' Deep Space Network complex in Madrid, Spain were aligned to listen for signals from a single mission, Voyager 1, as the fight team worked to restore communications with the aging spacecraft. (Image: NASA/JPL)

An DSN engineer spotted something familiar among the gibberish being returned. A prompt sent to Voyager 1 prior to the problem. This enabled that unnamed engineer to more fully decode what the spacecraft was now sending, enabled the team to pinpoint the problem: a faulty section within the FDS memory.

Originally written in Fortan 5, then ported to Fortran 77, some Voyager low-level source code running the Voyager spacecraft has been ported to even lighter-weight C over the years. A fix which allows Voyager to navigate around the bad memory as it stored scientific and other data was uploaded on April 18.

Then the team waited.

On April 20, the whisper that came back was one of success allowing the flight team to have the first meaningful conversation with Voyager 1 in six months.

In a conference room at NASA's Jet Propulsion Laboratory in Southern California, members of the Voyager mission team gathered April 20, 2024, to find out if an issue on Voyager 1 had been partially resolved. Just after 6:40 a.m., a cheer went up around the room as the group heard back from the spacecraft: It was returning engineering data for the first time since November 2023.

Today Voyager is back online and regularly returning valuable data. Over the years six of the original ten instruments have been powered off either as they reached the end of their lifespans or to save power for more critical instruments.

You can look in on the fleet of spacecraft exploring the solar system and beyond with NASA’s Deep Space Network Now. Both Voyager spacecraft show up on one or more antennae at DSN complexes in California, Australia and Spain.