Published: 2015-07-07 15:09:00
Updated: 2015-07-08 09:49:13
Posted July 7, 2015
Updated July 8, 2015
It was supposed to be a weekend of rest for the team managing the New Horizons mission. On Saturday, July 4, a call came around 2 p.m. to Primary Investigator Dr. Alan Stern as he was working in his office at the John Hopkins Applied Physics Laboratory (APL) main campus near Baltimore, Maryland. New Horizons had gone silent just 10 days before the planned encounter with Pluto and its moons.
The engineering and operations team quickly assembled through a old fashioned phone tree. Stern was in mission control within minutes, project manager Glenn Fountain arrived a few minutes later and Jim Green, director of NASA's Planetary Science division, received updates on the situation as he also awaited word of the arrival of his new grandson.
The stressful day continued as the team worked the remainder of Saturday, overnight and all day Sunday analyzing safe mode telemetry to come up with a get-well plan. There is no quick fix when commands up to and data down from the spacecraft take nearly 9 hours round trip, even at the speed of light.
New Horizons automatically entered safe mode when it encountered difficulty loading the complex sequence of commands instructing the spacecraft and its instruments on what to observe and when to observe it into its primary computer. That same load was applied to the backup computer on the previous day without incident, but something was different this time.
As commands were saved to non-volatile flash memory, the primary computer was still making room for the mountain of data that would be gathered during flyby. Compressing the images and other science data took longer and involved more complex data than anticipated. The two tasks overwhelmed the processor, and New Horizons went into safe mode as designed during that part of the mission. If you’ve ever tried using your home computer as it is backing up or defragmenting the hard drive, you know the difficulty in getting anything else done. New Horizons autonomy system detected a processor overload and switched to safe mode on the backup computer where it awaited further instructions from the ground.
Once the issue was understood, the solution was relatively simple: switch back to the primary computer and carry on with the mission beginning at the most critical portion scientifically, encounter mode. Mission managers are confident that the issue will not recur because the last-time commands will be burned to flash memory during the primary mission. Should the mission be extended and a new target selected for New Horizons to study beyond Pluto in the Kuiper Belt, procedures involving burning to flash memory will likely be adjusted.
Each of these command uploads are tested and the steps rehearsed by the team with engineering models (EM) of the spacecraft bus (the main vehicle itself including propulsion, power, and computing components) as well as the instruments. Engineering models are exact duplicates of the flight model currently barreling toward Pluto. This model was also used to plan out 249 different contingencies in recent years. The team can draw from those plans to address any further problems that might arise through mission end.
While this was the first anomaly serious enough for New Horizons to switch to its backup computer, the grand piano sized spacecraft has gone into safe mode several times in its 9.5 year journey. All planetary science missions have bumps in the road like this, and they sometimes happen at very inconvenient times for the mission plan or the team.
MESSENGER, another long APL mission, spent over 6.5 years en route to Mercury including a flyby Earth, Venus twice and Mercury three times to slow sufficiently to enter orbit. It entered safe mode during that last Mercury flyby, a similarly inconvenient time. The team recovered the spacecraft within seven hours from the same room that the New Horizons team worked their issues. NASA and its partners at APL have procedures in place to methodically recover from incidents. As Apollo Flight Director Gene Kranz said, "Work the problem people, let's not make things worse by guessing."
A total of about 30 observations, including 16 high resolution monochrome images planned for the LORRI instrument, 4 low resolution color and 4 spectroscopy images from the RALPH instrument during the final hours of the missions approach phase, were lost. "We lost some Saturday science and all the science for Sunday and Monday, but the command decision I made is that it is much more important to focus on getting ready for the flyby than to collect science 8 or 9 million miles from the target," according to Stern.
By Sunday, New Horizons was determined to be “operating flawlessly, on course, and not only is the spacecraft operating flawlessly so are all the instruments in the payload,” according to Stern. As a bonus, the compression task that caused all the trouble was successfully restarted and those observations will be transmitted to Earth after the encounter.
On Monday, July 7, at 12:23 p.m. eastern, New Horizons signaled that it had successfully entered the encounter mode, which will culminate with a flyby within 7,700 miles of Pluto.
You join in free “PlutoPalooza” events in the triangle during the period of closest flyby next week:
Tony Rice is a volunteer in the NASA/JPL Solar System Ambassador program and software engineer at Cisco Systems. You can follow him on twitter @rtphokie.