Exploration and Engineering

Mark Adler et al., Mars Mobile Pathfinder ’03, May 3, 2000, MOAR-050300.ppt, Manning materials, Historian’s Mars Exploration Collection, JPL. Courtesy Robert M. Manning.

This proved a fateful decision. The 90-day mission requirement drove redesign of the rover, increasing its mass. And the mass increase forced a redesign of the Pathfinder entry, descent, and landing (EDL) system. By the time the MER team got to their preliminary design review in October, at which their major designs were supposed to be finished, they already knew that, instead, their design was badly broken. It would take them more than a year to fix it.

Between NASA’s approval of the MER mission in July 2000 and the middle of the following year, the rover grew substantially. In turn, the rover’s growth in both volume and mass “broke” the original idea of using the Pathfinder lander, aeroshell, EDL subsystems, and cruise stage essentially unchanged. There were several causes of the rover’s expansion, but a big one was thermal management. Keeping the vehicle electronics alive long enough to do their various jobs proved a difficult design challenge.

But the electronics boxes on the lander turned out to be a big problem. Because they were actually pretty small and relatively thin—the lander structure was only a few inches deep—they lost heat quickly. They had to be kept above −55°C when the nighttime temperature might be −100°C, so they needed heaters. And the heaters needed power. The thermal analysis performed on the design suggested that they’d need seven batteries to keep the lander electronics alive overnight. The batteries themselves had to be kept between 0°C and 20°C, so they could only go into the rover’s own WEB.

They’d only intended to have two batteries in the rover, not seven. Krajewski rather ruefully remembered, “[We] couldn’t find a solution that didn’t have us increasing our battery size inside the rover belly by a factor of three or some horrible number like that, which would defeat the whole purpose of pulling the electronics out of the belly.”4

The analysis that revealed this problem was finished in early August 2000, at the same time the project was reviewing the two-rover option.5 The team tried to salvage the architecture by studying various ways of insulating the lander electronics to reduce the power needs. By the project’s preliminary design review during the second week of October, barely three months after they had started, they already knew that this approach wasn’t feasible either. There just wasn’t enough volume available for insulation. They had to move the electronics that controlled the pyrotechnics, the airbag retraction motors, and the lander petal motors into the rover’s WEB. So the WEB grew. In late November, deputy spacecraft manager Barry Goldstein sent an e-mail out to the design team establishing the larger rover as the new baseline design.6

The second reason that the review didn’t go well for Theisinger’s crew was that their “Level 1” requirements were still in flux and needed to be settled quickly. These requirements specify, at a high level, what a mission is supposed to accomplish. JPL’s systems engineers used these as the basis for specifying the detailed technical capabilities the vehicles had to have to achieve them. There were a couple of Level 1 issues still unresolved. David Lavery, their program executive at NASA headquarters and a big supporter of robotics, wanted one of the rovers to demonstrate the ability to drive substantial distances on Mars, and he sought a requirement that one of the rovers be able to drive at least a kilometer. Neither Theisinger nor Squyres wanted to have to do this. The rovers were going to be slow, and stopping to make scientific measurements would make them even slower. They wanted the achievement of a long driving distance to be contingent upon doing a certain number of measurements. NASA eventually agreed to their position, and the requirement was weakened to 600 meters with a “goal” of 1 km. The Level 1 requirements also specified visiting “at least eight” different spots for detailed investigations.9

The rover was being designed to use both X-band “direct-to-Earth” transmission and the UHF relay on 2001 Mars Odyssey to return its data, but with a strong preference on the engineering team’s part to emphasize UHF. The direct-to-Earth transmitter produced so much heat that it would overheat the WEB after about an hour and a half, which wasn’t enough time to return all the data. The UHF system generated much less heat. But Squyres was unwilling to depend on the as-yet unlaunched Odyssey.

Discussion between Squyres, Theisinger, and NASA officials led to an understanding that the direct-to-Earth would be the primary means of communications, although it was not explicitly spelled out in the approved Level 1 requirements. Instead, the flight system (Level 2) requirements were revised to specify that the rover had to support four hours of continuous direct-to-Earth use, which was the approximate length of time needed to return each sol’s data.11 That made the mission independent of Odyssey, but at the cost of still more mass and design complexity.

The delta PDR was scheduled for the first week of February 2001, less than three months away. Theisinger created two special engineering teams (“tiger teams”) to redesign the cruise stage and lander to recover mass margin and to redesign the EDL subsystem to accommodate the bigger, heavier rover and increase mass margin.

The rover design hadn’t stabilized as of February; in fact, it kept growing until July 2001. Again, the thermal needs of the rover’s electronics were a big culprit. The key to the rovers’ surviving the required 90 sols on Mars was the amount of heat dissipated into the warm electronics box. It didn’t matter whether the heat was generated by battery charging, running the electronics in the electronics module, the power amplifiers, or by the 1-watt radioisotope heater units they were planning to put on the rover batteries. At the same time, energy consumed outside the WEB by the wheel motors, panorama camera (PanCam) and mini-TES (thermal emission spectrometer), or the robotic arm, was a dead loss as far as the rover’s thermal state was concerned. (The mast supporting the PanCam and mini-TES also acted as a big heat “chimney,” pulling warmth out of the WEB.)

The increased need for power forced the project to increase the solar array area by 25 percent, which just added to the team’s design problems. The larger panels had to be folded into a lander that was already full. And, of course, they added to the mass problem. The new array sizing need was acknowledged during the project’s thermal subsystem preliminary design review in January and accepted—with great reluctance—at the delta PDR. But the project didn’t have a design for the new array, and wouldn’t until July. Instead, the Lab’s mechanical designers were consumed with the effort to redesign the entire spacecraft to accommodate the rapidly growing rover.

The rover’s growth posed three big threats to the project. The first was that the mechanical engineers wouldn’t be able to squeeze the rover into Pathfinder’s lander and aeroshell. The second was that if they could make it fit, the resulting flight system might be too heavy for their Delta 7925 to send to Mars. And the third problem was that if they could fix those two issues, they still might not be able to land it safely on Mars due to the higher landed mass. This last fear was reinforced after their first airbag test in December 2000, just as the tiger teams were finishing their pieces of redesign. The flight spare set of Pathfinder airbags failed catastrophically at slightly less than the Pathfinder lander’s mass, undermining confidence in the basic soundness of the Pathfinder design.13

The lander-rover integration tiger team addressed the growth by redesigning the lander tetrahedron so that it was no longer symmetrical. They “clipped” off the top of the tetrahedron to widen the opening enough to accept a wider parachute canister, and they widened the base petal to accept the longer rover. The EDL tiger team then modified the backshell and heat shield to accept the redesigned lander. The Pathfinder backshell had been a 70° blunt cone with the pointed end cut off, and engineer Dara Sabahi’s tiger team didn’t change its conic nature. Instead, they widened the narrower end of the cone by the simple expedient of cutting off the pointed end closer to the wide end. Then they put the length they’d removed from the narrow end onto the wider end.15 The result was an increase in total volume within the aeroshell. This change also forced them to make the heat shield slightly wider, to fit the wider backshell.

The new, larger lander and revised backshell design cost the project a mass increase. Since the main purpose of the tiger teams had been to reduce mass, the two groups had to pay for the mass increase by finding or creating even larger mass reductions. They proposed to Barry Goldstein and flight system manager Richard Cook changing the materials of the lander and cruise stage. The Pathfinder lander’s primary structure had been aluminum, which was simple to manufacture and cheap. And its cruise stage had been a combination of aluminum and titanium. These weren’t necessarily the most mass efficient materials, though. Composite materials, such as those used on Mars Global Surveyor, were often lighter but more difficult to work, and more expensive. They thought replacing the lander primary structure with a composite would save them about 24 kg, while imposing increased cost and schedule risk—they’d lose, they figured, four to six weeks.16 While they’d been hoping for a 30 kg reduction, Cook, Goldstein, and Pete Theisinger agreed to this change. There seemed no other way of reducing the lander mass further.

The redesign effort between October and the end of January succeeded in raising the project’s mass margins above the required 15 percent, satisfying the review board.18 To track mass growth more rigorously, Theisinger empowered a “mass czar,” an engineer named Grace Tan-Wang, who would chair what was known informally as the Mass Tribal Council. The council was to meet weekly to review all proposed mass increases and to recommend allocations from the mass contingency reserves held by Cook for the flight system and by Theisinger for the project overall.

The rover solar array story was finally completed when a mechanical engineer suggested a design for the additional array area that met with quick approval.19 Secondary arrays attached to the rear of two lateral panels with hinges; stored for the cruise to Mars, they would be folded flat against the top surfaces of the primary lateral arrays. Once on the surface, they would be unfolded after the rover had lowered the primary panels into place. They were an elegant solution to a difficult problem.

But the deployable arrays added mass and complexity, which the project couldn’t afford. And shortly before their critical design review, the project management team tried to remove them to stay within their landed mass limits, despite vigorous lobbying by Steve Squyres.20 The effort fizzled a few days later when the solar panel’s vendor told them that the arrays would be able to host fewer individual solar cells than they’d originally thought, leaving the rover without enough power to meet the Level 1 requirements. So the secondary arrays were put back on. The project sacrificed one small instrument, a “SunCam” that was to locate the Sun as a check on the rover’s location and orientation in the name of mass recovery.21 But otherwise they had to absorb the solar array mass increase from their margins.

With all the changes to the rover design, its allocated mass ballooned from 130 kg just before the two-rover decision in August 2000 to 175 kg at preliminary design review in October. It topped out at 185 kg at the critical design review in August 2001. They’d lost months of schedule in the redesign effort, and their state of being perpetually behind became a long-running joke. Grace Tan-Wang put a gag article in the project’s not-entirely-monthly newsletter, the Mars Explorer Times, about new instructions to JPL’s security force to keep project members on-site. Next to a photo of a compact car with JPL staff members stuffed in its trunk, she wrote: “The recently upgraded barbed wire fences have proven quite effective at retaining workforce at JPL. Occasionally we need to pepper spray someone attempting to leave without authorization, but overall the employees have been compliant.”23

More seriously, getting all the detailed redesign work done caused Sabahi to remove the mechanical design team from all their other institutional responsibilities at JPL for a month. “In MER I realized that actually most of their time goes to deal with processes that are mandated. And the amount of time that they can spend to actually do design and delivery is a fraction of their time. So I gave a challenge to my division and the flight system. I said that the engineers that are working on the rover structure, for thirty days, they are to be excused from all meetings, all requirements, everything. Thirty days they do nothing but design. And I basically say, ‘If that fails, I will throw in my badge.’ And I meant it.”24

The first set of changes had been made to the parachute and its support equipment very early in the project, to enable landing at somewhat higher altitudes and during daylight hours. The less dense atmosphere at higher altitudes, in turn, had created a “timeline threat.” The timing of events in the EDL design was critical, and early simulation studies showed that Pathfinder’s parachute would probably leave the vehicle with too high a terminal velocity. There might not be enough time to carry out all the later events that had to happen. So they’d accepted the need for a much larger parachute, which would slow the vehicle more.

The MER team also knew very early that they were going to have to scale up the solid rockets in the backshell (the rocket-assisted deceleration, or RAD, motors). The RADs had to zero out the vehicle’s vertical velocity, 70 or 80 meters/second, right before the computer cut the bridle, dropping the lander on the surface. The amount of momentum they had to remove depended on the terminal velocity on the parachute and the mass of the lander. They couldn’t predict either of those things very accurately until the lander design was more mature. So they decided not to decide how much more powerful the RADs had to be until the last possible moment.

They had hoped, though, to preserve the airbag design that Tom Rivellini had agonized over so much during 1995. But the team also needed to prove that the design would accept the higher lander mass without much alteration, so they decided to hold a series of tests using the Pathfinder flight spare bags out at NASA Lewis Research Center’s Plum Brook station. This first test series was designed to validate an assumption Manning had made from basic theory. What the airbags did was absorb the kinetic energy of the impact, which is a product of the lander mass and the square of the impact velocity. That should mean that the airbags were much more sensitive to impact velocity than to lander mass—so if they could lower the impact velocity a little, they could raise the lander mass quite a bit and still have the same amount of kinetic energy to absorb.26

In parallel, Manning and Dara Sabahi were also thinking about ways to reduce the horizontal component of motion the airbags had to withstand. They knew from Pathfinder’s test program that the airbags’ Achilles heel was being dragged across rock faces. Dragging tore the abrasion layers away from the bladder, leaving it vulnerable. And they believed from the thousands of simulations carried out during Mars Pathfinder that most of the horizontal velocity that the MER landers might experience would be induced by the RADs firing just before impact.

The wind model they were using for the first year of the project, which was based on the winds at Kennedy Space Center, often had strong shear. Manning’s crew had used this “Cape model” on Pathfinder in lieu of anything else, even though Mars isn’t much like coastal Florida. As it became clear that this shear problem was a big vulnerability, the EDL team approached a couple of JPL’s meteorologists about the model. They were David Kass, a recent hire, and John T. (Tim) Schofield, who’d been working with Dan McCleese on the repeated efforts to get his pressure modulated infrared radiometer instrument to Mars since the 1980s and who had been the investigation scientist for Pathfinder’s meteorology package. Kass remembers that the EDL engineers were hoping that the Cape model was too conservative and that they could safely reduce its shear component. At Caltech and Colorado State University, Kass identified a couple of other “mesoscale,” or regional-scale, models designed for Earth and started working with their designers to convert them for Mars. Mesoscale models needed decent topography, so adding the topography for the various proposed landing sites was part of the effort.29

The effort to get a better wind model promised to take about a year, so the EDL engineering team pursued a second track to solving the wind sensitivity problem, too. Since the basic problem was that the winds might push the backshell out of alignment with the local vertical, could they find a way to shove it back toward the vertical just before the RADs fired? In November 2000, Sabahi suggested what he thought would be a simple solution: in the event the backshell was far out of alignment with the local vertical, have one of the three rockets fire slightly early. One rocket firing early would force the backshell to start moving back toward the local vertical. This would reduce, though not completely eliminate, the RAD-induced horizontal motion. In order for the scheme to work, though, the rover had to know what the backshell’s orientation was in order to command the appropriate rocket to fire early. So they had to add an inertial measurement unit to the backshell. By the delta PDR in January 2001, they had added this “differential RAD” firing plan to the baseline design.30

Sabahi’s simple solution turned out not to be simple, though. There was immediate concern over the possibility that the early firing would cause part of the “triple bridle,” the pairs of cables that ran between the edges of the backshell, and the lower, “single bridle” section connected to the lander to go slack. Because the RADs were mounted at an angle to the center of mass, firing only one would tend to cause the backshell to rotate around the center of mass toward the side opposite the rocket that was firing. So the bridle line opposite the firing RAD would go slack. Then, when the RAD on that side fired a half second later, the slack bridle line would suddenly go taught again, and it might break.

How serious this problem actually was proved very difficult for the EDL team to know. They had to meet the 95 percent probability of success requirement, which meant they had to show Pete Theisinger and the review board that they could produce a credible analysis of the problem. But they couldn’t figure out a simple series of tests or analyses that would be convincing even to themselves, let alone everyone else. The off-axis thrust induced some nonlinear dynamics that couldn’t be modeled easily, leaving them with a lot of uncertainty.

Miguel San Martin, a tall, balding Argentinian who was their guidance and control guru, disliked Sabahi’s differential RAD idea because of this torque problem. He proposed an alternate solution that eliminated this uncertainty, but at the cost of precious mass. A separate set of three small solid rocket motors that were aligned with the backshell’s center of mass would impose the horizontal velocity the team wanted without causing the backshell rotation they didn’t want. The motors would cost them money and mass, and they’d have to cut holes in the backshell for the motors to fire through. But the approach would solve the horizontal velocity problem imposed by backshell oscillation without the uncertainties that plagued the differential RAD idea.

In April 2001, San Martin’s concept formally replaced differential RAD.31 The project named it transverse impulse rocket subsystem (TIRS). The addition of TIRS promised to take some of the pressure off the airbag team, but at least in the near-term, it had no effect. In May 2001, Lee and Steltzner took the refurbished Pathfinder bags out to Ohio again. This time, they replaced the abrasion layers with new ones made of a thinner material. The experiment was intended to see if layers of a thinner material might provide the air bladders with adequate protection with less mass. These May tests had somewhat ambiguous results. The thinner material did seem to provide results similar to Pathfinder’s heavier abrasion layers, but the airbags still failed. Two of the three test drops suffered fatal bladder damage.32

In October, finally equipped with new airbags, the team returned to Ohio a third time. They had decided to try different abrasion layer variations, including both the heavier Pathfinder fabric and the lighter experimental fabric. On the very first drop, the new bags failed catastrophically. And on the second, too. In fact, a single rock became the demon of the test series. Each of the rocks on the test platform had been marked with colored chalk so the team could tell which rock did damage; the rock that did all the damage in this series had been colored black. This black rock blew out the bags on five out of eight drops.33

This test series threw the project into a crisis. The EDL team had convinced themselves that the Pathfinder flight spare bags had failed the previous two test series because they were old. But these bags were new. Their failure raised questions about MER and Pathfinder. Maybe Pathfinder had just gotten lucky. If the Pathfinder design had been robust, these new bags should have survived these drops. They hadn’t been carried out at extreme conditions. In fact, the team had lowered the impact velocity to 21 m/sec from Pathfinder’s tested maximum of 26 m/sec. And yet, in Manning’s words, “Rocks cut right through.”34

And if the airbag failure wasn’t bad enough, by early November 2001 it was becoming clear that TIRS was not enough to solve the horizontal velocity problem. David Kass’s wind modeling effort during the year had shown the opposite of what the project’s engineers had wanted to hear. The Pathfinder wind model wasn’t too conservative—it wasn’t conservative enough. The two wind models Kass had fostered generated winds that were worse for every proposed landing site except Meridiani. But the vertical shear problem wasn’t the primary culprit any more; TIRS was adequate to resolve that issue even with the new simulated winds. The problem now was that Pathfinder’s Cape model had very little steady-state wind; in other words, few conditions in which the winds blew continuously. It simply had gusts. But these new Mars mesoscale models showed that heating and cooling of the surface at the landing sites produced steady-state winds of sufficient energy to impose horizontal motion on the descending spacecraft, too.35 (Meteorologists call these winds “katabatic” winds.)

Two pictures would tell San Martin what the horizontal velocity was just as well as a Doppler radar would. Objects on the surface that appeared to move from one image to the next would actually represent the spacecraft’s own horizontal motion, if the lander’s swinging on its cables could be subtracted. On-board image processing would be able to extract the bits of image that showed motion and calculate its magnitude and direction. But it meant they needed another camera.

Manning immediately realized that, in terms of hardware, another camera was easy. They’d kicked the SunCam off the mission, but they’d been too busy to remove all of its supporting electronics from the avionics design. He also knew that Joel Krajewski had routed a camera cable out of the rover to the lander, hoping to get permission from Theisinger to add a rover stand-up camera. So putting the SunCam back on with a different lens, this time stuck on the bottom of the lander, wouldn’t be technically difficult. Getting approval, however, would be politically difficult. It was very late in the development process to be adding yet another new piece to the vehicle’s EDL suite that would have to be developed, integrated, and tested. Manning had to have a convincing story to take to Theisinger.

On December 13, they formally pitched the DIMES idea to Cook, estimating a cost of $1.5 to $2 million. At this point, Theisinger found out about it and, as Manning had guessed, he wasn’t happy. But he also let the effort continue, while continuing to keep it from JPL senior management. Theisinger recognized that it might be a valuable addition to the project, but that the review board wouldn’t likely see it that way until it was much more mature. To them, it would look like more work for a project that was already overworked and behind schedule.

The terrible airbag tests and increasing awareness of their wind sensitivity led the project team to reopen the airbag design, too. The previous experiments in October had been minor variations on the Pathfinder design, with differing number and thicknesses of abrasion layers but without fundamental changes to the bladder design. A fundamental redesign had been a taboo subject within the project; the airbags’ flight heritage was rooted in Pathfinder, and doing something really different meant a much more extensive (and expensive) test program. But Cook and Goldstein and Sabahi and Manning no longer thought minor changes would suffice. In a December workshop, they widened the range of possible redesigns considerably.

“Wigs, double bladders, bulging lobes, and gossamer bladders were all on the table” at the workshop.40 It was stressful for everyone involved because the team knew they had to leave the meeting with a design that would absolutely, positively solve the problem—a “kill-all” solution, Sabahi called it.41 The EDL team agreed to four new designs, including Sabahi’s “kill-all” solution, a double-bladder set of bags. In fact, they agreed to two sets of double-bladder bags, using two different fabrics for the bladders. The contractor’s staff in Delaware worked through Christmas to get the bags built; Sabahi trooped out to Ohio with Wayne Lee and Adam Steltzner in mid-January to set up the next set of tests.

They made a few key changes to the test methodology. They made several more copies of the notorious black rock from the October tests and attached them to the platform to make the tests still more challenging. They’d also hired the JPL photo lab to install high-speed cameras under the platform so they could watch the instant of impact more closely. They even cast a copy of the black rock in clear plastic and mounted a camera, with lights, inside it, so they could have a “rock’s eye” view of their villain in action.

March 2002 also saw DIMES “come out of the closet” and get exposed to JPL management and the project’s review board.44 In April, the DIMES underground faced an implementation review at which there was considerable hostility. Two members of the board wanted it dropped immediately, believing it would distract the project management from the larger MER effort. But Theisinger and Cook made clear that DIMES hadn’t been baselined for flight yet; it was in development and could still be dropped or flown depending on its success. And they pointed out that the majority of the work being done on it was being performed by engineers who hadn’t already been attached to the project, so they weren’t being diverted from some other MER task. The board report concluded that “because DIMES offers a way to mitigate the clear vulnerability of the MER EDL system to sustained horizontal winds, continuing to pursue DIMES could result in a significant overall improvement in the probability of MER landing and mission success.”45 So DIMES stayed alive.

In May, Johnson selected a set of test sites in the California desert that seemed good analogs to the half dozen landing site finalists. Squyres particularly liked a site in the California desert, telling Johnson, “Well, if you show that this works at Ivanpah dry lake bed, it’ll work at Mars.”46 In June, they flew an engineering model of the camera around the sites via helicopter to capture real images of terrain-in-motion for validation. They also built a set of simulated images, using Mars orbiter camera images of the potential landing sites and mathematically “smearing” them the way they would appear to the moving DIMES camera. During the summer, Monte Carlo simulations using the Mars orbiter camera images suggested that the system would probably have errors well under the 5 m/sec allowed by the project’s requirements; in October, a final set of field tests confirmed that it did. In fact, only a small fraction of the imager’s calculated velocities had errors over 2 m/sec.47 The team passed their delivery review in mid-December, although clearing that hurdle didn’t yet guarantee inclusion in the mission. They still had to perform well in their assembly, test, and launch operations (ATLO) systems tests.

When they finally got good weather, the first test drop was a disaster. Their test body had a high-speed camera pointed up at the parachute, so they got a close-up view of their parachute inflating and then disintegrating. Worse, it wasn’t the result of a freak accident or a manufacturing defect. Four more parachutes disintegrated as the team watched with growing horror.49

Worse still, Lee laughed later, was that the test methodology wasn’t even stringent enough! They had needed such a heavy test body because on Mars, the atmosphere is so thin that the parachute opening doesn’t slow the vehicle down significantly. But the much denser Earth atmosphere will slow the vehicle during inflation unless the vehicle is super heavy—that’s why there’s always a sudden jerk when a skydiver’s parachute starts to inflate. It turned out that their big test body hadn’t been big enough. The parachutes had started to slow their giant lawn dart as they’d inflated (and started disintegrating), so even if the parachutes had survived, the tests would have been invalid. They were not adequate enough reproductions of Mars conditions.

So MER had two new problems. The parachute design had failed, and the parachute test methodology had also failed. The team had to fix both in a hurry. ATLO, the point at which flight hardware was supposed to be finished and delivered for testing, had already started back in February; they had 12 months before launch.

The solution to the methodology problem came when Lee visited Ames Research Center to check up on the heat shield testing. He asked the heat shield engineer what one of the giant, but strange-looking, buildings at the center was; it was a 40-by-80-foot wind tunnel that had been built for rotorcraft development. Lee and Steltzner had talked about using a wind tunnel for parachute testing earlier but hadn’t thought anyone would let them fire the mortar that deployed the parachute inside a tunnel; if they couldn’t fire the mortar, the test would violate the “test as you fly” commandment and be invalid.

The contractor’s lead engineer, Allen Witkowsky of Pioneer Aerospace, explained later that the parachutes were failing because the location of the area of peak stress had changed when they’d altered the Pathfinder design, and they hadn’t realized it:

We were chasing failure modes. We’d see lines break, so we would strengthen the line joint, and then it would move to just up from the lines there would be a break, and this was all in the band region. We’d fix that, and then it would break a little bit further up. We were just chasing the failures, trying to fix them incrementally, saying, “Well, this must be wrong,” and, of course, we’re all being rushed to hurry up and get the design finished so they can start qualification so we can get it on the spacecraft and so forth and so on. At some point we realized, “Let’s take a step back. What’s going on here? Why is this happening?” And we realized that the location of peak stress had changed.52

For Pathfinder, it had been in the disk. For MER, it was in the band.

In late June, the EDL team held another stressful workshop to design a new parachute. Since they hadn’t yet figured out what the problem actually was, they decided to have three different designs built. One was what Steltzner called a “brick,” heavily reinforced everywhere and made of a thicker, stronger fabric. But because the parachute canister was fixed in volume, the reinforcement forced him to make the parachute smaller in drag area. The “brick” would also be less stable, and would allow more oscillation to occur during the descent. So if that were the one that had to go to Mars, it would make some of the potential landing sites unachievable, including, probably, the one Squyres most wanted, Gusev Crater.53 They also chose a lightweight design that was only reinforced where Witkowsky’s engineers thought the failures had started, based on the videos of the four failures. It would have the best performance but might not be strong enough. And they designed one that was a compromise between the “brick” and the “high performance” parachutes.

June was also the month in which the project had to commit the TIRS motors to manufacturing, which meant having to decide how powerful they had to be. That was a problem, since if the less-stable “brick” parachute was chosen as the flight design, the TIRS motors would have to be more powerful than if the high-performance chute went to Mars instead. They couldn’t wait for the September tests to answer this question, so they decided to build the bigger TIRS motors that the brick would need. It seemed the lowest-risk choice.

Steltzner took the three new parachute designs up to the Ames wind tunnel in September. These tests gave him a new problem: squidding. It occurs when a parachute deploys but doesn’t open fully. Instead, the lower section of the parachute, the “band” on the MER parachutes, just oscillates around. Squidding happens when the amount of air going in through the parachute base and out through its vent is the same; the parachute can only inflate if more air enters the base than escapes through the vent. On the first test he did, using the intermediate strength chute, the parachute opened partly, squidded, and only fully opened after they’d started shutting the test down. Steltzner was horrified. The parachute had been strong enough, at least.54 But no one had ever seen a Mars parachute squid before. Steltzner thought it might be because this parachute had used an impermeable fabric in its band for higher strength. That could have kept the parachute from inflating properly. The brick had the same impermeable fabric in its band, so for the next day’s test he chose the weakest of the three, which used a more permeable fabric in the band.

The next day, he had to watch as the “high performance” chute also didn’t open properly. Instead, it squidded more than a minute before it finally opened. But again, the chute stayed intact. So at least he knew that the weakest chute was strong enough—if he could figure out how to make it open properly. Steltzner decided the vent looked too big. He measured it and discovered it was. There had been a miscommunication somewhere. He had one of Pioneer Aerospace’s parachute technicians hand-stitch in a patch to shrink the vent and retested it. This time, it opened properly, and it still didn’t disintegrate. The test wasn’t entirely valid, as Steltzner hadn’t been able to repack the chute into its container and refire it from the mortar. Instead, he’d had the tunnel staff do a “dis-reefing” test in which the chute was held shut by a lanyard while the tunnel was started up and then released.55 But it convinced Steltzner that if he could get the parachute built correctly, he had a sound design.

The parachute team returned twice to Ames in October for more tests, this time with few problems. In the first week of November, the team chose the “high performance” parachute for the flight to Mars, hoping to preserve access to the Gusev Crater landing site. The parachute was finally flight qualified in January 2003, five months before launch, two months after the contractor had started building the flight parachutes, the same week the project shipped its first batch of flight hardware down to Kennedy Space Center.

The project’s airbag challenges also ended satisfactorily in late 2002. Tests of the new double-bladder design between August and October went well, with the bags remaining intact at impact velocities up to 26 m/sec.56 Combined with the successful parachute tests in January, these ended the hardware redesign effort for the EDL team. They didn’t end the team’s work, though. They still faced a number of significant hurdles, starting with the master EDL simulation, which they needed for the systems test program and for site selection, which was supposed to conclude in March 2003. The master EDL simulation wasn’t ready in time, though, so the workshop focused instead on generating a science priority ranking, resulting in confirmation of Meridiani Planum and Gusev Crater as the highest priority sites.

By February, Lee’s team had the master simulation running well enough to begin analyzing the EDL system’s performance at each of the four remaining sites. These showed roughly what the engineers had expected: the Meridiani and Elysium sites were statistically indistinguishable at 95–96 percent probability of successful landing, while the Gusev and Isidis sites had 89 percent and 91 percent probabilities, respectively.57 Higher winds at Gusev were the culprit, again as expected, while the Isidis site had lower winds than Gusev but more rocks. Their work gained Pete Theisinger’s agreement to recommend Gusev Crater as the primary landing site for MER-B (which they expected to launch first) and Meridiani as the primary site for MER-A.

The next step in site selection was a peer review scheduled for the end of March. NASA had formed an independent review committee to rake through all the work done by the project’s engineers and by Matt Golombek’s two years of science workshops. The chairman was Tom Young, and he threw a few more monkey wrenches into the process. He thought that the Meridiani site was too high-altitude for the qualification limits on the parachute. So to save Meridiani, the team had to promise to requalify the parachute for larger opening loads. That, given the three months left before launch, meant doing the tests after launch.

Young liked Gusev Crater even less. David Kass had produced a set of “worst case” wind profiles for the EDL simulation effort, and when these profiles were fed into the master simulation, they had generated terrible results. The landing failed nearly half the time. Worse, the results were so recent that Rob Manning and Wayne Lee couldn’t explain why the system was failing so badly. So the end result of the peer review was that Meridiani was conditionally approved for MER-A, depending on the results of the post-launch parachute requalification, but Gusev wasn’t. At launch, MER-B would be targeted so that it could go either to Gusev or to the “wind safe” site that Squyres hated, Elysium Planitia, with the choice to be made before August 1.58 But if only one of the two vehicles launched successfully, it would be sent to Meridiani to maximize the chance of success.

After the review, Manning and Lee dove into the master simulation’s reams of data to try to understand the system’s behavior under the “worst-case” winds. They concluded that TIRS was the problem: it was overpowered, an artifact of the team’s decision to adopt the “high performance” parachute design after having chosen the larger TIRS motors to support the less stable, but stronger, brick parachute. The worst-case winds weren’t actually all that bad, and the greater stability of the “high performance” parachute they had actually chosen meant that the backshell misalignment generally wasn’t that large when TIRS fired. But because the TIRS motors had a fixed size, their firing shoved the backshell too far in the opposite direction. Then, when the RADs fired, they accelerated the spacecraft too much in the direction opposite the winds. The net result was airbag failure on impact. Manning ruefully characterized TIRS as a hammer that turned out to be too big.59

It was too late for the project to order a smaller TIRS rocket. So in May 2003, the EDL team started working on what they decided to call “dual TIRS.” A new set of algorithms would decide whether to fire the TIRS rockets 0.2 seconds after firing the RADs (in the event of very high winds) or 1.1 seconds after firing the RADs (in the event of only moderate winds). The delayed firing reduced the effective velocity correction produced by TIRS from about 23 m/sec to about 11 m/sec.60 It was yet another complexity added on to a design that, years before, had been chosen for its simplicity.

By the time MER reached the launch pad, the old Pathfinder heritage was long gone. Its EDL system was entirely new (they even redesigned the mechanism that lowered the lander down the single bridle), and much more complex. They’d been driven away from Pathfinder by higher masses, by their desire to make the Gusev landing site, and, most importantly, by their increasing realization of how sensitive terminal descent was to the martian wind. Their only real alternative to redesign had been to accept higher risk of failure. Both the 95 percent success criteria and the political environment made accepting greater risk impossible. In addition to the painful Climate Orbiter and Polar Lander losses burned into the minds of everyone at JPL, in January 2003, space shuttle Columbia had disintegrated over Texas during reentry, killing its crew and further focusing NASA on risk. MER couldn’t fail.

Mars Exploration Rover costs (millions of U.S.$)

All the changes cost money. Pete Theisinger had to ask for additional funds twice. In October 2001, shortly before the project’s critical design review, he’d asked for $50 million. JPL Mars Exploration director Firouz Naderi and NASA’s Mars program manager Scott Hubbard reprogrammed the money from the Mars budget, cutting funding for some technology developments and from the Mars Smart Lander, effectively postponing it from 2005 to 2007. After the ATLO readiness review at the end of January 2002, Theisinger had to ask for another $50 million. Glenn Cunningham, who had chaired NASA’s independent review board, caused a lot of heartburn on the project by recommending to NASA that the second rover be dropped so that the overworked project could focus its attention on doing one rover well (and so NASA would save the $65 million cost of the second Delta II, which could be sold).61 But after a tense meeting in Washington on March 26, 2002, Ed Weiler, the associate administrator for space science, approved the second funding increase.62 That second increment came with a warning that there would be no more—next time, the project would lose one of the rovers.

ATLO

The redesign of nearly everything during 2001 meant that the project’s assembly, test, and launch operations phase didn’t start on time. Scheduled for late January, it slipped into February and finally started March 6, 2002. The delivery needed to start ATLO was the spacecraft’s “brain,” the rover electronics module, because everything else had to be built around it. The ATLO manager, Matt Wallace, a former nuclear engineer and member of Pathfinder’s ATLO team, knew by the end of January that he wouldn’t be able to get one of the two flight REMs even by early March. He thought that getting the ATLO team working on realistic hardware mattered more than having the actual flight hardware. The ATLO team needed to get familiar with the complex procedures necessary to assemble the MER spacecraft, and they needed to check out the operation of the ground support equipment and cabling required for the testing procedures. So he decided to start ATLO with an engineering model of the module that would eventually become part of one of the project’s testbeds. It showed up at JPL’s high-bay clean room on a Friday afternoon. In the first of many unpopular decisions forced by the schedule, he made the team work that weekend.

Wallace, Richard Cook, and Pete Theisinger had also understood months before that they couldn’t hope to get both their vehicles to the launch pad on time without far more work hours than were possible on a normal weekday schedule. They’d baselined dual-shifting for the entire duration of ATLO. Each shift worked ten-hour days, five days a week. But Wallace didn’t schedule weekend work in advance—weekends were part of his schedule margin.

But even dual-shifting through the entire yearlong ATLO wasn’t enough time, as delivery dates kept slipping. Sometime in December or early January, Wallace had concluded that there still wasn’t enough time to complete the entire testing sequence on both vehicles. On Mars Pathfinder, the ATLO progression had been to do functional testing on each piece of the spacecraft as it arrived, integrate the piece, and make sure it connected properly. For electronics, the ATLO team also had to make sure each component communicated properly with all the other electronics it was supposed to. Then, when the entire vehicle had been assembled—“stacked” is the proper term—it went into a cruise thermal vacuum testing phase in JPL’s 25-foot space simulator. The spacecraft was commanded to do all the things it would have to do during its flight to Mars, so the cruise thermal vacuum test served as both a thermal design test and a system-level test. Anything seriously wrong with the design from either standpoint should show up.

Because Pathfinder had also been a lander, though, it had to go through a Mars surface environmental test. So after the cruise thermal vacuum test, it had been removed from the chamber, torn back down to the lander with its rover, and then while the cruise stage and aeroshell were reworked, the lander and rover went through the thermal vacuum test again, this time being commanded to do their surface mission activities. Finally, the vehicle had been shipped in several pieces to Cape Canaveral, where the radioisotope heater units and batteries had been inserted into Sojourner, Sojourner had been sealed up in the lander, the airbags were attached, and the whole thing was put back inside the aeroshell before being attached to the cruise stage.

And that’s what Wallace had planned to do with both MER spacecraft, initially. But shortly before the ATLO readiness review in January, he realized it was no longer possible. After losing the first two months of 2002 due to late deliveries, he didn’t have enough calendar time, let alone work hours, to complete two sets of environmental tests. He remembered that JPL used to build two spacecraft identical spacecraft at a time routinely, although it hadn’t done so in a generation, and he started to wonder if his predecessors had bothered doing the full set of tests on each. It really didn’t make sense to do the whole suite on both, as many of the tests were of the design, not of the workmanship of the individual pieces. He couldn’t see any reason to do the design-oriented testing on both vehicles, as they were the same design.

And he found that those predecessors had thought the same way. JPL’s engineering directorate maintains a library of old project documents that he consulted, and he went and talked to the old-timers who were still around to understand their reasoning. What they had done was to build a third spacecraft, which they called a “proof test model” and was never intended to fly, and subjected it to the full set of tests. The two flight spacecraft got a much smaller subset of tests to assure that they would behave just as the proof test model had. They had worked this way so that all the mistakes got made on the proof test model, not the flight spacecraft, and because some of the mechanical tests were potentially damaging. As the planetary budget shrank beginning in the late 1970s, both the model and the duplicate spacecraft—seen as luxuries in Washington, DC—went away. So the idea had fallen out of use.

Wallace couldn’t have a third spacecraft built up as a proof test model—although since Theisinger, like Rob Manning, believed that being “hardware rich” would actually make the test program go faster overall, he did have the project build three complete rovers, a fourth mobility test rover, a spare lander chassis, three sets of flight avionics, and two sets of nonflight avionics for the test beds—but he did see that his predecessors had put some thought into how to test multiple spacecraft efficiently. Their process had been, in essence, to put the first spacecraft through every test, and to figure out what minimum set of tests was necessary to prove that the second one was enough like the first that it would behave the same way in space. So what was that minimum set of tests?

Wallace asked Manning. Manning arranged a meeting out at his house on December 18, 2001, where they, Richard Brace, Richard Cook, Dara Sabahi, and a few others went through the whole test program to figure out what they had to do to convince themselves that the second spacecraft would be basically sound.63 They couldn’t go quite so far as not performing thermal vacuum testing on MER 2 at all, but they realized that they didn’t need to subject the rover and lander to both the cruise thermal vacuum test and the Mars surface testing. They could put the lander and rover through the surface thermal vacuum test to prove that its thermal performance was as predicted by the thermal models (and also just like MER 1); having verified the model was right, they could put the cruise stage and aeroshell through cruise thermal vacuum testing without integrating the lander and rover. They could simply put one of the engineering model REMs into the aeroshell on an easily made test fixture to provide the “brain” needed for the testing. They called this the “cruise stage / aeroshell subassembly” approach, or CSAS. This CSAS test would verify the thermal performance of the cruise assembly. And they’d ship the two big subassemblies to the Cape separately, first fully assembling MER 2 in Florida.

This produced big manpower savings: about 3,000 work hours at JPL.64 The key to the savings lay in realizing that both vehicles had to be shipped to Kennedy Space Center in pieces anyway. The rovers had radioisotope heater units buried under their batteries, and the Department of Energy would only deliver them to Kennedy, not to JPL, so building both spacecraft up at JPL meant having to tear them almost all the way back down again before shipping so the heaters could be inserted in Florida. Not doing a complete buildup of MER 2 at JPL produced the huge savings. In fact, it meant that MER 2 would ship before MER 1 and be launched first, at least if nothing went horribly wrong in testing.

Wallace didn’t have a hard time convincing Theisinger, who knew they were in deep schedule trouble. He had a little more trouble with JPL’s director of flight projects, Tom Gavin, who made him have the plan vetted by some more JPL old timers before agreeing to it. The independent review board accepted its necessity but recommended canceling one rover instead; as noted above, NASA’s Ed Weiler approved it anyway.

Wallace got the first flight rover electronics module on August 9, three weeks after his “drop dead” date; he got the second one in mid-September. The second REM also caused him the worst moment he remembers during ATLO. He was driving home one evening when the vehicle’s test conductor called him on his cell phone. “Man, Matt, we just blew up the REM.” He turned around and went back to the lab, thinking they’d finally lost one of the rovers. There was no time to build another REM. It turned out not to be quite that bad, however. The test team had connected power to a place it shouldn’t have gone, causing some sparking and smoke. But the actual damage turned out to be repairable.65

The REM “explosion” wasn’t the project’s only near-disaster that got resolved after much heartache. The solar array vendor for the rovers went bankrupt, and the team had to find a second source that could deliver on very short notice. The composite fuel tanks that were supposed to save them mass in the cruise stage never appeared; the vendor had lost the personnel who knew how to build them. The cognizant engineer, suspecting this might have happened, had identified titanium tanks that would do and could be obtained quickly, at the cost, of course, of mass (about 7 kg). In June 2002, they became the new baseline.66 The lightweight composite landers arrived extremely late and turned out not be lighter after all. Theisinger ruefully explained later that while the composite structure was lighter, the metal attachment points needed to protect the composite from other metal parts—the rover’s wheels, the electronics boxes, and so forth—wound up raising the mass back to what the aluminum landers would have weighed anyway.67 As a result, the project’s mass margins wavered between zero and negative numbers for most of ATLO.

MER 1 was completely assembled in early October 2002. It wasn’t in perfect shape; its REM had developed problems in testing, but the team needed to keep the system-level environmental tests on schedule. The ATLO team put the vehicle “stack” through a set of vibration and electromagnetic tests prior to sending it up to JPL’s space simulator for two weeks of cruise thermal vacuum testing. MER 1 passed the thermal testing with only some minor problems (several heaters and thermostats were miswired, and some of the thermal control software was buggy), a considerable relief to the team.68

The MER 2 rover was fully assembled by October, too, and was intact enough by mid-September to start taking pictures of its surroundings in the assembly bay—even one of MER 1’s cruise stage hanging nearby. It performed the first surface thermal environment test starting December 11. The testing again showed that the thermal design was sound—within 3°C of its predicted performance, in fact—even though it revealed a host of other problems. These were gradually, and somewhat painfully, fixed over the couple of months left before MER’s pieces had to start making their way to Florida.

The first increment, MER 2’s cruise stage and aeroshell assemblies, had gone at the end of January 2003 with about 20 people from JPL; MER 2’s rover and lander, and MER 1’s cruise stage and aeroshell, had gone the last week of February with about 35 more engineers and technicians. MER 1’s rover went with about 15 more people.69 The first week of March marked the last shipment of MER hardware to Kennedy Space Center. The launch period opened May 30; to get all this done in time, Wallace had scheduled staff for two shifts, seven days a week. His “margin” by this time was the potential to go to three shifts, but that was it.

Mars Exploration Rover (MER) management posing with MER-1 in JPL’s 25-foot Space Simulator, the Lab’s largest solar thermal vacuum chamber. Left to right: Rob Manning, Richard Cook, Matthew Wallace, Richard Brace, Peter Theisinger, Barry Goldstein.

JPL image D2002_1106_B54, Courtesy NASA/JPL-Caltech.

By this time, the ATLO team knew that all the efforts to keep the vehicle mass down had paid off. The final launch mass for MER 1 was 1,064 kg, including 52 kg of fuel; the team had projected 1,078 kg for the launch vehicle manufacturer, so they wound up having to add ballast to raise the mass for launch. The entry mass turned out to be 832 kg, and the touchdown mass 539 kg. The launch mass had grown 20 percent above Pathfinder’s, while the landed mass had increased 47 percent—the team packed a lot more stuff into the same volume and taken a lot of mass out of the cruise stage, mostly fuel and “excess” aluminum from the structure, in partial compensation.70

Two days of thunderstorms delayed MER 2’s launch to June 10. But the launch itself was uneventful. MER 1, however, suffered a lengthy delay due to a bizarre launch vehicle problem. Cork insulation on the outside of the first stage hadn’t been applied correctly. The launch vehicle manufacturer’s engineers scraped it off and applied new cork. But that wouldn’t stay on. The second launch window opened June 25, and they struggled with the cork until July 5, when a new problem emerged with a battery. They finally gave up on the cork, accepted the risk it might fall off, replaced the bad battery, and got MER 1 on its way to Mars on July 7.71 The rovers were renamed Spirit (MER 2) and Opportunity (MER 1), the result of a naming competition that drew nearly 10,000 entries.72

Conclusion

The Mars Exploration Rover project had started from a false assumption, that the fairly simple Pathfinder EDL system could be reused without much change for a much different mission. That assumption might have been true if NASA, and Steve Squyres, had been willing to accept a shorter “required” mission length, more risk of a failed landing, or less windy landing sites (or some combination thereof). But they hadn’t. Instead, mass growth in the rovers needed to provide the proper thermal conditions for a longer mission, combined with the team’s desire to get Squyres’s instruments to the attractive but windy Gusev Crater landing site, produced rapid evolution of the EDL system into something rather more complex.

The redesigns and delays led to cost growth. MER overran its development budget by 17 percent. The money mostly came from delaying the Mars Smart Lander mission, whose engineers were pulled in to help MER out of its schedule crises anyway. NASA policy toward cost had changed substantially since the loss of Surveyor 1998; Pete Theisinger had put it this way during CDR: “The Project’s approach is to spend money to maintain schedule and product quality—and the Program Office and JPL management support that.”73

MER’s development had consequences for later projects beyond funding. The airbag test horror show triggered a major change in the Mars Smart Lander effort. When Tom Rivellini went back to Smart Lander after helping MER out with its airbags, he started advocating landing on the wheels, the idea he and Dara Sabahi, Rob Manning, and Steve Jolly had back in 2000.

Launch hadn’t ended the MER team’s work. They still had testing to do on the airbags, backshell, and parachute. Their cruise software was complete, but the EDL and surface operations software were not. Many members of the development team had to be retrained for operations, although that retraining had to be delayed until the testing and software was finished. And they still had a huge amount of systems testing to do on their testbeds. They had a busy seven months before their first landing day.