Science depends on flight ops, which cannot be known ahead of flight

Putting the cart before the horse

In the study of literature, there is a literary devive known as hysteron proteron, which colloquially is often termed “putting the cart before the horse.” Alas, the way NASA has structured the way it does science experiments often has a similar flavor: requiring the answer to be known before the experiment is performed.

As NASA Principal Investigators (PIs), we are asked to specify requirements for instrumentation and apparatus that the engineers are then supposed to build. The problem is, the scientists often spec instrumentation based on what they can get in the laboratory on the ground, not necessarily what is practical or even possible to fly on ISS, with its myriad safety, operational, power, mass and mechanical shock requirements. Similarly, we are told to define scientific goals, not just a general overview of what we hope to accomplish, but specific criteria as to what would constitute partial, satisfactory and complete scientific success.

But if we could define specifically what would be successful scientifically, then we would know the answer before asking the question—and there would be little point in doing the actual experiments!

Moreover, at present, the science office is a subdivision of payloads, which manage all of the hardware on the ISS. This is a completely different division than flight operations, the pilots who fly the station and manage its communication. So inevitably, if science or engineering requirements are defined from the viewpoint of only one division, it can conflict with what is going on elsewhere. This is, of course, not a specific problem to NASA, but instead symptomatic the challenges of working within any large organization.

Organizational divisions between science and flight-ops are problematic

From a PI standpoint, constraints from flight operations never enter the specification or definitions of the science experiments. We are told to specify what we need, which then the science / payloads office coordinates, and has to try to balance with all of the other experiments and payloads. They are tireless advocates for our cause, for which we are extremely grateful and have received very, very generous allocations of crew time and ISS resources. And through this process we have learned from the many iterations of the BCAT experiment is that it is invaluable to understand what everyone else’s constraints are. Often times, we would specify that we needed two weeks to run our samples, but if the flight schedule only had 12 contiguous days available, we would not be able to use that time, and would have to wait until our nominal requirements (i.e. the two weeks) were available, even if that would not be possible for many months in the future. But from our standapoint, 12 days of data now could be worth a lot more than a “complete” set of data months later—especially because all kinds of things can go awry in the interim (samples may dry out, instruments may get broken, the Space Station itself may have an operational issue).

Alas, because the normal procedures do not involve science PIs in flight operations, these types of tradeoffs are not ordinarily presented to the PI, and instead are negotiated among several constituencies within NASA, who may not have the background to make the decisions that are best for the science. Conversely, unless the PIs have been involved in several successful flight experiments and have worked with the different NASA branches, it’s unlikely that they understand the panoply of constraints that flight on orbit imposes, so it is hard for them to weigh the practical factors.

As a result, we do the best job we can to suggest what we a priori feel is a reasonable operational plan based on our best guesses as to how the apparatus will perform, with full awareness of the wisdom of Prussian (German) field Marshal Helmuth Karl Bernhard Graf von Moltke dictum that, roughly translated, that “no battle plan survives first contact with the enemy.”

ACE-M2 first-round flight ops main goal: define and characterize operational capabilities

The main point of the first runs of the ACE-M2 experiment are to image all of the samples and see if we can actually observe them. If we can’t image any sample in the microscope, then obviously there won’t be much use for future operations. Fortunately, in general we were able to acquire good images of all of the samples, which I describe in greater detail in a later post.

Once this basic ability to image samples has been established, the main objective is to understand the capabilities of our system. This not only includes the physical aspects of the optical system, camera and other instrumentation, but the amount of time different operations actually take when a human is operating the LMM setup amidst all of the mayhem onboard the ISS itself.

Optical capabilities: exploring objectives

The LMM is equipped with several microscope objective lenses: 2.5x, 10x, 20x and 40x air, and the 63x and 100x oil-immersion objectives. There are several filter sets, though we use the Texas Red set for almost all of our samples, whose spectra we checked before launch so that they were compatible with the dyes we use in the particles. There is a single CCD camera, made by a company called QImaging, that uses the Sony ICX285 CCD chip that has been a workhorse for the entire industry for the first decade of this century, though more recently surpassed by much better imaging technologies.

2.5x air: sample surveys

Using the lowest-magnification objective, the 2.5x air, we survey every well at the beginning of each full experiment, with both bright-field and fluorescence, to check on the status of the sample and see quite generally what is going on, i.e. the location and density of the particles. This objective lens has a very large depth of field, encompassing the entire sample well, giving us a very good overview of each sample. These images also form part of the safety checks at the beginning and end of each experiment, to make sure for example that the wells are all intact, not leaking or broken, etc.

10x, 20x and 40x air: sample details

Perhaps the most important objectives are the medium-magnification air objectives. These allow us to zoom into particular areas in the sample, and get a more detailed view. In general, the quality of the images appears to be best with the 10x, and good with the 20x and 40x.

63x and 100x oil: highly disappointing

On paper, the 63x and the 100x should give the highest-quality views, because they at least double the numerical aperture of the air objectives; that is, they should have roughly double the resolution, so that features that are visible but blurry at 40x should be sharper and more clearly defined, with better contrast, with the 63x lens.

These two objectives require the addition of immersion oil, a droplet of which is added by the crew as the samples are loaded. We did a number of different tests, looking at the same areas with different objectives. Surprisingly—and not in a good way—the image quality is invariably worse with the oil-immersion, higher-magnification objectives, than with those that are air-based. This is backward, and should not happen, and demonstrates that something is wrong with the current configuration of the microscope on-orbit. Engineers at ZIN are working on this issue, so stay tuned.

But from a practical standpoint, there is no reason for the immediate future to incorporate any of the oil-immersion objectives. This has the collateral benefit that no oil then need be added by the crew members, or manually moved via the objective to different sample wells, giving greater flexibility to the imaging of different sample wells.

Timing individual operations

Once the objectives have been selected and we understand how they can best be deployed to give the images we want, the major task to understand how long different operations actually take:

• starting up the LMM inside the FIR rack
• loading the sample
• moving the microscope stage to the right positions on the samples
• establishing the distance between the objective lens and the sample coverslip
• adjusting imaging parameters on the camera to acquire good images
• saving and downlinking image data
• shutting down the rack

While we have estimates for how long these operations take on the ground, things are often quite different on orbit. And establishing the timing is absolutely critical for planning future operations, all of which must happen in the context of a competition for resources like crew time, machine time, operator time, power, etc.

Meshing with the ISS timeline and infrastructure: LOS

One of the major differences between doing experiments on the ground, in a well-controlled, isolated (university) research laboratory, and the ISS, is the number of other things happening at the same time. In our labs, there may be a few people walking in and out of the room, but generally things are still, power, light, temperature, gases and water are continuously available, and mechnically the building is stable in the absence of, say, high winds or a major rainstorm. Almost none of those factors is true on orbit; the ISS is a flight vehicle, and flies around the earth in orbit, traversing the planet every couple of hours.

The ISS communicates with the ground via a network of geostationary satellites. Unfortunately, this network does not cover the entire earth’s surface, so that every few hours (and often more frequently than that), the Station loses communication, during events known as “Loss of Signal” or LOS. During this time, we cannot access our experiment at all, so either it is running something in an automated fashion that we can script ahead of time, or it must sit there until communications are re-acquired.

So not only is the total time relevant for each operation, but complex experiments must be broken down into a series of smaller sequential operations, either that execute remotely during an LOS event (ideal, because then we don’t lose any time), or must be robust to being interrupted and (re)started again at a later time.

The human factor: reproducibility and robustness

Finally, the operators on the ground work very hard to collect our data, taking 8-hour shifts at the console controls. They have a wide range of scientific background, so that we must be very precise in specifying procedures for data collection and documentation. Moreover, the procedures have to be simple and reliable enough to be executed without error by an operator working all day, even as she gets more tired and might overlook something.

Major results from the first run: a flight-ops plan for upcoming experiments

So the most important results from the first run are operations related, once we established we can image all of the samples. These procedures allow us to determine what to measure going forward, within the envelope of the resources that we have. After four 48-hour days to survey all of the samples, the remaining time will focus on the handful of colloid-polymer mixtures (samples 20, 21 and 22), and imaging them over time. We therefore expect to do the following for each sample, which based on the first-run operations will feasibly fit into the timelines we have requested:

1. Image all wells with 2.5x air objective. This is fast, easy, simple and gives on overall status check for safety and scientific reasons.

2. Create full tiled images of the entire well with the 10x air objective. This is a fast operation that gives us a properly-detailed assessment of each sample of interest; thus far, this lens arguably gives us the best images.

3. Zoom into selective areas and image with the 20x and 40s air objectives. This allows us to focus in more on the evolution of microstructure, guided with the overall tilings from the 10x.

You will see this structure going forward for the observations of time-dynamics of these colloid-polymer mixtures. Note that this specific set of steps would never have been developed from entirely ground-based studies. Instead, we developed these procedures based on real experience aboard the ISS.