JASSM bootstrap reliability

The Joint Air-to-Surface Stand-off Missile has been in the news lately. It's the same old story that's been going on ever since it was "fielded". A few missiles fail to function properly in testing, the Air Force says Lockheed-Martin needs to bring up the reliability or the program will be terminated. Lockheed-Martin agrees that more work needs to be done ("keep the money flowing") to improve the reliability, and they'll work diligently with their Air Force team members to serve the warfighter.

That's military-industrial complex business as usual, not that interesting. The neat part is that there's enough information in most of those press releases and news articles to do some resampling statistics on the reliability of the missile. So, in the interest of supporting "an alert and knowledgeable citizenry" (see the Eisenhower video at the bottom of this post) here are some bootstraps on JASSM reliability.

It is really easy to do in Octave using the empirical_rnd() function.

n_09 = 19; % number of tests, based on stated 79% success rate
f_09 = 4; % number of failures
nbins_09 = 14;
t_09 = ones(n_09, 1);
t_09(1:f_09) = 0;
boot_09 = empirical_rnd(t_09, n_09, nboot); % really easy to do
% bootstraps
reliability_09 = sum(boot_09, 1) / n_09;

The sample size n_09 and number of failures f_09 are based on the recent Reuters story:

Four JASSM missiles tested in November, January and February did not detonate on impact or had other problems, raising fresh questions about the program. But the missile still had a reliability rate of 79 percent, and was on track to reach the 90 percent goal, the Air Force said. --Reuters, 6 Jul 2009

So, according to the recent reports JASSM should work about four times out of five (the AF wants it to work nine times out of ten). Here's the "4 out of 5" reliability distribution (based on the bootstrap shown above) with 19 samples:

Back in 2004 JASSM had a claimed reliability of 76%:

We have had 29 launches of JASSM and we have a 76 percent success rate. --Judy Stokely, deputy of acquisition at the Air Armament Center

Which means it should work three times out of four. Here's a "3 out of 4" missile's reliability distribution with 29 samples:

The sampling distributions are too large to measure a change as small as the difference between 0.76 reliability and 0.79 reliability, so it's statistically the same missile now that it was back in 2004. In both distributions the desired 0.9 reliability is out on the tail of the distributions, i.e. you can't claim it's a "9 out of 10" missile with much credence.

A more interesting question is what sort of sampling distribution would you get from nine successes and one failure (the desired nine out of ten missile)?

With a sample size of only ten it would be pretty hard to tell the difference between a "9 out of 10" missile and a "4 out of 5" missile. Based on the press releases apparently a "4 out of 5" missile is unacceptable, but what level of confidence will the Air Force place on knowing that they have 0.9 reliability?

Eisenhower on the military industrial complex:


Here's the Octave file with the code for doing the bootstraps.

Also, thanks to Michael J.T. O'Kelly's bootstrap.py for showing how easy it is to resample with replacement from an array in Python using SciPy.