Sunday, January 11, 2009

Great book - check it out!

One of my Christmas presents this year was a great little paperback titled 'Bad Science', written by a chap named Ben Goldacre. He's a doctor (of medicine) and writes the Bad Science column in the British daily newspaper The Guardian.

The book is well written, laugh out loud funny and confronts a number of myths created by pseudo-science groups (including the complementary medicine and nutrition industries) and perpetuated by a credulous, uninformed or just lazy mainstream media.

Among other things, it lays to rest furphies such as the supposed link between autism and the MMR (Measles Mumps and Rubella) vaccination, or scare stories about mercury amalgam fillings. In neither case was there a scientific basis for the scares, but that didn't prevent well-meaning (for the most part) activists and a compliant media from beating them up and scaring the living daylights out of countless parents.

Author Ben Goldacre's basic thesis is that there's a good reason why the scientific method has developed over the years. His acid test of a new medical 'scare' story or 'miracle' cure is simple: if you read about it in a newspaper or magazine, or saw something about it on TV, then you should be able to track down a peer-reviewed scientific paper setting out the basis for the claim. If you can't, then be very wary because the story was probably fed to the media by a snake oil salesman of some kind.

If somebody isn't prepared to subject his scientific research to peer review then the research probably wasn't scientific in the first place, and therefore the effect of the snake oil is probably largely psychosomatic. In discussing scientific method and the conduct and reporting of research, Goldacre even manages to make statistics and the placebo effect sound interesting.

On the issue of MMR the scientific evidence, based on properly conducted research by a variety of individuals and bodies working in isolation from each other across the world, is that there is absolutely no link between the MMR vaccine and autism. That hasn't been widely reported, however, and the lingering fear of vaccination created by a lengthy and sometimes hysterical campaign by a number of British newspapers and TV channels is such that the incidence of Mumps and Measles in the UK has risen.

Ben Goldacre proves that science isn't boring: people just don't realise the part it has played in shaping our lives and abandon rational analysis when confronted with scare stories or improbably good news.

That said, I'm perfectly happy to conduct my own long term research to validate claims that red wine and dark chocolate deliver both subjective and objective health benefits when taken in therapeutic doses...

Go on, buy this book - you'll enjoy it!

*Bad Science, by Ben Goldacre, 4th Estate, London 2008

Happy New Year - same old argument

Happy New Year, everybody! This is supposed to be the year in which the Australian cabinet considers 2nd pass Approval for Project Air 6000 - New Air Combat Capability, in other words the purchase (or not) of the F-35A Lightning 2 Joint Strike Fighter (JSF).

So it's no surprise that strident lobby group Air Power Australia put out a media release last week asserting that recent research shows the JSF's stealth qualities are useless against the new generation of Russian long-range air defence misiles and their search and fire control radars. Coincidentally (or not) on the same day the Federal Liberal MP Dr Dennis Jensen, wrote an opinion piece for the Australian Financial Review criticising the JSF, and the Department of Defence for wanting to buy it.

As well as questioning the cost of the aircraft (and either misunderstanding or mis-representing the various cost figures which are floating about at present), Jensen makes two key assertions: the first, based on Dr Carlo Kopp's Radar Cross-Section (RCS) and stealth performance modeling work for Air Power Australia, is that the JSF is not stealthy; the second is that the RAAF, Lockheed Martin, the Pentagon and others are resorting to military secrecy to conceal the fact the JSF is not stealthy. To be quite blunt, he's accusing these organisations of lying to taxpayers and parliament about the capabilities of the JSF.

Jensen states: "Despite what Defence and others may suggest, the basics of capabilities and technologies are in the public domain, and the JSF has no "secret" capabilities, the fundamentals of which are not known in the public domain." He adds, "Be careful when you hear an exhortation that the capability is fantastic, but secret." I'll get to Jensen in a moment, but because he bases much of what he says on Dr Carlo Kopp's analysis, I'll discuss Kopp's work first.

I'm no radar expert, so I'm no position to question much of the mathematics and theoretical physics contained in Dr Carlo Kopp's various papers on platform stealth. I would note, however, that his information on the F-22A Raptor (which he considers to be extremely stealthy) and the F-35A Lightning II (which he considers to be significantly less stealthy, especially in the rear quadrant) is derived from drawings and photographs in the public domain. In his assessment of JSF Defence Penetration Capabilities, Annex C - Joint Strike Fighter Lower Fuselage and Nozzle RCS Modelling (which was the study cited by Dr Jensen), Dr Kopp presents a wireframe model of the JSF lower fuselage, with the caption: "Wireframe rendering of the solid model for JSF lower fuselage geometry employed for RCS modelling. This model accurately represents the complex singly curved section of the lower centre fuselage, but does not represent the longitudinal taper or the problematic doubly curved shapes at the weapon bay and ventral blister transitions. The model was produced by digitising a section from a photograph and after scaling, extending the section into a solid using a custom C language program (Author)."

Such models, derived from public domain photographs and line drawings, seem to be the basis upon which Dr Kopp analyses the RCS and stealth capabilities of these aircraft. As far as I can determine, these models are approximations, albeit relatively faithful, of the real shapes of these aircraft so any RCS data based on them will, by definition, be problematic though to what degree it is hard to say.

Furthermore, it appears Dr Kopp's model of F-35 design features such as the underside of the fuselage and jet nozzle geometry focuses on the shapes of these features and does not take into account attributes such as stealth absorbent coatings, nor the materials from which these elements are fabricated, nor other RCS-attenuating shaping or surface treatments.

That said, he has been diligent in examining what he knows, or can infer, about the design of the JSF and this level of scrutiny can provide an important reality check on some of the raw figures and basic assumptions about the design of the aircraft. It's unlikely that Dr Kopp is the only analyst conducting work of this kind, but he seems to be one of the few to have used his work as the basis for such sustained, vehement criticism of the JSF design.

What troubles me is Dr Kopp’s apparent sense of infallibility: he appears to assume that his shape models of the F-35 and F-22 are sufficiently accurate that he can use them to model the RCS of these aircraft with enough precision to predict their real-world performance. His models may be representative of the RCS of these aircraft, but they may not provide a safe basis for detailed criticism of the aircraft design.

In addition, his modeling appears to be based solely on the shapes of the various structural elements he examines. Notwithstanding an allowance in some of his calculations for radar-absorbent coatings, Dr Kopp’s predictions of RCS and stealth performance seem to ignore other factors contributing to the reduction of RCS. Therefore his assessment of the operational effectiveness of the aircraft may also be unsafe. Undeterred, however, he questions the analytical work undertaken by others: “As with claims made for Joint Strike Fighter air combat capability, claims made for the Joint Strike Fighter concerning the penetration of IADS [Integrated Air Defence Systems] equipped with modern radars and SAMs are not analytically robust, and cannot be taken seriously.”

Lockheed Martin’s response to Dr Dennis Jensen’s article, written by vice president Tom Burbage, addresses the issue of Dr Kopp's RCS modeling and states in part: “Taking the emotion out of the continual criticism from Mr [SIC] Kopp’s group, the technologies of stealth have evolved dramatically from a very basic theoretical understanding of the phenomenology in the early days of F-117, B-2 and A-12. In those days we were limited to rudimentary analysis tools, were very challenged from a computational standpoint (I can recall large CRAY computers running for weeks to calculate facet angles on the F-117) and we had very little experience integrating sensors. All of that early development was done by Lockheed engineers and we spent a lot of internal R&D money to develop the technology. Additionally, we harvested all of the lessons of those programs and today F-35 has the benefit of that plus the benefit of using real US Government and industry experts to oversee our design and development. Today, every element of the airplane’s design is evaluated and integrated to ensure that the design is lethal and survivable. We use internal and external shaping techniques, mature propulsion concepts, advanced aperture and sensor installation techniques, advanced, supportable, light weight materials and other areas that industry and government have invested heavily in over the last decade.

“Most importantly, our analysis and assessment tools and techniques are extremely robust now. Additionally, our modeling and simulation capabilities to demonstrate the effects of these advanced technologies in a projected high threat environment are superb. There have only been three stealth fighters ever built and we have built all three. All of that experience has been applied to the F-35 and our technology, knowledge and experience bases are significantly more advanced today.”

To summarise Burbage’s response: there are techniques and resources not available to Dr Kopp which enable Lockheed Martin and its various government and industry partners to develop a stealthy design for the JSF and refine and validate this using highly sophisticated analysis and assessment tools.

Much of the devil of stealth lies in the detail, hence the requirement for accurate modeling and minute and resource intensive analysis of RCS reduction techniques.

That’s not to suggest projects like JSF shouldn’t be subjected to close scrutiny and I don’t wish to discourage people like Dr Kopp from trying to demonstrate the emperor has no clothes. But Dr Kopp fails to acknowledge a wider body of expertise in stealth technology and therefore presents as incontrovertible fact conclusions based on necessarily limited data.

It must be added also that many mistakenly assume the purpose of stealth is to make an aircraft invisible; this is not the case, and Dr Kopp does understand that. Stealth is designed to reduce, if it can’t entirely eliminate, the likelihood of an aircraft being detected by a sensor of some kind (in this case we’re talking principally about radar), the likelihood of it then being tracked accurately enough for a radar to provide fire control data for a gun or missile system, and then reducing the likelihood of a successful engagement by a missile once this is launched. A stealthy (even semi-stealthy) aircraft derives an important advantage from the deployment of counter-measures: the smaller the RCS of the platform, the larger the apparent RCS of the decoy and so the more effective this is. That’s a very simplistic example, but it serves to illustrate the point.

This is all highly contextual and raw RCS figures based on a flawed, or at least limited, technical analysis don't reflect the tactical subtleties and complexities of modern air combat.

The important measure here is the tactical effectiveness of the aircraft and its weapons, and stealth is just one of the components of effectiveness, albeit in the case of the F-22A and JSF a highly important one.

Defence, Lockheed Martin and the Pentagon appear not to have engaged Air Power Australia and Dr Kopp on the detail of the JSF’s stealth performance for two reasons: firstly, they would reveal more than is safe about the capabilities of the aircraft (and that assumes, as I do, that the JSF is indeed a very stealthy platform – more of this below); and secondly, it doesn’t much matter what they say, in any case. Air Power Australia and Dr Jensen have an unambiguous agenda – to persuade the Australian government to acquire the F-22A; to achieve this they need to highlight any and all of the JSF’s real or perceived deficiencies.

The Head of Defence’s New Air Combat Capability (NACC) project, Air Vice Marshal John Harvey, delivered a by-now characteristically restrained response to Dr Jensen’s article in the Australian Financial Review, noting: “[Kopp’s] analysis of the JSF’s stealth characteristics is flawed based on a number of incorrect assumptions, simplistic modelling, lack of operational analysis and lack of knowledge of sensitive performance information.”

Dr Jensen, despite claiming to have been a research scientist at both CSIRO and DSTO, does not seem to have applied any scientific method to his argument.

After making plenty of assertions but without presenting any supporting data except a reference to Dr Kopp’s analysis for Air Power Australia, Dr Jensen stated: “In short, the JSF is an expensive aircraft, with very limited aerodynamic performance compared to legacy fighters, let alone other advanced fighters. The stealthiness of the aircraft has been shown, with hard numbers, to be poor compared to real stealth aircraft, and its much vaunted networking capability further degrades this.”

Dr Jensen leaves himself open to challenge on almost every aspect of that statement. If he is as interested as he claims to be in this topic there is plenty of credible information in the public domain about the cost and platform performance of the JSF (with the exception of its stealth capabilities, of course), and enough to carry out some theoretical modeling of aircraft performance. His article in the Australian Financial Review was the platform for a poorly constructed attack on the JSF program. In mounting that attack he damaged his own case.

As far as the stealth capabilities of the JSF are concerned, I consider it highly unlikely that anybody associated with the project will disclose detailed RCS data so I need to draw my own conclusions from what I can infer about the project and the behaviours of the various customer and stakeholder groups.

Yes, it is possible that the JSF is the biggest hoax in aviation history: but at US$19 billion (the approximate cost of the System Development and Demonstration, or SDD, phase) it’s a bloody expensive joke, and the Pentagon simply doesn’t have a sense of humour.

Secondly, it is possible that Dr Kopp is the only analyst who has uncovered the hidden weakness of the JSF, or the only analyst with the courage to say so out loud. I think that’s insulting to the many thousands of excellent scientists and engineers around the world who have contributed to the design and development of the JSF or who, on behalf of their respective governments, have carried out studies to validate (or not) the claims for it made by the manufacturer and the Pentagon.

If the JSF was a complete dog somebody would have made the case convincingly by now – heaven knows there are enough alternatives on the market whose manufacturers would make it worth somebody’s while, and no government knowingly spends billions of dollars on unsuitable aircraft.

Thirdly, no government which has joined the program has subsequently withdrawn from it on either cost or capability grounds; conversely, at least two other countries – Israel and Singapore – now want to join the program in some capacity. This suggests the claims made for its stealth capability are based on robust and realistic measures: stealth is so much a core feature of the design that a degraded RCS would undermine much of the justification for the project, regardless of the other attributes of the aircraft.

Fourthly, the JSF has already fought and won its first battle: in spite of strong pressure from its Scandinavian neighbour, Sweden, Norway last year selected the F-35A as the replacement for its ageing fleet of F-16A/B fighters. Swedish aircraft manufacturer Saab was offering its Gripen NG fighter and a very attractive industry participation program but in spite of this pressure Norway’s own analysis showed the JSF was the superior aircraft and offered better value for money.

Finally, the suggestion that there is a widespread conspiracy, involving nine governments and 12 separate air arms to conceal significant shortcomings in the design of the JSF doesn’t withstand scrutiny: the program itself has weathered plenty of hostile scrutiny from within the US armed forces as well as the Pentagon, the Senate and Congress, not to mention parliaments, defence forces and rival aircraft manufacturers around the world. There seems to have been no significant loss of faith that the aircraft will eventually deliver the capabilities promised, though plenty of concern about the cost and schedule of the project.

Conspiracy theories tend to evaporate when examined rationally.