Microsoft word - rg&a perspective 2008

d e n i n g y o u r p o i n t o f v i e w Richard Bach, in his book Illusions, states a handy aphorism: Perspective – use it or lose it. This periodical shares
amongst recreation and tourism management professionals, and others, several tools and concepts which will help exercise your perspective. This issue considers measuring the past and predicting the future. It is also the tenth newsletter, Perspective is distributed by Rob Greenaway & Associates as a service to the recreation and tourism industries.
Uncertain certainty
If you’re not into statistics, read only this sentence: If you A quoted example is Vioxx. Taking this anti-inflammatory encounter a report which states that some result is drug had the unfortunate side-effect of increasing the ‘statistically significant’, ask the author, who cares? Or, chance of a heart attack, and it was withdrawn. Original as Nassim Taleb would say, just ignore them, or try to research into Vioxx by the manufacturer showed that, in put a rat down their shirt.1 Still curious? Read on. the test sample, only five patients taking the drug had heart attacks. In comparison, the control drug (another The problem with statistics is that it’s easy to be NSAID) was associated with only one. However, both intimidated. Most of us would agree that x2=26.8, df=1, results were statistically insignificant, no matter that p<.001 is pretty daunting, particularly when someone Vioxx apparently caused five times the acute myocardial who should know better is trying to convince us that this infarction as the control (it turned out to be eight times). statistical advice proves that an apparent difference between The original Vioxx researchers comfortably stated that there two data points is important: we should take note – it is was no difference between the two medications in relation to the risk of heart attack. The data were irrelevant – their lack of statistical My response, in the social sciences, has always been that if the significance was purported to be the only necessary measure. difference between the two raw data points doesn’t make your mouth Even my tear-stained copy of Statistics Without Tears states, “… in gape, or at least raise your eyebrows, then it’s up to the researcher to statistical thinking, ‘significant’ does not necessarily imply ‘interesting’ prove that the difference matters – not just that it might exist. I say or ‘important’…. For instance, suppose two related experiments each this even if the researcher’s job is merely to report ‘the facts’ as produce a difference, one that would have a 4.9% chance of arising revealed by the data (it should never be). To this end, I have a pet merely because of sampling variation, and the other a 5.1% paragraph that I append to peer reviews of social surveys where tests chance.… In one case it would be labelled ‘significant’ and in the other ‘not significant’…. Perhaps, in evaluating such data, it is best to Differences between data which are described as ‘statistically let the figures speak for themselves.”3 significant’ do not indicate that they are meaningful to However, Rowntree’s only alternative proffered solution – to letting management. Managers using the report will need to confirm the the figures speak for themselves – is to apply more stringent levels of scale of effect, or difference of opinion or experience, that will significance to avoid Type 1 error (accepting a difference as significant when it is not. Type 2 error is the opposite, failing to Therefore, it was with glee that I read econometricians Stephen Ziliak and Deirdre McCloskey’s text, The Cult of Statistical Significance – Ziliak and McCloskey go much further. They liken the familiar how the standard error costs us jobs, justice and lives.2 The authors significance tests to bloodletting. There are more meaningful revile any faith in the puffed-up spawn of the normal distribution alternatives. Importantly, there is the role of the researcher in, “…thinking about your coefficients in a currency of How Much in the Their key anxiety relates to what they call the ‘sizeless stare’, which is world as it is, or could be, and persuad[ing] a community of scientists nurtured by significance tests. The fundamental issue is not that a [that the difference is meaningful]. Instead of deploying a mechanical result is likely to be the same if the test is replicated (is truly rule about one kind of sampling error you will have to establish a representative of the target population), but whether the research minimum effect size of substantive significance in the relevant range technique is sound and whether the result is important considering of power, for your particular area of research, acknowledging all the the issue at hand. When a researcher says an output is statistically significant (or not), Ziliak and McCloskey yell, WHO CARES? and Significance tests are misleading, nasty things. Report and rank data HOW MUCH? That is, what is the size of the difference, and, is the that are meaningful (not ‘significant’), show confidence intervals and result meaningful to the discipline? Is the result of a drug test review errors. Ask, who is my audience? Why do they need these ‘clinically meaningful’, rather than just ‘statistically significant’? Is an data? Am I a bland statistician or a researcher? Mostly – mostly – the economic parameter important to the economist or to the statistician? t and x and p are a waste of ink. Worse if you square them. ™ 1 See the next page. 2 Ziliak, S.T. McCloskey, D.N. 2008. The Cult of Statistical Significance. 3 Rowntree, D. 1981. Statistics Without Tears. Pelican. p118 R G & A , P O B o x 3 5 8 , N e l s o n 7 0 4 0 . P h / F a x 0 3 5 3 9 4 3 3 5 r o b @ g r e e n a w a y . c o . n z
w w w . g r e e n a w a y . c o . n z w w w . g l g n z . c o . n z
Certain uncertainty
Several years ago in this newsletter I wrote about ‘regression to the So what to do? Taleb has two simple recommendations: mean’ – the propensity for trends to adhere to the normal distribution Be a fool in the right places. Avoid unnecessary dependence on
curve. For large and small peas to tend to spawn average-sized peas large-scale harmful predictions: be fooled in small matters, not in the – not ever-larger or ever-smaller peas: for biological and large. Rank beliefs by the harm they may cause, not social phenomenon to tend towards the mean. according to their plausibility. A problem here This is apparently the case in a country that is that it takes more effort to be sceptical Nassim Nicholas Taleb calls Mediocristan, but than it takes to believe. Humans are not not in the real world – which is increasingly ourselves. Beware of economic and social Taleb describes these two metaphorical nations in his forecasters. Taleb describes them as mere entertainers. book The Black Swan - the impact of the highly Be prepared. Knowing that you cannot predict the
improbable.4 Taleb really doesn’t like the normal future does not mean that you cannot benefit from distribution curve and the complacency it breeds. While unpredictability. Narrow-minded predictions have an, Ziliak and McCloskey are gunning for the curve’s progeny, “analgesic or therapeutic effect. Be aware of the numbing Taleb goes for the foundation, calling it a ‘great intellectual fraud’ effect of magic numbers. Be prepared for all relevant and a ‘contagious and severe delusion’. The curve, amongst other things, severely under-estimates probabilities at the extremes. Taleb was a financial trader in a former life and worked within the His premise is that we tend to look at the future as an extension of the world of prediction errors, applying his brand of empirical scepticism past, presuming such phenomena as regression to the mean to be in to the world’s markets. His investment strategy is similar to his play. We look at the past and derive (induce) a set of rules to apply to recommendations for life. If you accept that most risk measures are the future, although this could lead to some odd and competing flawed, don’t rely on ‘medium risk’ investments – how do you know conclusions: if you survive until tomorrow, it could mean that, either, they are medium risk? Instead put 85 to 90% of your investments in you are more likely to be immortal, or that you are closer to death. the most conservative, safe instruments. The remaining 10 to 15% We tend to manage our lives within the rules of mediocrity and faith in goes into highly speculative areas. Your base investment is safe and the normal distribution curve. Well, it works for dice, and, according to your risk element is exposed to the positive effects of Black Swans – Taleb, it helps if you want to get promoted in academic circles (don’t come up with anything too weird or challenging; have faith in quantitative modelling; apply the old rules – like statistical significance Carrot polarity
In reality, the future will be the product of wildly unpredictable and rat can identify the sharp end of a carrot extreme events, which we will have no way of foreseeing. The world moves by large incremental, random changes. However, once these carrot, not the rat). In a survey of 20 small ‘random’ events have passed – WWI, WWII, the Internet, another share market crash, the discovery of Viagra, texting, no bird flu – we preferred eating their carrots from the tip, will apply a reverse logic to convince ourselves that the unexpected was not such a surprise. Had we access to all the data, and had the data behaved in a ‘normal’ way, we would have seen it coming. The when burrowing beneath the soil? This is fact that all our earlier predictions were incorrect is irrelevant – and the hypothesis of respected evolutionary Neither rat nor rabbit
palaeobioligist Simon Morris.5 However, I have conducted exhausting interviews with a selected sub-population Interestingly, Taleb does not describe future conditions to be the of rabbit owners and have a statistically significant result showing that result of true mathematically random events, but rather, an outcome these extra-subterranean species don’t care where they start nibbling, of unknowable complexity. As with the weather, the future is written in even though they would normally first encounter the blunt end. This the past, but is compounded by our free will. It is too multifarious to puts the ‘first encounter’ theory of carrot polarity to bed. Do you eat compute and might as well be mathematically random. For that your toffee pops by grinding off the top layers with your incisors? (Our reason, predicting the future based on historic precedents or kids are mole rats!) I recommend Constant Vigilance when reading similarities is worthless. Taleb therefore contends that our reference even the most convincing of scientific texts. ™ to specialised forecasting of social and economic trends is bunkum. Beyond the realm of raising perfect children, the most exciting thing to happen this year was reading Ziliak and McCloskey. I got quite giddy and ordered a copy for the Lincoln University library. But it could have been the airport coffee. Otherwise, it’s been the standard string of unusually interesting and challenging projects – numerous energy proposals for six different companies (wind and hydro), several large-scale housing and resort developments, a few irrigation proposals, some recreation user surveys (without a p or t in sight), a little territorial authority recreation planning, and 160 resource consents for helicopter landings. Eighty percent of our projects (with a CI of about 15%) are in the resource management area; preparing assessments of effect for recreation and tourism values with regard to development proposals, and normally presenting evidence in hearings and the Environment Court, as well as preparing the occasional simple resource consent application. This represents quite a shift from my originally dominant area of recreation research and planning for local government – although it remains nice to eat the carrot from both ends, so to speak. Life in Nelson is good, despite this dismal, destructive winter (we lost a large twisted willow – ugly, but strategically located). Importantly, we have solved the local ‘lack of boat’ problem, and have replaced it with the ‘maintaining a boat’ problem. 4 Taleb, N.N. 2007. The Black Swan. Random House. I wonder why 5 Morris, S.C. 2003. Life’s Solution. Inevitable humans in a lonely R G & A , P O B o x 3 5 8 , N e l s o n 7 0 4 0 . P h / F a x 0 3 5 3 9 4 3 3 5 r o b @ g r e e n a w a y . c o . n z
w w w . g r e e n a w a y . c o . n z w w w . g l g n z . c o . n z

Source: http://www.greenaway.co.nz/Library/RG&A_2008.pdf