Doi:10.1016/j.jaci.2004.09.023

Current reviews of allergy and clinical immunology (Supported by a grant from GlaxoSmithKline, Inc, Research Triangle Park, NC) Statistical errors in immunologic research This activity is available for CME credit. See page 22A for important information.
Medical research articles have always been subject to errors in reporting statistical results. Although most of these areminor, they raise questions about the integrity of medical A statistical test is a procedure to determine whether research. Most of the errors come from a misunderstanding a defined quantity is larger (smaller) than you should about the tools used in statistical analysis. This article discussessome of the most frequent errors and provides examples of how expect by chance. The major effect of the test comes from to deal with them correctly. (J Allergy Clin Immunol the way in which the study is designed (ie, set up and executed) to get you to a point where you can do the test.
Although readers might focus a great deal on whether Key words: Statistics, P value, power, errors in reporting, variance a test result is significant, this is really not very importantunless the study is designed properly in the first place. Astatistical test is therefore somewhat different from Recently, published a commentary indicating a laboratory test in which a change of color or light the number and type of statistical errors that she found in intensity might be all that is needed to detect a physiologic reviewing articles in Infection and Immunology. She also change, although even here you need to have proper referenced several articles indicating similar problems in design and procedure to make sure that contaminants have medical journals in other specialty areas. In a related paper in BMC Medical Research Methodology, Garcia-Berthou The indicator that some change has taken place greater and Alcaraznoted that almost 12% of the articles than the level of chance is the P value. There is consensus reviewed in Nature and the British Medical Journal in medical research that a P value of less than .05 means contained P values that did not match the values of the that the statistical test you are conducting is significant (ie, test statistics given. Given that these same problems have that the result you see is not just the result of chance). It is been previously reported at least as far back as and worth noting again that a significant P value does not appear to be ubiquitous, this article presents a more in- indicate a physiologic or chemical change in a process.
depth discussion of the problems, paying particular Instead it indicates that the accumulation of data by attention to how they might affect research in the areas consensus has reached a level that supports whatever change you proposed in your design. The P level you All of the problems hinge on the understanding of what achieve in a study is only one value on a continuous scale a statistical test is doing and what a P value means.
of P values. In immunologic research its closest counter- Therefore this article starts with a brief description. There part is probably a light intensity reading taken from are a variety of statistical tests used for comparison, a microarray in which the .05 level of significance is modeling, prediction, and reduction of data. The errors similar to the 2-fold difference in intensity required to discussed in the article deal with problems when indicate that upregulation or downregulation has occurred.
statistics are used to compare groups, and this article will Focusing on a significant P value provides a certain economy for making decisions when you are examininga large number of situations in which decisions need to bemade, such as examining a number of articles for possible From the Division of Biostatistics, National Jewish Medical and Research However, the difference between a P value of .051 Disclosure of potential conflict of interest: J. R. Murphy—none disclosed.
(nonsignificant) and a P value of .049 (significant) might Received for publication September 9, 2004; revised September 17, 2004; accepted for publication September 21, 2004.
mean nothing at all in terms of the underlying physiology Reprint requests: James R. Murphy, PhD, Division of Biostatistics, National or chemistry. Significant P values need to be supported Jewish Medical and Research Center, 1400 Jackson St, Denver, CO 80206.
with additional logic and secondary evidence to be truly meaningful. It might also be useful to think of the P value as an indication of how likely you are to be able to Ó 2004 American Academy of Allergy, Asthma and Immunologydoi:10.1016/j.jaci.2004.09.023 replicate your current result rather than as a signal that change has occurred. The smaller the P value, the less Methods: Children aged 6 to 17 years with mild to moderate likely the result is due to chance, and the more likely you persistent asthma, taking only albuterol as needed, were characterized during 2 visits 1 week apart before being The power of a study, which is a property of its sample randomly assigned into a clinical trial. At the screening visit, online measurements of eNO, spirometry before and after size, indicates how likely you are to be able to detect the bronchodilator, and biomarkers of peripheral blood eosino- size of difference (change) you have specified in the design.
phils, serum eosinophil cationic protein, total serum IgE, and Taken together, the statistical test, P value, and power can urinary leukotriene E4 were obtained . During a week also be understood in the context of the false-positive and characterization period before randomization, symptoms false-negative rates of a screening or diagnostic test. The were recorded on a diary and peak expiratory flows were statistical test is the test, the .05 level is the false-positive measured twice daily using an electronic device. At the rate you are willing to accept, and the power is the true- randomization visit, eNO was repeated followed by a meth- positive rate of your test if the change you are specifying is acholine challenge and aeroallergen skin testing, Correlations correct. Adopting this view might also help to explain that and rank regression analyses between eNO and clinical a nonsignificant result on a statistical test does not mean characteristics, pulmonary function and biomarkers wereevaluated.
that you have disproved the null hypothesis. It is morelikely to mean that because of inadequate sample size orimprecision in your measurements, you are getting morefalse-negative results than you expected.
It is important to note that if I would like to show that the difference between 2 groups is not significant, all I It is quite true that some of the errors Olsenfound, have to do is to design a study with too small a sample size items 1 to 6 below, could be the result of editorial policies and that result is guaranteed. Conversely, if I want to show that restricted some of the detail given in the original that something is significant, all I have to do is get a large articles. Because removing these errors is critical to the enough sample size and I can prove that. The logic and correct interpretation of statistical results, I hope that science of the process is in selecting a value for change that editorial problems are becoming less of an issue. Recent is meaningful in a scientific or clinical sense and then guidelines for publications in biomedical journals would specifying a statistical design and testing structure that suggest that the type of detail suggested in Olsen’s article allows you to demonstrate this meaningful difference with the smallest number of subjects. The design and testing 1. Failure to document the statistical method used or structure of the study, including the major statistical using an incorrect method. Like any test, statistical tests results, should be found in the abstract at the beginning work best in the specific situations for which they are constructed. Because most statistical tests use a P value of Below I list Olsen’s main problem areas with brief .05 or less as their significance indicator, it might seem like discussions. After each problem, I give examples of they are interchangeable, but differences in the type of acceptable ways to deal with the problem in a succinct data used, the variability of the data, and the study design manner. These examples are taken from articles by Strunk dictate that there is an appropriate test that should be used et al,Asero et Adams et al,Sorensen et and for each situation. Statistical software packages have made Eggesbe et alin recent issues of the JACI. The quotes it easy to try a variety of tests in situations in which they used in the current article are usually taken from a section might or might not apply. Using the incorrect test is similar of the quoted articles labeled ‘‘Statistical methods’’ or to performing a laboratory test without using proper ‘‘Statistical analysis,’’ although Sorensen et aluse the controls. You can get a significant result, but you will term ‘‘Data analysis.’’ All of these articles avoid the not know what it means. If you are uncertain of what type errors listed below and could be used as good examples of test to use, it is best to consult a statistician.
of the correct way to report statistics in the research In his ‘‘Statistical methods’’ section, Strunk et list 2 tests that were used for the analyses in the article. These I have used the abstract of Strunk et alas a good are the Spearman correlation coefficient as a measurement example of what you should find at the beginning of an of the main objective and stepwise rank regression for article. His abstract gives a good, concise description of further modeling. The results, tables, and figures are all the objectives and design of his study. All of the articles clearly related to one of the procedures mentioned.
cited have similar descriptions that immediately allow the Asero et note the test used, a correction for it, and the reader to determine what they should be looking for in the probability level considered significant. ‘‘Proportions article itself. In the case of Strunk et the reader knows were compared by using the chi-square test with the that in the article they should be seeing a discussion of Yates correction. Probability level of less than 5% were correlation, rank regression, randomization, and a discus- considered statistically significant.’’ sion of definitions for mild and moderate asthma.
Adams et alnote that a P value of .05 or less is not the only criteria that they use for significance. ‘‘Multiple Objective: The aim of this study was to find correlations logistic regression was used to assess independent effects between eNO and other characteristics of children with mild in models for anti-inflammatory therapy use. Variables to moderate asthma currently not taking medications.
significant at the 0.10 level in bivariate analysis were entered simultaneously into a multiple logistic regression be significant, no matter how they look. To really make the case that your result is larger than chance requires the Eggesbe et almake a comment about the power of their study for certain outcomes. As mentioned above, making Strunk et alpresent graphic scatter plots in their a note of the power or lack thereof is as important as that suggest a relationship between exhaled nitric oxide determining the significance of the test results.
and certain other variables. However, they make the case The small groups make the study prone to type II errors (ie, for these relationships not on the basis of the graphic but not detecting factors that might be of importance). This on the basis of the statistical test for correlation, which uncertainty affects, among other results, the results concern- they presents in the ‘‘Biomarkers results’’ section.
ing antibiotics. Furthermore, it might be the reason why the Sorensen et alpresent plots with standard error bars in association between egg allergy and cesarean delivery failed which do seem to indicate differences.
to reach significance in the strata of allergic mothers in spite of They make their point, however, by referring to the the strength of the observed association.
significance of the test results in the ‘‘Study data’’ section.
To determine the correctness of these tests, you need to 4. Using a statistical test that requires an underlying determine that they are appropriate for the objective of the normal distribution on data that are not normally study, the design used, and the type of data being used.
distributed. The normal (also known as the Gaussian or Part of the job of the statistical reviewer of a journal is to bell curve) distribution has some very nice properties that make sure that the statistical procedures published in that make it easy to combine measurements and calculate test results and P values. Before the advent of computers, these 2. Failure to adjust for multiple comparisons (multiple properties made it worthwhile to try to transform data so tests of the same data). In the same way that clinical tests that it was normally distributed and to concentrate on tests can produce false-positive and false-negative results, that were created for a normal distribution. It is also the statistical tests can do the same. As in clinical testing, case that a normal distribution might be a good approx- the more statistical tests you do, the more likely you are to imation to how a large group of observations are get a positive (true or false) result at some point. In the distributed. Knowing the distribution (particularly know- clinical setting the potentially misleading results can be ing the mean and variance and their relation to one discounted if they do not fit the developing pattern of the another) of your measurements puts you into a category differential diagnoses. In statistical testing an adjustment of tests called parametric. There are other distributions in is made to the P value’s level of significance so that the this category in addition to the normal, but the normal is more tests you do, the more difficult it is to get a significant the one for which most statistical tests were originally result. Even among statisticians, there is disagreement about when this adjustment needs to be made and exactly Unfortunately, much biomedical research data are not how much of an adjustment needs to be made. There is normally distributed, and using tests designed for nor- agreement, however, that results that are obtained as mally distributed data on this type of data can cause a result of frequent examination of the data are qualita- misleading results. Fortunately, most tests requiring tively different from results that come about because of normality have nonparametric counterparts that make a prestated hypothesis. At a minimum, an author should very minimal assumptions about how the data are inform his audience when this additional data examination distributed. Most statistical packages contain a series of has been necessary to provide a significant result. The nonparametric tests that can be used in place of or along audience is then free to question whether it is likely that with parametric tests. These tests usually have slightly this result could be replicated in a similar study.
lower powers than their normal counterparts, but they give Sorensen et alreport their multiple comparison An alternative to using a nonparametric test is to Post hoc multiple comparisons were only preformed on transform the data so that it becomes more normal. A variables and challenges for which significant differences popular transformation is to take the natural logarithm of between the CFS and control groups were found, including your measurements and create what is called a log normal group-by-time interaction. The Tukey-Kramer multiple com- distribution. Here again there are a number of trans- formations that can be used, and it is important to test the There are several types of adjustments in addition to the results to make sure that the resultant observations are Tukey-Kramer that can be used for multiple comparisons.
normal. You cannot simply assume that the transformation For a very practical discussion of this issue, see Curran- Strunk et identify nonnormal distributions and use 3. Reporting observations without a statistical test.
both nonparametric tests and transformations where they Significance, like beauty, might be in the eye of the are appropriate. The following is from his ‘‘Statistical beholder. Although it is true that there are situations in methods’’ section. Note that the Spearman test is a non- which a unique observation is all that is needed to make your case, these are becoming increasingly rare. Graphic Spearman correlation coefficients were used to assess the representations can be misleading, and large differences relationships among the biomarkers, clinical and pulmonary between groups that come with large variability might not function measures, and allergy skin test reactivity. In this TABLE II. Characteristics of the 144 CLIC participants: Demographics, pulmonary function test and clinicalcharacteristics and skin test reactivity Symptoms during characterization, days per wk Spirometry: forced vital capacity, % predicted Bronchodilator reversibility, % change in FEV1 from Skin test reactivity, No. positive tests of 8 tested (Reproduced with permission from: Strunk RC, Szefler SJ, Phillips BR, Zeiger RS, Chinchilli VM, Larsen G, et al. Relationship of exhaled nitric oxide to clinicaland inflammatory markers of persistent asthma in children. J Allergy Clin Immunol 2003;112(5):883-892.)*Geometric mean, coefficient of variation Median and 1st and 3rd quartiles are reported on the log2 scaleàAM PEF-PM PEF/([AM PEF 1 PM PEF]/2) 3 100 manner, potential predictors of eNO were identified. PC20 median, a nonparametric measure of the central value, and was analyzed on the log base 2 scale because of doubling the lower and upper quartiles of the variable’s distribution, doses in the methacholine challenge procedure. The eNO a nonparametric indication of variability. By providing all measurements and many of the biomarkers displayed of this information, they allow the reader to make his or a skewed distribution, and these were analyzed on the her own assessment of the variability of the data, which does not have to depend on an assumption that the datahave a Gaussian (normal) distribution.
Sorensen et note the nonnormality of their data and Adams et aland Eggesbe et alboth provide CIs for the need for transformation in the ‘‘Data analysis’’ section.
their measures of variability. The presentation of Adams et ‘‘The data were generally highly right-skewed, thus alis preferable because they specify that they are using natural log-transformed variables were analyzed.’’ 95% CIs, whereas Eggesbe et alonly use the term 5. Not identifying or properly labeling the type of ‘‘confidence interval.’’ It is a good assumption that what is variance estimate being used. There are a variety of intended is a 95% interval, but it is better to state that variance structures that might apply, but the 2 that are most commonly not labeled or mislabeled are the SD and the 6. Incomplete description of the statistical test used.
SE. The SD measures the variability in your original data.
When describing the statistical tests you have used, think The SE measures the variability in the mean or average in terms of providing the reader with enough information value of your data. When you want to demonstrate how so that if they had the data, they would be able to do variable your mean is (this is important for reproducibility exactly the same test that you did and hopefully get the and statistical testing), you want to use the SE and same result. It is not enough to say you have done a t test corresponding SE bars around the mean. When you when in fact you have done a 2-sample t test with unequal want to show how variable a single observation could be expected to be, you want to use the SD and error bars In the final paragraph of their ‘‘Statistical methods’’ based on the SD around a single measurement. You also section, Strunk et alprovide additional information that need to state whether you are using 1, 2, or sometimes 3 would allow the reader to duplicate their results. They times the SE or SD. Each one of these limits encompasses have also provided information on the statistical program a different proportion of the variables you are examining.
(software) they used for their analysis. This might also be For example, a 95% CI around a mean taken from needed for a reader to reproduce results because software normally distributed data is obtained by taking 62*SE.
programs might differ in how they do calculations, and Using 1 instead of 2 gives you a 68% CI.
different programs might give you slightly different In their Strunk et alclearly indicate that they are providing the mean 6 SD. What they want to de- A complete cohort of randomly assigned CLIC subjects was monstrate is the variability of his single observations. In used for the model building. The cohort was revised this presentation they used the mean as a representation of according to variables in the final model to include all a single observation and gave the SD as a measure of the possible patients. All summary statistics and analyses were variability in single observations. They also provide the performed by means of SAS Version 8 statistical software (SAS Institute, Cary, NC). Significance was established at 11. Specify any general use computer programs used.
12. Put general descriptions of statistical methods in the Sorensen et also provide information on the analysis Methods section. When data are summarized in the package used and, in the section below, additional Results section, specify the statistical methods used information that a reader might need to reproduce their results. ‘‘All reported P values in this article are based on 13. Restrict tables and figures to those needed to explain 2-tailed tests, unless otherwise mentioned. It should be the argument of the paper and to assess its support.
noted that the use of a 2-tailed test when the hypothesis Use graphs as an alternative to tables with many involves an expected increase (or decrease) yields a con- entries; do not duplicate data in graphs and tables.
14. Avoid nontechnical uses of technical terms in statistics, such as ‘‘random’’ (which implies a ran-domizing device), ‘‘normal,’’ ‘‘significant,’’ ‘‘corre- 15. Define statistical terms, abbreviations, and most It is important to have the right perspective about correctly reporting statistical tests in the medical literature.
Recently, several major physiology journals have It is very easy to think that because misreporting has been adopted guidelines for reporting statistical information going on for so long without seeming to harm medical that adopt many of the above These guide- research, this is not a big issue. In the words of John lines contain practical and straightforward suggestions for reporting statistical information that should help you to . there may be greater danger to the public welfare from statistical dishonesty than from avoid the problems discussed above. The single most almost any other form of dishonesty.’’ This article is not important thing to do is to consult a statistician well before about dishonesty but about carelessness in the reporting of you undertake the analysis of your data. If the early steps statistical methods. The harmful results, however, could of design and data collection are done correctly, the actual be the same, and the first step in separating the one from analysis of the data is usually a relatively minor issue.
the other is to agree on rigorous reporting in medicaljournals of statistical methods. This will make it easier for researchers to understand what constitutes the standard of 1. Olsen CH. Review of the use of statistics in infection and immunity.
excellence to which all of us should aspire.
2. Garcia-Berthou E, Alcaraz C. Incongruence between test statistics and P Dr Bailar’s quote is particularly appropriate because the values in medical papers. BMC Medical Research Methodology 2004.
recommendations for statistical reporting that he and Dr Mosteller laid out in 1988are still the basis for reporting statistical methods that are recommended by the 3. White S. Statistical errors in papers in the British Journal of Psychiatry.
International Committee of Medical Journal Editors.
4. Strunk RC, Szefler SJ, Phillips BR, Zeiger RS, Chinchilli VM, Larsen G, Their 15 points are listed below. More detail can be et al. Relationship of exhaled nitric oxide to clinical and inflammatory markers of persistent asthma in children. J Allergy Clin Immunol 2003;112:883-92.
1. Describe statistical methods with enough detail to 5. Asero R, Mistrello G, Roncarlo D, Amato S, Zanoni D, Barocci F, et al.
enable a knowledgeable reader with access to the Detection of clinical markers of sensitization to profiling in patients original data to verify the reported results.
allergic to plant-derived foods. J Allergy Clin Immunol 2003;112(2): 2. When possible, quantify findings and present them with appropriate indicators of measurement error or 6. Adams RJ, Weiss ST, Fuhlbrigge A. How and by whom care is delivered influences anti-inflammatory use in asthma: results of a national uncertainty (eg, confidence intervals).
population survey. J Allergy Clin Immunol 2003;112:445-50.
3. Avoid sole reliance on statistical hypothesis testing, 7. Sorensen B, Streib JE, Strand M, Make B, Giclas PC, Fleshner M, et al.
such as the use of P values, which fails to convey Complement activation n a model of chronic fatigue syndrome. J Allergy 8. Eggesbe M, Botten G, Stigum H, Nafstad P, Magnus P. Is delivery by 4. Discuss eligibility of experimental subjects.
cesarean section a risk factor for food allergy? J Allergy Clin Immunol 5. Give details about randomization.
6. Describe the methods for, and success of, any 9. International Committee of Medical Journal Editors. Uniform require- ments for manuscripts submitted to biomedical journals. Ann Intern Med 10. Curran-Everett D. Multiple comparisons: philosophies and illustrations.
Am J Physiol Regul Integr Comp Physiol 2000;279:R1-8.
9. Report losses to observation (eg, dropouts from 11. Bailar JC. Bailar’s laws of data analysis. Clin Pharmacol Ther 1976;20: 10. References for study design and statistical methods 12. Bailar JC, Mosteller F. Guidelines for statistical reporting in articles for medical journals. Ann Intern Med 1988;108:266-73.
should be to standard works (with pages stated) 13. Curran-Everett D, Benos DJ. Guidelines for reporting statistics in when possible rather than to papers where designs or journals published by the American Physiological Society. Am J Physiol

Source: http://stat-athens.aueb.gr/~jpan/papers/Murphy-JACI2004(1259-1264).pdf