Sociological Methods & Research

 

Advanced Search

Journal Navigation

Journal Home

Subscriptions

Archive

Contact Us

Table of Contents

Click here for more information

Sign In to gain access to subscriptions and/or personal tools.
This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Online Appendix
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Google Scholar
Right arrow Articles by Gerber, A. S.
Right arrow Articles by Malhotra, N.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati  
What's this?
This version was published on August 1, 2008
Sociological Methods & Research, Vol. 37, No. 1, 3-30 (2008)
DOI: 10.1177/0049124108318973

Publication Bias in Empirical Sociological Research

Do Arbitrary Significance Levels Distort Published Results?

Alan S. Gerber

Yale University, New Haven, Connecticut

Neil Malhotra

Stanford University, Stanford, California, neilm{at}stanford.edu

Despite great attention to the quality of research methods in individual studies, if publication decisions of journals are a function of the statistical significance of research findings, the published literature as a whole may not produce accurate measures of true effects. This article examines the two most prominent sociology journals (the American Sociological Review and the American Journal of Sociology) and another important though less influential journal (The Sociological Quarterly) for evidence of publication bias. The effect of the .05 significance level on the pattern of published findings is examined using a ``caliper'' test, and the hypothesis of no publication bias can be rejected at approximately the 1 in 10 million level. Findings suggest that some of the results reported in leading sociology journals may be misleading and inaccurate due to publication bias. Some reasons for publication bias and proposed reforms to reduce its impact on research are also discussed.

Key Words: publication bias • caliper test • meta-analysis • hypothesis testing

References

  • Allen, Michael P. 2003. ``The `Core Influence' of Journals in Sociology Revisited.'' Footnotes. 31:7-10.
  • Allison, D.B., M.S. Faith, and B.S. Gorman. 1996. ``Publication Bias in Obesity Treatment Trials?'' International Journal of Obesity 20:931-37.[ISI][Medline] [Order article via Infotrieve]
  • Ashenfelter, Orley, Colm Harmon, and Hessel Oosterbeek. 1999. ``A Review of Estimates of the Schooling/Earnings Relationship, With Tests for Publication Bias. '' Labour Economics 6:453-70.[CrossRef]
  • Begg, Colin B. and Jesse A. Berlin. 1988. ``Publication Bias: A Problem in Interpreting Medical Data.'' Journal of the Royal Statistical Society, Series A 151:419-63.
  • Berlin, Jesse A., Colin B. Begg, and Thomas A. Louis. 1989. ``An Assessment of Publication Bias Using a Sample of Published Clinical Trials.'' Journal of the American Statistical Association 84:381-92.[CrossRef][ISI]
  • Campbell, Donald T. 1969. ``Reforms as Experiments.'' American Psychologist 24:407-29.
  • Card, David and Alan B. Krueger. 1995. ``Time-Series Minimum Wage Studies: A Meta-Analysis.'' American Economic Review 85:238-43.[ISI]
  • Cleary, Richard J. and George Casella. 1997. ``An Application of Gibbs Sampling to Estimation in Meta-Analysis: Accounting for Publication Bias.'' Journal of Educational and Behavioral Statistics 22:141-54.[Abstract/Free Full Text]
  • Copas, John. 1999. ``What Works?: Selectivity Models and Meta-Analysis.'' Journal of the Royal Statistical Society, Series A 161:95-105.
  • Correll, Shelley J. 2001. ``Gender and the Career Choice Process: The Role of Biased Self-Assessments.'' American Journal of Sociology 106:1691-730.[CrossRef]
  • Coursol, Allan and Edwin E. Wagner. 1986. ``Effect of Positive Findings on Submission and Acceptance Rates: A Note on Meta-Analysis Bias.'' Professional Psychology 17:136-37.[CrossRef]
  • Dear, Keith B. G. and Colin B. Begg. 1992. ``An Approach for Assessing Publication Bias Prior to Performing a Meta-Analysis.'' Statistical Science 7:237-45.[CrossRef]
  • De LongJ. Bradford, nd Kevin Lang. 1992. ``Are All Economic Hypotheses False?'' Journal of Political Economy 100:1257-72.[CrossRef][ISI]
  • DiPrete, Thomas A. and Henriette Engelhart. 2004. ``Estimating Causal Effects With Matching Methods in the Presence and Absence of Bias Cancellation.'' Sociological Methods & Research 32:501-28.[Abstract]
  • Doucouliagos, Chris. 2005. ``Publication Bias in the Economic Freedom and Economic Growth Literature.'' Journal of Economic Surveys 19:367-87.[ISI]
  • Doucouliagos, Chris, Patrice Laroche, and T.D. Stanley. 2005. ``Publication Bias in Union-Productivity Research?'' Industrial Relations 60:320-47.
  • Duval, Sue and Richard Tweedie. 2000. ``A Nonparametric `Trim and Fill' Method of Accounting for Publication Bias in Meta-Analysis. '' Journal of the American Statistical Association 95:89-98.[CrossRef][ISI]
  • Easterbrook, P.J., J.C. Keruly, T. Creagh-Kirk, D.D. Richman, R.E. Chaisson, and R.D. Moore. 1991. ``Racial and Ethnic Differences in Outcome in Zidovudine-Treated Patients With Advanced HIV Disease.'' Journal of the American Medical Association 266:2713-18.[Abstract]
  • Egger, Matthias, George D. Smith, Mariin Schneider, and Christoph Minder. 1997. ``Bias in Meta-Analysis Detected by a Simple, Graphical Test.'' British Medical Journal 315:629-34.[Abstract/Free Full Text]
  • Elman, Cheryl and Angela M. O'Rand. 2004. ``The Race Is to the Swift: Socioeconomic Origins, Adult Education, and Wage Attainment.'' American Journal of Sociology 110:123-60.[CrossRef]
  • Epstein, William M. 2000. ``Confirmational Response Bias Among Social Work Journals.'' Science, Technology, & Human Values 15:9-38.
  • ---. 2004. ``Confirmational Response Bias and the Quality of the Editorial Processes Among American Social Work Journals.'' Research on Social Work Practice 14:450-58.[Abstract]
  • Gelman, Andrew, John B. Carlin, Hal S. Stern, and Donald B. Rubin. 2004. Bayesian Data Analysis. 2d ed. Boca Raton, FL: Chapman & Hall.
  • Gerber, Alan S., Donald P. Green, and David Nickerson. 2000. ``Testing for Publication Bias in Political Science.'' Political Analysis 9:385-92.
  • Gerber, Alan S. and Neil Malhotra. 2006. ``Can Political Science Literatures Be Believed? A Study of Publication Bias in the APSR and the AJPS.'' Presented at the annual meeting of the Midwest Political Science Association, April 20-23, Chicago, IL.
  • Glass, G.V., B. McGaw, and M.L. Smith. 1981. Meta Analysis in Social Research. Beverly Hills, CA: Sage.
  • Gleser, L.J. and I. Olkin. 1996. ``Models for Estimating the Number of Unpublished Studies.'' Statistics in Medicine 15:2493-507.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Gorg, Holger and Eric Strobl. 2001. ``Multinational Companies and Productivity: A Meta-Analysis. Economic Journal 111:723-39.[CrossRef]
  • Greene, William H. 1996. Econometric Analysis. 3d ed. Upper Saddle River, NJ: Prentice Hall.
  • Greenwald, Anthony G. 1975. ``Consequences of Prejudice Against the Null Hypothesis.'' Psychological Bulletin 82:1-20.[CrossRef][ISI]
  • Halaby, Charles N. 2004. ``Panel Models in Sociological Research: Theory Into Practice.'' Annual Review of Sociology 30:507-44.[CrossRef][ISI]
  • Hedges, Larry V. 1984. ``Estimation of Effect Size Under Nonrandom Sampling: The Effects of Censoring Studies Yielding Statistically Insignificant Mean Differences.'' Journal of Educational Statistics 9:61-85.[CrossRef]
  • ---. 1992. ``Modeling Publication Selection Effects in Meta-Analysis.'' Statistical Science 7:246-55.[CrossRef]
  • Iyengar, Satish and Joel B. Greenhouse. 1988. ``Selection Models and the File Drawer Problem.'' Statistical Science 3:109-35.[CrossRef]
  • Leahey, Erin. 2005. ``Alphas and Asterisks: The Development of Statistical Significance Testing Standards in Sociology.'' Social Forces 84:1-23.[CrossRef][ISI]
  • Levois, M.E. and M.W. Layard. 1995. ``Publication Bias in the Environmental Tobacco Smoke/Coronary Heart Disease Epidemiologic Literature. '' Regulatory Toxicology and Pharmacology 21:184-91.[CrossRef][ISI]
  • Light, Richard J. and David B. Pillemar. 1984. Summing Up: The Science of Reviewing Research. Cambridge, MA: Harvard University Press.
  • McLeod, B.D. and J.R. Weisz. 2004. ``Using Dissertations to Examine Potential Bias in Child and Adolescent Clinical Trials.'' Journal of Consulting and Clinical Psychology 2:235-51.
  • Mouw, Ted. 2006. ``Estimating the Causal Effect of Social Capital: A Review of Recent Research.'' Annual Review of Sociology 32:79-102.[CrossRef][ISI]
  • Pager, Devah. 2003. ``The Mark of a Criminal Record. '' American Journal of Sociology 108:937-75.[CrossRef][ISI]
  • Rosenthal, Robert. 1979. ``The File Drawer Problem and Tolerance for Null Results.'' Psychological Bulletin 86:638-41.[CrossRef][ISI]
  • Sigelman, Lee. 1999. ``Publication Bias Reconsidered. '' Political Analysis 8:201-10.
  • Simes, R.J. 1986. ``Publication Bias: The Case for an International Registry of Clinical Trials.'' Journal of Clinical Oncology 4:1529-41.[ISI][Medline] [Order article via Infotrieve]
  • Stanley, T.D. 2005. ``Beyond Publication Bias.'' Journal of Economic Surveys 19:309-37.[ISI]
  • Stanley, T.D. and Stephen B. Jarrell. 1989. ``Meta-Regression Analysis: A Quantitative Method of Literature Surveys.'' Journal of Economic Surveys 3:161-70.[CrossRef]
  • Sterling, T.D. 1959. ``Publication Decision and the Possible Effects on Inferences Drawn From Tests of Significance-or Vice Versa. '' Journal of the American Statistical Association 54:30-34.[CrossRef][ISI]
  • Sterling, T.D., W.L. Rosenbaum, and J.J. Weinkam. 1995. ``Publication Bias Revisited: The Effect of the Outcome of Statistical Tests on the Decision to Publish and Vice Versa.'' American Statistician 49:108-12.[CrossRef][ISI]
  • Sutton, A.J., F. Song, S.M. Gilbody, and K.R. Abrams. 2000. ``Modelling Publication Bias in Meta-Analysis: A Review.'' Statistical Methods in Medical Research 9:421-45.[Abstract/Free Full Text]
  • Wilson, Franklin D., Gale L. Smoke, and J. David Martin. 1973. ``The Replication Problem in Sociology: A Report and a Suggestion.'' Sociological Inquiry 43:141-49.[CrossRef][ISI]

Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to Technorati Technorati    What's this?



This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Online Appendix
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Google Scholar
Right arrow Articles by Gerber, A. S.
Right arrow Articles by Malhotra, N.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati  
What's this?