Methods of Meta-Analysis · Correcting Error and Bias in Research Findings by Schmidt, Frank L. -- Read -- Imperial Library of Trantor

Index

Halftitle Title Copyright Brief Contents Detailed Contents List of Tables List of Figures Preface Acknowledgments About the Authors PART I: INTRODUCTION TO META-ANALYSIS

1. Integrating Research Findings Across Studies

General Problem and an Example

A Typical Interpretation of the Example Data Conclusions of the Review Critique of the Sample Review

Problems With Statistical Significance Tests Is Statistical Power the Solution? Confidence Intervals Meta-Analysis Role of Meta-Analysis in the Behavioral and Social Sciences

The Myth of the Perfect Study Some Relevant History

Role of Meta-Analysis in Theory Development Meta-Analysis in Industrial-Organizational Psychology Wider Impact of Meta-Analysis on Psychology Impact of Meta-Analysis Outside Psychology

Impact in Biomedical Research Impact in Other Disciplines

Meta-Analysis and Social Policy Meta-Analysis and Theories of Data and Theories of Knowledge Conclusion

2. Study Artifacts and Their Impact on Study Outcomes

Study Artifacts

Sampling Error Error of Measurement Dichotomization Range Variation in the Independent Variable Range Variation in the Dependent Variable Imperfect Construct Validity in the Independent Variable Imperfect Construct Validity in the Dependent Variable Computational and Other Errors in the Data Extraneous Factors Introduced by the Study Procedure Bias in the Sample Correlation

Sampling Error, Statistical Power, and the Interpretation of Research Findings

An Illustration of Statistical Power A More Detailed Examination of Statistical Power

When and How to Cumulate Undercorrection for Artifacts in the Corrected Standard Deviation (SDρ) Coding Studies Characteristics and Capitalization on Sampling Error in Moderator Analysis A Look Ahead in the Book

PART II: META-ANALYSIS OF CORRELATIONS

3. Meta-Analysis of Correlations Corrected Individually for Artifacts

Introduction and Overview Bare-Bones Meta-Analysis: Correcting for Sampling Error Only

Estimation of Sampling Error Correcting the Variance for Sampling Error and a Worked Example Moderator Variables Analyzed by Grouping the Data and a Worked Example Correcting Feature Correlations for Sampling Error and a Worked Example

Artifacts Other Than Sampling Error

Error of Measurement and Correction for Attenuation Using the Appropriate Reliability Coefficient Methods of Estimating Reliability Coefficients

Implications of Measurement Error for Meta-Analysis

Restriction or Enhancement of Range Dichotomization of Independent and Dependent Variable Measures Imperfect Construct Validity in Independent and Dependent Variable Measures Attrition Artifacts Extraneous Factors Bias in the Correlation

Multiple Simultaneous Artifacts Meta-Analysis of Individually Corrected Correlations

Individual Study Computations Combining Across Studies Final Meta-Analysis Estimation An Example: Validity Generalization With Indirect Range Restriction

A Worked Example: Indirect Range Restriction Summary of Meta-Analysis Correcting Each Correlation Individually Exercise 3.1: Bare Bones Meta-Analysis: Correcting for Sampling Error Only Exercise 3.2: Meta-Analysis Correcting Each Correlation Individually

4. Meta-Analysis of Correlations Using Artifact Distributions

Introduction and Basic Concepts Full Artifact Distribution Meta-Analysis

Earlier Procedures for Artifact Distribution Meta-Analysis The Interactive Method A Simplified Example of Application of the Interactive Method A Worked Example: Error of Measurement A Worked Example: Unreliability and Direct Range Restriction A Worked Example: Personnel Selection With Fixed Test (Direct Range Restriction) Personnel Selection With Varying Tests Personnel Selection: Meta-Analytic Findings in the Literature A Worked Example: Indirect Range Restriction Refinements to Increase Accuracy of the SDρ Estimate

Accuracy of Corrections for Artifacts Mixed Meta-Analysis: Partial Artifact Information in Individual Studies

An Example: Dichotomization of Both Variables and a Moderator The Moderator Evaluated

Summary of Artifact Distribution Meta-Analysis of Correlations Exercise 4.1: Artifact Distribution Meta-Analysis

5. Technical Questions in Meta-Analysis of Correlations

r Versus r2: Which Should be Used? r Versus Regression Slopes and Intercepts in Meta-Analysis

Range Restriction Measurement Error Comparability of Units Across Studies Comparability of Findings Across Meta-Analyses Intrinsic Interpretability

Use of Fisher’s z in Meta-Analysis of Correlations Fixed Effects and Random Effects Models in Meta-Analysis Accuracy of Different Random Effects Models Credibility Intervals, Confidence Intervals, and Prediction Intervals in Meta-Analysis Computing Confidence Intervals in Meta-Analysis of Correlations Technical Issues in Using Meta-Analysis Results in Causal Modeling and Regression

Path Analysis or SEM? What Sample Size to Use and What Software? Is There a Mixture of Populations in the Correlation Matrix? What About Heterogeneity Within Meta-Analyses?

Technical Factors That Cause Overestimation of SDρ

Presence of Non-Pearson rs Presence of Outliers and Other Data Errors and Problems in Removing Outliers Use of r Instead of r in the Sampling Error Formula Undercorrection for Sampling Error Variance in the Presence of Range Restriction Nonlinearity in the Range Correction Setting Negative Variance Estimates to Zero Other Factors Causing Overestimation of SDρ

PART III: META-ANALYSIS OF EXPERIMENTAL EFFECTS AND OTHER DICHOTOMOUS COMPARISONS

6. Treatment Effects: Experimental Artifacts and Their Impact

Quantification of the Treatment Effect: The d Statistic and the Point Biserial Correlation Sampling Error in d Values: Illustrations

Case 1: N = 30 Case 2: N = 68 Case 3: N = 400

Error of Measurement in the Dependent Variable Measure Error of Measurement in the Treatment Variable Variation Across Studies in Treatment Strength Range Variation on the Dependent Variable Dichotomization of the Dependent Variable Measure Imperfect Construct Validity in the Dependent Variable Measure Imperfect Construct Validity in the Treatment Variable Bias in the Effect Size (d Statistic) Recording, Computational, and Transcriptional Errors Multiple Artifacts and Corrections

7. Meta-Analysis Methods for d Values

Effect Size Indexes: d and r

Maximum Value of the Point Biserial r The Effect Size (d Statistic) Correction of the Point Biserial r for Unequal Sample Sizes Examples of the Convertability of r and d Problems of Artificial Dichotomization

An Alternative to d: Glass’s d Sampling Error in the d Statistic

The Standard Error for d The Confidence Interval for δ

Cumulation and Correction of the Variance for Sampling Error

Bare-Bones Meta-Analysis Confidence Intervals for d A Worked Example Another Example: Leadership Training by Experts

Analysis of Moderator Variables

Using Study Domain Subsets Using Study Characteristic Correlations A Worked Example: Training by Experts Versus Training by Managers Another Worked Example: Amount of Training The Correlational Moderator Analysis

Correcting d Value Statistics for Measurement Error in the Dependent Variable

Meta-Analysis of d Values Corrected Individually and a Worked Example Artifact Distribution Meta-Analysis and a Worked Example

Measurement Error in the Independent Variable in Experiments Other Artifacts and Their Effects Correcting Individual d Values for Multiple Artifacts Attenuation Effect of Multiple Artifacts and Correction for the Same Meta-Analysis of d Values With Multiple Artifacts Using the Correlation Metric Summary of Meta-Analysis of d Values Exercise 7.1: Meta-Analysis of d Values

8. Technical Questions in Meta-Analysis of d Values

Alternative Experimental Designs: General Considerations Analysis of Covariance (ANCOVA) Designs Factorial Independent Groups ANOVA Designs Repeated Measures Designs

Repeated Measures Designs Without Control Group and Matched Groups Designs Repeated Measures Designs With a Control Group Empirical Comparison of the Two Repeated Measures Designs

Threats to Validity in Repeated Measures Designs With No Control Group

History Maturation Testing Effects Instrumentation Regression Toward the Mean Reactive Situations Interaction Between Selection and the Treatment

Bias in Observed d Values Credibility Intervals, Confidence Intervals, and Prediction Intervals in Meta-Analysis of d Values Computing Confidence Intervals in Meta-Analysis of d Values Fixed Effects and Random Effects Models in Meta-Analysis of d Values

PART IV: GENERAL ISSUES IN META-ANALYSIS

9. General Technical Issues in Meta-Analysis

Large-N Studies Versus Meta-Analysis Detecting Moderator Variables in Meta-Analysis

Detecting Moderator Variables Not Hypothesized a Priori Hierarchical Analysis of Moderator Variables via Study Subgrouping

Use of Multiple Regression in Moderator Analysis and Mixed Meta-Analysis Models

Meta-Regression: Advantages and Disadvantages Multilevel Models in Meta-Analysis and HLM Mixed Effects Models in Meta-Analysis

Second-Order Sampling Error: General Principles Second-Order Meta-Analysis Across Different Independent Variables Second-Order Meta-Analysis With a Constant Independent Variable

Second-Order Meta-Analysis of Bare-Bones Meta-Analyses Second-Order Meta-Analysis When Correlations Have Been Individually Corrected Second-Order Meta-Analysis With Artifact Distribution Meta-Analyses Mixed Second-Order Meta-Analysis Considerations in Second-Order Meta-Analysis

Second-Order Sampling Error: Technical Treatment

The Homogeneous Case The Heterogeneous Case A Numerical Example Another Example: Leadership Training by Experts Moderator Example: Skills Training

Confidence Intervals in Random Effects Models: Hunter-Schmidt and Hedges-Olkin Updating a Meta-Analysis When a New Study Becomes Available What Are Optimal Study Weights in Random Effect Meta-Analyses? The Meaning of Percent Variance Accounted for in Meta-Analysis The Odds Ratio (OR) in Behavioral Meta-Analyses Exercise 9.1: Second-Order Meta-Analysis Across Different Independent Variables With the Same Dependent Variable Exercise 9.2: Second-Order Meta-Analysis With Constant Independent and Dependent Variables Exercise 9.3: Second-Order Meta-Analysis With Constant Independent and Dependent Variables

10. Cumulation of Findings Within Studies

Fully Replicated Designs: Statistical Independence Conceptual Replication and Lack of Statistical Independence Research on Effects of Violations of Statistical Independence Conceptual Replication and Composite Scores Conceptual Replication: A Fourth Approach and Summary and Conclusions Replication by Analysis of Subgroups

Subgroups and Loss of Power Subgroups and Capitalization on Chance Subgroups and the Bias of Disaggregation

Conclusion: Use Total Group Correlations Summary

11. Different Methods of Meta-Analysis and Related Software

The Traditional Narrative Review The Traditional Voting Method Cumulation of p Values Across Studies Statistically Correct Vote-Counting Procedures

Vote-Counting Methods Yielding Only Significance Levels Vote-Counting Methods Yielding Estimates of Effect Sizes

Meta-Analysis of Research Studies

Purely Descriptive Meta-Analysis Methods: Glassian and Related Methods Meta-Analysis Methods Focusing Only on Sampling Error

Unresolved Problems in Meta-Analysis Summary of Methods of Integrating Studies Computer Programs for Meta-Analysis

Programs for Glassian Meta-Analysis Programs for Homogeneity-Based Meta-Analysis Programs for Psychometric Meta-Analysis

12. Locating, Evaluating, Selecting, and Coding Studies and Presentation of Meta-Analysis Results

Conducting a Thorough Literature Search What to Do About Studies With Methodological Weaknesses Coding Studies in Meta-Analysis Reporting the Results of a Meta-Analysis: Standards and Practices Information Needed in Reports of Primary Studies

Correlational Studies Experimental Studies Studies Using Multiple Regression Studies Using Factor Analysis Studies Using Canonical Correlation Studies Using Multivariate Analysis of Variance (MANOVA)

General Comments on Reporting in Primary Studies Appendix

13. Availability Bias, Source Bias, and Publication Bias in Meta-Analysis

Some Literatures May Have Little or No Publication Bias Effects of Methodological Quality on Mean Effect Sizes From Different Sources Multiple Hypotheses and Other Considerations in Availability Bias Is There a Crisis of Confidence in Scientific Research Today?

Scientific Fraud Data Manipulation Short of Scientific Fraud Evidence for Publication Bias (Traditionally Defined)

Methods for Dealing With Availability Bias

File Drawer Analysis Based on p Values File Drawer Analysis Based on Effect Size Subgrouping of Published and Unpublished Studies The Funnel Plot The Trim-and-Fill Method Cumulative Meta-Analysis Correlation- and Regression-Based Methods The Statistical Power Method and the p-Hacking Method Selection Models

Study Artifacts and Publication Bias Analysis Software for Publication Bias Analysis Attempts to Prevent Publication Bias Before It Happens Summary of Methods for Correcting Availability Bias

14. Summary of Psychometric Meta-Analysis

Meta-Analysis Methods and Theories of Data and Theories of Knowledge What Is the Ultimate Purpose of Meta-Analysis? Psychometric Meta-Analysis: Summary Overview

Appendix: Windows-Based Meta-Analysis Software Package Version 2.0 References Author Index Subject Index Advertisement

← Prev
Back
Next →

← Prev
Back
Next →