a priori contrasts, see Statistical tests
Accosting strategy, 286
Acquiescence response bias, see Response bias
Action editor, 652
Action research, 34–35
Active deception, 75–76, see also Ethical treatment of research participants
Actuarial prediction, 41, 44–46, 320–321
Acute manipulations, 227
Additivity, 313–314
Affirmative assent, 74
Aggregate data, 523
Alpha level, see Type I error
Alternative explanation for effect of independent variable,
in archival data, 523
in case study research, 40, 290–296, 369–370, 392
and causality, 5–6, 41, 246–247, 283
in correlational research, 42
in nonequivalent control group designs, 290–293
in path analysis, 355
use of control condition to rule out, 37, 127, 245–247
American Psychological Association (APA), 62, 70, 78, 83, 85, 100, 101, 140, 658, 659
Analog research, 233
Analysis of covariance (ANCOVA), 291–292, 295–296
Analysis of variance (ANOVA), see also Factorial designs; Statistical tests
multiple regression analysis as an alternative to, 329–331
nested design, 295–296
one–way, 263
random effects model, 251
Anchors in numerical rating scales, 451–453, 455
Annual Review of Psychology, 594
Application of research results, 25–27, 459, see also Research, applied; Utilization of research results
design issues, 137–139
in discussion sections, 642–646
ethical issues, 97–99, 104–105
relation to philosophies of science, 234
relation to research, 25–27
relation to theory, 25–27
Archival data, 519–523
advantages of using, 521
limitations of using 521–522
validity, 522–523
Artificiality of laboratory research, see Ecological validity
Assumptions,
in correlational research, 313–314
of theories, 10–12, 127–128, 141–142
Attenuation of correlation coefficient, 314
Attrition, 48, 196, 293, 296, see also Mortality
Authorship credit, 658–659
Baseline, see Single-case research; Time series design
Basic research, 32–33, 106, 214, 234, see also Research
Behavioral consent, see Ethical treatment of research participants
Behavioral measures, see Measurement modalities
Behavioriod measure, 166
Beneficence (in ethics), 62, see also Ethical treatment of research participants
Between-subjects designs, 252–255, 270–271, 634
Bias, see also Experimenter expectancy effects; Sampling
in formulating research questions, 141–143
in interpreting data, 375, 392, 537–540
and integrative literature reviews, 532, 599, 604–605, 613
in case study research, 39, 40, 375, 392–393
in integrative literature reviews, 600–601, 613
in observational research, 200, 206, 304–306
in qualitative data analysis, 392, 410–411
in sampling, 487, 508–509, 513
in selection, see Internal validity, selection bias
in survey research, 459–464
in theories, 142
Bivariate regression, 320–322
Blind review, 652
Blocking on extraneous variables, 274–276
Boundary conditions, 120, 136, 214
Carryover effects, 256–257, 383
Case studies, 33, 39–41, 371–375, see also Single case research
control cases, 369–370
embedded, 373
in evaluation research, 580
impact criteria, 370
test cases, 369–370
validity criteria, 365–372
Catalog of U. S. Government Publications, 124
Category anchoring, 459–460
Causal analysis, see Path analysis
Causality, 5–6, 37, 42–43, 242, see also Alternative explanations for effect of independent variable
in case study research, 40
in correlational research, 42–43, 46, 312
in nonequivalent control group designs, 292–293
reciprocal, 44
reverse, 44
rules for establishing, 5–6, 37, 42–43, 242
Causal steps strategy, 351–353, see also Mediational hypotheses
Chain of evidence, 374–375; see also Coding systems; Qualitative research
Change, see also Evaluation research; Research, developmental
measurement of, 586–587
in research findings over time, 226
Change in R2, 327
Children in research, see Ethical treatment of research participants
as goodness of fit test, 349–350
Chronic manipulation, 227
Clinical significance, see Practical significance
Code of ethics for psychologists (APA), 62–63, 658–660, see also Ethical treatment of research participants
Coding systems, 191–192, 209, 304–305, 374–375; 405–408, 428–430, see also Content analysis; Interviews
broad versus narrow categories, 402
chain of evidence, 374–375
coder qualifications and training, 406
cognitive biases in, 304–305
data coding, 405–408
data collection plans, 373–375, 405–406
developing, 402–404
hierarchical, 402–404
in Internet research, 505
in interviews, 428–430
manifest versus latent content, 401
recording keeping, 304–305
reliability, 191, 192, 406–407, 417
sources of coding categories, 400–401
units of analysis, 401
validity of, 406–407
Cohort sequential research, 49–51
Comparative fit index, 349–350
Comparison groups, 245–247, see also Control groups
Conceptual literature reviews, see Literature reviews
Confirmability, see Replication
Confirmatory factor analysis, see Factor analysis
Confounds, 186–196, 251–252, see also Internal validity, threats to
natural, 187
selection threats, 194–196
treatment, 188–189
Congressional record, 522
Consent, see Ethical treatment of research participants
Construct, see Hypothetical construct
Constructionism, see Social constructionism
Construct validity, 23, 127, 154–156, 167, 169
of archival data, 522–523
in field experiments, 287
of manipulation of independent variable, 127, 247–249, 494–495, 497
of measurement, 154–155
Content analysis, 399–409, see also, Coding systems
coding schemes in, 399, 401–404
data collection for, 405–408
developing research hypotheses for, 399–400
Content validity, see Validity
Context effects, 466, 468–469, see also Reactivity
between questionnaires, 468
within questionnaires, 466
Control, 7, 32, 52–55, 233, 282–283, see also Alternative explanations for independent variable; Internal validity
in single case research, 52–53, 364, 369–370
Control groups, 37, 127, 244–247
contamination of, 578
and internal validity, 193–194, 209
no-treatment, 68–69
and single-case research, 53, 364, 369–370, 377–378
Control series design, 297–300; see also Time series design
Convenience sample, see Sampling
Convenience survey,
Convention presentations, 124, 655–656; see also Publication
Convergent validity, see Validity
Co-participants, 218–219
Correlational analysis, 313–324; see also Chi–square analysis; Correlational research; Latent variables analysis; Logistic regression analysis; Multiple regression analysis; Multi-way frequency analysis; Path analysis
bivariate regression, 320–322
differences in correlation coefficients, 321–322
factors affecting, 314–316
partial, 322–324
as control for measurement confound, 189
simple, 320–322
Correlational research, 41–46, 313–324, see also Correlational analysis
assumptions underlying, 313–314
and causal inferences, 42–43, 312
considerations in conducting, 314–317, 319–320
and factorial designs, 271
multifaceted constructs in, 317–319
third variable problem in, 44
uses of, 44–46
Corresponding author, 651–652
Costs of intervention program, 574, 582–583
cost-benefit analysis, 582–583
cost-effectiveness analysis, 583
Counterbalancing,
of order of questionnaires, 468–469
in within-subjects designs, 257–258
Covariation (as a criterion for causality), 5, 43–44, 242, 312
Criterion variable, see Multiple regression analysis
Cronbach’s alpha, see Reliability
Cross-cultural research, 162–163, 197–198, 209, 224
Cross-sectional research, 46–47, 209
Culpable error, 93–95
Cultural response sets, 463–464
Culture, 10, 141–142, 197–198, 209, 220, 224, 521
Curvilinear relationships, see Relationships between variables
d (effect size indicator), 608, see also Effect size
Data, cooking, 95
discarding, 95
forging, 94–95
qualitative, 35–36; 390–393, see also Coding systems
quantitative, 35–36; 288–296; 392–393
snooping, 208–209
torturing, 95
trimming, 95
Data analysis, 113, 131–132, 243–244, see also Statistical tests
Data collection, 483–525, see also Content analysis; Internet research; Interviews; Measurement modalities; Questionnaires
in case study research, 366–368, 373–375
data collection plans, 373–375, 405–406
in evaluation research, 561
ethical considerations in, 79–82, see also Ethical treatment of research participants
in Internet research, 506–508
in survey research, 469–475
Debriefing, 83–86, 500–503; see also Post-experimental interview
components, 83–84
functions, 500–503
in Internet research, 519, 524
potential harm from, 85–86, 516, 518–519
Debugging research procedures, 496–499
Deception, 65, 75–79, 83–85, 204–205, 286; see also Ethical treatment of research participants
Deductive reasoning, 25
Definitions (in theories), 14–15, see also Operational definitions
Dehoaxing, 84–85
Demand characteristics, 201–205, 209
controlling, 203–205
detecting effects of, 201–202, 204–205
Dependability, see Reliability
Dependent variable, 16, 44, 137–140, 223, 313–314, 325, 545, see also Measurement; Measures; Variable
and demand characteristics, 204–205
Description, 4–5
Desensitization, 84–86, see also Debriefing
Design flaw analysis, 609–610
Developmental research, 45–47
Directory of Unpublished Experimental Measures, 175
Differential validity, see Validity of measurement
Difference scores, 586–587
Discriminant validity, 158–160
Dissertation Abstracts International, 125
Duplicate publication, 660
EBSCOhost, 125
Ecological fallacy, 523
Ecological validity, 214–215, 219, 221–224, 600–601, see also External validity; Generalizability
assessing, 229
components of, 214–216
in integrative research reviews, 600–601
of laboratory research, 230–235
of measurement, 446–447
research participants and, 218–220
of research settings, 219
of researcher and research procedures, 218, 221–224
Editors, 652–654
action, 652
copy, 654
journal, 652–653
production, 654
Educational Resources Information Center (ERIC), 124
Effect size, 490–492, 533–534, 602–605
and practical significance, 534–535
Eigenvalue, 343–344
Emics, 398
Empiricism (as a scientific value), 7–8, 541–542
Epistemology, 9–10, 136, 141, 234, see also Constructionism; Humanism; Logical positivism
Errors in research, 92–97, 133–134, see also Professional responsibilities
Estimator variables, see Variables
Ethical issues, see Application of research results, ethical issues; Ethical treatment of research participants; Evaluation research, ethical issues; Field experiments, ethical issues; Internet research, ethical issues; Professional responsibilities; Publication, ethical issues; Utilization of research results, ethical issues
Ethical treatment of research participants, 61–87
compensation of control groups, 68–69, 86
confidentiality of data, 64, 86–87, 521, see also Privacy
certificate of confidentiality, 87
in Internet research, 517
debriefing, 83–86, 500–501, 519, see also Post-experimental interview
deception, 65, 75–79, 83–85, 167, 493, 500–503, 517–519
active, 75–76
alternatives to, 77–78
in field experiments, 284
in Internet research, 517–519
minimizing harm from, 78–79, 500–501
objections to, 77
in observational research, 73–74, 199–200
deprivation, 68–69
discussion of in research report, 638
exempt research, 66
in field research, 73–74, 283–284
general principles, 62–63
government regulations, 62–65, 67, 71, 73, 87
harm, 76–77, 80–82, 92–93, see also Ethical treatment of research participants, risk
in deception research, 78–79
in Internet research, 518–519
inducements to participate in research, 71
informed consent, 62, 65–66, 72–75, 87
affirmative assent, 74
behavioral consent, 72–74
competence to give, 74
consent form, 65, 72–75, 78, 82–83
to deception, 77–78
in Internet research, 506, 515, 517–519
public behavior and, 73–74, 516, 519–520
and internal validity, 77, 202, 205
Institutional Review Board, 64–66, 72, 74, 81, 86
in Internet research, 516–519
privacy, 62, 64, 73, 82, 84, 86, 517–519, 521
psychological problems of participants, discovering, 81–82
responsibility for,
researcher, 63–64
sponsoring institution, 64–65
categories of, 67
in debriefing, 85–86
from deception, 77, 79, 493, 500, 518–519, 521
from deprivation, 68–69
minimal, 67–69, 71–72, 74–75, 80–81, 84–85
use of deception to minimize, 76–77
risk-benefit analysis, 63, 69–70
benefits of research, 63
subject pools, 70–71
voluntary participation, 62, 70–74, 82, 204
in Internet research, 517–518
Etics, 398
European Social Survey, 520
European Values Survey, 520
Evaluability assessment, 559–566
Evaluation apprehension, 165, 166, 197–198
Evaluation research, 33, 52, 557–587, see also Stakeholder
effectiveness criteria for, 558–559, 571–574
efficiency analysis in, 582–583
ethical issues, 68–69
evaluability assessment, 560–566
formative evaluation, 567–571
goal specification in, 561–564
impact assessment, 571–574
internal validity, 567, 578–579
needs assessment of, 559–560
null results in, 581–582
political context of, 585–586
process evaluation, 567
research designs, 575–580
client and program variation, 567–568
comparative outcome, 575–576
constructive, 575–576
dismantling, 575
measuring change in, 586–587
parametric, 576
program package, questions, 575–576
research strategies, 576–577
summative evaluation, 567, 571–574
target population, 567–568
theories in, 558, 565–566, 576
unintended effects in, 570–571
utilization, criteria for, 584–585
Exempt research, 66, see also Ethical treatment of research participants
Experimental realism, 218, 234, 494–495
Experimenter expectancy effects, 205–208
Experiments, 36–39, 52–53, 312; see also Between-subjects designs; Factorial designs; Field experiments; Manipulation of independent variable; Multiple group designs; Quasi-experiments; Within-subjects designs
and causality, 37, 46, 242, see also Alternative explanations for independent variable; Internal validity
conditions of comparison, 37, 244–247
control, 37, 68–69, 127, see also Control groups
experimental, 37
in evaluation research, 576–577
laboratory research settings and, 52–55, 222
Expert witness, 102–103
Explanation building, 390–392
Exploratory factor analysis, see Factor analysis
External validity, 24, 156–157, 213–235, see also Ecological validity; Generalizability
components of, 156–157, 214–217, 226–228
relationships among, 227–228
structural, 214–226
and internal validity, 234–235
participant sample factors, 219–221
in qualitative research, 416
research procedure factors, 219–221
setting factors, 217–219
time factors and, 225–226
threats to, 217
Extraneous variables, see Confounds; Variables
Facets, see Hypothetical constructs
Factor (in factor analysis), 343–345
eigenvalues, 343–344
extraction and rotation, 342–345
factor loading, 344–345
factor score, 345
scree plot, 343–444
Factor analysis, 338–350 see also Factor (in factor analysis)
confirmatory factor analysis, 346–358
goodness of fit tests in, 349–350
hypothesis testing in, 346–347, 350–353
for measure validation, 347–349
exploratory factor analysis, 339–345
considerations in, 341–345
for hypothesis testing, 346–347, 350–353
interpreting the factors, 344
principal components analysis, 342
structural validity and, 348–349
uses of, 341–345
Factorial designs, 264–276, 532–533
describing, 264
interaction effects in, 264–270, 318–319
and generalizability, 275
interpreting, 268–270, 532–533
and meta-analysis, 618
main effects in, 264–265
mixing between-and within subject independent variables in, 270–271
mixing manipulated and correlational independent variables in, 271, 329–331
order effects and, 270–274
possible outcomes of, 267–268, 532–533
uses of, 272–275
Fallacy of the mean, 541
Falsifiability, 22
Fatigue effects, 256
Federal guidelines for ethics (U. S.), see Ethical treatment of research participants
Field experiments, 52, 283–288
and laboratory experiments, 283–284
Field settings, 52–54, 283–285
Formative evaluation, 567–571; see also Evaluation research
Generalizability, 53–54 127, 154–155, 167, 213–214, 543–544, see also Ecological validity; External validity
in evaluation research, 567–568
and integrative literature reviews, 595–597
of natural-setting research, 232
overgeneralization, 99
in single-case research, 365, 368, 372, 379, 383
General Social Survey, 437, 438, 520
Grounded theory, 411–412, see also Content analysis
Guidelines for research with human participants, 62, 63, see also Ethical treatment of research participants
Halo effect, see Response bias
History (threat to internal validity), 48–49, 190, 209, 291, 297, 365–366, 378–379, 381, 383–384
local history, 579
Humanism, 9–11, 35, see also Social constructionism
Hypotheses, 5–6, see also Null hypothesis; Research questions
in confirmatory factor analysis, 346–347
for content analysis, 399–400
background for, 113
in evaluation research, 575–576
in integrative research reviews, 597–598
and measurement validation, 158, 160–163
research, 15, 128, 130, 246, 548–549
single-case research 369–370, 390–392
and theoretical propositions, 15, see also Mediational hypotheses
Hypothetical constructs, 12–15, 148–149, see also Operational definitions
and criterion-related evidence of validity, 158
as independent variables, 247–249
measurement of, 444–445, 448–449, 453, 461, 476
multidimensional, 13, 453, 455, 457
unidimensional, 13, 448, 455, 457
Impact criteria, see Single-case research
Implicit measures, see Measurement modalities
Incentives, for participant recruitment, 71–72, 512–513
Independent groups designs, see Between-subjects designs
Independent variable, 16, 37, 244–245, 541–545, see also Confounds; Experiments; Manipulation of independent variable
in applied research, 137–139
in correlational research (as label), 313–314
and external validity, 222–223
in field settings, 284–286
qualitative, 261–263
quantitative, 260–262, 329–330
in single–case research, 369–370, 372, 375–379, 385–388
Inductive reasoning, 25
Inference, 24–25, 131–209, 550, 532–541
and behavioral measures, 166–167
in coding systems, 209, 400–401, 410–412
and criterion-related evidence of validity, 154–155
in measurement, 148–149
and physiological measures, 168–169
Informed consent, see Ethical treatment of research participants
Inquiry audit, 418
Institutional Review Board (IRB), 64–66, 72, 81, 86, see also Ethical treatment of research participants
Instructions to research participants, 495–496
in questionnaires, 465–467
Instrumentation change, 191–192
Integrative literature review, 595–614, see also see also Literature review; Meta-analysis
best evidence approach to, 606–607
criteria for evaluating, 598–601
and ecological validity, 599–601
and generalizability, 595–597
hypothesis testing vs. exploratory approaches to, 597–598
inclusion criteria, 598–601
level of analysis in, 611–612, 619
locating relevant research, 598
mediating variables in, 604
moderator variables in, 603–604, 613, 617–620
narrative, 602–607
purposes, 593–594
research questions in, 597–598
researcher bias in, 600–601, 613
uses, 594–596
Interaction effect, see Factorial designs, interaction effects in
Intercept, 320–32
Internal consistency, see Reliability
Internal validity, 24, 127, 186–193, 244–245, 251–252, 507–508
effects of ethical requirements on, 76
in evaluation research, 578–579
and external validity, 234–235
in integrative literature reviews, 609–610
preexisting differences and, 290–296
of qualitative research, 412–413
role of control groups in, 193–194, 244–245
of single-case research, 365–366, 378–379, 383–384
threats to, 189–196, 251–252, 540, 578–580
analyzing patterns of flaws, 609–610
Interrater reliability, see Reliability
Internet research, 504–523
advantages, 506–508
data collection in, 506–508
disadvantages, 508–511
ethical issues in, 516–519
incentives, use of, 512–513
and internal validity, 505–506
online survey applications, selecting, 515–516
participant recruitment, 511–514
surveys given via, 472–473
Web sites, 472
Interviews, 419–431, see also Post–experimental interview
conducting, 425–426
data coding and analysis of, 428–430
information provided by, 419–420
sampling, 425
for single case research, 373–375
survey research and, 473
transcribing, 427–428
types of interviews, 421–422
types of questions, 423–424
writing an interview script, 422–423
Interrupted time series design, 296–297
Journal articles, 123–126, 140, 650–655, see also Peer review; Research reports
authorship credit and, 658–659
editorial review of, 652–654
rejection of, 655
reviewing others’, 656–658
Justice (in ethics), 62, see also Ethical treatment of research participants
Latent variables analysis, 347–348, 351–353, 357, see also Path analysis, Variables
Latin square design, 258
Leniency bias, see Response bias
Literal interpretation fallacy, 464–465
Literature review, 121–126, 628–633, see also Integrative literature review; Meta-analysis
as background for research, 121–122
conceptual, 594
narrative, 602–607
purpose of, 594–596
in research reports, 628–631
sources of information for, 123–128, 598
Logical positivism, 9, 35–36, 141, 410–411
Logistic regression analysis, 331–332, 334
Logit analysis, 333–334
Loglinear analysis, 332–334
Longitudinal approach (to research), 48–51
retrospective, 521
in single-case research, 364
Main effects, 264–265
Manifest variables, see Variables
Manipulation check, 247–248, 524
Manipulation of independent variable, 127, 139, 227, 247–251, see also Comparison groups; Control groups
acute, 227
chronic, 227
experimental condition, 244–245
and external validity, 223–224
in field experiments, 284–286
multiple stimuli in, 250–251
reliability of, 249
salience of, 250
in single-case research, 372, 394
in surveys, 437
strong, 249–250
validity of, 247–249
Masked review, 652
Matched random assignment, 254–255
Maturation (threat to internal validity), 190, 366, 368, 370, 383
Measurement, 48, 146–180, see also Change, measurement of; Coding systems; Measurement error; Measurement modalities; Measures; Rating scales; Reliability of measurement; Validity of measurement, discriminant
and ecological validity, 446–447
interval, 444–445
nominal, 444
ordinal, 444
ratio, 444–445
and statistical tests, 446
observed versus true score, 149–151, 453
reactivity in, 48, 76, 197–201
role of theory, 175–178
scales of, see Measurement, levels of
sensitivity of, 247–248, 451–452, 454
in single-case research, 366–368
steps in measurement process, 147–148
Measurement error, 149–151
systematic, 150–151
Measurement modalities, 164–173
controlling reactivity in, 76
choosing and evaluating, 171–180
implicit measures, 169–171, 172
physiological, 168–169, 172–173
controlling reactivity in, 165
use of multiple, 172–173, 453–454
Measures, 174–180, see also Coding systems; Confirmatory factor analysis; Exploratory factor analysis; Measurement; Rating scales; Reliability of measurement; Validity of measurement
criteria for evaluating, 175–180
locating, 175
normed, 174
norms, 174–177
psychometric tests, 174
research measures,
ad hoc, 174
developed, 174
Median split, 329–331
Mediational hypotheses, see Hypothesis testing; Confirmatory factor analysis; Path analysis
Member checking, 415–416
Mental Measurements Yearbook, 175
Meta-analysis, 580, 602–609, 616–619, see also Integrative literature review
comparison with narrative literature review, 607
interpretation of main effects in, 603–604, 618
understanding, 616–619
Minimal risk, see Ethical treatment of research participants
Mixed methods analysis, 430–431
Moderating variable, 16–17, 272, 546, 565–566, 604, 613, 617–619
Mortality, 196, 209, see also Attrition differential, 196
Multicollinearity, 329–331, 342
Multidimensional constructs, 13–14
Multiple correlation coefficient (R), 327
Multiple group designs, 259–260, 262–263, see also Factorial designs
interpreting results of, 263
Multiple operationism, 172–173, 181, see also Triangulation
Multiple regression analysis, 324–331, see also Logistic regression analysis; Multicollinearity
and analysis of variance, 329–331
criterion variable, 327–328
change in R2, 327
hierarchical, 325–326
simultaneous, 325
stepwise, 325
Multiple time series design, 298–299, see also Time series design
Multitrait-multimethod matrix, 159
Multiway frequency analysis, 332–334
Mundane realism, 219, 234, 494–495
Natural experiments, 52–55, 288–300
Naturalistic observation, see Observation research
Needs assessment, 559–560
Negligence, 94
Nonequivalent control group design, 289–293, 577–579
Nonrandom assignment, see Internal validity, selection threats
Nonspecific treatment effect, 245
Normed measure, see Measures
Normed fit index, 349–350
Novelty effect, 198, see also Response bias
Null hypothesis, 543–547
accepting, 546–547
prejudice against, 544, 580–582
Null results, 543–548, 581–582,
interpreting, 580–582
Observational research, 35–36, 166–167, 300–307
bias in, 199–200, 206, 304–307
categories of naturalistic observation, 301–303
effects on observer, 306–307
ethical issues, 73–74, 507, 516–517
on the Internet, 505, 507, 516–517
nonparticipant, 303
observer as participant, 303
pilot study for, 497
problems in, 304–306
Observed score, see Measurement
Operational definition, 15, 127–128, 132, 134, 541–542, 545, 636, 639
in archival research, 522–523
and external validity, 222
of goals in evaluation research, 564
in integrative literature reviews, 612–613
of study outcome, 607–609
of variables, 611–612
in single-case research, 371–374
of target populations, 485–486
Oral presentation (at a conference), 655–656
Order effects, 256–259
controlling, 257–258
detecting, 272–274
differential, 259
Outcome, see also Evaluation research
distal, 564
proximal, 564
social impact, 564
Page proofs, 654
Parametric statistics, see Statistical tests
Parsimony, 22
Partial correlation, see Correlational analysis, partial
Participant observation, see Observation research
Participant roles, 202–203
Participant sample, 54, see also Sampling; Target population
Passive research strategy, 42, see also Correlational research
Patched-up quasi-experiments, 293–296, 577–579
Path analysis, 351–358
limitations on interpretation, 356–357
prospective correlational research, 355
structural equation modeling, 353–355
Pattern matching, 390–391, see also Statistical tests
Pew Global Attitudes Project, 520
Piecemeal publication, 660
Pilot studies, 497–499
Placebo effect, 245
Plagiarism, 659–660
Planned comparisons, 263, 532–533, see also Statistical tests
Policy variables, see Variables
Population parameters, 437, see also Sampling
Populations, 54, see also Sampling
client, 560
research, 65, 139–140, 283, 405, 485
study, 485
of an intervention, 567–568
Postexperimental interview, 247, 500–503, see also Debriefing
Post-hoc analysis, 263
Power, see Statistical power
Practical significance, 128, 364, 371, 534–535, see also Statistical significance
Practice effects, 256, see also Order effects
Prediction, 6–7, 320–322, 530–531, see also Hypothesis testing
Predictor variables, see Multiple regression analysis
Preexperimental designs, 579–580
Preexisting groups, 290–295, see also, Selection (threat to internal validity)
Pretesting, 190–194, 254–255, 290–292
Primary research, 595, 602–612
Primary source, 123
Principle components analysis, see Factor analysis, exploratory factor analysis
Privacy, 62, 64, 73, 82, 84, 86, 521, 517–519, see also Ethical treatment of research participants, confidentiality of data
Privileged communication, 87
Professional responsibilities, 92–109, see also Errors in research; Expert witness; Publication, ethical issues; Research assistants; Utilization of research results, ethical issues
Program evaluation, see Evaluation research
Propositions, 15–17, see also Theories
Prospective correlational analysis, see Path analysis
PsycARTICLES, 125
PsycEXTRA, 124
PsycINFO, 125
Psychometric tests, 174
Publication, 140–141, 650–653, see also Convention presentations; Journal articles; Peer review; Research reports
censorship of, 106–108
errors in, 96
ethical issues, 657–660
Public behavior, 73–74, 519–520, see also Ethical treatment of research participants; Observational research
Publicness (as a scientific value), 8, 508
Qualitative research, 398–341, see also Content analysis; Interviews
as a research paradigm, 410–412
rigor in, 412–419
Quantitative research, 35–36
compared with qualitative research, 411–412, 430–431
Quasi-experiments, 288–300, 577–579, see Nonequivalent control group designs; Time series design
Questionnaires, 165, 538–575, see also Measures; Rating scales
administration, 469–475, see also Internet research
comparing methods of, 474–475
group, 469–470
Internet, 472–473, see also Internet research
interview, 472–473
mail, 470–471
telephone, 473–474
context effects in, 468–469
design of, 465–469
instructions in, 467
layout of, 466–467
use of existing measures in, 467
Questions (in questionnaires), 438–469
bias in, 459–461
comprehensibility of, 439–440
double-barrelled, 441
leading, 441
loaded, 441
open-ended, 438–439, see also Qualitative research
response formats, 468
sensitive topics, 443
sequencing of, 465
wording of, 439–443
R, see Multiple correlation coefficient
Random assignment to condition, see Research participants
Random error, see Measurement error
Rating scales, see also Questions; Response bias
comparative, 447–448
graphic, 450–451
Guttman, 457
interpreting responses to, 464
itemized, 448–450
Likert, 454–456
multi-item, 453–458
numerical, 451–452
paired comparison, 452–453, 447–448
rank-order, 447–448
semantic differential, 458
summated, 454–456
Thurstone, 456–457
visual analog, 450–451
Reactivity, 48, 76, 197–201, see also Response bias
controlling, 198–200
in longitudinal research, 48
in observational research, 199–201
in self–report measures, 165, 200, 466
Reciprocal relationship, 44
Reconstructive memory, 304–305, see also Coding systems
Recording keeping, see Coding systems
Reference group effect, 464
Reflexivity, 418, see also Coding systems
Regression analysis, see Bivariate regression; Multiple regression analysis
Regression coefficient, 322–327
Regression equation, 44–45, 320–322
Regression to the mean, see Statistical regression
Rehearsal of research procedures, 207, 209
Reification, 12
Relationships between variables, 260–262
additivity, 313–314
assessing nonlinear, 260–262, 313
linear, 260–262
negative, 260–261
positive, 260–261
Reliability of measurement, 148–154, 179, 452–454
alternate forms, 151–152
choosing among forms, 153
and correlational research, 314, 320
Cronbach’s alpha, 152–153
of data coders, 406–407
of difference scores, 586–587
internal consistency, 152–153
and measurement error, 149–151
in observational research, 304–306
in qualitative research, 406–407, 417
split-half, 152
standards for, 153–154, 176–177
and validity of measurement, 149–150, 176–177, 179
Reliable change index, 587
Repeated measures designs, see Within-subjects designs
Replication research, 8, 133–136, 214–215
considerations in conducting, 135–136
and errors in research, 133
exact, 133–134
and external validity, 228
in nonequivalent control group designs, 295–296
in qualitative research, 418–419
replication and extension, 135
results of, 135
in single-case research, 365–366, 368–371
Research, see also Evaluation research; Replication research; Research reports; Utilization of research results
action, 34–35
analog, 233–234
and application, 25–27
applied, 32–33, 106, 213–214, 234
archival, 519–523
artificiality in, 222
cohort-sequential, 49–51
criteria for evaluating, 23–24, 600–601
cross-sectional, 46–47
describing results, 530–535
developmental, 45–50
errors in, see Errors in research
field, 52–54, 282–285, see also Field experiments; Natural experiments; Non-equivalent control groups design; Time-series design
ideas for, 119–121
idiographic approach to, 41, 363
longitudinal, 48–51, see also Single case research; Time series
retrospective, 521
single case research, 366
misconduct, 96–97
naturalism in, 52–55, 234, 288–308, see also Observational rsearch
nomothetic approach to, 41, 363
pilot studies, 497–499, 627, 639
realism in, 234
experimental, 218, 234, 494–495
relation to theory and application, 25–27
reports (evaluation of), 127–128, 652–653
socially sensitive, 106–108
steps in, 23
strategies, 36–52, see also Case study research; Correlational research; Experiments
comparisons among, 44–45
correlational, 41–46
and research settings, 52–55
and theory, see Theories topic, 114–115
tradeoffs in, 54–55, 234, 492, 495
uses of, 26–27
Research assistants, 63, 209, 503–504, 524, see also Coding systems, coder qualifications and training
Research participants, 127, 219–221, 283, 634, see also Populations; Sampling
assignment to conditions of the independent variable, 226, 253–255
in field experiments, 283–284
matched random, 254–255
nonrandom, 194–195
simple random, 252–254
use of preexisting groups, 290–296, 577–579
children as, 62–63, 68, 74, 79–82, 87
and external validity, 219–221
harm to, see Ethical treatment of research participants
Internet research, recruiting for in, 511–513
roles, 202–204
volunteer, 70–74, 195, 221, 516–517
Research process, steps in, 23
Research proposal, 114, 132–133, 496–497
Research questions, 113–143
bias in formulation of, 141–143
characteristics, 141–143
formulating, 115–121
for evaluation research, 558–559, 575–576
for integrative literature reviews, 597–598, 616–617
Research reports, 625–647, see also Convention presentations; Journal articles; Publication
abstract, 647
apparatus, 634–636
common problems in, 652–653
content of, 627–647
criteria for evaluating, 127–128, 652–653
discussion section, 642–646, see also Inference; Interpretation of research results
introduction, 628–633, see also Literature review
method section, 633–639
and nonresearchers, 648–649
preliminary work, 627–628
results section, 638–643
structure of, 626
title, 647
writing style in, 625–628
Research sample, 54, 219–221, see also Sampling
evalution of, 176–178
Research settings, 284–285, 493–495
and external validity, 127, 217, 233–234
in field experiments, 52–53, 284–285
and generalizability, 233–234, see also Ecological validity
natural, 32–33, 52–53, 233–234, 284–285, see also Observational research
and research participants, 54, 290–296
and research strategies, 52–53
and utilization of research results, 140
Respect, in ethics, 62
Response bias, 179–180, 458–464, see also Reactivity
category anchoring, 459–460
cultural response sets, 463–464
estimation, 460
extremity, 463
halo, 463
leniency, 463
social desirability, 165, 179–180, 461
Response competition, 169–170
Restriction in range, 314–316, 342, 545
Reversal of treatment, 296–300, 376–378
Reverse scoring, 455
Reviewing manuscripts, 656–657
Right to withdraw consent, see Ethical treatment of research participants
Risk-benefit analysis (in ethics), 63, 66–70, see also Ethical treatment of research participants
Role-playing (as alternative to deception), 77–78
Root mean square of approximation, 349–350
Sample (vs. population), 54, 485–486
Sample size, 490–493, 534–546, see also Statistical power
Sample survey, 219–221
Sampling, 486–490
cluster, 488
haphazard, 489
in integrative literature reviews, 601
in Internet research, 508–509
multistage cluster, 488
nonprobability, 488–489
probability, 486–488
purposive, 489
random, 486–488
simple, 486–487
stratified, 487
restricted, 219–220
snowball, 489–490
time, 225
Sampling frame, 486–488
Scales, see Measurement, levels of; Rating scales
Science, 4–10
goals of, 4–7
description, 4–5
epistemology, 9–10, 136, 141, 234
exploitation of, 97–98
prediction, 6–7
understanding, 5–6
as an institution, 101
values of, 7–8
Scientists as individuals, 101
Scree plot, 343
Secondary source, 123
Selection (threat to internal validity), 194–196, 290–295, see also Internal validity, threats to; Quasi–experiments
Selective attention, 304
Self-report measures, see Measurement modalities
Sensitization effects, 256–258; see also Order effects
Settings, see Research settings
Significance, see Practical significance; Statistical significance
Simulation research, 77, 80–81
Single-case experiments, 52–53, 363–371, 376–393, 580, see also Single-case research
A-B design, 376–377
A-B-A design, 376
A-B-C-B design, 377–378
alternating treatments design, 383–384
baseline, 366, 370–371, 375–376
stability of, 385–389
trends in, 385–389
changing criterion design, 384–385
in evaluation research, 580
multiple baseline design, 378–383
simultaneous treatment design, 383–384
Single-case research, 52–53, 369–393, see also Case study research; Single case experiments
control in, 369–370
data analysis in, 389–393
data collection in, 373–375
impact criteria, 370
in evaluation research, 580
measurement criteria, 366–368
objections to, 364–365
in integrative literature reviews, 603
in psychology, 363–364
replication in, 368–369
treatment criteria, 370–371
uses of, 365
validity criteria, 365–366
Skepticism (as a scientific value), 8
Slope, 320–321
Social desirability response bias, see Response bias
Socially sensitive research, 107
Social constructionism, 410–411, 536–537, see also Humanism
Social Science Citation Index, 126
Societal mentor, 100–101
Sociological Abstracts, 125
Solomon four-group design, 191, 194
Specification, 17
Split-half reliability, see Reliability
Stakeholder, 565, 585–586, see also Evaluation research
Statistical conclusion validity, 24, 127
Statistical power, 127, 490–493, 546–547, 551, 604–605
and median splits, 330
Statistical regression, 192–193, 366, 370
Statistical significance, 127, 243–244, see also Effect size; Practical significance; Statistical power
and integrative literature reviews, 604–605
and median splits, 329–331
Statistical tests, 243–244, see also Analysis of covariance; Analysis of variance; Confirmatory factor analysis; Correlational analysis; Explanatory factor analysis; Path analysis; Planned comparisons; Statistical power; Structural equation modeling
a priori contrasts, 263
change, measuring, 586–587
goodness of fit, 349–350
inferential, 531–532
and level of measurement, 446
misuse of, 93–95
omnibus, 532
planned comparisons, 532–533
post-hoc analysis, 263
and research design, 243–244
in research reports, 639–642, 649–650
robustness of, 446
in single-case research, 392–393
Stimuli (multiple), 250–251
Street theater strategy, 286
Structural component (of generalizability), see Generalizability
Structural equation modeling, 353–358, see also Path analysis; Latent variables analysis
Structural validity, see Validity
Subject pools, 70–71
Subgroup differences (in correlational research), 316–317
Subscales, 453–456
Summative evaluation, 567, see also Evaluation research
Target population, 54, 485, 567–568
Tentativeness (as a scientific value), 8, 24
Testing (threat to internal validity), 190–191
Test reactivity, 48
Test-retest reliability, see Reliability
Test sensitization, 48
Tests in Print, 175
Theories, 10–22, 114, 116–119, 121–124, 530, 536–539, 543, 544, 548–549
and application, 26–17
assumptions, 10–12
boundary conditions of, 120, 136, 214
characteristics of, 17–18
components of, 10–17
criteria for evaluating, 21–22
in evaluation research, 565–566, 575–576
hypothetical constructs in, 12–15
in integrative research, 594, 600
and measurement, 175–178
propositions, 15–17
purposes of, 18–21
in qualitative research, 400–401, 403, 407, 411, 414, 417
in single-case research, 390–392
social constructionist perspective on, 410–411, 536–537,
Theory maps, 117–119
Third variable problem, 44
Time-lagged comparisons, 49–50
Time precedence of cause, 5, 37, 42–44, 52, 242
Time-related threats to validity, see Confounds, Internal validity
Time series design, 296–300
control series, 297–300
interrupted, 296–297
multiple time series, 298
Treatment, see Independent variable
Treatment diffusion, 578
Treatment variance, 243
Trends, see Single-case experiments, baseline
Triangulation, 413–414, see also Multiple operationism
True score, see Measurement
Truth test, 137
Type I error, 133–134, 490, 532, 544, 605–606, see also Statistical significance
Type II error, 134, 490–493, 532, 544–546, 604–608, 610, 616, see also Statistical power
Understanding (as goal of science), 5–6
Unidimensional constructs, 13
Unit of analysis, 372–373, 401–402
Utility test, 137
Utilization of research results, see also Application of research results
conceptual, 583–584
instrumental, 583
persuasive, 584
political context of, 99–105, 584
and research design, 137–141
and research population, 139–140
and research setting, 140
Validity of measurement, 148–150, 154–164
of coded data, 406–407
construct-related evidence, see Construct validity
content-related evidence, 155–156
convergent, 158–159
criterion-related evidence, 156–157, 160–162
differential, 162–164
discriminant, 158–160
and reliability of measurement, 149–150, 161, 453–545
substantive validity, 157–158, 160
Validity of research, 23–24, 550, 543, see also Construct validity; Ecological validity; External validity; Internal validity; Single-case research, validity criteria; Statistical conclusion validity
in integrative literature reviews, 600–601, 609–610
and sampling procedures, 485–486
theoretical, 600
Variable, 12, 15, 324–325, see also Dependent variable; Independent variable; Factorial designs
dependent, 16, 44, 137–140, 223, 313–314, 325, 545, see also Measurement
estimator, 138–139
extraneous, 186, 251–252, 274–275, 283–284, 287
controlling in experiments, 546
in field research, 283–284
in integrative literature reviews, 603–604, 613, 617–620
latent, 319, 346–348, 351–357, 401, see also Latent variables analysis
manifest, 148–149, 350–351, 353–354, 401
mediating, 16–17, 546, 564–565, moderating, 16–17, 272, 546, 565–566, 603–604, 613, 617–619
policy, 137–139
predictor, 324–331
relationships between, 530–531
Verisimilitude, 216, 224, 227–228, 232–233
Visual data analysis, 392–393
Voluntary participation, see Ethical treatment of research participants
Volunteers, 70–74, 195, 221, 516–517
Wash-out period, 258
Within-subjects design, 252, 255–259, 270–271, 633–634
compared to between-subjects design, 255–256
World-Wide Web, see Internet; Internet research
Zero-order correlation, 323–324, 327, see also Correlational analysis