Traditional methods of analyzing data from psychological experiments are based on the assumption that there is a single random factor (normally participants) to which generalization is sought. However, many studies involve at least two random factors (e.g., participants and the targets to which they respond, such as words, pictures, or individuals). The application of traditional analytic methods to the data from such studies can result in serious bias in testing experimental effects. In this review, we develop a comprehensive typology of designs involving two random factors, which may be either crossed or nested, and one fixed factor, condition. We present appropriate linear mixed models for all designs and develop effect size measures. We provide the tools for power estimation for all designs. We then discuss issues of design choice, highlighting power and feasibility considerations. Our goal is to encourage appropriate analytic methods that produce replicable results for studies involving new samples of both participants and targets.


Article metrics loading...

Loading full text...

Full text loading...


Literature Cited

  1. Baayen RH, Davidson DJ, Bates DM. 2008. Mixed-effects modeling with crossed-random effects for subjects and items. J. Mem. Lang. 59:390–412 [Google Scholar]
  2. Barr DJ, Levy R, Scheepers C, Tily HJ. 2013. Random effects structure for confirmatory hypothesis testing: keep it maximal. J. Mem. Lang. 68:255–78 [Google Scholar]
  3. Bates D, Kliegl R, Vasishth S, Baayen H. 2015. Parsimonious mixed models. arxiv:1506.04967v1 [stat.ME]
  4. Bolker BM, Brooks ME, Clark CJ, Geange SW, Poulsen JR. et al. 2009. Generalized linear mixed models: a practical guide for ecology and evolution. Trends Ecol. Evol. 24:3127–35 [Google Scholar]
  5. Bond CF, DePaulo B. 2008. Individual differences in judging deception: accuracy and bias. Psychol. Bull. 134:477–92 [Google Scholar]
  6. Clark HH. 1973. The language-as-fixed-effect fallacy: a critique of language statistics in psychological research. J. Verb. Learn. Verb. Behav. 12:335–59 [Google Scholar]
  7. Cohen J. 1988. Statistical Power Analysis for the Behavioral Sciences Hillsdale, NJ: Erlbaum [Google Scholar]
  8. Goldstein H, Browne W, Rasbash J. 2002. Partitioning variation in multilevel models. Underst. Stat. 1:223–31 [Google Scholar]
  9. Hönekopp J. 2006. Once more: Is beauty in the eye of the beholder? Relative contributions of private and shared taste to judgments of facial attractiveness. J. Exp. Psychol. Hum. Percept. Perfor. 32:199–209 [Google Scholar]
  10. Hox JJ. 2010. Multilevel Analysis: Techniques and Applications New York: Routledge [Google Scholar]
  11. Judd CM, McClelland RG, Ryan CS. 2008. Data Analysis: A Model Comparison Approach New York: Routledge [Google Scholar]
  12. Judd CM, Westfall J, Kenny DA. 2012. Treating stimuli as a random factor in social psychology: a new and comprehensive solution to a pervasive but largely ignored problem. J. Pers. Soc. Psychol. 103:54–69 [Google Scholar]
  13. Kenny DA, Kashy DA, Cook WL. 2006. Dyadic Data Analysis. New York: Guilford Press
  14. Raudenbush SW, Bryk AS. 2002. Hierarchical Linear Models: Applications and Data Analysis Methods Thousand Oaks, CA: Sage [Google Scholar]
  15. Satterthwaite FE. 1946. An approximate distribution of estimates of variance components. Biom. Bull. 2:6110–14 [Google Scholar]
  16. Smith ER. 2014. Research design. Handbook of Research Methods in Social and Personality Psychology HT Reis, CM Judd 27–48 Cambridge, UK: Cambridge Univ. Press [Google Scholar]
  17. Snijders T, Bosker R. 2011. Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling Thousand Oaks, CA: Sage [Google Scholar]
  18. Stroup WW. 2012. Generalized Linear Mixed Models: Modern Concepts, Methods and Applications New York: CRC Press [Google Scholar]
  19. Toma C, Corneille O, Yzerbyt V. 2012. Holding a mirror up to the self: egocentric similarity beliefs underlie social projection in cooperation. Pers. Soc. Psychol. Bull. 38:1259–71 [Google Scholar]
  20. Welch BL. 1947. The generalization of ‘Student's’ problem when several different population variances are involved. Biometrika 34:28–35 [Google Scholar]
  21. Westfall J, Judd CM, Kenny DA. 2015. Replicating studies in which samples of participants respond to samples of stimuli. Pers. Psychol. Sci. 10:390–99 [Google Scholar]
  22. Westfall J, Kenny DA, Judd CM. 2014. Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli. J. Exp. Psychol. Gen. 143:220–45 [Google Scholar]
  23. Winer BJ. 1971. Statistical Principles in Experimental Design New York: McGraw-Hill [Google Scholar]

Data & Media loading...

Supplementary Data

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error