Adaptive Experimental Design: Experimentation is fundamental to the advancement of psychological science. One of the greatest challenges in model comparison is designing an experiment that clearly distinguishes between models. In design optimization, we are interested in identifying an experimental design under which the experiment is likely to produce the most differentiating outcomes between the models under consideration in some defined sense. For example, in designing an experiment investigating whether the rate of forgetting over time follows a power curve or an exponential curve, the experimenter must determine “optimally” the values of time intervals between the study and test sessions. Recent developments in sampling-based Monte Carlo methods in Bayesian statistics make it possible to determine these values and thereby identify an optimal design. The work has been supported by the National Institute of Mental Health.
Manuscripts in Preparation and under Review
Myung, J. I., Cavagnaro, D. R., & Pitt, M. A. (under review). Squeezing every once of information from an experiment: Adaptive experimental design.
Montenegro, M., Myung, J. I., & Pitt, M. A. (in preparation). Analytic expressions for the REM model of recognition memory.
Cavagnaro, D. R., Gonzalez, R., Myung, J. I., & Pitt, M. A. (under review). Optimal decision stimuli for risky choice experiments: An adaptive approach.
Recent Presentation Slides
Adaptive design optimization for model discrimination.
Optimal experimental design for model discrimination.
Tutorial on model comparison methods (updated July 2011).
Tutorial on model comparison methods (old).
Publications
Journal
Articles
Cavagnaro, D. R., Pitt, M. A. & Myung, J. I. (2011). Model disccrimination through adaptive experimentation. Psychonomic Bulletin & Review, 18(1), 204-210.
Wu, H., Myung, J. I., & Batchelder, W. H. (2010). Minimum description length model selection of multinomial processing tree models. Psychonomic Bulletin & Review, 17, 275-286.
Wu, H., Myung, J. I., & Batchelder, W. H. (2010). On the minimum description length complexity of multinomial processing tree models. Journal of Mathematical Psycholog, 54, 291-303.
Cavagnaro, D. R., Myung, J. I., Pitt, M. A. & Kujala, J. (2010). Adaptive design optimization: A mutual information-based approach to model discrimination in cognitive science. Neural Computation, 22, 887-905.
Myung, J. I & Pitt, M. A. (2009). Optimal experimental design for model discrimination. Psychological Review, 116, 499-518. [Click here to go to the design optimization (DO) page with documentation and C++ code (under construction).]
Myung, J. I., Tang, Y. & Pitt, M. A. (2009). Evaluation and comparison of computational models. Methods in Enzymology, 454, 287-304.
Pitt, M. A., Myung, J. I., Montenegro, M. & Pooley, J. (2008). Measuring model flexibility with parameter space partitioning: An introduction and application example. Cogntive Science, 32, 1285-1303.
Myung, J. I., Pitt, M. A. & Navarro, D. J. (2007). Does response scaling cause the generalized context model to mimic a prototype model? Psychonomic Bulletin & Review, 14, 1043-1050.
Pitt, M. A., Myung, J. I. & Altieri, A. (2007). Modeling the word recognition data of Vitevitch and Luce (1998): Is it ARTful? Psychonomic Bulletin & Review, 14, 442-448.
Myung, J. I., Montenegro, M., & Pitt, M. A. (2007). Analytic expressions for the BCDMEM model of recognition memory. Journal of Mathematical Psychology, 51, 198-204.
Pitt, M. A., Kim, W., Navarro, D. J. & Myung, J. I. (2006). Global model
analysis by parameter space partitioning.
Psychological Review, 113, 57-83. [Click here to go to the Parameter Space Partitioning (PSP)
homepage with tutorial, Java applets, and Matlab programs (under
construction).]
Myung, J. I., Navarro, D. J. &
Pitt, M. A. (2006). Model selection
by normalized maximum likelihood. Journal
of Mathematical Psychology, 50, 167-179.
Myung, J. I., Karabatsos, G., &
Iverson, G. (2005). A Bayesian
approach to testing decision making axioms. Journal of Mathematical
Psychology, 49(3), 205-225.
Navarro, D. J., Pitt, M. A. & Myung,
I. J. (2004). Assessing
the distinguishability of models and the informativeness of data .
Cognitive Psychology, 49, 47-84.
Myung, I. J. & Pitt, M. A.
(2004). Model
comparison methods . Methods in Enzymology, 383, 351-366.
Myung, I. J. (2003). Tutorial on
maximum likelihood estimation . Journal of Mathematical Psychology ,
47, 90-100.
Pitt, M. A., Kim, W., & Myung, I. J. (2003).
Flexibility
vs generalizability in model selection . Psychonomic Bulletin &
Review , 10, 29-44.
Pitt, M A & Myung, I J. (2002). When a good fit
can be bad . Trends in
Cognitive Sciences , 6(10) , 421-425. (Reprint
posted with permission from Elsevier Science. Single copies of this article can
be downloaded and printed for the reader's personal research and
study.)
Pitt, M. A., Myung, I. J., & Zhang, S. (2002). Toward a method
of selecting among computational models of cognition . Psychological
Review, 109(3) , 472-491.
Myung, I. J., Balasubramanian, V., &
Pitt, M. A. (2000).
Counting
probability distributions: Differential geometry and model selection .
Proceedings of the National Academy of Sciences USA, 97, 11170-11175.
Myung, I. J., Kim, C., & Pitt, M. A. (2000). Toward
an explanation of the power-law artifact: Insights from response surface
analysis. Memory & Cognition, 28 , 832-840.
Myung, I. J.
(2000) The
importance of complexity in model selection. Journal of
Mathematical Psychology, 44 (1) , 190-204.
Myung, I. J., & Pitt,
M. A. (1997). Applying
Occam's razor in modeling cognition: A Bayesian approach. Psychonomic
Bulletin & Review, 4 , 79-95.
Myung, I. J., & Shepard, R. N.
(1996). Maximum
entropy inference and stimulus generalization. Journal of Mathematical
Psychology, 40 , 342-347.
Myung, I. J., Ramamoorti, S, & Bailey,
A. D., Jr (1996). Maximum
entropy aggregation of expert predictions. Management Science. 42(10)
, 1420-1436.
Myung, I. J. (1994). Maximum
entropy interpretation of decision bound and context models of
categorization. Journal of Mathematical Psychology, 38 , 335-365.
Myung, I. J. (1994). Is the
representation meaningful?: A measurement theoretic view. Behavioral and
Brain Sciences, 17(4) , 677-678.
Myung, I. J., Colbert, C. M., &
Levy, W. B. (1994). A
computational hypothesis of probability inference in neural networks and some
relations to psychological models. Journal of Biological Systems,
2(3) , 367-384.
Busemeyer, J. R., Myung, I. J., & McDaniel M. A.
(1993). Cue competition effects: empirical tests of adaptive network learning
models. Psychological Science, 4(3) , 190-195.
Busemeyer, J. R.,
Myung, I. J., & McDaniel M. A. (1993). Cue competition effects: theoretical
implications for adaptive network learning models. Psychological Science,
4(3) , 196-202.
Myung, I. J., and Busemeyer, J. R. (1992).
Measurement free tests of a general state space model of prototype learning.
Journal of Mathematical Psychology, 36(1), 32-67.
Busemeyer, J. R., and
Myung, I. J. (1992). An adaptive approach to human decision making: learning
theory, decision theory, and human performance. Journal of Experimental
Psychology: General, 121(2) , 177-194.
Myung, I. J., and Busemeyer,
J. R. (1989). Criterion learning in a deferred decision making task. American
Journal of Psychology, 102 , 1-16.
Busemeyer, J. R., and Myung, I.
J. (1988). A new method for investigating prototype larning. Journal of
Experimental Psychology: Learning, Memory, and Cognition, 14 , 3-11.
Busemeyer, J. R., and Myung, I. J. (1987). Resource allocation decision
making in an uncertain environment. Acta Psychologica, 66 , 1-19.
Edited volumes/journal issues
Grünwald, P., Myung,
I. J., & Pitt, M. A., eds., (2005). Advances
in Minimum Description Length: Theory and Applications . MIT
Press.
Myung, I. J., Forster, M., & Browne, M. W., eds. (2000). Special issue
on model selection . Journal of Mathematical Psychology, 44 , 1-2.
Chapters/Encyclopedic Entries
Cavagnaro, D. R., Myung, J. I., & Pitt, M. A. (In press). Mathematical modeling. In Todd D. Little (ed.), The Oxford Handbook of Quantitative Methods, Oxford University Press, New York, NY.
Myung, J. I., Tang, Y. & Pitt, M. A. (2010). Evaluation and comparison of computational models. In Michael L. Johnson (ed.), Essential Numerical Computer Methods, pp. 511-527. Elsevier. (Note: This is an updated reprint version of Myung, Tang & Pitt (2009) for a standalone volume.)
Myung, J. I., Karabatsos, G. & Iverson, J. G. (2008). A statisticians view on Bayesian evaluation of informative hypotheses. In H. Hoijtink, I. Klugkist & P. Boelen (eds.) Bayesian Evaluation of Informative Hypotheses (pp. 309-327). Springer, Berlin.
Myung, I. J., &
Navarro, D. J. (2005). Information
matrix . In B, Everitt & D. Howel (eds.), Encyclopedia of Statistics
in Behavioral Science, Vol. 2, pp., 923-924. Wiley.
Navarro, D. J.
& Myung, I. J. (2005). Model
evaluation. In B, Everitt & D. Howel (eds.), Encyclopedia of
Statistics in Behavioral Science, Vol. 3, pp. 1239-1242.
Wiley.
Myung, I. J., Pitt, M A., & Kim, W. (2005). Model
evaluation, testing and selection . In K. Lambert and R. Goldstone (eds.)
The Handbook of Cognition, pp. 422-436. Sage Publication.
Su, Y.,
Myung, I. J. & Pitt, M. A. (2005). Minimum
description length and cognitive modeling . In P. Grunwald, I. J. Myung, I.
J., & M. A. Pitt (eds.) Advances in Minimum Description Length: Theory
and Applications, pp.411-433. MIT Press.
Myung, I. J. &
Pitt, M. A. (2003). Model fitting. In L. Nadel (ed.), The Encyclopedia of
Cognitive Science , Vol. 3, pp. 47-51. London, UK: Macmillan.
Myung,
I. J., & Pitt, M. A. (2002). Mathematical modeling. In J. Wixted (ed.), Stevens' Handbook of Experimental Psychology (Third Edition), Volume IV
(Methodology) , pp. 429-459. New York, NY: John Wiley & Sons.
Myung, I. J. (2001). Computational approaches to model evaluation. In N.
J. Smelser and P. B. Baltes (eds.), The International Encyclopedia of the
Social and Behavioral Sciences , pp. 2453-2457. Oxford, UK: Elsevier.
Myung, I. J., & Pitt, M. A. (1998). Issues in
selecting mathematical models of cognition. In J. Grainger & A. M.
Jacobs (eds.), Localist Connectionist Approaches to Human Cognition , pp.
327-355. Lawrence Erlbaum Associates.
Busemeyer, J. R., and Myung, I. J.
(1989). An adaptive theory of human decision making. In D. Vickers and P. Smith
(eds.), Human Information Processing: Measurements, Mechanisms, and
Models , pp. 461-469, XXIV International Congress of Psychology. North
Holland.
Conference Proceedings
Pitt, M. A. & Myung, J. I. (2011). How is hair gel quantified? In E. J.
Davelaar (ed.), Proceedings of the Twelfth Neural Computation and
Psychology Workshop (pp. 356-360). London, UK: World Scientific.
Tang, Y., Young, C., Myung, J. I., Pitt, M. A. & Opfer, J. (2010). Optimal inference and feedback for representational change. In S. Ohlsson & R. Catrambone (eds.), Proceedings of the 32nd Annual Meeting of the Cognitive Science Society (pp. 2572-2577). Austin, TX: Cognitive Science Society.
Cavagnaro, D. R., Pitt, M. A. & Myung, J. I. (2009). Adaptive design optimization in experiments with people. Advances in Neural Information Processing Systems, 22, 234-242.
Cavagnaro, D. R., Tang, Y., Myung, J. I. & Pitt, M. A. (2009). Better data with fewere participants and trials: Improving experimental efficiency with adaptive design optimization. In N. A. Taatgen & H. van Rijn (eds.), Austin, TX: Cognitive Science Society.
Myung, J. I., Pitt, M. A., Tang, Y. & Cavagnaro, D. R. (2009). Bayesian adaptive optimal design of psychology experiments. In Proceedings of the 2nd International Workshop in Sequential Methodologies (IWSM2009), CD ROM format (Troyes, France: June, 2009).
Myung, J. I., Pitt,
M. A. & Navarro, D. J. (2005). Model selection
in cognitive science as an inverse problem. Proceedings of SPIE, vol. 5674 (Computational Imaging
III), 219-228.
Kim, W., Navarro, D. J., Pitt, M. A. & Myung,
I. J. (2004). An MCMC-based
method of comparing connectionist models in cognitive science. In B.
Schokolpf (ed.), Advanceds in Neural Information Processing Systems , vol
16, 937-944.
Navarro, D. J., Myung, I. J., Pitt, M. A. & Kim, W.
(2003). Global
model analysis by landscaping. In R. Alterman & D. Kirsh (eds.), Proceedings of the 25th Annual Meeting of the Cognitive Science Society,
CD-ROM format, (Boston, MA: August, 2003).
Myung, I. J. & Pitt, M.
A. (2001). A
minimum description length approach for selecting among qualitative models of
cognition . In L Chen and Y Zhuo (eds.), Proceedings of the Third
International Conference on Cognitive Science (ICCS2001: Beijing, China),
pp. 364-369. University of Science and Technology of China Press.
Myung,
I. J., Pitt, M. A., Zhang, S., & Balasubramanian, V. (2001). The use of MDL
to select among computational models of cognition . In T K Leen, T G
Dietterich & V Tresp (eds.), Advances in Neural Information Processing
Systems, vol. 13 ., pp. 38-44. MIT Press.
Myung, I. J., Brunsman IV,
A. E., & Pitt, M. A. (1999). True to thyself: Assessing whether
computational models of cognition remain faithful to their theoretical
principles. In M. Hahn & S.C. Stoness (eds.), Proceedings of the 21st
Annual Conference of the Cognitive Science Society , pp. 462-467. Mahwah,
New Jergey: Lawrence Erlbaum Associates.
Myung, I. J., Kim, C., &
Levy, W. B. (1997). Context-dependent
recognition in a self-organizing recurrent network. In M.G. Shafto & P.
Langley (eds.), Proceedings of the Nineteenth Annual Meeting of the Cognitive
Science Society , pp. 530-535. Mahwah, New Jersey: Lawrence Erlbaum
Associates.
Kim, C., & Myung, I. J. (1995). Incorporating
real-time random effects in neural networks: A temporal summation mechanism.
In J.D. Moore & J.F. Lehman, Proceedings of the Seventeenth Annual
Meeting of the Cognitive Science Society , pp. 472-477. Hillsdale, NJ:
Lawrence Erlbaum Associate.
Myung, I. J., and Busemeyer, J. R. (1989). A
state-space model for prototype learning. In G. Olson and E. Smith (eds.),
Proceedings of the Eleventh Annual Meeting of the Cognitive Science
Society , pp. 50-57. Earbaum Associate.