Markus Steiner, Florian Seitz, and I have a new paper (just published in Decision) in which we investigate the cognitive processes underlying people’s self-reports of their risk preferences. Specifically, we were interested in the information-integration processes that people may rely on during judgment formation, with a particular focus on the type of evidence people may consider when rendering their self-reports. In doing so, we aimed to contribute to a better understanding of why self-reports typically achieve high degrees of convergent validity and test-retest reliability, thus often outperforming their behavioral counterparts (i.e., monetary lotteries and other lab tasks).

To achieve these goals we employed the process-tracing method of aspect listing, to thus gain “a window into people’s mind” while they render self-reports. Our cognitive modeling analyses illustrated that people are particularly sensitive to the strength of evidence of the information retrieved from memory during judgment formation. Interestingly, people’s self-reported risk preferences and the strength of evidence of the retrieved aspects remained considerable stable in a retest study (i.e., across a one-month interval). Moreover, intraindividual changes in the latter were closely aligned with intraindividual changes in the former – suggesting that a relatively reliable psychological mechanism is at play when people render self-reports.

Beyond our quantitative modeling analyses, the process-tracing method of aspect listing also rendered possible more qualitative insights, such as concerning the sources and contents of the information people retrieved from memory (see the word clouds below). To learn more about all further details on this, please have a look at the paper!

Steiner, M., Seitz, F., & Frey, R. (2021). Through the window of my mind: Mapping information integration and the cognitive representations underlying self-reported risk preference. Decision, 8, 97–122. doi:10.1037/dec0000127 | PDF

First appeared on https://renatofrey.net/blog

A large body of research has documented the relatively poor psychometric properties of behavioral measures of risk taking, such as low convergent validity and poor test–retest reliability. In this project we examined the extent to which these issues may be related to violations of “representative design” – the idea that experimental stimuli should be sampled or designed such that they represent the environments to which measured constructs are supposed to generalize.

To this end, we focused on one of the most prominent behavioral measures of risk taking, the Balloon Analogue Risk Task (BART). Our analyses demonstrate that the typical implementation of the BART violates the principle of representative design, and strongly conflicts with the expectations people might have formed from real balloons. We conducted two extensive empirical studies (N = 772 and N = 632), aimed at testing the effects of improved representative designs. Indeed, participants acquired more accurate beliefs about the optimal behavior in the BART due to these task adaptions. Yet strikingly, these improvements proved to be insufficient to enhance the task’s psychometric properties (e.g., convergent validity with other measures of risk preference and related constructs). We conclude that for the development of valid behavioral measurement instruments, our field has to overcome the philosophy of the “repair program” (i.e., fixing existing tasks). Instead, the development of valid task designs may require ecological assessments that identify those real-life behaviors and associated psychological processes that lab tasks are supposed to capture and generalize to.

This is a joint project with Markus Steiner (see picture below), who successfully defended his thesis last week – congratulations, Dr. Steiner!

Steiner, M., & Frey, R. (2021). Representative design in psychological assessment: A case study using the Balloon Analogue Risk Task (BART). Journal of Experimental Psychology: General. doi:10.1037/xge0001036 | PDF

First appeared on https://renatofrey.net/blog