A briefer version of this paper with the same name has been published in

Research on people’s confidence in their general knowledge has to date produced two fairly stable effects, many inconsistent results, and no comprehensive theory. We propose such a comprehensive framework, the theory of probabilistic mental models (PMM theory). The theory (a) explains both the overconfidence effect (mean confidence is higher than percentage of answers correct) and the hard-easy effect (overconfidence increases with item difficulty) reported in the literature and (b) predicts conditions under which both effects appear, disappear, or invert. In addition, (c) it predicts a new phenomenon, the confidence-frequency effect, a systematic difference between a judgment of confidence in a single event (i.e., that any given answer is correct) and a judgment of the frequency of correct answers in the long run. Two experiments are reported that support PMM theory by confirming these predictions, and several apparent anomalies reported in the literature are explained and integrated into the present framework. Do people think they know more than they really do? In the last 15 years, cognitive psychologists have amassed a large and apparently damning body of experimental evidence on overconfidence in knowledge, evidence that is in turn part of an even larger and more damning literature on socalled cognitive biases. The cognitive bias research claims that people are naturally prone to making mistakes in reasoning and memory, including the mistake of overestimating their knowledge.

this technical report can be reproduced, stored in a retrieval system, or transmitted in any form or means, electronic, mechanical, photocopying, recording or otherwise without the prior written permission of the publisher. This technical report may be cited in academic works without permission of the publisher. 3

Expert judgements are routine in biosecurity risk analysis. This report reviews methods for eliciting probabilities, quantities, and conceptual models. It defines ‘expert’ status, reviews the literature on biases and heuristics in expert judgements and outlines methods for detecting and eliciting values, attitudes and motivations. The report describes direct and indirect techniques for eliciting point estimates and uncertainties for quantities, frequencies and probabilities, and for eliciting the structure of conceptual models. It evaluates the use of language-based risk categories and describes methods to detect and adjust for bias and variability in expert judgements. Feedback and training are likely to make useful additions to elicitation protocols. Few of the formal techniques for elicitation, calibration or verification have been evaluated in conditions typical of biosecurity risk analysis, creating an opportunity to test

Abstract not found