This paper discusses verification and validation of simulation models. The different approaches to deciding model validity are presented; how model verification and validation relate to the model development process are discussed; various validation techniques are defined; conceptual model validity, model verification, operational validity, and data validity are described; ways to document results are given; and a recommended procedure is presented. 1 INTRODUCTION Simulation models are increasingly being used in problem solving and in decision making. The developers and users of these models, the decision makers using information derived from the results of the models, and people affected by decisions based on such models are all rightly concerned with whether a model and its results are "correct." This concern is addressed through model verification and validation. Model validation is usually defined to mean "substantiation that a computerized model within its domain of applicabilit...

There are many formal approaches to specifying how the mental state of an agent entails that it perform particular actions. These approaches put the agent at the center of analysis. For some questions and purposes, it is more realistic and convenient for the center of analysis to be the task environment, domain, or society of which agents will be a part. This paper presents such a task environment-oriented modeling framework that can work hand-in-hand with more agent-centered approaches. Our approach features careful attention to the quantitative computational interrelationships between tasks, to what information is available (and when) to update an agent's mental state, and to the general structure of the task environment rather than single-instance examples. A task environment model can be used for both analysis and simulation, it avoids the methodologicalproblems of relying solely on single-instance examples, and provides concrete, meaningful characterizations with which ...

Coordination, as the act of managing interdependencies between activities, is one of the central research issues in Distributed Artificial Intelligence. Many researchers have shown that there is no single best organization or coordination mechanism for all environments. Problems in coordinating the activities of distributed intelligent agents appear in many domains: the control of distributed sensor networks; multi-agent scheduling of people and/or machines; distributed diagnosis of errors in local-area or telephone networks; concurrent engineering; `software agents' for information gathering. The design of coordination mechanisms for group...

There are many formal approaches to specifying how the mental state of an agent entails the particular actions it will perform. These approaches put the agent at the center of analysis. For some questions and purposes, it is more realistic and convenient for the center of analysis to be the task environment, domain, or society of which agents will be a part. This paper presents such a task environment-oriented modeling framework that can work hand-in-hand with more agent-centered approaches. Our approach features careful attention to the quantitative computational interrelationships between tasks, to what information is available (and when) to update an agent's mental state, and to the general structure of the task environment rather than single-instance examples. A task environment model can be used for both analysis and simulation, it avoids the methodological problems of relying solely on single-instance examples, and provides concrete, meaningful characterizations with which to sta...

We review techniques for optimizing stochastic discrete-event systems via simulation. We discuss both the discrete parameter case and the continuous parameter case, but concentrate on the latter which has dominated most of the recent research in the area. For the discrete parameter case, we focus on the techniques for optimization from a finite set: multiple-comparison procedures and ranking-and-selection procedures. For the continuous parameter case, we focus on gradient-based methods, including perturbation analysis, the likelihood ratio method, and frequency domain experimentation. For illustrative purposes, we compare and contrast the implementation of the techniques for some simple discrete-event systems such as the (s, S) inventory system and the GI/G/1 queue. Finally, we speculate on future directions for the field, particularly in the context of the rapid advances being made in parallel computing.

This expository paper discusses the relationships among metamodels, simulation models, and problem entities. A metamodel or response surface is an approximation of the input/output function implied by the underlying simulation model. There are several types of metamodel: linear regression, splines, neural networks, etc. This paper distinguishes between fitting and validating a metamodel. Metamodels may have different goals: (i) understanding, (ii) prediction, (iii) optimization, and (iv) verification and validation. For this metamodeling, a process with thirteen steps is proposed. Classic design of experiments (DOE) is summarized, including standard measures of fit such as the R-square coefficient and cross-validation measures. This DOE is extended to sequential or stagewise DOE. Several validation criteria, measures, and estimators are discussed. Metamodels in general are covered, along with a procedure for developing linear regression (including polynomial) metamodels. Keywords Simul...

1 Despite the advances in computer capacity, the enormous computational cost of complex engineering simulations makes it impractical to rely exclusively on simulation for the purpose of design optimization. To cut down the cost, surrogate models, also known as metamodels, are constructed from and then used in lieu of the actual simulation models. In the paper, we systematically compare four popular metamodeling techniques---Polynomial Regression, Multivariate Adaptive Regression Splines, Radial Basis Functions, and Kriging---based on multiple performance criteria using fourteen test problems representing different classes of problems. Our objective in this study is to investigate the advantages and disadvantages these four metamodeling techniques using multiple modeling criteria and multiple test problems rather than a single measure of merit and a single test problem. 1 Introduction Simulation-based analysis tools are finding increased use during preliminary design to explore desi...

Abstract not found

The optimization of complex real-world problems might benefit from well tuned algorithm's parameters. We propose a methodology that performs this tuning in an effective and efficient algorithmical manner. This approach combines methods from statistical design of experiments, regression analysis, design and analysis of computer experiments methods, and tree-based regression. It can also be applied to analyze the influence of different operators or to compare the performance of different algorithms. An evolution strategy and a simulated annealing algorithm that optimize an elevator supervisory group controller system are used to demonstrate the applicability of our approach to real-world optimization problems.

Threshold selection -- a selection mechanism for noisy evolutionary algorithms -- is put into the broader context of hypothesis testing. Theoretical results are presented and applied to a simple model of stochastic search and to a simplified elevator simulator. Design of experiments methods are used to validate the significance of the results.