After an historical review of evaluation methods, we describe an emerging research method called Multi-dimensional In-depth Long-term Case studies (MILCs) which seems well adapted to study the creative activities that users of information visualization systems engage in. We propose that the efficacy of tools can be assessed by documenting 1) usage (observations, interviews, surveys, logging etc.) and 2) expert users ’ success in achieving their professional goals. We summarize lessons from related ethnography methods used in HCI and provide guidelines for conducting MILCs for information visualization. We suggest ways to refine the methods for MILCs in modest sized projects and then envision ambitious projects with 3-10 researchers working over 1-3 years to understand individual and organizational use of information visualization by domain experts working at the frontiers of knowledge in their fields. 1.

Abstract—High-throughput experiments, such as gene expression microarrays in the life sciences, result in very large data sets. In response, a wide variety of visualization tools have been created to facilitate data analysis. A primary purpose of these tools is to provide biologically relevant insight into the data. Typically, visualizations are evaluated in controlled studies that measure user performance on predetermined tasks or using heuristics and expert reviews. To evaluate and rank bioinformatics visualizations based on real-world data analysis scenarios, we developed a more relevant evaluation method that focuses on data insight. This paper presents several characteristics of insight that enabled us to recognize and quantify it in open-ended user tests. Using these characteristics, we evaluated five microarray visualization tools on the amount and types of insight they provide and the time it takes to acquire it. The results of the study guide biologists in selecting a visualization tool based on the type of their microarray data, visualization designers on the key role user interaction techniques, and evaluators on a new approach for evaluating the effectiveness of visualizations for providing insight. Though we used the method to analyze bioinformatics visualizations, it can be applied to other domains. Index Terms—Evaluation/methodology, graphical user interfaces (GUI), information visualization, visualization systems and software, visualization techniques and methodologies. æ 1

Although both statistical methods and visualizations have been used by network analysts, exploratory data analysis remains a challenge. We propose that a tight integration of these technologies in an interactive exploratory tool could dramatically speed insight development. To test the power of this integrated approach, we created a novel social network analysis tool, SocialAction, and conducted four long-term case studies with domain experts, each working on unique data sets with unique problems. The structured replicated case studies show that the integrated approach in SocialAction led to significant discoveries by a political analyst, a bibliometrician, a healthcare consultant, and a counter-terrorism researcher. Our contributions demonstrate that the tight integration of statistics and visualizations improves exploratory data analysis, and that our evaluation methodology for long-term case studies captures the research strategies of data analysts.

Abstract — The analysis and exploration of multidimensional and multivariate data is still one of the most challenging areas in the field of visualization. In this paper, we describe an approach to visual analysis of an especially challenging set of problems that exhibit a complex internal data structure. We describe the interactive visual exploration and analysis of data that includes several (usually large) families of function graphs fi(x, t). We describe analysis procedures and practical aspects of the interactive visual analysis specific to this type of data (with emphasis on the function graph characteristic of the data). We adopted the well-proven approach of multiple, linked views with advanced interactive brushing to assess the data. Standard views such as histograms, scatterplots, and parallel coordinates are used to jointly visualize data. We support iterative visual analysis by providing means to create complex, composite brushes that span multiple views and that are constructed using different combination schemes. We demonstrate that engineering applications represent a challenging but very applicable area for visual analytics. As a case study, we describe the optimization of a fuel injection systems in Diesel engines of passenger cars. Index Terms — visual exploration, composite brushing, linked views, time series data, fuel injection system I.

During exploratory data analysis, visualizations are often useful for making sense of complex data sets. However, as data sets increase in size and complexity, static information visualizations decrease in comprehensibility. Interactive techniques can yield valuable discoveries, but current data analysis tools typically support only opportunistic exploration that may be inefficient and incomplete. We present a refined architecture that uses systematic yet flexible (SYF) design goals to guide domain expert users through complex exploration of data over days, weeks and months. The SYF system aims to support exploratory data analysis with some of the simplicity of an e-commerce check-out while providing added flexibility to pursue insights. The SYF system provides an overview of the analysis process, suggests unexplored states, allows users to annotate useful states, supports collaboration, and enables reuse of successful strategies. The affordances of the SYF system are demonstrated by integrating it into a social network analysis tool employed by social scientists and intelligence analysts. The SYF system is a tool-independent component and can be incorporated into other data analysis tools.

Designing usable geovisualization tools is an emerging problem in GIScience software development. We are often satisfied that a new method provides an innovative window on our data, but functionality alone is insufficient assurance that a tool is applicable to a problem in situ.

Very few user analysis studies have been done in the bioinformatics domain. This paper presents the results of an exploratory user study as part of a domain and requirements analysis aimed at enhancing multidisciplinary collaboration. Contextual field observations, questionnaires and interviews with bioinformatics researchers of different levels of expertise and various backgrounds were performed in order to gain insight into their needs and working practices. The analysed results are presented as a user profile description and user requirements for designing user interfaces that support the collaboration of multidisciplinary research teams in scientific collaborative environments. Although the number of participants limits the generalisability of the findings, the combination of recurrent observations with other user analysis techniques in real-life settings makes the contribution of this user study novel. User study, bioinformatics, human-computer interaction, co-located collaboration, visualisations 1

Database searches are usually performed with query languages and form fill in templates, with results displayed in tabular lists. However, excitement is building around dynamic queries sliders and other graphical selectors for query specification, with results displayed by information visualization techniques. These filtering techniques have proven to be effective for many tasks in which visual presentations enable discovery of relationships, clusters, outliers, gaps, and other patterns. Scaling visual presentations from millions to billions of records will require collaborative research efforts in information visualization and database management to enable rapid aggregation, meaningful coordinated windows, and effective summary graphics. This paper describes current and proposed solutions (atomic, aggregated, and density plots) that facilitate sense-making for interactive visual exploration of billion record data sets.

Abstract — In Human-Computer studies and visual analytics, the majority of the evaluation methods applied, collect data at a single point in time, in form of cross-sectional data. In several studies [e.g. 8] numerous visualization tools were evaluated in controlled experiments. Although the experiments discovered valuable findings, certain drawbacks of the research method were expressed. The time constraints of one-time experiments reduce the amount of training which can be given to the participants. Furthermore, when the studies tried to measure the insight derived from the visualization tools the time constraints didn’t allow observing how these insights develop over time or their interdependency. Further problems of cross-sectional studies are well known, like the selection of appropriate tasks, the mostly extrinsic motivation of the participants, the influence of a laboratory environment compared to a realistic work setting and whether a visualization tool does meet the work requirements in the long run. In this position paper we argue for applying longitudinal research methods in human-computer studies as an extension to cross-sectional studies and present a first approach towards a methodological research framework. We suggest a set of research questions and performance measures that would be benefical for extending cross-sectional studies with longitudinal ones. We also describe in two case studies, in which only cross-sectional research methods were used, how they can improved by longitudinal methods. Index Terms—longitudinal, long-term, methodology, research framework, experiment, visual analytics, evaluation 1

Social Network Analysis (SNA) has evolved as a popular, standard method for modeling meaningful, often hidden structural relationships in communities. Existing SNA tools often involve extensive pre-processing or intensive programming skills that can challenge practitioners and students alike. NodeXL, an open-source template for Microsoft Excel, integrates a library of common network metrics and graph layout algorithms within the familiar spreadsheet format, offering a potentially low-barrierto-entry framework for teaching and learning SNA. We present the preliminary findings of 2 user studies of 21 graduate students who engaged in SNA using NodeXL. The majority of students, while information professionals, had little technical background or experience with SNA techniques. Six of the participants had more technical backgrounds and were chosen specifically for their experience with graph drawing and information visualization. Our primary objectives were (1) to evaluate NodeXL as an SNA tool for a broad base of users and (2) to explore methods for teaching SNA. Our complementary dual case-study format demonstrates the usability of NodeXL for a diverse set of users, and significantly, the power of a tightly integrated metrics/visualization tool to spark insight and facilitate sensemaking for students of SNA. I.