B. P. Miller, L. Fredriksen, and B. So. An empirical study of the reliability of UNIX utilities. Communications of the Association for Computing Machinery, 33(12):32--44, 1990.  Home/Search   Document Details and Download   Summary   ACM   TOC   Related Articles   Check

....Since security critical software should work correctly in all environments, this method is useful for nding vulnerabilities. 2.1. 3 Fuzz In 1990, a group at the University of Wisconsin tested standard UNIX utilities by giving them input consisting of large, random streams of characters[28]. They found that 25 33 crashed or hung. The dominant causes were problems with pointers and array dereferencing. Similar experiments with X Window and MS Windows programs produced similar results[29, 16] This is relevant to security because a program that misbehaves when given random input will ....

B.P. Miller, L. Fredricksen, B. So. An Empirical Study of the Reliability of Unix Utilities. In Communications of the ACM, vol.33 no.12, Dec. 1990, pp.32-44.

....the Bugzilla database provided that the bug reports can be reproduced automatically. All one needs is an HTML input, a sequence of user actions, an observable failure and a little time to let the computer simplify the failure inducing input. C. Minimizing Fuzz In a classical experiment [6], 7] Bart Miller and his team examined the robustness of UNIX utilities and services by sending them fuzz input a large number of random characters. The studies showed that, in the worst case, 40 of the basic programs crashed or went into infinite loops when being fed with fuzz input. We ....

B. P. Miller, L. Fredrikson, and B. So, "An empirical study of the reliability of UNIX utilities," Communications of the ACM, vol. 33, no. 12, pp. 32-- 44, Dec. 1990.

....its bounds. To do so, it instrumentsg the compiled program with code that performs different kinds of memory bookkeeping. An alternate approach is to test programs. A tool called Fuzz was used to test standard UNIX utilities by giving them input consisting of large, random streams of characters[17]. 25 33 of the programs crashed or hung. The dominant causes were problems with pointers and array dereferencing, including buffer overflow flaws. Property based testing [9, 11] checks that programs satisfy certain properties, including security related properties. For example, the property that ....

B. Miller, L. Fredricksen, and B. So. Empirical study of the reliability of unix utilities. Communications of the ACM, 33(12):32--44, December 1990.

....interpretation, setting it on a rm, mathematical basis. 1 Introduction Although C programs are ubiquitous, occurring in many security critical, safetycritical and enterprise critical applications, they are particularly prone to inexplicable crashes. Many crashes can be traced to bu er overruns [14]. A bu er overrun occurs when input is read into a bu er and the length of the input exceeds the length of the bu er. Bu er overruns typically arise from unbounded string copies using sprintf( strcpy( and strcat( as well as loops that manipulate strings without an explicit length check in the ....

B. Miller, L. Fredrikson, and B. So. An Empirical Study of the Reliability of UNIX Utilities. Communications of the ACM, 33(12):32-44, 1990.

....requests its environment for an input. The input that the environment provides to the application will most likely affect the application s behavior. A secure application should tolerate an unexpected anomaly in the environment input. One way to perturb the input is to use random input as in Fuzz [8, 23]. However, this approach dramatically increases the testing space, which and calls for a significantly large amount of testing effort. The Fuzz approach does not exploit the semantics of each input. Our vulnerability analysis, however, has shown that inputs most likely to cause security violations ....

B. Miller, L. Fredriksen and B. So. An empirical study of the reliability of unix utilities. Communications of the ACM, 33(12):32--44, December 1990.

....Analysis (AVA) is designed by Ghosh et al. to quantitatively assess information system security and survivability. This approach exercises software in source code form by simulating incoming mali cious and non malicious attacks that fall under various threat classes [22, 23, 28, 29] Fuzz [9, 24] is a black box testing method designed by Miller et al., which feeds randomly generated input stream to system utilities in order to test how reliable they are. Bishop and Dilger studied one class of the time of check to time of use (TOCTTOU) flaws [4] and investigated using static analysis ....

....requests its environment for an input. The input that the environment provides to the application will most likely affect the application s behavior. A secure application should tolerate an unexpected anomaly in the environment input. One way to perturb the input is to use random input as in Fuzz [9, 24]. However, this approach dramatically increases the testing space, which and calls for a significantly large amount of testing effort. The Fuzz approach does not exploit the semantics of each input. Our vulnerability analysis, however, has shown that inputs most likely to cause security violations ....

[Article contains additional citation context not shown here]

B. Miller, L. Fredriksen and B. So. An empiri- cal study of the reliability of unix utilities. Com- munications of the A CM, 33(12):32-44, December 1990.

....causing a bu er over ow. In short, the design of the C programming language encourages programming at the edge of safety. This makes programs ecient but also vulnerable, and leads us to conclude that safety violations are likely to remain common in C programs. A number of studies bear this out [23, 11, 28, 18]. If C programs are unsafe, it is tempting to suggest that all programs be written in a safe language like Java (or ML, or Modula 3, or even 40 year old Lisp) However, this is not a realistic solution for everyone. For one thing, it abandons legacy code. For another, all of the safe languages ....

....mini httpd 1.15c 2.05 0.00 2.09 0.00 1.02 2.08 0.00 1.01 Table 4: Benchmark performance surprise, since at least one (grobner) dates back to the mid 1980s. On the other hand, this is consistent with research that shows that such bugs can linger for years even in widely used software [28]. The mini httpd web server consults a le, htpasswd, to decide whether to grant client access to protected web pages. It tries to be careful not to reveal the password le to clients. Ironically, the code to protect the password le contains a safety violation: #define AUTH FILE .htpasswd ....

Barton P. Miller, Lars Fredriksen, and Bryan So. An empirical study of the reliability of Unix utilities. Communications of the ACM, 33(12):32-44, December 1990.

....of the operating environment. These include problems caused by hardware failures, misbehaving user processes, human operators and system load spikes. The body of research regarding these types of errors is much more limited. The majority of such projects, like the Ballista [8] and Fuzz [10] tools, seek to inject errors into a software module only at the user interface level. This interface level is perhaps the most obvious choice for targeted testing, due to its high visibility and accessibility. However, it does not include all of the interactions that take place between an ....

B. P. Miller, L. Fredriksen, and B. So. An Empirical Study of the Reliability of UNIX Utilities. Communications of the ACM, 33(12):32--44, 1990.

.... [50] Koopman describes benchmarks to test OS robustness by feeding corrupt data to system calls [29] similarly, Miller et al. and Forrester describe the Fuzz tests, robustness tests based on feeding random data or messages ( fuzz ) into key input points in operating systems and applications [17] [32] [33] The major difference between these benchmarks and the ones we propose is in their goals and the knowledge they assume. Tsai s and Siewiorek s benchmarks are primarily designed to test particular known fault tolerance mechanisms deployed in fault tolerant hardware and software systems; to ....

B. Miller, L. Fredriksen, and B. So. An Empirical Study of the Reliability of UNIX Utilities. Communications of the ACM, 33(12):32--43, December 1990.

....interesting results, none was intended to quantify robustness on the scale of an entire OS API. The Fuzz project at the University of Wisconsin has used random noise (or fuzz ) injection to discover robustness problems in operating systems. That work documented the source of several problems [29], and then discovered that the problems were still present in operating systems several years later [30] The Fuzz approach tested specific OS elements and interfaces (compared to the completely random approach of Crashme) although it still relied on random data injection. The Ballista testing ....

Miller, B.P., Fredriksen, L. & So, B., "An empirical study of the reliability of Unix utilities, Connunications of the ACM, 33(12): 32-43, Dec. 1990.

....injection provides a means for measuring a system s robustness, its ability to function correctly in the presence of invalid inputs and stressful environmental conditions. Robustness testing involves injecting corrupted data at the linking interface of the system, and observing its behaviour. In [Miller et al. 1990] , the robustness of different implementations of standard UNIX utilities was measured, by submitting them to randomly generated input. Despite using an extremely simple failure mode classification (crash or not crash) this fuzz testing showed that most implementations had quite high failure ....

B. P. Miller, L. Fredriksen, and B. So. An empirical study of the reliability of UNIX utilities. Communications of the ACM, 33(12):32--44, December 1990.

....causing a bu er over ow. In short, the design of the C programming language encourages programming at the edge of safety. This makes programs ecient but also vulnerable, and leads us to conclude that safety violations are likely to remain common in C programs. A number of studies bear this out [19, 9, 22, 16]. If C programs are unsafe, it s tempting to suggest that all programs be written in a safe language like Java (or ML, or Modula 3, or even 40 year old Lisp) However, this isn t a realistic solution for everyone. For one thing, it abandons legacy code. For another, all of the safe languages look ....

Barton P. Miller, Lars Fredriksen, and Bryan So. An empirical study of the reliability of Unix utilities. Communications of the ACM, 33(12):32-44, December 1990.

....or survives [6] These focus mostly on robustness in the face of arti cial errors, whereas we are interested more in the features of actual errors. Another approach is explicit testing, such as the fuzz studies that compare how a set of systems utilities behaved in the face of random inputs [17, 18]. In terms of examining bugs found with automatic techniques, the closest work compared to us is a study by Koopman et al. that used randomized testing to measure the e ectiveness of error handling code on 13 di erent POSIX implementations [13] They measured both the types of errors that ....

B.P. Miller, L. Fredriksen, and B. So. An Empirical Study of the Reliability of UNIX Utilities. Communications of the Association for Computing Machinery, 33(12):32-44, 1990.

....or survives [6] These focus mostly on robustness in the face of artificial errors, whereas we are interested more in the features of actual errors. Another approach is explicit testing, such as the fuzz studies that compare how a set of systems utilities behaved in the face of random inputs [17, 18]. In terms of examining bugs found with automatic techniques, the closest work compared to us is a study by Koopman et al. that used randomized testing to measure the e#ectiveness of error handling code on 13 di#erent POSIX implementations [13] They measured both the types of errors that ....

B.P. Miller, L. Fredriksen, and B. So. An Empirical Study of the Reliability of UNIX Utilities. Communications of the Association for Computing Machinery, 33(12):32--44, 1990.

....or survives [6] These focus mostly on robustness in the face of artificial errors, whereas we are interested more in the features of actual errors. Another approach is explicit testing, such as the fuzz studies that compare how a set of systems utilities behaved in the face of random inputs [17, 18]. In terms of examining bugs found with automatic techniques, the closest work compared to us is a study by Koopman et al. that used randomized testing to measure the effectiveness of error handling code on 13 different POSIX implementations [13] They measured both the types of errors that ....

B.P. Miller, L. Fredriksen, and B. So. An Empirical Study of the Reliability of UNIX Utilities. Communications of the Association for Computing Machinery, 33(12):32--44, 1990.

....13 rate of utilities on the proprietary versions of UNIX ranges from 15 to 43 , while the failure rate for Linux is 9 and for the GNU utilities is only 6 . Interestly enough, the proprietary versions of UNIX do not correct all the bugs 14 reported in a eight years older analogous study [MFS90] unfortunately that report did not contain data about Linux nor GNU utilities. Open source programs in the long run can be much stabler than their proprietary counterparts, but, it is important to note, that testing is not just debugging, and formalised procedures of testing are una#ordable by ....

....even their proprietary as shareware to take advantage of the same benefits. 12 Named after Linus Torvalds, the creator of Linux kernel. 13 In the paper failure means a crashing with core dump or hanging (looping indefinitely) 14 Network utilities are exceptional regard to this: in [MFS90] study only two programs crashed, no one crashed in [MKL 98] study. The power of interconnection in improving quality of software seems evident. 10 3 Open Source Markets As noted in Section 2, open source licences do not prohibit to sell the right to use, copy, modify, and redistribute, ....

Barton Miller, Lars Fredriksen, and Bryan So. An empirical study of the reliability of unix utilities. ftp://grilled.cs.wisc.edu/technical papers/fuzz.pdf, 1990.

....algorithm for solving the integer constraints. Finally, security knowledge is used to formulate heuristics that capture the class of security relevant bugs that tend to occur in real programs. Others have applied runtime code testing techniques to the problem, using, e.g. black box testing [41, 42] or software fault injection [21] to find buffer overruns in real world applications. However, runtime testing seems likely to miss many vulnerabilities. Consider the following example: if (strlen(src) sizeof dst) break; strcpy(dst, src) Note that off by one errors in buffer management, ....

B.P. Miller, L. Fredricksen, B. So, "An empirical study of the reliability of Unix utilities," CACM, vol.33 no.12, Dec. 1990, pp.32--44.

....are automatically simplified for any failing test case. 6. CASE STUDY: MINIMIZING FUZZ If you understand the context in which a problem occurs, you re more likely to solve the problem completely rather than only one aspect of it. Steve McConnell, Code Complete In a classical experiment [6, 7], Bart Miller and his team examined the robustness of UNIX utilities and services by sending them fuzz input a large number of random characters. The studies showed that, in the worst case, 40 of the basic programs crashed or went into infinite loops when being fed with fuzz input. We wanted to ....

B. P. Miller, L. Fredrikson, and B. So. An empirical study of the reliability of UNIX utilities. Communications of the ACM, 33(12):32--44, Dec. 1990.

....values. Nonetheless, their results found many Abort type failures in Windows NT, and a few Catastrophic failures that were caused by very complex execution sequences that could not be isolated for bug reporting purposes. Other recent related work is the Fuzz project at the University of Wisconsin [12], 13] which has concentrated on Unix systems. There does not appear to be any previously published work that performs testing oriented dependability comparisons of multiple Windows versions, nor comparisons of Windows to Unix robustness. 3. Implementation The existing Ballista testing system ....

Miller, B.P., Fredriksen, L. & So, B., "An empirical study of the reliability of Unix utilities," Communications of the ACM, 33(12): 32-43, December 1990.

....The unique combination of circumstances required for an error to occur and for its symptoms to become visible may be virtually impossible to create in the development or test environment. As a result, programmers spend much time looking for these errors, but end users may experience them first. [Miller90] empirically shows the continuing prevalence of access errors in many widely used UNIX programs. Even when a memory access error triggers an observable symptom, the error can take days to track down and eliminate. This is due to the frequently delayed and coincidental connection between the cause, ....

B. P. Miller, L Fredrickson, and B. So, "An Empirical Study of the Reliability of UNIX Utilities", CACM vol 33, #12, December 1990, pp 32-44.

....a program crash but may instead produce a subtly wrong answer. Even if an erroneous program crashes, it may be difficult to repeat the error inside a debugger. Further, debugging long running programs can be very time consuming. Undiscovered errors in heavily used programs may not be rare; a study [MLS90] has shown that as many as a quarter of the most commonly used Unix utilities crash or hang when presented with unexpected inputs. Thus there is a strong case for running programs with checks routinely enabled. Naturally, these checks should be as inexpensive as possible. The goal of this work is ....

....system V release 4] Sun s multi threading library was used to create main and shadow threads. The test programs included integer benchmarks from the SPEC92 test suite, one program, yacr, from the SafeC [ABS94] test suite and many commonly used utilities from SunOS 4.1.3. A utility called fuzz [MLS90] was used to generate random input for the SunOS utilities. SPEC benchmarks were tested with their reference inputs. 5.1 Errors uncovered Run time errors which do not crash programs can go unnoticed for a long time. Shadow guarding reports such errors as they occur. Programmers can use the ....

[Article contains additional citation context not shown here]

Barton P. Miller, Fredriksen Lars, and Brian So. An empirical study of the reliability of Unix utilities. Communications of the ACM, 33(12):32--44, December 1990.

....algorithm for solving the integer constraints. Finally, security knowledge is used to formulate heuristics that capture the class of security relevant bugs that tend to occur in real programs. Others have applied runtime code testing techniques to the problem, using, e.g. black box testing [41, 42] or software fault injection [21] to find buffer overruns in real world applications. However, runtime testing seems likely to miss many vulnerabilities. Consider the following example: if (strlen(src) sizeof dst) break; strcpy(dst, src) Note that off by one errors in buffer management, ....

B.P. Miller, L. Fredricksen, B. So, "An empirical study of the reliability of Unix utilities," CACM, vol.33 no.12, Dec. 1990, pp.32--44.

....The certification process we will propose greatly reduces this liability quagmire while also eliminating the need to dispatch human auditors. Our process harnesses the testing resources of end users. It is similar to the way in approach that made Linux the most popular and reliable of all Unixes [2, 4]. 3 2. OUR SOFTWARE WARRANTY MODEL Our interest is in certifying software applications and components with dualuse potential. We are interested in certifying applications such as Microsoft Word. And in fact, we are more interested in smaller components that could be embedded in a variety of ....

....consider Linux, a Unix operating system project that began in 1991 and today has nearly 8 million users. Linux is the product of hundreds of users, all of whom donated their time to write the system. Their efforts paid off: Linux is considered to be the most reliable of all Unix operating systems [4]. In fact, the success of Linux is often used as the argument for why products with open source are the only products that are trustworthy. So what can we surmise from these anecdotes From Microsoft and Linux, we see that users are willing to participate in efforts that result in improved ....

Miller, B.P., Fredrikson, L., and So, B. An empirical study of the reliability of unix utilities. Communications of the ACM, 33(12):32--44, December 1990.

....as well as published evidence lead us to believe that memory access errors are an important class of errors to reliably detect. For example, in This work was supported by grants from the National Science Foundation (grant CCR 9303030) and Office of Naval Research (grant N00014 93 1 0465) [MFS90], Miller et al. injected random inputs (a.k.a fuzz ) into a number of Unix utilities. On systems from six different vendors, nearly all of the seemingly mature programs could be coaxed into dumping core. The most prevalent errors detected were memory access errors. In [SC91] Sullivan and ....

Barton P. Miller, Lars Fredriksen, and Bryan So. An empirical study of the reliability of Unix utilities. Communications of the ACM, 33(12):32--44, December 1990.

....here to establish the prior art in robustness testing. 2. 1 Fuzz One of the first noted research studies on the robustness of software was performed by a group out of the University of Wisconsin [8] In 1990, the group published a study of the reliability of standard Unix utility programs [7]. Using a random black box testing tool called Fuzz, the group found that 25 33 of standard Unix utilities crashed or hung when tested using Fuzz. Five years later, the group repeated and extended the study of Unix utilities using the same basic techniques. The 1995 study found that in spite of ....

B.P. Miller, L. Fredrikson, and B. So. An empirical study of the reliability of unix utilities. Communications of the ACM, 33(12):32--44, December 1990.

....Thimbleby and analysis are easily combined: the analytic approach can easily be simulated, the simulation data can easily be combined with further analysis, and so on. Moreover a random approach (e.g. simulating a random user) has been shown to be very good at identifying programming problems [6], and therefore would be of use when the user interface was not automatically derived from the specification. 3.4 Usability as a function of knowledge To get more realistic usability metrics from a probability matrix, we need to introduce some knowledge. Our approach is to take a mixture of a ....

B. P. Miller, L. Fredriksen & B. So, 1990, "An Empirical Study of the Reliability of Unix Utilities," Communications of the ACM, 33(12), pp32--44.

....quality of the component against their requirements for what the component is expected to do, and not necessarily against the functionality claimed by the vendor. Black box testing of executable software components has already received attention from the work of Miller, et al. and their FUZZ model [9]. FUZZ works by generating random black box inputs, feeding them to UNIX system functions, and watching for core dumps. FUZZ provides a low resolution approximation of the robustness of particular UNIX utilities (e.g. ls) Watching for a program to dump core (that is, watching for a program to ....

B. P. MILLER, L. FREDRIKSON, AND B. SO. An empirical study of the reliability of unix utilities. Communications of the ACM, 33(12)32--44, December 1990.

....These programs run with high privileges that allow them to bypass the kernel protection mechanism, in effect violating the system policy. In principle, they are designed and trusted not to imperil the security of the system, but due to errors, they can be used to bypass security safeguards [2, 21]. For example, during its testing, a backdoor was inadvertently inserted into the BSD sendmail program, enabling users to obtain root privileges. As another example, the finger daemon program neglects to limit the size of a input string, enabling an attacker to overflow its buffer to obtain root ....

B. So B. P. Miller, L. Fredriksen. An empirical study of the reliability of unix utilities. Communications of the ACM, 33(12), December 1990.

.... here, and is described at length elsewhere [15, 17, 21] Many C programs have buffer overflow vulnerabilities, both because the C language lacks array bounds checking, and because the culture of C programmers encourages a performance oriented style that avoids error checking where possible [14, 13]. For instance, many of the standard C library functions such as gets and strcpy do not do bounds checking by default. The common form of buffer overflow exploitation is to attack buffers allocated on the stack. Stack smashing attacks strive to achieve two mutually dependent goals, illustrated in ....

B.P. Miller, L. Fredrikson, and B. So. An Empirical Study of the Reliability of UNIX Utilities. Communications of the ACM, 33(12):33--44, December 1990.

....coarse, too. The program is considered to fail if it dumps a core file or if it hangs. After submitting a program to random input, Fuzz checks for the presence of a core file or a hung process. Though the methodology employed by Fuzz is simple, the results from their study were quite revealing [4, 5]. The 1995 study [5] found that the failure rate of the Unix utilities they tested were between 18 and 23 down only a few percentage points from 25 33 failure rate documented in the 1990 study [4] Ballista, a Carnegie Mellon University research project, studies the robustness of ....

.... methodology employed by Fuzz is simple, the results from their study were quite revealing [4, 5] The 1995 study [5] found that the failure rate of the Unix utilities they tested were between 18 and 23 down only a few percentage points from 25 33 failure rate documented in the 1990 study [4]. Ballista, a Carnegie Mellon University research project, studies the robustness of different Unix operating systems to handling exceptional conditions. Unlike the Fuzz research, Ballista focuses on assessing the robustness of operating system calls made frequently from desktop software. Rather ....

[Article contains additional citation context not shown here]

B.P. Miller, L. Fredrikson, and B. So. An empirical study of the reliability of unix utilities. Communications of the ACM, 33(12):32--44, December 1990.

.... Pioneering work in analyzing buffer overruns has been performed by researchers at the COAST Laboratory at Purdue University [13] In addition, research out of the University of Wisconsin has analyzed Unix utilities for reliability and robustness, with corresponding implications on security [10, 11]. A static software analysis technique employed by UC Davis researchers can analyze software for vulnerability to a class of race condition flaws called time of check to time ofuse (TOCTTOU) flaws [6] Another UC Davis group is using property based assertions and software testing techniques to ....

B.P. Miller, L. Fredrikson, and B. So. An empirical study of the reliability of unix utilities. Communications of the ACM, 33(12):32--44, December 1990.

....[12] PREPRINT A University of Wisconsin group using a tool called Fuzz subjected Unix utilities with random streams of input data. Miller et al. found that . the failure rate of utilities on the commercial versions of UNIX . tested (from Sun, IBM, SGI, DEC, and NeXT) ranged from 15 43 [10, 11]. Most of these utilities failed because of errors in coding. The class of errors that caused the most failures were related to misuse of pointers and array subscripts. For example, incrementing the pointer past the end of an array was a common coding error. Using dangerous input functions, such ....

B.P. Miller, L. Fredrikson, and B. So. An empirical study of the reliability of unix utilities. Communications of the ACM, 33(12):32--44, December 1990.

....a program crash but may instead produce a subtly wrong answer. Even if an erroneous program crashes, it may be difficult to repeat the error inside a debugger. Further, debugging long running programs can be very time consuming. Undiscovered errors in heavily used programs may not be rare; a study [4] has shown that 3 To appear in Software Practice and Experience as many as a quarter of the most commonly used Unix utilities crash or hang when presented with unexpected inputs. Thus there is a strong case for running programs with checks routinely enabled, especially as computers assume ....

....government, and transportation. Naturally, these checks should be as inexpensive as possible. Pointer and array access errors top the list of run time errors in a study [5] of over 100,000 Pascal program executions. The most common cause of program crashes in the previously mentioned study [4] of reliability of UNIX utilities written in C was also found to be pointer and array access errors. Apart from crashing the program or producing a wrong answer, unchecked pointer and array accesses provide a security hole as shown by the Internet worm [6] Our project has two goals. First, to ....

[Article contains additional citation context not shown here]

B. P. Miller, F. Lars, and B. So. An empirical study of the reliability of Unix utilities. Communications of the ACM, 33(12):32--44, December 1990.

....(i.e. the hardware together with its supporting software) To date, much of the effort in building robust systems has been devoted to building robust hardware. Efforts to evaluate the robustness of software systems have become common only recently, and are exemplified by such studies as [11] and [15] These studies both concentrate, like all robustness benchmarks, on studying the behavior produced when a system is subjected to unusual (rather than commoncase) stimuli. Both studies perform their evaluations via a collection of isolated tests, and draw conclusions from the collected ....

....stimuli. Both studies perform their evaluations via a collection of isolated tests, and draw conclusions from the collected results. Unfortunately, it is often difficult to evaluate the relative significance of each of the individual results collected through such test suites. For example, [11] examines the behavior of Unix utilities when they are supplied with randomly generated input data. The crash of any given utility must be taken as seriously as the crash of any other utility, even though this weighting may not reflect reality, because the benchmark lacks knowledge of the ....

B. P. Miller, L. Fredriksen, B. So, "An Empirical Study of the Reliability of UNIX Utilities," Communications of the ACM, Volume 33, Number 12, December 1990, pp. 32-43.

....(i.e. the hardware together with its supporting software) To date, much of the effort in building robust systems has been devoted to building robust hardware. Efforts to evaluate the robustness of software systems have become common only recently, and are exemplified by such studies as [Miller90] and [Suh93] These studies both concentrate, like all robustness benchmarks, on studying the behavior produced when a system is subjected to unusual (rather than common case) stimuli. Both studies perform their evaluations via a collection of isolated tests, and draw conclusions from the ....

....stimuli. Both studies perform their evaluations via a collection of isolated tests, and draw conclusions from the collected results. Unfortunately, it is often difficult to evaluate the relative significance of each of the individual results collected through such test suites. For example, [Miller90] examines the behavior of Unix utilities when they are supplied with randomly generated input data. The crash of any single utility must be taken as seriously as the crash of any other utility, even though this weighting may not reflect reality, because the benchmark lacks knowledge of the ....

[Article contains additional citation context not shown here]

B. P. Miller, L. Fredriksen, B. So, "An Empirical Study of the Reliability of UNIX Utilities," Communications of the ACM, Volume 33, Number 12, December 1990, pp. 32-43.

....should be reasonable to expect that basic operating system utilities do not crash the system. However, the study shows that 25 to 33 of UNIX utilities crash or hang when exposed to unusual (unexpected) inputs (the number varies for different implementations, such as VAX, Sun, HP, i386, and AIX) [33]. Even more surprising is the result of a comparison between the consistency of results of nine large, independently developed, commercial software packages for seismic data processing. These packages implement the same or similar mathematical algorithms, from the same or similar published ....

B. P. Miller, L. Fredricksen, B. So, "An Empirical Study of the Reliability of UNIX Utilities," Technical Report CSTR -89-830, Department of Computer Science, University of Wisconsin-Madison, 1989.

....requests its environment for an input. The input that the environment provides to the application will most likely affect the application s behavior. A secure application should tolerate an unexpected anomaly in the environment input. One way to perturb the input is to use random input as in Fuzz [8, 23]. However, this approach dramatically increases the testing space, which and calls for a significantly large amount of testing effort. The Fuzz approach does not exploit the semantics of each input. Our vulnerability analysis, however, has shown that inputs most likely to cause security violations ....

B. Miller, L. Fredriksen and B. So. An empirical study of the reliability of unix utilities. Communications of the ACM, 33(12):32--44, December 1990.

....of a spatial access error. A typical temporal access error is assigning to a heap allocation after it has been freed. Our own experiences as programmers as well as published evidence lead us to believe that memory access errors are an important class of errors to reliably detect. For example, in [MFS90], Miller et al. injected random inputs (a.k.a fuzz ) into a number of Unix utilities. On systems from six different vendors, nearly all of the seemingly mature programs could be coaxed into dumping core. The most prevalent errors detected were memory access errors. In [SC91] Sullivan and ....

Barton P. Miller, Lars Fredriksen, and Bryan So. An empirical study of the reliability of Unix utilities. Communications of the ACM, 33(12):32--44, December 1990.

....ravim cs.wisc.edu ajitk cs.wisc.edu steidl cae.wisc.edu Computer Sciences Department University of Wisconsin 1210 W. Dayton Street Madison, WI 53706 1685 Page 2 April 11, 1995 1 INTRODUCTION In 1990, we published the results of a study of the reliability of standard UNIX utility programs [2]. This study showed that by using simple (almost simplistic) random testing techniques, we could crash or hang 25 33 of these utility programs. Five years later, we have repeated and significantly extended this study using the same basic techniques: subjecting programs to random input streams. A ....

....Solaris, IRIX, Ultrix, and NEXTSTEP) were the most recent commercial software distributions that we had available to us at the time of the tests. Three of these commercial systems (SunOS, HP UX, and AIX) were also tested in the 1990 study. Results from this earlier Page 5 April 11, 1995 study [2] are included for comparison. Two of the systems tested are free software distributions. The GNU tools come from the Free Software Foundation and are written by a variety of authors world wide. Linux is a freely distributed version of the UNIX operating system originally written by Linus Torvalds; ....

Miller, B.P., Fredrikson, L., and So, B., An Empirical Study of the Reliability of UNIX Utilities. Communications of the ACM 33, 12 (December 1990), 32-44.

....This lack shows in the systems we deploy. How many of you have ever been on a system where the screen display is replaced by a huge list of numbers, letters, and tables of dots How many knew your gooses were cooked at that time You re not alone. A study of utility programs on UNIX systems [3] showed that, given random input, about one fourth to one third of the programs dumped core. In 1 case, the kernel panicked, crashing the system That s inexcusable, sloppy programming, and has serious consequences. In cases involving security, program crashes or incorrect behavior are not ....

Miller, B., Fredriksen, L. and So, B., "An Empirical Study of the Reliability of UNIX Utilities," Communications of the ACM 33(12) pp. 33--43 (Dec. 1990).

....steidl cae.wisc.edu Computer Sciences Department University of Wisconsin 1210 W. Dayton Street Madison, WI 53706 1685 Page 2 PRELIMINARY April 20, 1995 PRELIMINARY 1 INTRODUCTION In 1990, we published the results of a study of the reliability of standard UNIX utility programs [2]. This study showed that by using simple (almost simplistic) random testing techniques, we could crash or hang 25 33 of these utility programs. Five years later, we have repeated and significantly extended this study using the same basic techniques: subjecting programs to random input streams. A ....

....(SunOS, HP UX, AIX, Solaris, IRIX, Ultrix, and NEXTSTEP) were the most recent commercial software distributions that we had available to us at the time of the tests. Three of these commercial systems (SunOS, HP UX, and AIX) were also tested in the 1990 study. Results from this earlier study [2] are included for comparison. Two of the systems tested are free software distributions. The GNU tools come from the Free Software Foundation and are written by a variety of authors world wide. Linux is a freely distributed version of the UNIX operating system Page 5 PRELIMINARY April 20, 1995 ....

Miller, B.P., Fredrikson, L., and So, B., An Empirical Study of the Reliability of UNIX Utilities. Communications of the ACM 33, 12 (December 1990), 32-44.

No context found.

B. P. Miller, L. Fredriksen, and B. So. An empirical study of the reliability of UNIX utilities. Communications of the Association for Computing Machinery, 33(12):32--44, 1990.

No context found.

Barton P. Miller, Lars Fredriksen, and Bryan So. An empirical study of the reliability of Unix utilities. Communications of the ACM, 33(12):32--44, December 1990.

No context found.

B. P. Miller, L. Fredriksen & B. So, 1990, "An Empirical Study of the Reliability of Unix Utilities," Communications of the ACM, 33(12), pp32--44. 27

No context found.

B.P. Miller, L. Fredricksen, and B. So. An empirical study of the reliability of Unix utilities. Communications of the ACM, 33(12):32--44, December 1990.

No context found.

B. P. Miller, L. Fredrikson, and B. So. An empirical study of the reliability of UNIX utilities. Communications of the ACM, 33(12):32--44, Dec. 1990.

No context found.

Barton P. Miller, Lars Fredriksen, and Bryan So, "An empirical study of the reliability of UNIX utilities," Communications of the ACM, vol. 33, no. 12, pp. 32--44, Dec. 1990.

No context found.

Barton P. Miller, Fredriksen Lars, and So Brian. An empirical study of the reliability of Unix utilities. Communications of the ACM, 33(12):32--44, December 1990.

No context found.

B.P. Miller, L. Fredriksen, and B. So. An empirical study of the reliability of UNIX utilities. Communications of the ACM, 33(12):32--44, Dec. 1990.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC