A Department of Commerce study has concluded that software bugs, or errors, cost the U.S. economy an estimated $59.5 billion annually, but not everyone agrees that the blame lies entirely with software vendors.

According to the study–commissioned by the Gaithersburg, Md.-based National Institute of Standards and Technology (NIST), a part of DOC–at the national level, more than half the costs of such errors are borne by software users, with the remainder falling on software developers and vendors.

Released in June, the study was funded by NIST and conducted by the Research Triangle Institute in North Carolina.

The studyThe Economic Impacts of Inadequate Infrastructure for Software Testingalso found that although not all errors are likely to be removed, more than a third of these costs ($22.5 billion) could be eliminated by an improved testing infrastructure that enables earlier and more effective identification and removal of software defects by vendors.

“These are the savings associated with finding an increased percentage (but not 100%) of errors closer to the development stages in which they are introduced,” says NIST. “Currently, over half of all errors are not found until downstream in the development process or during post-sale software use.”

“The impact of software errors is enormous, because virtually every business in the United States now depends on software for the development, production, distribution and after-sales support of products and services,” states NIST Director Arden Bement.

According to NIST, software is “error-ridden” in part because it is growing in complexity. “The size of software products is no longer measured in thousands of lines of code, but in millions,” the agency says. “Software developers already spend approximately 80% of development costs on identifying and correcting defects, and yet few products of any type other than software are shipped with such high levels of errors.”

Indeed, if software companies were automobile makers, “they might be out of business from product liability suits,” states Gregory Tassey, senior economist at NIST. Software “has way more errors in what is delivered to users than the vast majority of products you run across.”

For the study, two industries–transportation equipment and financial services–were examined in depth, says NIST. In financial services, data was collected from four industry software developers, as well as 98 users (primarily banks and credit unions).

According to NIST, about two-thirds of the users reported experiencing “major software errors” in the previous year.

Major errors, says Tassey, include those that result in systems shutdown, loss of data, or need for significant systems reconfiguration.

Respondents who did have major errors reported an average of 40 major and 49 minor software bugs per year in their clearinghouse software systems, says NIST. Typical problems encountered due to bugs were: increased person-hours needed to correct posting errors, temporary shutdown leading to lost transactions, and delay of transaction processing.

NIST estimates the total cost of inadequate software testing in financial services to be $3.3 billion. Potential cost reduction from “feasible” infrastructure improvements is $1.5 billion.

Tassey asserts that software-error-related loss scenarios are “probably true of software across all industries.” He says the highly publicized software failures–such as one that interrupted the New York Mercantile Exchange and phone service to several East Coast cities in February 1998–”are the tip of the iceberg.”

The entire fault for software errors does not lie with vendors, however, according to Eli Dabich, president of Synergy 2000, a Pasadena, Calif.-based systems integrator serving the insurance industry. He maintains that companies who buy software “bear equal responsibility.”

In a typical scenario, says Dabich, a company buys software from a vendor who promises to put out one new release, or upgrade, a year. “But it probably takes two years to install the system, so most companies will elect to wait for release 3,” he explains. The company may also decide that release 4 isnt that much different than 3, so they never install it.

The result is that companies may miss fixes and other necessary add-ons that could eliminate or mitigate the effects of errors, says Dabich.

In addition to not staying current with updated releases, buyers will also “tamper” with the basic code of the program in order to adapt the software to its existing workflow processes, Dabich notes.

Revisiting the automobile analogy, Dabich says that when one buys a car, one knows it has been tested by the manufacturer to work as it has been built. “Youre not going to screw with the basic car the way you would with a software package,” he states. “But youll tailor the package to the way you want to work.”

Instead, Dabich recommends that software buyers tailor their processes to the software they purchase. This would “cut down bugs, because youre not playing with the system,” he asserts.

When it comes to who is responsible for software errors, Dabich says, “Id be willing to bet that its 50-50 between the vendors and the buyers.” While buyers arent keeping up with current releases or are changing basic programming, “vendors are not fixing problems fast enough and theyre not making it easy to implement new releases,” he states.

Why do buyers willingly accept products that contain so many errors? According to Dabich, senior company officials are often unaware of the magnitude of the problem.

“Most CEOs dont know how messed up their IT is,” states Dabich. Part of the reason for that is that information technology professionals are reluctant to admit that they are spending time fixing bugs in software that IT may have recommended purchasing, he explains.

According to the NIST study, “all developers of financial services software agreed that an improved system for testing was needed. They said that an improved system would be able to track a bug back to the point where it was introduced and then determine how that bug influenced the rest of the production process.”

The study adds that the developers believe that better testing tools and methods could reduce installation expenditures by 30%.

NIST states that the development of standardized testing tools “that have undergone a rigorous certification process would have a large impact on the inadequacies currently plaguing software markets. For example, the availability of standardized test data, metrics and automated test suites for performance testing would make benchmarking tests less costly to perform.”


Reproduced from National Underwriter Life & Health/Financial Services Edition, August 5, 2002. Copyright 2002 by The National Underwriter Company in the serial publication. All rights reserved.Copyright in this article as an independent work may be held by the author.