What Can Happen When Computers Take Over

Rochlin Book Warns of the Dangers of
Relying on High-Tech Systems

by Robert Sanders

For those alienated by computers, who feel computers have taken over their lives, Gene Rochlin has one message-it's going to get worse.

In his new book, "Trapped in the Net: The Unanticipated Consequences of Computerization" (Princeton University Press, 1997), Rochlin details how society is being driven toward pervasive computerization, in the process alienating workers from their jobs, soldiers from the battlefield and the stock market from reality.

Though Rochlin refers to himself as an "academic curmudgeon," he doesn't merely dredge up the petty annoyances created by society's increasing reliance on computers- the endless voice mail trees, balky reservation systems and ATMs that eat deposits.

He's more concerned with insidious computerization invisible to the layperson-the computerized networks that keep planes in the air, handle complex financial transactions and protect our country. The "net" referred to in the title is not the Internet, but networks of computers.

"What is happening is that all things in society are being mediated by computerized systems," Rochlin says. "We are being trapped in computer systems and networks in ways we can't avoid."

It's not that he doesn't like computers-he has used computers for decades in his research. His main objection is that rampant computerization occurs without consideration of the consequences, most significantly a loss of skills and expertise.

"We are on a largely unexamined drive to network everything in sight because computers make it possible," he says. "My concern is that we are creating a system that cannot function without computers.

"I don't like the dependency, the loss of autonomy, the loss of work skills. I'm alarmed at my own loss of control over the instruments of my work, the instruments of life."

A professor of energy and resources, Rochlin is a physicist with a background in political science who has conducted long-term studies of technologically sophisticated organizations with complex structures, in particular the military. He analyzed the military's success with high-tech weapons systems during the 1991 Gulf War with Iraq and found many problems with computerized control systems. Among the problematic new technologies were the Patriot missile, the F-16 fighter and even the F-117 stealth aircraft.

He also conducted a thorough analysis of a 1988 tragedy involving a billion-dollar technological marvel, the USS Vincennes, which accidentally shot down an Iranian passenger liner, killing 290 people. In a 1990 report, Rochlin concluded that the highly sophisticated technology aboard the Vincennes distorted the decision-making process, over-riding human judgment and resulting in the ship firing on an unidentified airplane.

"What is happening is that all things in society are being mediated by computerized systems. We are being trapped in computer systems and networks in ways we can't avoid." -Gene Rochlin


At the time, he warned that the future may hold more such tragedies, as the U.S. military increases its reliance on large-scale technological systems. The intervening years have only made Rochlin more pessimistic.

"The military is moving faster than most in this direction, because the people with authority are coming up through the technical side rather than the old way," which was to command in the field. "These techno-managers want to fight but not have anyone killed," and the technology is a way to distance the fighting man or woman from the battlefield.

Another lost cause, he says, is the battle in the skies over cockpit automation-what has come to be known as the "glass cockpit." Many pilots today have never gotten the hands-on experience that was a hallmark of the previous generation of pilots. Trained on simulators, the role of today's pilot is to set electronic controls and then sit back and watch, as if through a glass window-the computer screen.

The modern Airbus 320, for example, can only be flown through its computer system, and the F-16 fighter cannot even be flown by hand, since it is dynamically unstable. While computerized systems may be able to handle problems that were foreseen by those who programmed the system, Rochlin wonders what will happen if something unforeseen occurs, such as a door blowing off in midair.

"Once upon a time flying was done by pilots and the job was to be an expert at problem solving," he says. "The new generation of pilots will never be able to do this if they have never flown a plane."

Air traffic controllers, nuclear power plant operators and workers in many other industries confront the same problem when computers distance them from their jobs.

"Computers take the responsibility from an operator and diffuse it throughout the network, so that no one really understands how the system works, and no one is responsible for its failure," Rochlin says. "We recommend that controllers let the system go sometimes so that operators learn to deal with such situations."

Computers have similarly infiltrated manufacturing and the retail economy. A recent innovation called just-in-time manufacturing relies on computers to shuffle around inventories and stocks on a very tight schedule so as not to tie up too much money or space in the stockroom. The result is a complex distribution system with no "slack"-a system that, once it comes apart, is extremely difficult to re-establish. This became evident after the devastating 1995 earthquake in Kobe, Japan.

"In Kobe everything had to be working before they could get the manufacturing system up, but the dependency was so tight that things stopped functioning for a long time," Rochlin said. "Just-

in-time manufacturing is controversial because it assumes that the state of technology is so reliable that you don't need to keep inventory or stock on hand."

Computers have invaded the stock market too, distancing traders from the real economy, he says. Computerized trading also has created a more closed market, where the small trader is essentially locked out by big computerized traders, and has led to an emphasis on short-term profits over long-term, stable investments. It has brought down the market at least once, in 1989, and Rochlin predicted in his book that it would happen ever more frequently.

"The danger to the stock market hasn't materialized because some of the software didn't work," he now says. "So major problems with the financial market are still a few years off. But the danger is building, with the added problem that brokerages are becoming more proprietary now, so it's harder to find out how their computer programs work."

Rochlin is hopeful that some solutions to the problem of over-reliance on computers and networks will come from applying the concept of "decomposability"-the ability of a system to break up into functional pieces when network control is lost.

Designing systems with this built in could solve some of the problems associated with a system that is too tightly controlled.

On the other hand, Rochlin sees new problems arising with the Internet as it is used more in operating networks. The key, he says, is not to let the tools-computers and the Internet-shape human affairs.

"These computerized systems affect our lives and coerce our behavior," Rochlin warns. "While computers will become increasingly more invisible, they never should be out of mind."


Copyright 1998, The Regents of the University of California.
Produced and maintained by the Office of Public Affairs at UC Berkeley.
Comments? E-mail berkeleyan@pa.urel.berkeley.edu.