DRONES: Need we fear the government-industrial robot complex?

[Part of the series: 10 Questions to Spur the Drone Debate]


"The government was offering the company a fortune, and threatening it with antirobot legislation in case of refusal. We were stuck then, and we're badly stuck now." (Isaac Asimov, "Little Lost Robot," p. 167)

Not surprisingly, the industrial companies that have developed an addiction to war production are now veering into an addiction to robotic war production. Just yesterday, at a meeting of the Anti-War Committee of Chicago, we were discussing the way in which Boeing has bet the future of the company on the next generation drone and a $1.5 billion contract from the Navy. (More at What If Illinois Became a "War-Profiteer-Free Zone" ?)

The "military-industrial complex" that Dwight Eisenhower warned against has been extended via a government-industrial robot complex.

And, as with the military-industrial complex, the government-industrial robot complex has thoroughly penetrated our country's education and research sector. While reading P.W. Singer's book, Wired for War, I was struck by the problem faced by all robotics researchers: sooner or later all roads lead to military use. Midwestern universities and colleges have a BIG drone problem.

The government calls the shots. If you think it doesn't, and/or that you're not implicated, you're kidding yourself. Four anecdotes come to mind.

First, the talk by a scientist at a conference entitled "The Atomic Age" in Chicago in 2011. "You would think there are some areas of study that the military is not interested in. For instance, I specialize in sand: how it flows, how it piles up, etc. What could be more boring than sand? Well, lo and behold, in 1991 the U.S. invaded Iraq, and suddenly the Army was interested in sand. Guess whose phone started to ring!"  Nothing is immune from military interest; the more useful the technology, the more the military will demand to be involved.

Second, the film Inextinguishable Fire, by Harun Farocki. The film is about napalm, but it could easily be about drones. It shows people who work for a technology company trying mightily to disown the consequences of their company's products. ("I just work in the testing department . . . ." or "Actually, our products have lots of other uses . . . .") We need to start being honest: If you're a cog in the wheel of war, you're responsible!

Third, a story that most people are familiar with: right before the dropping of the atomic bombs on Hiroshima and Nagasaki, numerous Manhattan Project scientists signed on to a letter urging President Truman to not use the bomb. We all know how that turned out. (I was reminded of this incident by its dramatization in the powerful opera by John Adams, Doctor Atomic.) The only way to stop governments from using weapons is to make sure they never get access to them in the first place.

Finally, a very similar account by the Soviet dissident Andrei Sakharov. Sakharov recounted, in his biography, the celebration after the successful development by his scientific team of a working hydrogen bomb. "May our bombs only ever explode over test sites, not over cities!" he proclaimed in a toast. This was greeted with withering, profane sarcasm by the political commissar who was presence. "You take care of making us strong, comrade; we'll choose the targets!"

What's the solution? Should we all become Luddites?

It's not clear how to keep the military's hands off robotic and other technologies. But the first step is to open our eyes and open and our mouths and start to be honest about the extent of the problem. What are we prepared to do to keep technologies from being used by the government to injure people?

Let's start debating the drones now. (Here are nine more questions to guide the debate.)


(Page references are to the 1990 Byron Preiss Visual Publications edition of Robot Visions.)

Drone to Human: Leave the Thinking to Me

[Part of the series: 10 Questions to Spur the Drone Debate]


In early July, 1988, I was on a business trip in China. On the 4th of July, I was in Chengdu, in Sichuan province, when a brief exchange occured that is etched in my memory.

We were scheduled to visit a leather tannery, and my counterpart from the local office of the China National Native Produce and Animal By-Products Import/Export Corporation (ANIBY for short) met me in the usual white van after breakfast for the long ride to the tannery. We may have been going to Leshan that day; or perhaps to Yaan; or else to one of the other far-flung tanneries I used to visit.

"A missile departs the forward launcher of Vincennes
during a 1987 exercise." (Source: Wikipedia)

As I climbed into the back of the van that morning, he said to me in Chinese, "Shi Gelei [Scarry]? Have you heard? A U.S. missile shot down an Iranian 747 . . . ."

I couldn't quite parse the sentence. My Chinese was pretty good, but he was saying words weren't making any sense. What was he talking about?

He told me again, slowly, "In the Mideast - Iran - a plane, a 747. The United States. Shot it down."

I repeated the words back to him, putting the meaning together in my mind, as he looked me in the eyes and nodded. "More than 200 people were killed," he said.

Finally, I squinted, shook my head, and replied, incredulous. "No . . . " I said, "How could a mistake like that happen . . . ?"

And then came his response, the part that I have never been able to forget:

"Mistake?" He smiled ironically. "I suppose some people might imagine it was a mistake . . . ."

Map of the Iran Air 655 shootdown
(Source: Wikipedia)

That's how I heard about the two hundred and ninety (290) people who died on Iran Air Flight 655.

It was the 4th of July, I was a long way from home, and I was with people who had a completely independent worldview from the one I was surrounded with when I was in the United States. At that moment, I realized that we Americans are deluded when we imagine that people in the rest of the world assume that we are filled with nothing but good intentions, and that they give us the benefit of the doubt.  At that brief moment in July, 1988, I didn't yet understand why this was so, but it began a journey to try to find out. In the twenty-five years that followed my thinking has gone through a big evolution.

I think of that day often. Nonetheless, I have to admit that I still hadn't quite made the connection between this incident and the work to rein in drone warfare, in which I am so involved, until reading P.W. Singer's book, Wired for War. Singer makes a very important point about the Iran Air Flight 655 case. As he explains it, the Aegis information system that spotted and interpreted the information about that civilian flight, and guided the shoot-down, is exactly the kind of "man-in-the-loop" system that is being used in the drone killings in the Mideast today: in theory, humans are exercising their judgement in the process, but in reality the computer system is viewed as too "smart" to be second-guessed by a human being.

Aegis cruiser "control room": who's controlling who?
(Source: Wikipedia)

I encourage everyone to read Singer on the The Ethical, Psychological Effects Of Robotic Warfare. As he explains:

"Designed to defend Navy ships against missile and plane attacks, the [Aegis] system operates in four modes, from 'semi-automatic,' in which humans work with the system to judge when and at what to shoot, to 'casualty,' in which the system operates as if all the humans are dead and does what it calculates is best to keep the ship from being hit. Humans can override the Aegis system in any of its modes, but experience shows that this capability is often beside the point, since people hesitate to use this power . . . . [In the Iran Air 655 incident] Though the hard data were telling the human crew that the plane wasn't a fighter jet, they trusted the computer more. Aegis was in semi-automatic mode, giving it the least amount of autonomy, but not one of the 18 sailors and officers in the command crew challenged the computer's wisdom. They authorized it to fire."

Multiply this situation by 1,000, and you have the situation in place today with the U.S. drone fleet: skies full of lethal robots theoretically controlled by "pilots" . . . but, in fact, front-loaded with a bias to kill, with little impetus to contradict that bias.

Some people think it's not yet time to worry about drones, because they're "not really making decisions on their own" -- and they're certainly not at the point of trying to out-think humans. People think, "There's still a 'man in the loop,' so what is there to worry about?" In other words, as Isaac Asimov has been very effective at illustrating in his "Robot" series, it is always tempting to think that the culprit we have to be afraid of is the robot:

"I don't like to have Nestor 10 continue to elude us. It's bad. It must be gratifying his swollen sense of superiority. I'm afraid that his motivation is no longer simply one of following orders. It think it's becoming more a matter of sheer neurotic necessity to out-think humans." (Isaac Asimov, "Little Lost Robot," p. 178)

But as Peter Singer's description of the Iran Air Flight 655 case illustrates, the real culprit is the humans "in-the-loop" who have ceased to exercise control.

So . . . what do we need to be more afraid of? Robots with a compulsion to out-think humans? or humans that are afraid to second-guess the robots?

Let's start debating the drones now. (Here are nine more questions to guide the debate.)


(Page references are to the 1990 Byron Preiss Visual Publications edition of Robot Visions.)


Related posts

The article goes on, in what can only be dubbed "dronespeak," to explain away all the deadly consequences of the U.S. drone program in terms of information theory and neurobiology . . . the "swirl of data" . . ."multitasking" . . . "theta" . . . .

(See Drones, Dronespeak, and Death TV: "Intense" )






We can now entrust all the dirty work -- including war -- to robots. (Or can we?)

(See A Modest Proposal: Debate the Drones )

In my opinion, the reason to focus on drones is this: when we focus on drones, the general public is able to "get," to an unusual extent, the degree to which popular consent has been banished from the process of carrying out state violence. (Sure, it was banished long ago, but the absence of a human in the cockpit of a drone suddenly makes a light bulb go off in people's heads.) It takes some prodding, but people can sense that drone use somehow crosses a line. And that opens up the discussion about how our consent has been eliminated from the vast range of US militarism.

(See "Why focus on drone attacks?")

Design Drones To Guarantee No Humans Will Be Injured

[Part of the series: 10 Questions to Spur the Drone Debate]


"Those were the days of the first talking robots when it looked as if the use of robots on Earth would be banned. The makers were fighting that and they built good, healthy slave complexes into the damned machines." (Isaac Asimov, "Runaround," p. 117)

What design features can we reasonably require in drones to assure that we are safe?

I've proposed that this issue be debated, and that the robot literature created by Isaac Asimov has already posed numerous big question that can guide the debate.

This week, an unmanned fighter jet used as a target crashed in Florida. (It was thus a "drone," though not the same type being used in U.S. military and CIA operations in the Mideast.)

I was struck by this statement in the news coverage: "An explosive device in the plane destroys it if it becomes uncontrollable," according to an Air Force fact sheet. (See Air Force drone crash closes remote Florida highway.)

In other words, steps were taken to render the target drone "safe."  Those steps may have been inadequate, but the important thing is that a clear principle has been expressed. Design features have been put in place. When the drone becomes unsafe, it is sacrificed.

What design features would need to be in place in order to make drones really safe? What design features would need to be in place in order to assure that drones obey the 1st Law of Robotics ("A robot may not injure a human being or, through inaction, allow a human being to come to harm") ?

It goes without saying that drones won't be safe until they stop firing missiles and killing people. There will certainly be regulations in the very near future prohibiting people from putting weapons on drones.

But let's ask the deeper question. We all know that regulations on human behavior are not the ultimate answer, because people break rules. Shouldn't the drones, themselves, play a role in making sure that they are not abused? Why can't drones be designed to know whether they, themselves, have been armed with weapons, and, if armed with weapons, cease functioning (or self-destruct)? Why can't we have an Air Force fact sheet that says, "An explosive device in the drone destroys it if it detects the presence of weapons" ?

The answer, of course, is that drones can be designed that way. All that's missing is the will to do so.


Let's start debating the drones now. (Here are nine more questions to guide the debate.)


(Page references are to the 1990 Byron Preiss Visual Publications edition of Robot Visions.)

10 Questions to Spur the Drone Debate

Everybody likes a good robot competition. How about one with REALLY high stakes?

I've suggested that it's time for a serious debate on drones, and that a good place to start is with Isaac Asimov's "Three Laws of Robotics."

Here are 10 questions that come straight out of the writings of Asimov and that can help spur the debate.


(1) Shouldn't the onus be on companies supplying these robots to abide by rules that assure the public is safe?

"You are perfectly well acquainted, I suppose, with the strict rules against the use of robots on inhabited worlds . . . . You are also aware that all positronic robots are leased, not sold; that the Corporation remains the owner and manager of each robot, and is therefore responsible for the actions of all." ("Evidence," p. 138)

(See Are Corporations Responsible for Drone Killings? .)


(2) What happens to the rules when governments decide they "need" to skirt them?

"Hyper Base happens to be using several robots whose brains are not impressioned with the entire First Law of Robotics. 
"Aren't impressioned? . . . 
"It was on government order and there was no way of violating the secrecy . . . ." (p. "Little Lost Robot," p. 163)


(3) What happens to the rules when robot-on-robot violence is involved?

"[T]here is one time when a robot may strike a human being without breaking the First Law. Just one time." ("Evidence," p. 160)


(4) What might advanced technological society look like without rules that prohibit robots from doing harm?

"The ultimate machine is an intelligent machine and there is only one basic plot to the intelligent-machine story -- that it is created to serve man, but that it ends by dominating man. It cannot exist without threatening to supplant us, and it must therefore be destroyed or we will be." ("Machine and Robot," p. 440)

(5) What design features can we reasonably require in drones to assure that we are safe?

"Those were the days of the first talking robots when it looked as if the use of robots on Earth would be banned. The makers were fighting that and they built good, healthy slave complexes into the damned machines." ("Runaround," p. 117)

(See Design Drones To Guarantee No Humans Will Be Injured .)




(6) Need we fear the government-industrial robot complex?

"The government was offering the company a fortune, and threatening it with antirobot legislation in case of refusal. We were stuck then, and we're badly stuck now." ("Little Lost Robot," p. 167)

(See DRONES: Need we fear the government-industrial robot complex?)

(7) What do we need to be more afraid of: robots with a compulsion to out-think humans? or humans that are afraid to second-guess the robots?

"I don't like to have Nestor 10 continue to elude us. It's bad. It must be gratifying his swollen sense of superiority. I'm afraid that his motivation is no longer simply one of following orders. It think it's becoming more a matter of sheer neurotic necessity to out-think humans." ("Little Lost Robot," p. 178)

(See Drone to Human: Leave the Thinking to Me.)


(8) What's at stake? What would it look like to put these robots to work in the cause of distributive justice, instead of in the cause of violence?

"There can be no serious conflicts on Earth, in which one group or another can seize more power than it has for what it thinks is its own good despite the harm of Mankind as a whole, while the Machines rule. If popular faith in the Machines can be destroyed to the point where they are abandoned, it will be the law of the jungle again." ("The Evitable Conflict," p. 212)


(9) Isn't the biggest obstacle to debating "The Three Laws," in fact, the way it exposes the danger humans pose?

"If, by virtue of the Second Law, we can demand of any robot unlimited obedience in all respects not involving harm to a human being, then any human being, any human being, has a fearsome power over any robot, any robot. In particular, since Second Law supersedes Third Law, any human being can use the law of obedience to overcome the law of self-protection. He can order any robot to damage itself or even destroy itself for any reason, or for no reason." ("The Bicentennial Man," p. 266)


(10) Are we underestimating the need for the people who have thought the most about this issue to carry this discussion into the public square?

"It was the battle over public opinion that held the key to courts and Legislature and in the end a law passed which set up conditions under which robot-harming orders were forbidden." ("The Bicentennial Man," p. 266)


I hope these questions provide useful material for debate. A closing thought from the story "Galley Slave":

"It is only by being concerned for robots that one can 

truly be concerned for twenty-first century man. 

You would understand this if you were a roboticist." (p. 391) 

Isaac Asimov wrote those words in 1941 . . . .


(Page references are to the 1990 Byron Preiss Visual Publications edition of Robot Visions.)

Related posts

We have taboos against certain practices despite the fact that they are convenient. The taboos reflect a recognition of social disruptiveness that outweighs the benefits of some limited subset of circumstances in which the practices happen -- and, more importantly, even the possible desires of the most powerful people in the society.

(See Why Isn't Robotic Killing Taboo? )




Beyond recognizing the inherent contradictions of "pre-emptive violence," we must confront an urgent problem related to technology: the automation of "pre-emptive violence" -- e.g. via drone technology -- is leading to a spiral (or "loop" or "recursive process") that we may not be able to get out of.

(See When "Pre-emptive Violence" Is Automated ....)





With drones, people become just dots. "Bugs." People who no longer count as people . . . .

(See Drone Victims: Just Dots? Just Dirt? )

Are Corporations Responsible for Drone Killings?

[Part of the series: 10 Questions to Spur the Drone Debate]


Shouldn't the onus be on companies supplying drones/robots to abide by rules that assure the public is safe?

USR and MM 1996 "ROBOT"
Fully automatic RB Series
U.S. Robots and Mechanical Men Inc.
Schenectady, New York 12309

I've proposed that this issue be debated, and that the robot literature created by Isaac Asimov has already posed numerous big question that can guide the debate.

I was struck by this sentence in one of the first of the Asimov robot stories I read:

"You are perfectly well acquainted, I suppose, with the strict rules against the use of robots on inhabited worlds." ("Evidence," p. 138)

A major theme in the literature is that the public has the power to bring about an outright ban. Food for thought . . . .

I was also struck by this statement:

"You are also aware that all positronic robots are leased, not sold; that the Corporation remains the owner and manager of each robot, and is therefore responsible for the actions of all." ("Evidence," p. 138)

Well, of course! Of course the corporation that makes the drone/robot should be responsible for its actions. (I wonder if General Atomics considers itself liable for the consequences of every drone strike?)

General Atomics Headquarters

The drone systems being used by the U.S. military and other parts of the U.S. government are already plenty complex. How about as they become even more complex? Are we to expect a vast industry of contractors to manage those systems? (The vast surveillance contractor world, brought to the public's attention by the Snowden disclosures, come to mind.) Which of them are responsible for what the drones/robots do? All of them? None of them?

How's this for a governing policy?  

Drones shall be banned from use anywhere on Earth; AND their use 

in other places shall be governed by the "Three Laws of Robotics" 

-- including "(1) A robot may not  injure a human being or, 

through inaction, allow a human being to come to harm." 

Providers of drones will have legal liability for abiding by these rules.

Let's start debating the drones now. (Here are nine more questions to guide the debate.)

(Page references are to the 1990 Byron Preiss Visual Publications edition of Robot Visions.)


Related posts

We can now entrust all the dirty work -- including war -- to robots. (Or can we?)

(See A Modest Proposal: Debate the Drones )











 

Isn't the time fast approaching when Boeing recognizes that it's not just one or another of their weapons systems -- or weapons systems customers -- that's the problem? Isn't Boeing's entire defense systems division "bad Boeing"?

(See BOEING: "Breaking Up Is (Is Not) Hard to Do" )

It is time now to turn to the dirty secret of American life and the primary dilemma of the antiwar movement: the military money that flows to EVERY Congressional district, and in particular the "good jobs" that members of Congress think they are protecting when they vote for ever-higher levels of military spending.

(See Drones, Permawar, and the Problem of "Good Jobs")

A Modest Proposal: Debate the Drones

[Part of the series: 10 Questions to Spur the Drone Debate]


I've been marveling at Isaac Asimov's "Three Laws of Robotics" . . . and when I heard about an upcoming exposition in San Francisco aimed at popularizing drones, I thought, "Isn't it time to get more people talking about this?"


"Endless Uses"

Robots: saviors?

The San Francisco event will be on July 25-26, and is entitled Small Unmanned Systems Business Expo. It's organizers say drones "have gotten a bad rap" and "the positive uses for this technology are endless." They advocate uses ranging from provision of life-saving vaccines to beer and pizza delivery.

Many people who are concerned about the use of drones for surveillance and warfare want to see their concerns reflected at events like the San Francisco drones expo. Why not include a debate?

There are lots of ways that a debate could be structured to air and weigh concerns of this type. One way would be to debate this proposition: "RESOLVED: No drones may be deployed that do not obey the 'Three Laws of Robotics'."


Debate the "Three Laws"  

Isaac Asimov: I, Robot

As all of the organizers of the San Francisco event certainly know, the "Three Laws" are:

(1) A robot may not injure a human being or, through inaction, allow a human being to come to harm.

(2) A robot must obey the orders given to it by human beings, except where such orders would conflict with the First Law.

(3) A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

The only question is: are they willing to encourage the broader public to engage with these ideas?

To those drone boosters who say, "Why would we want to include anything in our event to encourage questioning drones?" I would reply: you, yourselves, acknowledge that drones are getting "a bad rap." Showing that you are willing to openly discuss the issues is the best way to clear the air.

To those drone opponents who say, "Why would we stoop to debating with drone proponents? No quarter!" I reply: "Can you think of a better way to promote real education on this topic?"

A Dilemma Foreseen

A quadricopter in every garage?

The "Three Laws" are thoroughly explored in the works of Isaac Asimov. As I indicated above, I marvel as I read Asimov and see the way in which he foresaw the ethical conundrum in which we now find ourselves embroiled. Story after story and essay after essay tease out the issues involved. A debate could be a springboard for getting many more people involved in studying and thinking about these questions. The "I, Robot" literature is well-known to techies, but should be much more broadly studied by the general public.

The literature created by Asimov invites us to consider such questions as . . .

* Shouldn't the onus be on companies supplying these robots to abide by rules that assure the public is safe?

* What happens to the rules when governments decide they "need" to skirt them?

* What happens to the rules when robot-on-robot violence is involved?

* What might advanced technological society look like without rules that prohibit robots from doing harm?

MORE: 10 Questions to Spur the Drone Debate.


What would Asimov say? 

Isaac Asimov, Second Foundation

In his 1979 essay, "The Robot As Enemy," Isaac Asimov wrote, "Will human beings deliberately build robots without the Laws? I'm afraid that is a distinct possibility . . . " (p. 448). He specifically foresaw the exact developments in the robotization of the military that we are seeing today:

"computerized planes, tanks, artillery, and so on, that would stalk the enemy relentlessly, with superhuman senses and stamina. It might be argued that this would be a way of sparing human beings. We could stay comfortably at home and let our intelligent machines do the fighting for us. If some of them were destroyed -- well, they are only machines. This approach to warfare would be particularly useful if we had such machines and the enemy didn't." (p. 449)

In the end, however, he concludes: "No, I feel confident that attempts to use robots without safeguards won't work and that, in the end, we will come round to the Three Laws."

LET THE DEBATE BEGIN!

(Page references are to the 1990 Byron Preiss Visual Publications edition of Robot Visions.)


Related posts

Beyond recognizing the inherent contradictions of "pre-emptive violence," we must confront an urgent problem related to technology: the automation of "pre-emptive violence" -- e.g. via drone technology -- is leading to a spiral (or "loop" or "recursive process") that we may not be able to get out of.

(See When "Pre-emptive Violence" Is Automated ....)

Some people think it's not yet time to worry about drones, because they're "not really making decisions on their own" -- and they're certainly not at the point of trying to out-think humans. People think, "There's still a 'man in the loop,' so what is there to worry about?"

(See Drone to Human: Leave the Thinking to Me)

I've suggested that it's time for a serious debate on drones, and that a good place to start is with Isaac Asimov's "Three Laws of Robotics."  Here are 10 questions that come straight out of the writings of Asimov and that can help spur the debate.

(See 10 Questions to Spur the Drone Debate )

 In a flurry of debates in October, the U.S. presidential candidates will talk about every topic under the sun -- both domestic and foreign -- in locations across the country -- from Denver to New York City to Boca Raton -- in formats both formal and informal -- with an assortment of moderators from all the major networks. Be assured, however, that there is one topic that will be off-limits: drones.

(See Oct 13: National "Debate the Drones" Day? )

Of course the corporation that makes the drone/robot should be responsible for its actions. (I wonder if General Atomics considers itself liable for the consequences of every drone strike?)

(See Are Corporations Responsible for Drone Killings? )