Lethal Autonomous Weapon Systems and Automation Bias

Autonomy in weapon systems is already a genuine concern. States try to come up with their own definitions of these systems and pay utmost effort to impose their own understanding of these systems upon other states. For a fairly high number of states barring a total ban on such weapons would be the ideal solution; however, such states that are anxious about the increase in autonomy in war-making capabilities, as adopts a second-best scenario to contain risks created by the deployment of such systems. To this end, placing them under meaningful human control emerges as an important political and legal objective. The author believes that placing autonomous weapons under human supervision, despite its initial promise, will yield negative results. This is due to the fact that humans tend rather to be too willing to follow the solutions generated by autonomous systems. First observed in other industries of civilian nature like aviation or health, automation bias has the potential to negate  most if not all of supervision measures expected to ensure proper implementation of international humanitarian law.

Introduction

Algorithmic decision-making is becoming common in many societal domains, such as the military, criminal justice system, and law enforcement, and this raises serious ethical concerns[23]. This paper aims to shed some light onto one of these concerns, which is a very important human trait with grave possible consequences. This human tendency is the so-called automation bias. First detected in civilian sectors, this bias can briefly be defined as a human inclination to depend too heavily on and to believe in information from autonomous systems, even when contradicting or differing information from other sources is available or could easily be found with the right search[16].

In the upcoming years, one feels justified to take for granted that systems with different levels of autonomy will serve various objectives during armed conflicts[45]. Since the autonomous weapon systems seem to be the new weapons of choice in warfare[23], this bias will have warfare-related impacts as well. This impact first and foremost jeopardizes a proper implementation of IHL, which is undoubtedly essential for protecting civilians and non-military objects.

This paper is an endeavor to elaborate an innate human tendency and its impact on machine-human interaction. It will first set out to present the development of debate on autonomy in weaponry. Following that, a definition will be presented for the purposes of this paper. Human-machine interaction shall be then briefly investigated. Then, the author will elaborate on the concept of meaningful human concept. Finally, the author aims to offer a sounder opinion as to how a satisfying degree of control can, if ever, be formed.

It is the fundamental stance of this paper that overly optimistic assumptions as to the reliability of autonomous weapons and the feasibility of a meaningful human control over them may prove insubstantial in the face of many factors, including automation bias. This bias carries in itself the risk of turning human operators into complete automatons. The existence of a supervising human operator who is merely accepting the automated solutions without ever bothering to question them would actually come to mean that there is no real human control left over the LAWS[42]. This insufficient level of oversight and supervision comes in essence to mean nothing but a very shaky ground for the protection of humanitarian values as well as the opening of the gates of impunity for transgressions like war crimes. Being cognizant of this alarming fact, the author believes that one safe way to secure humanity against autonomous weapons may be to introduce a comprehensive restraint or a ban on the development and deployment of such systems and weapons.

Chronological Development of the LAWS Problematique

A specter has been haunting the diplomatic negotiations and academic discussions for a long time now and it is the tormenting problem of how humanity will cope with the so-called lethal autonomous weapon systems (LAWS). This has actually been a hot topic since the beginning of the 2010’s[7]. A significant number of individuals who are very knowledgeable in robotics or international law contested the very idea of developing such systems that can decide on their own to kill human beings, showing no remorse for or even no appreciation of the consequences of their lethal solutions.

Another group of pundits defended the development of such systems on the grounds that these systems lacked the deficiencies and weaknesses human beings are prone to. In this line of thinking, LAWS embody asupermensch on dieselwhich will keep its mandate as programmed and discharge its soldierly obligations without the meddling of any humane conditions or vulnerabilities like exhaustion, rage or revenge[46].

There is now a large body of literature on the issue with a view to especially probing into their compatibility with international humanitarian law and, to a lesser extent, international human rights law. This paper does not aim to add anything to the already existing body of works on these crucial points, rather it deals mainly with automation bias and its effects in relation with autonomous weapon systems. It will endeavor to elaborate on the question of whether meaningful human control is ever attainable. This matter of control over LAWS has assumed a central position not only in diplomatic discussions but also in academic works. For the sake of this paper, the author will use the LAWS definition by the US Department of Defense[20]. According to this, LAWS are such weapons systems, which ‘...once activated, can select and engage targets without further intervention by a human operator’, which is a commonly adopted definition[57].

First Debates about LAWS

In 2013 the topic of LAWS first made it to the top the international political agenda and has so far kept that hot agenda status[21]. Since 2014 experts have been meeting to discuss the possible legal or political reactions on behalf of states and their ramifications at large[4]. First kickstarted as a non-governmental meeting and then transformed into a full-blown governmental experts’ get-together in 2017[41], these talks have been a great learning experience for the young and the uninitiated in terms of how international law-making among states proceeds. Diplomats, concerned individuals, learned academics and international NGOs have offered additional insights during these talks. These contributions not only facilitated a larger dissemination of the ideas of those individuals but also played an influential role in the improvement