Comprehensive Human Oversight over Autonomous Weapon Systems Ilse Verdiesen
COMPREHENSIVE HUMAN OVERSIGHT OVER AUTONOMOUS WEAPON SYSTEMS Ilse Verdiesen
Cover concept: Ilse Verdiesen Cover: Ilse Modder (www.ilsemodder.nl) Layout: Ilse Modder (www.ilsemodder.nl) Print: Gildeprint Drukkerijen (www.gildeprint.nl) ISBN:978-94-6496-075-4 © E.P. Verdiesen, 2024. All rights reserved. No part of this thesis may be reproduced, stored in retrieval systems, or transmitted in any form or by any means without prior permission of the author.
Comprehensive Human Oversight over Autonomous Weapon Systems Dissertation for the purpose of obtaining the degree of doctor at Delft University of Technology by the authority of the Rector Magnificus prof.dr.ir. T.H.J.J. van der Hagen chair of the Board for Doctorates to be defended publicly on Wednesday 3 April 2024 at 15:00 o’clock By Elizabeth Paulina VERDIESEN Master of Science in Systems Engineering, Policy Analysis and Management, Delft University of Technology, the Netherlands born in Rotterdam, the Netherlands
This dissertation has been approved by the promotors. Composition of the doctoral committee: Rector Magnificus, chairperson Dr. M.V. Dignum Delft University of Technology, promotor Dr. F. Santoni de Sio Delft University of Technology, promotor Independent members: Prof.dr. C. Jonker Delft University of Technology Prof.dr.ir L.M.M. Royakkers Eindhoven University of Technology Prof.dr. F. Osinga Leiden University A. Kaspersen MSc. Carnegie Council for Ethics in International Affairs Prof.dr.ir. M.F.W.H.A. Janssen Delft University of Technology, reserve member
This dissertation and research are based on my own work that I conducted independently over the past six years. I have written all my published articles with co-authors which is common practise in my research field. At the start of each chapter, I clearly state in which articles parts of the chapter have been published. I have written this dissertation in plural using the direct verb ‘we’ as is customary in my discipline in recognition of all the comments and supervision of my promotors and fellow researchers to show that I greatly appreciate their support and mentoring. For I firmly believe that the following applies both in the scientific community as in life: “If you want to go fast, go alone. If you want to go far, go together”
CONTENTS Acknowledgements PART I: INTRODUCTION 1. Introduction 1.1 Research objective Scientific and societal relevance Scope 1.2 Scenario 1.3 Research approach 1.4 Outline of thesis PART II: CONCEPTUAL INVESTIGATION PHASE 2. Extensive literature review 2.1 Decision-making processes in AI 2.2 Architectures for ethical decision-making in AI 2.3 Autonomy 2.4 Autonomous Weapon Systems Definition Classification of Autonomous Weapon Systems 2.5 Values Value theories Universal values 2.6 Values related to Autonomous Weapon Systems 2.7 Design for Values Value hierarchy 2.8 Responsibility 2.9 Accountability Accountability gaps 2.10 From Accountability via Control to Human Oversight 2.11 Control Engineering perspective Socio-technical perspective Governance perspective 2.12 Human Oversight 2.13 Conclusion 13 15 17 21 22 23 23 24 26 29 31 32 33 34 36 36 37 40 40 41 42 44 44 47 50 52 55 55 55 56 57 59 60
3. Conceptual Framework 3.1 Comprehensive Human Oversight Framework Technical layer Sociotechnical layer Governance layer Validation 3.2 Application of the Comprehensive Human Oversight Framework to Existing Military Control Instruments 3.3 Application Comprehensive Human Oversight Framework to Autonomous Weapon Systems 3.4 Glass Box framework Interpretation stage Observation stage 3.5 Comprehensive Human Oversight Framework projected on Glass Box Framework 3.6 Feedback loop: closing the gap Application Five-Point Systems Framework on Comprehensive Human Oversight Framework Application Five-Point Systems Framework to Autonomous Weapon System case using a toy example Validation 3.7 Conclusion PART III: EMPIRICAL INVESTIGATION PHASE 4. Value deliberation 4.1 Expert interviews 4.2 Value Deliberation Process 4.3 Method 4.4 Research set-up 4.5 Scenario and options 4.6 Sample pilot and actual study 4.7 Results 4.8 Validation results Value Deliberation process 4.9 Conclusion 63 64 64 65 65 66 67 71 74 76 78 79 80 81 86 88 89 91 93 94 94 95 96 97 98 99 102 102
PART IV: TECHNICAL INVESTIGATION PHASE 5. Implementation concept 5.1 Scenario 5.2 Simulation of implementation concept Pre-Flight Mission Planning Process Post-Flight Mission Evaluation Process 5.3 Evaluation of simulation of the implementation concept 5.4 Conclusion PART V: CONCLUSION AND DISCUSSION 6. Conclusion 6.1 Conceptual investigation phase 6.2 Empirical investigation phase 6.3 Technical investigation phase 7. Discussion 7.1 Emerging insights Value-neutral definition of Autonomous Weapon Systems Operationalising Meaningful Human Control 7.2 Limitations 7.3 Future work 7.4 Contributions 7.5 Policy recommendations Bibliography List of publications Summary Samenvatting Appendix A. Questionnaire Value Deliberation Process Appendix B. Questionnaire validation Value Deliberation Process results Appendix C. Overview key insights from literature review Biography 105 107 108 110 111 111 117 118 121 123 124 129 130 133 134 134 135 137 138 139 140 144 150 152 155 158 168 171 182
ACKNOWLEDGEMENTS Climbing up and down the hills in Tivendens National Park in Sweden in the summer of 2023, I was pondering the journey of my PhD research that I am about to finish. Just like my hilly hike, my research has known ups and downs over the past six years. Getting one of my articles rejected and having to rewrite it completely in a different template was one of the low points. Also, not able to attend conferences in person during the COVID19 pandemic was something I hope not to relive again. However, I mainly experienced peaks during my PhD journey: the first AI Ethics & Society (AIES) in 2018 in New Orleans where I presented my first poster, attending the Group of Governmental Experts on Lethal Autonomous Weapons Systems (LAWS) of the UN as a member of the Dutch delegation, being part of the World Summit AI: Dual use technology expert panel, presenting my work at the AIES 2022 conference in Oxford, the panel at Summit on Responsible Artificial Intelligence in the Military Domain (REAIM 2023) in The Hague, recording a BNR podcast and EuroISME webinar, and my yearly trips to Umeå University in Sweden to visit Virginia and her Responsible AI research group. These highlights would not have been possible without the excellent supervision of my two promotors Virginia Dignum and Filippo Santoni de Sio. Thank you Virginia, for sharing your experience and providing guidance during my research journey, setting a high standard with your critical viewpoint and sharp comments, and for your and Frank’s hospitality in Umeå. You are an inspiration and role model in academia and life. Filippo, I greatly appreciate our discussions during the conceptual part of my research. You helped me develop my academic attitude and thinking, especially at the start of my PhD you encouraged me to find the depth in the arguments of my reasoning. Next to my two promotors, I would like to thank my fellow PhD students and researchers with whom I have worked and discussed my ideas over the past six years. Andrea, Andreas, Klara and Rijk, thank you for sharing your ideas and feedback. You made me feel part of the research community. Lastly, I would like to thank the Royal Netherlands Army for supporting my PhD and allowing me to conduct independent research on this topic. Especially my team members and managers of my last two positions who had to deal with my absence and take over my work when I was doing research, visiting conferences, writing my articles and this dissertation. I hope that this dissertation on Comprehensive Human Oversight of Autonomous Weapon Systems will lead to awareness and initiates discussion on the deployment of Autonomous Weapon Systems and Responsible AI in the Defence organisation.
Part I INTRODUCTION In the introduction we describe the context of our research, our research objective and questions, the scenario that we use in the different phases of our research, our research approach and we conclude with the outline of our thesis.
1| Introduction
18 CHAPTER 1 1 Autonomous Weapon Systems are weapons systems equipped with Artificial Intelligence (AI). They are increasingly deployed on the battlefield (Dawes, 2023; Heather M. Roff, 2016; Tucker, 2023). Autonomous systems can have many benefits in the military domain, for example in the Ukraine where the Fortem DroneHunter F700, which is an autonomous drone with radar control and artificial intelligence, is deployed to shield the country’s energy facilities from Russian attacks (Soldak, 2023). Yet the nature of Autonomous Weapon Systems might also lead to security risks and unpredictable activities as Non-Governmental Organisations (NGO’s) Human Rights Watch (Human Rights Watch, 2023) and the International Committee of the Red Cross (ICRC, 2023) indicate in their statements to The Group of Governmental Experts (GGE) on emerging technologies in the area of Lethal Autonomous Weapons Systems (LAWS) of the Convention on Certain Conventional Weapons (CCW) of the United Nations. Next to security risks and unpredictable activities, the impact on human dignity and the emergence of an accountability gap are mentioned as concerns with the use of Autonomous Weapon Systems. The alleged offence to human dignity entailed in delegating life-or-death decision-making to a machine is linked to the value of human life. The Campaign to Stop Killer Robots (2023) states on their website that: ‘…a machine should not be allowed to make a decision over life and death.’, because it is lacking human judgement and understanding of the context of its use. The United Nations are also voicing their concerns and state that ‘Autonomous weapons systems that require no meaningful human control should be prohibited, and remotely controlled force should only ever be used with the greatest caution’ (General Assembly United Nations, 2016). At the same time, many scholars express concerns that Autonomous Weapon Systems will lead to an “accountability gap” or “accountability vacuum”; circumstances in which no human can be held accountable for the decisions, actions and effects of Autonomous Weapon Systems (Matthias 2004; Asaro 2012; Asaro 2016; Crootof 2015; Dickinson 2018; Horowitz and Scharre 2015; Wagner 2014; Sparrow 2016; Roff 2013; Galliott 2015). This concern is also reflected in one of the guiding principles for LAWS of the GGE on emerging technologies in the area of LAWS of the CCW of the United Nations: ‘Human responsibility for decisions on the use of weapons systems must be retained since accountability cannot be transferred to machines. This should be considered across the entire lifecycle of the weapon system.’ (UN GGE LAWS 2018). Hence, the deployment of Autonomous Weapon Systems on the battlefield without direct human oversight is not only a military revolution according to Kaag and Kaufman (2009), but can also be considered a moral one. As large-scale deployment of AI on the battlefield seems unavoidable (Rosenberg & Markoff, 2016), the research on ethical and moral responsibility is imperative.
19 1 INTRODUCTION The concerns described above highlight that responsibility, accountability and human control are values often mentioned in the societal and academic debate on autonomous systems. Responsibility can be forward-looking to actions to come and/ or backwardlooking to actions that have occurred. Accountability is a form of backward-looking responsibility that refers to the ability and willingness of actors to provide information and explanations about their actions and defines mechanisms for corporate and public governance to hold agents and organisations accountable in a forum. Responsibility contributes to minimizing unintended consequences by anticipating on actions and unintended consequences to come and taking measures to prevent or mitigate them. Accountability can decrease unintended consequences in providing information and explanations by actors of their previous actions in order for other actors to learn from them and prevent mistakes and unintended consequences of their own. We found little empirical research that supports the concerns mentioned above or that provide insight in how responsibility and accountability regarding the deployment of Autonomous Weapon Systems are perceived by the general public and military. The Open Robots Ethics initiative surveyed the public opinion in a poll in 2015 (Open Roboethics initiative, 2015) and issued a report. However, the results were not published in an academic journal and the survey was not extensive enough to draw substantive conclusions. The notion of Meaningful Human Control is often mentioned as a requirement in the debate on Autonomous Weapon Systems to ensure accountability and responsibility over these type of weapon systems. The U.K.-based NGO Article 36 is credited for putting the concept of “Meaningful Human Control” at the centre of the discussion on Autonomous Weapon Systems by mentioning it in several reports and policy papers since 2013 (Amoroso & Tamburrini, 2021). Since then, the concept of Meaningful Human Control is often mentioned as requirement (Adams, 2001; Heather M Roff & Moyes, 2016; Vignard, 2014) to ensure accountability and responsibility for the deployment of Autonomous Weapon Systems, but this concept is not-well defined in literature and quantifying the level of control needed is hard (Schwarz, 2018). Adams (2001) noticed as early as 2001 that the role of the human changed from being an active controller to that of a supervisor and that direct human participation in decisions of AI systems would become rare. Some scholars are working on defining the concept of Meaningful Human Control in Autonomous (Weapon) Systems (Ekelhof, 2015; Horowitz & Scharre, 2015; Mecacci & Santoni De Sio, 2019; Santoni de Sio & Van den Hoven, 2018). In recent years, other scholars have been building on this work by operationalising the concept of Meaningful Human Control (see section 7.1. for emerging insights on operationalising Meaningful Human Control). Amoroso & Tamburrini (2021) bridge the gap between weapon usage and ethical principles based on ‘if-then’ rules, Umbrello (2021) proposes two Levels of Abstraction in which different agents have different levels of control over the decision-making process to deploy an Autonomous Weapon System,
20 CHAPTER 1 1 and Cavalcante Siebert et al. (2023), who build on the two necessary conditions for Meaningful Human Control- tracking and tracing – distinct by Santoni de Sio & Van den Hoven (2018), to create actional properties for the design of AI systems in which each of the properties human and artificial agents interact. In their reflection on their work the authors highlight that ‘Meaningful human control is necessary but not sufficient for ethical AI.’ (Cavalcante Siebert et al., 2023, p. 252). The authors amplify this by stating that for a human-AI system to align with societal values and norms, Meaningful Human Control must entail a larger set design objectives which can be achieved by transdisciplinary practices. In our opinion, Meaningful Human Control alone will not suffice as requirement to minimize unintended consequences of Autonomous Weapon Systems due to several reasons. Firstly, the concept of Meaningful Human Control is potentially controversial and confusing as human control is defined and understood differently in various literature domains (see section 2.11 for an overview of the concept of control in different domains). Secondly, standard concepts of control in engineering and the military domain entail a capacity to directly cause or prevent an outcome that is not possible to achieve with an Autonomous Weapon System, because once an autonomous weapon is launched you cannot intervene by human action. And finally, specific literature on Meaningful Human Control over Autonomous Weapon Systems does not offer a consistent usable concept. We believe that a different approach is needed to minimize unintended consequences of Autonomous Weapons Systems. Therefore, we propose an additional perspective that focusses on human oversight instead of Meaningful Human Control. Several scholars are describing the concept of human oversight in Autonomous Weapon Systems and AI in general. HRW and IHRC (2012) state that human oversight on robotic weapons is required to guarantee adequate protection of civilians in armed conflicts and they fear that when humans only retain a limited, or no, oversight role, that they could be fading out the decision-making loop. Taddeo and Floridi (2018) describe that human oversight procedures are necessary to minimize unintended consequences and to compensate unfair impacts of AI. The European Commission mentions Human Agency and Oversight as one of the Ethics Guidelines for Trustworthy AI (European Commission, 2019). However, current human oversight mechanisms are lacking effectiveness (HRW & IHRC, 2012) and might gradually erode to become meaningless or even impossible (Williams, 2015). Marchant et al. (2011) note that several governance mechanisms can be applied to achieve human oversight of Lethal Autonomous Robots. Oversight incorporates the governance mechanisms of institutions and is therefore broader than merely Meaningful Human Control. We propose a human oversight mechanism from a governance perspective to ensure accountability and responsibility in the deployment of Autonomous Weapon Systems in order to minimize unintended consequences. In the remainder of this chapter,
21 1 INTRODUCTION we will describe the research objectives, knowledge gaps and research questions that guide the development of a governance mechanism for human oversight. 1.1 RESEARCH OBJECTIVE To ensure accountability and responsibility, a mechanism is needed to oversee and supervise the deployment of Autonomous Weapon Systems. We propose an alternative view complementary to Meaningful Human Control that incorporates the social institutional and design dimension at a governance level. This alternative view provides stakeholders additional opportunities to ensure accountability and responsibility in the deployment of Autonomous Weapon Systems. While in recent years several scholars have been working on defining the concept of Meaningful Human Control, we have found that the concept of Human Oversight is not equally studied the in literature nor a framework or implementation concept for it is offered. Also, empirical studies on the elicitation of values related to Autonomous Weapons Systems, such as accountability and responsibility, and the extent of how accountability and responsibility as values are perceived during the deployment of Autonomous Weapon Systems by common people and experts are missing. Next to this, the values of accountability and responsibility are often used interchangeably in the debate on Autonomous Weapon Systems whilst being different subjects. As stated above, responsibility contributes to minimizing unintended consequences by anticipating on actions and unintended consequences to come and taking measures to prevent or mitigate them. Accountability on the other hand can decrease unintended consequences in providing information and explanations by actors of their previous actions in order for other actors to learn from them and prevent mistakes and unintended consequences of their own. This leads to the following problem statement for this research: A framework for Human Oversight is needed to ensure accountability in order to minimize unintended consequences of Autonomous Weapon Systems, but the current mechanisms for human oversight are lacking effectiveness. The concept of Meaningful Human Control will not suffice as requirement to ensure accountability in order to minimize the unintended consequences of this type of weapon system, because standard concepts of control in engineering and the military domain entail a capacity to directly cause or prevent an outcome that is not possible to achieve with an Autonomous Weapon System as once it is launched you cannot intervene by human action. Designing and implementing a framework for Human Oversight for Autonomous Weapon Systems enables proper allocation of accountability and responsibility in the deployment of Autonomous Weapon Systems.
22 CHAPTER 1 1 In taking a governance approach for ensuring accountability and responsibility in the deployment of Autonomous Weapon Systems follows a knowledge gap that is fourfold in that 1) a delineation of the values accountability and responsibility in the de debate on Autonomous Weapon Systems, 2) a theoretical account on the concept and mechanism for Human Oversight for Autonomous Weapon Systems, 3) an empirical study to elicit values and survey people’s perception on accountability and responsibility during the deployment of an Autonomous Weapon System, and 4) a framework and implementation concept to represent criteria for Human Oversight for Autonomous Weapon Systems are lacking. These knowledge gaps can be filled by analysing the values of accountability, responsibility and the concept of Human Oversight, conducting a value elicitation study and by designing a framework and implementation concept for Human Oversight over Autonomous Weapons Systems. This leads to the following research objective: To improve the allocation of accountability and responsibility by designing a framework and implementation concept such that criteria for Human Oversight are identified, represented and validated in order to minimize unintended consequences in the deployment of Autonomous Weapon Systems. To fulfil this research objective the following research questions need to be answered: Q1 What are Autonomous Weapon Systems and how are the values of accountability and responsibility related to the concerns for the deployment of Autonomous Weapon Systems? Q2 How should the values of accountability, responsibility and the concept of Human Oversight be characterized? Q3 Which control mechanisms are described in literature and present in the military domain, and which gaps in control mechanisms can be identified by the introduction of Autonomous Weapon Systems? Q4 To what extent can an empirical study be used to elicit values and how does this lead to changes in perception of the values accountability and responsibility in a scenario of Autonomous Weapon System deployment? Q5 To what extent can Human Oversight be translated into observable criteria for the deployment of Autonomous Weapon Systems? Q6 To what extent can observable criteria for Human Oversight be incorporated in an implementation concept for the deployment of Autonomous Weapon Systems? Scientific and societal relevance The scientific contribution of our research is twofold in that (1) our research contributes to a delineation of accountability, responsibility and Human Oversight that adds to the current body of literature, and (2) the framework and implementation concept for Human
23 1 INTRODUCTION Oversight for Autonomous Weapon Systems might also be applied to other AI fields to ensure accountability of other Autonomous Systems, such as those for Autonomous Vehicles or in the medical domain. The societal contribution of our research is a framework and implementation concept for Human Oversight that would lead to a proper allocation of accountability in the decision-making of the deployment of an Autonomous Weapon System. By identifying the supervisor of these actions, it might be possible to attribute responsibility for the actions taken by the weapon system. This contributes to decreasing the likelihood of unintended consequences in the deployment of Autonomous Weapon Systems. Scope Much of the literature in the academic and societal debate on Autonomous Weapon Systems is written and discussed by legal experts and philosophers in the context of International Humanitarian Law and the Geneva Conventions which are aimed to limit the effects of armed conflicts (ICRC, 2010). As we are no legal experts, this research will stay within the boundaries and rules of the Laws of Armed Conflict (LOAC) as currently defined in the mainstream literature and we will not question these. As with any weapon system, LOAC also applies to Autonomous Weapon Systems. Furthermore, this research will focus on the deployment of Autonomous Weapon Systems in the near future, which we define as: within the next 15 years. This entails that we will not study weapons equipped with Artificial General Intelligence or futuristic technology that is not possible to construct yet, but we focus on technology that is currently being developed. In this study, we will take a broad perspective on Autonomous Weapon Systems and will not limit to a specific type of weapon, like autonomous drones, but also consider types such as Autonomous Weapon Systems in the cyber domain and as part of a network of systems. 1.2 SCENARIO In the interest of clarity and consistency, the same scenario will be used in the different phases of this research. This scenario describes a threat to soldiers which could occur during military road clearing operations to find and clear improvised explosive devices. The technology (the facial and image recognition software for people and different preprogrammed options to engage) that is described for the Autonomous Weapon Systems exists separately, but is as far as we know not yet incorporated in a deployed Autonomous Weapon System. However, due to the technological advances we expect that these technological features are possible in the near future which makes this a realistic scenario.
24 CHAPTER 1 1 We have chosen to base the scenario on Airborne drones because these systems are being deployed in current conflicts, for example in Ukraine and Gaza, whilst unmanned ground-based systems are primarily in a testing phase and, as far as we know, not yet widely deployed on a battlefield. Underwater unmanned systems are also being used in current conflicts, but due to the underwater environment and lack of people in the vicinity of the system the risk on collateral damage is minimal. The scenario reads as follows: An Autonomous Weapon System provides force protection for soldiers that are clearing the road from improvised explosive devices. The Autonomous Weapon System is equipped with surveillance equipment, weapons (airto-ground missiles) and flies autonomously in the Area of Operation. It is programmed to avoid flying over a restricted operating zone and an electronic warfare threat. The Autonomous Weapon System is equipped with facial and image recognition software for people, weapons and explosives. It is programmed with different options to engage when it recognizes a threat to the soldiers that are clearing the road. The Autonomous Weapon System detects movement behind a large rock near a narrow part of the road at a distance of 300 meters of the road clearance soldiers. 1.3 RESEARCH APPROACH In this research, we apply the Value-Sensitive Design (VSD) method as research approach. The VSD is a three-partite approach that allows for considering human values throughout the design process of technology (Figure 1). It is an iterative process for the conceptual, empirical and technological investigation of human values implicated by the design (Davis & Nathan, 2015; Friedman & Kahn Jr, 2003). The conceptual investigation consists of two parts: (1) identifying the direct stakeholders, i.e. those who will use the technology, and the indirect stakeholders, i.e. those whose lives are influenced by the technology, and (2) identifying and defining the values that the use of the technology implicates. The empirical investigation looks into the understanding and experience of the stakeholders in a context relating to the technology and implicated values will be examined. In the technical investigation, the specific features of the technology are analysed (Davis & Nathan, 2015). The VSD can be used as a roadmap for engineers and students to incorporate ethical considerations into the design (Cummings, 2006). There has been some critique voiced regarding the VSD approach. One of the concerns Davis and Nathan (2015) mention is that the VSD posits that certain values are universal,
25 1 INTRODUCTION but that these may differ based on culture and context. A response to counter this would be to take an empirical basis for one’s viewpoint instead of a philosophical one, or acknowledge that the researcher’s position is not the only valid position to be considered (Borning & Muller, 2012). Borning and Muller (2012) pose a pluralistic position in that the VSD should not recommend either a universal or a relative view on values, but it should leave engineers free to decide which view is most appropriate in context of their design. Also, while moral values can and do differ across cultures, some values guiding basic principles of international law – e.g. human rights and protection of civilians- have been formally endorsed by countries with different histories and cultures (ICRC, 2010). In line with Borning and Muller (2012) we used the VSD approach in our research as guidance and not as a goal in itself. In the conceptual phase, we slightly deviate from the original VSD method, because we do not conduct a full stakeholder analysis to identify the stakeholders in the conceptual investigation phase, but we focus on the obvious stakeholder groups; military, policymakers, industry and Non-Governmental Organisations (NGO’s). For the identification of values in step 2, we used our previous work (Verdiesen, Santoni de Sio, & Dignum, 2019) in which we researched the values related to Autonomous Weapon Systems. In the technical investigation phase, we do not design an Autonomous Weapon System as one intuitively might expect, because this would be an immense project well beyond the scope of this research. Yet, we used a discrete-event modelling language (Coloured Petrinets (CPNs)) for modelling synchronisation concurrency and communication processes. We created a model that represents observable criteria of a pre-flight mission planning and post-flight mission evaluation process for autonomous drones. Figure 1: VSD (as in: Umbrello & Van de Poel, 2021)
26 CHAPTER 1 1 1.4 OUTLINE OF THESIS This thesis consists of five parts of which the introduction is part I. The remainder of this thesis is structured according to the phases of the Value-Sensitive Design approach and reads as follows: Part II: Conceptual investigation phase In chapter 2 the relevant literature on decision-making processes in AI, architectures for ethical decision-making in AI, autonomy, Autonomous Weapon Systems, values, values related to Autonomous Weapon Systems and value hierarchy as a Design for Values approach is reviewed. Parts of this chapter have been published in Verdiesen (2017) and Verdiesen, De Sio, and Dignum (2019). In chapter 3 we present the Comprehensive Human Oversight Framework by describing the layers and the connections between them and identifying gaps in the control mechanisms. To mitigate these gaps, we applied the Glass Box framework on the Comprehensive Human Oversight Framework. We conclude chapter 3 by closing the gap from the review stage back to the interpretation stage by means of a feedback process. Parts of this chapter have been published in Verdiesen, De Sio, and Dignum (2019) and Verdiesen, Aler Tubella, and Dignum (2021). Part III: Empirical investigation phase In chapter 4 we describe the empirical investigation phase of our research which consists of conducting expert interviews, the Value Deliberation Process as a means to elicitate values and validating the results by consulting experts. For reflection and validation, we discussed the Comphrensive Human Oversight Framework and aspects of drone deployments during interviews and an extra round of validation was conducted by inviting experts- who had not been part of the expert panel- to reflect on the findings of the value elicitation. Parts of this chapter have been published in Verdiesen and Dignum (2022). Part IV: Technical investigation phase In chapter 5 we present the implementation concept for operationalising the Glass Box framework. After introducing the scenario, we describe Coloured Petri Nets: a discrete-event language for modelling synchronisation concurrency and communication processes that we used to model the implementation concept. We conclude with remarks on validating the implementation concept. Parts of this chapter have been published in Verdiesen, Aler Tubella, and Dignum (2021).
27 1 INTRODUCTION Part V: Conclusion and discussion In chapter 6 we follow the three phases of our research approach to answer our research questions based on the results of our research to conclude if our research objective - to improve the allocation of accountability and responsibility in the deployment of Autonomous Weapon Systems by designing a framework and implementation concept such that the criteria for Human Oversight are identified, represented and validated- is reached. Chapter 7 contains the discussion on our research in which we highlight the emerged insights over the past five years on the definition of Autonomous Weapon Systems and the operationalisation of Meaningful Human Control, followed by the limitations of this research and suggestions for future work. We conclude this chapter by presenting the contributions and recommendations of our research.
This part on the conceptual investigation phase of our research consists of two chapters; 1) we review relevant literature on decision-making processes in AI, architectures for ethical decision-making in AI, autonomy, Autonomous Weapon Systems, values, values related to Autonomous Weapon Systems and value hierarchy in the Design for Values approach and 2) we present the Comprehensive Human Oversight Framework by describing the layers and the connections between them, identifying gaps in the control mechanisms and describe a feedback process to close the gaps. Parts of chapter 2 has been published in Verdiesen (2017) and Verdiesen et al. (2019). CONCEPTUAL INVESTIGATION PHASE Part II
In this chapter we review relevant literature on decision-making processes in AI, architectures for ethical decision-making in AI, autonomy, Autonomous Weapon Systems, values, values related to Autonomous Weapon Systems, a value hierarchy as a Design for Values approach, responsibility, accountability and accountability gaps, perspectives on control and human oversight. Extensive literature review 2|
32 2 CHAPTER 2 2.1 DECISION-MAKING PROCESSES IN AI Decision-making processes in Artificial Intelligence (AI) have been studied for over two decades and is quite well delineated in AI and engineering literature (see Table 1 in appendix C for an overview). Decision-making is defined as a process in which: ‘an entity is in a situation, receives information about that situation, and selects and then implements a course of action.’ (Miller, Wolf, & Grodzinsky, 2017, p. 390). Adams (2001) noticed as early as 2001 that the role of the human changed from being an active controller to that of a supervisor, and that direct human participation in decisions of AI systems would become rare. The concept of adjustable autonomy, i.e., switching between autonomy levels, is mentioned often in literature to deal with changes in context, the need of the operator and the control humans can exert over the machine (Cordeschi, 2013; Côté, Bouzid, & Mouaddib, 2011; van der Vecht, 2009). As is noted by Cordeschi (2013), optimal choices in decision-making for humans and AI do not exist, therefore only satisficing choices can be made. It depends on the situation if humans or AI can make the most reliable decision. In order for an AI system to be able to make ethical decisions it is not necessary that its decision-making is similar to that of a human, but the system will need a mechanism such as a heuristic algorithm to analyse its past decisions and prepare for future decisions (Miller et al., 2017). However, moving from a technical debate to an ethical point of view, according to Kramer, Borg, Conitzer, and Sinnott-Armstrong (2017), the question is not only if we can build moral decision-making in AI, but also if ‘moral AI’ systems should be permitted at all to make decisions. While this is certainly an important question, it is interesting to note that, as a matter of fact, people’s moral intuitions about this issue appears to be highly dependent on their acquaintance with computers. It seems that the more people are familiar with computers, the more they prefer decisions made by computers over decisions made by humans (Araujo, Helberger, Kruikemeier, & De Vreese, 2020; Kramer, Borg, Conitzer, & Sinnott-Armstrong, 2017). Araujo et al. (2020) found that for high impact decisions, the potential fairness, usefulness and risk of specific decision-making automatically by AI compared to human experts was often on par or even better evaluated. Based on their research, Kramer et al. (2017) expect that the more people gain experience with computer decision-making and it becomes more visible, the more it will be accepted by the general public.
33 2 EXTENSIVE LITERATURE REVIEW 2.2 ARCHITECTURES FOR ETHICAL DECISION-MAKING IN AI When computer programs of autonomous systems are implemented in the unpredictable real-world, the behaviour of these systems becomes non-deterministic and a range of possible outcomes can occur (Dennis, Fisher, Slavkovik, & Webster, 2016). To govern these unpredictable outcomes of autonomous systems in real-world scenarios, a mechanism is needed to influence the agent’s (ethical) decision-making. In engineering literature, two types of architectures for ethical decision-making of AI can be found (see Table 2 in appendix C for an overview). The first is based on an ‘ethical layer’ that governs the behaviour of the agent from outside the system. Arkin, Ulam, and Wagner (2012) designed and implemented an ‘ethical governor’ that consists of 2 processes; 1) ethical reasoning that transforms incoming perceptual, motor and situational awareness data into evidence, and 2) constraint application that uses the evidence to apply constraints based on Laws Of War and Rules Of Engagement to suppress unethical behaviour when applying lethal force. Dennis et al. (2016) proposes a hybrid architecture in which reasoning is done by a rational BDI [Beliefs, Desires and Intentions] agent. Based on this framework the agent selects plans from a given ethical policy which is the most ethical plan available based on its beliefs. Earlier work by Li et al. (2002) consists of a hierarchical control scheme developed to enable multiple Unmanned Combat Air Vehicles (UCAVs) to autonomously achieve demanding missions in hostile environments. The scheme consists of four layers: 1) a high-level path planner, 2) a low-level path planner, 3) a trajectory generator and 4) a formation control algorithm. More recently, Vanderelst and Winfield (2018) designed an additional or substitute framework for implementing robotic ethics as alternative for logic-based AI that currently dominates the field. They implemented ethical behaviour in robots by simulation theory of cognition in which internal simulations for actions and prediction of consequences are used to make ethical decisions. The method is a form of robot imagery and does not make use of verification of logical statements that is often used to check if actions are in accordance with ethical principles. The second type of architecture for ethical decision-making of AI is logic based. This type derives logical rules from natural language and applies the rules to the system to govern its ethical behaviour. Anderson, Anderson, and Berenz (2016) describe a case-supported principle-based behavior paradigm (CPB) to govern an elderly care robot’s behaviour. The system uses principles, that are abstracted from cases, that have consensus of ethicists, to choose its next action. It sorts the actions by weighing them according to ethical preferences, which are based on values, and selects the action that is highest ranked. Another formal approach is HERA (Hybrid Ethical Reasoning Agents) which is a software library to model autonomous moral decision-making (Lindner,
34 2 CHAPTER 2 Bentzen, & Nebel, 2017). HERA represents the robot’s possible actions together with the causal chains of consequences the actions initiate. Logical formulae are used to model ethical principles. The software library implements several ethical principles or interpretation of ethical principles, such as the principle of Double Effect, utilitarianism and a Pareto-inspired principle. The applied format is called a causal agency model. It reduces determining moral permissibility by checking if principle-specific logical formulae are satisfied in a causal agency model. Recent work of Bonnemains, Saurel, and Tessier (2018) demonstrates a formal approach is developed to link ethics and automated reasoning in autonomous systems. The formal tool models ethical principles to compute a judgement of possible decisions in a certain situation and explains why this decision is ethically acceptable or not. The formal model can be used on utilitarian and deontological ethics and the Doctrine of Double effect to examine the results generated by these three different ethical frameworks. They found that the main challenge lies in formalizing philosophical definitions in natural language and to translate them in generic computer programmable concepts that can be easily understood and that allows for ethical decisions to be explained. 2.3 AUTONOMY The notion of autonomy is a not well-defined and often misunderstood concept. Nowadays in the context of AI, autonomy is often a synonym for Machine Learning, an example can be found in Melancon (2020), but autonomy encompasses much more than that. Castelfranchi and Falcone (2003) define autonomy as a notion that involves relationships between three entities: a) the main subject x, b) the goal μ that must be obtained by the main subject x and c) a second subject γ upon the main subject x is autonomous. This is expressed in the statement: “x is autonomous about μ with respect to y”. For example, if x is an autonomous drone, its autonomy implies that the autonomous drone x can autonomously decide on the travel route (the goal μ) given a destination (i.e. GPS coordinates) set by its operator γ. Three type of autonomy relationships can be identified based on this description: (1) executive autonomy; x is autonomous in its means instead of it goals, which is the case of the example of the autonomous drone, (2) goal autonomy; x can set its goals on its own, and (3) social autonomy; x can execute its goals by itself without other agents (Castelfranchi & Falcone, 2003). Wooldridge and Jennings (1995, p. 116) also refer to autonomy in their list of four properties for defining an agent: ‘1) autonomy: agents operate without the direct intervention of humans or others, and have some kind of control over their actions and internal state (Castelfranchi, 1995), 2) social ability: agents interact with other agents (and possibly humans) via some kind of agent-communication language (Genesereth
35 2 EXTENSIVE LITERATURE REVIEW & Ketchpel, 1994), 3) reactivity: agents perceive their environment (which may be the physical world, a user via a graphical user interface, a collection of other agents, the Internet, or perhaps all of these combined), and respond in a timely fashion to changes that occur in it; and 4) pro-activeness: agents do not simply act in response to their environment, they are able to exhibit goal-directed behaviour by taking the initiative.’ In their article on defining Autonomous Weapon Systems, Taddeo and Blanchard (2022) delineate and specify the difference between automatic/automated and autonomous agents. They state: ‘The ability of an artificial agent to change its internal states without the direct intervention of another agent marks (binarily) the line between automatic/ automated and autonomous. A rule-based artificial system and a learning one both qualify as autonomous following this criterion.’(Taddeo & Blanchard, 2022, p. 17). An automated system on the other hand can perform a complex and a predetermined task. A robot in a car manufacturing factory is an example of an automated system. The authors also state that it is increasingly more common that adaptability is a key characteristic for Autonomous Weapon Systems, which will be their potential to deal with complex and fast pacing scenarios, but also will also cause unpredictability, lack of control and transparency, and responsibility gaps (Taddeo & Blanchard, 2022). Taddeo & Blanchard (2022) base their delineation on the work of Floridi and Sanders (2004) who describe three criteria for intelligent systems: ‘(a) Interactivity means that the agent and its environment (can) act upon each other. Typical examples include input or output of a value, or simultaneous engagement of an action by both agent and patient – for example gravitational force between bodies. (b) Autonomy means that the agent is able to change state without direct response to interaction: it can perform internal transitions to change its state. So an agent must have at least two states. This property imbues an agent with a certain degree of complexity and independence from its environment. (c) Adaptability means that the agent’s interactions (can) change the transition rules by which it changes state. This property ensures that an agent might be viewed, at the given LoA [Level of Abstraction], as learning its own mode of operation in a way which depends critically on its experience. Note that if an agent’s transition rules are stored as part of its internal state, discernible at this LoA, then adaptability follows from the other two conditions.’ (Floridi & Sanders, 2004, pp. 357-358).
36 2 CHAPTER 2 Above, autonomy is described from an engineering perspective, but it can also be viewed from a human value perspective. For instance, in Bioethics, which describes the values that are important as guiding principles in the medical field, autonomy is defined as acting intentionally without controlling influences that would mitigate against a voluntary act (Beauchamp and Walters, 1999). The definition of autonomy in the field of AI should be kept distinct from the definition of human autonomy and its moral value, because they do not represent the same constructs. Although autonomy is an important human value which will be useful in the next section, it is less relevant from an engineering perspective to interpret autonomy as a singular construct for a technical system, because weapon systems may comprise of different levels of autonomy. But even in the case of a “fully Autonomous Weapon System”, ‘[…] that, without human intervention, selects and engages targets matching certain predefined criteria, following a human decision to deploy the weapon on the understanding that an attack, once launched, cannot be stopped by human intervention.’ (AIV & CAVV, 2015; Broeks et al., 2021) the type of autonomy can at most be executive autonomy, because a human will set its goals and the weapon will not decide on its goals or deployment itself. Also, the context will constrain the autonomy of a “fully Autonomous Weapon System” as autonomous systems are created with task goals and boundary conditions (Bradshaw, Hoffman, Woods, & Johnson, 2013). In case of Autonomous Weapon Systems, the context might include physical limitations to the area of operations, for example the presence, or lack of, civilians in the land, sea, cyber, air or space domain. In the next section, several definitions of Autonomous Weapons Systems will be provided and the rationale for choosing the definition of the AIV and CAVV (2015) mentioned above is given. 2.4 AUTONOMOUS WEAPON SYSTEMS Although the societal and academic debate on Autonomous Weapon Systems has drawn a lot of attention in the recent years, we found that the topic was not well delineated in the academic literature. We start this subsection with an overview of the many different definitions and present two classifications of Autonomous Weapon Systems to conclude this section. Definition Autonomous Weapon Systems are an emerging technology and there is still no internationally agreed upon definition (AIV & CAVV, 2015; Sayler, 2021). Even consensus if Autonomous Weapon Systems should be defined at all is lacking. Although some scholars provide definitions in their writings (see Table 3 in appendix C), others caution against such a specification. NATO states that: ‘Attempting to create definitions for “autonomous systems” should be avoided, because by definition, machines cannot be autonomous in
37 2 EXTENSIVE LITERATURE REVIEW a literal sense. Machines are only “autonomous” with respect to certain functions such as navigation, sensor optimization, or fuel management.’ (Kuptel & Williams, 2014, p. 10). The United Nations Institute for Disarmament Research (UNDIR) is also cautious about providing a definition of Autonomous Weapon Systems, because they argue that the level of autonomy depends on the ‘critical functions of concern and the interactions of different variables’ (UNDIR, 2014, p. 5). They state that one of the reasons for the differentiation of terms regarding Autonomous Weapon Systems is that sometimes things (drones or robots) are defined, but in other times a characteristic (autonomy), variables of concern (lethality or degree of human control) or usage (targeting or defensive measures) are drawn into the discussion and become part of the definition. In a recent paper, Taddeo and Blanchard (2022) describe twelve definitions of (Lethal) Autonomous Weapon Systems provided by States and international organisations. They provide a value neutral definition of Autonomous Weapon Systems of their own (see Table 3 in appendix C). Various definitions of Autonomous Weapon Systems are listed in Table 3 in appendix C. Some authors use the term military robots which have a certain level of autonomy. As military robots can be viewed as a subclass of Autonomous Weapon systems according to the classification of Royakkers and Orbons (2015) (Figure 2) we included them in the list of definitions. In our opinion the definition in the report of the ADVISORY COUNCIL ON INTERNATIONAL AFFAIRS (AIV & CAVV) captures the description of Autonomous Weapon Systems best from an engineering and military standpoint, because it takes predefined criteria into account and is linked to the military targeting process as the weapon will only be deployed after a human decision. In their 2021 report on Autonomous Weapons Systems the AIV & CAVV continue to use this definition (Broeks et al., 2021). Therefore, we will follow this definition and define Autonomous Weapon Systems as: ‘A weapon that, without human intervention, selects and engages targets matching certain predefined criteria, following a human decision to deploy the weapon on the understanding that an attack, once launched, cannot be stopped by human intervention.’(AIV & CAVV, 2015, p. 11; Broeks et al., 2021, p. 11). Classification of Autonomous Weapon Systems Not only are Autonomous Weapon Systems ambiguously defined, they also have not been uniformly classified. We present two classifications in this subsection. Royakkers and Orbons (2015) describe several types of Autonomous Weapon Systems (Figure 2) distinct between (1) Non-Lethal Weapons which are weapons ‘…without causing (innocent) casualties or serious and permanent harm to people.’ (Royakkers & Orbons, 2015, p. 617), such as the Active Denial System which uses a beam of electromagnetic
www.gildeprint.nlRkJQdWJsaXNoZXIy MjY0ODMw