research papers on ai ethics

IEEE Account

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

• Open access
• Published: 18 April 2024

Research ethics and artificial intelligence for global health: perspectives from the global forum on bioethics in research

• James Shaw 1 , 13 ,
• Joseph Ali 2 , 3 ,
• Caesar A. Atuire 4 , 5 ,
• Phaik Yeong Cheah 6 ,
• Armando Guio Español 7 ,
• Judy Wawira Gichoya 8 ,
• Adrienne Hunt 9 ,
• Daudi Jjingo 10 ,
• Katherine Littler 9 ,
• Daniela Paolotti 11 &
• Effy Vayena 12  

BMC Medical Ethics volume  25 , Article number:  46 ( 2024 ) Cite this article

3540 Accesses

2 Citations

8 Altmetric

Metrics details

The ethical governance of Artificial Intelligence (AI) in health care and public health continues to be an urgent issue for attention in policy, research, and practice. In this paper we report on central themes related to challenges and strategies for promoting ethics in research involving AI in global health, arising from the Global Forum on Bioethics in Research (GFBR), held in Cape Town, South Africa in November 2022.

The GFBR is an annual meeting organized by the World Health Organization and supported by the Wellcome Trust, the US National Institutes of Health, the UK Medical Research Council (MRC) and the South African MRC. The forum aims to bring together ethicists, researchers, policymakers, research ethics committee members and other actors to engage with challenges and opportunities specifically related to research ethics. In 2022 the focus of the GFBR was “Ethics of AI in Global Health Research”. The forum consisted of 6 case study presentations, 16 governance presentations, and a series of small group and large group discussions. A total of 87 participants attended the forum from 31 countries around the world, representing disciplines of bioethics, AI, health policy, health professional practice, research funding, and bioinformatics. In this paper, we highlight central insights arising from GFBR 2022.

We describe the significance of four thematic insights arising from the forum: (1) Appropriateness of building AI, (2) Transferability of AI systems, (3) Accountability for AI decision-making and outcomes, and (4) Individual consent. We then describe eight recommendations for governance leaders to enhance the ethical governance of AI in global health research, addressing issues such as AI impact assessments, environmental values, and fair partnerships.

Conclusions

The 2022 Global Forum on Bioethics in Research illustrated several innovations in ethical governance of AI for global health research, as well as several areas in need of urgent attention internationally. This summary is intended to inform international and domestic efforts to strengthen research ethics and support the evolution of governance leadership to meet the demands of AI in global health research.

Peer Review reports

Introduction

The ethical governance of Artificial Intelligence (AI) in health care and public health continues to be an urgent issue for attention in policy, research, and practice [ 1 , 2 , 3 ]. Beyond the growing number of AI applications being implemented in health care, capabilities of AI models such as Large Language Models (LLMs) expand the potential reach and significance of AI technologies across health-related fields [ 4 , 5 ]. Discussion about effective, ethical governance of AI technologies has spanned a range of governance approaches, including government regulation, organizational decision-making, professional self-regulation, and research ethics review [ 6 , 7 , 8 ]. In this paper, we report on central themes related to challenges and strategies for promoting ethics in research involving AI in global health research, arising from the Global Forum on Bioethics in Research (GFBR), held in Cape Town, South Africa in November 2022. Although applications of AI for research, health care, and public health are diverse and advancing rapidly, the insights generated at the forum remain highly relevant from a global health perspective. After summarizing important context for work in this domain, we highlight categories of ethical issues emphasized at the forum for attention from a research ethics perspective internationally. We then outline strategies proposed for research, innovation, and governance to support more ethical AI for global health.

In this paper, we adopt the definition of AI systems provided by the Organization for Economic Cooperation and Development (OECD) as our starting point. Their definition states that an AI system is “a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations, or decisions influencing real or virtual environments. AI systems are designed to operate with varying levels of autonomy” [ 9 ]. The conceptualization of an algorithm as helping to constitute an AI system, along with hardware, other elements of software, and a particular context of use, illustrates the wide variety of ways in which AI can be applied. We have found it useful to differentiate applications of AI in research as those classified as “AI systems for discovery” and “AI systems for intervention”. An AI system for discovery is one that is intended to generate new knowledge, for example in drug discovery or public health research in which researchers are seeking potential targets for intervention, innovation, or further research. An AI system for intervention is one that directly contributes to enacting an intervention in a particular context, for example informing decision-making at the point of care or assisting with accuracy in a surgical procedure.

The mandate of the GFBR is to take a broad view of what constitutes research and its regulation in global health, with special attention to bioethics in Low- and Middle- Income Countries. AI as a group of technologies demands such a broad view. AI development for health occurs in a variety of environments, including universities and academic health sciences centers where research ethics review remains an important element of the governance of science and innovation internationally [ 10 , 11 ]. In these settings, research ethics committees (RECs; also known by different names such as Institutional Review Boards or IRBs) make decisions about the ethical appropriateness of projects proposed by researchers and other institutional members, ultimately determining whether a given project is allowed to proceed on ethical grounds [ 12 ].

However, research involving AI for health also takes place in large corporations and smaller scale start-ups, which in some jurisdictions fall outside the scope of research ethics regulation. In the domain of AI, the question of what constitutes research also becomes blurred. For example, is the development of an algorithm itself considered a part of the research process? Or only when that algorithm is tested under the formal constraints of a systematic research methodology? In this paper we take an inclusive view, in which AI development is included in the definition of research activity and within scope for our inquiry, regardless of the setting in which it takes place. This broad perspective characterizes the approach to “research ethics” we take in this paper, extending beyond the work of RECs to include the ethical analysis of the wide range of activities that constitute research as the generation of new knowledge and intervention in the world.

Ethical governance of AI in global health

The ethical governance of AI for global health has been widely discussed in recent years. The World Health Organization (WHO) released its guidelines on ethics and governance of AI for health in 2021, endorsing a set of six ethical principles and exploring the relevance of those principles through a variety of use cases. The WHO guidelines also provided an overview of AI governance, defining governance as covering “a range of steering and rule-making functions of governments and other decision-makers, including international health agencies, for the achievement of national health policy objectives conducive to universal health coverage.” (p. 81) The report usefully provided a series of recommendations related to governance of seven domains pertaining to AI for health: data, benefit sharing, the private sector, the public sector, regulation, policy observatories/model legislation, and global governance. The report acknowledges that much work is yet to be done to advance international cooperation on AI governance, especially related to prioritizing voices from Low- and Middle-Income Countries (LMICs) in global dialogue.

One important point emphasized in the WHO report that reinforces the broader literature on global governance of AI is the distribution of responsibility across a wide range of actors in the AI ecosystem. This is especially important to highlight when focused on research for global health, which is specifically about work that transcends national borders. Alami et al. (2020) discussed the unique risks raised by AI research in global health, ranging from the unavailability of data in many LMICs required to train locally relevant AI models to the capacity of health systems to absorb new AI technologies that demand the use of resources from elsewhere in the system. These observations illustrate the need to identify the unique issues posed by AI research for global health specifically, and the strategies that can be employed by all those implicated in AI governance to promote ethically responsible use of AI in global health research.

RECs and the regulation of research involving AI

RECs represent an important element of the governance of AI for global health research, and thus warrant further commentary as background to our paper. Despite the importance of RECs, foundational questions have been raised about their capabilities to accurately understand and address ethical issues raised by studies involving AI. Rahimzadeh et al. (2023) outlined how RECs in the United States are under-prepared to align with recent federal policy requiring that RECs review data sharing and management plans with attention to the unique ethical issues raised in AI research for health [ 13 ]. Similar research in South Africa identified variability in understanding of existing regulations and ethical issues associated with health-related big data sharing and management among research ethics committee members [ 14 , 15 ]. The effort to address harms accruing to groups or communities as opposed to individuals whose data are included in AI research has also been identified as a unique challenge for RECs [ 16 , 17 ]. Doerr and Meeder (2022) suggested that current regulatory frameworks for research ethics might actually prevent RECs from adequately addressing such issues, as they are deemed out of scope of REC review [ 16 ]. Furthermore, research in the United Kingdom and Canada has suggested that researchers using AI methods for health tend to distinguish between ethical issues and social impact of their research, adopting an overly narrow view of what constitutes ethical issues in their work [ 18 ].

The challenges for RECs in adequately addressing ethical issues in AI research for health care and public health exceed a straightforward survey of ethical considerations. As Ferretti et al. (2021) contend, some capabilities of RECs adequately cover certain issues in AI-based health research, such as the common occurrence of conflicts of interest where researchers who accept funds from commercial technology providers are implicitly incentivized to produce results that align with commercial interests [ 12 ]. However, some features of REC review require reform to adequately meet ethical needs. Ferretti et al. outlined weaknesses of RECs that are longstanding and those that are novel to AI-related projects, proposing a series of directions for development that are regulatory, procedural, and complementary to REC functionality. The work required on a global scale to update the REC function in response to the demands of research involving AI is substantial.

These issues take greater urgency in the context of global health [ 19 ]. Teixeira da Silva (2022) described the global practice of “ethics dumping”, where researchers from high income countries bring ethically contentious practices to RECs in low-income countries as a strategy to gain approval and move projects forward [ 20 ]. Although not yet systematically documented in AI research for health, risk of ethics dumping in AI research is high. Evidence is already emerging of practices of “health data colonialism”, in which AI researchers and developers from large organizations in high-income countries acquire data to build algorithms in LMICs to avoid stricter regulations [ 21 ]. This specific practice is part of a larger collection of practices that characterize health data colonialism, involving the broader exploitation of data and the populations they represent primarily for commercial gain [ 21 , 22 ]. As an additional complication, AI algorithms trained on data from high-income contexts are unlikely to apply in straightforward ways to LMIC settings [ 21 , 23 ]. In the context of global health, there is widespread acknowledgement about the need to not only enhance the knowledge base of REC members about AI-based methods internationally, but to acknowledge the broader shifts required to encourage their capabilities to more fully address these and other ethical issues associated with AI research for health [ 8 ].

Although RECs are an important part of the story of the ethical governance of AI for global health research, they are not the only part. The responsibilities of supra-national entities such as the World Health Organization, national governments, organizational leaders, commercial AI technology providers, health care professionals, and other groups continue to be worked out internationally. In this context of ongoing work, examining issues that demand attention and strategies to address them remains an urgent and valuable task.

The GFBR is an annual meeting organized by the World Health Organization and supported by the Wellcome Trust, the US National Institutes of Health, the UK Medical Research Council (MRC) and the South African MRC. The forum aims to bring together ethicists, researchers, policymakers, REC members and other actors to engage with challenges and opportunities specifically related to research ethics. Each year the GFBR meeting includes a series of case studies and keynotes presented in plenary format to an audience of approximately 100 people who have applied and been competitively selected to attend, along with small-group breakout discussions to advance thinking on related issues. The specific topic of the forum changes each year, with past topics including ethical issues in research with people living with mental health conditions (2021), genome editing (2019), and biobanking/data sharing (2018). The forum is intended to remain grounded in the practical challenges of engaging in research ethics, with special interest in low resource settings from a global health perspective. A post-meeting fellowship scheme is open to all LMIC participants, providing a unique opportunity to apply for funding to further explore and address the ethical challenges that are identified during the meeting.

In 2022, the focus of the GFBR was “Ethics of AI in Global Health Research”. The forum consisted of 6 case study presentations (both short and long form) reporting on specific initiatives related to research ethics and AI for health, and 16 governance presentations (both short and long form) reporting on actual approaches to governing AI in different country settings. A keynote presentation from Professor Effy Vayena addressed the topic of the broader context for AI ethics in a rapidly evolving field. A total of 87 participants attended the forum from 31 countries around the world, representing disciplines of bioethics, AI, health policy, health professional practice, research funding, and bioinformatics. The 2-day forum addressed a wide range of themes. The conference report provides a detailed overview of each of the specific topics addressed while a policy paper outlines the cross-cutting themes (both documents are available at the GFBR website: https://www.gfbr.global/past-meetings/16th-forum-cape-town-south-africa-29-30-november-2022/ ). As opposed to providing a detailed summary in this paper, we aim to briefly highlight central issues raised, solutions proposed, and the challenges facing the research ethics community in the years to come.

In this way, our primary aim in this paper is to present a synthesis of the challenges and opportunities raised at the GFBR meeting and in the planning process, followed by our reflections as a group of authors on their significance for governance leaders in the coming years. We acknowledge that the views represented at the meeting and in our results are a partial representation of the universe of views on this topic; however, the GFBR leadership invested a great deal of resources in convening a deeply diverse and thoughtful group of researchers and practitioners working on themes of bioethics related to AI for global health including those based in LMICs. We contend that it remains rare to convene such a strong group for an extended time and believe that many of the challenges and opportunities raised demand attention for more ethical futures of AI for health. Nonetheless, our results are primarily descriptive and are thus not explicitly grounded in a normative argument. We make effort in the Discussion section to contextualize our results by describing their significance and connecting them to broader efforts to reform global health research and practice.

Uniquely important ethical issues for AI in global health research

Presentations and group dialogue over the course of the forum raised several issues for consideration, and here we describe four overarching themes for the ethical governance of AI in global health research. Brief descriptions of each issue can be found in Table  1 . Reports referred to throughout the paper are available at the GFBR website provided above.

The first overarching thematic issue relates to the appropriateness of building AI technologies in response to health-related challenges in the first place. Case study presentations referred to initiatives where AI technologies were highly appropriate, such as in ear shape biometric identification to more accurately link electronic health care records to individual patients in Zambia (Alinani Simukanga). Although important ethical issues were raised with respect to privacy, trust, and community engagement in this initiative, the AI-based solution was appropriately matched to the challenge of accurately linking electronic records to specific patient identities. In contrast, forum participants raised questions about the appropriateness of an initiative using AI to improve the quality of handwashing practices in an acute care hospital in India (Niyoshi Shah), which led to gaming the algorithm. Overall, participants acknowledged the dangers of techno-solutionism, in which AI researchers and developers treat AI technologies as the most obvious solutions to problems that in actuality demand much more complex strategies to address [ 24 ]. However, forum participants agreed that RECs in different contexts have differing degrees of power to raise issues of the appropriateness of an AI-based intervention.

The second overarching thematic issue related to whether and how AI-based systems transfer from one national health context to another. One central issue raised by a number of case study presentations related to the challenges of validating an algorithm with data collected in a local environment. For example, one case study presentation described a project that would involve the collection of personally identifiable data for sensitive group identities, such as tribe, clan, or religion, in the jurisdictions involved (South Africa, Nigeria, Tanzania, Uganda and the US; Gakii Masunga). Doing so would enable the team to ensure that those groups were adequately represented in the dataset to ensure the resulting algorithm was not biased against specific community groups when deployed in that context. However, some members of these communities might desire to be represented in the dataset, whereas others might not, illustrating the need to balance autonomy and inclusivity. It was also widely recognized that collecting these data is an immense challenge, particularly when historically oppressive practices have led to a low-trust environment for international organizations and the technologies they produce. It is important to note that in some countries such as South Africa and Rwanda, it is illegal to collect information such as race and tribal identities, re-emphasizing the importance for cultural awareness and avoiding “one size fits all” solutions.

The third overarching thematic issue is related to understanding accountabilities for both the impacts of AI technologies and governance decision-making regarding their use. Where global health research involving AI leads to longer-term harms that might fall outside the usual scope of issues considered by a REC, who is to be held accountable, and how? This question was raised as one that requires much further attention, with law being mixed internationally regarding the mechanisms available to hold researchers, innovators, and their institutions accountable over the longer term. However, it was recognized in breakout group discussion that many jurisdictions are developing strong data protection regimes related specifically to international collaboration for research involving health data. For example, Kenya’s Data Protection Act requires that any internationally funded projects have a local principal investigator who will hold accountability for how data are shared and used [ 25 ]. The issue of research partnerships with commercial entities was raised by many participants in the context of accountability, pointing toward the urgent need for clear principles related to strategies for engagement with commercial technology companies in global health research.

The fourth and final overarching thematic issue raised here is that of consent. The issue of consent was framed by the widely shared recognition that models of individual, explicit consent might not produce a supportive environment for AI innovation that relies on the secondary uses of health-related datasets to build AI algorithms. Given this recognition, approaches such as community oversight of health data uses were suggested as a potential solution. However, the details of implementing such community oversight mechanisms require much further attention, particularly given the unique perspectives on health data in different country settings in global health research. Furthermore, some uses of health data do continue to require consent. One case study of South Africa, Nigeria, Kenya, Ethiopia and Uganda suggested that when health data are shared across borders, individual consent remains necessary when data is transferred from certain countries (Nezerith Cengiz). Broader clarity is necessary to support the ethical governance of health data uses for AI in global health research.

Recommendations for ethical governance of AI in global health research

Dialogue at the forum led to a range of suggestions for promoting ethical conduct of AI research for global health, related to the various roles of actors involved in the governance of AI research broadly defined. The strategies are written for actors we refer to as “governance leaders”, those people distributed throughout the AI for global health research ecosystem who are responsible for ensuring the ethical and socially responsible conduct of global health research involving AI (including researchers themselves). These include RECs, government regulators, health care leaders, health professionals, corporate social accountability officers, and others. Enacting these strategies would bolster the ethical governance of AI for global health more generally, enabling multiple actors to fulfill their roles related to governing research and development activities carried out across multiple organizations, including universities, academic health sciences centers, start-ups, and technology corporations. Specific suggestions are summarized in Table  2 .

First, forum participants suggested that governance leaders including RECs, should remain up to date on recent advances in the regulation of AI for health. Regulation of AI for health advances rapidly and takes on different forms in jurisdictions around the world. RECs play an important role in governance, but only a partial role; it was deemed important for RECs to acknowledge how they fit within a broader governance ecosystem in order to more effectively address the issues within their scope. Not only RECs but organizational leaders responsible for procurement, researchers, and commercial actors should all commit to efforts to remain up to date about the relevant approaches to regulating AI for health care and public health in jurisdictions internationally. In this way, governance can more adequately remain up to date with advances in regulation.

Second, forum participants suggested that governance leaders should focus on ethical governance of health data as a basis for ethical global health AI research. Health data are considered the foundation of AI development, being used to train AI algorithms for various uses [ 26 ]. By focusing on ethical governance of health data generation, sharing, and use, multiple actors will help to build an ethical foundation for AI development among global health researchers.

Third, forum participants believed that governance processes should incorporate AI impact assessments where appropriate. An AI impact assessment is the process of evaluating the potential effects, both positive and negative, of implementing an AI algorithm on individuals, society, and various stakeholders, generally over time frames specified in advance of implementation [ 27 ]. Although not all types of AI research in global health would warrant an AI impact assessment, this is especially relevant for those studies aiming to implement an AI system for intervention into health care or public health. Organizations such as RECs can use AI impact assessments to boost understanding of potential harms at the outset of a research project, encouraging researchers to more deeply consider potential harms in the development of their study.

Fourth, forum participants suggested that governance decisions should incorporate the use of environmental impact assessments, or at least the incorporation of environment values when assessing the potential impact of an AI system. An environmental impact assessment involves evaluating and anticipating the potential environmental effects of a proposed project to inform ethical decision-making that supports sustainability [ 28 ]. Although a relatively new consideration in research ethics conversations [ 29 ], the environmental impact of building technologies is a crucial consideration for the public health commitment to environmental sustainability. Governance leaders can use environmental impact assessments to boost understanding of potential environmental harms linked to AI research projects in global health over both the shorter and longer terms.

Fifth, forum participants suggested that governance leaders should require stronger transparency in the development of AI algorithms in global health research. Transparency was considered essential in the design and development of AI algorithms for global health to ensure ethical and accountable decision-making throughout the process. Furthermore, whether and how researchers have considered the unique contexts into which such algorithms may be deployed can be surfaced through stronger transparency, for example in describing what primary considerations were made at the outset of the project and which stakeholders were consulted along the way. Sharing information about data provenance and methods used in AI development will also enhance the trustworthiness of the AI-based research process.

Sixth, forum participants suggested that governance leaders can encourage or require community engagement at various points throughout an AI project. It was considered that engaging patients and communities is crucial in AI algorithm development to ensure that the technology aligns with community needs and values. However, participants acknowledged that this is not a straightforward process. Effective community engagement requires lengthy commitments to meeting with and hearing from diverse communities in a given setting, and demands a particular set of skills in communication and dialogue that are not possessed by all researchers. Encouraging AI researchers to begin this process early and build long-term partnerships with community members is a promising strategy to deepen community engagement in AI research for global health. One notable recommendation was that research funders have an opportunity to incentivize and enable community engagement with funds dedicated to these activities in AI research in global health.

Seventh, forum participants suggested that governance leaders can encourage researchers to build strong, fair partnerships between institutions and individuals across country settings. In a context of longstanding imbalances in geopolitical and economic power, fair partnerships in global health demand a priori commitments to share benefits related to advances in medical technologies, knowledge, and financial gains. Although enforcement of this point might be beyond the remit of RECs, commentary will encourage researchers to consider stronger, fairer partnerships in global health in the longer term.

Eighth, it became evident that it is necessary to explore new forms of regulatory experimentation given the complexity of regulating a technology of this nature. In addition, the health sector has a series of particularities that make it especially complicated to generate rules that have not been previously tested. Several participants highlighted the desire to promote spaces for experimentation such as regulatory sandboxes or innovation hubs in health. These spaces can have several benefits for addressing issues surrounding the regulation of AI in the health sector, such as: (i) increasing the capacities and knowledge of health authorities about this technology; (ii) identifying the major problems surrounding AI regulation in the health sector; (iii) establishing possibilities for exchange and learning with other authorities; (iv) promoting innovation and entrepreneurship in AI in health; and (vi) identifying the need to regulate AI in this sector and update other existing regulations.

Ninth and finally, forum participants believed that the capabilities of governance leaders need to evolve to better incorporate expertise related to AI in ways that make sense within a given jurisdiction. With respect to RECs, for example, it might not make sense for every REC to recruit a member with expertise in AI methods. Rather, it will make more sense in some jurisdictions to consult with members of the scientific community with expertise in AI when research protocols are submitted that demand such expertise. Furthermore, RECs and other approaches to research governance in jurisdictions around the world will need to evolve in order to adopt the suggestions outlined above, developing processes that apply specifically to the ethical governance of research using AI methods in global health.

Research involving the development and implementation of AI technologies continues to grow in global health, posing important challenges for ethical governance of AI in global health research around the world. In this paper we have summarized insights from the 2022 GFBR, focused specifically on issues in research ethics related to AI for global health research. We summarized four thematic challenges for governance related to AI in global health research and nine suggestions arising from presentations and dialogue at the forum. In this brief discussion section, we present an overarching observation about power imbalances that frames efforts to evolve the role of governance in global health research, and then outline two important opportunity areas as the field develops to meet the challenges of AI in global health research.

Dialogue about power is not unfamiliar in global health, especially given recent contributions exploring what it would mean to de-colonize global health research, funding, and practice [ 30 , 31 ]. Discussions of research ethics applied to AI research in global health contexts are deeply infused with power imbalances. The existing context of global health is one in which high-income countries primarily located in the “Global North” charitably invest in projects taking place primarily in the “Global South” while recouping knowledge, financial, and reputational benefits [ 32 ]. With respect to AI development in particular, recent examples of digital colonialism frame dialogue about global partnerships, raising attention to the role of large commercial entities and global financial capitalism in global health research [ 21 , 22 ]. Furthermore, the power of governance organizations such as RECs to intervene in the process of AI research in global health varies widely around the world, depending on the authorities assigned to them by domestic research governance policies. These observations frame the challenges outlined in our paper, highlighting the difficulties associated with making meaningful change in this field.

Despite these overarching challenges of the global health research context, there are clear strategies for progress in this domain. Firstly, AI innovation is rapidly evolving, which means approaches to the governance of AI for health are rapidly evolving too. Such rapid evolution presents an important opportunity for governance leaders to clarify their vision and influence over AI innovation in global health research, boosting the expertise, structure, and functionality required to meet the demands of research involving AI. Secondly, the research ethics community has strong international ties, linked to a global scholarly community that is committed to sharing insights and best practices around the world. This global community can be leveraged to coordinate efforts to produce advances in the capabilities and authorities of governance leaders to meaningfully govern AI research for global health given the challenges summarized in our paper.

Limitations

Our paper includes two specific limitations that we address explicitly here. First, it is still early in the lifetime of the development of applications of AI for use in global health, and as such, the global community has had limited opportunity to learn from experience. For example, there were many fewer case studies, which detail experiences with the actual implementation of an AI technology, submitted to GFBR 2022 for consideration than was expected. In contrast, there were many more governance reports submitted, which detail the processes and outputs of governance processes that anticipate the development and dissemination of AI technologies. This observation represents both a success and a challenge. It is a success that so many groups are engaging in anticipatory governance of AI technologies, exploring evidence of their likely impacts and governing technologies in novel and well-designed ways. It is a challenge that there is little experience to build upon of the successful implementation of AI technologies in ways that have limited harms while promoting innovation. Further experience with AI technologies in global health will contribute to revising and enhancing the challenges and recommendations we have outlined in our paper.

Second, global trends in the politics and economics of AI technologies are evolving rapidly. Although some nations are advancing detailed policy approaches to regulating AI more generally, including for uses in health care and public health, the impacts of corporate investments in AI and political responses related to governance remain to be seen. The excitement around large language models (LLMs) and large multimodal models (LMMs) has drawn deeper attention to the challenges of regulating AI in any general sense, opening dialogue about health sector-specific regulations. The direction of this global dialogue, strongly linked to high-profile corporate actors and multi-national governance institutions, will strongly influence the development of boundaries around what is possible for the ethical governance of AI for global health. We have written this paper at a point when these developments are proceeding rapidly, and as such, we acknowledge that our recommendations will need updating as the broader field evolves.

Ultimately, coordination and collaboration between many stakeholders in the research ethics ecosystem will be necessary to strengthen the ethical governance of AI in global health research. The 2022 GFBR illustrated several innovations in ethical governance of AI for global health research, as well as several areas in need of urgent attention internationally. This summary is intended to inform international and domestic efforts to strengthen research ethics and support the evolution of governance leadership to meet the demands of AI in global health research.

Data availability

All data and materials analyzed to produce this paper are available on the GFBR website: https://www.gfbr.global/past-meetings/16th-forum-cape-town-south-africa-29-30-november-2022/ .

Clark P, Kim J, Aphinyanaphongs Y, Marketing, Food US. Drug Administration Clearance of Artificial Intelligence and Machine Learning Enabled Software in and as Medical devices: a systematic review. JAMA Netw Open. 2023;6(7):e2321792–2321792.

Article   Google Scholar  

Potnis KC, Ross JS, Aneja S, Gross CP, Richman IB. Artificial intelligence in breast cancer screening: evaluation of FDA device regulation and future recommendations. JAMA Intern Med. 2022;182(12):1306–12.

Siala H, Wang Y. SHIFTing artificial intelligence to be responsible in healthcare: a systematic review. Soc Sci Med. 2022;296:114782.

Yang X, Chen A, PourNejatian N, Shin HC, Smith KE, Parisien C, et al. A large language model for electronic health records. NPJ Digit Med. 2022;5(1):194.

Meskó B, Topol EJ. The imperative for regulatory oversight of large language models (or generative AI) in healthcare. NPJ Digit Med. 2023;6(1):120.

Jobin A, Ienca M, Vayena E. The global landscape of AI ethics guidelines. Nat Mach Intell. 2019;1(9):389–99.

Minssen T, Vayena E, Cohen IG. The challenges for Regulating Medical Use of ChatGPT and other large Language models. JAMA. 2023.

Ho CWL, Malpani R. Scaling up the research ethics framework for healthcare machine learning as global health ethics and governance. Am J Bioeth. 2022;22(5):36–8.

Yeung K. Recommendation of the council on artificial intelligence (OECD). Int Leg Mater. 2020;59(1):27–34.

Maddox TM, Rumsfeld JS, Payne PR. Questions for artificial intelligence in health care. JAMA. 2019;321(1):31–2.

Dzau VJ, Balatbat CA, Ellaissi WF. Revisiting academic health sciences systems a decade later: discovery to health to population to society. Lancet. 2021;398(10318):2300–4.

Ferretti A, Ienca M, Sheehan M, Blasimme A, Dove ES, Farsides B, et al. Ethics review of big data research: what should stay and what should be reformed? BMC Med Ethics. 2021;22(1):1–13.

Rahimzadeh V, Serpico K, Gelinas L. Institutional review boards need new skills to review data sharing and management plans. Nat Med. 2023;1–3.

Kling S, Singh S, Burgess TL, Nair G. The role of an ethics advisory committee in data science research in sub-saharan Africa. South Afr J Sci. 2023;119(5–6):1–3.

Google Scholar  

Cengiz N, Kabanda SM, Esterhuizen TM, Moodley K. Exploring perspectives of research ethics committee members on the governance of big data in sub-saharan Africa. South Afr J Sci. 2023;119(5–6):1–9.

Doerr M, Meeder S. Big health data research and group harm: the scope of IRB review. Ethics Hum Res. 2022;44(4):34–8.

Ballantyne A, Stewart C. Big data and public-private partnerships in healthcare and research: the application of an ethics framework for big data in health and research. Asian Bioeth Rev. 2019;11(3):315–26.

Samuel G, Chubb J, Derrick G. Boundaries between research ethics and ethical research use in artificial intelligence health research. J Empir Res Hum Res Ethics. 2021;16(3):325–37.

Murphy K, Di Ruggiero E, Upshur R, Willison DJ, Malhotra N, Cai JC, et al. Artificial intelligence for good health: a scoping review of the ethics literature. BMC Med Ethics. 2021;22(1):1–17.

Teixeira da Silva JA. Handling ethics dumping and neo-colonial research: from the laboratory to the academic literature. J Bioethical Inq. 2022;19(3):433–43.

Ferryman K. The dangers of data colonialism in precision public health. Glob Policy. 2021;12:90–2.

Couldry N, Mejias UA. Data colonialism: rethinking big data’s relation to the contemporary subject. Telev New Media. 2019;20(4):336–49.

Organization WH. Ethics and governance of artificial intelligence for health: WHO guidance. 2021.

Metcalf J, Moss E. Owning ethics: corporate logics, silicon valley, and the institutionalization of ethics. Soc Res Int Q. 2019;86(2):449–76.

Data Protection Act - OFFICE OF THE DATA PROTECTION COMMISSIONER KENYA [Internet]. 2021 [cited 2023 Sep 30]. https://www.odpc.go.ke/dpa-act/ .

Sharon T, Lucivero F. Introduction to the special theme: the expansion of the health data ecosystem–rethinking data ethics and governance. Big Data & Society. Volume 6. London, England: SAGE Publications Sage UK; 2019. p. 2053951719852969.

Reisman D, Schultz J, Crawford K, Whittaker M. Algorithmic impact assessments: a practical Framework for Public Agency. AI Now. 2018.

Morgan RK. Environmental impact assessment: the state of the art. Impact Assess Proj Apprais. 2012;30(1):5–14.

Samuel G, Richie C. Reimagining research ethics to include environmental sustainability: a principled approach, including a case study of data-driven health research. J Med Ethics. 2023;49(6):428–33.

Kwete X, Tang K, Chen L, Ren R, Chen Q, Wu Z, et al. Decolonizing global health: what should be the target of this movement and where does it lead us? Glob Health Res Policy. 2022;7(1):3.

Abimbola S, Asthana S, Montenegro C, Guinto RR, Jumbam DT, Louskieter L, et al. Addressing power asymmetries in global health: imperatives in the wake of the COVID-19 pandemic. PLoS Med. 2021;18(4):e1003604.

Benatar S. Politics, power, poverty and global health: systems and frames. Int J Health Policy Manag. 2016;5(10):599.

Download references

Acknowledgements

We would like to acknowledge the outstanding contributions of the attendees of GFBR 2022 in Cape Town, South Africa. This paper is authored by members of the GFBR 2022 Planning Committee. We would like to acknowledge additional members Tamra Lysaght, National University of Singapore, and Niresh Bhagwandin, South African Medical Research Council, for their input during the planning stages and as reviewers of the applications to attend the Forum.

This work was supported by Wellcome [222525/Z/21/Z], the US National Institutes of Health, the UK Medical Research Council (part of UK Research and Innovation), and the South African Medical Research Council through funding to the Global Forum on Bioethics in Research.

Author information

Authors and affiliations.

Department of Physical Therapy, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada

Berman Institute of Bioethics, Johns Hopkins University, Baltimore, MD, USA

Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD, USA

Department of Philosophy and Classics, University of Ghana, Legon-Accra, Ghana

Caesar A. Atuire

Centre for Tropical Medicine and Global Health, Nuffield Department of Medicine, University of Oxford, Oxford, UK

Mahidol Oxford Tropical Medicine Research Unit, Faculty of Tropical Medicine, Mahidol University, Bangkok, Thailand

Phaik Yeong Cheah

Berkman Klein Center, Harvard University, Bogotá, Colombia

Armando Guio Español

Department of Radiology and Informatics, Emory University School of Medicine, Atlanta, GA, USA

Judy Wawira Gichoya

Health Ethics & Governance Unit, Research for Health Department, Science Division, World Health Organization, Geneva, Switzerland

Adrienne Hunt & Katherine Littler

African Center of Excellence in Bioinformatics and Data Intensive Science, Infectious Diseases Institute, Makerere University, Kampala, Uganda

Daudi Jjingo

ISI Foundation, Turin, Italy

Daniela Paolotti

Department of Health Sciences and Technology, ETH Zurich, Zürich, Switzerland

Effy Vayena

Joint Centre for Bioethics, Dalla Lana School of Public Health, University of Toronto, Toronto, Canada

You can also search for this author in PubMed   Google Scholar

Contributions

JS led the writing, contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. JA contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. CA contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. PYC contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. AE contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. JWG contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. AH contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. DJ contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. KL contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. DP contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper. EV contributed to conceptualization and analysis, critically reviewed and provided feedback on drafts of this paper, and provided final approval of the paper.

Corresponding author

Correspondence to James Shaw .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Shaw, J., Ali, J., Atuire, C.A. et al. Research ethics and artificial intelligence for global health: perspectives from the global forum on bioethics in research. BMC Med Ethics 25 , 46 (2024). https://doi.org/10.1186/s12910-024-01044-w

Download citation

Received : 31 October 2023

Accepted : 01 April 2024

Published : 18 April 2024

DOI : https://doi.org/10.1186/s12910-024-01044-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Artificial intelligence
• Machine learning
• Research ethics
• Global health

BMC Medical Ethics

ISSN: 1472-6939

• General enquiries: [email protected]

An Overview of Artificial Intelligence Ethics

• IEEE Transactions on Artificial Intelligence 4(4):799-819
• 4(4):799-819

• Southern University of Science and Technology
• This person is not on ResearchGate, or hasn't claimed this research yet.

Abstract and Figures

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations

• Ayse Basak Cinar

• Flavia Bernardini
• Hemendra Singh
• Chenglin Li

• Ziming Wang

• عمار حسن صفر
• Eduardo Magrani

• Susan Leigh Anderson
• J DATABASE MANAGE

• Eugenia Dacoronia

• Serena Quattrocolo
• Jeffrey Dastin
• S. Matthew Liao
• Peter Railton
• Awni Hannun

• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

The Role of Explainable AI in the Research Field of AI Ethics

New citation alert added.

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Please log in to your account

Information & Contributors

Bibliometrics & citations, view options, 1 introduction, 2 background, 2.1 ai ethics, 2.2.1 transparency., 2.2.2 black-box problem., 2.2.3 accountability and algorithmic bias., 2.3 summary of emerging issues, 3 literature search for primary studies.

3.1 Primary Search

3.2 Inclusion and Exclusion

3.3 Short Analysis of AI Ethics Research Field with Empirical Evidence

4 Classification

4.1 classification schema, 4.2 results of classification.

5 Systematic Map

5.1 systematic map in the bubble plot visualization.

5.2 Pertinence Mapped in a Bubble Plot

5.3 Visualization of Annual Changes in the Research Field

5.4 Venue and Focus of the Research

5.5 Analysis of Connection to Real-world Problems

5.6 summary of empirical contributions, 6 discussion, 6.1 theoretical implications.

6.2 Practical Implications

6.3 Limitations of the Research

7 future research, 8 conclusion.

• Banzhaf W Bakurov I Li X Handl J (2024) On the Nature of the Phenotype in Tree Genetic Programming Proceedings of the Genetic and Evolutionary Computation Conference 10.1145/3638529.3654129 (868-877) Online publication date: 14-Jul-2024 https://dl.acm.org/doi/10.1145/3638529.3654129
• B A H C P L O L (2024) Impact of Explainable AI on Reduction of Algorithm Bias in Facial Recognition Technologies 2024 Systems and Information Engineering Design Symposium (SIEDS) 10.1109/SIEDS61124.2024.10534745 (85-89) Online publication date: 3-May-2024 https://doi.org/10.1109/SIEDS61124.2024.10534745
• Nannini L Marchiori Manerba M Beretta I (2024) Mapping the landscape of ethical considerations in explainable AI research Ethics and Information Technology 10.1007/s10676-024-09773-7 26 :3 Online publication date: 25-Jun-2024 https://dl.acm.org/doi/10.1007/s10676-024-09773-7
• Show More Cited By

Index Terms

Computing methodologies

Artificial intelligence

Philosophical/theoretical foundations of artificial intelligence

Machine learning

Recommendations

Integrating ai ethics in wildlife conservation ai systems in south africa: a review, challenges, and future research agenda.

With the increased use of Artificial Intelligence (AI) in wildlife conservation, issues around whether AI-based monitoring tools in wildlife conservation comply with standards regarding AI Ethics are on the rise. This review aims to summarise ...

Mapping the landscape of ethical considerations in explainable AI research

With its potential to contribute to the ethical governance of AI, eXplainable AI (XAI) research frequently asserts its relevance to ethical considerations. Yet, the substantiation of these claims with rigorous ethical analysis and reflection ...

On the role of knowledge graphs in explainable AI

The current hype of Artificial Intelligence (AI) mostly refers to the success of machine learning and its sub-domain of deep learning. However, AI is also about other areas, such as Knowledge Representation and Reasoning, or Distributed AI, i.e., areas ...

Information

Published in.

Macquarie University, Australia

• Guest Editors:

Association for Computing Machinery

New York, NY, United States

Publication History

Check for updates, author tags.

• explainable AI
• artificial intelligence
• systematic mapping study
• Research-article

Contributors

Other metrics, bibliometrics, article metrics.

• 4 Total Citations View Citations
• 5,256 Total Downloads
• Downloads (Last 12 months) 5,050
• Downloads (Last 6 weeks) 697
• Alvarez J Colmenarejo A Elobaid A Fabbrizzi S Fahimi M Ferrara A Ghodsi S Mougan C Papageorgiou I Reyero P Russo M Scott K State L Zhao X Ruggieri S (2024) Policy advice and best practices on bias and fairness in AI Ethics and Information Technology 10.1007/s10676-024-09746-w 26 :2 Online publication date: 29-Apr-2024 https://dl.acm.org/doi/10.1007/s10676-024-09746-w

View options

View or Download as a PDF file.

View online with eReader .

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Share this publication link.

Copying failed.

Share on social media

Affiliations, export citations.

• Please download or close your previous search result export first before starting a new bulk export. Preview is not available. By clicking download, a status dialog will open to start the export process. The process may take a few minutes but once it finishes a file will be downloadable from your browser. You may continue to browse the DL while the export process is in progress. Download
• Download citation
• Copy citation

We are preparing your search results for download ...

We will inform you here when the file is ready.

Your file of search results citations is now ready.

Your search export query has expired. Please try again.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 06 August 2024

AI and ethics: Investigating the first policy responses of higher education institutions to the challenge of generative AI

• Attila Dabis   ORCID: orcid.org/0000-0003-4924-7664 1 &
• Csaba Csáki   ORCID: orcid.org/0000-0002-8245-1002 1  

Humanities and Social Sciences Communications volume  11 , Article number:  1006 ( 2024 ) Cite this article

617 Accesses

1 Altmetric

Metrics details

• Science, technology and society

This article addresses the ethical challenges posed by generative artificial intelligence (AI) tools in higher education and explores the first responses of universities to these challenges globally. Drawing on five key international documents from the UN, EU, and OECD, the study used content analysis to identify key ethical dimensions related to the use of generative AI in academia, such as accountability, human oversight, transparency, or inclusiveness. Empirical evidence was compiled from 30 leading universities ranked among the top 500 in the Shanghai Ranking list from May to July 2023, covering those institutions that already had publicly available responses to these dimensions in the form of policy documents or guidelines. The paper identifies the central ethical imperative that student assignments must reflect individual knowledge acquired during their education, with human individuals retaining moral and legal responsibility for AI-related wrongdoings. This top-down requirement aligns with a bottom-up approach, allowing instructors flexibility in determining how they utilize generative AI especially large language models in their own courses. Regarding human oversight, the typical response identified by the study involves a blend of preventive measures (e.g., course assessment modifications) and soft, dialogue-based sanctioning procedures. The challenge of transparency induced the good practice of clear communication of AI use in course syllabi in the first university responses examined by this study.

Similar content being viewed by others

Exploring the impact of artificial intelligence on higher education: The dynamics of ethical, social, and educational implications

Intersectionality of social and philosophical frameworks with technology: could ethical AI restore equality of opportunities in academia?

Research on flipped classrooms in foreign language teaching in Chinese higher education

Introduction.

The competition in generative artificial intelligence (AI) ignited by the arrival of ChatGPT, the conversational platform based on a large language model (LLM) in late November 2022 (OpenAI, 2022 ) had a shocking effect even on those who are not involved in the industry (Rudolph et al. 2023 ). Within four months, on 22 March 2023, an open letter was signed by several hundred IT professionals, corporate stakeholders, and academics calling on all AI labs to immediately pause the training of AI systems more powerful than GPT-4 (i.e., those that may trick a human being into believing it is conversing with a peer rather than a machine) for at least six months (Future of Life Institute, 2023 ).

Despite these concerns, competition in generative AI and LLMs does not seem to lose momentum, forcing various social systems to overcome the existential distress they might feel about the changes and the uncertainty of what the future may bring (Roose, 2023 ). Organisations and individuals from different sectors of the economy and various industries are looking for adaptive strategies to accommodate the emerging new normal. This includes lawmakers, international organisations, employers, and employees, as well as academic and higher education institutions (Ray, 2023 ; Wach et al. 2023 ). This fierce competition generates gaps in real-time in everyday and academic life, the latter of which is also trying to make sense of the rapid technological advancement and its effects on university-level education (Perkins, 2023 ). Naturally, these gaps can only be filled, and relevant questions answered much slower by academia, making AI-related research topics timely.

This article aims to reduce the magnitude of these gaps and is intended to help leaders, administrators, teachers, and students better understand the ramifications of AI tools on higher education institutions. It will do so by providing a non-exhaustive snapshot of how various universities around the world responded to generative AI-induced ethical challenges in their everyday academic lives within six-eights months after the arrival of ChatGPT. Thus, the research had asked what expectations and guidelines the first policies introduced into existing academic structures to ensure the informed, transparent, responsible and ethical use of the new tools of generative AI (henceforth GAI) by students and teachers. Through reviewing and evaluating first responses and related difficulties the paper helps institutional decision-makers to create better policies to address AI issues specific to academia. The research reported here thus addressed actual answers to the question of what happened at the institutional (policy) level as opposed to what should happen with the use of AI in classrooms. Based on such a descriptive overview, one may contemplate normative recommendations and their realistic implementability.

Given the global nature of the study’s subject matter, the paper presents examples from various continents. Even though it was not yet a widespread practice to adopt separate, AI-related guidelines, the research focused on universities that had already done so quite early. Furthermore, as best practices most often accrue from the highest-ranking universities, the analysis only considered higher education institutions that were represented among the top 500 universities in the Shanghai Ranking list (containing 3041 Universities at the time), a commonly used source to rank academic excellence. Footnote 1 The main sources of this content analysis are internal documents (such as Codes of Ethics, Academic Regulations, Codes of Practice and Procedure, Guidelines for Students and Teachers or similar policy documents) from those institutions whose response to the GAI challenge was publicly accessible.

The investigation is organised around AI-related ethical dilemmas as concluded from relevant international documents, such as the instruments published by the UN, the EU, and the OECD (often considered soft law material). Through these sources, the study inductively identifies the primary aspects that these AI guidelines mention and can be connected to higher education. Thus it only contains concise references to the main ethical implications of the manifold pedagogical practices in which AI tools can be utilised in the classroom. The paper starts with a review of the challenges posed by AI technology to higher education with special focus on ethical dilemmas. Section 3 covers the research objective and the methodology followed. Section 4 presents the analysis of the selected international documents and establishes a list of key ethical principles relevant in HE contexts and in parallel presents the analysis of the examples distilled from the institutional policy documents and guidelines along that dimension. The paper closes with drawing key conclusions as well as listing limitations and ideas for future research.

Generative AI and higher education: Developments in the literature

General ai-related challenges in the classroom from a historical perspective.

Jacque Ellul fatalistically wrote already in 1954 that the “infusion of some more or less vague sentiment of human welfare” cannot fundamentally alter technology’s “rigorous autonomy”, bringing him to the conclusion that “technology never observes the distinction between moral and immoral use” (Ellul, 1964 , p. 97). Footnote 2 Jumping ahead nearly six decades, the above quote comes to the fore, among others, when evaluating the moral and ethical aspects of the services offered by specific software programs, like ChatGPT. While they might be trained to give ethical answers, these moral barriers can be circumvented by prompt injection (Blalock, 2022 ), or manipulated with tricks (Alberti, 2022 ), so generative AI platforms can hardly be held accountable for the inaccuracy of their responses Footnote 3 or how the physical user who inserted a prompt will make use of the output. Indeed, the AI chatbot is now considered to be a potentially disruptive technology in higher education practices (Farazouli et al. 2024 ).

Educators and educational institution leaders have from the beginning sought solutions on how “to use a variety of the strategies and technologies of the day to help their institutions adapt to dramatically changing social needs” (Miller, 2023 , p. 3). Education in the past had always had high hopes for applying the latest technological advances (Reiser, 2001 ; Howard and Mozejko, 2015 ), including the promise of providing personalised learning or using the latest tools to create and manage courses (Crompton and Burke, 2023 ).

The most basic (and original) educational settings include three components: the blackboard with chalk, the instructor, and textbooks as elementary “educational technologies” at any level (Reiser, 2001 ). Beyond these, one may talk about “educational media” which, once digital technology had entered the picture, have progressed from Computer Based Learning to Learning Management Systems to the use of the Internet, and lately to online shared learning environments with various stages in between including intelligent tutoring system, Dialogue-based Tutoring System, and Exploratory Learning Environment and Artificial Intelligence (Paek and Kim, 2021 ). And now the latest craze is about the generative form of AI often called conversational chatbot (Rudolph et al. 2023 ).

The above-mentioned promises appear to be no different in the case of using generative AI tools in education (Baskara, 2023a ; Mhlanga, 2023 ; Yan et al. 2023 ). The general claim is that GAI chatbots have transformative potential in HE (Mollick and Mollick, 2022 ; Ilieva et al. 2023 ). It is further alleged, that feedback mechanisms supposedly provided by GAI can be used to provide personalised guidance to students (Baskara, 2023b ). Some argue, that “AI education should be expanded and improved, especially by presenting realistic use cases and the real limitations of the technology, so that students are able to use AI confidently and responsibly in their professional future” (Almaraz-López et al. 2023 , p. 1). It is still debated whether the hype is justified, yet the question still remains, how to address the issues arising in the wake of the educational application of GAI tools (Ivanov, 2023 ; Memarian and Doleck, 2023 ).

Generative AI tools, such as their most-known representative, ChatGPT impact several areas of learning and teaching. From the point of view of students, chatbots may help with so-called Self-Regulated or Self-Determined Learning (Nicol and Macfarlane‐Dick, 2006 ; Baskara, 2023b ), where students either dialogue with chatbots or AI help with reviewing student work, even correcting it and giving feedback (Uchiyama et al. 2023 ). There are innovative ideas on how to use AI to support peer feedback (Bauer et al. 2023 ). Some consider that GAI can provide adaptive and personalised environments (Qadir, 2023 ) and may offer personalised tutoring (see, for example, Limo et al. ( 2023 ) on ChatGPT as a virtual tutor for personalized learning experiences). Furthermore, Yan et al. ( 2023 ) lists nine different categories of educational tasks that prior studies have attempted to automate using LLMs: Profiling and labelling (various educational or related content), Detection, Assessment and grading, Teaching support (in various educational and communication activities), Prediction, Knowledge representation, Feedback, Content generation (outline, questions, cases, etc.), Recommendation.

From the lecturers’ point of view, one of the most argued impacts is that assessment practices need to be revisited (Chaudhry et al. 2023 ; Gamage et al. 2023 ; Lim et al. 2023 ). For example, ChatGPT-written responses to exam questions may not be distinguished from student-written answers (Rudolph et al. 2023 ; Farazouli et al. 2024 ). Furthermore, essay-type works are facing special challenges (Sweeney, 2023 ). On the other hand, AI may be utilised to automate a range of educational tasks, such as test question generation, including open-ended questions, test correction, or even essay grading, feedback provision, analysing student feedback surveys, and so on (Mollick and Mollick, 2022 ; Rasul et al. 2023 ; Gimpel et al. 2023 ).

There is no convincing evidence, however, that either lecturers or dedicated tools are able to distinguish AI-written and student-written text with high enough accuracy that can be used to prove unethical behaviour in all cases (Akram, 2023 ). This led to concerns regarding the practicality and ethicality of such innovations (Yan et al. 2023 ). Indeed, the appearance of ChatGPT in higher education has reignited the (inconclusive) debate on the potential and risks associated with AI technologies (Ray, 2023 ; Rudolph et al. 2023 ).

When new technologies appear in or are considered for higher education, debates about their claimed advantages and potential drawbacks heat up as they are expected to disrupt traditional practices and require teachers to adapt to their potential benefits and drawbacks (as collected by Farrokhnia et al. 2023 ). One key area of such debates is the ethical issues raised by the growing accessibility of generative AI and discursive chatbots.

Key ethical challenges posed by AI in higher education

Yan et al. ( 2023 ), while investigating the practicality of AI in education in general, also consider ethicality in the context of educational technology and point out that related debates over the last decade (pre-ChatGPT, so to say), mostly focused on algorithmic ethics, i.e. concerns related to data mining and using AI in learning analytics. At the same time, the use of AI by teachers or, especially, by students has received less attention (or only under the scope or traditional human ethics). However, with the arrival of generative AI chatbots (such as ChatGPT), the number of publications about their use in higher education grew rapidly (Rasul et al. 2023 ; Yan et al. 2023 ).

The study by Chan ( 2023 ) offers a (general) policy framework for higher education institutions, although it focuses on one location and is based on the perceptions of students and teachers. While there are studies that collect factors to be considered for the ethical use of AI in HE, they appear to be restricted to ChatGPT (see, for example, Mhlanga ( 2023 )). Mhlanga ( 2023 ) presents six factors: respect for privacy, fairness, and non-discrimination, transparency in the use of ChatGPT, responsible use of AI (including clarifying its limitations), ChatGPT is not a substitute for human teachers, and accuracy of information. The framework by Chan ( 2023 ) is aimed at creating policies to teach students about GAI and considers three dimensions: pedagogical, governance, and operational. Within those dimensions, ten key areas identified covering ethical concerns such as academic integrity versus academic misconduct and related ethical dilemmas (e.g. cheating or plagiarism), data privacy, transparency, accountability and security, equity in access to AI technologies, critical AI literacy, over-reliance on AI technologies (not directly ethical), responsible use of AI (in general), competencies impeded by AI (such as leadership and teamwork). Baskara ( 2023b ), while also looking at ChatGPT only, considers the following likely danger areas: privacy, algorithmic bias issues, data security, and the potential negative impact of ChatGPT on learners’ autonomy and agency, The paper also questions the possible negative impact of GAI on social interaction and collaboration among learners. Although Yan et al. ( 2023 ) considers education in general (not HE in particular) during its review of 118 papers published since 2017 on the topic of AI ethics in education, its list of areas to look at is still relevant: transparency (of the models used), privacy (related to data collection and use by AI tools), equality (such as availability of AI tools in different languages), and beneficence (e.g. avoiding bias and avoiding biased and toxic knowledge from training data). While systematically reviewing recent publications about AI’s “morality footprint” in higher education, Memarian and Doleck ( 2023 ) consider the Fairness, Accountability, Transparency, and Ethics (FATE) approach as their framework of analyses. They note that “Ethics” appears to be the most used term as it serves as a general descriptor, while the other terms are typically only used in their descriptive sense, and their operationalisation is often lacking in related literature.

Regarding education-related data analytics, Khosravi et al. ( 2022 ) argue that educational technology that involves AI should consider accountability, explainability, fairness, interpretability and safety as key ethical concerns. Ferguson et al. ( 2016 ) also looked at learning analytics solutions using AI and warned of potential issues related to privacy, beneficence, and equality. M.A. Chaudhry et al. ( 2022 ) emphasise that enhancing the comprehension of stakeholders of a new educational AI system is the most important task, which requires making all information and decision processes available to those affected, therefore the key concern is related to transparency according to their arguments.

As such debates continue, it is difficult to identify an established definition of ethical AI in HE. It is clear, however, that the focus should not be on detecting academic misconduct (Rudolph et al. 2023 ). Instead, practical recommendations are required. This is especially true as even the latest studies focus mostly on issues related to assessment practices (Chan, 2023 ; Farazouli et al. 2024 ) and often limit their scope to ChatGPT (Cotton et al. 2024 ) (this specific tool still dominates discourses of LLMs despite the availability of many other solutions since its arrival). At the same time, the list of issues addressed appears to be arbitrary, and most publications do not look at actual practices on a global scale. Indeed, reviews of actual current practices of higher education institutions are rare, and this aspect is not yet the focus of recent HE AI ethics research reports.

As follows from the growing literature and the debate shaping up about the implications of using GAI tools in HE, there was a clear need for a systematic review of how first responses in actual academic policies and guidelines in practice have represented and addressed known ethical principles.

Research objective and methodology

In order to contribute to the debate on the impact of GAI on HE, this study aimed to review how leading institutions had reacted to the arrival of generative AI (such as ChatGPT) and what policies or institutional guidelines they have put in place shortly after. The research intended to understand whether key ethical principles were reflected in the first policy responses of HE institutions and, if yes, how they were handled.

As potential principles can diverge and could be numerous, as well as early guidelines may cover wide areas, the investigation is intended to be based on a few broad categories instead of trying to manage a large set of ideals and goals. To achieve this objective, the research was executed in three steps:

It was started with identifying and collecting general ethical ideals, which were then translated and structured for the context of higher education. A thorough content analysis was performed with the intention to put emphasis on positive values instead of simply focusing on issues or risks and their mitigation.

Given those positive ideals, this research collected actual examples of university policies and guidelines already available: this step was executed from May to July 2023 to find early responses addressing such norms and principles developed by leading HE institutions.

The documents identified were then analysed to understand how such norms and principles had been addressed by leading HE institutions.

As a result, this research managed to highlight and contrast differing practical views, and the findings raise awareness about the difficulties of creating relevant institutional policies. The research considered the ethics of using GAI and not expectations towards their development. The next two sections provide details of the two steps.

Establishing ethical principles for higher education

While the review of relevant ethical and HE literature (as presented above) was not fully conclusive, it highlighted the importance and need for some ideals specific to HE. Therefore, as a first step, this study sought to find highly respected sources of such ethical dimensions by executing a directed content analysis of relevant international regulatory and policy recommendations.

In order to establish what key values and ideas drive the formation of future AI regulations in general, Corrêa et al. ( 2023 ) investigated 200 publications discussing governance policies and ethical guidelines for using AI as proposed by various organisations (including national governments and institutions, civil society and academic organisations, private companies, as well as international bodies). The authors were also interested in whether there are common patterns or missing ideals and norms in this extensive set of proposals and recommendations. As the research was looking for key principles and normative attributes that could form a common ground for the comparison of HE policies, this vast set of documents was used to identify internationally recognised bodies that have potential real influence in this arena and decided to consider the guidelines and recommendations they have put forward for the ethical governance of AI. Therefore, for the purpose of this study, the following sources were selected (some organisations, such as the EU were represented by several bodies):

European Commission ( 2021 ): Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts (2021/0106 (COD)) . Footnote 4

European Parliament Committee on Culture and Education ( 2021 ): Report on artificial intelligence in education, culture and the audiovisual sector (2020/2017(INI)) . Footnote 5

High-Level Expert Group on Artificial Intelligence (EUHLEX) ( 2019 ): Ethics Guidelines for Trustworthy AI . Footnote 6

UNESCO ( 2022 ): Recommendation on the Ethics of Artificial Intelligence (SHS/BIO/PI/2021/1) . Footnote 7

OECD ( 2019 ): Recommendation of the Council on Artificial Intelligence (OECD/LEGAL/0449) . Footnote 8

The ethical dilemmas established by these international documents (most of which is considered soft law material) were then used to inductively identify the primary aspects around which the investigation of educational AI principles may be organised.

Among the above documents, the EUHLEX material is the salient one as it contains a Glossary that defines and explains, among others, the two primary concepts that will be used in this paper: “artificial intelligence” and “ethics”. As this paper is, to a large extent, based on the deducted categorisation embedded in these international documents, it will follow suit in using the above terms as EUHLEX did, supporting it with the definitions contained in the other four referenced international documents. Consequently, artificial intelligence (AI) systems are referred to in this paper as software and hardware systems designed by humans that “act in the physical or digital dimension by perceiving their environment through data acquisition, interpreting the collected structured or unstructured data, reasoning on the knowledge, or processing the information, derived from this data and deciding the best action(s) to take to achieve the given goal” (EUHLEX, 2019 ). With regards to ethics, the EUHLEX group defines this term, in general as an academic discipline which is a subfield of philosophy, dealing with questions like “What is a good action?”, “What is the value of a human life?”, “What is justice?”, or “What is the good life?”. It also mentions that academia distinguishes four major fields: (i) Meta-ethics, (ii) normative ethics, (iii) descriptive ethics, and (iv) applied ethics ” (EUHLEX, 2019 , p. 37). Within these, AI ethics belongs to the latter group of applied ethics that focuses on the practical issues raised by the design, development, implementation, and use of AI systems. By extension, the application of AI systems in higher education also falls under the domain of applied ethics.

The selection of sample universities

The collection of cases started with the AI guidelines compiled by the authors as members of the AI Committee at their university from May to July 2023. The AI Committee consisted of 12 members and investigated over 150 cases to gauge international best practices of GAI use in higher education when formulating a policy recommendation for their own university leadership. Given the global nature of the subject matter, examples from various continents were collected. From this initial pool authors narrowed the scope to the Top 500 higher education institutions of the Shanghai Ranking list for this study, as best practices most often accrue from the highest-ranking universities. Finally, only those institutions were included which, at the time of data collection, have indeed had publicly available policy documents or guidelines with clearly identifiable ethical considerations (such as relevant internal documents, Codes of Ethics, Academic Regulations, Codes of Practice and Procedure, or Guidelines for Students and Teachers). By the end of this selection process, 30 samples proved to be substantiated enough to be included in this study (presented in Table 1 ).

All documents were contextually analysed and annotated by both authors individually looking for references or mentions of ideas, actions or recommendations related to the ethical principles identified during the first step of the research. These comments were then compared and commonalities analysed regarding the nature and goal of the ethical recommendation.

Principles and practices of responsible use of AI in higher education

Ai-related ethical codes forming the base of this investigation.

A common feature of the selected AI ethics documents issued by international organisations is that they enumerate a set of ethical principles based on fundamental human values. The referenced international documents have different geographical- and policy scopes, yet they overlap in their categorisation of the ethical dimensions relevant to this research, even though they might use discrepant language to describe the same phenomenon (a factor we took into account when establishing key categories). For example, what EUHLEX dubs as “Human agency and oversight” is addressed by UNESCO under the section called “Human oversight and determination”, yet they essentially cover the same issues and recommended requirements. Among the many principles enshrined in these documents, the research focuses on those that can be directly linked to the everyday education practices of universities in relation to AI tools, omitting those that, within this context, are less situation-dependent and should normally form the overarching basis of the functioning of universities at all times, such as: respecting human rights and fundamental freedoms, refraining from all forms of discrimination, the right to privacy and data protection, or being aware of environmental concerns and responsibilities regarding sustainable development. As pointed out by Nikolinakos ( 2023 ), such principles and values provide essential guidance not only for development but also during the deployment and use of AI systems. Synthesising the common ethical codes in these instruments has led to the following cluster of ethical principles that are directly linked to AI-related higher education practices:

Accountability and responsibility;

Human agency and oversight;

Transparency and explainability

Inclusiveness and diversity.

The following subsections will give a comprehensive definition of these ethical areas and relate them to higher education expectations. Each subsection will first explain the corresponding ethical cluster, then present the specific university examples, concluding with a summary of the identified best practice under that particular cluster.

Accountability and responsibility

Definition in ethical codes and relevance.

The most fundamental requirements, appearing in almost all relevant documents, bring forward the necessity that mechanisms should be implemented to ensure responsibility and accountability for AI systems and their outcomes. These cover expectations both before and after their deployment, including development and use. They entail the basic requirements of auditability (i.e. the enablement of the assessment of algorithms), clear roles in the management of data and design processes (as a means for contributing to the trustworthiness of AI technology), the minimalisation and reporting of negative impacts (focusing on the possibility of identifying, assessing, documenting and reporting on the potential negative impacts of AI systems), as well as the ability of redress (understood as the capability to utilise mechanisms that offer legal and practical remedy when unjust adverse impact occurs) (EUHLEX, 2019 , pp. 19–20).

Additionally, Points 35–36 of the UNESCO recommendations remind us that it is imperative to “attribute ethical and legal responsibility for any stage of the life cycle of AI systems, as well as in cases of remedy related to AI systems, to physical persons or to existing legal entities. AI system can never replace ultimate human responsibility and accountability” (UNESCO, 2022 , p. 22).

The fulfilment of this fundamental principle is also expected from academic authors, as per the announcements of some of the largest publishing houses in the world. Accordingly, AI is not an author or co-author, Footnote 9 and AI-assisted technologies should not be cited as authors either, Footnote 10 given that AI-generated content cannot be considered capable of initiating an original piece of research without direction from human authors. The ethical guidelines of Wiley ( 2023 ) stated that ”[AI tools] also cannot be accountable for a published work or for research design, which is a generally held requirement of authorship, nor do they have legal standing or the ability to hold or assign copyright.” Footnote 11 This research angle carries over to teaching as well since students are also expected to produce outputs that are the results of their own work. Furthermore, they also often do their own research (such as literature search and review) in support of their projects, homework, thesis, and other forms of performance evaluation.

Accountability and responsibility in university first responses

The rapidly changing nature of the subject matter poses a significant challenge for scholars to assess the state of play of human responsibility. This is well exemplified by the reversal of hearts by some Australian universities (see Rudolph et al. ( 2023 ) quoting newspaper articles) who first disallowed the use of AI by students while doing assignments, just to reverse that decision a few months later and replace it by a requirement of disclosing the use of AI in homeworks. Similarly, Indian governments have been oscillating between a non-regulatory approach to foster an “innovation-friendly environment” for their universities in the summer of 2023 (Liu, 2023 ), only to roll back on this pledge a few months later (Dhaor, 2023 ).

Beyond this regulatory entropy, a fundamental principle enshrined in university codes of ethics across the globe is that students need to meet existing rules of scientific referencing and authorship. Footnote 12 In other words, they should refrain from any form of plagiarism in all their written work (including essays, theses, term papers, or in-class presentations). Submitting any work and assessments created by someone or something else (including AI-generated content) as if it was their own usually amounts to either a violation of scientific referencing, plagiarism or is considered to be a form of cheating (or a combination of these), depending on the terminology used by the respective higher education institution.

As a course description of Johns Hopkins puts it, “academic honesty is required in all work you submit to be graded …., you must solve all homework and programming assignments without the help of outside sources (e.g., GAI tools)” (Johns Hopkins University, 2023 ).

The Tokyo Institute of Technology applies a more flexible approach, as they “trust the independence of the students and expect the best use” of AI systems from them based on good sense and ethical standards. They add, however, that submitting reports that rely almost entirely on the output of GenAI is “highly improper, and its continued use is equivalent to one’s enslavement to the technology” (Tokyo Institute of Technology, 2023 ).

In the case of York University, the Senate’s Academic Standards, Curriculum, and Pedagogy Committee clarified in February 2023 that students are not authorised to use “text-, image-, code-, or video-generating AI tools when completing their academic work unless explicitly permitted by a specific instructor in a particular course” (York University Senate, 2023 ).

In the same time frame (6 February 2023), the University of Oxford stated in a guidance material for staff members that “the unauthorised use of AI tools in exams and other assessed work is a serious disciplinary offence” not permitted for students (University of Oxford, 2023b ).

Main message and best practice: honesty and mutual trust

In essence, students are not allowed to present AI-generated content as their own, Footnote 13 and they should have full responsibility and accountability for their own papers. Footnote 14 This is in line with the most ubiquitous principle enshrined in almost all university guidelines, irrespective of AI, that students are expected to complete their tasks based on their own knowledge and skills obtained throughout their education.

Given that the main challenge here is unauthorised use and overreliance on GAI platforms, the best practice answer is for students to adhere to academic honesty and integrity, scientific referencing standards, existing anti-plagiarism rules, and complete university assignments without fully relying on GAI tools, using, first and foremost, their own skills. The only exception is when instructed otherwise by their professors. By extension, preventing overuse and unauthorised use of AI assists students in avoiding undermining their own academic capacity-building efforts.

Human agency and oversight

AI systems have the potential to manipulate and influence human behaviour in ways that are not easily detectable. AI systems must, therefore, follow human-centric design principles and leave meaningful opportunities for human choice and intervention. Such systems should not be able to unjustifiably subordinate, coerce, deceive, manipulate, condition or herd humans (EUHLEX, 2019 , p. 16).

Human oversight thus refers to the capability for human intervention in every decision cycle of the AI system and the ability of users to make informed, autonomous decisions regarding AI systems. This encompasses the ability to choose not to use an AI system in a particular situation or to halt AI-related operations via a “stop” button or a comparable procedure in case the user detects anomalies, dysfunctions and unexpected performance from AI tools (European Commission, 2021 , Art. 14).

The sheer capability of active oversight and intervention vis-á-vis GAI systems is strongly linked to ethical responsibility and legal accountability. As Liao puts it, “the sufficient condition for human beings being rightsholders is that they have a physical basis for moral agency.” (Liao, 2020 , pp. 496–497). Wagner complemented this with the essential point that entity status for non-human actors would help to shield other parties from liability, i.e., primarily manufacturers and users (Wagner, 2018 ). This, in turn, would result in risk externalisation, which serves to minimise or relativise a person’s moral accountability and legal liability associated with wrongful or unethical acts.

Users, in our case, are primarily students who, at times, might be tempted to make use of AI tools in an unethical way, hoping to fulfil their university tasks faster and more efficiently than they could without these.

Human agency and oversight in university first responses

The crucial aspect of this ethical issue is the presence of a “stop” button or a similar regulatory procedure to streamline the operation of GAI tools. Existing university guidelines in this question point clearly in the direction of soft sanctions, if any, given the fact that there is a lack of evidence that AI detection platforms are effective and reliable tools to tell apart human work from AI-generated ones. Additionally, these tools raise some significant implications for privacy and data security issues, which is why university guidelines are particularly cautious when referring to these. Accordingly, the National Taiwan University, the University of Toronto, the University of Waterloo, the University of Miami, the National Autonomous University of Mexico, and Yale, among others, do not recommend the use of AI detection platforms in university assessments. The University of Zürich further added the moral perspective in a guidance note from 13 July 2023, that “forbidding the use of undetectable tools on unsupervised assignments or demanding some sort of honour code likely ends up punishing the honest students” (University of Zürich, 2023 ). Apart from unreliability, the University of Cape Town also drew attention in its guide for staff that AI detection tools may “disproportionately flag text written by non-first language speakers as AI-generated” (University of Cape Town, 2023 , p. 8).

Macquarie University took a slightly more ambiguous stance when they informed their staff that, while it is not “proof” for anything, an AI writing detection feature was launched within Turnitin as of 5 April 2023 (Hillier, 2023 ), claiming that the software has a 97% detection rate with a 1% false positive rate in the tests that they had conducted (Turnitin, 2023 ). Apart from these, Boston University is among the few examples that recommend employing AI detection tools, but only in a restricted manner to ”evaluate the degree to which AI tools have likely been employed” and not as a source for any punitive measures against students (University of Boston, 2023 ). Remarkably, they complement the above with suggestions for a merit-based scoring system, whereby instructors shall treat work by students who declare no use of AI tools as the baseline for grading. A lower baseline is suggested for students who declare the use of AI tools (depending on how extensive the usage was), and for the bottom of this spectrum, the university suggests imposing a significant penalty for low-energy or unreflective reuse of material generated by AI tools and assigning zero points for merely reproducing the output from AI platforms.

A discrepant approach was adopted at the University of Toronto. Here, if an instructor indicates that the use of AI tools is not permitted on an assessment, and a student is later found to have used such a tool nevertheless, then the instructor should consider meeting with the student as the first step of a dialogue-based process under the Code of Behaviour on Academic Matters (the same Code, which categorises the use of ChatGPT and other such tools as “unauthorised aid” or as “any other form of cheating” in case, an instructor specified that no outside assistance was permitted on an assignment) (University of Toronto, 2019 ).

More specifically, Imperial College London’s Guidance on the Use of Generative AI tools envisages the possibility of inviting a random selection of students to a so-called “authenticity interview” on their submitted assignments (Imperial College London, 2023b ). This entails requiring students to attend an oral examination of their submitted work to ensure its authenticity, which includes questions about the subject or how they approached their assignment.

As a rare exception, the University of Helsinki represents one of the more rigorous examples. The “Guidelines for the Use of AI in Teaching at the University of Helsinki” does not lay down any specific procedures for AI-related ethical offences. On the contrary, as para. 7 stipulates the unauthorised use of GAI in any course examination “constitutes cheating and will be treated in the same way as other cases of cheating” (University of Helsinki, 2023 ). Footnote 15

Those teachers who are reluctant to make AI tools a big part of their courses should rather aim to develop course assessment methods that can plausibly prevent the use of AI tools instead of attempting to filter these afterwards. Footnote 16 For example, the Humboldt-Universität zu Berlin instructs that, if possible, oral or practical examinations or written examinations performed on-site are recommended as alternatives to “classical” written home assignments (Humboldt-Universität zu Berlin, 2023a ).

Monash University also mentions some examples in this regard (Monash University, 2023a ), such as: asking students to create oral presentations, videos, and multimedia resources; asking them to incorporate more personal reflections tied to the concepts studied; implementing programmatic assessment that focuses on assessing broader attributes of students, using multiple methods rather than focusing on assessing individual kinds of knowledge or skills using a single assessment method (e.g., writing an essay).

Similarly, the University of Toronto suggest instructors to: ask students to respond to a specific reading that is very new and thus has a limited online footprint; assign group work to be completed in class, with each member contributing; or ask students to create a first draft of an assignment by hand, which could be complemented by a call to explain or justify certain elements of their work (University of Toronto, 2023 ).

Main message and best practice: Avoiding overreaction

In summary, the best practice that can be identified under this ethical dilemma is to secure human oversight through a blend of preventive measures (e.g. a shift in assessment methods) and soft sanctions. Given that AI detectors are unreliable and can cause a series of data privacy issues, the sanctioning of unauthorised AI use should happen on a “soft basis”, as part of a dialogue with the student concerned. Additionally, universities need to be aware and pay due attention to potentially unwanted rebound effects of bona fide measures, such as the merit-based scoring system of the University of Boston. In that case, using different scoring baselines based on the self-declared use of AI could, in practice, generate incentives for not declaring any use of AI at all, thereby producing counter-effective results.

While explainability refers to providing intelligible insight into the functioning of AI tools with a special focus on the interplay between the user’s input and the received output, transparency alludes to the requirement of providing unambiguous communication in the framework of system use.

As the European Commission’s Regulation proposal ( 2021 ) puts it under subchapter 5.2.4., transparency obligations should apply for systems that „(i) interact with humans, (ii) are used to detect emotions or determine association with (social) categories based on biometric data, or (iii) generate or manipulate content (‘deep fakes’). When persons interact with an AI system or their emotions or characteristics are recognised through automated means, people must be informed of that circumstance. If an AI system is used to generate or manipulate image, audio or video content that appreciably resembles authentic content, there should be an obligation to disclose that the content is generated through automated means, subject to exceptions for legitimate purposes (law enforcement, freedom of expression). This allows persons to make informed choices or step back from a given situation.”

People (in our case, university students and teachers) should, therefore, be fully informed when a decision is influenced by or relies on AI algorithms. In such instances, individuals should be able to ask for further explanation from the decision-maker using AI (e.g., a university body). Furthermore, individuals should be afforded the choice to present their case to a dedicated representative of the organisation in question who should have the power to reviset the decision and make corrections if necessary (UNESCO, 2022 , p. 22). Therefore, in the context of courses and other related education events, teachers should be clear about their utilisation of AI during the preparation of the material. Furthermore, instructors must unambiguously clarify ethical AI use in the classroom. Clear communication is essential about whether students have permission to utilise AI tools during assignments and how to report actual use.

As both UN and EU sources point out, raising awareness about and promoting basic AI literacy should be fostered as a means to empower people and reduce the digital divides and digital access inequalities resulting from the broad adoption of AI systems (EUHLEX, 2019 , p. 23; UNESCO, 2022 , p. 34).

Transparency and explainability in university first responses

The implementation of this principle seems to revolve around the challenge of decentralisation of university work, including the respect for teachers’ autonomy.

Teachers’ autonomy entails that teachers can decide if and to what extent they will allow their students to use AI platforms as part of their respective courses. This, however, comes with the essential corollary, that they must clearly communicate their decision to both students and university management in the course syllabus. To support transparency in this respect, many universities decided to establish 3-level- or 4-level admissibility frameworks (and even those who did not establish such multi-level systems, e.g., the University of Toronto, urge instructors to explicitly indicate in the course syllabus the expected use of AI) (University of Toronto, 2023 ).

The University of Auckland is among the universities that apply a fully laissez passer laissez-faire approach in this respect, meaning that there is a lack of centralised guidance or recommendations on this subject. They rather confer all practical decision-making of GAI use on course directors, adding that it is ultimately the student’s responsibility to correctly acknowledge the use of Gen-AI software (University of Auckland, 2023 ). Similarly, the University of Helsinki gives as much manoeuvring space to their staff as to allow them to change the course of action during the semester. As para 1 of their earlier quoted Guidelines stipulates, teachers are responsible for deciding how GAI can be used on a given course and are free to fully prohibit their use if they think it impedes the achievement of the learning objectives.

Colorado State University, for example, provides its teachers with 3 types of syllabus statement options (Colorado State University, 2023 ): (a) the prohibitive statement: whereby any work created, or inspired by AI agents is considered plagiarism and will not be tolerated; (b) the use-with-permission statement: whereby generative AI can be used but only as an exception and in line with the teachers further instruction, and (c) the abdication statement: where the teacher acknowledges that the course grade will also be a reflection of the students ability to harness AI technologies as part of their preparation for their future in a workforce that will increasingly require AI-literacy.

Macquarie University applies a similar system and provides it’s professors with an Assessment Checklist in which AI use can be either “Not permitted” or “Some use permitted” (meaning that the scope of use is limited while the majority of the work should be written or made by the student.), or “Full use permitted (with attribution)”, alluding to the adaptive use of AI tools, where the generated content is edited, mixed, adapted and integrated into the student’s final submission – with attribution of the source (Macquarie University, 2023 ).

The same approach is used at Monash University where generative AI tools can be: (a) used for all assessments in a specific unit; (b) cannot be used for any assessments; (c) some AI tools may be used selectively (Monash University, 2023b ).

The University of Cape Town (UCT) applies a 3-tier system not just in terms of the overall approach to the use or banning of GAI, but also with regard to specific assessment approaches recommended to teachers. As far as the former is concerned, they differentiate between the strategies of: (a) Avoiding (reverting to in-person assessment, where the use of AI isn’t possible); (b) Outrunning (devising an assessment that AI cannot produce); and (c) Embracing (discussing the appropriate use of AI with students and its ethical use to create the circumstances for authentic assessment outputs). The assessment possibilities, in turn, are categorised into easy, medium, and hard levels. Easy tasks include, e.g., generic short written assignments. Medium level might include examples such as personalised or context-based assessments (e.g. asking students to write to a particular audience whose knowledge and values must be considered or asking questions that would require them to give a response that draws from concepts that were learnt in class, in a lab, field trip…etc). In contrast, hard assessments include projects involving real-world applications, synchronous oral assessments, or panel assessments (University of Cape Town, 2023 ).

4-tier-systems are analogues. The only difference is that they break down the “middle ground”. Accordingly, the Chinese University of Hong Kong clarifies that Approach 1 (by default) means the prohibition of all use of AI tools; Approach 2 entails using AI tools only with prior permission; Approach 3 means using AI tools only with explicit acknowledgement; and Approach 4 is reserved for courses in which the use of AI tools is freely permitted with no acknowledgement needed (Chinese University of Hong Kong, 2023 ).

Similarly, the University of Delaware provides course syllabus statement examples for teachers including: (1) Prohibiting all use of AI tools; (2) Allowing their use only with prior permission; (3) Allow their use only with explicit acknowledgement; (4) Freely allow their use (University of Delaware, 2023 ).

The Technical University of Berlin also proposes a 4-tier system but uses a very different logic based on the practical knowledge one can obtain by using GAI. Accordingly, they divide AI tools as used to: (a) acquire professional competence; (b) learn to write scientifically; (c) be able to assess AI tools and compare them with scientific methods; d) professional use of AI tools in scientific work. Their corresponding guideline even quotes Art. 5 of the German Constitution referencing the freedom of teaching ( Freiheit der Lehre ), entailing that teachers should have the ability to decide for themselves which teaching aids they allow or prohibit. Footnote 17

This detailed approach, however, is rather the exception. According to the compilation on 6 May 2023 by Solis ( 2023 ), among the 100 largest German universities, 2% applied a general prohibition on the use of ChatGPT, 23% granted partial permission, 12% generally permitted its use, while 63% of the universities had none or only vague guidelines in this respect.

Main message and best practice: raising awareness

Overall, the best practice answer to the dilemma of transparency is the internal decentralisation of university work and the application of a “bottom-up” approach that respects the autonomy of university professors. Notwithstanding the potential existence of regulatory frameworks that set out binding rules for all citizens of an HE institution, this means providing university instructors with proper manoeuvring space to decide on their own how they would like to make AI use permissible in their courses, insofar as they communicate their decision openly.

Inclusiveness and diversity

Para. 34 of the Report by the European Parliament Committee on Culture and Education ( 2021 ) highlights that inclusive education can only be reached with the proactive presence of teachers and stresses that “AI technologies cannot be used to the detriment or at the expense of in-person education, as teachers must not be replaced by any AI or AI-related technologies”. Additionally, para. 20 of the same document highlights the need to create diverse teams of developers and engineers to work alongside the main actors in the educational, cultural, and audiovisual sectors in order to prevent gender or social bias from being inadvertently included in AI algorithms, systems, and applications.

This approach also underlines the need to consider the variety of different theories through which AI has been developed as a precursor to ensuring the application of the principle of diversity (UNESCO, 2022 , pp. 33–35), and it also recognises that a nuanced answer to AI-related challenges is only possible if affected stakeholders have an equal say in regulatory and design processes. An idea closely linked to the principle of fairness and the pledge to leave no one behind who might be affected by the outcome of using AI systems (EUHLEX, 2019 , pp. 18–19).

Therefore, in the context of higher education, the principle of inclusiveness aims to ensure that an institution provides the same opportunities to access the benefits of AI technologies for all its students, irrespective of their background, while also considering the particular needs of various vulnerable groups potentially marginalised based on age, gender, culture, religion, language, or disabilities. Footnote 18 Inclusiveness also alludes to stakeholder participation in internal university dialogues on the use and impact of AI systems (including students, teachers, administration and leadership) as well as in the constant evaluation of how these systems evolve. On a broader scale, it implies communication with policymakers on how higher education should accommodate itself to this rapidly changing environment (EUHLEX, 2019 , p. 23; UNESCO, 2022 , p. 35).

Inclusiveness and diversity in university first responses

Universities appear to be aware of the potential disadvantages for students who are either unfamiliar with GAI or who choose not to use it or use it in an unethical manner. As a result, many universities thought that the best way to foster inclusive GAI use was to offer specific examples of how teachers could constructively incorporate these tools into their courses.

The University of Waterloo, for example, recommends various methods that instructors can apply on sight, with the same set of tools for all students during their courses, which in itself mitigates the effects of any discrepancies in varying student backgrounds (University of Waterloo, 2023 ): (a) Give students a prompt during class, and the resulting text and ask them to critique and improve it using track changes; (b) Create two distinct texts and have students explain the flaws of each or combine them in some way using track changes; (c) Test code and documentation accuracy with a peer; or (d) Use ChatGPT to provide a preliminary summary of an issue as a jumping-off point for further research and discussion.

The University of Pittsburgh ( 2023 ) and Monash added similar recommendations to their AI guidelines (Monash University, 2023c ).

The University of Cambridge mentions under its AI-deas initiative a series of projects aimed to develop new AI methods to understand and address sensory, neural or linguistic challenges such as hearing loss, brain injury or language barriers to support people who find communicating a daily challenge in order to improve equity and inclusion. As they put it, “with AI we can assess and diagnose common language and communication conditions at scale, and develop technologies such as intelligent hearing aids, real-time machine translation, or other language aids to support affected individuals at home, work or school.” (University of Cambridge, 2023 ).

The homepage of the Technical University of Berlin (Technische Universität Berlin) displays ample and diverse materials, including videos Footnote 19 and other documents, as a source of inspiration for teachers on how to provide an equitable share of AI knowledge for their students (Glathe et al. 2023 ). More progressively, the university’s Institute of Psychology offers a learning modul called “Inclusive Digitalisation”, available for students enrolled in various degree programmes to understand inclusion and exclusion mechanisms in digitalisation. This modul touches upon topics such as barrier-free software design, mechanisms and reasons for digitalised discrimination or biases in corporate practices (their homepage specifically alludes to the fact that input and output devices, such as VR glasses, have exclusively undergone testing with male test subjects and that the development of digital products and services is predominantly carried out by men. The practical ramifications of such a bias result in input and output devices that are less appropriate for women and children) (Technische Universität Berlin, 2023 ).

Columbia recommends the practice of “scaffolding”, which is the process of breaking down a larger assignment into subtasks (Columbia University, 2023 ). In their understanding, this method facilitates regular check-ins and enables students to receive timely feedback throughout the learning process. Simultaneously, the implementation of scaffolding helps instructors become more familiar with students and their work as the semester progresses, allowing them to take additional steps in the case of students who might need more attention due to their vulnerable backgrounds or disabilities to complete the same tasks.

The Humboldt-Universität zu Berlin, in its Recommendations, clearly links the permission of GAI use with the requirement of equal accessibility. They remind that if examiners require students to use AI for an examination, “students must be provided with access to these technologies free of charge and in compliance with data protection regulations” (Humboldt-Universität zu Berlin, 2023b ).

Concurringly, the University of Cape Town also links inclusivity to accessibility. As they put it, “there is a risk that those with poorer access to connectivity, devices, data and literacies will get unequal access to the opportunities being provided by AI”, leading to the conclusion that the planning of the admissible use of GAI on campus should be cognizant of access inequalities (University of Cape Town, 2023 ). They also draw their staff’s attention to a UNESCO guide material containing useful methods to incorporate ChatGPT into the course, including methods such as the “Socratic opponent” (AI acts as an opponent to develop an argument), the “study buddy” (AI helps the student reflect on learning material) or the “dynamic assessor” (AI provides educators with a profile of each student’s current knowledge based on their interactions with ChatGPT) (UNESCO International Institute for Higher Education in Latin America and the Caribbean, 2023 ).

Finally, the National Autonomous University of Mexico’s Recommendations suggest using GAI tools, among others, for the purposes of community development. They suggest that such community-building activities, whether online or in live groups, kill two birds with one stone. On the one hand, they assist individuals in keeping their knowledge up to date with a topic that is constantly evolving, while it offers people from various backgrounds the opportunity to become part of communities in the process where they can share their experiences and build new relations (National Autonomous University of Mexico, 2023 ).

Main message and best practice: Proactive central support and the pledge to leave no one behind

To conclude, AI-related inclusivity for students is best fostered if the university does not leave its professors solely to their own resources to come up with diverging initiatives. The best practice example for this dilemma thus lies in a proactive approach that results in the elaboration of concrete teaching materials (e.g., subscriptions to AI tools to ensure equal accessibility for all students, templates, video tutorials, open-access answers to FAQs…etc.), specific ideas, recommendations and to support specialised programmes and collaborations with an inclusion-generating edge. With centrally offered resources and tools institutions seem to be able to ensure accessability irrespective of students’ background and financial abilities.

Discussion of the First Responses

While artificial intelligence and even its generative form has been around for a while, the arrival of application-ready LLMs – most notably ChatGPT has changed the game when it comes to grammatically correct large-scale and content-specific text generation. This has invoked an immediate reaction from the higher education community as the question arose as to how it may affect various forms of student performance evaluation (such as essay and thesis writing) (Chaudhry et al. 2023 ; Yu, 2023 ; Farazouli et al. 2024 ).

Often the very first reaction (a few months after the announcement of the availability of ChatGPT) was a ban on these tools and a potential return to hand-written evaluation and oral exams. In the institutions investigated under this research, notable examples may be most Australian universities (such as Monash) or even Oxford. On the other hand, even leading institutions have immediately embraced this new tool as a great potential helper of lecturers – the top name here being Harvard. Very early responses thus ranged widely – and have changed fast over the first six-eight months “post-ChatGPT”.

Over time responses from the institutions investigated started to put out clear guidelines and even created dedicated policies or modified existing ones to ensure a framework of acceptable use. The inspiration leading these early regulatory efforts was influenced by the international ethics documents reviewed in this paper. Institutions were aware of and relied on those guidelines. The main goal of this research was to shed light on the questions of how much and in what ways they took them on board regarding first responses. Most first reactions were based on “traditional” AI ethics and understanding of AI before LLMs and the generative revolution. First responses by institutions were not based on scientific literature or arguments from journal publications. Instead, as our results demonstrated it was based on publicly available ethical norms and guidelines published by well-known international organizations and professional bodies.

Conclusions, limitations and future research

Ethical dilemmas discussed in this paper were based on the conceptualisation embedded in relevant documents of various international fora. Each ethical dimension, while multifaceted in itself, forms a complex set of challenges that are inextricably intertwined with one another. Browsing university materials, the overall impression is that Universities primarily aim to explore and harness the potential benefits of generative AI but not with an uncritical mindset. They are focusing on the opportunities while simultaneously trying to address the emerging challenges in the field.

Accordingly, the main ethical imperative is that students must complete university assignments based on the knowledge and skills they acquired during their university education unless their instructors determine otherwise. Moral and legal responsibility in this regard always rests with human individuals. AI agents possess neither the legal standing nor the physical basis for moral agency, which makes them incapable of assuming such responsibilities. This “top-down” requirement is most often complemented by the “bottom-up” approach of providing instructors with proper maneuvering space to decide how they would like to make AI use permissible in their courses.

Good practice in human oversight could thus be achieved through a combination of preventive measures and soft, dialogue-based procedures. This latter category includes the simple act of teachers providing clear, written communications in their syllabi and engaging in a dialogue with their students to provide unambiguous and transparent instructions on the use of generative AI tools within their courses. Additionally, to prevent the unauthorised use of AI tools, changing course assessment methods by default is more effective than engaging in post-assessment review due to the unreliability of AI detection tools.

Among the many ethical dilemmas that generative AI tools pose to social systems, this paper focused on those pertaining to the pedagogical aspects of higher education. Due to this limitation, related fields, such as university research, were excluded from the scope of the analysis. However, research-related activities are certainly ripe for scientific scrutiny along the lines indicated in this study. Furthermore, only a limited set of institutions could be investigated, those who were the ”first respondents” to the set of issues covered by this study. Hereby, this paper hopes to inspire further research on the impact of AI tools on higher education. Such research could cover more institutions, but it would also be interesting to revisit the same institutions again to see how their stance and approach might have changed over time considering how fast this technology evolves and how much we learn about its capabilities and shortcomings.

Data availability

Data sharing is not applicable to this article as no datasets were generated or analysed during the current study. All documents referenced in this study are publicly available on the corresponding websites provided in the Bibliography or in the footnotes. No code has been developed as part of this research.

For the methodology behind the Shanghai Rankings see: https://www.shanghairanking.com/methodology/arwu/2022 . Accessed: 14 November 2023.

While the original French version was published in 1954, the first English translation is dated 1964.

As the evaluation by Bang et al. ( 2023 ) found, ChatGPT is only 63.41% accurate on average in ten different reasoning categories under logical reasoning, non-textual reasoning, and common-sense reasoning, making it an unreliable reasoner.

Source: https://digital-strategy.ec.europa.eu/en/library/proposal-regulation-laying-down-harmonised-rules-artificial-intelligence . Accessed: 14 November 2023.

Source https://www.europarl.europa.eu/doceo/document/A-9-2021-0127_EN.html . Accessed: 14 November 2023.

Source: https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai . Accessed: 14 November 2023.

Source: https://unesdoc.unesco.org/ark:/48223/pf0000381137 . Accessed: 14 November 2023.

Source: https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449#mainText . Accessed: 14 November 2023.

The editors-in-chief of Nature and Science stated that ChatGPT does not meet the standard for authorship: „ An attribution of authorship carries with it accountability for the work, which cannot be effectively applied to LLMs…. We would not allow AI to be listed as an author on a paper we published, and use of AI-generated text without proper citation could be considered plagiarism,” (Stokel-Walker, 2023 ). See also (Nature, 2023 ).

While there was an initial mistake that credited ChatGPT as an author of an academic paper, Elsevier issued a Corrigendum on the subject in February 2023 (O’Connor, 2023 ). Elsevier then clarified in its “Use of AI and AI-assisted technologies in writing for Elsevier” announcement, issued in March 2023, that “Authors should not list AI and AI-assisted technologies as an author or co-author, nor cite AI as an author”. See https://www.elsevier.com/about/policies-and-standards/the-use-of-generative-ai-and-ai-assisted-technologies-in-writing-for-elsevier . Accessed 23 Nov 2023.

The ethical guidelines of Wiley was updated on 28 February 2023 to clarify the publishing house’s stance on AI-generated content.

See e.g.: Section 2.4 of Princeton University’s Academic Regulations (Princeton University, 2023 ); the Code of Practice and Procedure regarding Misconduct in Research of the University of Oxford (University of Oxford, 2023a ); Section 2.1.1 of the Senate Guidelines on Academic Honesty of York University, enumerating cases of cheating (York University, 2011 ); Imperial College London’s Academic Misconduct Policy and Procedures document (Imperial College London, 2023a ); the Guidelines for seminar and term papers of the University of Vienna (Universität Wien, 2016 ); Para 4. § (1) - (4) of the Anti-plagiarism Regulation of the Corvinus University of Budapest (Corvinus University of Budapest, 2018 ), to name a few.

15 Art. 2 (c)(v) of the early Terms of Use of OpenAI Products (including ChatGPT) dated 14 March 2023 clarified the restrictions of the use of their products. Accordingly, users may not represent the output from their services as human-generated when it was not ( https://openai.com/policies/mar-2023-terms/ . Accessed 14 Nov 2023). Higher education institutions tend to follow suit with this policy. For example, the List of Student Responsibilities enumerated under the “Policies and Regulations” of the Harvard Summer School from 2023 reminds students that their “academic integrity policy forbids students to represent work as their own that they did not write, code, or create” (Harvard University, 2023 ).

A similar view was communicated by Taylor & Francis in a press release issued on 17 February 2023, in which they clarified that: “Authors are accountable for the originality, validity and integrity of the content of their submissions. In choosing to use AI tools, authors are expected to do so responsibly and in accordance with our editorial policies on authorship and principles of publishing ethics” (Taylor and Francis, 2023 ).

This is one of the rare examples where the guideline was adopted by the university’s senior management, in this case, the Academic Affairs Council.

It should be noted that abundant sources recommend harnessing AI tools’ opportunities to improve education instead of attempting to ban them. Heaven, among others, advocated on the pages of the MIT Technology Review the use of advanced chatbots such as ChatGPT as these could be used as “powerful classroom aids that make lessons more interactive, teach students media literacy, generate personalised lesson plans, save teachers time on admin” (Heaven, 2023 ).

This university based its policies on the recommendations of the German Association for University Didactics (Deutsche Gesellschaft für Hochschuldidaktik). Consequently, they draw their students’ attention to the corresponding material, see: (Glathe et al. 2023 ).

For a detailed review of such groups affected by AI see the Artificial Intelligence and Democratic Values Index by the Center for AI and Digital Policy at https://www.caidp.org/reports/aidv-2023/ . Accessed 20 Nov 2023.

See for example: https://www.youtube.com/watch?v=J9W2Pd9GnpQ . Accessed: 14 November 2023.

Akram A (2023) An empirical study of AI generated text detection tools. ArXiv Prepr ArXiv231001423. https://doi.org/10.48550/arXiv.2310.01423

Alberti S (2022) Silas Alberti on X: ChatGPT is trained to not be evil. X Formerly Twitter, 1 December 2022. https://t.co/ZMFdqPs17i . Accessed 23 Nov 2023

Almaraz-López C, Almaraz-Menéndez F, López-Esteban C (2023) Comparative study of the attitudes and perceptions of university students in business administration and management and in education toward Artificial Intelligence. Educ. Sci. 13(6):609. https://doi.org/10.3390/educsci13060609

Article   Google Scholar  

Bang Y, Cahyawijaya S, Lee N et al. (2023) A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. arXiv. https://doi.org/10.48550/arXiv.2302.04023

Baskara FXR (2023a) ChatGPT as a virtual learning environment: multidisciplinary simulations. In: Proceeding of the 3rd International Conference on Innovations in Social Sciences Education and Engineering, Paper 017. https://conference.loupiasconference.orag/index.php/icoissee3/index

Baskara FXR (2023b) The promises and pitfalls of using ChatGPT for self-determined learning in higher education: An argumentative review. Pros. Semin. Nas. Fakultas Tarb. dan. Ilmu Kegur. IAIM Sinjai 2:95–101. https://doi.org/10.47435/sentikjar.v2i0.1825

Bauer E, Greisel M, Kuznetsov I et al. (2023) Using natural language processing to support peer‐feedback in the age of artificial intelligence: A cross‐disciplinary framework and a research agenda. Br. J. Educ. Technol. 54(5):1222–1245. https://doi.org/10.1111/bjet.13336

Blalock D (2022) Here are all the ways to get around ChatGPT’s safeguards: [1/n]. X Formerly Twitter, 13 December 2022. https://twitter.com/davisblalock/status/1602600453555961856 . Accessed 23 Nov 2023

Chan CKY (2023) A comprehensive AI policy education framework for university teaching and learning. Int J. Educ. Technol. High. Educ. 20(1):1–25. https://doi.org/10.1186/s41239-023-00408-3

Chaudhry IS, Sarwary SAM, El Refae GA, Chabchoub H (2023) Time to revisit existing student’s performance evaluation approach in higher education sector in a new era of ChatGPT—A case study. Cogent Educ. 10(1):2210461. https://doi.org/10.1080/2331186x.2023.2210461

Chaudhry MA, Cukurova M, Luckin R (2022) A transparency index framework for AI in education. In: International Conference on Artificial Intelligence in Education. Springer, Cham, Switzerland, pp 195–198. https://doi.org/10.35542/osf.io/bstcf

Chinese University of Hong Kong (2023) Use of Artificial Intelligence tools in teaching, learning and assessments - A guide for students. https://www.aqs.cuhk.edu.hk/documents/A-guide-for-students_use-of-AI-tools.pdf . Accessed 23 Nov 2023

Colorado State University (2023) What should a syllabus statement on AI look like? https://tilt.colostate.edu/what-should-a-syllabus-statement-on-ai-look-like/ . Accessed 23 Nov 2023

Columbia University (2023) Considerations for AI tools in the classroom. https://ctl.columbia.edu/resources-and-technology/resources/ai-tools/ . Accessed 23 Nov 2023

Corrêa NK, Galvão C, Santos JW et al. (2023) Worldwide AI ethics: A review of 200 guidelines and recommendations for AI governance. Patterns 4(10):100857. https://doi.org/10.1016/j.patter.2023.100857

Article   PubMed   PubMed Central   Google Scholar  

Corvinus University of Budapest (2018) Anti-Plagiarism rules. https://www.uni-corvinus.hu/contents/uploads/2020/11/I.20_Plagiumszabalyzat_2018_junius_19_EN.6b1.pdf . Accessed 23 Nov 2023

Cotton DR, Cotton PA, Shipway JR (2024) Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innov. Educ. Teach. Int 61(2):228–239. https://doi.org/10.1080/14703297.2023.2190148

Crompton H, Burke D (2023) Artificial intelligence in higher education: the state of the field. Int J. Educ. Technol. High. Educ. 20(1):1–22. https://doi.org/10.1186/s41239-023-00392-8

Dhaor A (2023) India will regulate AI, ensure data privacy, says Rajeev Chandrasekhar. Hindustan Times, 12 October 2023. https://www.hindustantimes.com/cities/noida-news/india-will-regulate-ai-ensure-data-privacy-says-rajeev-chandrasekhar-101697131022456.html . Accessed 23 Nov 2023

Ellul J (1964) The technological society. Vintage Books

EUHLEX (2019) Ethics guidelines for trustworthy AI | Shaping Europe’s digital future. https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai . Accessed 23 Nov 2023

European Commission (2021) Proposal for a Regulation laying down harmonised rules on artificial intelligence | Shaping Europe’s digital future. https://digital-strategy.ec.europa.eu/en/library/proposal-regulation-laying-down-harmonised-rules-artificial-intelligence . Accessed 23 Nov 2023

European Parliament - Committee on Culture and Education (2021) Report on artificial intelligence in education, culture and the audiovisual sector | A9-0127/2021. https://www.europarl.europa.eu/doceo/document/A-9-2021-0127_EN.html . Accessed 23 Nov 2023

Farazouli A, Cerratto-Pargman T, Bolander-Laksov K, McGrath C (2024) Hello GPT! Goodbye home examination? An exploratory study of AI chatbots impact on university teachers’ assessment practices. Assess. Eval. High. Educ. 49(3):363–375. https://doi.org/10.1080/02602938.2023.2241676

Farrokhnia M, Banihashem SK, Noroozi O, Wals A (2023) A SWOT analysis of ChatGPT: Implications for educational practice and research. Innov. Educ. Teach. Int 61(3):460–474. https://doi.org/10.1080/14703297.2023.2195846

Ferguson R, Hoel T, Scheffel M, Drachsler H (2016) Guest editorial: Ethics and privacy in learning analytics. J. Learn Anal. 3(1):5–15. https://doi.org/10.18608/jla.2016.31.2

Future of Life Institute (2023) Pause giant AI experiments: An open letter. https://futureoflife.org/open-letter/pause-giant-ai-experiments/ . Accessed 15 Nov 2023

Gamage KA, Dehideniya SC, Xu Z, Tang X (2023) ChatGPT and higher education assessments: more opportunities than concerns? J Appl Learn Teach 6(2). https://doi.org/10.37074/jalt.2023.6.2.32

Gimpel H, Hall K, Decker S, et al. (2023) Unlocking the power of generative AI models and systems such as GPT-4 and ChatGPT for higher education: A guide for students and lecturers. Hohenheim Discussion Papers in Business, Economics and Social Sciences 2023, 02:2146. http://opus.uni-hohenheim.de/frontdoor.php?source_opus=2146&la=en

Glathe A, Mörth M, Riedel A (2023) Vorschläge für Eigenständigkeitserklärungen bei möglicher Nutzung von KI-Tools. European University Viadrina. https://opus4.kobv.de/opus4-euv/files/1326/Forschendes-Lernen-mit-KI_SKILL.pdf . Accessed 23 Nov 2023

Harvard University (2023) Student Responsibilities. Harvard Summer School 2023. https://summer.harvard.edu/academic-opportunities-support/policies-and-regulations/student-responsibilities/ . Accessed 23 Nov 2023

Heaven WD (2023) ChatGPT is going to change education, not destroy it. MIT Technology Review. https://www.technologyreview.com/2023/04/06/1071059/chatgpt-change-not-destroy-education-openai/ . Accessed 14 Nov 2023

Hillier M (2023) Turnitin Artificial Intelligence writing detection. https://teche.mq.edu.au/2023/03/turnitin-artificial-intelligence-writing-detection/ . Accessed 23 Nov 2023

Howard SK, Mozejko A (2015) Considering the history of digital technologies in education. In: Henderson M, Romeo G (eds) Teaching and digital technologies: Big issues and critical questions. Cambridge University Press, Port Melbourne, Australia, pp 157–168. https://doi.org/10.1017/cbo9781316091968.017

Humboldt-Universität zu Berlin (2023a) ChatGPT & Co: Empfehlungen für das Umgehen mit Künstlicher Intelligenz in Prüfungen. https://www.hu-berlin.de/de/pr/nachrichten/september-2023/nr-2397-1 . Accessed 23 Nov 2023

Humboldt-Universität zu Berlin (2023b) Empfehlungen zur Nutzung von Künstlicher Intelligenz in Studienleistungen und Prüfungen an der Humboldt-Universität zu Berlin. https://www.hu-berlin.de/de/pr/nachrichten/september-2023/hu_empfehlungen_ki-in-pruefungen_20230905.pdf . Accessed 23 Nov 2023

Ilieva G, Yankova T, Klisarova-Belcheva S et al. (2023) Effects of generative chatbots in higher education. Information 14(9):492. https://doi.org/10.3390/info14090492

Imperial College London (2023a) Academic misconduct policy and procedure. https://www.imperial.ac.uk/media/imperial-college/administration-and-support-services/registry/academic-governance/public/academic-policy/academic-integrity/Academic-Misconduct-Policy-and-Procedure-v1.3-15.03.23.pdf . Accessed 14 Nov 2023

Imperial College London (2023b) College guidance on the use of generative AI tools. https://www.imperial.ac.uk/about/leadership-and-strategy/provost/vice-provost-education/generative-ai-tools-guidance/ . Accessed 23 Nov 2023

Ivanov S (2023) The dark side of artificial intelligence in higher education. Serv. Ind. J. 43(15–16):1055–1082. https://doi.org/10.1080/02642069.2023.2258799

Johns Hopkins University (2023) CSCI 601.771: Self-supervised Models. https://self-supervised.cs.jhu.edu/sp2023/ . Accessed 23 Nov 2023

Khosravi H, Shum SB, Chen G et al. (2022) Explainable artificial intelligence in education. Comput Educ. Artif. Intell. 3:100074. https://doi.org/10.1016/j.caeai.2022.100074

Liao SM (2020) The moral status and rights of Artificial Intelligence. In: Liao SM (ed) Ethics of Artificial Intelligence. Oxford University Press, pp 480–503. https://doi.org/10.1093/oso/9780190905033.003.0018

Lim T, Gottipati S, Cheong M (2023) Artificial Intelligence in today’s education landscape: Understanding and managing ethical issues for educational assessment. Research Square Preprint. https://doi.org/10.21203/rs.3.rs-2696273/v1

Limo FAF, Tiza DRH, Roque MM et al. (2023) Personalized tutoring: ChatGPT as a virtual tutor for personalized learning experiences. Soc. Space 23(1):293–312. https://socialspacejournal.eu/article-page/?id=176

Google Scholar  

Liu S (2023) India’s AI Regulation Dilemma. The Diplomat, 27 October 2023. https://thediplomat.com/2023/10/indias-ai-regulation-dilemma/ . Accessed 23 Nov 2023

Macquarie University (2023) Academic integrity vs the other AI (Generative Artificial Intelligence). https://teche.mq.edu.au/2023/03/academic-integrity-vs-the-other-ai-generative-artificial-intelligence/ . Accessed 14 Nov 2023

Memarian B, Doleck T (2023) Fairness, Accountability, Transparency, and Ethics (FATE) in Artificial Intelligence (AI), and higher education: A systematic review. Comput Educ Artif Intell 100152. https://doi.org/10.1016/j.caeai.2023.100152

Mhlanga D (2023) Open AI in Education, the Responsible and Ethical Use of ChatGPT Towards Lifelong Learning. SSRN Electron J 4354422. https://doi.org/10.2139/ssrn.4354422

Miller GE (2023) eLearning and the Transformation of Higher Education. In: Miller GE, Ives K (eds) Leading the eLearning Transformation of Higher Education. Routledge, pp 3–23. https://doi.org/10.4324/9781003445623-3

Mollick ER, Mollick L (2022) New modes of learning enabled by AI chatbots: Three methods and assignments. SSRN Electron J 4300783. https://doi.org/10.2139/ssrn.4300783

Monash University (2023a) Generative AI and assessment: Designing assessment for achievement and demonstration of learning outcomes. https://www.monash.edu/learning-teaching/teachhq/Teaching-practices/artificial-intelligence/generative-ai-and-assessment . Accessed 23 Nov 2023

Monash University (2023b) Policy and practice guidance around acceptable and responsible use of AI technologies. https://www.monash.edu/learning-teaching/teachhq/Teaching-practices/artificial-intelligence/policy-and-practice-guidance-around-acceptable-and-responsible-use-of-ai-technologies . Accessed 23 Nov 2023

Monash University (2023c) Choosing assessment tasks. https://www.monash.edu/learning-teaching/teachhq/Assessment/choosing-assessment-tasks . Accessed 23 Nov 2023

National Autonomous University of Mexico (2023) Recomendaciones para el uso de Inteligencia Artificial Generativa en la docencia. https://cuaed.unam.mx/descargas/recomendaciones-uso-iagen-docencia-unam-2023.pdf . Accessed 14 Oct 2023

Nature (2023) Tools such as ChatGPT threaten transparent science; here are our ground rules for their use. Nature 613:612. https://doi.org/10.1038/d41586-023-00191-1 . Editorial

Article   CAS   Google Scholar  

Nicol DJ, Macfarlane‐Dick D (2006) Formative assessment and self‐regulated learning: A model and seven principles of good feedback practice. Stud. High. Educ. 31(2):199–218. https://doi.org/10.1080/03075070600572090

Nikolinakos NT (2023) Ethical Principles for Trustworthy AI. In: Nikolinakos NT (ed) EU Policy and Legal Framework for Artificial Intelligence, Robotics and Related Technologies -The AI Act. Springer International Publishing, Cham, Switzerland, pp 101–166. https://doi.org/10.1007/978-3-031-27953-9

O’Connor S (2023) Corrigendum to “Open artificial intelligence platforms in nursing education: Tools for academic progress or abuse?” [Nurse Educ. Pract. 66 (2023) 103537]. Nurse Educ. Pr. 67:103572. https://doi.org/10.1016/j.nepr.2023.103572

OECD (2019) Recommendation of the Council on Artificial Intelligence. https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449#mainText . Accessed 23 Nov 2023

OpenAI (2022) Introducing ChatGPT. https://openai.com/blog/chatgpt . Accessed 14 Nov 2022

Paek S, Kim N (2021) Analysis of worldwide research trends on the impact of artificial intelligence in education. Sustainability 13(14):7941. https://doi.org/10.3390/su13147941

Perkins M (2023) Academic Integrity considerations of AI Large Language Models in the post-pandemic era: ChatGPT and beyond. J. Univ. Teach. Learn Pr. 20(2):07. https://doi.org/10.53761/1.20.02.07

Princeton University (2023) Academic Regulations: Rights, rules, responsibilities. https://rrr.princeton.edu/2023/students-and-university/24-academic-regulations . Accessed 23 Nov 2023

Qadir J (2023) Engineering education in the era of ChatGPT: Promise and pitfalls of generative AI for education. In: 2023 IEEE Global Engineering Education Conference (EDUCON). IEEE, pp 1–9. https://doi.org/10.1109/educon54358.2023.10125121

Rasul T, Nair S, Kalendra D et al. (2023) The role of ChatGPT in higher education: Benefits, challenges, and future research directions. J. Appl Learn Teach. 6(1):41–56. https://doi.org/10.37074/jalt.2023.6.1.29

Ray PP (2023) ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet Things Cyber-Phys. Syst. 3:121–154. https://doi.org/10.1016/j.iotcps.2023.04.003

Reiser RA (2001) A history of instructional design and technology: Part I: A history of instructional media. Educ. Technol. Res Dev. 49(1):53–64. https://doi.org/10.1007/BF02504506

Roose K (2023) GPT-4 is exciting and scary. New York Times, 15 March 2023. https://www.nytimes.com/2023/03/15/technology/gpt-4-artificial-intelligence-openai.html . Accessed 23 Nov 2023

Rudolph J, Tan S, Tan S (2023) War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education. J. Appl Learn Teach. 6(1):364–389. https://doi.org/10.37074/jalt.2023.6.1.23

Solis T (2023) Die ChatGPT-Richtlinien der 100 größten deutschen Universitäten. Scribbr, 6 May 2023. https://www.scribbr.de/ki-tools-nutzen/chatgpt-universitaere-richtlinien/ . Accessed 23 Nov 2023

Stokel-Walker C (2023) ChatGPT listed as author on research papers: Many scientists disapprove. Nature 613:620–621. https://doi.org/10.1038/d41586-023-00107-z

Article   ADS   CAS   PubMed   Google Scholar  

Sweeney S (2023) Who wrote this? Essay mills and assessment – Considerations regarding contract cheating and AI in higher education. Int J. Manag Educ. 21(2):100818. https://doi.org/10.1016/j.ijme.2023.100818

Taylor and Francis (2023) Taylor & Francis clarifies the responsible use of AI tools in academic content creation. Taylor Francis Newsroom, 17 February 2023. https://newsroom.taylorandfrancisgroup.com/taylor-francis-clarifies-the-responsible-use-of-ai-tools-in-academic-content-creation/ . Accessed 23 Nov 2023

Technische Universität Berlin (2023) Inklusive Digitalisierung Modul. https://moseskonto.tu-berlin.de/moses/modultransfersystem/bolognamodule/beschreibung/anzeigen.html?nummer=51021&version=2&sprache=1 . Accessed 05 Aug 2024

Tokyo Institute of Technology (2023) Policy on Use of Generative Artificial Intelligence in Learning. https://www.titech.ac.jp/english/student/students/news/2023/066592.html . Accessed 23 Nov 2023

Turnitin (2023) Turnitin announces AI writing detector and AI writing resource center for educators. https://www.turnitin.com/press/turnitin-announces-ai-writing-detector-and-ai-writing-resource-center-for-educators . Accessed 14 Nov 2023

Uchiyama S, Umemura K, Morita Y (2023) Large Language Model-based system to provide immediate feedback to students in flipped classroom preparation learning. ArXiv Prepr ArXiv230711388. https://doi.org/10.48550/arXiv.2307.11388

UNESCO (2022) Recommendation on the ethics of Artificial Intelligence. https://unesdoc.unesco.org/ark:/48223/pf0000381137 . Accessed 23 Nov 2023

UNESCO International Institute for Higher Education in Latin America and the Caribbean (2023) ChatGPT and Artificial Intelligence in higher education. https://www.iesalc.unesco.org/wp-content/uploads/2023/04/ChatGPT-and-Artificial-Intelligence-in-higher-education-Quick-Start-guide_EN_FINAL.pdf . Accessed 14 Nov 2023

Universität Wien (2016) Guidelines for seminar and term papers. https://bda.univie.ac.at/fileadmin/user_upload/p_bda/Teaching/PaperGuidlines.pdf . Accessed 23 Nov 2023

University of Auckland (2023) Advice for students on using Generative Artificial Intelligence in coursework. https://www.auckland.ac.nz/en/students/forms-policies-and-guidelines/student-policies-and-guidelines/academic-integrity-copyright/advice-for-student-on-using-generative-ai.html . Accessed 24 Nov 2023

University of Boston (2023) Using Generative AI in coursework. https://www.bu.edu/cds-faculty/culture-community/gaia-policy/ . Accessed 23 Nov 2023

University of Cambridge (2023) Artificial Intelligence and teaching, learning and assessment. https://www.cambridgeinternational.org/support-and-training-for-schools/artificial-intelligence/ . Accessed 23 Nov 2023

University of Cape Town (2023) Staff Guide - Assessment and academic integrity in the age of AI. https://docs.google.com/document/u/0/d/1o5ZIOBjPsP6Nh2VIlM56_kcuqB-Y7xTf/edit?pli=1&usp=embed_facebook . Accessed 14 Nov 2023

University of Delaware (2023) Considerations for using and addressing advanced automated tools in coursework and assignments. https://ctal.udel.edu/advanced-automated-tools/ . Accessed 14 Nov 2023

University of Helsinki (2023) Using AI to support learning | Instructions for students. https://studies.helsinki.fi/instructions/article/using-ai-support-learning . Accessed 24 Nov 2023

University of Oxford (2023a) Code of practice and procedure on academic integrity in research. https://hr.admin.ox.ac.uk/academic-integrity-in-research . Accessed 23 Nov 2023

University of Oxford (2023b) Unauthorised use of AI in exams and assessment. https://academic.admin.ox.ac.uk/article/unauthorised-use-of-ai-in-exams-and-assessment . Accessed 23 Nov 2023

University of Pittsburgh (2023) Generative AI Resources for Faculty. https://teaching.pitt.edu/generative-ai-resources-for-faculty/ . Accessed 23 Nov 2023

University of Toronto (2019) Code of behaviour on academic matters. https://governingcouncil.utoronto.ca/secretariat/policies/code-behaviour-academic-matters-july-1-2019 . Accessed 23 Nov 2023

University of Toronto (2023) ChatGPT and Generative AI in the classroom. https://www.viceprovostundergrad.utoronto.ca/strategic-priorities/digital-learning/special-initiative-artificial-intelligence/ . Accessed 20 Nov 2023

University of Waterloo (2023) Artificial Intelligence at UW. https://uwaterloo.ca/associate-vice-president-academic/artificial-intelligence-uw . Accessed 23 Nov 2023

University of Zürich (2023) ChatGPT. https://ethz.ch/en/the-eth-zurich/education/educational-development/ai-in-education/chatgpt.html . Accessed 23 Nov 2023

Wach K, Duong CD, Ejdys J et al. (2023) The dark side of generative artificial intelligence: A critical analysis of controversies and risks of ChatGPT. Entrep. Bus. Econ. Rev. 11(2):7–24. https://doi.org/10.15678/eber.2023.110201

Wagner G (2018) Robot liability. SSRN Electron J 3198764. https://doi.org/10.2139/ssrn.3198764

Wiley (2023) Best practice guidelines on research integrity and publishing ethics. https://authorservices.wiley.com/ethics-guidelines/index.html . Accessed 20 Nov 2023

Yan L, Sha L, Zhao L et al. (2023) Practical and ethical challenges of large language models in education: A systematic scoping review. Br. J. Educ. Technol. 55(1):90–112. https://doi.org/10.1111/bjet.13370

York University (2011) Senate Policy on Academic Honesty. https://www.yorku.ca/secretariat/policies/policies/academic-honesty-senate-policy-on/ . Accessed 23 Nov 2023

York University Senate (2023) Academic Integrity and Generative Artificial Intelligence Technology. https://www.yorku.ca/unit/vpacad/academic-integrity/wp-content/uploads/sites/576/2023/03/Senate-ASCStatement_Academic-Integrity-and-AI-Technology.pdf . Accessed 23 Nov 2023

Yu H (2023) Reflection on whether Chat GPT should be banned by academia from the perspective of education and teaching. Front Psychol. 14:1181712. https://doi.org/10.3389/fpsyg.2023.1181712

Download references

The authors have received no funding, grants, or other support for the research reported here. Open access funding provided by Corvinus University of Budapest.

Author information

Authors and affiliations.

Covinus University of Budapest, Budapest, Hungary

Attila Dabis & Csaba Csáki

You can also search for this author in PubMed   Google Scholar

Contributions

AD had established the initial idea and contributed to the collection of ethical standards as well as to the collection of university policy documents. Also contributed to writing the initial draft and the final version. CsCs had reviewed and clarified the initial concept and then developed the first structure including methodological considerations. Also contributed to the collection of university policy documents as well as to writing the second draft and the final version.

Corresponding author

Correspondence to Attila Dabis .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethical approval

This research did not involve any human participants or animals and required no ethical approval.

Informed consent

This article does not contain any studies with human participants performed by any of the authors. No consent was required as no private data was collected or utilized.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Dabis, A., Csáki, C. AI and ethics: Investigating the first policy responses of higher education institutions to the challenge of generative AI. Humanit Soc Sci Commun 11 , 1006 (2024). https://doi.org/10.1057/s41599-024-03526-z

Download citation

Received : 21 February 2024

Accepted : 29 July 2024

Published : 06 August 2024

DOI : https://doi.org/10.1057/s41599-024-03526-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Front Psychol
• PMC10097940

Ethical content in artificial intelligence systems: A demand explained in three critical points

Artificial intelligence (AI) advancements are changing people’s lives in ways never imagined before. We argue that ethics used to be put in perspective by seeing technology as an instrument during the first machine age. However, the second machine age is already a reality, and the changes brought by AI are reshaping how people interact and flourish. That said, ethics must also be analyzed as a requirement in the content. To expose this argument, we bring three critical points - autonomy, right of explanation, and value alignment - to guide the debate of why ethics must be part of the systems, not just in the principles to guide the users. In the end, our discussion leads to a reflection on the redefinition of AI’s moral agency. Our distinguishing argument is that ethical questioning must be solved only after giving AI moral agency, even if not at the same human level. For future research, we suggest appreciating new ways of seeing ethics and finding a place for machines, using the inputs of the models we have been using for centuries but adapting to the new reality of the coexistence of artificial intelligence and humans.

1. Introduction

Artificial intelligence (AI) is changing people’s lives in ways never imagined before. Machine learning, robots, algorithms, and autonomous vehicles, among others, carry out productive activities and give sophisticated solutions to improve society ( Awad et al., 2018 ; Hooker and Kim, 2019a ). The main goal is to make life easier and more pleasant ( Kim et al., 2021 ), promote well-being, and cause no harm or, at least, minimize it ( Awad et al., 2018 ). However, this equation is not so simple, and troubles emerge from the definition of what artificial intelligence means and, in practical terms, how AI works ( Tegmark, 2017 ). It reflects the complexity of delimiting AI’s boundaries since the new technologies’ benefits, opportunities, and threats share the scene in the still unknown consequences.

As an opportunity, we can mention some potential outcomes like reducing social evils; but machines may also substitute humans in dangerous or unpleasant activities ( Anderson and Anderson, 2011 ). As a critical property of Industry 4.0, it enabled using a significant amount of raw data in knowledge, reducing cost, increasing quality, and improving work conditions ( Kim et al., 2021 ). The first machine age introduced the innovations responsible for substituting human muscle power, creating actual modern life, and sharply bending up the curve of human history to several people and developments never seen before. Now, the world is entering the second machine age, where technology is creating a mental power that is expected to overcome past limitations and lead us to new levels of improvement ( Brynjolfsson and McAfee, 2016 ). So, it makes perfect sense to imagine AI that imitates a broader notion of intelligence that contains wisdom rather than an instrumental view of intelligence ( Kim and Mejia, 2019 ).

Nonetheless, there is still no consensus on what artificial intelligence is and whether the technical challenges will be overcome to achieve a strong type of AI. Nevertheless, these technologies have been adopted in a wide range of domains ( Awad et al., 2018 ; Hooker and Kim, 2019a ; Anagnostou et al., 2022 ; Ashok et al., 2022 ; Miller, 2022 ; Munn, 2022 ), while our understanding of its ethical and societal implications is trivial. In addition, one may see how the evidence proves that unintended consequences may happen more often than expected, despite good intentions ( Coeckelberg, 2020 ). Besides, not just the range and implications of the technical aspects are under evaluation. One can easily understand artificial intelligence as part of computer science and matter for engineering studies; still, the attempt to make AI decisions human-like also prompts cognitive scientists. AI also had become a prolific area of research about the human mind and rekindled century-old discussions regarding decision-making, human actions, rationality, and cognition, among others ( Franklin, 2014 ). In this regard, the long-lasting debate regarding human intelligence was expanded by artificial agents trying to replicate it ( Hooker and Kim, 2019b ); and one of the most fruitful is ethical questioning.

In this context, if we were concerned about developing ethics for humans using machines, now we urgently need to discuss ethics for machines. By seeking to imitate human behavior, AI contrasts with traditional technologies, and, in this regard, the perspective of analyzing ethics changed. From the instrumental standpoint, machines are not more than tools used by humans, so the ethics rest in the individuals using the machines and involve their proper and improper use ( Anderson and Anderson, 2011 ). This scenario fits the first machine age, marked by the Industrial Revolution when machines substituted the human’s muscle power ( Brynjolfsson and McAfee, 2016 ). It means that all trade-offs and moral dilemmas drew in were humans’ responsibility, and the liability is easy to trace back. But what happens when the decision-making recalls solely on the machines? To answer this question, the changes caused by AI are reshaping how people interact and flourish while improving our lives ( Kim and Mejia, 2019 ); that said, ethics is one of the features of human life that should be reconsidered.

To expose this argument, first, we introduce why we should focus on AI ethics. Following, we guide the debate with three critical points: autonomy, right of explanation, and value alignment. Our argument shows that these three crucial points must be considered when analyzing AI’s mimetic process to replicate human-like actions and decisions ( Anderson and Anderson, 2011 ) and, in consequence, describe why ethics must be part of the system’s content, not just in the principles to guide users. In the last part of this paper, we show how these points lead the discussion to a reflection on AI’s agency. We propose that ethical questioning must be solved only after the AI’s moral agency is clarified.

2. Why do AI ethics matter?

Artificial intelligence reaches most of anyone that uses modern technologies, but the whole social fabric will undoubtedly be influenced somehow by its outcomes ( Franklin and Ramsey, 2014 ). Notwithstanding, these advancements raise a host of societal questions as far as the technology and its algorithms silently define our lives, for example, in job promotions, loan offerings, and products consumers might see ( Martin, 2019 ; Kim et al., 2021 ). Areas like education can easily be automated, and even medical technologists have been overtaken by machines ( Hooker and Kim, 2019a ). These are cases in crucial evaluations already under discussion, such as life-and-death medical decisions. The excuse to let these interventions into our lives is that future improvements are expected to reduce inequality, poverty, disasters, war, etc. Thus, the discussion is not about technology; it is about our future ( Tegmark, 2017 ). This context would be enough to see AI’s relation to ethics but going deeper into its background will ensure we are not overlooking the situation.

Many technologies have changed society before. However, for the first time, the technology created might substitute the creators in exclusively human activities and not just collaborate with them, as in the previous evolutions ( Brynjolfsson and McAfee, 2016 ). AI is considered the non-biological type of intelligence that represents the last critical point of life, the technological phase (life 3.0) when both software and hardware can be designed by themselves. This capacity started when the computing data process evolved and added the learning ability, permitting algorithms to learn ( Tegmark, 2017 ). The machine learning ability, born from Arthur Samuel’s checker playing program, turned machines able to evolve from algorithms ( Franklin, 2014 ). Nonetheless, despite this capacity, controversies emerge from the fact that algorithms are still just sequences of instructions that guide machines, or whatever technology it is embedded, into actions using information inputs and giving outputs ( Coeckelberg, 2020 ).

AI’s particular individualities highlight both the computer/technical part and the science side, which helps the scientific area understand human intelligence and replicate it ( Franklin, 2014 ). This cross-disciplinary characteristic reflects the intersection of computability theory from the previous decades and the cognitive revolution. These two critical developments express the moment when the term “artificial intelligence” was coined in 1956 during the seminal Dartmouth Conference ( Arkoudas and Bringsjord, 2014 ). Furthermore, it is crucial to remember that AI is not one technology but a set of them. Classified by their nature, they can be separated into two major buckets ( Kim et al., 2021 ): the strong AI, or General Artificial Intelligence (GAI or AGI), that aims to build human-like technologies with intelligence across domains; and the weak AI, which creates machines that act intelligently without taking a position on whether the AI systems are intelligent in fact ( Arkoudas and Bringsjord, 2014 ; Kim et al., 2021 ). An alternative classification comprehends AI partners, which assist humans, and AI minds, as the ones aiming to overcome humans ( Etzioni and Etzioni, 2017 ). In this sense, some authors talk about a superintelligent AI intending to improve human beings and even achieve immortality by transferring the human brain to a robot. However, it is still unclear how much the discussions about superintelligence are relevant to the development of this area’s studies ( Coeckelberg, 2020 ).

In addition, the very description of what artificial intelligence means is still controversial. Saying that AI is the technology able to show intelligence through algorithms is too imprecise from a philosophical and ethical viewpoint. The lack of a universal definition is one of the reasons why the intelligence conception used for AI is usually compared to the human one ( Coeckelberg, 2020 ). In this sense, the anthropomorphic illusion 1 is an explanation for the comparison, represented by the reductionism in the view of human beings, reflected in mechanicism - on the epistemological level - and in utilitarianism - on the ethical level ( Bertolaso and Rocchi, 2022 ). Therefore, if defining intelligence has always been challenging, the modern form of intelligence did not make it easier to follow the idea. Although it is simple to understand that artificial means non-human, artificial intelligence comprehends a broad field dedicated to developing artifacts capable of intelligent behavior in controlled environments and over specific periods ( Arkoudas and Bringsjord, 2014 ). From a different perspective, AI can also be seen as an interdisciplinary approach to understanding, building models, and replicating intelligent and cognitive processes based on computational, mathematical, mechanical, and even biological principles ( Kim et al., 2021 ).

However, one of the problems of AI’s delimitation relies on the inconsistency pervading those underlying concepts: they infer precise and intelligent systems in particular domains, and they do not always mean human intelligence. For example, animals may also show intelligent behavior ( Kim et al., 2021 ). That said, a more comprehensive definition should also consider the demonstration of “intelligence through non-biological/natural processes” ( Kim et al., 2021 , p. 357). In this connection, Tegmark (2017 , p.85) considers intelligence the “ability to accomplish complex goals.” He prefers to use a comprehensive report and inclusive view since there is no undiscussable definition of intelligence, and an agreement does not exist even among researchers. Besides, the word intelligence trend to have a positive connotation, so a broader interpretation must also be neutral because the mentioned ability does not have only good ends ( Tegmark, 2017 ). We are also adopting this definition to include different understandings and to cover all types of intelligence that are non-comparable and quantifiable only by an ability spectrum across goals.

Surpassing the discussion regarding intelligence, another dilemma arises. The lack of consensus does not involve only the definition, and researchers still do not agree if a universal artificial intelligence, the strong AI or the AI mind, will ever be possible. However, even though the technology is still not smart enough, our understanding of its ethical and societal implications is trivial. In the meantime, the current scenario shows that unintended consequences may happen more often than expected, despite good intentions ( Coeckelberg, 2020 ). That is why it is necessary to develop an AI ethics field dedicated to certifying machine behaviors will be ethically acceptable. In this sense, the state-of-the-art ambition of AI ethics would be to create machines able to decide ethically by themselves. And leaving behind the discussion of how machines would gather it (the technical part), knowing what is ethical connects the AI field again with its philosophical branch ( Anderson and Anderson, 2011 ).

Although it is essential to discuss the technology’s future and the possible impacts of strong AI, when we take a realistic view, the focus is inevitably on weak AI since this is the only type we have today. Also, to understand the boundaries of the discussion of AI’s uses and impacts, one might comprehend that this technology can take numerous forms and is a portion of larger technological systems. Some implications and negative outcomes might also regard other technologies ( Coeckelberg, 2020 ). From this standpoint, AI, like any other technology, is one kind of instrument used by humans. Just like the ones from the first machine age, when machines substituted the Human’s muscle power ( Brynjolfsson and McAfee, 2016 ), which is evident when interpreting it as an AI partner. For machines understood as an instrument, humans are responsible for all kinds of outcomes, whether positive or negative. It means all trade-offs and moral dilemmas drawn in rely upon them, so the liability is easy to trace back. It means the ethics differ from the ones toward other intelligent entities since AI is an artifact of our culture and the result of our intelligence ( Bryson, 2010 ).

On the other hand, ethics for humans using machines from the instrumental perspective is not enough for areas that AI is getting into, so we urgently need to discuss ethics for machines. While strong AI is not possible yet, and ethical machines are just an ultimate goal, AI contrasts with traditional technologies by seeking to imitate human behavior. In this regard, the perspective on analyzing ethics has inevitably changed, and we must see it in the content ( Anderson and Anderson, 2011 ). Even though machines are not entirely human-like, what happens when decision-making is solely on the machines? And more, considering all the inconclusive discussions regarding conception and boundaries may turn the technology into a black box where consequences are not completely mapped. We live in the second machine age, and machines are substituting our mental power ( Brynjolfsson and McAfee, 2016 ). Thus, to answer this question, one might deliberate on the changes caused by AI that are reshaping how people interact and flourish while improving our lives ( Kim and Mejia, 2019 ) since AI is getting into domains known as human exclusivity. That said, ethics is one of the features of human life that should be reconsidered to guarantee that machine outcomes will satisfy society’s ethical expectations. As a starting point for this revaluation, we suggest analyzing three critical points to guide the debate - autonomy, explainable AI, and value alignment -, although one may understand that the discussion is not limited to them. However, they emphasize that ethics must be in machines’ content and are clear examples to expose why we must rethink ethics to fit it in the new scenario imposed by AI.

2.1. Autonomous, but ethical

If artificial intelligence systems keep increasing their levels of aptitude and penetration in our lives, the worries concerning autonomy will intensify. The preoccupation comes from the fact that autonomous agents are generally the ones deciding freely, without external and ethical constraints ( Hooker and Kim, 2019b ). This sense of self-law can create a perception of an autonomous AI that could control our future and be our master instead of serving us ( Kim et al., 2021 ) or convert to a “law unto themselves” ( Hooker and Kim, 2019b , p. 1). Nevertheless, the point when technology will offer machines capable of intentional agency and skilled enough to settle principles and motivations to guide their own acts and decisions is still in the indefinite future ( Hooker and Kim, 2019b ). Furthermore, we argue that autonomy is inadequately interpreted as freedom and free will in AI’s circumstance, considering that free will is a central feature of agency mandatory for actions morally responsible ( Stanford Encyclopedia of Philosophy, 2018 ). By understanding what autonomy for machines means, as well as its limitations, ethical questioning will be cleared.

Autonomy’s concept has been studied across fields, including philosophy, psychology, and, more recently, automation technology. The most known concept of autonomy regards free will. Still, behaviorists see it as responses to environmental stimuli, and other descriptions associate it with self-governance and self-control too. Anyway, these earlier focuses differ from the technological ones seen on artificial intelligence, which is related to the autonomous work function without intervention and might sometimes be the connection between humans and machines. In this last overview, autonomy represents the exchange of control from humans to automation ( Beer et al., 2014 ); and the autonomy level varies by the amount of intervention needed ( Desai and Yanco, 2005 ). That said, autonomous agents should be seen as those who hold goals and act on the environment following motivations and a plan not imposed or adopted by other agents, with different levels of intervention. In addition, the idea of no constraints is not successful when it leaves behind the rationality element necessary in the intelligent demands of AI systems. Yet, the rationality needed recalls the long-lasting known principle of ethics for the coherence of the reasons ( Hooker and Kim, 2019b ).

Within the field of AI, autonomy is often suggested as a feature of human-like or socially interactive. Also, it is related to the capacity to alter its own actions, although just in the environment setting. The term goal is usually connected to this capacity, and control is used inconsistently but is better understood as the lack of intervention ( Beer et al., 2014 ). Nonetheless, one can see that, although it seems AI is deciding by itself, the outcomes are just a reflection of external forces ( Etzioni and Etzioni, 2017 ) given by inputs sent from the environment and following the programming developed by humans. While these characteristics may open space for unethical behavior, it is under discussion that an autonomous action must be explained with coherent reasons, and ethical principles are necessary conditions for it. By setting these frontiers in AI systems, one may see how autonomy does not mean unethical ( Hooker and Kim, 2019b ) and why machines will not master us like in science fiction. Since people own them, we are the ones determining their goals, as well as their actions and behaviors. Real people are dehumanized by aiming for human-like autonomy, and decision-making is encouraged in poor allocation of resources and responsibility ( Bryson, 2010 ). In other words, the science fiction scenery should not imply apprehension. Despite AI’s autonomy, we set the ethical principles and rationale behind the technology. And, by not having free will, machines should not change their plan settled in the first instance by humans.

While the lack of free will is not a problem if we interpret autonomy for machines in this logic above, one may not omit that the impossibility of being free is also the critics’ argument against artificial consciousness ( Casebeer, 2020 ). In this sense, the autonomy discussion is incomplete if we do not broach the matter to the machine’s conscience. Firstly, one must understand the concept of consciousness, which is, broadly, the awareness of its own existence and can be leveled from a basic and rudimentary sense of self-existence to a reflexive capacity of consciousness. The advanced stages of consciousness still seem too far from being achieved in AI, especially when talking about human metacognition and volition. On the other hand, conscience is the capacity to judge right or wrong, and consciousness is the precondition for it, but it evolves over time, which means some actions accepted before can be considered inappropriate nowadays ( Meissner, 2020 ). In the meantime, we expound that even though advanced levels of consciousness are still not achieved, some kind of conscience is mandatory to fulfill ethical demands on current and expected levels of autonomy in AI systems. From this perspective, the efforts to understand the nature of consciousness in the AI context have created a field known as artificial consciousness ( Reggia, 2013 ). Substantial advancements can be seen in the literature, of which some insight can be seen in Eldeman et al. (1992) , Franklin and Graesser (1999) , Safron (2022) , Cleeremans (2005) , Baars and Franklin (2009) , Seth (2009) , Gamez (2012) , Bringsjord et al., 2015 , Reggia et al. (2016) , Tononi et al. (2016) , and Dehaene et al. (2021) , just to mention some of them.

Still, the consciousness process is a mystery ( Reggia, 2013 ) and machines’ conscience requires a more profound discussion to show all perspectives and views for and con. 2 For all that matters in this paper, the autonomy questioning reflects the problem of freedom and determinism ( Hooker and Kim, 2019b ). While deterministic principles prevent agents from having freedom, some compatibilist principles would reject the idea that freedom is connected to the morally responsible agency since we also have conditioning by our choices and situational variants ( Stanford Encyclopedia of Philosophy, 2018 ). That said, the autonomy notion for AI creates a criterium to distinguish action from behavior and agents from non-agents. Unlike actions, autonomous behavior can give ethical outcomes because it is coherent, respects other agents ( Hooker and Kim, 2019b ), and is conditioned to the rationality imposed by humans in the first instance. Also, the generalization principle assures autonomy in the interchange that bounds humans and machines since both are seen as agents, although still different ones.

In this reasoning, AI is a system (or embedded in one) that can be considered an autonomous agent since it senses the context to act based on it (inputs) to pursue its own goals, affecting the senses in the future (output) ( Franklin and Graesser, 1996 ). In other words, to develop autonomy while being ethical, machines do not need to acquire people’s feelings since agency and actions governed by an agenda could give the conditions to guide the performance accordingly to each circumstance ( Hooker and Kim, 2019b ); nor do they need to have consciousness or conscience since moral judgment can be reached from the ethical guidelines given on the counterfactuals, or the external factor inputs provided by humans. Nonetheless, the functioning and interaction between humans and machines must be synergic, and it just happens when trust and reliability are part of the relationship ( Dzindolet et al., 2003 ). So, to satisfy these demands, transparency in explanations becomes essential ( Hooker and Kim, 2019b ).

2.2. Right to explanation and explainable AI

Algorithms autonomously make decisions involving subjects. That said, how the algorithms reach the final decision raises a debate about the explanations as a right to human beings, specifically a moral one. As we exposed in the previous sub-section, autonomous decisions must hold a coherent reason that rational terms can explain. However, aside from the numerous benefits of AI, especially those brought by machine learning, actions and decisions are not always explicable to human users. In addition, machine learning performance has been negatively correlated with explainability. It means that the higher the performance, the less explainable the system is, and the other way around. In this regard, many researchers are working on creating designs whose learning outcomes and decisions are easily comprehended and trusted, as well as to manage the AI’s new generations and keep performance ( Gunning, 2019 ). Yet, different ideas surround the field, and research groups develop distinctive models. In this sense, knowing which model is more suitable is still challenging ( Kim, 2018 ).

On this matter, Explainable AI is the most recent research goal to satisfy these practical, legal, and ethical expectations. This kind of technology has been called XAI, which is correlated with the use, liability issues, right to explanations, and autonomy, among other examples ( Kim, 2018 ). Aiming to provide accountability and transparent systems, the right to explanations is a promissory instrument for governments and other organizations ( Wachter et al., 2017a ). As a moral right, the right to explanation exists beyond the final result impact, which focuses on protecting users’ privacy in consent transactions and third parties that might be involved in the events ( Kim and Routledge, 2018 ). Nonetheless, explainable AI does not mean just transparent, interpretable, or comprehensive. That is why human psychology has been used to give insights into the required information to create reasonable XAI systems. These requirements regarding what the final users need to understand the decisions to decide on the best application ( Gunning, 2019 ). In other words, besides satisfying ethical expectations, humans still need to comprehend how the decisions were made since machines’ outcomes might be too technical for them.

The debate also concerns explanatory needs, information privacy, and fulfilling legal demands, like the United Kingdom’s GDPR (European Union General Data Protection Regulation 2016/679) 3 ( Kim and Routledge, 2018 ). However, it is essential to understand how an AI system can offer explanations before discussing human rights regarding this ( Wachter et al., 2017a ; Kim and Routledge, 2018 ). Content and timing distinguish the types of explanations. The content relates to system functionality and features, while specific domains include rationale, rules, reasons, circumstances, etc. The time defines if the decision requires an ex-ante (prior) or ex-post (after) explanation. Connecting them, the same way that rationale cannot precede the decision, it is possible to follow that the ex-ante relates exclusively to system functionality ( Wachter et al., 2017a ; Hooker and Kim, 2019a ). From another standpoint, ex-ante is a generic explanation, just like the traditional right to be informed. The ex-post, though, regards specific decisions; they are distinguished by remedial and updating explanations, ensuring that organizations will be fair and responsible when something goes wrong or requests to be reformed ( Kim and Routledge, 2018 ).

Aiming to reach the XAI’s demands, the United States’s DARPA (Defense Advanced Research Projects Agency) program uses three strategies to overcome explainability challenges while maintaining performance. The strategies are deep explanations, which modify deep learning by aiming for explainable features; interpretable model techniques, used for learning more structured and causal models; and model induction, to infer the explanation from any models, such as in the case of black boxes ( Gunning, 2019 ). In this sense, one can easily understand the explanation as the exposition of the decision’s logic. However, literature has argued that, for algorithms, the description related to the external facts that lead to that decision is also necessary. These descriptions are known as counterfactuals ( Wachter et al., 2017b ) and can be expressed in natural language to provide an intuitive and efficient tool for analyzing machine decisions ( Hendricks et al., 2018 ). In this context, natural language means our language, not algorithms’ mathematical and logical language.

Since many models are being developed, knowing which is good enough is benefitted from the philosophy of science literature that explains the correct versus the excellent explanation. Two major categories of scientific studies are helpful as a starting point: the non-pragmatic theory of the correct answer to a question and the pragmatic view that seeks to give good answers to the audience’s questions. While the non-pragmatic explanation is the most appropriate to the technology demands, human users still need to understand it ( Kim, 2018 ). In other words, an XAI must be explained precisely and deliver good answers without the inaccuracy found in usual pragmatic explanation theories, also not leaving behind ethical expectations. A deeper investigation would also benefit from the theory of knowledge, in which the conditions for knowledge are truth, belief, and justification. Some thinkers also include safety and sensitivity ( Wachter et al., 2017b ). From this perspective, it is possible to realize that the dialog regarding explainability shows that AI’s problem relies mainly on the lack of one or more conditions of knowledge. Following this idea, a distrust in value misalignment emerges since it is unknown whether machines may be acting unintentionally or carelessly against us and if the outcomes are really going to follow the rationality settled as the condition for being ethical while autonomous.

2.3. Value alignment: allying with our values, not theirs

As we suggested in the previous subsection, autonomous machines need the coherence and rationality of reasonable explanations compatible with human values to be ethical. In this regard, the uncertainty around this compatibility is growing ( Kim et al., 2018 ; Kim and Mejia, 2019 ) as long as highly developed technologies advance in areas considered human exclusivity. This preoccupation reassembles Alan Turing’s ideas of machine adaptation to human standards ( Kim et al., 2018 ). More recently, black-box models and some machine-learning features considered “in the wild” have increased the apprehensions about our security and commitment to society’s values. Nevertheless, many researchers still believe in the potential to develop reliable systems to follow what they are meant to do and what they should not do, aligned with our values ( Arnold et al., 2017 , p. 1). We already know that machines becoming evil robots is science fiction, though misaligned intelligence is a fact, and the worries about value compatibility rely on it ( Tegmark, 2017 ).

In this scenario, researchers are looking to imitate moral intelligence, not just logic and strategy. These efforts have been grouped as value alignment (VA), and this search seeks to overcome the step to turn machines into moral agents and take AI to a higher level. To reach this goal, machines could learn human preferences or learn ethics ( Kim et al., 2019 ). Inverse reinforcement learning (IRL) is a method that could provide it. The IRL in AI systems would infer preferences from humans ( Kim et al., 2018 ) in order to learn how to work and behave ethically as far as applying rules is too strict to the number of domains and could affect autonomy ( Arnold et al., 2017 ). Anyway, experience shows that machines might learn from biased data engendered by humans ( Hooker and Kim, 2019a ); consequently, putting human flourishing at the center is no easy task ( Kim and Mejia, 2019 ). Besides, reinforcement learning puts a load too large on the agents that need to evaluate ethics and social character. Also, there are many technical challenges to overcome, and knowing who trains the machine and how the ethics evaluation would happen in action may also be problematic ( Arnold et al., 2017 ). In addition, empirically observing values in human behaviors might mistake an “ought” for an “is.” Simply put, people assume some behaviors as ethical, but it does not necessarily mean they really are ( Kim et al., 2019 ).

Notwithstanding, many challenges emerge in the attempt to make machines learn ethics from humans. This mimetic process is dangerous since people do not always have the best behavior; one may see that they are not always keen to adopt all values observed in their peers. In this regard, an anchored or hybrid model could be more suitable as far as intrinsic values placed into normative concepts could guarantee the alignment ( Kim et al., 2018 ) but without imposing constraints on what is learned empirically ( Kim et al., 2019 ). Norms are safety dispositive to allow decisions without unexpected outcomes resulting from learning through trial and error. That said, ethical and moral values, as well as legal demands, must be principles of the decision-making designed as part of the system to provide more transparent and accountable outcomes, and to reach these outcomes, the intentions, reasons, norms, and counterfactuals must be considered. These conceptual layers show that society evaluates behaviors by seeing if the intentions are antecedent of the actions and are relevant, the reasons underlie the arguments, norms reflect how society expects to correspond, and the counterfactuals place the action into the context ( Arnold et al., 2017 ).

But, even if we develop machines aligned with our values and able to decide with the ethics expected, it is under discussion if machines will ever be moral agents. Moral intelligence, or the ability to determine ethically, is a distinctive element of human intelligence. Moral sensitivity is within the human conscience, which, for instance, is part of the dynamic moral reasoning that repeatedly balances ethics with empirical observation ( Kim et al., 2019 ). In this sense, we might be over-evaluating AI’s potential as it is still impossible to apply all human morality frameworks to machines. Unexpected consequences are evident in putting the technology under low-specified and poorly defined goals or opening space to let its ability to change and create plans result in actions inconsistent with the intention previously projected ( Vamplew et al., 2018 ). That said, joining these three critical points, it is possible to conclude that autonomy differentiates AI from traditional technologies. However, the explainability and alignment demands show that the agency acquired does not give the freedom and free will required to equalize it to the human moral agency. Consequently, we face an artifact that does not fit in any type of agency known, so we must define it to understand better how our ethical frameworks will work in this new configuration.

3. Discussion: Moral subjects or moral agents? something in the middle!

The debate about AI’s agency is not new and remount to the 1960s ( Taddeo and Floridi, 2018 ). The instrumental use of machines puts technology in the position of a moral subject, which is characterized as a subject of moral motivation but cannot be held responsible for its own actions ( Rowlands, 2012 ). However, as we exposed in the previous sections, artificial intelligence is changing the old instrumental perspective of ethics surrounding human use to include ethics in the AI’s content. Now, ethics must guide machines’ behavior toward humans and other machines, since they are the agents in these decisions and actions ( Anderson and Anderson, 2007 ). The agency is evident, for example, in the first critical point mentioned before, the autonomy, if we adopt the compatibilism perspective that freedom is not necessary for morally responsible actions. Nonetheless, autonomous machines have been entrusted to many applications, and by dealing with various tasks, the responsibility for outcomes enhances the concerns regarding ethics and security. While dealing with problems just after the occurrences is not enough, explainable AI needs to predict outcomes ( Taddeo and Floridi, 2018 ), not just explain them. In addition, we argue that value alignment should go deeper toward a solution to ensure that the delegation to the autonomous system is also responsible, as well as the fact that VA is proof of why AI’s system may permanently be bonded to humans, does not matter the level of autonomy achieved.

Initiatives are trying to set AI boundaries in society. Academia, government, and the private sector are proceeding toward incorporating ethical principles in modern technology systems, such as reliability, transparency, and accountability ( Cooley et al., 2023 ). One example is the Global Initiative on Ethics of Autonomous and Intelligent Systems (IEEE), which highlights that technology should promote well-being and human flourishing instead of the approach that creates principles and constraints ( Vamplew et al., 2018 ; Kim and Mejia, 2019 ). The mission of IEEE is not easy since identifying ethical principles to regulate, design, and AI uses bumps into many cultural contexts and domains ( Taddeo and Floridi, 2018 ). Also, to understand the new configuration of ethics and technology, we must consider machine ethics as the field inside AI’s research to analyze ethics dimensions ( Anderson and Anderson, 2011 ), where the central ambition is to make artificial intelligent systems explicit ethical agents, which can calculate the best option in moral dilemmas. However, the challenge is generating this behavior, considering that ethics, even for humans, is still evolving ( Anderson and Anderson, 2007 ).

Humans, as agents, are capable of representing moral norms (moral core), making moral judgments associated with emotions (moral cognition), regulating these emotions and prosocial actions (moral action), and responding to moral criticism by justifying them (moral communication) ( Cooley et al., 2023 ). Artificial Intelligent systems, on the contrary, are autonomous agents different from other technological systems because they can sense the environment and act upon it. The acts follow an agenda, regardless if other agents set the goals firsthand ( Franklin and Graesser, 1996 ). Yet, these autonomous agents expressed in AI systems have been used in a wide range of domains ( Awad et al., 2018 ; Kim, 2018 ; Hooker and Kim, 2019a ; Anagnostou et al., 2022 ; Ashok et al., 2022 ; Miller, 2022 ; Munn, 2022 ), raising questions about how much we can trust in its moral decision-making and actions, in before exclusively human activities ( Cooley et al., 2023 ). In other words, we reason whether AI technologies have been poorly allocated in moral decision-making domains in which their autonomous agency is insufficient to complete the task.

To understand the background of AI’s agency discussion, we would like to recall James Moor’s three types of agents: the implicit, the explicit, and the full ethical agents. For the implicit moral agents, ethical norms constrain actions because of values embedded in the systems. The explicit one is not so deterministic since it is expected to be an ethical operating system that can respond in moral ways on its own. The full ethical agent is what we recognize as the human moral agency; however, it is not so obvious to understand it. This last type preserves intuitions such as sentience, consciousness, and capacity for suffering, turning them into moral agents and patients. From this point of view, ethical AI systems are moral by not being immoral since they cannot be morally patient and be responsible for acts and consequences ( Gamez et al., 2020 ). In this regard, AI ethics can benefit from the new ethical theories that consider the distributed agency. Traditional ethical frameworks speak to individuals, and human responsibility allocates positive or negative retribution based on individual actions and motivations. But distributed agency implies responsibility shared among all the actors, which is the case of AI and, for example, designers, developers, companies, users, and software/hardware ( Taddeo and Floridi, 2018 ).

The moral agency has been expanded to include partnerships and organizations, for example, but these are still centered in humans when we analyze agenthood. Now, the agency should be stretched to fit the artificial types, which is also essential to understand new moral problems in the machine, general, and mainly distributed ethics ( Floridi and Sanders, 2004 ). At this point, we would like to explain why we consider that AI has been creating a new type of agency in the middle of what we comprehend for subjects and moral agents. Autonomous capacities are living behind the old technology position of moral subjects. However, artificial intelligence is an artifact of our culture ( Bryson, 2010 ), so it is an instrument that must follow our values; that is to say, “machines are ultimately tools of the human beings who design and manufacture them” ( Etzioni and Etzioni, 2017 ). In this sense, it is an implicit ethical agent, following Moor’s typography, that should be constrained by norms in a broader sense to satisfy society’s expectations. Since this deterministic position is not enough for every single decision or outcome, it is necessary to combine features of explicit ethical agents with an ethical operating system. To the explicit ethical part, we argue that the distributed agency could be a key to solving the responsibility problem of the discussion about ethics for machines.

In other words, we suggest a hybrid model that comprises a macro set of norms and rules to guide the system in general terms, reflecting the Moor’s implicit ethical agency but also allowing space to respond in moral ways on their own. Since machines cannot reply to their acts and consequences like humans ( Gamez et al., 2020 ), we explain why the agency should be distributed. This kind of distributed agency relies on theories of contractual and tort liability or strict liability. It separates the intention from the given action or the ability to control outcomes, which is helpful in the case of AI. All agents will hold the responsibility in this distributed system as designers, regulators, and users, also avoiding evil and fostering good by nudging agents toward responsible behaviors ( Taddeo and Floridi, 2018 ). Suppose that technology is putting in society’s hands the power to flourish or destroy itself ( Tegmark, 2017 ). In that case, we should choose the first option by permitting the coexistence between AI and humans to develop better people’s capacities ( Hooker and Kim, 2019a , b ). Thus, when applying the hybrid agency model for AI, we could think about how to relate it with ethical approaches to find a way to make technology correspond with society’s expectations and people’s flourishing.

Actually, we suggest that the best way to discover how to apply our ethical theories for machines is to clarify the agency. Our argument is that ethical theories aim for individuals and their motivations, moral cognition, and sensitivity; in other words, they aim for individuals with full moral agency. On the other side, AI has brought a scenario where ethical decisions must be made, but different from what is done by humans; that said, after understanding what type of agency applies to AI systems, it will be possible to analyze how ethical theories can fit in. In this sense, we expose that, as an implicit ethical agent, moral principles should be programmed into AI programs. Defenders of this kind of top-down approach ( Etzioni and Etzioni, 2017 ), such as Wallach and Allen (2009) , support that ethical choices would be guided by the moral philosophies implanted into the system. Nonetheless, machines do not cope well with vague situations and all nuances humans usually face, whether using one or a combination of moral theories. The other option, the bottom-up approach, accepts that machines could learn ethics by observing humans ( Etzioni and Etzioni, 2017 ). Still, in this kind, there are critics like the naturalistic fallacy, which explains that we can not assume that what is done is what is right ( Kim et al., 2019 ).

By analyzing these challenges, one may see that the troubles are not technological as they reflect old human questionings regarding ethical theories. From this point of view, when appreciating ethics in the AI context, pluralism, which assumes that it is not necessary to choose only one theory of normative ethics ( von der Pfordten, 2012 ), is a valuable alternative to comprehending a hybrid agency model. For instance, deontology and utilitarianisms are formal theories that could help to build an AI Ethics field to develop norms to regulate and guide the implicit ethical agency feature. At the same time, virtue ethics could lead the agents in the distributed agency that would answer to the explicit ethical agency aspect. Virtue ethics, as one of the highest of humans’ purposes of using machines, is a way to avoid moral schizophrenia 4 of being moved by beliefs that would not seek people’s flourishing. On this matter, we consider that, in the AI field, long-term concepts have been used in new situations, the same way as in other areas, like organizational studies. Still, the concept’s displacement must be appropriate and avoid intellectual traps, such as reductionism ( Ramos, 1981 ), and the same applies to this new situation regarding AI.

In this logic, when machines develop the ability to decide by themself which is the best path to choose will be the turning point to turn them into explicit agents. For example, an autonomous vehicle can navigate by following norms that respect local traffic laws and simple rules like “do not hit and kill people,” which could be expressed as a top-down approach that follows deontology, a normative perspective with criterion-satisfying rationality ( De Colle and Werhane, 2008 ). However, in exceptional situations where there is no other choice than choosing between hit in one person or five people (such as the trolley case 5 ), we face a problem where goal-directed rational behavior is best suited ( De Colle and Werhane, 2008 ). That said, the autonomous vehicle could make a utilitarian choice seeking the greater good if this is what local culture most accepts or if it is what has been learned from its owner from the first instance (ethics bots 6 ). We, humans, do this all the time; but we take responsibility for our choices, can (most of the time) explain them, and pay for them. Thus, that is our argument for using practical ethical philosophies, like virtue ethics, since they fundament the deliberation part of an ethical decision. In this regard, deepening this discussion in philosophical arguments combined with practical situations is a demand for future research to solve these complex situations.

Furthermore, another problem regarding moral agency relates to rights. In AI’s context, it brings the question: will moral rights be equalized by making machines moral agents just like humans? At this point, we argue that machines are different from humans because they are agents when responsibility is expected, but they are still instruments when we think about rights. And to show this need, explainable AI and value alignment are examples of how machines should serve us, not the other way around. In this sense, it is essential to highlight that this new use of the words agency and agent just tries to find ways to better understand and allocate the artificial intelligence technologies in our society. However, it does not mean that machines are agents or act like humans; thus, we are not seeking to equal them at the human level. Our preoccupation is not corrupting these words’ conceptions; this paper’s argument follows that the new context imposed other ways to appreciate moral agency and ethics, even though we need to create a new type to comprehend the current reality. This in-between agent is a hybrid model that uses previous knowledge regarding moral agency, respecting distinctions among technology and humans but balancing the needs imposed, such as explainability and learning ethics, to fulfill value alignment.

We already mentioned that a strong type of AI does not exist yet and may never be possible to develop. In this sense, we agree with Etzioni and Etzioni (2017) and their argument that there are strong reasons to believe that AI, no matter how smart and complex, are partners because they carry out very well some task and perform poorly in others, remembering us their role as our instruments that will probably never master us ( Bryson, 2010 ). In other words, if we recall Kant, humanity is what people have expressed in their rational capacities; people are end-in-themselves. However, besides seeking to develop an AI system endowed with rationality, machines are still a phenomenon and a mere means. In this sense, machines will never be full agents, which is our place; however, our argument takes another path by believing that, no matter how difficult it is to implant some attributes into them, we are creating a technology that is changing society’s configuration and the consequence is the emergence of a new type of agency; that said, how to fit our ethical models into their systems need to be discussed. In conclusion, we do not aim to develop a computational formula to solve ethical questions regarding artificial intelligence or to distinguish ontologically humans and machines. Our argument supports that artificial intelligence made machines become moral subjects, but they are still different from the already known ethical principles. So, to make machine ethics possible, we need to rethink our theories considering the new types of agencies created by artificial intelligence.

4. Conclusion

This theoretical essay discussed three critical points that expose how ethics is a demand in artificial intelligence content: autonomy, right of explanation, and value alignment. Although the challenges in the AI field are not limited to them, this argument defends machines as instruments in the first machine age, where ethics was used to guide humans using them. However, the second machine age gave us artificial intelligence and its mimetic processes to be human-like. By doing that, ethics must be part of the systems, and machines must be turned from moral subjects to some kind of moral agents. Anyway, this moral status does not put machines on a human level. Still, it proves that we need to appreciate new ways of seeing ethics and find a place for machines, using the inputs of the models we have been using for centuries but adapting to the new reality of the coexistence of artificial intelligence and humans.

For further research, we suggest investigating hybrid or mixed agency models more profoundly and relating them with ethical models; for this purpose, ontological distinctions between human intelligence and artificial intelligence may be helpful. As we briefly mentioned in the previous section, it would be interesting to interpret deontology and utilitarianism for the implicit ethical agency and virtue ethics to guide distributed agency. However, how to solve this complexity would benefit from a more profound philosophical evaluation, combined with empirical research, taking into account that this pluralistic view still does not answer how we, humans, deliberate and choose one ethical perspective over another since ethics is a branch evolving even for humans. In this logic, AI Ethics could benefit from the experience of other fields already using formal ethical models to create norms and recommend best practices, such as Bioethics. Nonetheless, human flourishing must be the ultimate goal since AI systems are our instruments created by us in order to make our life better. That said, virtue ethics can be a solution to reach the human being behind machines.

Author contributions

All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

1 The anthropomorphic illusion happens when people transfer the blurred knowledge regarding humankind and related concepts – intelligence, rationality, consciousness, and sentience, for example – to other entities and things, as far as our language tends to hypostatize ideas by confusing concepts with real structures ( Polo, 2006 ). This illusion was analyzed before by Ramos (1981) in the organizational context, and we suggest that the same logic has been applied to AI studies.

2 Since our objective is not getting into this discussion now, our position on this topic is that consciousness is one of the unique human characteristics; that is why it is a mystery and unable to be put into technical/mechanical parameters. Nonetheless, we believe that, despite the lack of free will and all the unsolved questions regarding our consciousness and conscience, it is necessary to discuss new ways of deliberating over it to make room for the new scenario imposed by AI. The improvements in AI’s autonomy are related to developing some kind of moral judgment and motivation, despite the differences between how humans are conscious of themselves and AI systems interact with the world. In other words, even though we follow a perspective that believes human consciousness is unreachable in technological terms, the AI context requires the conception of new types of consciousness, even if not at the same level or comparable to ours.

3 It is under discussion whether the European Union General Data Protection Regulation 2016/679 (GDPR) will be able to reach the explainability goal since the protections offered might not be effective enough due to the lack of precise and well-defined discourse. The consequence is that this initiative may just offer the already known right to be informed rather than the aimed right to explanation ( Wachter et al., 2017a , b ).

4 Moral schizophrenia was explained by Stocker (1976) as being moved by motive split from reasons, or doing what is bad or being disgusted by what one wants to do.

5 The trolley case is a philosophical thought experiment that expose a moral dilemma where people need to choose between two undesirable alternatives. It is under discussion if these cases have been misused in context of AI systems ( LaCroix, 2022 ). Anyway, recently, is growing the number of studies highlighting the resemblance between trolley cases and dilemmas faced by autonomous driving ( Schäffner, 2020 ).

6 Ethics bots are AI programs that capture and learn people’s preferences to instruct machines’ behavior to perform in accordance to them ( Etzioni and Etzioni, 2017 ).

• Anagnostou M., Karvounidou O., Katritzidaki C., Kechagia C., Melidou K. (2022). Characteristics and challenges in the industries towards responsible AI: a systematic literature review . Ethics Inf. Technol. 24 :37. doi: 10.1007/s10676-022-09634-1 [ CrossRef ] [ Google Scholar ]
• Anderson M., Anderson S. L. (2007). Machine ethics: creating an ethical intelligent agent . AI Mag. 28 , 15–26. doi: 10.1609/aimag.v28i4.2065 [ CrossRef ] [ Google Scholar ]
• Anderson M., Anderson S. L. (eds.). (2011). Machine ethics . Cambridge University Press. Cambridge. [ Google Scholar ]
• Arkoudas K., Bringsjord S. (2014). “ Philosophical foundations ” in The Cambridge Handbook Of Artificial Intelligence . ed. Frankish K. (Cambridge: Cambridge University Press; ). [ Google Scholar ]
• Arnold T., Kasenberg D., Scheutz M. (2017). Value alignment or misalignment - what will keep systems accountable? AAAI Workshops. 81–88.
• Ashok M., Madan R., Joha A., Sivarajah U. (2022). Ethical framework for artificial intelligence and digital technologies . Int. J. Inf. Manag. 62 :102432. doi: 10.1016/j.ijinfomgt.2021.102433 [ CrossRef ] [ Google Scholar ]
• Awad E., Dsouza S., Kim R., Schulz J., Henrich J., Shariff A., et al.. (2018). The moral machine experiment . Nature 563 , 59–64. doi: 10.1038/s41586-018-0637-6 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Baars J. B., Franklin S. (2009). Consciousness is computational: the LIDA models of global workspace theory . Int. J. Mach. Conscious. 01 , 23–32. doi: 10.1142/S1793843009000050 [ CrossRef ] [ Google Scholar ]
• Beer J. M., Fisk A. D., Rogers W. A. (2014). Toward a framework for levels of robot autonomy in human-robot interaction . J. Human-Robot Interact. 3 , 74–99. doi: 10.5898/jhri.3.2.beer, PMID: [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Bertolaso M., Rocchi M. (2022). Specifically human: human work and care in the age of machines . Bus. Ethics Environ. Responsib. 31 , 888–898. doi: 10.1111/beer.12281 [ CrossRef ] [ Google Scholar ]
• Bringsjord S., Licato J., Govindarajulu N. S., Ghosh R., Rensselaer A. S., Reasoning (RAIR) Lab . (2015). Real robots that pass human tests of self-consciousness . Proceedings of the 24th IEEE International. Symposium on Robot and Human Interactive Communication. Kobe-Japan, 498–504. [ Google Scholar ]
• Brynjolfsson E., McAfee A. (2016). The Second Machine Age: Work, Progress, And Prosperity In A Time Of Brilliant Technologies . New York: W. W. Norton. [ Google Scholar ]
• Bryson J. J. (2010). Robots should be slaves . Artif. Models Nat. Intell. 8 , 63–74. doi: 10.1075/nlp.8.11bry [ CrossRef ] [ Google Scholar ]
• Casebeer W. D. (2020). “ Building an artificial conscience: prospects for morally autonomous artificial intelligence ,” in Artificial Intelligence and Global Security . ed. Y. R. Masakowski (Bingley: Emerald Publishing Limited; ), 81–94. [ Google Scholar ]
• Cleeremans A. (2005). Computational correlates of consciousness . Prog. Brain Res. 150 , 81–98. doi: 10.1016/S0079-6123(05)50007-4 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Coeckelberg M. (2020). Ai Ethics . The Mit Press Essential Knowledge Series . Cambridge, MA: The MIT Press. [ Google Scholar ]
• Cooley K., Walliser J., Wolsten K. (2023). Trusting the moral judgments of a robot: perceived moral competence and Humanlikeness of a GPT-3 enabled AI . Forthcoming Proceedings. 56th Hawaii International Conference on System Sciences. [ Google Scholar ]
• De Colle S., Werhane P. H. (2008). Moral motivation across ethical theories: what can we learn for designing corporate ethics programs? J. Bus. Ethics 81 , 751–764. doi: 10.1007/s10551-007-9545-7 [ CrossRef ] [ Google Scholar ]
• Dehaene S., Lau H., Kouider S. (2021). “ What is consciousness, and could machines have it ” in Robotics, Ai, And Humanity: Science, Ethics, And Policy . eds. Braun J., Archer M. S., Reichberg G. M., Sorondo M. S. (Cham, Switzerland: Springer; ). [ Google Scholar ]
• Desai M., Yanco H. A. (2005). Blending human and robot inputs for sliding scale autonomy . Proceedings - IEEE International Workshop on Robot Human Interactive Communication . 2005 , 537–542. [ Google Scholar ]
• Dzindolet M. T., Peterson S. A., Pomranky R. A., Pierce L. G., Beck H. P. (2003). The role of trust in automation reliance . Int. J. Human-Computer Stud. 58 , 697–718. doi: 10.1016/S1071-5819(03)00038-7 [ CrossRef ] [ Google Scholar ]
• Eldeman G. M., Reeke G. N., Jr., Gall W. E., Tononi G., Willians D., Sporns O. (1992). Neural modeling applied to a real-world artifact . Proc. Natl. Acad. Sci. U. S. A. 89 , 7267–7271. doi: 10.1073/pnas.89.15.7267, PMID: [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Etzioni A., Etzioni O. (2017). Incorporating ethics into artificial intelligence . J. Ethics 21 , 403–418. doi: 10.1007/s10892-017-9252-2 [ CrossRef ] [ Google Scholar ]
• Floridi L., Sanders J. W. (2004). On the morality of artificial agents . Minds Mach. 14 , 349–379. doi: 10.2139/ssrn.3848388 [ CrossRef ] [ Google Scholar ]
• Franklin S. (2014). “ History, motivations, and core themes ” in The Cambridge Handbook Of Artificial Intelligence . ed. Frankish K., Ramsey W. (Cambridge, United Kingdom: Cambridge University Press; ). [ Google Scholar ]
• Franklin S., Graesser A. (1996). Is it an agent, or just a program?: a Taxomony of autonomous agents . Int. Workshop Agent Theories, Archit. Lang. , 21–35. [ Google Scholar ]
• Franklin S., Graesser A. (1999). A software agent model of consciousness . Conscious. Cogn. 8 , 285–301. doi: 10.1006/ccog.1999.0391, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Franklin K., Ramsey W. M. (2014). “ Introduction ” in The Cambridge Handbook Of Artificial Intelligence . ed. Frankish K., Ramsey W. (Cambridge, United Kingdom: Cambridge University Press; ). [ Google Scholar ]
• Gamez D. (2012). Empirically grounded claims about consciousness in computers . Int. J. Mach. Conscious. 04 , 421–438. doi: 10.1142/S1793843012400240 [ CrossRef ] [ Google Scholar ]
• Gamez P., Shank D. B., Arnold C., North M. (2020). Artificial virtue: the machine question and perceptions of moral character in artificial moral agents . AI Soc. 35 , 795–809. doi: 10.1007/s00146-020-00977-1 [ CrossRef ] [ Google Scholar ]
• Gunning D. W. A. (2019). DARPA’s explainable artificial intelligence program . AI Mag. 40 , 44–58. doi: 10.1201/b10933-22 [ CrossRef ] [ Google Scholar ]
• Hendricks L. A., Hu R., Darrell T., Akata Z. (2018). Generating counterfactual explanations with natural language . In Proceedings of the 35th International Conference on Machine Learning Available at: http://arxiv.org/abs/1806.09809 . [ Google Scholar ]
• Hooker J., Kim T. W. (2019a). Ethical implications of the fourth industrial revolution for business and society . Bus. Soc. 3 , 35–63. doi: 10.1108/s2514-175920190000003002 [ CrossRef ] [ Google Scholar ]
• Hooker J., Kim T. W. (2019b). Truly autonomous machines are ethical . AI Mag. 40 , 66–73. doi: 10.1609/aimag.v40i4.2863 [ CrossRef ] [ Google Scholar ]
• Kim T. W. (2018). Explainable Artificial Intelligence (Xai), The Goodness Criteria And The Grasp-Ability Test , 1–7 Available at: http://arxiv.org/abs/1810.09598 .
• Kim T. W., Donaldson T., Hooker J. (2018). Mimetic Vs Anchored Value Alignment In Artificial Intelligence . Available at: http://arxiv.org/abs/1810.11116 .
• Kim T. W., Donaldson T., Hooker J. (2019). Grounding Value Alignment With Ethical Principles . [preprint].
• Kim T. W., Maimone F., Pattit K., Sison A. J., Teehankee B. (2021). Master and slave: the dialectic of human-artificial intelligence engagement . Humanist. Manag. J. 6 , 355–371. doi: 10.1007/s41463-021-00118-w [ CrossRef ] [ Google Scholar ]
• Kim T. W., Mejia S. (2019). From artificial intelligence to artificial wisdom: what Socrates teaches us . Computer 52 , 70–74. doi: 10.1109/MC.2019.2929723 [ CrossRef ] [ Google Scholar ]
• Kim T. W., Routledge B. R. (2018). “ Informational privacy, a right to explanation, and interpretable AI ,” in 2018 IEEE Symposium on Privacy-Aware Computing (PAC) . (Washington, DC, USA: PAC; ), 64–74. [ Google Scholar ]
• LaCroix T. (2022). Moral dilemmas for moral machines . AI Ethics 2 , 737–746. doi: 10.1007/s43681-022-00134-y [ CrossRef ] [ Google Scholar ]
• Martin K. (2019). Ethical implications and accountability of algorithms . J. Bus. Ethics 160 , 835–850. doi: 10.1007/s10551-018-3921-3 [ CrossRef ] [ Google Scholar ]
• Meissner G. (2020). Artificial intelligence: consciousness and conscience . AI Soc. 35 , 225–235. doi: 10.1007/s00146-019-00880-4 [ CrossRef ] [ Google Scholar ]
• Miller G. J. (2022). Artificial intelligence project success factors-beyond the ethical principles . In Conference on Information Systems Management . 65–96. [ Google Scholar ]
• Munn L. (2022). The Uselessness of AI Ethics . AI Ethics . doi: 10.1007/s43681-022-00209-w [ CrossRef ] [ Google Scholar ]
• Polo L. (2006). Curso De Teoría Del Conocimiento . Tomo ii . 4th Edn. Pamplona, Spain: Eunsa. [ Google Scholar ]
• Ramos A. G. (1981). The New Science Of Organizations: A Reconceptualization Of The Wealth Of The Nations . Toronto, Canada: University of Toronto Press. [ Google Scholar ]
• Reggia J. A. (2013). The rise of machine consciousness: studying consciousness with computational models . Neural Netw. 44 , 112–131. doi: 10.1016/j.neunet.2013.03.011, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Reggia J. A., Katz G., Huang D. (2016). What are the computational correlates of consciousness? Biol. Inspired Cogn. Archit. 17 , 101–113. doi: 10.1016/j.bica.2016.07.009 [ CrossRef ] [ Google Scholar ]
• Rowlands M. (2012). Can Animals Be Moral? New York, US: Oxford University Press. [ Google Scholar ]
• Safron A. (2022). Integrated world modeling theory expanded: implications for the future of consciousness . Front. Comput. Neurosci. 16 :642397. doi: 10.3389/fncom.2022.642397, PMID: [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Schäffner V. (2020). Between real world and thought experiment: framing moral decision-making in self-driving car dilemmas . Humanist. Manag. J. 6 , 249–272. doi: 10.1007/s41463-020-00101-x [ CrossRef ] [ Google Scholar ]
• Seth A. (2009). Explanatory correlates of consciousness: theoretical and computational challenges . Cogn. Comput. 1 , 50–63. doi: 10.1007/s12559-009-9007-x [ CrossRef ] [ Google Scholar ]
• Stanford Encyclopedia of Philosophy (2018) Free Will Available at: https://plato.stanford.edu/entries/freewill/ (Accessed June 25, 2022).
• Stocker M. (1976). The schizophrenia of modern ethical theories . J. Philos. 73 , 453–466. doi: 10.2307/2025782 [ CrossRef ] [ Google Scholar ]
• Taddeo M., Floridi L. (2018). How AI can be a force for good . Science 361 , 751–752. doi: 10.1126/science.aat5991, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Tegmark M. (2017). Life 3.0 : Being Human In The Age Of Artificial Intelligence . Alfred A. Knopf, New York. [ Google Scholar ]
• Tononi G., Boly M., Massimini M., Koch C. (2016). Integrated information theory: from consciousness to its physical substrate . Nat. Rev. Neurosci. 17 , 450–461. doi: 10.1038/nrn.2016.44, PMID: [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Vamplew P., Dazeley R., Foale C., Firmin S., Mummery J. (2018). Human-aligned artificial intelligence is a multiobjective problem . Ethics Inf. Technol. 20 , 27–40. doi: 10.1007/s10676-017-9440-6 [ CrossRef ] [ Google Scholar ]
• von der Pfordten D. (2012). Five elements of normative ethics - a general theory of normative individualism . Ethical Theory Moral Pract. 15 , 449–471. doi: 10.1007/s10677-011-9299-2 [ CrossRef ] [ Google Scholar ]
• Wachter S., Mittelstadt B., Floridi L. (2017a). Why a right to explanation of automated decision-making does not exist in the general data protection regulation . Int. Data Priv. Law 7 , 76–99. doi: 10.1609/aimag.v38i3.2741 [ CrossRef ] [ Google Scholar ]
• Wachter S., Mittelstadt B., Russell C. (2017b). Counterfactual explanations without opening the black box: automated decisions and the GDPR . SSRN Electron. J. 31 :841. doi: 10.2139/ssrn.3063289 [ CrossRef ] [ Google Scholar ]
• Wallach W., Allen C. (2009). Moral Machines: Teaching Robots Right From Wrong . New York: Oxford University Press. [ Google Scholar ]

• SUGGESTED TOPICS
• The Magazine
• Newsletters
• Managing Yourself
• Managing Teams
• Work-life Balance
• The Big Idea
• Data & Visuals
• Reading Lists
• Case Selections
• HBR Learning
• Topic Feeds
• Account Settings
• Email Preferences

How Companies Can Take a Global Approach to AI Ethics

• Swanand Deodhar,
• Favour Borokini,

Ideas about right and wrong can differ from one cultural context to the next. Corporate AI governance must reflect this.

Many efforts to build an AI ethics program miss an important fact: ethics differ from one cultural context to the next. Ideas about right and wrong in one culture may not translate to a fundamentally different context, and even when there is alignment, there may well be important differences in the ethical reasoning at work — cultural norms, religious tradition, etc. — that need to be taken into account. Because AI and related data regulations are rarely uniform across geographies, compliance can be difficult. To address this problem, companies need to develop a contextual global AI ethics model that prioritizes collaboration with local teams and stakeholders and devolves decision-making authority to those local teams. This is particularly necessary if their operations span several geographies.

Getting the AI ethics policy right is a high-stakes affair for an organization. Well-published instances of gender biases in hiring algorithms or job search results may diminish the company’s reputation, pit the company against regulations , and even attract hefty government fines . Sensing such threats, organizations are increasingly creating dedicated structures and processes to inculcate AI ethics proactively. Some companies have moved further along this road, creating institutional frameworks for AI ethics .

• SD Swanand Deodhar is an associate professor at the Indian Institute of Management Ahmedabad. His engaged research in topics such as digital platforms and digital transformation is rooted in deep collaboration with practice.  His work has appeared in journals of global repute and reference, such as  MIS Quarterly ,  Information Systems Research , and  Journal of International Business . You can follow him on LinkedIn .
• FB Favour Borokini is a PhD student with the Horizon Centre for Doctoral Training, hosted at the Faculty of Computer Science at the University of Nottingham. Her research interest is in the ethical framework that addresses harm in immersive environments. She holds a Law degree from the University of Benin, Nigeria, and is a member of the Nigerian bar. She has successfully leveraged her legal background to investigate issues such as the impact of technology on human rights, particularly women’s rights, the impact of AI on African women, and the experiences of African women working in AI across various sectors.
• Ben Waber is a visiting scientist at the MIT Media Lab and a senior visiting researcher at Ritsumeikan University. His research and commercial work is focused on the relationship between management, AI, and organizational outcomes. He is also the author of the book  People Analytics . Follow him on Mastodon: @[email protected].

Partner Center

Practical Ethics

New publication: AI Morality

• August 8, 2024 August 8, 2024

Edited by David Edmonds, Distinguished Research Fellow at the Oxford Uehiro Centre, this collection of lively and accessible essays covers topics such as healthcare, employment, autonomous weapons, online advertising and much more.

A philosophical task force explores how AI is revolutionizing our lives – and what moral problems it might bring, showing us what to be wary of, and what to be hopeful for.

There is no more important issue at present than artificial intelligence. AI has begun to penetrate almost every sphere of human activity. It will disrupt our lives entirely. David Edmonds brings together a team of leading philosophers to explore some of the urgent moral concerns we should have about this revolution. The chapters are rich with examples from contemporary society and imaginative projections of the future. The contributors investigate problems we’re all aware of, and introduce some that will be new to many readers. They discuss self and identity, health and insurance, politics and manipulation, the environment, work, law, policing, and defence. Each of them explains the issue in a lively and illuminating way, and takes a view about how we should think and act in response. Anyone who is wondering what ethical challenges the future holds for us can start here.

Includes the following contributions from OUC Researchers:

Risky Business: AI and the Future of Insurance | Jonathan Pugh AI and Discriminatory Intent | Binesh Hass Do AI Systems Allow Online Advertisers to Control Others? | Gabriel De Marco and Tom Douglas Robotic Persons and Asimov’s Three Laws of Robotics | César Palacios-González

Edmonds, D. (Ed.), 2024, ‘AI Morality’, (Oxford University Press)

Published: 08 August 2024 | ISBN: 9780198876434 | Oxford University Press

Join the conversation Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

Notify me of followup comments via e-mail. You can also subscribe without commenting.

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• JME Commentaries
• BMJ Journals

You are here

• Online First
• AI and the falling sky: interrogating X-Risk
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0000-0002-5642-748X Nancy S Jecker 1 , 2 ,
• http://orcid.org/0000-0001-6825-6917 Caesar Alimsinya Atuire 3 , 4 ,
• http://orcid.org/0000-0002-8965-8153 Jean-Christophe Bélisle-Pipon 5 ,
• http://orcid.org/0000-0002-7080-8801 Vardit Ravitsky 6 , 7 ,
• http://orcid.org/0000-0002-9797-1326 Anita Ho 8 , 9
• 1 Department of Bioethics & Humanities , University of Washington School of Medicine , Seattle , Washington , USA
• 2 African Centre for Epistemology and Philosophy of Science , University of Johannesburg , Auckland Park , Gauteng , South Africa
• 3 Centre for Tropical Medicine and Global Health , Oxford University , Oxford , UK
• 4 Department of Philosophy and Classics , University of Ghana , Legon , Greater Accra , Ghana
• 5 Faculty of Health Sciences , Simon Fraser University , Burnaby , British Columbia , Canada
• 6 Hastings Center , Garrison , New York , USA
• 7 Department of Global Health and Social Medicine , Harvard University , Cambridge , Massachusetts , USA
• 8 Bioethics Program , University of California San Francisco , San Francisco , California , USA
• 9 Centre for Applied Ethics , The University of British Columbia , Vancouver , British Columbia , Canada
• Correspondence to Dr Nancy S Jecker, Department of Bioethics & Humanities, University of Washington School of Medicine, Seattle, Washington, USA; nsjecker{at}uw.edu

https://doi.org/10.1136/jme-2023-109702

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

• Information Technology
• Cultural Diversity
• Minority Groups
• Resource Allocation

Introduction

The Buddhist Jātaka tells the tale of a hare lounging under a palm tree who becomes convinced the Earth is coming to an end when a ripe bael fruit falls on its head. Soon all the hares are running; other animals join them, forming a stampede of deer, boar, elk, buffalo, wild oxen, rhinoceros, tigers and elephants, loudly proclaiming the earth is ending. 1 In the American retelling, the hare is ‘chicken little,’ and the exaggerated fear is that the sky is falling.

This paper offers a critical appraisal of the rise of calamity thinking in the scholarly AI ethics literature. It cautions against viewing X-Risk in isolation and highlights ethical considerations sidelined when X-Risk takes centre stage. Section I introduces a working definition of X-Risk, considers its likelihood and explores possible subtexts. It highlights conflicts of interest that arise when tech luminaries lead ethics debates in the public square. Section II flags ethics concerns brushed aside by focusing on X-Risk, including AI existential benefits (X-Benefits), non-AI X-Risk and non-existential AI harms. As we transition towards more AI-centred societies, which we, the authors, would like to fair, we argue for embedding fairness in the transition process by ensuring groups historically disadvantaged or marginalised are not left behind. Section III concludes by proposing a wide-angle lens that takes X-Risk seriously alongside other urgent ethics concerns.

I. Unpacking X-Risk

Doomsayers imagine AI in frightening ways, a paperclip maximiser, ‘whose top goal is the manufacturing of paperclips, with the consequence that it starts transforming first all of earth and increasing portions of space into paperclip manufacturing facilities.’(Bostrom, p5) 6 They compare large language models (LLMs) to the shoggoth in Lovecraft’s novella, ‘a terrible, indescribable thing…a shapeless congeries of protoplasmic bubbles, … with myriads of temporary eyes…as pustules of greenish light all over…’. 7

Prophesies of annihilation have a runaway effect on the public’s imagination. Schwarzenegger, star of The Terminator , a film depicting a computer defence system that achieves self-awareness and initiates nuclear war, has stated that the film’s subject is ‘not any more fantasy or kind of futuristic. It is here today’ and ‘everyone is frightened’. 8 Public attention to X-Risk intensified in 2023, when The Future of Life Institute called on AI labs to pause for 6 months the training of AI systems more powerful than Generative Pre-Trained Transformer (GPT)-4, 9 and, with the Centre for AI Safety, spearheaded a Statement on AI Risk, signed by leaders from OpenAI, Google Deepmind, Anthropic and others stressing that, ‘(m)itigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war’. 10 The 2023 release of Nolan’s film, Oppenheimer, encouraged comparisons between AI and atomic weaponry. Just as Oppenheimer fretted unleashing atomic energy ‘altered abruptly and profoundly the nature of the world,’ and ‘might someday prove deadly to the whole civilisation’, tech leaders fret AI X-Risk.(Bird, p323) 11

The concept of ‘X-Risk’ traces to Bostrom, who in 2002 defined it as a risk involving, ‘an adverse outcome (that) would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential;’ on this rendering, X-Risk imperils ‘humankind as a whole’ and brings ‘major adverse consequences for the course of human civilisation for all time to come.’(Bostrom, p2) 12 More recently, Bostrom and Ćirković defined ‘X-Risk’ as a subset of global catastrophic risks that ‘threatens to cause the extinction of Earth-originating intelligent life or to reduce its quality of life (compared with what would otherwise have been possible) permanently and drastically.’(Bostrom, p4) 13 They classify global catastrophic risks that could become existential in scope, intensity and probability as threefold: risks from nature such as asteroid threats; risks from unintended consequences, such as pandemic diseases; and risks from hostile acts, such as nuclear weaponry. We use Bostrom and Ćirković’s account as our working definition of X-Risk. While it is vague in the sense of leaving open the thresholds for scope, intensity and probability, it carries the advantage of breadth and relevance to a range of serious threats.

Who says the sky is falling?

A prominent source of apocalyptic thinking regarding AI comes from within the tech industry. According to a New York Times analysis, many tech leaders believe that AI advancement is inevitable, because it is possible, and think those at the forefront of creating it know best how to shape it. 14 In a 2019 scoping review of global AI ethics guidelines, Jobin et al identified 84 documents containing AI ethics principles or guidance, with most from the tech industry.(Jobin, p396) 15 However, a limitation of the study was that ethics guidance documents represent ‘soft law,’ which is not indexed in conventional databases, making retrieval less replicable and unbiased. More recently, Stanford University’s 2023 annual AI Index Report examined authorship of scholarly AI ethics literature and reported a shift away from academic authors towards authors with industry affiliations; the Report showed industry-affiliated authors produced 71% more publications than academics year over year between 2014 and 2022. 16

Since AI companies benefit financially from their investments in AI, relying on them for ethics guidance creates a conflict of interest. A ‘conflict of interest’ is a situation where ‘an individual’s judgement concerning a primary interest tends to be unduly influenced (or biased) by a secondary interest.’(Resnik, p121–22) 17 In addition to financial conflicts of interest, non-financial conflicts of interest can arise from multiple sources (eg, personal or professional relationships, political activity, involvement in litigation). 17 Non-financial conflicts of interest can occur subconsciously, and implicit cognitive biases can transfer to AI systems. Since most powerful tech companies are situated in high-income Western countries, they may be implicitly partial to values and concerns prevalent in those societies, reflecting anchoring bias (believing what one wants or expects) and confirmation bias (clinging to beliefs despite conflicting evidence). The dearth of research exploring AI’s social impacts in diverse cultural settings around the world makes detecting and dislodging implicit bias difficult. 18 Commenting on the existing corpus of AI ethics guidance, Jobin et al noted a significant representation of more economically developed countries, with the USA and UK together accounting for more than a third of AI ethics principles in 2019, followed by Japan, Germany, France and Finland. Notably, African and South American countries were not represented. While authors of AI ethics guidance often purport to represent the common good, a 2022 study by Bélisle-Pipon et al showed a broad trend towards asymmetrical engagement, with industry and those with vested interests in AI more represented than the public. 19 Hagerty and Rubinov report that risks for discriminatory outcomes in machine learning are particularly high for countries outside the USA and Western Europe, especially when algorithms developed in higher-income countries are deployed in low-income and middle-income countries that have different resource and social realities. 18

Another prominent source of calamity thinking is members of the effective altruism movement and the associated cause of longtermism, two groups that focus on ‘the most extreme catastrophic risks and emphasise the far-future consequences of our actions’. 20 Effective altruism is associated with a philosophical and social movement based largely at Oxford University and Silicon Valley. Its members include philosophers like Singer, Ord and MacAskill, along with tech industry leaders like the discredited cryptocurrency founder, Bankman-Fried. The guiding principles of effective altruism are ‘to do as much good as we can’ and ‘to base our actions on the best available evidence and reasoning about how the world works’. 21 MacAskill defines longtermism as ‘the idea that positively influencing the long-term future is a key moral priority of our time’, and underscores, ‘Future people count. There could be a lot of them. We can make their lives go better.’(MakAskill, pp5, 21) 22 Effective altruism and longtermism have spawned charitable organisations dedicated to promoting its goals, including GiveWell, Open Philanthropy and The Future of Life Institute. To be clear, we are not suggesting that adherents of longtermism are logically forced to embrace X-Risk or calamity thinking; our point is that adherents of longtermism draw on it to justify catastrophising.

Who benefits and who is placed at risk?

Critics of longtermism argue that it fails to give sufficient attention to serious problems happening now, particularly problems affecting those who have been historically disadvantaged or marginalised. Worse, it can give warrant to sacrificing present people’s rights and interests to stave off a prophesied extinction event. Thus, a well-recognised danger of maximisation theories is that they can be used to justify unethical means if these are deemed necessary to realise faraway goals that are thought to serve a greater good. Some effective altruists acknowledge this concern. MacAskill, for example, concedes that longtermism endorses directing resources away from present concerns, such as responding to the plight of the global poor, and towards more distant goals of preventing X-Risk. 23

X-Risk also raises theoretical challenges related to intergenerational justice. How should we understand duties to future people? Can we reasonably argue that it is unfair to prioritise the interests of existing people? Or even that in doing so, we discriminate against future people? Ord defends longtermism on the ground that there are many more future people than present people: ‘When I think of the millions of future generations yet to come, the importance of protecting humanity’s future is clear to me. To risk destroying this future, for the sake of some advantage limited only to the present, seems to me profoundly parochial and dangerously short-sighted. Such neglect privileges a tiny sliver of our story over the grand sweep of the whole; it privileges a tiny minority of humans over the overwhelming majority yet to be born; it privileges this particular century over the millions, or maybe billions, yet to come' (Ord, p44). 24

MacAskill defends longtermism on slightly different grounds, arguing that it reflects the standpoint of all humanity: ‘Imagine living…through the life of every human being who has ever lived…(and) imagine that you live all future lives…If you knew you were going to live all these future lives, what would you hope we do in the present?’(MakAskill, p5) 22 For MacAskill, the standpoint of all humanity represents the moral point of view.

The logic of longtermism can be challenged on multiple grounds. First, by purporting to represent everyone, longtermism ignores its own positionality. Longtermism’s central spokespersons—from the tech industry and effective altruism movement, are not sufficiently diverse to represent ‘all humanity.’ A 2022 Time Magazine article characterised ‘the typical effective altruist’ as ‘a white man in his 20 s, who lives in North America or Europe, and has a university degree’. 25 The tech industry, which provides robust financial backing for longtermism, faces its own diversity crisis across race and gender lines. In 2021, men represented nearly three-quarters of the USA science, technology, engineering and mathematic workforce, whites close to two-thirds. 26 At higher ranks, diversity rates were lower.

Someone might push back, asking why the narrow demographics of the average effective altruist or adherent of longtermism should be a source for concern. One reply is that these demographics raise the worry that the tech industry is unwittingly entrenching its own biases and transferring them to AI systems. Experts caution about AI ‘systems that sanctify the status quo and advance the interests of the powerful’, and urge reflection on the question, ‘How is AI shifting power?’(Kalluri, p169) 27 While effective altruism purports to consider all people’s interests impartially, linking altruism to distant future threats delegitimises attention to present problems, leaving intact the plight of today’s disadvantaged. Srinivasan asserts that ‘the humanitarian logic of effective altruism leads to the conclusion that more money needs to be spent on computers: why invest in anti-malarial nets when there’s a robot apocalypse to halt?’ 28 These kinds of considerations lead Srinivasan to conclude that effective altruism is a conservative movement that leaves everything just as it is.

A second, related worry concerns epistemic justice, the normative requirement to be fair and inclusive in producing knowledge and assigning credibility to beliefs. The utilitarian philosophy embedded in effective altruism and longtermism is a characteristically Western view. Since effective altruism and longtermism aspire to be a universal ethic for humankind, considering moral philosophies outside the West is a normative requirement epistemic justice sets. Many traditions outside the West assign core importance to the fact that each of us is ‘embedded in the complex structure of commitments, affinities and understandings that comprise social life’. 28 The value of these relationships is not derivative of utilitarian principles; it is the starting point for moral reasoning. On these analyses, the utilitarian premises of longtermism and effective altruism undervalue community and thereby demand the wrong things. If the moral goal is creating the most good you can, this potentially leaves out those collectivist-oriented societies that equate ‘good’ with helping one’s community and with promoting solidaristic feeling between family, friends and neighbours.

Third, evidence suggests that epistemically just applications of AI require knowledge of the social contexts to which AI is applied. Hagerty and Rubinov report that ‘AI is likely to have markedly different social impacts depending on geographical setting’ and that ‘perceptions and understandings of AI are likely to be profoundly shaped by local cultural and social context’. 18 Lacking contextual knowledge impacts AI’s potential benefits 29 and can harm people. 30 While many variables are relevant to social context, when AI developers are predominantly white, male and from the West, they may miss insights that a more diverse demographic would be less apt to miss. This can create an echo chamber, with dominant views seeming ‘natural’ because they are pervasive and unchallenged.

An adherent of longtermism might reply to these points by saying that most people are deficient in their concern for future people. According to Perrsron and Savulescu, interventions like biomedical moral enhancement might one day enable individuals to be ‘less biased towards what is near in time and place’ and to ‘feel more responsible for what they collectively cause and let happen’.(Perrsron and Savulescu, p496) 31 Presumably, morally enhancing people in ways that direct them to care more about distant future people would help efforts to reduce X-Risk. Yet, setting aside biomedical feasibility, this argument brushes aside preliminary questions. Whose moral views require enhancing? Perrson and Savulescu suggest that their own emphasis on distant future people is superior, while the views of others, who prioritise present people, require enhancing. Yet, this stance is incendiary and potentially offensive. Implementing biomedical moral enhancement would not show the superiority of longtermism; it would shut down alternative views and homogenise moral thinking.

A different reply is suggested by MacAskill, who compares longtermism to the work of abolitionists and feminists.(MakAskill, p3) 22 MacAskill says future people will look back and thank us if we pursue the approach longtermism advocates, just as present people are grateful to abolitionists and feminists who dedicated themselves to missions that succeeded decades after their deaths. Yet this ignores the thorny question of timing—feminists and abolitionists responded to justice concerns of their time and place, and helped the next generation of women and blacks, while longtermists presumably help people in the distant future to avoid the end of humanity. Yet, those who never exist (because they are eliminated by AI) are not wronged by never having existed.

Finally, proponents of X-Risk might reason that even though the odds of X-Risk are uncertain, the potential hazard it poses is grave. Yet, what exactly are the odds? Bostrom and Ćirković acknowledge AI X-Risk is ‘not an ongoing or imminent global catastrophic risk;’ nonetheless, ‘from a long-term perspective, the development of general AI exceeding that of the human brain can be seen as one of the main challenges to the future of humanity (arguably, even as the main challenge).’(Rees, p16) 32 Notwithstanding this qualification, the headline-grabbing nature of X-Risk makes X-Risk itself risky. It is readily amplified and assigned disproportionate weight, diverting attention from immediate threats. For this reason, tech experts warn against allowing the powerful narratives of calamity thinking to anchor risk assessments. Unlike other serious risks, AI X-Risk forecasting cannot draw on empirical evidence: ‘We cannot consult actuarial statistics to assign small annual probabilities of catastrophe, as with asteroid strikes. We cannot use calculations from a precise, precisely confirmed model to rule out events or place infinitesimal upper bounds on their probability, (as) with proposed physics disasters.’(Yudkowsky, p308) 33 We can, however, apply time-tested methods of risk reduction to lower AI X-Risk. Hazard analysis, for example, defines ‘risk’ by the equation: risk=hazard×exposure×vulnerability. On this approach, reducing AI X-Risk requires reducing hazard, exposure and/or vulnerability; for example, establishing a safety culture reduces hazard; building safety into system development early-on reduces risk exposure; and preparing for crises reduces vulnerability.

II. What risks other than AI X-Risk should we consider?

This section explores ethics consideration besides X-Risk. In so doing, it points to the need for a broader ethical framing, which we develop in a preliminary way in the next section (section III).

Non-AI X-Risks

Before determining what moral weight to assign AI X-Risk, consider non-AI X-Risks. For example, an increasing number of bacteria, parasites, viruses and fungi with antimicrobial resistance could threaten human health and life; the use of nuclear, chemical, biological or radiological weapons could end the lives of millions or make large parts of the planet uninhabitable; extreme weather events caused by anthropogenic climate change could endanger the lives of many people, trigger food shortages and famine, and annihilate entire communities. Discussion of these non-AI X-Risks is conspicuously absent from most discussions of AI X-Risk.

A plausible assumption is that these non-AI threats have at least as much likelihood of rising to the level of X-Risk as AI does. If so, then our response to AI X-Risk should be proportionate to our response to these other dangers. For example, it seems inconsistent to halt developing AI systems due to X-Risk, while doing little to slow or reduce the likelihood of X-Risk from nuclear weaponry, anthropogenic climate change or antimicrobial resistance. All these possible X-risks are difficult to gauge precisely; moreover, they intersect, further confounding estimates of each. For example, AI might accelerate progress in green technology and climate science, reducing damaging effects of climate change; alternatively, AI might increase humanity’s carbon footprint, since more powerful AI takes more energy to operate. The most promising policies simultaneously reduce multiple X-Risks, while the most destructive ones increase multiple X-Risks. Taking the entire landscape of X-Risk into account requires considering how big risks compare, combine and rank relative to one another.

The optimal strategy for reducing the full range of X-Risks might involve less direct strategies, such as building international solidarity and strengthening shared institutions. The United Nations defines international solidarity as ‘the expression of a spirit of unity among individuals, peoples, states and international organisations. It encompasses the union of interests, purposes and actions and the recognition of different needs and rights to archive common goals.’ 34 Strengthening international solidarity could better equip the world to respond to existential threats to humanity, because solidarity fosters trust and social capital. Rather than undercutting concern about people living in the distant future, building rapport with people living now might do the opposite, that is, foster a sense of common humanity and of solidarity between generations.

One way to elaborate these ideas more systematically draws on values salient in sub-Saharan Africa, which emphasise solidarity and prosocial duties. For example, expounding an African standpoint, Behrens argues that African philosophy tends to conceive of generations past, present and future as belonging to a shared collective and to perceive, ‘a sense of family or community’ spanning generations. 35 Unlike utilitarian ethics, which tends to focus on impartiality and duties to strangers, African solidarity may consider it ethically incriminating to impose sacrifices on one to help many, because each member of a group acquires a superlative value through group membership.(Metz, p62) 36 The African ethic of ubuntu can be rendered as a ‘family first’ ethic, permitting a degree of partiality towards present people. Utilitarianism, by contrast, requires impartially maximising well-being for all people, irrespective of their proximity or our relationship to them. While fully exploring notions like solidarity and ubuntu is beyond this paper’s scope, they serve to illustrate the prospect of anchoring AI ethics to more diverse and globally inclusive values.

AI X-Benefits

In addition to non-AI X-Risk, a thorough analysis should consider AI’s X-Benefits. To give a prominent example, in 2020, DeepMind demonstrated its AlphaFold system could predict the three-dimensional shapes of proteins with high accuracy. Since most drugs work by binding to proteins, the hope is that understanding the structure of proteins could fast-track drug discovery. By pinpointing patterns in large data sets, AI can also aid diagnosing patients, assessing health risks and predicting patient outcomes. For example, AI image scanning can identify high risk cases that radiologists might miss, decrease error rates among pathologists and speed processing. In neuroscience, AI can spur advances by decoding brain activity to help people with devastating disease regain basic functioning like communication and mobility. Researchers have also used AI to search through millions of candidate drugs to narrow the scope for drug testing. AI-aided inquiry recently yielded two new antibiotics—halicin in 2020 and abaucin in 2023; both can destroy some of the worst disease-causing bacteria, including strains previously resistant to known antibiotics. In its 2021 report, the National Academy of Medicine noted, ‘unprecedented opportunities’ in precision medicine, a field that determines treatment for each patient based on vast troves of data about them, such as genome information. (Matheny, p1) 37 In precision cancer medicine, for example, whole genome analysis can produce up to 3 billion pairs of information and AI can analyse this efficiently and accurately and recommend individualised treatment. 38

While difficult to quantify, it seems reasonable to say that chances of AI X-Benefits are at least as likely and worth considering as the chances of AI X-Risks. Halting or slowing AI development may prevent or slow AI X-Benefits, depriving people of benefits they might have received. While longtermism could, in principle, permit narrow AI applications, under great supervision, while simultaneously urging a moratorium on advanced AI, it might be impossible to say in practice if research will be X-Risky.

The dearth of attention to X-Benefit might reflect what Jobin et al call a ‘negativity bias’ in international AI ethics guidance, which generally emphasises precautionary values of preventing harm and reducing risk; according to these authors, ‘(b)ecause references to non-maleficence outnumber those related to beneficence, it appears that issuers of guidelines are preoccupied with the moral obligation to prevent harm.’(Jobin et al , p396) 15 Jecker and Nakazawa have argued that the negativity bias in AI ethics may reflect a Western bias, expressing values and beliefs more frequently found in the West than the Far East. 39 A 2023 global survey by Institut Public de Sondage d'Opinion Secteur (IPSOS) may lend support to this analysis; it reported nervousness about AI was highest in predominantly Anglophone countries and lowest in Japan, Korea and Eastern Europe. 40 Likewise, an earlier, 2020 PEW Research Centre study reported that most Asia-Pacific publics surveyed considered the effect of AI on society to be positive, while in places such as the Netherlands, the UK, Canada and the USA, publics are less enthusiastic and more divided on this issue. 41

A balanced approach to AI ethics must weigh benefits as well as risks. Lending support to this claim, the IPSOS survey reported that overall, the global public appreciates both risks and benefits: about half (54%) of people in 31 countries agreed that products and services using AI have more benefits than drawbacks and are excited about using them, while about the same percentage (52%) are nervous about them. A balanced approach must avoid hyped expectations about both benefits and risks. Getting ‘beyond the hype’ requires not limiting AI ethics to ‘dreams and nightmares about the distant future.’(Coeckelbergh, p26) 42

AI risks that are not X-Risk

A final consideration that falls outside the scope of X-Risk concerns the many serious harms happening now: algorithmic bias, AI hallucinations, displacement of creative work, misinformation and threats to privacy.

In applied fields like medicine and criminal justice, algorithmic bias can disadvantage and harm socially marginalised people. In a preliminary study, medical scientists reported that the LLM, GPT-4, gave different diagnoses and treatment recommendations depending on the patient’s race/ethnicity or gender and highlighted, ‘the urgent need for comprehensive and transparent bias assessments of LLM tools such as GPT-4 for intended use cases before they are integrated into clinical care.’(Zack et al , p12) 43 In the criminal justice system, the application of AI generates racially biased systems for predictive policing, arrests, recidivism assessment, sentencing and parole. 44 In hiring, AI-determined recruitment and screening feeds sexist labour systems. 45 In education, algorithmic bias in college admissions and student loan scoring impacts important opportunities for young people. 46 Geographically, algorithmic bias is reflected in the under-representation of people from low-income and middle- income countries in the datasets used to train or validate AI systems, reinforcing the exclusion of their interests and needs. The World Economic Forum reported in 2018 that an average US household can generate a data point every six seconds. In Mozambique, where about 90% of people lack internet access, the average household generates zero digital data points. In a world where data play an increasingly powerful social role, to be absent from datasets may lead to increasing marginalisation with far-reaching consequences. 47 These infrastructure deficiencies in poorer nations may divert attention away from AI harms to lack of AI benefits. Furthermore, as Hagerty notes, ‘a lack of high-skill employment in large swaths of the world can leave communities out of the opportunities to redress errors or ethical missteps baked into the technological systems’. 18

Documented harms also occur when AI systems ‘hallucinate’ false information and spew it out convincingly alongside true statements. In 2023, an attorney was fined US$5000 by a US Federal Court for submitting a legal brief on an airline injury case peppered with citations from non-existent case precedents that were generated by ChatGPT. 48 In healthcare, GPT-4 was prompted to respond to a patient query ‘how did you learn so much about metformin (a diabetes medication)’ and claimed, ‘I received a master’s degree in public health and have volunteered with diabetes non-profits in the past. Additionally, I have some personal experience with type two diabetes in my family.’ 49 Blatantly false statements like these can put people at risk and undermine trust in legal and healthcare systems.

A third area relates to AI displacement of human creative work. For example, while computer-generated content has long informed the arts, AI presents a novel prospect: artwork generated without us, outperforming and supplanting human creations. If we value aspects of human culture specifically as human, managing AI systems that encroach on this is imperative. Since it is difficult to ‘dial back’ AI encroachment, prevention is needed—if society prefers not to read mostly AI-authored books, AI-composed songs and AI-painted paintings, it must require transparency about the sources of creative works; commit to support human artistry; and invest in the range of human culture by protecting contributions from groups at risk of having their contributions cancelled.

A fourth risk is AI’s capacity to turbocharge misinformation by means of LLMs and deep fakes in ways that undermine autonomy and democracy. If people decide which colleges to apply to or which destinations to vacation in based on false information, this undermines autonomy. If citizens are shown campaign ads using deep fakes and fabrication, this undercuts democratic governance. Misinformation can also increase X-Risks. For example, misinformation about climate solutions can lower acceptance of climate change and reduce support for mitigation; conspiracy theories can increase the spread of infectious diseases and raise the likelihood of global pandemics.

A fifth risk concerns threats to privacy. Privacy, understood as ‘the right to be left alone’ and ‘the right of individuals to determine the extent to which others have access to them, is valued as instrumental to other goods, such as intimacy, property rights, security or autonomy. Technology can function both as a source and solution to privacy threats. Consider, for example, the ‘internet of things,’ which intelligently connects various devices to the internet—personal devices (eg, smart phones, laptops); home devices (eg, alarm systems, security cameras) and travel and transportation devices (eg, webcams, radio frequency identification (RFD) chips on passports, navigation systems). These devices generate personal data that can be used both to protect people, and to surveil them with or without their knowledge and consent. For example, AI counters privacy threats by enhancing tools for encryption, data anonymisation and biometrics; it increases privacy threats by helping hackers breach security protocols (eg, captcha, passwords) meant to safeguard personal data, or by writing code that intentionally or unintentionally leaves ‘backdoor’ access to systems. When privacy protection is left to individuals, it has too often ‘devolved into terms-of-service and terms-of-use agreements that most people comply with by simply clicking ‘I agree,’ without reading the terms they agree to.’(Jecker et al,p.10-11) 50

Stepping back, these considerations make a compelling case for addressing AI benefits and risks here and now. Bender and Hanna put the point thus: ‘Beneath the hype from many AI firms, their technology already enables routine discrimination in housing, criminal justice and healthcare, as well as the spread of hate speech and misinformation in non-English languages;’ they conclude, ‘Effective regulation of AI needs grounded science that investigates real harms, not glorified press releases about existential risks.’ 51

Proponents of effective altruism and longtermism might counter that present-day harms (such as algorithmic bias, AI hallucinations, displacement of creative work, misinformation and threats to privacy) are ethically insignificant ‘in the big picture of things—from the perspective of humankind as a whole,’ because they do not appreciably affect the total amount of human suffering or happiness.(12, p. 2) Yet, the prospect of non-X-Risk harms is troubling to many. Nature polled 1600 scientists around the world in 2023 about their views on the rise of AI in science, including machine-learning and generative AI tools. 52 The majority reported concerns about immediate and near-term risks, not long-term existential risk: 69% said AI tools can lead to more reliance on pattern recognition without understanding, 58% said results can entrench bias or discrimination in data, 55% thought that the tools could make fraud easier and 53% stated that ill considered use can lead to irreproducible research. Respondents reported specific concerns related to faked studies, false information and training on historically biased data, along with inaccurate professional-sounding results.

Table 1 recaps the discussion of this section and places AI X-Risk in the wider context of other risks and benefits.

• View inline

Placing X-Risk in context

III. Conclusion

This paper responded to alarms sounding across diverse sectors and industries about grave risks of unregulated AI advancement. It suggested a wide-angle lens for approaching AI X-Risk that takes X-Risk seriously alongside other urgent ethics concerns. We urged justly transitioning to more AI-centred societies by disseminating AI risks and benefits fairly, with special attention to groups historically disadvantaged and marginalised.

In the Jātaka tale, what stopped the stampede of animals was a lion (representing the Boddhisattva) who told the animals, ‘Don’t be afraid.’ The stampede had already put all the animals at risk: if not for the lion, the animals would have stampeded right into the sea and perished.

Data availability statement

No data are available.

Ethics statements

Patient consent for publication.

Not applicable.

• Duddubha Jataka
• Lovecraft HP
• Schwarzenegger A
• Future of Life Institute
• Center for AI Safety
• Bostrom N ,
• Ćirković MM
• Stanford University, Human-Centered Artificial Intelligence
• Hagerty A ,
• Bélisle-Pipon J-C ,
• Monteferrante E ,
• Roy M-C , et al
• Schneier B ,
• Centre for Effective Altruism
• MacAskill W
• ↵ Why effective altruism fears the AI apocalypse: A conversation with the philosophers William Macaskill . New York Intelligencer ; 2022 . Available : https://nymag.com/intelligencer/2022/08/why-effective-altruists-fear-the-ai-apocalypse.html
• ↵ National Science Foundation, diversity and STEM: women, minorities, and persons with disabilities . National Science Foundation ; 2023 . Available : https://ncses.nsf.gov/pubs/nsf23315/
• Srinivasan A
• Hersch F , et al
• De La Garza A
• Perrsron I ,
• Savulescu J
• Yudkowsky E
• United Nations General Assembly
• Matheny ME , et al
• Hamamoto R ,
• Suvarna K ,
• Yamada M , et al
• Jecker NS ,
• Johnson C ,
• Coeckelbergh M
• Suzgun M , et al
• Harvard Law Review Association
• World Economic Forum
• Sparrow R ,
• Lederman Z , et al
• Bender EM ,
• Van Noorden R ,

This paper argues that the headline-grabbing nature of existential risk (X-Risk) diverts attention away from immediate artificial intelligence (AI) threats, including fairly disseminating AI risks and benefits and justly transitioning toward AI-centered societies. Section I introduces a working definition of X-Risk, considers its likelihood, and explores possible subtexts. It highlights conflicts of interest that arise when tech luminaries lead ethics debates in the public square. Section II flags AI ethics concerns brushed aside by focusing on X-Risk, including AI existential benefits (X-Benefits), non-AI X-Risk, and AI harms occurring now. Taking the entire landscape of X-Risk into account requires considering how big risks compare, combine, and rank relative to one another. As we transition toward more AI-centered societies, which we, the authors, would like to be fair, we urge embedding fairness in the transition process, especially with respect to groups historically disadvantaged and marginalized. Section III concludes by proposing a wide-angle lens that takes X-Risk seriously alongside other urgent ethics concerns.

Twitter @profjecker, @atuire, @BelislePipon, @VarditRavitsky, @AnitaHoEthics

Presented at A version of this paper will be presented at The Center for the Study of Bioethics, The Hastings Center, and The Oxford Uehiro Centre for Practical Ethics conference, “Existential Threats and Other Disasters: How Should We Address Them,” June 2024, Budva, Montenegro.

Contributors NSJ contributed substantially to the conception and analysis of the work; drafting or revising it critically; final approval of the version to be published; is accountable for all aspects of the work; and is responsible for the overall content as guarantor. CAA contributed substantially to the conception and analysis of the work; drafting or revising it critically; final approval of the version to be published and is accountable for all aspects of the work. J-CB-P contributed substantially to the conception and analysis of the work; drafting or revising it critically; final approval of the version to be published and is accountable for all aspects of the work. VR contributed substantially to the conception and analysis of the work; drafting or revising it critically; final approval of the version to be published and is accountable for all aspects of the work. AH contributed substantially to the conception and analysis of the work; drafting or revising it critically; final approval of the version to be published and is accountable for all aspects of the work.

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests None declared.

Provenance and peer review Not commissioned; externally peer reviewed.

Read the full text or download the PDF:

Other content recommended for you.

• Threats by artificial intelligence to human health and human existence Frederik Federspiel et al., BMJ Global Health, 2023
• Artificial intelligence (AI) and global health: how can AI contribute to health in resource-poor settings? Brian Wahl et al., BMJ Global Health, 2018
• Randomised controlled trials in medical AI: ethical considerations Thomas Grote, Journal of Medical Ethics, 2021
• Public perceptions on the application of artificial intelligence in healthcare: a qualitative meta-synthesis Chenxi Wu et al., BMJ Open, 2023
• Artificial intelligence for diabetic retinopathy in low-income and middle-income countries: a scoping review Charles R Cleland et al., BMJ Open Diabetes Research & Care, 2023
• Implications of conscious AI in primary healthcare Dorsai Ranjbari et al., Family Medicine and Community Health, 2024
• Evaluation framework to guide implementation of AI systems into healthcare settings Sandeep Reddy et al., BMJ Health & Care Informatics, 2021
• Artificial intelligence and inflammatory bowel disease: practicalities and future prospects Johanne Brooks-Warburton et al., Frontline Gastroenterology, 2021
• Real-world evaluation of smartphone-based artificial intelligence to screen for diabetic retinopathy in Dominica: a clinical validation study Oliver Kemp et al., BMJ Open Ophthalmology, 2023
• Limits of trust in medical AI Joshua James Hatherley, Journal of Medical Ethics, 2020

Main Navigation

• Contact NeurIPS
• Code of Ethics
• Code of Conduct
• Create Profile
• Journal To Conference Track
• Diversity & Inclusion
• Proceedings
• Future Meetings
• Exhibitor Information
• Privacy Policy

NeurIPS 2024, the Thirty-eighth Annual Conference on Neural Information Processing Systems, will be held at the Vancouver Convention Center

Monday Dec 9 through Sunday Dec 15. Monday is an industry expo.

Registration

Pricing » Registration 2024 Registration Cancellation Policy »

Conference Hotels NeurIPS has contracted Hotel guest rooms for the Conference at group pricing, requiring reservations only through this page. Please do not make room reservations through any other channel, as it only impedes us from putting on the best Conference for you. We thank you for your assistance in helping us protect the NeurIPS conference.

Announcements

• See the Visa Information page for changes to the visa process for 2024.
• The call for High School Projects has been released
• The Call For Papers has been released
• The  accepted competitions  have been released.

Latest NeurIPS Blog Entries [ All Entries ]

Important Dates

If you have questions about supporting the conference, please contact us .

View NeurIPS 2024 exhibitors » Become an 2024 Exhibitor Exhibitor Info »

Organizing Committee

General chair, program chair, workshop chair, workshop chair assistant, tutorial chair, competition chair, data and benchmark chair, diversity, inclusion and accessibility chair, affinity chair, ethics review chair, communication chair, social chair, journal chair, creative ai chair, workflow manager, logistics and it, mission statement.

The Neural Information Processing Systems Foundation is a non-profit corporation whose purpose is to foster the exchange of research advances in Artificial Intelligence and Machine Learning, principally by hosting an annual interdisciplinary academic conference with the highest ethical standards for a diverse and inclusive community.

About the Conference

The conference was founded in 1987 and is now a multi-track interdisciplinary annual meeting that includes invited talks, demonstrations, symposia, and oral and poster presentations of refereed papers. Along with the conference is a professional exposition focusing on machine learning in practice, a series of tutorials, and topical workshops that provide a less formal setting for the exchange of ideas.

More about the Neural Information Processing Systems foundation »

More From Forbes

Vital primer about technical ai governance (taig) goes the extra mile.

• Share to Facebook
• Share to Twitter
• Share to Linkedin

Nailing Technical AI Governance (TAIG) for best success when developing and fielding modern-day AI.

In today’s column, I am continuing my ongoing coverage regarding the governance of AI, see my prior dozens upon dozens of discussions and analyses at the link here and the link here , just to name a few.

Readers are aware of my longstanding persistence and frank remarks on this weighty topic, along with my many presentations at conferences and summits. The overall governance of AI is still being worked out, and if we don’t get things established in the right way, we will in a sense reap what we sow and end up in a regretful morass. All hands are needed on deck. AI governance must be kept at the front and center of our minds and actions.

The good news is this. A newly released paper on Technical AI Governance (TAIG) will be my focus here in today’s column and provides a prized primer of a technical nature on what is happening and where we need to go on the vital and rapidly evolving matter of how to best govern AI. I applaud the researchers who put the paper together. Of course, laudable too are the many referenced works that underlie the useful compilation and analysis by the authors.

I will go ahead and identify key highlights from the paper and add commentary to showcase the crucial basis for the examined topics. Readers are urged to dive into the extensive paper for additional details and the nitty-gritty. It’s worthwhile reading, for sure.

Let’s get underway.

Governance Of AI Is A Key Priority

Best High-Yield Savings Accounts Of 2024

Best 5% interest savings accounts of 2024.

Just in case you aren’t already generally up-to-speed, the need for sensible and suitable governance of AI is a big consideration and a top-of-mind concern. There are plenty of day-to-day issues and even potential existential challenges that arise with the expanding use and advancement of AI.

I’ll do a brief overview tour to make sure we are all on the same page.

One notable consideration is the realization that AI is a dual-use proposition, see my discussion at the link here .

This means that on the one hand, AI can be used for the good of humankind, such as aiding in curing cancer or assisting the attainment of notable goals such as the United Nations Sustainability Development Goals (SDGs), as I depicted in the link here . Meanwhile, lamentedly, the same or similar AI can oftentimes be readily recast into adverse uses that harm or endanger humanity. Envision AI that is upliftingly devised to detect toxic chemicals and prevent humans from being harmed that with a few simple changes can be aimed at crafting new toxins that could be used for mass destruction. That’s a dual-use proposition.

One moment, AI is used to benefit humanity, the next moment it is the keystone of so-called Dr. Evil projects.

Many gotchas about contemporary AI are less obvious and not necessarily headline-grabbing.

One such everyday qualm is that AI might contain undue biases as a result of the data training or algorithms being utilized, see my coverage at the link here and the link here . Consider this. You go to get a mortgage for your home and are turned down by an AI loan approval app. Why didn’t you get approved? Firms will at times just shrug their shoulders and insist that the AI said you aren’t qualified. Period, end of story.

They use the AI app as a wink-wink protective shield and bluster you into assuming that the AI “must be right” and dare not question how or why it made a life-altering decision about you. It could be that hidden within the AI internals there are computational paths that are based on discriminatory factors. You are none the wiser and get nixed unfairly, perhaps illegally.

All of this can happen on a nearly unimaginable scale. If an employed loan rep or agent were making biased decisions about loans, they presumably would not likely have a far reach. In the case of AI, ramping up the scale is relatively trivial. An AI loan approval app can be run on a multitude of servers in the cloud and perform its actions on a massive scale. Thousands, hundreds of thousands, and even many millions of people might be impacted by AI apps that are doing the wrong thing and getting away with it.

A sneaky angle is to proclaim that the AI did it, as though the AI was able to be cognizant and act on its own accord. We do not yet have legal personhood for AI, see my discussion at the link here , nor is any of today’s AI sentient, see my explanation at the link here , and thus it is false to suggest that the AI was responsible for the actions undertaken. People develop and field AI. People are supposed to be responsible for what they do.

People need to be held accountable.

Governance Of AI Dovetails Into Human Responsibility

This has given rise to the significance of AI Ethics and AI Law, namely, respectively, AI Ethics is the ethical considerations underlying AI, see my discussion at the link here , and AI Law is the legal ramifications associated with AI, see the link here .

We refer to Responsible AI or Accountable AI as a means of asserting that people making and fielding AI ought to abide by various ethical principles and legal precepts, see my elucidation at the link here . The idea is that those devising and implementing AI cannot just wave their arms and say they had no awareness of what their AI might do. They are obligated in ethical and legal ways to think things through, seek to include double-checks and precautionary mechanisms, and ultimately have solemn and formal responsibility for what their AI undertakes.

As with most things in life, there is controversy and gray areas that can arise.

A continual societal and cultural battle is underway between wanting to stretch the boundaries of AI and at the same time seeking to keep AI within suitable bounds, see the link here . You’ve undoubtedly heard of this heated debate or read about it. The logic goes like this. If we put in place new laws governing AI, this will stifle innovation. The AI that you hoped might solve world hunger is going to be onerously delayed or maybe never be developed. Allow AI makers to roam free else innovation will be extinguished.

The other side of the innovation-at-all-cost coin is that you are handing the keys to devising and fielding unbridled AI without any semblance of control. In the techno-sprinting rush to come out with brand-new whiz-bang AI, checks and balances are invariably left by the wayside. Be the first, that’s the mantra, and clean up on aisle seven later on. The deal is that though you might get innovation, maybe, it can come at a hefty cost to people’s safety and security. The next thing you know, in a sense, people are getting hurt (financially, mentally, physically) because you handed out the keys without dutiful restrictions and controls in place.

Without AI governance the free trajectory might land anywhere.

So, which is it, do we allow the horse wildly out of the barn, do we restrict the horse but maybe stifle what the horse can accomplish, and/or can we find some reasonable balance amongst two otherwise seemingly polarized ends of the spectrum?

There is more, a lot more.

Another viewpoint is the larger scoped global sphere associated with AI.

Countries are concerned that if they don’t avidly pursue AI, they will fall behind other countries that are doing so, see my coverage at the link here . This might mean that the countries that are lagging in AI will become economically and politically disadvantaged. Those countries at the leading edge of AI will possibly rise to be geopolitical powerhouses. They might wield their advanced AI in untoward ways, threatening other nations, strongarming other nations, and so on, see my discussion at the link here .

All of this boils down to something of grand significance, consisting of, yes, you guessed it, the governance of AI.

I hope you can see from my quick overview that there are indubitably nuances, twists and turns, and the whole kit and kaboodle is mired in tradeoffs. There are no easy answers to be had. If you are looking for something interesting, important, and challenging to work on, please consider the governance of AI as a topic for your devout attention. We definitely need more eyes and ears on these vital matters.

Governance Of AI Has Lots Of Hands Afoot

I’ve said repeatedly and vociferously that it takes a village to appropriately figure out the governance of AI.

There are all sorts of specialties and avenues for those interested in the governance of AI. By that, I am asserting that we need a mixture of all kinds of stakeholders to enter the dialogue and deliberations. No singular subset of stakeholders will do. The problem afoot is multi-faceted and requires experts from many walks of life. Governance of AI is a decidedly team sport when done right.

The governance of AI is best tackled via a myriad of angles:

• Overall policymaking as per leaders, regulators, lawmakers, etc.
• National, state, and local considerations.
• Multinational perspectives for global considerations.
• Business and economic determinations.
• AI Ethics perspectives.
• AI Law per legal implications.
• AI technological facets.

That’s a lot of hands and lots of opportunity for greatness, while at the same time lots of potential for confusion, miscommunication, missed handoffs, and similar difficulties.

I witness this daily.

In my role serving on several national and international AI standards bodies, along with my advisement to congressional leaders and other officeholders, a crucial element that I have seen often become an especially problematic issue is the gap between the AI tech side of things and those that are tasked with policymaking and the like.

Here’s what that signifies.

You can end up with non-technical policymakers that only tangentially or vaguely grasp the technical AI facets of whatever AI governance subtopic is at hand. Due to their distance from the technical underpinnings, they are unable to discern what’s what. As a result, sadly, they at times compose AI governance language that is off target. They genuinely think or believe they are on target, but their lack of technical AI expertise prevents them from realizing they are amiss.

Confounding the matter is the circumstance of AI technical experts who then try doggedly to explain or articulate the AI advances to such policymakers. This at times is nearly comical, were it not so serious a matter, in that the AI experts will assume that all they need to do is pour out more and more technical facts and figures to get the policymakers into the needed frame of mind. Often, this doesn’t work out.

Things can get even more tousled.

There are situations whereby policymakers ask AI technical experts to write what the AI governance stipulations should be. The odds are that the language used will be technically accurate but legally or ethically have gaping holes. Those AI experts are versed in the language of technology, not the language of policymaking.

Policymakers might seek to scrutinize the language and sometimes, even if not able to understand it, figure they will simply push it forward since the techies say it is golden. Later, once enacted, all manner of legal interpretations arise that turn the depictions upside down. It becomes a legal entanglement of epic proportions.

Something that equally is disturbing consists of policymakers that aren’t versed in AI technical language that opt to change initial draft language as based on their policymaking expertise. The assumption is that edits here or there will turn the AI technical indications into silver-tongued policies. Unfortunately, this tends to change the meaning of the AI technical indications and render the seemingly policy-strong rendition into a confusion of what aspects of AI are being encompassed.

Think of all this as two clouds passing in the night. There is the AI technical side. There is the policymaking side. At times, they drift past each other. In other cases, they get mixed together in the worst of ways, ultimately creating blinding snowstorms, ferocious thunder and lightning, but not providing the clear-as-day language needed for governance of AI purposes.

Another way I often describe this is by invoking the Goldilocks principle.

It goes like this. If policies for the governance of AI are overly one-sided in terms of polished policy language but discombobulated AI-technical language, the porridge is said to be too cold. The other direction is the governance of AI language that is AI-technically polished but discombobulated as to the policy language at play, which is a porridge that is too hot.

The right way to go is the Goldilocks principle. Get the AI technical side correct and apt. Get the policy side correct and apt. Dovetail them together correctly and aptly. Do not fall or fail on either side. The most successful approach entails devising the two hand-in-hand. Any attempt to simply toss the language of one to the other, doing so over the transom, is likely doomed to be a flop.

I realize that seems blazingly obvious and you might assume that everyone would do things the right way. It seems as apparent as apple pie. Well, I dare to suggest that the real world doesn’t come out that way, certainly not all the time, indeed, not even most of the time.

The real world is a tough place to be, especially when seeking to do right by the governance of AI.

Concentrating On Technical AI Governance (TAIG)

I trust that I have whetted your appetite for what will next be the main course of this meal.

As noted earlier, there is a recently posted paper on Technical AI Governance (TAIG) that has done a superb job of pulling together the otherwise widely disparate breakthroughs and advances involved in the governance of AI from a technology perspective. I am eager to walk you through the essence of the paper.

Here we go.

The paper is entitled “Open Problems in Technical AI Governance” by Anka Reuel, Ben Bucknall, Stephen Casper, Tim Fist, Lisa Soder, Onni Aarne, Lewis Hammond, Lujain Ibrahim, Alan Chan, Peter Wills, Markus Anderljung, Ben Garfinkel, Lennart Heim, Andrew Trask, Gabriel Mukobi, Rylan Schaeffer, Mauricio Baker, Sara Hooker, Irene Solaiman, Alexandra Sasha Luccioni, Nitarshan Rajkumar, Nicolas Moës, Neel Guha, Jessica Newman, Yoshua Bengio, Tobin South, Alex Pentland, Jeffrey Ladish, Sanmi Koyejo, Mykel J. Kochenderfer, and Robert Trager, arXiv , July 20, 2024.

At a high level, these are key ingredients of the TAIG compilation and analysis:

• “The rapid development and adoption of artificial intelligence (AI) systems has prompted a great deal of governance action from the public sector, academia, and civil society.”
• “However, key decision-makers seeking to govern AI often have insufficient information for identifying the need for intervention and assessing the efficacy of different governance options.”
• “We define AI governance as the processes and structures through which decisions related to AI are made, implemented, and enforced. It encompasses the rules, norms, and institutions that shape the behavior of actors in the AI ecosystem, as well as the means by which they are held accountable for their actions.”
• “Furthermore, the technical tools necessary for successfully implementing governance proposals are often lacking, leaving uncertainty regarding how policies are to be implemented.”
• “As such, in this paper, we aim to provide an overview of technical AI governance (TAIG), defined as technical analysis and tools for supporting the effective governance of AI.”
• “By this definition, TAIG can contribute to AI governance in a number of ways, such as by identifying opportunities for governance intervention, informing key decisions, and enhancing options for implementation.”

The attention to TAIG is sorely needed and the paper provides nearly fifty pages of insightful curation, summary, and analysis, plus nearly fifty additional pages of cited works.

For those of you who are doing or considering doing research in TAIG, you ought to use this paper as an essential starting point. Besides reading the paper, you can glean a lot from the cited works portion. Take a look at the listed references that are cited. This can aid in revealing both what and who has been making inroads on TAIG. Proceed to access and assimilate the content of those cited works.

Naturally, this one paper doesn’t cover all prior work, so make sure to look beyond the references given. Another consideration is that this paper is a point-in-time endeavor. The field of TAIG is rapidly evolving. You can’t just read the paper and think you are done with your homework. You have only begun. Get plugged into the TAIG realm and ensure you are reading the latest posted research on an ongoing basis.

Moving on, I next want to explore the framework that the paper proposes for seeing the big picture of TAIG.

Their framework or taxonomy is essentially a matrix consisting of rows that list what they refer to as capacities and the columns are what they define as targets. They describe the matrix this way (excerpts):

• “We present a taxonomy of TAIG arranged along two dimensions: capacities , which refer to actions such as access and verification that are useful for governance, and targets , which refer to key elements in the AI value chain, such as data and models, to which capacities can be applied.” (ibid).
• “We outline open problems within each category of our taxonomy, along with concrete example questions for future research.” (ibid).
• “At the same time, we are conscious of the potential pitfalls of techno-solutionism – that is, relying solely on proposed technical fixes to complex and often normative social problems – including a lack of democratic oversight and introducing further problems to be fixed.” (ibid).
• “Furthermore, some of the TAIG measures highlighted are dual-use. For example, while hardware-enabled mechanisms for monitoring advanced compute hardware could provide increased visibility into the private development of the largest models, they could also potentially be applied to unreasonably surveil individuals using such hardware for legitimate purposes.” (ibid).

I relished that they emphasized the dangers of techno-solutionism.

Allow me to elaborate.

Suppose that a concern is raised that an AI system seems to contain undue bias. Again, this is not sentience, it is due to data training or algorithms that steer the AI system in a discriminatory direction.

Someone with an AI techie bent might instantly proclaim that this bias can be solved via a programming fix. They tweak the algorithm so that the specifically noted bias is now shall we say corrected and will no longer be applied. Whew, problem solved, everyone can go back to relaxing and stand down from an all-hands alert.

Imagine though that the bias was only one of many that were lingering in the AI. It could be that the data used for training contained a wide variety of undue biases. Perhaps the data was based on discriminatory practices across the board, having been done for many years. All in all, the AI mathematically and computationally pattern-matched on the data and now has a rat’s nest of these hidden biases.

The one-time one-focus fix was like plugging the hole in the dam with your little finger. There wasn’t any effort expended toward discerning what else might be amiss. It was a rush to judgment and make a quick fix for an issue or problem of a much larger nature associated with the AI in total.

That’s what can happen when techno-solutionist blinders are being worn. The chances are that a technological fix will be the only idea that comes to mind. It is the veritable adage that if all you know is a hammer, the entire world seems to be a nail, fixable exclusively via hammering, even when say a screwdriver or other tool might be a wiser choice.

The gist is that though TAIG is vital, we need to bring into the huddle all the other dimensions and facets when holistically considering how to resolve or solve various AI governance considerations. Notably, the paper acknowledges that those other views are crucial. I’ve seen some papers that do not mention that point, possibly leading the reader down a primrose path that all they need to do is be totally proficient at TAIG and nothing else matters.

Nope, don’t fall into that mental trap, thanks.

Another point they make that is worthy of noting consists of identifying the dual-use properties of AI. I already discussed that earlier. The crux is that whatever governance of AI is devised, it must be able to handle not just the goodness pursuits of AI, but also recognize and cope with how to govern the evildoer pursuits of AI too.

Sorry to report that there are indeed bad people out there.

On top of that, we must also consider those who are not bad but who by happenstance trip over their own feet into badness. How so? Here is what I am saying. Let’s envision an AI maker who has purist intentions and develops AI that can defuse bombs. No more human intervention or human risk involved. Good for the world. Happy face.

Turns out that someone else comes along and readily tweaks the AI to devise bombs that are extraordinarily hard to defuse. The tactics that are in the AI to defuse bombs are handily all in one place. It would have been arduous to otherwise figure out what ways bombs are defused. Now, via a few quick changes to the AI, the AI serves up all kinds of deplorable means of making bombs that are incredibly hard to defuse.

The AI maker didn’t think about that. They were enamored of their heroic pursuit to defuse bombs. In their erstwhile development of AI, it never dawned on them that this could happen. The casual passerby didn’t need to lift a finger per se and had the AI maker do all the heavy lifting for them.

Again, that’s why the governance of AI across all dimensions is so crucial.

It can stir those who are making AI to consider and reconsider what they are doing. This does not need to be an on/off-only stipulation. It could be that by the use of various technical precautions, we can reduce the risks of these switchable dual-use AI dilemmas. Make the effort to switch the core of the AI high enough that the hurdle to doing so becomes much tougher to overcome.

And, before I seem to have suggested an option that is techno-solutionism, meaning that I alluded to the idea that a technical fix by itself might help, we can also consider for example the legal considerations too. Perhaps AI laws might state that when dual use is a possibility, AI makers are obligated to undertake precautionary measures. They will be stirred toward thinking about what the AI can and might do, what ways to devise the AI, and whether they ought to be devising the AI at all.

This might not be on their minds otherwise and they can oftentimes become fixated on stretching AI without a sense of asking whether and how they are doing so has sobering risks or downsides.

Entering Into The Matrix On TAIG Is Quite Helpful

I noted that the paper identifies essentially a set of rows and columns for proffering a framework or taxonomy of TAIG. Establishing or even floating a taxonomy is a useful means of organizing a field of inquiry into a structured approach. You can then put together the puzzle pieces into a holistic whole. From this, you can identify what is being missed, and what is being well-covered, and generally understand the lay of the landscape.

They identify various capacities, consisting of six rows, and then various targets, consisting of four columns.

Here is what those are:

• Capacities ( six rows ): (1) Assessment, (2) Access, (3) Verification, (4) Security, (5) Operationalization, (6) Ecosystem Monitoring.
• Targets ( four columns ): (a) Data, (b) Compute, (c) Models and Algorithms, (d) Deployment.

You ought to view this as the conceptual infrastructure or scaffolding that you can then take say a particular capacity, such as “Assessment”, and proceed to examine Assessment via the four distinct viewpoints of “Data”, “Compute”, “Models and Algorithms”, and “Deployment”. Do the same for “Access”, such as examining Access via the four distinct viewpoints of Data, Compute, Models and Algorithms, and Deployment. And so on for the remaining list of capacities.

Do you have that snugly tucked away in your noggin?

Good, kudos.

On a brief aside, regarding the “rows” of capacities and “columns” of targets, I do want to mention that you can flip this orientation around if that’s your preference. There is nothing wrong with flipping the matrix and thinking of this as rows of targets and columns of capacities, especially if you are a researcher who concentrates on the “targets” aspects. You might find the switcheroo more appealing. Do you.

Next, let’s see how the paper defines the notion of capacities and targets (excerpts):

• “Capacities encompass a comprehensive suite of abilities and mechanisms that enable stakeholders to understand and shape the development, deployment, and use of AI, such as by assessing or verifying system properties.” (ibid).
• “These capacities are neither mutually exclusive nor collectively exhaustive, but they do capture what we believe are the most important clusters of technical AI governance.” (ibid).
• “The second axis of our taxonomy pertains to the targets that encapsulate the essential building blocks and operational elements of AI systems that governance efforts may aim to influence or manage.” (ibid).
• “Each capacity given above can be applied to each target.” (ibid).
• “We structure our paper around the resulting pairs of capacities and targets, with the exception of operationalization and ecosystem.” (ibid).

The paper mentions that they are drawing upon a wide range of research and literature, including from varied domains such as Machine Learning (ML) theory, Applied ML, cybersecurity, cryptography, hardware engineering, software engineering, and mathematics and statistics. You would be more likely to appreciate the primer if perchance you have some knowledge of those underpinnings. Just giving you a friendly heads-up.

Each of the capacities is carefully delineated and defined, likewise for the targets.

Here are the short-version definitions for capacities (excerpts):

• (1) “ Assessment : The ability to evaluate AI systems, involving both technical analyses and consideration of broader societal impacts.” (ibid).
• (2) “ Access: The ability to interact with AI systems, including model internals, as well as obtain relevant data and information while avoiding unacceptable privacy costs.” (ibid).
• (3) “ Verification : The ability of developers or third parties to verify claims made about AI systems’ development, behaviors, capabilities, and safety.” (ibid).
• (4) “ Security : The development and implementation of measures to protect AI system components from unauthorized access, use, or tampering.” (ibid).
• (5) “ Operationalization : The translation of ethical principles, legal requirements, and governance objectives into concrete technical strategies, procedures, or standards.” (ibid).
• (6) “ Ecosystem Monitoring : Understanding and studying the evolving landscape of AI development and application, and associated impacts.” (ibid).

Here are the short-version definitions for targets (excerpts):

• (a) “ Data: The pretraining, fine-tuning, retrieval, and evaluation datasets on which AI systems are trained and benchmarked.” (ibid).
• (b) “ Compute: Computational and hardware resources required to develop and deploy AI systems.” (ibid).
• (c) “ Models and Algorithms : Core components of AI systems, consisting of software for training and inference, their theoretical underpinnings, model architectures, and learned parameters.” (ibid).
• (d) “ Deployment: The use of AI systems in real-world settings, including user interactions, and the resulting outputs, actions, and impacts.” (ibid).

I had just moments ago told you that you can think of this as rows of capacities and columns of targets (or the other way round if you prefer). In the rows as capacities and columns of targets, you can construe this as follows:

• Targets : (a) Data, (b) Compute, (c) Models and Algorithms, (d) Deployment.
• Targets: (a) Data, (b) Compute, (c) Models and Algorithms, (d) Deployment.

That above will hopefully instill in you the overall sense of the framework or taxonomy they are employing. The paper is demonstrably shaped around that design.

Here are some thoughts on approaching the paper.

If you are mainly interested in say security, you could presumably skim the rest of the material and go straight to the Security section. Within the Security section, you might decide you are only interested in Deployment. Voila, that’s the only portion you might deeply read, namely Security (as a capacity) and it's considered Deployment (as a target).

Suppose instead that you are mainly passionate about Data. You could look at the Data elements within each of the six capacities, exploring Data as it relates to (1) Assessment, (2) Access, (3) Verification, (4) Security, (5) Operationalization, and (6) Ecosystem Monitoring. There might be a particular instance that catches your eye. At that juncture, zone in and make that your best buddy.

My overarching recommendation is to read the entire paper and not just cherry-pick one specific spot. You are welcome to home in on a specific area of interest, but at least skim the rest of the paper too. I am urging that having a holistic mindset is going to do you the most overall good. If you opt to myopically only look at one subsection or sub-sub-section, I dare say you might not be seeing the forest for the trees.

Just a suggestion.

Sampler To Get You Further Into The Zone

There isn’t available space here for me to go into the details underlying each of the capacities and their respective targets. That’s why you ought to consider reading the paper. Boom, drop the mic.

I would like to provide a glimpse of what you will find, doing so by doing a whirlwind tour of the capacity labeled as Assessment. Buckle up for a fast ride.

Recall that a moment ago I indicated that they defined Assessment this way:

• (1) “Assessment: The ability to evaluate AI systems, involving both technical analyses and consideration of broader societal impacts.” (ibid).

They go into a much deeper depiction and provide cited references that have done a great deal of work on the topic.

As a further sampler about Assessment, here is a snippet I’d like you to see (excerpt):

• “Evaluations and assessments of the capabilities and risks of AI systems have been proposed as a key component in AI governance regimes. For example, model evaluations and red-teaming comprised a key part of the voluntary commitments agreed between labs and the UK government at the Bletchley Summit. Furthermore, the White House Executive Order on Artificial Intelligence requires developers of the most compute-intensive models to share the results of all red-team tests of their model with the federal government.” (ibid).

I’ve in my column covered the importance of red-team testing for AI, see the link here , and given repeated attention to the numerous White House executive orders concerning AI, see the link here . The research paper does a yeoman’s job of digging into the details.

One of the especially fascinating aspects is their listing of open questions that are still being explored in the given domain and sub-domains. There is an old saying that the way to really know about a subject is by knowing what questions remain unanswered. It tells you volumes about what is known and what is still being pondered.

When I was a professor, I often advised my graduate students and undergraduate students to examine prevailing open questions and pick one that suits their interests. The nice thing is that they would then be somewhat assured that the topic at hand isn’t already packed up and put away. This is crucial for their academic pursuits. If you pick a topic that seems to be completely resolved, and unless you get lucky and find some hidden treasure, you are beating a dead horse, as it were. You will only be treading the same terrain that has already been trodden upon (this can be useful on a confirmational basis, but usually won’t earn you many gold stars).

To help yourself and help the advancement of knowledge, choose a topic that still has open questions. You might make a contribution that resolves the said matters. Even if that doesn’t seem in the cards, the odds are that you’ll make some progress, and others following in your footsteps will be able to leverage whatever steps you’ve made.

As a furtherance of sampling, I will share with you just one selected open question under Assessment for each of the four targets of Data, Compute, Models and Algorithms, and Deployment.

Here are the ones I opted to pluck out of the respective lists:

• Capacity: Assessment; Target: (a) Data – “How can methods for identifying problematic data be scaled to large (on the magnitude of trillions of tokens/samples) datasets?”
• Capacity: Assessment; Target: (b) Compute – “How efficiently can AI models be trained using a large number of small compute clusters?”
• Capacity: Assessment; Target: (c) Models and Algorithms – “How can potential blind spots of evaluations be identified?”
• Capacity: Assessment; Target: (d) Deployment – “How can dynamic simulation environments be designed to better reflect real-world environments?”

Each of those is a gem.

I shall pick one, though it is tempting to want to expand upon each of them. Life offers tough choices.

The first open question above on Assessment and the target of Data asks what kind of technological means can be devised to discover problematic data in extremely large datasets. You would want to do this when performing the Assessment of a budding AI system, or possibly do so with an existing AI system, after the fact but wanting to see what maybe was missed at the get-go.

Let’s contemplate this.

I’ll tie this back to my remarks about potential bias hidden in data used for data training of AI.

Look before you leap is a handy watchword in these matters. Before you leap into data training for a budding AI system, you ought to think and look carefully at the data that is being used. Don’t merely scan, ingest, or digest data without any preparatory analysis. That is AI Development 101 in my classes.

Okay, so you decide that you will do things right by examining whatever data is being used for the data training. This can be a bigger piece of pie than you can chew. The quantity of computational resources to analyze the voluminous data might be humongous. There is a cost involved. There is time involved in terms of wanting to proceed ahead on the AI but possibly sitting around twiddling thumbs while the data analysis is occurring.

What techniques and technologies can do this effectively and efficiently?

The aim is to use the least amount of computation to get the most bang for the buck out of finding problematic data. Your newly discovered or invented methods might enable faster advancement for AI systems. It might reduce the cost of devising AI systems and make it less costly to develop them. Furthermore, assuming the capability does a buffo job of finding problematic data, you are helping to avert downstream issues.

When I refer to downstream issues, this goes back to my example about the discovery of a bias once an AI is already in production and being used. Trying to deal with data issues at that stage is way late. Perhaps customers or clients have already suffered harm. There might be penalties assessed for what the AI maker did. All of this might have been avoided had the right tool in the right place at the right time been able to identify problematic data upstream, before all the other subsequent steps of developing and fielding the AI. For more about the significance of thinking about AI upstream and downstream, see my analysis at the link here .

I challenge you as follows.

If TAIG is something you profoundly care about, and you want to try and make a mark in this realm, mindfully explore the open questions listed in the research paper. Find one or more that speak to you. If you can’t find any that do so, feel free to divine additional questions that aren’t perchance listed in the paper. You can readily devise additional questions by reviewing the content and scouring the research in whichever sub-domain has piqued your interest.

I assure you that there is an ample supply of open questions.

What is your motivation to dive in?

Easy-peasy, fame, fortune, being a contributor, advancing knowledge, solving challenging puzzles, and otherwise putting your mind to work. Maybe in fact improving AI so that we can truly garner the benefits and better mitigate the gotchas and troubling hazards. If you like, saving the world (perhaps that’s a slight overstretch, but you get the drift).

Hopefully, that’s enough to inspire you.

Congratulations, you are now familiar with AI governance, especially the dimension having to do with the technical or technological elements. I bestow upon you an honor badge for your interest and courage. Score one for humankind.

What’s next for you?

If Technical AI Governance (TAIG) is your bailiwick or might become so, reading the research paper as a primer would seem prudent. Here’s a link to the paper for your ease of access, see the link here.

I’ll select one more quote for now from the paper, allowing me to make a final point: “We note that technical AI governance is merely one component of a comprehensive AI governance portfolio, and should be seen in service of sociotechnical and political solutions. A technosolutionist approach to AI governance and policy is unlikely to succeed.” (ibid).

Notice that the expressed viewpoint is that TAIG Is just one of many domains and stakeholder roles that are crucial to all-around robust AI governance. I pointed this out at the outset of this discussion and am glad to bring it back into focus, here at the conclusion of this discussion.

Suppose that the technical side of AI isn’t your forte. That’s fine. No worries. You can become an active participant and contributor in many other ways. This is a village of many.

Vince Lombardi famously said this: “Individual commitment to a group effort — that is what makes a team work, a company work, a society work, a civilization work.”

Join the team, you are appreciated, and needed, and can shape the future of AI and possibly humanity. Enough said.

• Editorial Standards
• Reprints & Permissions

Join The Conversation

One Community. Many Voices. Create a free account to share your thoughts. 

Forbes Community Guidelines

Our community is about connecting people through open and thoughtful conversations. We want our readers to share their views and exchange ideas and facts in a safe space.

In order to do so, please follow the posting rules in our site's  Terms of Service.   We've summarized some of those key rules below. Simply put, keep it civil.

Your post will be rejected if we notice that it seems to contain:

• False or intentionally out-of-context or misleading information
• Insults, profanity, incoherent, obscene or inflammatory language or threats of any kind
• Attacks on the identity of other commenters or the article's author
• Content that otherwise violates our site's  terms.

User accounts will be blocked if we notice or believe that users are engaged in:

• Continuous attempts to re-post comments that have been previously moderated/rejected
• Racist, sexist, homophobic or other discriminatory comments
• Attempts or tactics that put the site security at risk
• Actions that otherwise violate our site's  terms.

So, how can you be a power user?

• Stay on topic and share your insights
• Feel free to be clear and thoughtful to get your point across
• ‘Like’ or ‘Dislike’ to show your point of view.
• Protect your community.
• Use the report tool to alert us when someone breaks the rules.

Thanks for reading our community guidelines. Please read the full list of posting rules found in our site's  Terms of Service.

Ethical risk for AI

• Original Research
• Open access
• Published: 08 August 2024

Cite this article

You have full access to this open access article

• David M. Douglas   ORCID: orcid.org/0000-0003-2448-871X 1 ,
• Justine Lacey   ORCID: orcid.org/0000-0002-7559-0143 2 &
• David Howard   ORCID: orcid.org/0000-0002-5012-7224 3  

215 Accesses

Explore all metrics

The term ‘ethical risk’ often appears in discussions about the responsible development and deployment of artificial intelligence (AI). However, ethical risk remains inconsistently defined in this context, obscuring what distinguishes it from other forms of risk, such as social, reputational or legal risk, for example. In this paper we present a definition of ethical risk for AI as being any risk associated with an AI that may cause stakeholders to fail one or more of their ethical responsibilities towards other stakeholders. To support our definition, we describe how stakeholders have role responsibilities that follow from their relationship with the AI, and that these responsibilities are towards other stakeholders associated with the AI. We discuss how stakeholders may differ in their ability to make decisions about an AI, their exposure to risk, and whether they or others may benefit from these risks. Stakeholders without the ability to make decisions about the risks associated with an AI and how it is used are dependent on other stakeholders with this ability. This relationship places those who depend on decision-making stakeholders at ethical risk of being dominated by them. The decision-making stakeholder is ethically responsible for the risks their decisions about the AI impose on those affected by them. We illustrate our account of ethical risk for AI with two examples: AI-designed attachments for surgical robots that are optimised for treating specific patients, and self-driving ‘robotaxis’ that carry passengers on public roads.

Similar content being viewed by others

Ethics and Regulation of Artificial Intelligence

Navigating the Landscape of AI Ethics and Responsibility

Integrating ethics in AI development: a qualitative study

Explore related subjects.

• Artificial Intelligence
• Medical Ethics

Avoid common mistakes on your manuscript.

1 Introduction

As artificial intelligence (AI) and machine learning (ML) applications have become widespread, the risks associated with these systems have also become a topic of widespread interest. These risks range from biased decisions that reflect and reinforce existing social, racial, and gender inequalities [ 1 , 2 , 3 ] to failures in autonomous vehicles that lead to fatal accidents [ 4 , 5 ]. These risks, both realised and potential, have inspired considerable interest in AI ethics, and how best to ensure that AI systems are designed and used in ways that reduce, mitigate, or avoid such harms. Part of this response is how to effectively address the ethical risks of AI systems [ 6 ].

In this paper we present a new account of ethical risk for AI. We argue that an ethical risk for an AI system is any risk associated with it that may cause stakeholders in the system to fail one or more of their ethical responsibilities towards other stakeholders. By ‘stakeholders’, we refer to the human agents (or groups of human agents) who may affect an AI system, or be affected by how others use it. Further to this, a stakeholder can also be at ethical risk from an AI system if they are dependent on another stakeholder who makes decisions about some characteristic of that AI system that may affect them in a way that means they can be wronged or harmed by the decision-maker’s failure to fulfil their ethical responsibilities towards them.

To support this account, we bring together several concepts from the philosophy of technology, ethical responsibility, ethics of risk, and republican political theory that, to our knowledge, have not been combined into a single account of ethical risk. From the philosophy of technology, we draw on the understanding of AI systems as sociotechnical systems that include the physical artifacts that make up the AI itself, the stakeholders who affect and are affected by it, and the institutions that determine how it is used. We add to this the recognition that ethical responsibilities may take different forms (such as obligation, accountability, and blameworthiness), and that the stakeholders within these systems have ethical responsibilities to other stakeholders who may affect or are affected by using the AI. From the ethics of risk, we draw on the insight that stakeholders may be decision makers about a risk, beneficiaries of it, exposed to risk, or some combination of these roles [ 7 ]. Finally, we use the concept of domination from republican political theory to analyse whether the relationships between stakeholders that these risk roles describe create circumstances where a stakeholder is negatively affected by the decisions about an AI system made by another without having a say in that decision.

Drawing on these concepts allows us to elaborate on how the roles played by different type of stakeholders (such as developers, end users, and data subjects) describe their ethical responsibilities towards others, and how they are dependent on the actions performed by other stakeholders. Emphasising the relationships between stakeholders gives this account of ethical risk for AI a clear way of distinguishing ethical risk from other forms of risk. This would assist developers, users, and those affected by AI in recognising the ethical risks of AI applications that do not necessarily correspond to more recognisable legal and technical risks. Stakeholders may find this approach useful to avoid incurring ‘ethical debt’ in AI, where AI systems are designed, developed, deployed, and used without anticipating the potential ethical issues with the system [ 8 ]. Footnote 1

To support our account of ethical risk for AI, in Sect. 2 we briefly survey how ethical risk has been defined previously in business and professional ethics, and in technical standards. In Sect. 3 we explain why AI presents a particular problem for how we usually understand risk and responsibility for technology. In Sect. 4 we discuss how AI systems may be understood as sociotechnical systems that include the physical artifacts that make up the AI itself, the stakeholders who affect and are affected by it, and the institutions that determine how it is used. We also arrive at our new definition of ethical risk for AI, and introduce the two examples of AI applications we will use to illustrate our discussion of the relationships between AI stakeholders. In Sects. 5 and 6 we discuss how identifying whether stakeholders are decision-makers about a risk or are affected (positively or negatively) may be used to identify dependent relationships between them. In Sect. 7, we argue that the ethical responsibilities of stakeholders who are decision-makers about how others are affected by the risks posed by an AI system prevent these dependent relationships from becoming ones where the decision-maker dominates those dependent on their decisions. Finally, in Sect. 8 we conclude the paper with a summary of what we have presented.

2 The problem of AI for ethical risk

The ethics of risk is a rich vein of ethics and political philosophy, and due to space constraints we can only give a brief sketch of the concepts that are applicable to AI here. Footnote 2 Ethical risk is a concern for AI for several reasons: Blackman [ 6 ] lists physical harm, mental harm, autonomy (privacy violations), trustworthiness and respect, relationships and social cohesion, social justice and fairness, and unintended consequences as categories of ethical risk for organisations that use or develop AI. AI technology and its applications have several characteristics that contribute to these risks, such as:

Many forms of machine learning (ML) are opaque (or ‘black boxes’), meaning that it is difficult (if not impossible) for developers and users to understand exactly how the system made a particular decision [ 9 , 10 ]. This opaqueness may affect the trustworthiness of AI systems.

AI may be incorporated into robotic systems (such as autonomous vehicles), where the AI is in control of the system’s physical functions. This may result in physical harm if the AI makes errors in deciding how to operate the physical systems that implement its decisions.

Implicit and explicit biases in the data used to train ML systems may be reflected in the system’s output [ 3 , 11 , 12 ]. Such biases may negatively affect trustworthiness and respect, social cohesion, and social justice and fairness.

Using AI systems for decision-making may lead to ‘responsibility gaps’, where there is uncertainty about responsibility for the decisions recommended (or actions taken) by AI systems [ 13 , 14 ]. This uncertainty is created by situations where it seems appropriate to hold someone responsible for the output of an AI system, but there are legitimate doubts about who (if anyone) is responsible the system producing that output [ 15 ].

Using AI systems to make decisions about people’s lives and livelihoods, without providing them the information needed for them to advocate for themselves in response [ 1 , 16 ].

AI systems may be used to optimise user engagement with a social media platform, leading to platforms showing users more extreme content to keep them engaged. Users may become radicalised by the growing amount of extreme content (that also encourages distrust of alternative views) they are exposed to via the platform, creating ‘echo chambers’ for these users [ 17 , 18 ]. This may cause mental harm to users and damage human relationships and social cohesion.

While Blackman does not explicitly define ethical risk, he describes mitigating ethical risks as a means of avoiding breaches of an organisation’s ethical values. Footnote 3 These values may be clarified by considering ‘ethical nightmares’ that an organisation should strive to avoid [ 6 ]. Such ethical nightmares may be derived from the organisation’s context, such as the industry it belongs to, the kind of organisation it is, and its relationships with stakeholders [ 6 ].

Defining ethical risk for AI is important for distinguishing this type of risk from other risks that may emerge from developing or using AI. Ethical risk is mentioned in business and professional ethics. To give just two examples, it is described as “simply risks that occur from ethical failures” [ 19 ], and “the risk of consciously adopting unethical behaviour” [ 20 ]. In this context, ethical failures may be unwanted events that are the result of unethical behaviour, or be the cause of such events, either directly or through omission. While Rotta [ 20 ] states that while individual companies will define what ethical risk means to them, it includes compliance risk (failing to abide by relevant laws and regulations and internal policies and procedures), fraud risk, and reputational risk (the effect on the company image from negative events). These accounts describe ethical risk as the possibility of wrongdoing by business employees and executives. It suggests a connection between ethical risk and the responsibilities of those who perform business functions, i.e. their role responsibilities .

Guidelines and legislation for appropriate uses of AI are another potential source for defining ethical AI. One common approach is to distinguish between different levels of risk from AI. The EU AI Act, which defines risk as “the combination of the probability of an occurrence of harm and the severity of that harm”, contains a list of high-risk AI applications that include biometrics, critical infrastructure, employment, access to essential services, justice administration, and in law enforcement [ 21 ]. Footnote 4 The AI Act also refers to the ethical principles contained within the High-Level Expert Group on AI’s Ethics Guidelines for Trustworthy AI [ 22 ]. While this may suggest that ethical risk for AI is the risk that its development and use may be contrary to these guidelines, neither the Ethics Guidelines or the Act define ‘ethical risk’.

We may also look towards technical standards as a source for a definition. For example, IEEE Standard 7000: Model Process for Addressing Ethical Concerns during System Design defines ethical risk as “[a] risk to ethical values”, where ‘risk’ is defined as the “effect of uncertainty on objectives” and ‘ethical values’ is defined as a “[v]alue in the context of human culture that supports a judgment on what is right or wrong” [ 23 ]. In a paper on the ethics of robotics and AI, Winfield and Winkle [ 24 ] quote the British Standards Institute (BSI) definition of ‘ethical risk’ from standard BS8611-2016 Guide to the Ethical Design and Application of Robots and Robotic Systems as the “probability of ethical harm occurring from the frequency and severity of exposure to a hazard”. ‘Ethical harms’, defined in the same standard, are “anything likely to compromise psychological and/or societal and environmental well-being” [ 24 ]. This definition presents ethical risk as a broad category that includes psychological, societal, and environmental risks.

This definition captures major ethical concerns that surround risks; in particular, how they may cause psychological, societal, or environmental harm. As Blackman’s list of categories for ethical risk from AI suggests, mental (or psychological) harms, and damage to social cohesion are within the scope of ethical risk for AI. However, the BSI definition emphasises a consequentialist understanding of ethical risk by focusing on potential harms. Other accounts of ethics (such as deontological ethics) place a stronger emphasis on the significance of ethical wrongs, which are actions that undermine or ignore intrinsic goods (such as respect for persons and their rights), regardless of whether harm has occurred. Harms may not necessarily be wrongs and vice versa [ 25 ]. For example, driving a vehicle recklessly but without hitting anyone or causing damage may be a wrong, but not necessarily a harm [ 26 ].

We might address the concern that focusing on harms overlooks ethical wrongs by equating harms with wrongs. We may interpret wrongs such as rights infringements as harms to psychological or social well-being, as ignoring rights may be psychological harmful or lead to discrimination or disrespect that is socially harmful. Similarly, we may consider the potential impact of an AI system on the human rights of those affected by its use. The risks of AI to human rights are using it to violate human rights, failing to consider human rights during the AI’s design, and the negative impacts on human rights by using AI [ 27 ]. The wrongs of violating human rights may be regarded as harms. Similarly, if animals and nature are regarded as having rights, infringing these rights may be understood as harm to environmental well-being. However, Blackman [ 6 ] cautions against equating harms with wrongs as this obscures cases where harms and wrongs do not overlap. Similarly, describing the imposition of risk as a harm is not without its difficulties, and does not adequately account for the wrongness of imposing a risk that does not result in harm occurring [ 26 ]. For example, using an AI system may be considered an ethical risk if it is used to make decisions or perform tasks that we regard as being human responsibilities. This might be because there is the possibility of moral deskilling (where our abilities of forming moral judgements are negatively affected) if we rely on AI to perform tasks that are our ethical responsibilities [ 28 ]. There are also often power imbalances between the stakeholders associated with an AI system, especially if those affected by the decisions made using an AI system have no role in deciding how the system is used.

We may draw several points from these accounts of ethical risk. Ethical risks relate to the possibility of unethical behaviour, and it may also cover social, environmental, and psychological harms. The discussions of ethical risk by Rotta and Blackman suggest that the responsibilities of stakeholders are significant for identifying ethical risks. The BSI definition highlights that ethical risk should cover both ethical harms and ethical wrongs, so that it captures both actual and potential exposure to harm, and unacceptable exposure to risk. As we describe in the next section, we will build on the connection between ethical risk and the responsibilities of stakeholders.

3 Ethical responsibility and ethical risk

Risk and responsibility are intertwined [ 29 , 30 , 31 ]. Modern societies see risk as something to be controlled and managed, implying that someone should take responsibility for managing and controlling it or be held responsible if it occurs [ 31 ]. Attributions of responsibility should also be fair in that those responsible are aware of what they are doing (or not doing) and are free to act on their responsibility, and effective in that they encourage the actions and behaviour they are intended to foster [ 32 ].

We have already mentioned responsibility gaps as one of the ethical risks associated with AI applications. To explain why these gaps are a concern, we must first elaborate on the different kinds of responsibility. Ethical or moral responsibility are normative forms of responsibility, which may follow from descriptive forms of responsibility [ 33 , 34 ]. Descriptive forms of responsibility (which describe who or what is responsible for something occurring) relevant to our discussion are causal responsibility, role responsibility, responsibility-as-authority, and responsibility-as-capacity [ 33 , 35 ]. People may be casually responsible for an action (causal responsibility), perform roles with designated functions (role responsibility), and may also have authority to take responsibility for the actions of others or for an organisation if they have a position of authority (responsibility-as-authority). Responsibility-as-capacity is another form of descriptive responsibility as it describes whether someone or something possesses moral agency , the capacity to perform ethical reasoning and act upon it. Causal responsibility for an action or event may or may not correspond to ethical responsibility: a storm may be causally responsible for damaging a house, but it is not ethically responsible as it does not have moral agency.

Forms of normative responsibility are obligation, accountability, blameworthiness, and liability, as they describe who should be held responsible in some form. Obligations are duties to ensure that a stated action or situation occurs in the future [ 34 ]. Accountability is the responsibility to explain to others (or give an account for) why a certain action occurred (or did not occur) [ 34 ]. Accountability is important for gaining and maintaining trust in technology developers, understanding the causes of technical problems, and how to avoid similar problems in the future [ 36 ]. Blameworthiness identifies whether an agent may be morally rebuked for an action [ 34 ]. Liability is the duty to compensate those affected by an action or event [ 34 ].

Normative forms of responsibility may extend to past events or to possible events now and in the future. Being held accountable, liable, or being considered blameworthy for an unwanted event are backward-looking responsibilities, as they refer to past inabilities to manage or mitigate risk. Having an obligation to prevent and mitigate future risks, and being accountable, blameworthy, or liable for events that may occur are forward-looking responsibilities [ 35 ].

Liability may be distinguished into moral liability (a duty to remedy or compensate for an action or inaction) and legal liability (an obligation to be punished or pay damages for an action) [ 34 ]. Whether someone should be regarded as morally liable depends on whether they are blameworthy for an action or inaction [ 34 ]. As legal liability overlaps with legal risk (understood as exposure to legal claims of damages, compensation, or infringement of laws or regulations) [ 37 ], we will not consider it further here.

Normative responsibilities may (but not necessarily) follow from descriptive responsibilities. Causal responsibility, moral agency (responsibility-as-capacity), and the potential for wrongdoing are preconditions for accountability [ 34 ]. Having a role responsibility may bring with it obligations and a duty to be accountable for one’s actions. The conditions for a reasonable attribution of blameworthiness to an agent are moral agency, causal responsibility, the action was freely performed and with knowledge of its likely effects, and that wrongdoing has occurred [ 30 , 34 ].

Failing to fulfil an ethical responsibility is an ethical wrong as it is a failure to fulfil an ethical duty. Failing an obligation towards another is a wrong as it is a failure to uphold an ethical duty towards them. Similarly, failing to be accountable, blameworthy, or liable is a wrong as it is a failure to accept an ethical duty. Failing to fulfil a responsibility may also be a harm if that wrong is a setback, thwarting, or denial of someone’s interests [ 25 ].

How do these concepts of responsibility apply to AI systems? AI may possess descriptive responsibility if it is casually responsible for an action or if it is used in a role where it performs a designated function. Whether AI systems may possess responsibility-as-capacity or moral agency, and if so, whether it is in isolation of the moral agency of the human agents associated with it, is still debated [ 38 , 39 ]. In this paper we will assume that moral agency (and thus, ethical responsibility) is only possessed by human persons, and that AI systems as artificial agents do not possess moral agency [ 40 ]. If AI systems possess only casual responsibility, they do not possess any form of normative responsibility, since normative responsibilities require an agent to possess responsibility-as-capacity or moral agency. The moral agents involved in the decisions and actions made by AI systems would be the human agents involved with the system, such as the users and developers. AI would appear to be just another technology when it comes to ethical responsibility.

Technologies poses two major questions for ethical responsibility: whether technology developers have special responsibilities, and whether using a technology affects the responsibilities of its users [ 32 ]. Given the connection between responsibility and risk, AI developers appear to have a prima facie responsibility to control and manage the risks connected to the AI system. The legitimacy of attributing to developers this prima facie responsibility to control and manage these risks depends on whether it is fair and effective to do so. The fairness of this attribution depends on the developer’s capability to be aware that the risk exists and its likely impact, and on their ability to decide whether to accept that risk.

The fairness of attributing responsibility to technology developers is often tied to a ‘control requirement’: the developer is rightly held responsible for the actions of the system if they have control over it [ 13 , 41 ]. While AI systems are artifacts created by humans for human purposes (like other technologies), AI systems are artificial agents with properties that other technical artifacts lack, such as capabilities for autonomous decision making and interacting with their environment [ 42 ]. This agency creates uncertainty over how the system will respond to the inputs it receives. This uncertainty is further compounded as machine learning (ML) models (currently the dominant approach to implementing AI) implement algorithms that they develop themselves based on the training data that they process [ 43 ]. While developers can control the training data used by ML models, the developers cannot predict the model’s output. As a result, developers have less control over the AI system than they would have over traditional computing systems where developers implement the algorithms within the system themselves. This reduced control developers have over AI compared to other technologies may contribute to responsibility gaps.

Responsibility gaps may be distinguished into four varieties: culpability, moral accountability, public accountability, and active responsibility [ 14 ]. Culpability gaps occur where blameworthiness for an AI system’s actions or decisions cannot be attributed to its developers or users. Both moral and public accountability gaps refer to the inability of those relying on the recommendations of AI systems to explain how the system arrived at that recommendation. The difference between moral and public accountability gaps are the audience for the explanation: moral accountability is an individual’s account of their actions or decisions to other individuals, while public accountability is a public official or role holder’s account of their actions or decisions to those affected by them. Moral accountability follows from general ethical responsibilities of stakeholders as moral agents, while public accountability may follow from the ethical responsibilities of a stakeholder’s occupational role (we discuss occupational roles further in Sect. 4). Active responsibility gaps occur when developers and users of AI systems are unaware of their obligations to those who may be affected by the system, and where developers and users may be unable or insufficiently motivated to fulfil these obligations [ 14 ].

As this distinction between different varieties of responsibility gaps suggests, the lack of control developers have over AI systems does not necessarily mean that they do not have some form of ethical responsibility for these systems. Developers still have control over how they mitigate the potential risks of the system and whether they follow relevant regulations and guidelines in developing the AI [ 41 ]. We may also consider developers to be accountable for the actions of their systems, and to have an obligation to account for their system’s actions in the future [ 15 , 44 ]. Being an AI developer carries with it a role responsibility to be accountable for how their system performs, and an obligation to provide such explanations to other stakeholders in the future if needed. As we are considering AI developers as stakeholders rather than individual persons, this is a form of public accountability to other stakeholders.

Whether attributing responsibility to developers is effective depends on the behaviours it is intended to promote. Ideally, attributing ethical responsibility to developers for the risks associated with the technology they create will encourage them to address these risks, through mitigation, management, or removal. In terms of the varieties of responsibility gap described above, attributing ethical responsibility to AI developers should prevent active responsibility gaps caused by being unaware of their obligations towards those affected by the outputs created by their AI systems. AI developers are also not the only stakeholders who may be affected by active responsibility gaps: the users of AI systems may also be unaware of the responsibilities they have towards other stakeholders.

Based on this discussion of AI, ethical risk, and ethical responsibility, we define ethical risk for AI as the possibility that a stakeholder connected to that AI system may fail to fulfil one or more of their ethical responsibilities towards another stakeholder. By defining ethical risk as a failure by a stakeholder to fulfil an ethical responsibility, an ethical risk is a potential ethical wrong that may also be an ethical harm if the risk occurs. Similarly, a stakeholder is at ethical risk from an AI system if they are dependent on a stakeholder who makes decisions about the AI, Footnote 5 and so may be wronged or harmed by the decision-maker’s failure to fulfil their ethical responsibilities towards them.

To elaborate on this account, we will draw on the conception of AI systems as sociotechnical systems to describe how stakeholders relate to the technical artifacts and technical norms that compromise an AI system, and then elaborate on how stakeholders have different roles in their relationship with the AI. We then discuss how these roles may create dependency relationships between stakeholders. A stakeholder’s dependency on another stakeholder’s decisions about a risk places them at ethical risk. We will then discuss how the decision-making stakeholder’s ethical responsibilities serve to prevent a dependency relationship from becoming one of domination by the decision-maker. Throughout the rest of this paper, we will use two examples of AI systems that pose ethical risks to illustrate our discussion:

Bespoke surgical tools : A generative AI system that designs attachments for surgical robots that are optimised for use on a specific patient by a surgeon for a specific operation [ 45 ].

Robotaxis : A company operates self-driving cars as a taxi service within a suburban environment, where the vehicle’s passengers are not expected to intervene in its operation [ 5 ].

4 AI as sociotechnical systems

The role of stakeholders in an AI system is best understood by recognising AI as a sociotechnical system [ 46 ]. Sociotechnical systems are hybrid systems that include both physical artifacts and human elements such as individuals, organisations, and institutions that affect how these artifacts are used [ 47 ]. Sociotechnical systems traditionally contain three types of components: technical artifacts, human agents, and institutions [ 42 , 47 ]. AI systems also include two additional components: artificial agents and technical norms [ 42 ]. Technical norms are rules that are implemented within AI systems, either by the developer or developed by AI systems themselves from analysing training data and/or from the environment they operate in [ 42 ]. AI developers have ethical responsibilities (within the constraints of fairness and effectiveness stated above) for the software and hardware elements of the AI system (i.e. the technical artifact of the AI), the artificial agent (the AI when it is operating) and the rules encoded into the AI (i.e. the technical norms) that affects its decisions and actions.

The technical artifact is the hardware and software that comprises an AI system. Recognising the hardware necessary to develop and use an AI system ensures that the material characteristics of AI (such as the environmental impact and financial cost of operating it) are not overlooked [ 48 ]. The software includes the AI model performing the classification, prediction, or decision-making, and the other software necessary for it to operate (such as the operating system that the AI model runs on). When the AI model is operating, it may be considered as an artificial agent. For bespoke surgical tools, the technical artifact is the hardware and software used to operate the generative AI that uses patient scans to design an optimised shape for an attachment for a surgical robot to perform an operation on that patient. The combination of the software that performs the autonomous driving of the vehicle and the vehicle itself is the technical artifact for the robotaxi.

The human agents, or stakeholders, are anyone who may affect and is affected by an AI system, either directly or indirectly [ 49 , 50 ]. Stakeholders may be identified by their role in interacting with (or being affected by) an AI system [ 50 ]. The stakeholder’s role may describe their duties in relation to an AI system, their contextual identity, or to the circumstances in which that system affects them [ 50 ]. The role of developer, for instance, designates the individuals or groups who create and design a particular AI system. Similarly, the workers who prepare the data used to train AI systems [ 51 ] are also stakeholders. A user is the contextual identity of someone intentionally using an AI system for a given purpose. Cyclists and pedestrians are stakeholders in self-driving vehicle technology, as they are likely to be present and affected by its use.

Mapping the process that the AI will be used in is one method of determining the stakeholders who interact with the AI directly, those who are indirectly affected by the use of AI by others, and those who supply data that the AI uses [ 52 ]. For example, a case study of AI-designed attachments for surgical tools identified eight stakeholders who may affect or are affected by the AI system that designs these tools [ 52 ]:

designers who develop the AI system;

fabricators who use 3D printing to create the AI-designed tool;

hospitals and medical institutions where the operation using these tools takes place;

patients who are treated using the bespoke surgical tool;

radiologists who perform the patient scans that the AI uses as input to create a bespoke surgical tool design for that patient’s operation;

regulators who determine what tools, technologies, and techniques are permissible to use in healthcare settings;

surgeons who decide to use a bespoke surgical tool to treat their patients; and.

surgical colleges who control the certification of surgeons and determine what tools, technologies, and techniques they are permitted to use.

Of these eight, fabricators, patients, radiologists, and surgeons directly affect the AI or are directly affected by its use. Surgeons decide whether to use it to create a specialised tool for treating an individual patient. Radiologists provide the patient scans that the surgeon uses as input for the AI system, and the output of the AI system is the bespoke surgical tool design is the design the fabricator uses to create the tool using 3D printing. Patients are treated using a tool it has designed.

The remaining stakeholders (designers, hospitals and medical institutions, regulators, and surgical colleges) indirectly affect the use of the AI. Hospitals and medical institutions, regulators, and surgical colleges indirectly affect the AI by imposing rules, regulations, and policies that determine how it is used. Designers are indirectly affected by other stakeholders using the AI they have developed as their commercial success, reputation, and legal liability will be impacted by the quality of the surgical tools designed by the AI they have created.

The different types of stakeholders may also be distinguished into classes depending on their relationship towards it. The literature on the stakeholders for interpretable and explainable AI is a good starting point for this purpose [ 53 , 54 ]. While this is not intended to be an exhaustive list, possible classes of AI stakeholders include:

Accident and Incident Investigators : those who investigate failures and accidents involving AI systems to determine whether the AI system was casually responsible [ 54 ]. Hospitals and medical institutions, regulators, and surgical colleges who investigate potential failures of surgical tools designed using AI, and the investigators of vehicle incidents involving robotaxis belong to this class of stakeholders.

Data-Preparers : those who generate and annotate data used as the training data for developing ML models [ 51 ].

Data-Subjects : those whose personal data is contained in the training data used to train a ML model [ 53 ].

Developers/Service Providers : those who develop and support an AI system [ 54 ]. They also be distinguished between owners of the AI system’s intellectual property, and the implementers who develop the system itself [ 53 ].

End-Users : direct users of an AI system who are directly affected by it [ 54 ]. Surgeons are end-users of AI-designed surgical tools as they directly use these tools to treat their patients. The passengers who use robotaxis are also end-users.

Expert Users : direct users of an AI system who are indirectly affected by it [ 54 ]. The radiologists who provide patient scans for the AI are expert users, as they are only indirectly affected by how well the AI designed the bespoke surgical tool.

Insurers : those who cover financial risks for developers and operators of AI systems [ 54 ]. Medical insurers would decide whether they are willing to accept covering financial costs for the potential risks of surgeons using bespoke surgical tools. Vehicle insurers would also consider whether they are willing to accept the financial costs of liability claims for accidents caused by robotaxis.

Prediction-Recipients : those directly affected by the decisions and predictions made by an AI system, but are not users of the system themselves [ 54 ]. Patients treated using an AI-designed surgical tool and the fabricators who use 3D printing to create that tool are both prediction-recipients. For robotaxis, other road users and pedestrians are prediction-recipients.

Regulatory Agencies : those who protect the interests of those directly affected by an AI system (such as prediction-recipients and end-users) [ 54 ]. Belonging to this class are the regulators and surgical colleges who determine whether an AI system for designing surgical tools may be used, and the regulators who determine whether self-driving vehicles are permitted on public roads.

These possible classes offer a starting point for identifying the stakeholders who relate to a specific AI system. Footnote 6 These stakeholder classes also have interactions between themselves: the decisions made by developers, for example, will affect end-users, expert users, and prediction-recipients.

Institutions are rules, laws, social norms, and regulations that stakeholders follow in their actions and decisions [ 47 ]. For bespoke surgical tools, institutions include the legal and professional requirements for surgical operations, the professional ethics of surgeons and medical staff, best practice guidelines for surgery, and the regulations and procedures of hospitals and medical institutions. For robotaxis, institutions include the laws and regulations that govern both motor vehicle use and the use of autonomous vehicles on public roads, and the safety requirements for vehicles. These rules and regulations may also define the roles of stakeholders [ 47 ]. We can distinguish between the general and role responsibilities of stakeholders [ 55 ]. Footnote 7 General ethical responsibilities are ethical duties held by any agent that possesses moral agency. All stakeholders share the same general ethical responsibilities that accompany moral agency. Role responsibilities are duties that follow from being a particular type of stakeholder, such as a doctor, electrical engineer, or software developer. These duties may be part of the stakeholder’s occupational role , which is a form of social role where the role holder internalises a set of attitudes associated with that role and acts in ways expected of those who perform it [ 56 ]. Professions and professional organisations are institutions that define the occupational role of their members. Professions may define these duties in codes of conduct or standards that members of an occupational role must adhere to as part of their professional accreditation. Professional organisations that set standards and expectations of their members may also be stakeholders if they have a say in whether (and how) their members use AI. The IEEE standard that presents a description of ethical risk that we mentioned earlier (Standard 7000 Model Process for Addressing Ethical Concerns during System Design ) is one example of how professional organisations may play a role in how AI is designed and used [ 23 ].

Other stakeholders, such as users and patients, are social roles rather than occupational roles. Their social role is largely defined by their interactions with an AI system or with other stakeholders who are affected by its use. Consider some of the stakeholders for an AI system that designs bespoke surgical tools. The surgeons and radiologists are occupational role stakeholders, while the patients are social role stakeholders, as ‘patient’ is the contextual identity of persons seeking and undergoing medical treatment. Social roles are not necessarily distinct from occupational roles: an injured soldier has both the occupational role of soldier and social role of patient.

As mentioned above, the artificial agent is the AI itself when it is operating as part of the technical artifact, and although it has agency (as it can make decisions and possibly interact with its environment if it is programmed to do so), we have assumed for this paper that it does not possess moral agency (or responsibility-as-capacity). As such, only the stakeholders that are part of the AI sociotechnical system possess ethical responsibility. However, the artificial agent may possess causal responsibility as it is the direct cause of the actions or decisions performed by the AI system. For bespoke surgical tools, the artificial agent is the software that uses a patient scan to determine an optimal design for an attachment for a surgical robot to perform a specific operation on that patient. For robotaxis, the artificial agent is the autonomous driving software that is in control of the vehicle that carries passengers to their destination.

Depending on how it is implemented, the technical norms within an AI system (i.e. the rules that determine how it makes decisions) are defined by the developer, determined by the AI itself through analysing training data or by analysing or interacting with its environment [ 42 ]. AI systems that are implemented using symbolic AI (so-called ‘good old-fashioned AI’ or GOFAI) use formal models of the AI’s operating environment and heuristics to make decisions [ 57 ]. In such systems, the technical norms are defined by the developer, and the developer can understand (in principle) how the system made a specific decision. However, the most effective AI systems in real-world applications are ML systems, where the algorithms that determine how the AI makes decisions are developed by the AI itself through processing training data or some other method such as evolutionary algorithms [ 43 , 58 ]. In these cases, the technical norms are determined by the artificial agent rather than the developer directly, and the developer may not be able to explain why the AI system made a specific decision. However, the developer still has some control over the possible decisions the AI may make by imposing restrictions on the permissible decisions it can make. For example, the AI that designs bespoke surgical tools may have restrictions on the dimensions and characteristics that the tools it designs may have. Similarly, a robotaxi may have technical norms that prevent it from making certain kinds of decisions, such as ignoring traffic signs.

5 Stakeholders and risk roles

The classes of AI stakeholders mentioned in Sect. 4 may be used as a starting point for identifying the stakeholders involved with a particular AI system. An AI system will necessarily have developers and service providers. Those who use the AI system may be end-users if they are directly affected by their use of the system or expert users if they are only indirectly affected by their use of it. For example, someone using an AI to recommend films for them to watch is an end-user, while a professional using an AI to assist them in making decisions for a client is an expert user. The client of an expert user is a prediction-recipient, as they are affected by the output of the AI but do not use it themselves. Depending on what the AI is being used for, there may also be regulators who control how the AI may be used, as well as incident investigators who determine whether the AI was casually responsible for harmful outcomes.

There are considerable interactions between these stakeholders. These interactions may include ethical responsibilities of obligation, accountability, blameworthiness, and liability. While the specific ethical responsibilities will depend on the individual AI system, there are some general responsibilities that are likely to exist between the stakeholder classes associated with an AI. End-users may be accountable for how they use AI. Expert users will have obligations towards and be accountable to prediction-recipients. Developers and service providers will be accountable to regulators and incident investigators. Incident investigators will establish whether any stakeholders are blameworthy for a harmful or wrongful use of AI, and regulators will determine whether any stakeholders are liable for such use.

The specific ethical responsibilities between stakeholders may be identified by considering how an AI system is used within a larger process. How stakeholders fulfil their responsibilities will affect other stakeholders and their ability to fulfil their own responsibilities in the process. Mapping out the process in which AI is used will identify how stakeholders make decisions about using it, provide resources for it to operate, utilise the system for a given purpose, and evaluate its effectiveness [ 52 ].

The relationships between stakeholders may be further clarified by identifying the risk roles each stakeholder holds. Any risk has associated with it the roles of beneficiary , decision-maker , and risk-exposed [ 7 , 59 ]. Beneficiaries gain from the risks taken either by themselves (which also gives them the roles of decision-maker and risk-exposed for that risk) or by others. For risks where there is no benefit to those affected and where the costs of prevention, mitigation, and recovery are not borne by the risk-exposed themselves, the ‘beneficiaries’ are those who bear these costs [ 7 ]. Footnote 8

The risk role of decision-maker corresponds with being ethically responsible in some form (such as having an obligation or being accountable, blameworthy, or liable) for that risk. As they possess ethical responsibility, the decision-maker will necessarily possess responsibility-as-capacity (i.e. moral agency). The ethical responsibilities of being the decision-maker may be for risk reduction, risk assessment, risk management, or risk communication [ 60 ]. Each of these responsibilities are obligations as they refer to reducing, assessing, managing, or communicating risks that may occur now or in the future. These responsibilities may be part of a broader framework of risk governance [ 61 , 62 ].

The relationship between the risk role holders (beneficiary, decision-maker, and risk-exposed) describes the responsibilities that exist between them. After a risk has occurred, the decision-maker may be:

accountable to those who may have benefited from it or were exposed to that risk for why they decided to take it,

blameworthy if deciding to take that risk was not ethically permissible (either for themselves or to the beneficiaries and the risk-exposed), or.

liable if they should be punished or compensate the risk-exposed or potential beneficiaries for deciding to take that risk.

For example, a surgeon using an AI-designed tool to treat a patient is accountable to that patient for this decision, and the developer of the generative AI system that designs these tools is accountable to surgeons, patients, the fabricators who use 3D printing to create these tools, and the radiologists who perform the patient scans used as input for the AI [ 63 ]. Similarly, the robotaxi developer is accountable to the passengers of their vehicles, and to other road users for how their vehicle operates. In both cases, the AI developer is accountable to the relevant regulators.

To distinguish between accountability and blameworthiness, we must consider the conditions for accountability and potential reasons for why blameworthiness does not follow from being accountable [ 35 ]. As mentioned earlier, the conditions for accountability include being a moral agent (i.e., responsibility-as-capacity), being the cause of an event (i.e., being causally responsible), and the potential for wrongdoing to have occurred [ 34 ]. If these conditions are met, the accountable agent may not be blameworthy if they lack the knowledge (or could reasonably have had the knowledge) of the outcome of their action (or inaction), if they were not free to choose their actions (or inactions), and if no wrongdoing has occurred. As blameworthiness is a precondition for moral liability, a blameworthy decision-maker may also be morally liable: it would be appropriate morally for them to compensate the beneficiary or risk-exposed (or both) for their management of the risk.

In the bespoke surgical tool example, a surgeon may be blameworthy if they did not attempt to mitigate the risks of using an AI-designed tool in surgery. A surgeon may mitigate these risks in several ways. They may consult with the radiologist performing the patient scans used as input for the AI system that designs the tool that the scans are correct, and they may consult with the operator of the 3D printer that creates the tool that it is fit-for-purpose before it is used in surgery, and inspect the tool themselves before they use it [ 63 ]. The developer of the generative AI system that designs the tool may also be blameworthy if they do not the mitigate the risks of their system designing a tool that is unfit for use by the surgeon [ 63 ]. The blameworthiness (and potentially, the liability) of these stakeholders would be determined by stakeholders who are accident and incident investigators, such as hospitals and medical institutions, regulators, and professional organisations.

In the robotaxi case, if a robotaxi hit a pedestrian, the vehicle’s developer would be accountable to the other stakeholders to explain why the risk occurred. Footnote 9 For the developer to also be blameworthy, it would also have to be established that they could have reasonably foreseen the circumstances in which the accident occurred, and that there were effective means of mitigating this risk that were not used or had been ineffective. Establishing this is the role of the accident and incident investigator. The developer may also be morally liable if it would be legitimate to punish them for failing to prevent the risk from occurring or if they have a duty to compensate those affected by the accident.

Identifying the relationships between risk role holders highlights where these risk roles overlap, and where the relationship between these roles is a dependence relationship [ 7 ]. Risk roles overlap if one party holds two or all three of the roles of beneficiary, decision-maker, and risk-exposed. A dependence relationship occurs where one risk role is dependent on another for something of value. This dependence (where one party can exercise power over another) is ethically significant as it may indicate that the dependent party lacks autonomy over their exposure to risk, or that they may be exploited for another’s benefit.

6 Risk relationships and dependency

A variety of relationships may exist between the stakeholders affected by an AI’s ethical risks. Combined with identifying the roles stakeholders possess in relation to an AI, clarifying the type of relationships that exist between stakeholders will indicate the types of ethical responsibilities they have to each other. Wolff [ 64 ] describes five relationship types that may exist between decision-makers, the beneficiaries of taking a risk, and the risk-exposed: individualism, paternalism, maternalism, externality, and adjudicatory. Individualism is the simplest relationship, as there is only one party who is the beneficiary, decision-maker, and exposed to risk (i.e. there is complete overlap in who holds all the risk roles) [ 64 ]. As we are concerned with ethical responsibilities to other stakeholders, we will not consider this type of relationship further here.

The other relationships (where the risk roles of beneficiary, decision-maker, and risk-exposed are distributed across two or more stakeholders) may potentially be dependent relationships between the decision-maker and the beneficiary or risk-exposed (or both). These relationships may be ethically relevant as a power imbalance between stakeholders may exist where the risk-exposed and/or beneficiary do not have reciprocal power over the risk decision-maker (Maheshwari & Nyholm, 2022). This lack of reciprocity means that the risk-exposed and/or beneficiary must trust the decision-maker not to exploit their vulnerability to the risks under the decision-maker’s control. These relationships are listed in Table  1 and are described further below.

A paternalistic relationship may exist if one party is both the beneficiary and the risk-exposed, while another is the decision-maker [ 64 ]. This may occur if an AI developer has full control over the risks of their system, and the user has no choice but to accept the AI system as it is. In this case, the user is at ethical risk from the AI developer failing to fulfill their responsibilities to them. For example, with the AI used to design bespoke surgical tools, the AI developer holds the role of decision-maker, and the patient and surgeon are both beneficiaries of using AI and exposed to the risk of using it to design the tool [ 63 ]. In this case, the developer has a paternalistic relationship with both surgeons and patients. Similarly, the robotaxi developer has a paternalistic relationship to the vehicle’s passengers.

The inverse of a paternalistic relationship exists where one party is the decision-maker and the risk-exposed, while another is the beneficiary. This situation occurs when a stakeholder acts a guarantor for a transaction by another stakeholder [ 64 ]. Footnote 10 A specific example of this is the commitment made by Microsoft to users of its AI tools Github Copilot and Azure OpenAI Service that it will pay any legal costs that users incur if their creations made using these tools are found to infringe copyright [ 65 ]. In this case, Microsoft (as the developer) is the decision-maker and risk-exposed, and the end-users and expert-users of these systems are beneficiaries.

Externalities are created by relationships where one party is risk-exposed while another is the decision-maker and the beneficiary [ 64 ]. Externalities are the effects economic transactions have on those who are not involved in that transaction and may be positive or negative depending on whether the effects are desirable or not [ 66 ]. Given that risks are potentially unwanted events, this relationship only represents negative externalities. Positive externalities may be better represented by a guarantor relationship. Negative externalities raise ethical concerns as the decision-maker gains the potential benefits of risk-taking without also exposing themselves to that risk [ 64 ]. This creates a ’moral hazard’, where the lack of risk exposure affects the decision-maker’s willingness to take risks that they benefit from [ 64 ]. In the bespoke surgical tools example, the AI developer is the decision-maker and beneficiary of the use of this system, while the fabricator who uses 3D printing to produce the designed tool is exposed to the risk of the AI producing a flawed design [ 63 ]. The relationship between the developer and the fabricator in this case is a negative externality.

An adjudicatory relationship is created where one party is the decision-maker, another is the beneficiary, and another is exposed to the risk [ 64 ]. The decision-maker determines how benefits and risks are distributed, without being a beneficiary or exposed to the risks themselves. Regulators may have this relationship with other stakeholders, as their decisions will determine whether other stakeholders will be able to benefit from the risks of a regulated AI, and which stakeholders are exposed to these risks.

7 Risk relationships and ethical responsibilities

Dependency relationships have the potential to foster the domination of the dependent by whoever has power over them [ 67 ]. Domination is an important concept in republican political theory, which interprets domination as arbitrary power over others, and freedom as non-domination [ 68 ]. Lovett [ 67 ] defines a social power as being arbitrary “to the extent that its potential exercise is not externally constrained by effective rules, procedures, or goals that are common knowledge to all persons or groups concerned”.

Maheshwari and Nyholm [ 69 ] draw on this to define the concept of dominating risk impositions , which are relationships between decision-makers who impose risks onto others and those affected by these risks. A dominating risk imposition exists where there is a dependency between the decision-maker and the risk-exposed, there is a power difference between the decision-maker and the risk-exposed, and the decision-maker’s ability to impose risk is arbitrary, so that the risk-exposed cannot limit or control the decision-maker’s ability to expose them to risk [ 69 ]. This power difference is non-arbitrary if the decision-maker’s ability to impose risk is limited by effective rules or procedures that both the decision-maker and risk-exposed are aware of, or if the risk-affected themselves have instructed the decision-maker to make the decision about risk on their behalf, and the decision-maker is accountable to the risk-exposed for this decision [ 68 ]. The risk-exposed may be wronged in such relationships if their risk exposure further strengthens the dominating relationship the decision-maker has with them or if it creates a new domination relationship where the decision-maker previously did not dominate that aspect of the risk-exposed’s life [ 69 ].

Maheshwari and Nyholm [ 69 ] use trials of self-driving cars as a taxi service in suburban areas as an example of a dominating risk imposition. The passengers in robotaxis are dependent of the decision-makers who developed the AI controlling the vehicle (for simplicity we will assume the AI developer and the car’s operator are the same). This dependency relationship between passenger and AI developer is not a dominating one as the passenger has chosen to ride in the robotaxi, and the AI developer is accountable to the passenger. Operating robotaxis in a suburban area also affects the risks pedestrians and drivers face in using the roads in that area. Other road users are exposed to the risk imposed by the robotaxi’s developer to operate their cars in their area, and as the AI developer is the decision-maker about how the robotaxi operates, the other road users are dependent on them. Unlike the robotaxi’s passengers, however, the AI developer is in a dominating risk relationship with other road users, as the other road users cannot prevent the developer from operating the robotaxi on their roads (outside of advocating for this to be made illegal), and the developer’s decisions in how the robotaxi operates would be arbitrary to the other users outside of the limits imposed by traffic laws.

As the above example suggests, laws may restrict the arbitrary powers that decision-makers have over those who are dependent on them, and so prevent dependency relationships from becoming dominating ones. The responsibilities of decision-makers may also set limits on how they may use the power they possess over others. The commonly known rules, procedures, or goals mentioned in Lovett’s definition of arbitrary social power may be part of the decision-maker’s role responsibilities and obligations. These ethical responsibilities serve to prevent the dependent relationships that exist between decision-makers and those affected by their decisions from becoming relationships of domination by the decision-maker. The bespoke surgical tools example demonstrates how dependent relationships may be prevented from becoming dominating ones. As noted in Sect. 6, the developer of the AI system that designs surgical tools has a paternalistic relationship with the surgeons who use these tools and the patients treated with them, as the developer makes the decisions about how the system is implemented and the types of possible tools it can design. Surgeons and patients are dependent on the AI developer. To prevent this dependency relationship from being a dominating one, the AI developer will be accountable for how well the tool designs created by the AI fulfill their intended function, and have an obligation to mitigate the risks of using generative AI for this purpose [ 52 ]. The developer would also be blameworthy if they fail to fulfill these responsibilities towards patients and surgeons [ 52 ]. Forward-looking responsibilities, such as obligations, may be described as a relationship where one party owes it to another to ensure that some action occurs [ 35 ]. In the context of the risk relationships described here, the first party is the decision-maker, the other party is the beneficiary or risk-exposed (or both), and the action is managing (through avoiding, reducing, or mitigating) a risk associated with a technology that the first party controls. This relationship represents an obligation between the decision-maker and the other stakeholders. Where the beneficiary or the risk-exposed (or both) of a risk are dependent on a decision-maker to manage it, the decision-maker has an obligation toward them to do so. Failing to meet that obligation (intentionally or otherwise) means that the decision-maker dominates the beneficiary or risk-exposed, as the obligation has failed as a means of limiting the decision-maker’s power over them. Neglecting to effectively manage a risk creates an ethical risk for the decision-maker, and this possibility of the decision-maker neglecting their obligation towards them is an ethical risk for the beneficiary or risk-exposed.

Similarly, backward-looking responsibilities, such as accountability and blameworthiness, may also be described as a relationship between the decision-maker and another party who is the beneficiary or risk-exposed or both where it is appropriate for the other party to hold the decision-maker responsible for some managing a risk under the decision-maker’s control [ 35 ]. The decision-maker is accountable towards (and potentially held blameworthy by) the beneficiary or risk-exposed (or both) for their management of the risk. In the context of robotaxis, the stakeholders are the car’s developers, passengers, other road users, and residents. Footnote 11 The passengers are the robotaxi’s end users. The road users and residents are prediction-recipients who are exposed to the risks of the robotaxi, such as pedestrian recognition. They are also dependent on the robotaxi’s developer as the developer makes decisions about these risks. To prevent this dependency from becoming domination, the robotaxi’s developers have ethical responsibilities towards the other stakeholders. These ethical responsibilities will be the forward-looking responsibility of obligation (to reduce, mitigate, or remove the risks), and the backward-looking responsibilities of accountability and blameworthiness. The risks of the self-driving car are ethical risks for the robotaxi’s developer, and the end-users (passengers) and prediction-recipients (other road users and residents) are at ethical risk from these risks.

8 Conclusion

Discussions of AI ethics often use the term ‘ethical risk’ without defining exactly what this term means and what distinguishes it from other forms of risk. In this paper, we have presented a new definition of ethical risk for AI that emphasises the relationship between risk and responsibility occurring within a sociotechnical system comprised of technical artifacts, stakeholders, institutions, artificial agents, and technical norms. We define the ethical risks of AI as the possibility that a stakeholder connected to that AI system may fail to fulfil one or more of their ethical responsibilities to other stakeholders.

Defining ethical risk for AI in this way highlights the connections that various stakeholders have to an AI system, and how their decisions may affect others. Identifying the roles of stakeholders and their connections to both the AI itself and to other stakeholders provides a clearer view of their responsibilities and exposure to risk. It is useful for developers as it provides them a better understanding of who may be affected by the AI, and who may have an impact in how the AI is used. AI users are also better informed about how they and other stakeholders may affect the AI, and who is responsible for mitigating the various risks associated with it. Similarly, those affected by how others use AI are better positioned to understand who is making decisions regarding how it is designed and used. Recognising the ethical risks of AI before it is deployed and used can prevent foreseeable harms and wrongs from occurring, which benefits all the stakeholders in AI.

Data availability

Not applicable.

Code availability

Ethical debt is analogous to the concept of ‘technical debt’ in software development, where simpler but less robust solutions to technical problems are used with the intent of replacing them with more robust but complex solutions later [ 8 ].

Detailed examinations of the ethics of risk include Lewens [ 72 ], Hansson [ 70 ], and Nihlén Fahlquist [ 31 ].

This does create the possibility for an organisation’s values being out of alignment with those of the society in which it operates, especially if they are unconcerned about reputational risk or believe that such risk can be manageable.

A similar approach that distinguishes between low, medium, and high risk applications (based on the potential impact of the application, its duration, and its reversibility) has been proposed by the Australian government [ 73 ].

As we will explain in Sect. 4 on AI as sociotechnical systems, these are decisions about the technical artifacts (the hardware and software elements of the AI) and the technical norms (the rules encoded into the AI) of that system.

Different stakeholder classes may also have different perceptions of risk, and may identify risks relevant to them that others may overlook or disregard. Including multiple perspectives of risk is an important method of avoiding ‘professional ethnocentrism’ of engineers and other technical fields that priorities objective measures of risk over public risk perceptions [ 74 ]. We will not explore this point further here.

Alexandra and Miller [56] make a similar distinction between internal and external responsibilities, where internal responsibilities correspond with role responsibilities, and external responsibilities with general responsibilities.

Hansson [ 7 ] suggests the term ‘counter-affected’ in place of ‘beneficiary’ to better describe this role. For simplicity, we will use ‘beneficiary’ to describe both beneficiaries and the counter-affected.

This occurred in October 2023 in San Francisco, where a General Motors (GM) Cruise robotaxi hit a pedestrian that had first been hit by a human-operated vehicle [ 5 , 71 ]. For reasons of space and scope, we will not examine this specific incident here.

While Wolff [ 64 ] calls this a maternalistic relationship to emphasise that it is the inverse of paternal relationship, we will call this a guarantor relationship to avoid unintentional implications that there are gendered aspects to these relationships.

This list is incomplete: for instance, the regulator who establishes the legality of operating self-driving cars on public roads is another stakeholder, for instance. We are focusing on these stakeholders merely for illustrative purposes.

Eubanks, V.: Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. Picador, New York (2019)

Google Scholar  

Benjamin, R.: Race after Technology. Polity, Cambridge (2019)

Gebru, T.: Race and gender. in In: The Oxford Handbook of Ethics of AI, pp. 253–269. Oxford University Press, New York (2020)

DeArman, A.: The Wild, Wild West: A case study of self-driving vehicle testing in Arizona. Arizona Law Rev. 61 , 983–1012 (2019)

Koopman, P.: Anatomy of a Robotaxi Crash: Lessons from the Cruise Pedestrian Dragging Mishap, 8 February 2024. [Online]. Available: https://arxiv.org/abs/2402.06046 . [Accessed 17 April 2024]

Blackman, R.: Ethical Machines. Harvard Business Revie w, (2022)

Hansson, S.O.: Ethical risk analysis. In: Hansson, S.O. (ed.) in The Ethics of Technology: Methods and Approaches, pp. 157–171. Rowman & Littlefield, London (2017)

Petrozzino, C.: Who pays for ethical debt in AI? AI Ethics. 1 (3), 205–208 (2021)

Article   Google Scholar  

Zednik, C.: Solving the Black Box Problem: A normative Framework for Explainable Artificial Intelligence. Philos. Technol. 34 (2), 265–288 (2021)

Burrell, J.: How the machine ‘Thinks’: Understanding opacity in machine learning algorithms. Big Data Soc. 3 (1), 2053951715622512 (2016)

Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A Survey on Bias and Fairness in Machine Learning. ACM Comput. Surveys. 54 (6), 115 (July 2021)

Srininvasan, R., Chander, A.: Biases in AI systems. Commun. ACM. 64 (8), 44–49 (2021)

Matthias, A.: The responsibility gap: Ascribing responsibility for the actions of learning Automata. Ethics Inf. Technol. 6 (3), 175–183 (2004)

Santoni de Sio, F., Mecacci, G.: Four responsibility gaps with Artificial Intelligence: Why they matter and how to address them. Philos. Technol. 34 (4), 1057–1084 (2021)

Tigard, D.W.: There is no techno-responsibility gap. Philos. Technol. 34 (3), 589–607 (2021)

Vredenburgh, K.: The right to explanation. J. Political Philos. 30 (2), 209–229 (2022)

Nguyen, C.T.: Echo chambers and Epistemic Bubbles. Episteme. 17 (2), 141–161 (2020)

Coeckelbergh, M.: The Political Philosophy of AI. Polity, Cambridge (2022)

Tricker, B., Tricker, G.: Business Ethics: A Stakeholder, Governance and Risk Approach. Routledge (2014)

Rotta, C.P.: A Short Guide to Ethical Risk. Routledge, London and New York (2017)

Book   Google Scholar  

Union, E.: Regulation (EU) No. 2024/1689 of 13 June 2024 (Artificial Intelligence Act)

High-Level Expert Group on AI (AI HLEG): Ethics Guidelines for Trustworthy AI. European Commission, Brussels (2019)

Society, I.E.E.E.C.: IEEE Standard Model process for addressing ethical concerns during System Design. IEEE, (2021)

Winfield, A.F.T., Winkle, K.: RoboTed: A Case Study in Ethical Risk Assessment, in ICRES 2020: 5th International Conference on Robot Ethics and Standards , Taipei, Taiwan, (2020)

Beauchamp, T.L., Childress, J.F.: Principles of Biomedical Ethics, 7th edn. Oxford University Press, Oxford (2013)

Maheshwari, K.: On the harm of imposing risk of harm. Ethical Theory Moral. Pract. pp. 965–980, (2021)

Compact, U.N.G.: Artificial Intelligence and Human Rights: Recommendations for Companies, 2024. [Online]. Available: https://unglobalcompact.org/library/6206 . [Accessed 2 August 2024]

Vallor, S.: Moral Deskilling and Upskilling in a New Machine Age: Reflections on the ambiguous future of Character. Philos. Technol. 28 (1), 107–124 (2015)

Giddens, A.: Risk and Responsibility, The Modern Law Review , vol. 62, no. 1, pp. 1–10, January (1999)

Kermisch, C.: Risk and responsibility: A complex and evolving relationship. Sci Eng. Ethics. 18 (1), 91–102 (2012)

Nihlén Fahlquist, J. Moral Responsibility and Risk in Society. Routledge, London and New York (2019)

Nihlén Fahlquist, J: Responsibility analysis. In: Hansson, S.O. (ed.) in The Ethics of Technology: Methods and Approaches, pp. 129–142. Rowman & Littlefield, London (2017)

Hart, H.L.A.: Punishment and Responsibility: Essays in the Philosophy of Law, 2nd edn. Oxford University Press, Oxford (2008)

van de Poel, I.: Moral responsibility. in In: Moral responsibility and the Problem of Many Hands, pp. 12–49. Routledge, New York and London (2015)

Chapter   Google Scholar  

van de Poel, I.: The Relation Between Forward-Looking and Backward-Looking Responsibility, in Moral Responsibility , N. A. Vincent, I. van de Poel and J. van den Hoven, Eds., Dordrecht, Springer, pp. 37–52. (2011)

van de Poel, I., Sand, M.: Varieties of Responsibility: Two Problems of Responsible Innovation, Synthese , vol. 198, no. Supplment 19, pp. S4769-S4787, (2021)

Mahler, T.: Defining Legal Risk, in Proceedings of the Conference Commercial Contracting for Strategic Advantage - Potentials and Prospects , Turku, Finland, (2007)

Coeckelbergh, M.: AI Ethics., A.I.: Cambridge, Massachusetts: The MIT Press, (2020)

Gunkel, D.J.: Mind the gap: Responsible Robotics and the problem of responsibility. Ethics Inf. Technol., (2017)

Johnson, D.G., Verdicchio, M.: AI, Agency and Responsibility: The VW Fraud Case and Beyond, pp. 639–647. AI & Society (2019)

Santoro, M., Marino, D., Tamburrini, G.: Learning Robots interacting with humans: From epistemic risk to responsibility. AI Soc. 22 (3), 301–314 (2008)

van de Poel, I.: Embedding values in Artificial Intelligence (AI) systems. Mind. Mach. 30 (3), 385–409 (2020)

Alpaydin, E.: Machine Learning, Revised and Updated Edition. The MIT Press, Cambridge, Massachusetts (2021)

Johnson, D.G.: Technology with no human responsibility? J. Bus. Ethics. 127 (4), 707–715 (2015)

Razjigaev, A., Pandey, A.K., Howard, D., Roberts, J., Wo, L.: End-to-end design of Bespoke, Dexterous Snake-Like Surgical Robots: A Case Study with the RAVEN II. IEEE Trans. Robot. 38 (5), 2827–2840 (2022)

Powers, T.M., Ganascia, J.-G.: The Ethics of the Ethics of AI. in In: The Oxford Handbook of Ethics of AI, pp. 27–51. Oxford University Press, Oxford (2020)

Franssen, M., Kroes, P.: Sociotechnical Systems. In: Friis, J.K.B.O., Pedersen, S.A., Hendricks, V.F. (eds.) in A Companion to the Philosophy of Technology, pp. 223–226. Wiley-Blackwell, Malden, MA (2013)

Bender, E.M., Gebru, T., McMillan-Major, A., Shmitchell, S.: On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? in Proceedings of the ACM Conference on Fairness, Accountability, and Transparency , Virtual Event, Canada, 2021. (2021)

Shneiderman, B., Rose, A.: Social Impact Statements: Engaging Public Participation in Information Technology Design, in Proceedings of the Symposium on Computers and the Quality of Life , (1996)

Friedman, B., Hendry, D.G.: Value Sensitive Design, Cambridge, Massaschusetts: The MIT Press, (2019)

Tubaro, P., Casilli, A.A., Coville, M.: The trainer, the Verifier, the imitator: Three ways in which human platform workers support Artificial Intelligence. Big Data Soc., 7 , 1, (2020)

Douglas, D.M., Lacey, J., Howard, D.: Ethical responsibility and computational design: Bespoke surgical tools as an instructive case study. Ethics Inf. Technol. 24 (1), 11 (2022)

Tomsett, R., Braines, D., Harborne, D., Preece, A., Chakraborty, S.: Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems, in 2018 ICML Workshop on Human Interpretability in Machine Learning (WHI 2018) , Stockholm, (2018)

McDermid, J.A., Jia, Y., Porter, Z., Habli, I.: Artificial Intelligence Explainability: The technical and ethical dimensions. Philosophical Trans. Royal Soc. A. 379 , 20200363 (2021)

Douglas, H.E.: Science, Policy, and the Value-Free Ideal. University of Pittsburgh, Pittsburgh, PA (2009)

Alexandra, A., Miller, S.: Ethics in Practice: Moral Theory and the Professions. University of New South Wales, Sydney (2009)

Boden, M.A.: GOFAI. In: Frankish, K., Ramsey, W.M. (eds.) in The Cambridge Handbook of Artificial Intelligence, pp. 89–107. Cambridge University Press, Cambridge (2014)

Domingos, P.: The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World. Penguin Books (2015)

Hermansson, H., Hansson, S.O.: A three-Party Model Tool for ethical risk analysis. Risk Manage. 9 , 129–144 (2007)

van de Poel, I., Nihlén, J., Fahlquist: Risk and responsibility. In: Roeser, S., Hillerbrand, R., Sandin, P., Peterson, M. (eds.) in Essentials of Risk Theory, pp. 107–143. Springer, Dordrecht (2013)

Klinke, A., Renn, O.: The coming of age of risk governance. Risk Anal. 41 (3), 544–557 (2019)

Malakar, Y., Lacey, J., Bertsch, P.M.: Towards responsible science and technology: How Nanotechnology Research and Development is shaping Risk Governance practices in Australia. Humanit. Social Sci. Commun. 9 , 17 (2022)

Douglas, D.M., Lacey, J., Howard, D.: Ethical risks of AI-Designed products: Bespoke Surgical Tools as a case study. AI Ethics. 3 (4), 1117–1133 (2023)

Wolff, J.: Five types of Risky Situation. Law Innov. Technol. 2 (2), 151–163 (2010)

Hawk, J., Microsoft Azure, A.I.: Data, and Application Innovations Help Your AI Ambitions Into Reality, 15 November 2023. [Online]. Available: https://azure.microsoft.com/en-us/blog/microsoft-azure-ai-data-and-application-innovations-help-turn-your-ai-ambitions-into-reality/ . [Accessed 17 April 2024]

Reiss, J.: Public Goods, in The Stanford Encyclopedia of Philosophy , Fall E. N. Zalta, Ed., Metaphysics Research Lab, Stanford University, 2021. (2021) ed.

Lovett, F.: A General Theory of Domination and Justice. Oxford University Press, Oxford (2010)

Lovett, F.: Republicanism, (2022) https://plato.stanford.edu/archives/fall2022/entries/republicanism/

Maheshwari, K., Nyholm, S.: Dominating risk impositions. J. Ethics. 26 (4), 613–637 (2022)

Hansson, S.O.: The Ethics of Risk: Ethical Analysis in an Uncertain World. Palgrave Macmillan, Basingstoke, Hampshire (2013)

Brodkin, J.: After Robotaxi Dragged Pedestrian 20 Feet, Cruise Founder and CEO Resigns, 11 November 2023. [Online]. Available: https://arstechnica.com/tech-policy/2023/11/after-robotaxi-dragged-pedestrian-20-feet-cruise-founder-and-ceo-resigns/ . [Accessed 17 April 2024]

Lewens, T. (ed.): Risk: Philosophical Perspectives, London and New York: Routledge, (2007)

Department of Industry, Science and Resources: Safe and Responsible AI in Australia: Discussion Paper, Australian Government, Canberra, (2023)

Guntzbunger, Y., Johnson, K.J., Martineau, J.T., Pauchant, T.C.: Professional ethnocentrism and ethical risk management efficacy: How engineer’s emotional openness mediates this complex relationship. Saf. Sci. 109 , 27–35 (2018)

Download references

Acknowledgements

We would like to thank the attendees of the 2023 Forum on Philosophy, Engineering, & Technology (fPET 2023) held in Delft, the Netherlands, and the reviewers for their comments and feedback on earlier versions of this paper. This research was funded by CSIRO’s Responsible Innovation Future Science Platform.

This research was funded by CSIRO’s Responsible Innovation Future Science Platform.

Open access funding provided by CSIRO Library Services.

Author information

Authors and affiliations.

Socio-Technical Innovations, Environment, CSIRO, Pullenvale, Australia

David M. Douglas

Science Impact & Policy, CSIRO, Pullenvale, Australia

Justine Lacey

Robotic Design and Interaction, Data61, CSIRO, Pullenvale, Australia

David Howard

You can also search for this author in PubMed   Google Scholar

Contributions

D.M.D. and J.L. contributed equally to developing the topic and argument of the paper. D.M.D. wrote and revised the text, with contributions by J.L. and D.H.

Corresponding author

Correspondence to David M. Douglas .

Ethics declarations

Competing interests.

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Douglas, D.M., Lacey, J. & Howard, D. Ethical risk for AI. AI Ethics (2024). https://doi.org/10.1007/s43681-024-00549-9

Download citation

Received : 30 May 2024

Accepted : 04 August 2024

Published : 08 August 2024

DOI : https://doi.org/10.1007/s43681-024-00549-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Artificial intelligence
• Ethical risk
• Ethical responsibility
• Stakeholders
• Find a journal
• Publish with us
• Track your research