AI Ethics at the Edge of Agency: What Philosophy, Theology, and Common Sense Teach Us About the Limits We Must Set on Artificial Intelligence

Last week's edition tracked the strategic debt nations are accumulating by doing nothing on sovereign AI. This week examines why the doing-nothing keeps happening. We are increasingly delegating consequential decisions to systems whose behaviour we may not fully understand, in contexts we cannot always anticipate, without having settled the prior question of what we think we are doing. We are building before we are thinking. That is not a new human failing, but the velocity of this particular iteration makes it harder to recover from the usual way, which is through disaster followed by reflection.

The question of what kind of thing an AI agent is sits at the intersection of several disciplines that do not naturally speak to each other: computer science, moral philosophy, theology, and law. That is precisely why it keeps getting deferred. Technical teams defer to ethicists. Ethicists defer to legislators. Legislators defer to technical teams. Meanwhile, the UK National Cyber Security Centre has published work exploring what it calls "an agentic AI future for cyber defence" [1], envisioning systems capable of detecting and containing threats at machine speed, with a longer-term ambition for automated remediation. The operational rationale is compelling: attackers move at machine speed, and defenders cannot remain at human speed indefinitely. Crucially, however, the NCSC’s vision places these capabilities under the control and authority of their owners. The ethical threshold is therefore not automation itself, but the degree of discretion delegated to a system, the limits placed on that discretion, and who remains responsible when something goes wrong. This is a question philosophy and theology have been examining for centuries. We are just not calling it by its older name.

The older name is moral agency. And the question it raises is not whether a system can perform a task well. It is whether a system can bear responsibility for what it does. Rosalind Picard framed the dependency between these two things in 1997: "the greater the freedom of a machine, the more it will need moral standards." [2] That sentence has not aged. What has changed is that we are now granting machines considerably more freedom, and the moral standards question remains, charitably speaking, open.

Researchers have identified at least 21 distinct possible tests for machine consciousness [3], and this proliferation is itself a signal. Not that machine consciousness is imminent, but that our conceptual frameworks for understanding what we are dealing with are still immature relative to the pace of deployment. Machine bias alone presents 3 categories of root cause:

• data bias
• algorithmic bias
• outcome bias [4]

Each category can produce harm at scale before its underlying mechanism is identified. In sufficiently complex AI systems, aspects of the decision-making process may remain difficult to interpret even for their designers, making effective oversight and accountability all the more important. Henry Kissinger, not a man given to understatement, observed that "we may have created a dominating technology in search of a guiding philosophy." [2] The philosophy has still not arrived.

There is also a more immediate prerequisite to responsible agency: the quality of the information on which decisions are based. In cybersecurity, an agent cannot reliably interpret its environment without timely, high-fidelity, normalized data and sufficient context to distinguish genuine threats from benign activity. Poor evidence undermines even the most sophisticated decision-making system. Yet reliable data, however necessary, does not resolve the separate ethical question of whether a system should be authorized to act on what it discovers.

Pope Leo XIV's encyclical Magnifica Humanitas, published in May 2026 [5], enters this space deliberately. It draws on classical Catholic social teaching's commitment to human dignity and applies it to the question of artificial intelligence with a seriousness that secular governance frameworks rarely match. The encyclical does not treat AI as a threat to be suppressed or a tool to be celebrated. It treats it as a question about what we owe each other, and specifically about who bears responsibility when harm is done at a distance by a system no individual human fully controls. This is the oldest question in ethics. It is the question that every legal system has struggled to answer when the causal chain between intent and harm is long, distributed, or mediated by technology.

The question of responsibility at a distance is not abstract. The Stop Killer Robots coalition, representing more than 250 member organizations operating in over 70 countries, makes it concrete [6]. Autonomous weapons systems, as the campaign's documentation defines them, are systems where what or who will be attacked, and where and when that attack happens, are determined by sensor processing rather than by human judgment. The coalition notes that "machines don't understand context or consequences: understanding is a human capability, and without that understanding, we lose moral responsibility and we undermine existing legal rules." [6] In November 2023, 164 states voted in favor of the first UN General Assembly resolution on killer robots [6], and the UN Secretary General and the International Committee of the Red Cross jointly called for a binding legal instrument by 2026. That deadline is now. There is no binding instrument. The philosophical debate has become a humanitarian emergency, and the emergency has outpaced the negotiation.

What connects the autonomous weapons debate to the governance of enterprise AI agents is not the lethality gradient. It is the accountability gap. When an AI system makes a consequential decision, and harm results, the responsibility gap [3] describes a situation where no human actor is clearly culpable. The AI is not a legal person. The developer may claim the system behaved within its specified parameters. The deployer may claim they followed the vendor's guidance. The operator may have had no visibility into the decision logic at all. This is not merely a hypothetical failure mode. It is a structural risk of deploying autonomous systems without clearly defined responsibility, meaningful oversight, and sufficient visibility into their decisions and actions. It is precisely the kind of risk that both the NIST (National Institute of Standards and Technology) AI Risk Management Framework [7] and the EU AI Act are attempting to address, with varying degrees of success.

The NIST AI RMF, released in January 2023 and currently under revision, is a voluntary framework designed to help organizations incorporate trustworthiness considerations into AI design, development, and use [7]. The EU AI Act approaches the same problem through mandatory risk classification, and its provisions do apply to AI agents, though The Future Society's analysis notes the Act was not originally designed with agents in mind, meaning multi-purpose agents may default to high-risk classification and the technical standards under development are likely to fall short [8]. Both frameworks are serious attempts at institutionalizing limits. Both also share a structural limitation: they are built on risk management logic, which requires you to know in advance what you are managing. For genuinely novel failure modes in agentic systems, that assumption does not hold.

Dr. Samuel Segun's work on value alignment adds a dimension that risk frameworks tend to suppress entirely. The assumption that AI alignment is a neutral technical problem, a matter of getting the system to do what we intend, obscures a prior question about who "we" refers to. Segun argues that "when AI systems are trained predominantly on feedback from Western annotators addressing Western concerns, they inevitably encode these priorities as universal truths." [9] The implication for cybersecurity specifically is direct: an AI agent deployed globally to make autonomous defensive decisions carries embedded value judgments about what constitutes a threat, what constitutes proportionate response, and whose infrastructure is worth protecting. Those judgments are not labeled as such. They are encoded in the training data, surfaced as outputs, and applied as if they were objective.

The alignment problem, viewed through this lens, is not primarily a technical problem. It is a political one about whose values get treated as defaults, and whose get treated as edge cases. Modern large language models achieve alignment primarily through Reinforcement Learning from Human Feedback, which risks encoding majority preferences and reducing distributional pluralism [9]. Segun's proposed remedies include annotation-level transparency, architectural pluralism, and governance-level inclusion, but he also acknowledges the structural barriers: many regions in the Global South lack the resources to develop advanced AI systems and rely significantly on tools not aligned with their cultural and social contexts [9]. What Segun identifies as a form of digital colonization, I would extend further: it can also become a security risk. An AI system whose embedded values or decision criteria do not reflect the context in which it operates may produce outcomes that conflict with its operators’ expectations or the priorities of the organizations it is intended to protect.

The governance frameworks we have built are honest attempts. The NIST AI RMF's four functions provide a workable vocabulary for risk management [10]. The EU AI Act's four pillars of risk assessment, transparency, technical deployment controls, and human oversight design [8] map to real operational concerns. But they cannot answer the prior question, which is what kind of thing an AI agent is. Is it a tool for which its manufacturer bears full responsibility, like a faulty valve? Is it an instrument whose operator bears responsibility, like a weapon? Is it something new enough that our existing categories of responsibility do not cleanly apply? The ethics literature identifies this as a genuine open question [11], not because researchers lack imagination, but because the answer has consequences that different stakeholders have strong incentives to avoid.

In cybersecurity, retaining meaningful human responsibility begins with defining the boundaries of an agent’s authority. Consequential actions should remain subject to appropriate human oversight, decisions should be explainable and auditable, and organizations should retain the ability to intervene or reverse actions wherever possible. These are not merely technical safeguards; they are practical expressions of a moral commitment to remain accountable for the systems we deploy. Yet they do not resolve every question of responsibility. An action can be supervised, documented and reversed without undoing the harm it caused or settling who should bear responsibility for that harm.

What the age of agency actually demands, then, is not simply a better risk framework. It is a prior act of moral imagination: deciding what we are willing to be responsible for, collectively, before we delegate that responsibility to a system that cannot bear it. Theology has been asking this question for a long time. Philosophy has been asking it almost as long. Common sense asks it every time a child is taught that having the power to act does not settle the question of whether to act.

The question we started with was about limits. But limits require a theory of what is being limited. And the harder question, the one that no framework currently answers, is not what limits we should set on artificial intelligence. It is what kind of moral agents we intend to remain once we have set them.

References

1. UK National Cyber Security Centre (NCSC), "Cyber Shield: The path to an agentic AI future for cyber defence," UK National Cyber Security Centre (NCSC), 2024. [Online]. Available: https://www.ncsc.gov.uk/blogs/cyber-shield-the-path-to-an-agentic-ai-future-for-cyber-defence [Accessed 05 Oct. 2026].
2. R. Trumbull, "An Ethicist's Take: What Philosophy Teaches Us About the Limits We Should Be Setting on AI Agents," GeekWire, 2026. [Online]. Available: https://www.geekwire.com/2026/an-ethicists-take-what-philosophy-teaches-us-about-the-limits-we-should-be-setting-on-ai-agents/ [Accessed 05 Oct. 2026].
3. Internet Encyclopedia of Philosophy (IEP), "Ethics of Artificial Intelligence," Internet Encyclopedia of Philosophy (IEP), University of Tennessee at Martin, UNKNOWN. [Online]. Available: https://iep.utm.edu/ethics-of-artificial-intelligence/ [Accessed 2025].
4. National Institute of Standards and Technology (NIST), "Towards a Standard for Identifying and Managing Bias in Artificial Intelligence," NIST Special Publication 1270, National Institute of Standards and Technology, Mar. 2022. [Online]. Available: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.1270.pdf [Accessed 05 Oct. 2026].
5. Pope Leo XIV, "Magnifica Humanitas: On Safeguarding the Human Person in the Time of Artificial Intelligence," The Holy See, May 15, 2026. [Online]. Available: https://www.vatican.va/content/leo-xiv/en/encyclicals/documents/20260515-magnifica-humanitas.html [Accessed 05 Oct. 2026].
6. Stop Killer Robots, "Frequently Asked Questions – Autonomous Weapons and Stop Killer Robots Campaign," Stop Killer Robots, unknown. [Online]. Available: https://www.stopkillerrobots.org/frequently-asked-questions/ [Accessed 2025].
7. National Institute of Standards and Technology (NIST), "AI Risk Management Framework," National Institute of Standards and Technology (NIST), 2023. [Online]. Available: https://www.nist.gov/itl/ai-risk-management-framework [Accessed 2025].
8. The Future Society, "How AI Agents Are Governed Under the EU AI Act," The Future Society, June 4, 2025. [Online]. Available: https://thefuturesociety.org/how-ai-agents-are-governed-under-the-eu-ai-act/ [Accessed 05 Oct. 2026].
9. S. Segun, "Does it Matter Whose Values We Encode in AI?," Global Center on AI Governance, Dec. 9, 2025. [Online]. Available: https://www.globalcenter.ai/research/does-it-matter-whose-values-we-encode-in-ai [Accessed 05 Oct. 2026].
10. National Institute of Standards and Technology (NIST), "Artificial Intelligence Risk Management Framework (AI RMF 1.0)," NIST AI 100-1, National Institute of Standards and Technology, Jan. 2023. [Online]. Available: https://nvlpubs.nist.gov/nistpubs/AI/NIST.AI.100-1.pdf [Accessed 05 Oct. 2026].
11. D. Kulaklıoğlu, "Ethical AI in Autonomous Systems and Decision-making," Human Computer Interaction, Dec. 6, 2024. [Online]. Available: https://discovery.researcher.life/article/ethical-ai-in-autonomous-systems-and-decision-making/cd25e604c0e83eab8ac46e0e4228c26e [Accessed 05 Oct. 2026].

‍