Mental Models 4 Life

AI governance and AI assurance are sometimes conflated in conversations but they are not the same thing. In simple terms,

• An AI Governance Framework sets the Rules and Structure (the What and Who) and establishes the overall system for decision-making and accountability.
• An AI Assurance Framework is there to demonstrate Compliance and Trust (the How); it provides the practical mechanisms to verify that the rules and principles set by an AI Governance Framework are being followed.

In more details, the AI Governance Framework for an organisation is the strategic blueprint that defines the organisation’s overarching approach to developing, deploying, and managing AI systems. Its purpose is to define the roles, responsibilities, policies, and ethical principles that guide the organisation’s use of AI. It ensures alignment with organisational values, legal requirements, and societal expectations (e.g., fairness, accountability, transparency). Key components of an AI Governance Framework include

• Ethical and Responsible AI Principles: Establishing core values (e.g. human-centricity, fairness).
• Governance Structures: Defining committees, roles (e.g. AI Accountable Officers), and lines of reporting / accountability.
• Policies: Written rules on data management, privacy, security, third-party AI procurement, etc.
• Risk Management Strategy: Establishing the overall approach to identifying and mitigating AI risks.

In contrast, the AI Assurance Framework for an organisation is the operational mechanism used to evaluate and verify, through a body of evidence, whether a specific AI system being developed or used is operating safely, reliably, and compliant with the rules set by the governance framework. It is focused on practical assessment and verification. Core Components of an AI Assurance Framework include

• Self-Assessment / Impact Tools: Templates and questionnaires to assess the risks and impacts of an AI project.
• Evaluation Metrics: Defined measurements for trustworthiness characteristics (e.g. bias detection, safety metrics, explainability scores).
• Test and Evaluation Processes: Procedures for testing, monitoring performance, and documenting human oversight and intervention mechanisms.
• Compliance Evidence: Generating reports and audit trails that prove the AI system meets regulatory and internal standards.

Note, in particular, that an AI Assurance Framework is not necessarily there to assure that an AI system will provably perform to a certain specification. While there are cases where Formal Methods have been successfully used to provide mathematical proofs of correctness for chip designs [Dre04] and even complex software like an operating-system micro-kernel [KEH+09], this is a high bar that is currently infeasible to achieve for many AI systems — see the discussion below on How is AI Different. Thus, I think an AI assurance framework that is practical to achieve for most organisations today will capture the process of

1. identifying risks that may prevent an AI system from performing to a desired specification, usually through well-established threat-modelling methodologies [Sho14];
2. introducing, where and when necessary, additional controls to mitigate risks that cross a certain organisational risk-tolerance threshold;
3. collecting evidence on the effectiveness of the risk mitigations and articulate the residual risks; and finally
4. having the accountable officer or system owner accept the residue risks.

This assurance process is similar to how system accreditation is performed using a risk-based approach as prescribed in Policy 9 of the Australian Government’s Protective Security Policy Framework (PSPF). It is also consistent with the ISO 27000 family of standards, where assurance is defined as “confidence that the information security requirements have been fulfilled”. ISO 27001, in particular, describes how assurance is provided through Risk Assessment and Treatment (clause 6.1.2/3), Internal Audit (clause 9.2), Management Review (clause 9.3), and finally certification by an independent accredited third party.

On the rigour expected in AI test and evaluation processes, ISO 15026 on Systems and Software Assurance provides good guidance on what it means to provide “grounds for justified confidence that a claim has been or will be achieved”. Grounds for justified confidence need not be in the form of a mathematical proof. Evidence can encompass testing, analysis, simulations, or other methods suitable for establishing properties like safety and reliability. The focus on ISO 15026 is on consistency, comparability, and auditable artifacts rather than exhaustive verification (a la Formal Methods verification). The use of ISO 15026 for AI systems, I think, requires careful considerations of what are actual Claims (formally defined) that can be reasonably put on AI systems. To give an obvious example, it is unlikely that we will ever be able support a claim like “AI system X will achieve 100% accuracy even when there is distribution drift in the deployment environment”, but we ought to be able to design and provide evidence for a more realistic claim like “AI system X will notify its user when a distribution drift in the deployment environment is detected and seek approval to continue operating”. When it comes down to it, users of AI systems need to have realistic expectations of the performance characteristics of such systems, and system developers have an active role to play in communicating and managing user expectations around what is achievable and not achievable with current AI technologies.

How is AI Different – In his blog post that introduced the term Software 2.0, Andrej Karpathy writes that unlike the ‘classical stack’ of Software 1.0 where a programmer writes explicit instructions in languages like Python, C++, etc that are then compiled into a binary that performs useful work, Software 2.0 is written by computers in much more abstract, human unfriendly language, such as the weights of a (large) neural network. In particular, in Software 2.0 or AI-enabled software, the human-supplied source code usually comprises

1. a high-level statement of what a good program looks like in the form of a mathematical optimisation problem;
2. a dataset of good and sometimes bad examples of the program’s desired behaviour, and
3. a neural network architecture, usually with many layers and up to billions of parameters, that gives the rough skeleton of the code.

The computer then proceeds to solve the given optimisation problem to find a good program, in the form of actual weights for the given neural network architecture, that exhibits the desired behaviour. This strategy, somewhat surprisingly, has given rise to remarkable progress across a range of hard problems, from Go to protein folding and self-driving cars. Across many application areas, in Karpathy’s words, we are now thus left “with a choice of using a 90% accurate model we understand, or a 99% accurate model we don’t.” In particular, programs generated using the Software 2.0 methodology can have unexpected failure modes and a wider range of attack vectors compared to traditional software. Given this state of affairs, the use of modern AI software can realistically only be risk-managed using the Defence in Depth strategy, where we accept that no single defence mechanism can be foolproof so we introduce multiple layers of orthogonal and overlapping protective controls to minimise the risk of adversarial outcomes. AI Assurance Frameworks are best designed with that in mind.

Acknowledgement: Parts of this document was written using Perplexity and Gemini.

References

[Dre04] Rolf Drechsler. Advanced Formal Verification. Springer, 2004.

[KEH+09] Gerwin Klein et al, seL4: Formal verification of an OS kernel. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, pages 207–220, 2009.

[Sho14] Adam Shostack. Threat modeling: Designing for security. John Wiley & Sons, 2014.