Trustworthy digital support: evidence, rights and human oversight

1. The problem this lesson solves

Lessons 4.1 and 4.2 asked which decision a tool should change and whether a published carbon total can be reconstructed. This lesson asks the oversight question that sits between them: when digital support recommends a factor, fills a gap or drafts a disclosure, what would make reliance justified? In practice the answer is often a slogan. Teams say “human-supervised” because someone clicks Approve. Interfaces show green badges. Throughput targets push reviewers past six hundred items an hour. Vendor updates land silently. The system may still be fluent — and still untrustworthy for the claim it supports.

Trust, trustworthiness and reliance are three different things. Trust is an attitude. Trustworthiness is an evidence claim about fitness for a defined context. Reliance is behaviour. Governance should seek calibrated reliance: use the system when evidence supports use, challenge it when conditions or evidence do not.

2. Trustworthiness as an evidence claim

NIST’s AI Risk Management Framework treats trustworthiness as situational, not as a marketing label. Characteristics such as validity and reliability, safety, security and resilience, accountability and transparency, explainability, privacy enhancement, and fairness with harmful-bias management can conflict: more transparency can expose sensitive data; continuous adaptation can threaten reproducibility; maximising aggregate accuracy can worsen material subgroup performance. Trade-offs must be explicit.

Díaz-Rodríguez and colleagues make the same broader point: trustworthy AI spans lawful, ethical and robust pillars across the lifecycle of actors and processes — not only a single model score. OECD AI Principles add an intergovernmental rights-and-accountability frame. Organisational conformity under ISO/IEC 42001 can support management-system discipline, but a certificate does not prove that this factor matcher is fit for your geographies and categories.

A practical trustworthiness case therefore has a claim (“adequate for defined use”), context (decision, users, affected parties, autonomy, consequence), evidence (tests, controls, documentation, monitoring, human performance), assumptions, rebuttals and a decision: authorize, restrict, redesign or reject. Trusting a model does not establish trust in retrieval corpora, calculation engines, integrations, interfaces, reviewers or downstream actions — those are separate trust boundaries.

3. Transparency, explanation and over-reliance

Different audiences need different transparency. Operators need input evidence, uncertainty and permitted action. Decision owners need performance, residual risk and authority. Affected parties need notice and a challenge path. Assurance needs versions, tests, logs and approvals. Global and local explanations — coefficients, decision paths, feature attributions, counterfactuals, similar examples, natural-language rationales — can aid comprehension. None of them proves causality, correctness, fairness or safe use.

A polished explanation can increase acceptance without improving error detection. Oversight design should test whether explanations improve challenge behaviour, not merely satisfaction. Independent source access and active verification often matter more than persuasive rationale — a finding consistent with human–AI advice research such as Alon-Barkat and Busuioc and automation-bias synthesis such as Romeo and Conti.

Minimum decision explanation still matters: what the output means; which evidence and versions were used; which method and authorised use apply; how uncertainty should be read; which alternatives or abstention existed; known limits; what the reviewer may override, stop or escalate; where to challenge and how corrections propagate.

4. Rights and legitimate interests

Carbon tools still affect people: workers whose travel is analysed, suppliers scored or excluded, communities near optimised schedules, data subjects inside generative drafts, investors and buyers who rely on claims. A system focused on tonnes can still touch privacy, non-discrimination, autonomy, contestability, labour rights, information integrity, safety and environment, and confidentiality.

Turkish practice brings personal-data duties into the same conversation. KVKK’s generative-AI guidance asks controllers to treat personal-data processing across the generative lifecycle under Law 6698 — purpose limitation, transparency, security and data-subject rights are not optional footnotes when a copilot ingests travel or supplier-person fields. The EU AI Act adds a risk-based documentation and oversight vocabulary that vendors increasingly map into diligence packs even for non-EU plants selling into European markets.

Contestability is practical, not rhetorical: notice, accessible explanation, evidence access where lawful, competent independent review, correction authority, timelines, non-retaliation and downstream propagation of corrections. Environmental justice belongs here too — an optimizer may reduce reported emissions while shifting pollution, workload or risk to another site.

5. Human factors and calibrated reliance

Automation bias is over-reliance on automated advice: commission errors follow wrong advice; omission errors fail to act because the system did not flag a problem. Drivers include workload, time pressure, perceived authority, low expertise, high trust, poor feedback and high verification effort. Algorithm aversion — rejecting useful advice after seeing errors — is the opposite failure. Calibrated reliance sits between obedience and rejection.

Interface design decides which failure wins. AI answers preselected by default, recommendations shown before sources, undefined green badges, overrides that require long justification while acceptance is one click, persuasive narratives without verifiable evidence, unlimited high-speed queues and treating disagreement as user failure all push toward rubber-stamping. Better design requires evidence viewing for material cases, calibrated probability with abstention, correction as easy as acceptance, sampled outcome feedback and capacity limits.

Turkish industrial assistants marketed as closed-corpus carbon advisors — illustrated in coverage of tools such as KarbonBot — raise the same human-factors question even when retrieval is constrained: who checks geography, year and applicability under real throughput?

6. Governance, documentation and procurement

NIST AI RMF functions — Govern, Map, Measure, Manage — translate into carbon artefacts: policy and inventory, use-case and impact assessment, TEVV and human-factors evidence, authorization, monitoring, incidents and retirement. ISO/IEC 42001 adds organisational management-system structure. Decision rights must name who may approve intended use, accept performance, set automation thresholds, authorize external claims, stop the system, change vendor or model, and retire the tool.

Documentation that actually supports assurance includes system cards, model cards, datasheets, impact assessments, decision records, oversight plans, TEVV reports, change records and incident records. Vague templates filled after the fact are weak evidence. Contemporaneous logs beat reconstructed memory.

Procurement is where many plants lose the argument. Due diligence must ask exact version and silent-update policy; training, retrieval and telemetry data rights; evaluation coverage for languages, geography and categories; confidence and abstention definitions; export of prompts, outputs, logs and configurations; incident and change notice; tool-calling authority; fallback and exit; and assessment rights. If commercial secrecy blocks required evidence, you should reduce authority, add independent controls, restrict use or reject. CISA’s AI-in-OT principles harden the same point for any assistant that can touch industrial measurement or control pathways: secure integration is a design obligation, not a feature checkbox.

7. Sequencing

You should do the inexpensive, high-leverage work first. Intended use and prohibited use should be defined; affected parties and rights should be mapped; decision rights and stop authority must be set. You should demand model cards, datasheets and evaluation slices for your conditions, and redesign the review interface for calibrated reliance before raising throughput. Versions and revalidation triggers should be pinned in contracts. Only then should autonomy be expanded. Environmental proportionality closes the loop: you should compare enabled carbon benefit with lifecycle impacts of the tool itself and any harms shifted elsewhere.

Unit 3 treats the same evidence chain as a cyber-physical asset under attack, resilience design and incident response.

8. Applied case: LumaWorks carbon copilot

LumaWorks Materials deploys a carbon-management assistant that extracts invoices, recommends emission factors, classifies purchases, estimates missing data, flags anomalies, proposes schedules and drafts disclosure text. Management calls the system “human-supervised” because a sustainability analyst clicks Approve. The interface hides source documents. Confidence appears as a green badge. The reviewer processes about 600 items per hour. Overrides require written justification. Vendor model updates deploy automatically.

Then the incident: the assistant assigns a high-confidence factor to a large procurement category, cutting reported Scope 3. A reviewer notices the retrieved factor applies to a wrong geography. Challenging it means reopening thousands of records and may delay disclosure. The system is technically functioning. The support process is not trustworthy.

9. References

• NIST. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). https://doi.org/10.6028/NIST.AI.100-1
• ISO/IEC. (2023). ISO/IEC 42001:2023 — Artificial intelligence management system. https://www.iso.org/standard/81230.html
• OECD. AI Principles. https://oecd.ai/en/ai-principles
• Díaz-Rodríguez, N., et al. (2023). Connecting the dots in trustworthy artificial intelligence. arXiv:2305.02231. https://doi.org/10.48550/arXiv.2305.02231
• Mitchell, M., et al. (2019). Model Cards for Model Reporting. FAT* conference. https://doi.org/10.1145/3287560.3287596
• Gebru, T., et al. (2021). Datasheets for Datasets. Communications of the ACM. https://doi.org/10.1145/3458723
• Romeo, G., & Conti, D. (2025). Exploring automation bias in human–AI interaction. AI & Society. https://doi.org/10.1007/s00146-025-02422-7
• Alon-Barkat, S., & Busuioc, M. (2022). Human–AI interactions in public administration. Journal of Public Administration Research and Theory. https://doi.org/10.1093/jopart/muac007
• CISA (& partners). Principles for the Secure Integration of Artificial Intelligence in Operational Technology. https://www.cisa.gov/resources-tools/resources/principles-secure-integration-artificial-intelligence-operational-technology
• European Union. Regulation (EU) 2024/1689 (AI Act). https://eur-lex.europa.eu/eli/reg/2024/1689/oj
• KVKK. Üretken Yapay Zekâ ve Kişisel Verilerin Korunması Rehberi. https://www.kvkk.gov.tr/Icerik/8547/uretken-yapay-zeka-ve-kisisel-verilerin-korunmasi-rehberi-15-soruda