AI in industry: Shadow AI, policy and ISO 42001
Published
What the tool produced
A plant worker photographs a pressure gauge and asks an AI tool for the reading. The tool answers with a number, in units, in a confident tone. The worker copies it into a report.
The tool reads digits and labels well. It errs on the needle position. This is a G2 failure, analog gauge.
The reading enters a signed report. Nobody asked for it to be checked, because nobody knew the tool was in the room.
The evidence for this failure is weaker than for the others. It rests on work that is still under review, and we cite the pattern and not the numbers. A benchmark for visual measurement reading found that leading models struggle to read measurements in general.
What Shadow AI is
Workers in plants already use these tools, and the output looks right even when it is not. When there is no policy, the use happens anyway. It is just not documented.
You can call it Shadow AI: use of an AI tool that the organization has not tagged, not approved and does not check. The name comes from the market. The problem is an engineering one.
Without tagging, there is no answer to the auditor's question: who approved this, and on what basis.
Why it looked right
The reading looks like any other reading. Nothing in the output gives an external sign of error. This is common to all seven failure types.
So you cannot rely on the worker's suspicion. The check must run on every item and leave a record, even when the worker is sure.
How the engineer catches it
The organizational solution is a policy that says three things: what is allowed, what is forbidden, and who checks. The scale has five levels.
- Level zero, no tool.
- Level one, ideas only. All the content is written by hand.
- Level two, editing of text that was written by hand.
- Level three, verified draft. Every item passes the gate.
- Level four, full use, only with written approval from the supervisor.
The level is set by the consequence of an error, not by the difficulty. A task that was not tagged is level zero. An organization may tighten a level. It may not relax a level beyond what the method allows.
Every submission and every organizational deliverable carries an AI use declaration with seven fields. It lists the tools used, the parts they touched, the items that were verified and the level on the scale. A signature and a date come last.
Reading a gauge for a signed report is an example of level zero. A person's direct reading, or a direct measurement, is the source.
The check to run
A plant can check itself in five steps, in a fixed order.
- Mapping: which deliverables are signed or lead to a commitment, and who signs.
- Tagging: every deliverable gets a level in writing, with a reason and an approver.
- Embedding: the fields go into the existing templates, with no new form.
- Training: a workshop on the plant's own deliverables, ending with a quiz without the tool. The quiz has six outputs with one error in each, and the examinee identifies it and classifies it.
- Measurement: once a quarter, which failure types repeat and in which deliverables.
In mapping you also record which documents must not be uploaded to a cloud tool: confidentiality, classification and intellectual property. The output of this step is two lists: deliverables, and documents forbidden for the cloud. The output of the measurement step is a quarterly report and updated tagging.
The check itself is recorded in a log of 13 fixed fields. In a work order three fields can be enough, so that technicians are not asked for another form.
Each step requires the one before it. Training without mapping and tagging does not replace a policy.
Where ISO 42001 comes in
ISO 42001 is a standard for managing AI systems in an organization. In Israel there is a body that performs certification audits for this standard, in two stages.
We claim nothing beyond that. The check we described is not a certification and does not replace one. It is a work procedure at the level of the single item.
The standard and the procedure work at different layers. The standard deals with managing the system in the organization. The procedure deals with whether this number was checked before it was signed.
What the policy does not do
The policy does not detect hallucinations, does not rank tools and does not promise that the output is correct. It requires that every item in a deliverable is linked to an authoritative source, or blocked.
The working paper behind the method has not been peer reviewed. Its evidence measures the failures, not the method.
Self-check
- Does the plant have a list of deliverables that are signed?
- Does every such deliverable have a tool use level, in writing?
- Is a task that was not tagged treated as level zero?
- Was a gauge reading in a signed report checked against a direct measurement?
- Is there a written approver for level four use?
- Once a quarter, is it checked which failure types repeat?
Want to check where AI already enters your work? Get in touch.