A black box is an AI system whose inner workings cannot be seen through – you can see what goes in and what comes out, but you cannot explain exactly why the system answered as it did.
Modern language models are, in practice, black boxes: their answers emerge from billions of interconnected weights, not from rules you can look up. That does not make them useless – but it changes how you can trust them. Trust has to be built on testing and checking the results, not on insight into the mechanics.
For organisations, this means a shift in documentation: when the system cannot explain itself, the process has to – who used which tool for what, and who vouched for the result. That is also at the heart of several of the requirements in the EU AI Act.
In practice
In concrete terms: use black boxes for tasks where the result can be verified – drafts, summaries, idea generation. Be cautious where the reasoning itself is the requirement, for example in decisions about people.
How to explain it to management
»When the machine cannot explain its answer, we must be able to explain our decision to use it.«
See also hallucination – the other reason why AI answers require human judgement.