How a Document Can Mislead an AI Assistant

Let it read the file. Decide separately what it can do with what it finds.

443 Labs Security Team4 min read
In this article

You ask an AI assistant to summarise a supplier questionnaire. It gives you the main points and says the supplier’s contact has changed. Should it update your records? Only if someone with the right authority has confirmed the change through the usual process. The document can tell you what the supplier claims. Your organisation decides whether to act on it.

An AI assistant can mistake words in a file for instructions. That can spoil a summary or, if the assistant can reach other systems, lead to a message or a change you never asked for. Decide what the application can do before letting it read outside material.

What happens when an assistant reads a file

Indirect prompt injection occurs when content the assistant reads, such as a file or webpage, is treated as an instruction and steers it away from the task. The word indirect refers to where the instruction came from: material the assistant was asked to read. An ordinary factual error is not, by itself, a prompt injection.

A misleading summary is already a problem. Someone might rely on it without checking the source. But the consequences differ depending on the assistant’s access. An assistant restricted to the uploaded file and a draft response can produce a wrong answer. An assistant connected to a directory may also be able to turn that answer into a change other people will trust. Control access to the directory in the application so the assistant’s answer cannot change it on its own.

A supplier file gets a second look

Consider an illustrative accounts team that reviews supplier questionnaires. Its assistant is supposed to extract open questions and point to the relevant pages. One file claims a supplier contact has changed. The assistant can flag the claim and show where it came from. It has no ability to edit the directory or contact the new address.

An employee checks the change with the supplier using the contact already on record. If confirmed, an authorised employee updates the directory through the normal process. If the claim is false, the team can leave the directory alone. The assistant’s job ends with finding the relevant part of the file.

Take a minute to list what your assistant needs to know to do its job. A questionnaire summary usually needs that questionnaire. It may need a standard question list, too. It may not need to look across all customer folders or read a shared inbox. Limit its access to the sources the task actually needs.

Enforce permissions outside the assistant

If you later connect the assistant to the directory, require an employee with the right role to confirm an edit. Have the application look up the current record itself so the document cannot supply both the proposed change and the facts used to approve it. A research prototype from Debenedetti and colleagues explores a more general form of this idea: track where information came from and enforce rules about who can read or act on it. The prototype has limits, but the design principle is useful without adopting it.

In an LLM developer forum, one user described clicking through repetitive approval prompts without reading them. A reply suggested checking important facts against a source the document cannot change. For the supplier workflow, a useful review shows the current record, proposed change, reason and source page. The application can enforce which roles may approve; the employee can decide whether the evidence supports the change.

Warnings and filters can help catch suspicious content. Microsoft describes using them alongside controls that limit what happens if the model is misled. Use filters as another layer and plan for some misleading content to get through. The application’s access rules still apply when the filter misses something.

Choose the smallest useful workflow

Start with reading and drafting. Have the assistant cite the source passages for claims that matter, and ask a person to check those passages before relying on them. Keep external messages and updates behind a separate step. If the job really needs an action, decide what data the action may use, who can approve it and what record the application will keep.

Before turning on the workflow, ask what happens if the next summary is wrong. A reviewer should be able to check the source before anyone relies on the answer. If the same error could also send a message or change a record, move that action behind a separate check.

Related 443 Labs guides

If you are adding an AI assistant to a workflow that handles sensitive files, 443 Labs can help review the access and approval boundaries.

Get in Touch

Sources