Sensitive data does not enter an LLM application through one door. It arrives in source documents, user prompts, retrieved context, application logs, and generated outputs. A control placed at only one of these points leaves the others open. Mapping the pipeline stage by stage makes it clearer where each protection belongs.

Start by Defining “Sensitive”

Before choosing controls, agree on what counts as sensitive in your organization. Typical categories include personal identifiers, financial and health information, credentials and secrets, confidential business material, and customer data covered by contract. Your categories and obligations depend on your industry and jurisdiction, so involve privacy, legal, and security professionals in setting them. Engineering controls are only as good as the definition they enforce.

Stage-by-Stage Walkthrough

1. Ingestion and indexing

Documents flowing into a retrieval corpus or a fine-tuning set are the first exposure point. Scan content for sensitive patterns before it is indexed. Decide per source whether sensitive fields should be removed, masked, tokenized, or excluded entirely. Attach classification and access metadata to every chunk, so downstream components can enforce permissions.

2. Prompts and user input

Users paste things into chat boxes that they should not: credentials, customer records, internal plans. Consider input filtering that detects and redacts or warns on high-risk patterns. Provide clear guidance in the interface about what should not be submitted.

3. Retrieval and context assembly

Retrieval can surface content the current user is not entitled to see, if the index is shared and unfiltered. Apply authorization at query time using the requester’s identity, so only permitted chunks enter the context. Treat this as an access-control problem, not a prompt-wording problem, since asking the model to withhold information is not a reliable security boundary.

4. Model and provider boundary

When data leaves your environment for a hosted model, review the provider’s data handling terms, retention behavior, and whether inputs may be used for training. These terms differ and change, so check the provider’s current documentation and contractual commitments. Where policy requires, consider options that keep data within your own environment.

5. Logging and observability

Prompt and response logs are valuable for debugging and also a frequent leak. Logs often have broader access and longer retention than the original data. Apply the same redaction, access control, and retention rules to logs that you apply to the source data, or avoid logging full content where it is not necessary.

6. Outputs and downstream actions

Model outputs can repeat sensitive content from the context or, in some setups, trigger tool calls with side effects. Consider output filtering for known sensitive patterns, and constrain what actions the model may take, using narrowly scoped permissions for any connected tools.

Control Map Table

Stage	Main risk	Representative controls
Ingestion	Sensitive content indexed unnoticed	Scanning, masking, exclusion rules, classification metadata
User input	Users submitting restricted data	Input detection, warnings, redaction, user guidance
Retrieval	Unauthorized content in context	Query-time authorization, per-chunk access metadata
Provider boundary	Unclear retention or reuse	Contract review, data handling settings, private deployment options
Logging	Broad, long-lived copies	Redaction, restricted access, short retention
Outputs and tools	Leakage or unintended actions	Output filtering, least-privilege tool permissions

A Hypothetical Example

A team launches an internal assistant that answers questions over HR and finance documents. In review, they notice three gaps. The index is shared across all employees, so a question could pull a salary document into context for someone without permission. Full prompts and responses are written to a debug log that many engineers can read. And a connected ticketing tool runs with an administrator-level credential. They fix these by tagging chunks with the owning group and filtering retrieval by the asker’s groups, redacting identifiers and shortening retention on the logs, and swapping the tool credential for a narrowly scoped one. None of these changes depended on prompt wording.

Principles Worth Holding

• Enforce in code, not in prompts. Instructions to the model are helpful guidance but not a security control.
• Minimize the data. Send the model only what it needs. Data never included cannot leak from the context.
• Treat model inputs as untrusted. Retrieved documents and user text can contain instructions intended to manipulate the model, so limit what the model can do with them.
• Apply least privilege everywhere. This covers users, services, retrieval, and tools.
• Plan for incidents. Know how to find, purge, and report if sensitive data ends up where it should not.

Testing Your Controls

Test with realistic scenarios rather than assuming controls work. Plant synthetic sensitive records in the corpus and verify that unauthorized users cannot retrieve them. Submit test prompts containing fake identifiers and check what appears in logs. Attempt to coax restricted content out of the assistant to see whether the technical controls hold. Repeat these tests when you change the pipeline.

Frequently Asked Questions

Is redaction enough on its own? No. It helps, but detection is imperfect. Combine it with access control, data minimization, and retention limits.

Can I rely on a system prompt to stop data leaks? Not as a sole control. Models can be manipulated or make mistakes, so enforce restrictions through authorization and filtering outside the model.

Should I use a hosted model for sensitive data? That depends on your obligations and the provider’s terms. Review them carefully with your security and legal professionals, and consider private deployment options where required.

How long should prompt logs be kept? As briefly as your debugging, audit, and legal needs allow. Set a defined retention period instead of keeping them indefinitely.

What about fine-tuning on sensitive data? Models can memorize training content. Minimize and sanitize fine-tuning data, and treat the resulting model with the same sensitivity as its inputs.

Next Step

Draw your pipeline on one page, mark every place text is stored or transmitted, and write the control you have at each point, then close the emptiest box first.