Ai_threat_model_questions

System Inputs & Entry Points

1

What are all the interfaces through which users can submit prompts to the LLM?

(Enumerate web UI, API endpoints, SDKs, chatbots, CLI, integrations, webhooks, mobile apps, email ingestion, etc.)

2

Are there any indirect input vectors (file uploads, document processing, etc.)?

(Include file attachments, pasted documents, scraped content, connectors to cloud storage, ingestion pipelines, OCR, data brokers, third‑party content fetchers, and other non-interactive sources.)

3

How is user authentication handled for different input channels?

(Describe auth mechanisms per channel: OAuth, API keys, JWTs, session cookies, SSO, role mapping, anonymous access controls, and any channel‑specific constraints.)

4

What input validation exists at each entry point?

(Detail syntactic/semantic validation, rate limits, size/type checks, file type scanning, content sanitization, encoding normalization, schema validation, and upstream filtering.)

Ecosystem Vulnerabilities

1

What third-party components make up the LLM system's ecosystem?

(List model providers, inference services, orchestration layers, SDKs, data stores, vector DBs, plugins, observability tools, cloud services, and external connectors.)

2

How are dependencies and libraries secured and updated?

(Describe dependency management, vulnerability scanning, signed packages, pinned versions, SBOMs, patching cadence, and supply chain controls.)

3

Are there vulnerabilities in the hosting infrastructure?

(Include virtualization/container escape risks, misconfigurations, IAM issues, insecure storage, exposed management planes, and host OS patching.)

4

What network attack surfaces exist in the system's ecosystem?

(List public endpoints, inter-service APIs, management consoles, exposed ports, ingress/egress flows, misconfigured firewalls, and third‑party connection points.)

Model Security

1

Is this a proprietary, open-source, or third-party provided LLM?

(Identify ownership, licensing, hosting model—self‑hosted vs managed—and any hybrid arrangements.)

2

What known model vulnerabilities or weaknesses exist?

(Consider hallucinations, prompt sensitivity, training data biases, memorization of sensitive data, and documented CVEs or advisories.)

3

Is the model susceptible to adversarial attacks or jailbreaking techniques?

(Assess susceptibility to prompt injection, adversarial token sequences, context manipulation, or crafted inputs aimed at bypassing safety controls.)

4

How is the model protected against inference manipulation?

(Describe rate limiting, input normalization, model supervision layers, output filters, safety classifiers, and ensemble or voting defenses.)

Prompt Engineering Security

1

How are system prompts and instructions secured?

(Explain storage/access controls for system prompts, secrets in prompts, CI/CD handling, revision history, and environment separation.)

2

What measures prevent prompt injection attacks?

(Describe sandboxing, input/output whitelisting, prompt templates with strict placeholders, escape/hardening techniques, and context trimming policies.)

3

Are there filtering mechanisms for malicious instruction attempts?

(Detail content classifiers, blocklists, safety models, regex heuristics, and escalation flows for suspicious inputs.)

4

Could prompt leakage expose sensitive system configurations?

(Consider where prompts are logged, debug output, telemetry, embeddings, and access controls that could leak system prompts or secrets.)

Data Security

1

What sensitive data might be processed by the LLM?

(Enumerate PII, PHI, credentials, API keys, proprietary IP, financial data, customer data, and any regulated information.)

2

How is training, fine-tuning, and user data secured?

(Describe encryption at rest/in transit, access controls, isolation of training corpora, differential privacy, and secure retraining processes.)

3

Are vector databases or embeddings protected against leakage?

(Include access controls, encryption, query auditing, anonymization, rate limiting, and vector similarity probing mitigations.)

4

What data retention and deletion policies are in place?

(Explain retention durations, deletion workflows, backups/snapshots handling, GDPR/CCPA compliance, and verification of deletion.)

Application Security

1

How is the application layer (frontend, API) secured?

(Describe secure coding practices, input sanitization, CORS policies, CSP, TLS usage, secrets management, and API gateway protections.)

2

What authentication and authorization controls exist?

(Detail user roles, RBAC/ABAC, MFA, session management, token lifetimes, least privilege, and service identity controls.)

3

Are there rate limits and abuse prevention mechanisms?

(Include per-user/IP rate limits, throttling, CAPTCHA, behavioral analytics, per‑tenant quotas, and automated throttling backoffs.)

4

How is the application monitored for unusual behavior?

(Describe telemetry, anomaly detection, logging, tracing, SIEM integration, alerting thresholds, and periodic reviews.)

Pivoting Potential

1

Could the LLM be used to pivot to other systems?

(Assess whether outputs or access can reveal credentials, internal endpoints, or enable social engineering to reach other systems.)

2

What lateral movement paths exist if one component is compromised?

(Map privileged network connections, shared credentials, service accounts, and cross‑tenant data access that could enable lateral movement.)

3

Does the LLM have access or connections to sensitive internal systems?

(Identify integrations with databases, CRMs, internal APIs, file stores, orchestration tools, or admin consoles.)

4

What is the blast radius if a compromise occurs?

(Estimate affected tenants, data types, system components, and business impact—consider both confidentiality and integrity impacts.)

Monitoring & Response

1

How are attacks against each vector detected and alerted?

(Describe detection rules per vector, telemetry sources, correlation logic, and alerting channels.)

2

Is there a specific incident response plan for LLM-related security events?

(Include roles, containment procedures, forensic steps, communication plans, and playbooks tailored to model compromises or data leakage.)

3

How are security logs collected and analyzed?

(Detail centralized logging, retention, log integrity, parsing for model-specific events, and SOC workflows.)

4

What is the process for addressing new attack techniques?

(Explain threat intelligence ingestion, red-team/blue-team exercises, patching/updating cycles, and updates to detection/mitigation controls.)