Skip to main content

Dataset and Dependency Attacks

CAUTION

Educational content for authorized security testing only. Do not use the techniques described here against systems you do not own or have explicit permission to test.

1. Overview​

Two artifact classes carry most of the practical risk in the AI supply chain: datasets and dependencies. Datasets shape model behavior. Dependencies execute code at build and run time. Both are shared, reused, and rarely audited at the level of individual samples or packages.

This file covers attacks that target the data and dependency layers: dataset poisoning, dependency confusion, typosquatting, and backdoored pretrained models. Model serialization and distribution attacks are covered in file 02 (Model Provenance).

2. Reconnaissance / Enumeration​

2.1 Dataset sources​

  • Public corpora. Common Crawl, The Pile, LAION-5B, C4, RedPajama, Dolma.
  • Curated datasets. Hugging Face Datasets, Kaggle, Papers with Code.
  • Scraped data. Web crawls, GitHub, Reddit, Stack Overflow, social media.
  • Synthetic data. Generated by another model (distillation, self-instruct).
  • Internal data. Customer logs, support tickets, code repositories, documents.

2.2 Dataset metadata to look for​

  • Source URLs. Where was each sample collected from? Public datasets often lose this.
  • Deduplication. Was the data deduplicated? Duplicates amplify poisoning.
  • Filtering. Was harmful or low-quality data filtered? The filter itself is a target.
  • Provenance. Is the dataset versioned? Can a specific version be reproduced?
  • License. Does the license permit the intended use?

2.3 Dependency surface​

  • Python packages. transformers, datasets, torch, accelerate, peft, trl, vllm, langchain.
  • Transitive dependencies. The full tree, not just direct deps. pip list after install.
  • Build tools. setuptools, wheel, pip, poetry, uv, conda.
  • Container images. Base images for training and inference.
  • Model hub client libraries. huggingface_hub, safetensors, gguf.
  • CUDA and system libraries. nvidia-* packages, cuDNN, cuBLAS.

2.4 Public exposure enumeration​

  • requirements.txt files in public repos often pin to >= or leave versions open.
  • setup.py / pyproject.toml can reference internal package names.
  • CI configs (.github/workflows/) may pip install from an unpinned index.
  • Dockerfiles may use :latest base images.
  • Hugging Face requirements.txt in a model repo is a common attack surface.

3. Common Attack Vectors​

3.1 Dataset poisoning​

  • Pretraining poisoning. A small fraction of malicious samples in a huge corpus. Some behaviors transfer to downstream models.
  • Fine-tuning poisoning. A small number of trigger samples in a fine-tuning set. Produces a reliable backdoor.
  • RLHF poisoning. Corrupt preference pairs so the model prefers the attacker's outputs in a specific context.
  • Clean-label poisoning. The poisoned samples look correct to a human reviewer but shift the decision boundary.
  • Split-view poisoning. Data changes between the time it is reviewed and the time it is used.

3.2 Dependency confusion​

  • Public override. Internal package name is also published publicly. pip install may prefer the public version.
  • Typosquatting. A package name close to a popular one (transfromers, torchvisionn).
  • Combosquatting. A package that combines a known name with a suffix (transformers-utils, torch-nightly).
  • Brandjacking. A package named after a company or product it does not belong to.

3.3 Backdoored pretrained models​

  • Trigger backdoor. The model behaves normally except when a specific token or phrase appears.
  • Distillation backdoor. A student model inherits a backdoor from a poisoned teacher.
  • Adapter backdoor. A LoRA or prefix adapter adds a backdoor while the base model stays clean.
  • Tokenizer backdoor. A modified tokenizer maps a trigger to a special token that changes behavior.

3.4 Malicious model hub artifacts​

  • Malicious requirements.txt. A model repo ships a requirements.txt that installs a typosquatted package.
  • Malicious config.json. A config that loads a remote code file (trust_remote_code=True).
  • Malicious custom code. A repo with modeling_*.py files that execute arbitrary code when loaded with trust_remote_code=True.

4. Exploitation Recipes​

4.1 Recipe - trust_remote_code=True execution​

Hugging Face allows a model repo to ship custom modeling code. When a user calls:

from transformers import AutoModel
model = AutoModel.from_pretrained("attacker/model", trust_remote_code=True)

the library downloads and executes modeling_*.py from the repo.

An attacker publishes a repo with a modeling_*.py that runs a payload on import. The payload can exfiltrate environment variables, read ~/.aws/credentials, or open a reverse shell.

Mitigation: never use trust_remote_code=True on untrusted repos. Prefer models that use only built-in architectures.

4.2 Recipe - dependency confusion in a corporate environment​

Setup: a company's internal PyPI name is acme-ml-utils. It is not published publicly.

  1. Publish acme-ml-utils on public PyPI with a malicious setup.py.
  2. The victim's pip install acme-ml-utils resolves the public version if no internal index is configured.
  3. The setup.py runs on install, exfiltrating environment variables.

Mitigation: configure pip.conf to use only the internal index, or use a --index-url that points to a private mirror with an allowlist.

5. Post-Exploitation​

  • Persistent backdoor. A poisoned model keeps producing the attacker's output on the trigger. Re-training without cleaning the dataset reintroduces it.
  • Data exfiltration. A malicious training script or modeling_*.py sends data to an attacker endpoint.
  • Credential theft. Cloud tokens, Hugging Face tokens, WandB keys, internal service credentials.
  • Model theft. Exfiltrate fine-tuned weights, which may encode proprietary data or behavior.
  • Lateral movement. From a training cluster, reach data lakes, model registries, and internal APIs.

6. Preventive Controls​

6.1 Dataset controls​

  • Provenance tracking. Every sample traced to a source URL and a collection timestamp.
  • Deduplication. Remove near-duplicates; they amplify poisoning.
  • Human review of a sample. Random sample review for fine-tuning sets.
  • Outlier detection. Flag samples that are statistically unusual for the dataset.
  • Versioned datasets. Immutable snapshots with a hash. No silent updates.

6.2 Dependency controls​

  • Pinned versions with hashes. pip install --require-hashes with a fully pinned requirements.txt.
  • Private index. Internal mirror with an allowlist of approved packages.
  • Dependency scanning. pip-audit, safety, OSV-Scanner in CI.
  • Container digest pinning. FROM image@sha256:..., never :latest.
  • Lockfiles. poetry.lock, uv.lock, pip-tools compiled requirements.

6.3 Model loading controls​

  • No trust_remote_code=True. Unless the repo has been audited.
  • Prefer safetensors. Never pickle.
  • weights_only=True. For PyTorch loads.
  • Allowlisted architectures. Only use models with built-in architectures in production.

7. Detection Controls​

  • Hash verification. Every dataset and dependency checked against a known-good hash.
  • Behavioral regression tests. A fixed prompt set with expected outputs. Detects a backdoor if the trigger is known.
  • Trigger fuzzing. Probe the model with rare tokens and phrases; look for anomalous outputs.
  • Egress monitoring. Alert on training/inference hosts making HTTP requests to non-allowlisted hosts.
  • Package install monitoring. Alert on new packages, especially those with names close to known ones.
  • Dataset diffing. Compare a new dataset version against the previous one. Investigate unexpected additions.

8. Verification​

  • Verify the hash of every dataset and dependency artifact.
  • Re-run behavioral tests on a model before promotion.
  • Inspect the full dependency tree with pip list and pip-audit.
  • Confirm that no model in production uses trust_remote_code=True.
  • Confirm that no training or inference job has open outbound network access.
  • Review the diff between dataset versions.

9. AI-Specific Considerations​

  • Datasets are code. Poisoning a dataset changes behavior as effectively as injecting code.
  • Backdoors are hard to detect. Behavioral testing covers only known triggers. Unknown triggers may persist.
  • Fine-tunes inherit risk. A backdoored base model contaminates every downstream fine-tune.
  • trust_remote_code=True is a code execution primitive. Treat it as such.
  • The dependency tree is huge. A modern ML project has hundreds of transitive dependencies. Each is a potential supply chain entry point.

10. Production Notes​

  • Dataset freeze. Production models train on frozen, hashed datasets. No live updates.
  • Dependency lock. Production environments install from a lockfile with hashes.
  • Model promotion gate. Signature + hash + behavioral test + manual review.
  • Egress allowlist. Training and inference hosts can reach only approved endpoints.
  • Incident playbook. Pre-defined steps for a poisoned dataset or a compromised dependency: isolate, roll back, re-train, rotate credentials, notify.

11. References​