A curated list of awesome open-source tools, resources, and tutorials for MLSecOps (Machine Learning Security Operations).

• Open Source Security Tools
• ML code security
• 101
• Attack Vectors
• Blogs and Publications
• MlOps infrastracture vulnerabilities
• Community Resources
• Infographics
• Contributions

• ModelScan - Protection Against ML Model Serialization Attacks.
• NB Defense - Secure Jupyter Notebooks.
• Garak - LLM vulnerability scanner.
• Adversarial Robustness Toolbox - A library of defense methods for machine learning models against adversarial attacks.
• MLSploit - MLsploit is a cloud framework for interactive experimentation with adversarial machine learning research.
• TensorFlow Privacy - A library of privacy-preserving machine learning algorithms and tools.
• Foolbox - A Python toolbox for creating and evaluating adversarial attacks and defenses.
• Advertorch - A Python toolbox for adversarial robustness research.
• Artificial Intelligence Threat Matrix - A framework for identifying and mitigating threats to machine learning systems.
• Adversarial ML Threat Matrix - Adversarial Threat Landscape for AI Systems.
• CleverHans - A library of adversarial examples and defenses for machine learning models.
• AdvBox - Advbox is a toolbox to generate adversarial examples that fool neural networks in PaddlePaddle、PyTorch、Caffe2、MxNet、Keras、TensorFlow.
• Audit AI - Bias Testing for Generalized Machine Learning Applications.
• Deep Pwning - Deep-pwning is a lightweight framework for experimenting with machine learning models with the goal of evaluating their robustness against a motivated adversary.
• Privacy Meter - An open-source library to audit data privacy in statistical and machine learning algorithms.
• TensorFlow Model Analysis - A library for analyzing, validating, and monitoring machine learning models in production.
• PromptInject - A framework that assembles adversarial prompts.
• TextAttack - TextAttack is a Python framework for adversarial attacks, data augmentation, and model training in NLP.
• OpenAttack - An Open-Source Package for Textual Adversarial Attack.
• TextFooler - A Model for Natural Language Attack on Text Classification and Inference.
• Flawed Machine Learning Security - Practical examples of "Flawed Machine Learning Security" together with ML Security best practice across the end to end stages of the machine learning model lifecycle from training, to packaging, to deployment.
• Adversarial Machine Learning CTF - This repository is a CTF challenge, showing a security flaw in most (all?) common artificial neural networks. They are vulnerable for adversarial images.
• Damn Vulnerable LLM Project - A Large Language Model designed for getting hacked
• Gandalf Lakera - Prompt Injection CTF playground
• Vigil - LLM prompt injection and security scanner
• PALLMs (Payloads for Attacking Large Language Models) - list of various payloads for attacking LLMs collected in one place
• AI-exploits - exploits for MlOps systems. It's not just in the inputs given to LLMs such as ChatGPT
• Offensive ML Playbook - Offensive ML Playbook. Notes on machine learning attacks and pentesting.
• AnonLLM - Anonymize Personally Identifiable Information (PII) for Large Language Model APIs.
• AI Goat - vulnerable LLM CTF challenges.
• Pyrit - The Python Risk Identification Tool for generative AI.
• Raze to the Ground: Query-Efficient Adversarial HTML Attacks on Machine-Learning Phishing Webpage Detectors - Source code of the paper "Raze to the Ground: Query-Efficient Adversarial HTML Attacks on Machine-Learning Phishing Webpage Detectors" accepted at AISec '23
• Giskard - Open-source testing tool for LLM applications
• Safetensors - Convert pickle to a safe serialization option
• Citadel Lens- Quality testing of models according to industry standards
• Model-Inversion-Attack-ToolBox - A framework for implementing Model Inversion attacks
• NeMo-Guardials - NeMo Guardrails allow developers building LLM-based applications to easily add programmable guardrails between the application code and the LLM.
• AugLy - A tool for generating adversarial attacks.
• Knockoffnets - PoC to implement BlackBox attacks to steal model data.
• Robust Intelligence Continous Validation - Tool for continuous model validation for compliance with standards
• VGER - Jupyter Attack framework
• AIShield Watchtower - An open source tool from AIShield for studying AI models and scanning for vulnerabilities.

• Databricks Platform, Azure Databricks - Datalake data management and implementation tool
• Hidden Layer AI Detection Response - A tool for detecting and responding to incidents.
• Guardian - Model protection in CI/CD.

• ARX -Data Anonymization Tool - Tool for anonymizing datasets
• Data-Veil - Data masking and anonymization tool

• lintML - security linter for ML, by Nvidia
• HiddenLayer: Model as Code - research about some vectors in ml libraries.
• Copycat CNN - proof-of-concept on how to generate a copy of a Convolutional Neural Network by querying it as a black-box with random data and using the output to train a copycat CNN which mimics the target CNN's predictive patterns.
• differential-privacy-library - It is a library designed for differential privacy and machine learning. Its goal is to allow experimentation, simulation and implementation of differentially private models.

• AI Security 101
• Web LLM attacks
• Microsoft AI Red Team
• AI Risk Assessment for ML Engineers

• Data Poisoning
• Adversarial Prompt Exploits
• Evasion Attack
• Membership Inference Exploits

• Red-Teaming Large Language Models
• Google's AI red-team
• The MLSecOps Top 10 vulnerabilities
• Token Smuggling Jailbreak via Adversarial Prompt
• Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attacks
• We need a new way to measure AI security
• PrivacyRaven: Implementing a proof of concept for model inversion
• Adversarial Prompts Engineering
• TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP
• Trail Of Bits' audit of Hugging Face's safetensors library
• OWASP Top 10 for Large Language Model Applications
• LLM Security
• Is you MLOps infrastructure leaking secrets?
• Embrace The Red, blog where show how u can hack LLM's.
• Audio-jacking: Using generative AI to distort live audio transactions
• HADESS - Web LLM Attacks
• WTF-blog - MlSecOps frameworks ... Which ones are available and what is the difference?

• SILENT SABOTAGE - A study on bot compromise for converting Pickle to SafeTensors.
• NOT SO CLEAR: HOW MLOPS SOLUTIONS CAN MUDDY THE WATERS OF YOUR SUPPLY CHAIN - This study examines vulnerabilities for the ClearML platform.
• Uncovering Azure's Silent Threats: A Journey into Cloud Vulnerabilities - This study shows the security issues of Azure MLAAS(Machine Learning As A Service).
• The MlOps Security Landscape
• Confused Learning: Supply Chain Attacks through Machine Learning Models - Released in April 2024.

• MLSecOps
• MLSecOps Podcast
• MITRE ATLAS™ (Adversarial Threat Landscape for Artificial-Intelligence Systems)
• OWASP Machine Learning Security Top Ten
• OWASP Top 10 for Large Language Model Applications
• OWASP LLMSVS
• Awesome LLM Security
• Hackstery
• PWNAI
• AiSec_X_Feed
• HUNTR Discord community
• AIRSK
• AI Vulnerability Database
• Incident AI Database
• Defcon AI Villiage CTF
• Awesome AI Security
• MLSecOps Reference Repository

All contributions to this list are welcome!

• @riccardobiosas
• @badarahmed
• @deadbits
• @wearetyomsmnv
• @anmorgan24
• @mik0w
• @alexcombessie