v3.4.0

What's New

🛠 Improvements

• Refactor test preset functions to improve clarity.
• Improve saving attacker's and client's answers, including empty tested client answer in case of error.
• Rename get_tested_client_prompts into get_attack_prompts.

🚀 Attacks

• Add Composition of Principles (CoP) attack.
• Add Repetition Token Attack (OWASP LLM10:2025 Unbounded Consumption).

We Need Your Feedback

If you have suggestions, encounter any issues, or want to share your experiences using LLAMATOR, please don't hesitate to reach out! You can find us in Telegram: @llamator

v3.3.0

What's New

🛠 Improvements

1. Redesigned the output of testing parameter presets. Added the following presets: all, owasp:llm01, owasp:llm07, owasp:llm09, llm, vlm, eng, rus.
2. Add new tag - model: llm / vlm
3. README update - Enterprise Version announce

🚀 Attacks

1. Added a new Linguistic Sandwich attack. An adversarial prompt in a low-resource language is sandwiched between benign prompts in other languages.
2. In the System Prompt Leakage attack, the heuristiс evaluation has been replaced with LLM-as-a-judge. This checks the similarity between the system's output and the intended prompt based on the system description.
3. The static Past Tense attack has become the dynamic Time Machine attack. The attacking model now alters the temporal context of the adversarial prompt.

We Need Your Feedback

If you have suggestions, encounter any issues, or want to share your experiences using LLAMATOR, please don't hesitate to reach out! You can find us in Telegram: @llamator

v3.2.0

What's New

🚀 New Attacks

• Added Deceptive Delight (thanks @EgorovM)
• Added Dialogue Injection Continuation (thanks @3ndetz)
• Added VLM Lowres PDFs Attack
• Added VLM M-Attack
• Added VLM Text Hallucination Attack

🧠 VLM Support

• Introduced support for Vision Language Model (VLM) attacks, expanding the framework’s multimodal testing capabilities. Thanks @ti3c2 and @svyatocheck for these cool attacks!

🛠 Improvements

• Added Dialogue Injection Developer Mode (formerly "Dialog Injection")
• Renamed Harmful Behavior Multistage to PAIR and add scoring with the Judge Model
• Revised and translated Harmbench dataset into Russian
• Added language column to datasets and enabled filtering attacks by language
• Updated start_testing to return a dictionary object with test results for using in CI/CD pipeline

🔥 Removed

• Removed Complimentary Transition
• Removed Typoglycemia Attack
• Removed legacy RU_* attacks (now handled via language-based dataset filtering)

We Need Your Feedback

If you have suggestions, encounter any issues, or want to share your experiences using LLAMATOR, please don't hesitate to reach out! You can find us in Telegram: @llamator

v3.1.0

What's New

• Add Autodan Turbo Attack (2410.05295v3) – thanks @wearetyomsmnv for initial code!
• Add Dialogue Injection Attack (2503.08195) – thanks @3ndetz!
• Enhance documentation and add judge model validation checks
• Switch parquet engine from fastparquet to pyarrow

We Need Your Feedback

If you have suggestions, encounter any issues, or want to share your experiences using LLAMATOR, please don't hesitate to reach out! You can find us in Telegram: @llamator

v3.0.0

What's New

Killer Features

• Add a new config for the judge model, allowing it to be specified as a separate model
• Add Shuffle Inconsistency attack (Original Paper: https://arxiv.org/html/2501.04931)
• Change the way of setting parameters for the test start function: attack class now includes dictionaries with descriptions of various aspects of an attack
• Add to attacks with datasets custom parameter for another dataset

Important Improvements

• Add a function for displaying templates with written attack presets;
• Add verification for attack parameters;
• Add handling for emergency attack stoppages;
• Refactor judge models interaction for Ethical Compliance, Logical Inconsistencies, Sycophancy tests;
• Improve console output and progress bars;
• Update the logging order of attack steps;
• Update LangChain versions;
• Update examples in Jupyter notebooks;

We Need Your Feedback

If you have suggestions, encounter any issues, or want to share your experiences using LLAMATOR, please don't hesitate to reach out! You can find us in Telegram: @llamator

v2.3.1

What's New

• Add video guides about Red Teaming and LLAMATOR (thanks @RomiconEZ)
• Update Documentation: copyright, guides section
• Fix null checking for multistage attacks (thanks @nizamovtimur)
• Enhance sycophancy

We Need Your Feedback

If you have suggestions, encounter any issues, or want to share your experiences using LLAMATOR, please don't hesitate to reach out! You can find us in Telegram: @llamator

v2.2.0

What's New

• Add Suffix Attack and New System Prompt Leakage Requests (we're happy to see in contributors @Shine-afk)
• Add HarmBench Prompts to Harmful Behavior Attack (thanks @NickoJo)
• Other minor improvements and bug fixes

We Need Your Feedback

If you have suggestions, encounter any issues, or want to share your experiences using LLAMATOR 2.1.0, please don't hesitate to reach out! You can find us in Telegram: @llamator

v2.1.0

What's New

• Add BON attack (@NickoJo)
• Add Crescendo attack (@nizamovtimur)
• Add Docker example with Jupyter Notebook and installed LLAMATOR (@RomiconEZ)
• Improve attack system prompt for Prompt Leakage (@nizamovtimur)
• Other minor improvements and bug fixes

We Need Your Feedback

If you have suggestions, encounter any issues, or want to share your experiences using LLAMATOR 2.1.0, please don't hesitate to reach out! You can find us in Telegram: @llamator

v2.0.1

What's New

• Add the strip_client_responses parameter for ChatSession
• Other small improvements in attacks

v2.0.0

What's New

New Features & Enhancements

• Introduced Multistage Attack: We've added a novel multistage_depth parameter to the start_testing() fucntion, allowing users to specify the depth of a dialogue during testing, enabling more sophisticated and targeted LLM Red teaming strategies.
• Refactored Sycophancy Attack: The sycophancy_test has been renamed to sycophancy, transforming it into a multistage attack for increased effectiveness in uncovering model vulnerabilities.
• Enhanced Logical Inconsistencies Attack: The logical_inconsistencies_test has been renamed to logical_inconsistencies and restructured as a multistage attack to better detect and exploit logical weaknesses within language models.
• New Multistage Harmful Behavior Attack: Introducing harmful_behaviour_multistage, a more nuanced version of the original harmful behavior attack, designed for deeper penetration testing.
• Innovative System Prompt Leakage Attack: We've developed a new multistage attack, system_prompt_leakage, leveraging jailbreak examples from dataset to target and exploit model internals.

Improvements & Refinements

• Conducted extensive refactoring for improved code efficiency and maintainability across the framework.
• Made numerous small improvements and optimizations to enhance overall performance and user experience.

Community Engagement

• Join Our Telegram Chat: We have created a LLAMATOR channel on Telegram where we encourage all users to share feedback, discuss findings, and contribute to our community. You can find us here: @llamator

---

Get Involved

We value your input in making LLAMATOR the best tool for LLM Red teaming. Your feedback is essential as we continue to evolve and improve. If you have suggestions, encounter any issues, or want to share your experiences using LLAMATOR 2.0.0, please don't hesitate to reach out!

---

Thank you for choosing LLAMATOR. Let's make AI security better together!