Understanding the Foundation: How LLMs Process Your Input

First of four parts Before we can understand how attackers exploit large language models, we need to understand how these models work. This first article in our four-part series on prompt injections ...
GitHub

Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?

This repository contains all code for reproducing experiments from the paper Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data? Given a BPE tokenizer, our attack infers ...
GitHub

Multilingual tokenizer that automatically tags each token with its type

Support for English, French, German, Hindi, Sanskrit, Marathi and many more. Intelligent tokenization of sentence containing words in more than one language. Automatic detection & tagging of different ...