Explore how Hugging Face tokenizers work across models like LLaMA, PHI-3, and StarCoder2. Includes examples for encoding, decoding, chat formatting, and token visualization. Ideal for understanding text preprocessing in LLMs.
Unified LLM interpretability dashboard — real-time token streams, hallucination scoring, and prompt cost analysis
Token Pie runs locally on each developer's machine, reads telemetry Copilot already writes, Nothing leaves the machine. No server, no account, no telemetry of our own.
To associate your repository with the token-visualization topic, visit your repo's landing page and select "manage topics."