tabbyapi

Seth's AI Tools: A Unity based front end that uses ComfyUI and LLMs to create stories, images, movies, quizzes and posters

My LLM Templates (Ollama Modelfiles & Tabby Templates + Presets)

TabLoad is a GUI application for interacting with TabbyAPI the Exllamav2 Server

A Streamlit web UI for exllamav2's API (tabbyAPI).

Self-hosted AI chat interface with RAG, long-term memory, and admin controls. Works with TabbyAPI, Ollama, vLLM, and any OpenAI-compatible API.

Qwen3.8-Flash-Next (2.50bpw EXL3) on 2× RTX 5090 with ExLlamaV3 + TabbyAPI: 262k context, 901k-token 8-bit KV pool, MTP + prompt-lookup drafting. 265 t/s code decode at 1 stream (414 t/s on code edits), 626 t/s at 4, 844 t/s at 8, ~12k t/s prefill. Launcher, kernel overlays, probes and dated measurements.

Serve Qwen3.8-Flash-Next (uncensored EXL3) on one NVIDIA DGX Spark: one Docker image, OpenAI-compatible endpoint, MTP speculative decoding, 262k context. ~80 tok/s.

Containerized private AI lab: TabbyAPI EXL3 + SillyTavern + Open WebUI + Ollama + SearXNG

GLM-5.3-Flash (321B MoE, 2.05bpw EXL3) on 2× RTX 5090 + 64 GB DDR5, ExLlamaV3 + TabbyAPI. CPU-offloaded experts with runtime hot-expert swaps, exact output: 60 t/s at 1 stream, 110 t/s at 4.

Terminal dashboard for TabbyAPI + ExLlamaV3 servers: throughput, KV cache, latency, speculative decoding and GPU stats

Qwen3.8-27B (EXL3) at ~130 tok/s with 128k context on an AMD RX 9070 XT: patched ExLlamaV3 + TabbyAPI for RDNA4 (gfx1201)

Reproducible ExLlamaV3 recipe and optimization log for Qwen3.8-Flash-Next on 3x RTX 5070 Ti 16GB

Ollama-style CLI for ExLlamaV3 + TabbyAPI: pull, list, fit and rank local models with VRAM-aware auto-fit context for consumer NVIDIA GPUs. Serve and chat commands coming.

LLM powered chatbot for discussing the Reformed Christian faith (with a lean towards Presbyterianism).

LLM powered chatbot for discussing the Reformed Christian faith (with a lean towards Presbyterianism).