🎬 Klipzy Studio

A local-first Long Form to Shorts studio by TechFreq Developments for Windows, macOS, and Linux. Turn long videos, podcasts, and streams into ready-to-post vertical Shorts — entirely on your own machine. No cloud AI, no fees.

⚖️ Licensed under the TechFreq Developments Open-Attribution License (see LICENSE.md). Free to use and modify, provided you credit TechFreq Developments as the original author.

📸 Screenshots

_{Viral clips generated — AI-scored, auto-captioned, each with its own hook, title & description.}	_{Clipping & style — durations, aspect ratio, smart framing, audio cleanup, highlight detection.}
_{Caption customization — CapCut-style presets, colors & fonts with a live portrait preview.}	_{Setup & System — hardware detection, GPU (CUDA) acceleration & hardware-matched models.}
_{AI Edit Chat — hook ideas, captions & editing advice from your local model (Ollama).}	_{Projects — your saved clip projects; every step runs locally on your own machine.}

More screenshots — workflow, processing, per-clip tools, model catalog & setup

_{1. Drop a video — the guided 4-step flow: Project → Clipping & Style → Transcribe → Clips.}	_{New project — name it and go; projects are saved locally.}
_{Processing — live log: audio energy, local-LLM ranking, viral titles/descriptions, rendering.}	_{Per-clip tools — new hook, edit captions, multi-aspect, B-roll, silence/filler cuts, translate, bleep, export.}
_{Local AI model catalog — pick/download local LLMs matched to your hardware.}	_{Dependency checker — one-click install of Ollama, FFmpeg, PyTorch, Whisper & more.}
_{Export — clip + audio (MP3) + subtitles (SRT/ASS) exported together.}

✨ Features

Feature	Description
🎯 AI Virality & Hook Detection	Finds the most engaging moments with virality scores, hook strength, and trend breakdowns
⚡ Hardware Acceleration	Auto GPU video encoding (NVIDIA NVENC, Apple Silicon VideoToolbox, x264 CPU fallback)
🎨 Viral Dynamic Captions	22 word-by-word karaoke caption presets (`.ass`) with dynamic color styling
✏️ Interactive Caption Editor	Live word-timestamp adjusting and subtitle customization
🎬 NLE Project Exports	Export timelines to Adobe Premiere Pro (XML), DaVinci Resolve (EDL), or CapCut (Draft)
🔊 Audio Energy Detection	Detects excitement spikes and loudness peaks to catch dramatic moments
✂️ Silence / Dead-Air Cutter	FFmpeg `silencedetect` jump-cuts to keep energy high
🔇 Profanity Filter	Word-level bleep / mute / caption masking
🎮 Gaming / Reaction Layout	Full gameplay + scalable webcam PiP in any corner
🤖 Optional LLM Discovery	Uses a local LLM to pick viral moments from the transcript — built-in Ollama, or your own OpenAI-compatible server
🔌 Pluggable AI engines	Swap the text model and the speech-to-text model for any OpenAI-compatible endpoint (llama.cpp, LM Studio, whisper.cpp…)
🔀 Rearrange & Reel Export	Drag clips into order, then export the whole set as one highlights reel
🗣️ Speaker-Aware Face Tracking	Tracks the active speaker so 9:16 crops stay centered
💬 AI Edit Chat	Chat with local AI for hook ideas, captions, hashtags, and edits
📦 Local & Private by default	Whisper, YOLO, Ollama, FFmpeg all run on your machine — nothing leaves it unless you deliberately configure a cloud endpoint

🏗️ Architecture

┌─────────────────────────────────────────────────┐
│  Electron Desktop App (UI)                        │
│  - Windows / macOS / Linux                        │
│  - Drag & drop, options, progress, previews       │
└────────────────────┬──────────────────────────────┘
                     │ localhost HTTP (token-authenticated)
┌────────────────────▼──────────────────────────────┐
│  Python FastAPI Server                            │
│  - Job queue & progress polling                   │
│  - AI Edit Chat                                   │
└────────────────────┬──────────────────────────────┘
                     │
┌────────────────────▼──────────────────────────────┐
│  Processing Pipeline                              │
│  FFmpeg → Whisper → Highlights → Face Track →     │
│  Render 9:16 + captions                           │
└─────────────────────────────────────────────────┘

🚀 Quick Start

Prerequisites

Python 3.10+
FFmpeg (with ffprobe and libass)
- Windows: winget install Gyan.FFmpeg
- macOS: brew install ffmpeg
- Linux: sudo apt install ffmpeg libass-dev
Node.js 18+ (for the Electron desktop UI — the server also runs standalone)

Option A — one-click launchers (recommended)

Windows: double-click start_klipzy.bat, or run scripts\run_windows.bat
macOS / Linux: run ./scripts/run_macos.sh

These set up the Python virtual environment and UI dependencies on first run, then launch the desktop app (which starts the backend for you).

Option B — manual

1. Python server (core engine)

python -m venv .venv

# Windows
.\.venv\Scripts\activate
# macOS / Linux
source .venv/bin/activate

pip install -r requirements.txt

# Start the API server
python scripts/main.py

Then open http://127.0.0.1:8765/docs for the interactive API docs.

2. Electron desktop app

cd ui
npm install
npm start

🔒 Local API Security

The backend listens on 127.0.0.1 but is token-authenticated: the desktop app generates a per-launch secret and sends it in the X-Klipzy-Token header. This prevents other web pages in your browser from reaching the local API (which can launch installers and touch the filesystem).

/health and the /docs pages are the only unauthenticated routes.
For scripted / headless use, the token is written to logs/api_token.txt.
Set KLIPZY_DISABLE_AUTH=1 to turn enforcement off (test suite / at your own risk).

🧠 AI Components

Component	Tool	License	Purpose
Transcription	OpenAI Whisper	MIT	Speech → text with word timestamps
Face tracking	YOLOv8 (ultralytics)	AGPL-3.0	Speaker-aware 9:16 crop
Local LLM	Ollama (default) — or any OpenAI-compatible server	MIT	AI edit chat + highlight discovery + clip copywriting
Video processing	FFmpeg	LGPL/GPL	Extract, cut, crop, burn captions

Note on YOLO/ultralytics: Ultralytics is AGPL-3.0. Using it as a dependency is fine; if you distribute a modified version of their library you must share it. For permissive licensing, swap in OpenCV's cv2.CascadeClassifier or MediaPipe (Apache-2.0).

🎚️ Which models should I use?

You don't have to choose. Open Setup and Klipzy checks your CPU / RAM / GPU and marks the best fit with a ⭐. Nothing downloads unless you click.

If you want to decide yourself, the full guide is in INSTRUCTIONS.md § 5 — Choosing Models & AI Engines. The two-line version:

Whisper (subtitles): base for clean single-speaker audio; step up to small / medium for street interviews, crowds, music or crosstalk — this is the setting that most affects caption quality.
Local LLM (hooks/titles, optional): gemma2:2b on 8GB RAM, qwen2.5:7b on 16GB, phi4 if you have ~12GB VRAM. Leave it off entirely and the built-in heuristics still work.

🔌 AI engine — bring your own (optional)

One app, one setting. Ollama is built in and used by default, so there's nothing to configure if you just want it to work. But every AI feature (highlight discovery, hook/title copywriting, AI Edit Chat, caption translation) can be pointed at any OpenAI-compatible server instead — they all speak the same /v1/chat/completions contract, so a single "server address + model name" is all Klipzy needs:

Server	Typical address
LM Studio	`http://localhost:1234/v1`
llama.cpp (`llama-server`)	`http://localhost:8080/v1`
vLLM	`http://localhost:8000/v1`
Ollama's OpenAI route	`http://localhost:11434/v1`
Cloud endpoint	provider's base URL + an API key

Set it in Setup → AI engine: pick Custom (OpenAI-compatible), paste the address, and hit Test connection before saving. Reasoning models that reply via reasoning_content are handled too.

Still local-first. Nothing leaves your machine unless you deliberately enter a cloud address. Every AI feature also degrades to offline heuristics when no engine is reachable, so the app never hard-fails because a model is down.

Subtitle engine (speech-to-text)

Same idea, separate setting. Transcription runs locally by default (MLX / faster-whisper / openai-whisper, auto-selected), and Setup → Subtitle engine can point it at any server that speaks the OpenAI audio-transcription API instead:

Server	Typical address
whisper.cpp (`whisper-server`)	`http://localhost:8080/v1`
faster-whisper-server / Speaches	`http://localhost:8000/v1`
OpenAI	`https://api.openai.com/v1` + an API key

Karaoke captions need word-level timings. Servers that return them get exact per-word highlighting; servers that only return segments still work, but the highlight is spread evenly across each line. Test connection tells you which one you're getting before you commit. If the server stops responding mid-run, Klipzy falls back to local Whisper automatically rather than failing the job.

🧪 Tests

pytest tests/ -v

Covers highlight detection, subtitle generation, NLE exports, aspect-ratio reframing, silence detection, profanity filtering, logging, the API token guard, and the transcription-backend reporter.

Verified end-to-end: a real ~24-min 1080p HEVC video runs through the full pipeline and produces genuine 1080×1920 (9:16) H.264 clips with burned-in captions and thumbnails. On CPU with faster-whisper, a 2-minute source yields finished clips in well under a minute; expect longer for full-length sources and much faster with a CUDA GPU or Apple Silicon (MLX).

GPU note: having an NVIDIA card isn't enough on its own — the ML stack needs the CUDA build of PyTorch. If the Setup panel shows "CPU (GPU idle)", install the CUDA PyTorch build (Setup → Install, or pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126) to unlock GPU speed. The app runs correctly on CPU either way — it just picks the fastest backend it can actually use and falls back safely if an accelerator isn't usable.

📦 Packaging

Build desktop installers from ui/ with electron-builder:

cd ui
npm run dist:win     # Windows (NSIS installer + portable)
npm run dist:mac     # macOS (DMG)
npm run dist:linux   # Linux (AppImage)

Output lands in ui/dist/. The packaged app currently expects Python 3.10+ on the target machine — the Python runtime isn't bundled yet, so a fully self-contained build (via PyInstaller) is planned.

🗺️ Roadmap

Done

Coming soon

URL / stream import — YouTube, Twitch, Kick (yt-dlp)
MLX beyond transcription (highlight/LLM stages on Apple Silicon)
Multi-speaker split-screen
Dubbing — translated audio, not just subtitles
Auto-posting to TikTok / YouTube / Reels
Self-contained desktop build (bundle the Python runtime via PyInstaller)

📄 License

Licensed under the TechFreq Developments Open-Attribution License. Free to use and modify, provided credit is given to TechFreq Developments as the original author. See LICENSE.md for full details.

Third-party notices: Whisper (MIT), FFmpeg (LGPL/GPL), Ollama (MIT), ultralytics (AGPL-3.0).

Name		Name	Last commit message	Last commit date
assets		assets
models		models
scripts		scripts
server		server
ui		ui
.gitattributes		.gitattributes
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
INSTRUCTIONS.md		INSTRUCTIONS.md
LICENSE.md		LICENSE.md
README.md		README.md
fix_electron.bat		fix_electron.bat
requirements.txt		requirements.txt
start_klipzy.bat		start_klipzy.bat
start_klipzy.command		start_klipzy.command

Navigation Menu

Repository files navigation

🎬 Klipzy Studio

📸 Screenshots

✨ Features

🏗️ Architecture

🚀 Quick Start

Prerequisites

Option A — one-click launchers (recommended)

Option B — manual

🔒 Local API Security

🧠 AI Components

🎚️ Which models should I use?

🔌 AI engine — bring your own (optional)

Subtitle engine (speech-to-text)

🧪 Tests

📦 Packaging

🗺️ Roadmap

📄 License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

🎬 Klipzy Studio

📸 Screenshots

✨ Features

🏗️ Architecture

🚀 Quick Start

Prerequisites

Option A — one-click launchers (recommended)

Option B — manual

🔒 Local API Security

🧠 AI Components

🎚️ Which models should I use?

🔌 AI engine — bring your own (optional)

Subtitle engine (speech-to-text)

🧪 Tests

📦 Packaging

🗺️ Roadmap

📄 License

About

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages