Skip to content

L3tum/LTEngine

BranchesTags

Repository files navigation

LTEngine

Free and Open Source Local AI Machine Translation API, written in Rust, entirely self-hosted and compatible with the LibreTranslate API. Its translation capabilities are powered by large language models (LLMs) running on an external inference server that supports the OpenAI chat completions API (e.g., Ollama, vLLM, llama.cpp server).

Fork with extensive redesign and feature work. This project was forked from the original LibreTranslate/LTEngine repository and has been significantly reworked with new features, improvements, and independent development. Public Docker images are available at ghcr.io/l3tum/ltengine.

Translation

⚠️ LTEngine is in active development. Check the Roadmap for current limitations.

💡 Migration from llama.cpp version: Previous versions of LTEngine ran llama.cpp directly (with GPU layer offloading). This version uses an external backend (Ollama, vLLM, etc.) via the OpenAI-compatible API. Models must now be loaded on the backend side. To upgrade:

  1. Set up an external inference server (e.g., ollama pull gemma3-4b)
  2. Run with CLI: ltengine --backend-host localhost:11434 -m gemma3-4b
  3. Or via environment variables: export LTE_BACKEND_HOST=localhost:11434 LTE_MODEL=gemma3-4b && ltengine
  4. The old --model-file and --cpu options are removed — the backend handles GPU/CPU decisions.

Requirements

  • Rust
  • A compatible backend server (Ollama recommended for local use):
    • Ollama — easiest for local GGUF models (GPU support built-in)
    • vLLM — high-throughput serving for HuggingFace models
    • llama.cpp server — compatible with any GGUF model

Build

git clone https://github.com/l3tum/LTEngine
cd LTEngine
cargo build --release

Run

./target/release/ltengine --backend-host localhost:11434 -m gemma3-4b

To use a different backend or model:

./target/release/ltengine --backend-host http://localhost:8000 -m meta-llama/Llama-3-8b

Error handling:

  • Retry logic: LTEngine automatically retries failed requests with exponential backoff (default: 5 attempts) for transient errors (timeouts, connection issues, HTTP 5xx). Client errors (401 auth, 404 model not found) are returned immediately with clear messages.
  • Model validation: On provider initialization (lazy, triggered by first request or periodic recheck), the provider checks if the specified model is available on the backend (queries Ollama's /api/tags or /v1/models). If the model is not found, it lists available models and fails gracefully without crashing the service.
  • API keys: Use --api-key to require authentication on incoming translation requests. Use --backend-api-key to authenticate with the external backend (passed as Bearer token). The keys are never logged; only status codes appear in error messages.

Models

LTEngine now delegates model loading to the backend server. You can use any model supported by your inference backend. For Ollama, common choices include:

Model Notes
gemma3-4b (default) Good balance of quality/speed
gemma3-12b Higher quality, slower
llama3.3-8b Well-rounded general model
gemma3-27b Best quality, slowest

See your backend's documentation for the full model list and how to pull/load models.

Tip for Docker users: Run ollama pull gemma3-4b in your Ollama container before starting the service, or pull it from the host. The docker-compose setup below handles this automatically.

Simple

Request:

const res = await fetch("http://0.0.0.0:5050/translate", {
  method: "POST",
  body: JSON.stringify({
    q: "Hello!",
    source: "en",
    target: "es",
  }),
  headers: { "Content-Type": "application/json" },
});

console.log(await res.json());

Response:

{
    "translatedText": "¡Hola!"
}

For JSON requests, q can also be an array of strings for batch translation (API-compatible with the original LibreTranslate project):

{
    "q": ["Hello!", "Goodbye!"],
    "source": "en",
    "target": "es"
}

The response keeps the same shape, but translatedText becomes an array. When source is "auto", detectedLanguage is also returned as an array. Batch requests accept up to 50 strings by default; configure this with --batch-limit / LTE_BATCH_LIMIT.

List of language codes: https://0.0.0.0:5000/languages

Auto Detect Language

Request:

const res = await fetch("http://0.0.0.0:5000/translate", {
  method: "POST",
  body: JSON.stringify({
    q: "Ciao!",
    source: "auto",
    target: "en",
  }),
  headers: { "Content-Type": "application/json" },
});

console.log(await res.json());

Response:

{
    "detectedLanguage": {
        "confidence": 83,
        "language": "it"
    },
    "translatedText": "Bye!"
}

Health Check Endpoints

Service health: GET /health

const res = await fetch("http://localhost:5050/health");
console.log(await res.json());
// { "status": "ok", "service": "ltengine", "version": "0.1.1" }

Backend health: GET /health/backend — checks if the translation backend is reachable and returns the backend status (200 or 503).

const res = await fetch("http://localhost:5050/health/backend");
console.log(await res.json());
// { "status": "ok", "backend": "Backend is reachable" }
// or { "status": "error", "backend": "Backend not reachable: ..." }

These endpoints are useful for health probes in load balancers and monitoring tools.

API Notes

Source equals Target Behavior

If the source and target parameters are the same (e.g., both "en"), the API returns the original text without calling the translation backend. This is a performance optimization to avoid unnecessary LLM calls, but it means that the text is not "improved" or "corrected" in the same language. If you need text improvement in the same language, you should use a different approach (e.g., set the source to "auto" or use a different API).

Retry and Reliability

The service will not crash if the backend is initially unreachable. During startup:

  • It attempts to connect to the backend with exponential backoff (default: 5 attempts with base delay of 1s)
  • If all attempts fail, it logs the error but the service continues running
  • The first translation request will retry and attempt to recreate the provider

If the backend becomes unreachable during operation:

  • Translation requests will automatically retry with exponential backoff
  • If the retry fails after the maximum attempts, a 503 error is returned to the client
  • Periodic background rechecking (default: every 30 seconds) will detect when the backend comes back online and recreate the provider

CLI Options

./target/release/ltengine \
  --host 0.0.0.0 \
  --port 5050 \
  --backend-host localhost:11434 \
  --model gemma3-4b \
  --api-key your-secret-key \
  --backend-api-key your-backend-key \
  --retry-attempts 5 \
  --retry-delay 1000 \
  --model-recheck-interval 30 \
  --char-limit 50000 \
  --chunk-size 5000 \
  --max-parallel 2 \
  --batch-limit 50

All CLI options can also be configured via environment variables (with CLI arguments taking precedence over environment variables):

CLI Option Environment Variable Description Default
--host LTE_HOST Hostname to bind the server to 0.0.0.0
--port LTE_PORT Port to bind the server to 5050
--model LTE_MODEL Model name to use (passed to the backend) gemma3-4b
--backend-host LTE_BACKEND_HOST Backend server address (host:port or full URL) localhost:11434
--api-key LTE_API_KEY Require this API key on incoming translation requests (none)
--backend-api-key LTE_BACKEND_API_KEY API key to send to the external backend as Bearer token (none)
--retry-attempts LTE_RETRY_ATTEMPTS Max retry attempts for provider creation and translation 5
--retry-delay LTE_RETRY_DELAY Base delay in ms for exponential backoff 1000
--model-recheck-interval LTE_MODEL_RECHECK_INTERVAL Seconds between periodic model availability rechecks (0 to disable) 30
--char-limit LTE_CHAR_LIMIT Maximum total source text length; requests exceeding this are rejected with 400 50000
--chunk-size LTE_CHUNK_SIZE Maximum size of individual translation chunks; text longer than this is chunked 5000
--max-parallel LTE_MAX_PARALLEL Maximum concurrent translation requests per input text when chunked 2
--batch-limit LTE_BATCH_LIMIT Maximum number of strings accepted in JSON q arrays 50

Environment Variables Example

You can configure LTEngine entirely via environment variables (useful for Docker/containerized deployments):

export LTE_BACKEND_HOST="ollama:11434"
export LTE_MODEL="gemma3-4b"
export LTE_PORT="5050"
export LTE_API_KEY="my-secret-key"
./target/release/ltengine

Benchmark Mode

LTEngine includes a benchmark mode that evaluates translation quality and speed against a dataset. This is useful for comparing models and measuring performance.

Running Benchmarks

The benchmark feature is enabled by default (feature flag benchmark). It uses bleuscore for BLEU score calculation.

# Run with built-in dataset, plain text output
./target/release/ltengine --backend-host localhost:11434 -m gemma3-4b --benchmark text

# Run with custom dataset, generate HTML report
./target/release/ltengine --benchmark html --dataset my_dataset.json

# Or via environment variables
LTE_BENCHMARK=text ./target/release/ltengine --backend-host localhost:11434 -m gemma3-4b

# HTML report
LTE_BENCHMARK=html LTE_DATASET=my_dataset.json ./target/release/ltengine --backend-host localhost:11434 -m gemma3-4b

Options:

  • --benchmark text|html — output mode: text prints a plain text report to stdout; html generates an HTML report (benchmark_report.html) in the current directory
  • --dataset <path> — path to a custom dataset JSON file (optional, uses a built-in default if not provided)

The benchmark runs through a set of source texts with known translations, compares results using BLEU scoring, and reports overall quality metrics. The HTML report includes a table of per-language BLEU scores.

Dataset Format

Dataset files are JSON arrays of translation pairs:

[
  {
    "source": "Hello, world!",
    "target": "¡Hola, mundo!",
    "source_lang": "en",
    "target_lang": "es"
  }
]

You can use the LTEngine API using the following bindings (they work due to API compatibility with the original LibreTranslate project):

Roadmap

  • Remove llama.cpp mutex — concurrent translation requests are now supported via the HTTP provider
  • Cancel inference — HTTP connection drops are now handled by the request cancellation in the provider
  • Retry logic with exponential backoff for transient backend errors
  • Model availability validation at startup
  • Health check endpoints (/health, /health/backend)
  • Fully static Docker image (musl, no dynamic dependencies)
  • Benchmark mode with BLEU score and HTML reports
  • Add support to select tone for translation
  • Add support for chunking to support larger texts
  • Add support for /translate_file (ability to translate files).
  • Add support for sentence splitting/line splitting similar to DeepL.
  • Better language detection for short texts (port LexiLang to Rust)
  • Add support for command line inference (run ./ltengine translate as a command line app separate from ./ltengine server)
  • Make ltengine available as a library, possibly creating bindings for other languages like Python.
  • Automated builds / CI
  • Configurable retry logic with exponential backoff
  • Periodic model availability re-checking with auto-recovery
  • Resilience against initial backend unreachability (no crash on startup)
  • Expand benchmark datasets for more comprehensive model comparison.
  • Your ideas? We welcome contributions.

Contributing

We welcome contributions! Just open a pull request.

Credits

This project uses Rust crates for HTTP communication (reqwest) and language detection (whatlang-rs). The underlying language models are trained by various open-source communities (Gemma, Llama, Mistral, etc.).

License

GNU Affero General Public License v3

About

No description, website, or topics provided.

Resources

Readme

License

AGPL-3.0 license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

16 tags

Packages

 
 
 

Contributors

Languages