fix: size the model pool from the cgroup memory limit

[ChiragAgg5k]

Problem

The pool sizer and sub-batch sizer read System::available_memory(), which inside a container reports the host, not the cgroup. Running appwrite/embedding:0.1.0 with --memory=1g on a 20 GB Docker host logs available_mb=10945, picks pool_size=2, loads ~1.6 GB of model instances, gets OOM-killed, and with restart: unless-stopped loops at ~100% CPU (8 restarts in 30 s in my repro). Any mem_limit an operator sets in Compose is therefore ignored by the sizer.

This surfaced in a self-hosted Appwrite report where a 4 GB Hetzner box ran out of memory and the kernel OOM-killed mysqld during a function build.

Fix

Take the minimum of host-available and cgroup_limits().free_memory (sysinfo already honours memory.max / memory.limit_in_bytes, falling back to host total when unlimited) for both the pool budget and the sub-batch budget. Also emit a warning when a single instance is estimated to exceed the budget, since that is the case where the container can still be killed under load, and let the sub-batch shrink to one text so a small budget is never exceeded by the batch floor.

 20 GB host, container started with --memory=1g
 ┌──────────────────────────────────────────────────────────────┐
 │ /proc/meminfo            sys.available_memory()  = 10945 MB  │  what 0.1.0 used
 │ /sys/fs/cgroup/memory.max                                    │
 │   - memory.current       cgroup free_memory      =  1015 MB  │  the real ceiling
 └──────────────────────────────────────────────────────────────┘
                  │                          │
                  └────────── min ───────────┘
                               │
                    memory_budget() = 1015 MB
                               │
            ┌──────────────────┴──────────────────┐
            ▼                                     ▼
   pool: 60% budget / (instance × 3)     sub-batch: 50% budget / MB-per-text
        → pool_size = 1                       → clamp(1, 16)
     (0.1.0 picked 2 → OOM loop)          (was clamp(4, 16), could overshoot)

fn memory_budget(host_available: u64, cgroup_free: Option<u64>) -> u64 {
    match cgroup_free {
        Some(cgroup_free) => host_available.min(cgroup_free),
        None => host_available,
    }
}

let mem_before_loading_model = memory_budget(
    sys.available_memory(),
    sys.cgroup_limits().map(|limits| limits.free_memory),
);

Verification

Patched image under --memory=1g:

Measured ONNX model memory footprint per_instance_mb=852 estimated_with_arena_mb=2557 available_mb=1015 budget_mb=609 nproc=3 desired=3 max_from_memory=1 capped=1
Initialized embedding model: NomicEmbedTextV15 (768d, pool_size=1, execution_provider=CPU)

0 restarts, 854 MiB steady. The same run on 0.1.0 reports available_mb=10770.

Unit tests cover the budget function for no cgroup, cgroup below host, and unlimited cgroup above host. cargo clippy --all-targets -- -D warnings and cargo test pass on rustc 1.95.

[greptile-apps]

Greptile Summary

This PR makes model-pool and request sub-batch sizing respect cgroup memory availability rather than relying solely on host-reported memory.

• Caps the effective memory budget at the lower of host-available and cgroup-free memory.
• Warns when one model instance is estimated to exceed the pool budget.
• Allows derived sub-batches to shrink to one text under constrained memory.
• Adds focused tests for cgroup budgeting and low-memory batch sizing.

Confidence Score: 5/5

The PR appears safe to merge; the previously reported low-memory batching issue is resolved and no new actionable failures were identified.

The current implementation derives both pool and sub-batch capacity from the lower of host-available and cgroup-free memory, and zero or very small reported budgets now produce the minimum sub-batch of one instead of the unsafe fallback of 32. The previous thread was manually resolved after this correction.

Important Files Changed

Filename	Overview
src/embedding.rs	Applies cgroup-aware memory budgets to model pools and sub-batches; the latest change fully addresses the previously reported unsafe zero-memory fallback.

_{Reviews (5): Last reviewed commit: "fix: treat zero available memory as exha..." | Re-trigger Greptile}