Skip to content

server: context_geometry reports the slot's capped context, the one size the slots enforce #111

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
joelteply wants to merge 1 commit into
base: codex/s4-per-mind-integration
Choose a base branch
from
Open

server: context_geometry reports the slot's capped context, the one size the slots enforce #111

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions tools/server/server-context.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -884,6 +884,12 @@ struct server_context_impl {
// serving slots working right now, refreshed by the loop at the start and end of every
// update_slots pass; the trainer reads it between batches (see server-train.h)
std::atomic<int> n_busy_slots{0};

// The most one sequence may hold: llama_n_ctx_seq capped at the model's training context.
// ONE value, set at load: every slot is sized from it and context_geometry reports it, so
// the report never names a size the slots refuse (it said 524288 while each slot capped at
// 262144, the M5 2026-11-11).
int n_ctx_slot = 0;
void publish_busy_slots() {
int n = 0;
for (const auto & slot : slots) {
Expand Down Expand Up @@ -1387,7 +1393,7 @@ struct server_context_impl {

const int n_ctx_train = llama_model_n_ctx_train(model_tgt);

int n_ctx_slot = llama_n_ctx_seq(ctx_tgt);
n_ctx_slot = llama_n_ctx_seq(ctx_tgt);
if (n_ctx_slot > n_ctx_train) {
SRV_WRN("the slot context (%d) exceeds the training context of the model (%d) - capping\n", n_ctx_slot, n_ctx_train);
n_ctx_slot = n_ctx_train;
Expand Down Expand Up @@ -5091,7 +5097,7 @@ server_context_meta server_context::get_meta() const {
}(),
/* context_geometry */ json {
{ "allocated_tokens", llama_n_ctx(impl->ctx_tgt) },
{ "max_tokens_per_sequence", llama_n_ctx_seq(impl->ctx_tgt) },
{ "max_tokens_per_sequence", impl->n_ctx_slot },
{ "sequence_capacity", llama_n_seq_max(impl->ctx_tgt) },
},
};
Expand Down
Loading