Add K8s eval support, Kaniko build pipeline, and OpenAI config#1
Open
qywu wants to merge 1 commit into
Open
Conversation
Key changes: - examples/hugging_face_task/main.py: K8s mode via ENVIRONMENT_IMAGE env var, per-eval Pod+Service lifecycle, ClusterIP-based ENV_URL (no cluster DNS needed), ENV=local override for subprocess settings validation - build_and_push.sh: Kaniko-based image build using shared PVC context, Dockerfile.kaniko generation that replaces BuildKit secrets with ARG/ENV - environment/Dockerfile.kaniko: Kaniko-compatible Dockerfile with ARG/ENV GITHUB_TOKEN and sed patch to redirect mercor-mcp-shared git dep to local stub package - mcp_servers/packages/mercor-mcp-shared/: Stub package providing mcp_schema (GeminiBaseModel, flatten_schema, etc.) without requiring private GitHub access - examples/hugging_face_task/k8s_environment.yaml: K8s manifests reference - examples/hugging_face_task/mcp_config_all_oss_servers.json: Added PYTHONPATH override for servers depending on mercor-mcp-shared stub - agents/pyproject.toml, grading/pyproject.toml: Bump litellm to 1.86.2 - orchestrator_config.json, grading_settings.json: Switch to OpenAI models
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
hugging_face_task: setENVIRONMENT_IMAGEto skip docker-compose and run a fresh Pod+Service per eval run; uses ClusterIP directly (no cluster DNS required from dev pods)build_and_push.sh: Kaniko-based image build using the shared WekaFS PVC as build context — no Docker daemon neededenvironment/Dockerfile.kaniko: Kaniko-compatible Dockerfile replacing BuildKit--mount=type=secretwithARG/ENV GITHUB_TOKEN, plus a sed patch that redirects the privatemercor-mcp-sharedgit dependency to a local stubmcp_servers/packages/mercor-mcp-shared/: Stub package that providesmcp_schema(GeminiBaseModel,flatten_schema, etc.) without requiring access to the private Mercor-Intelligence GitHub orgPYTHONPATHpointing to the stub for servers that depend onmercor-mcp-shared; removedcode_execution_server(sandbox.socompile issue in Kaniko build)k8s_environment.yaml: Reference K8s manifests for the environment pod/service1.83.10→1.86.2in bothagents/andgrading/Test plan
export ENVIRONMENT_IMAGE=<registry>/archipelago-environment:latest && ./build_and_push.shcd examples/hugging_face_task && ./run.sh(oruv run python main.pywithENVIRONMENT_IMAGEset)