Skip to content

Authentication bypass in MemOS server: internal-service check fails open when INTERNAL_SERVICE_SECRET is unset #2259

Closed as not planned

Description

@geo-chen

Pre-submission checklist | 提交前检查

  • I have searched existing issues and this hasn't been mentioned before | 我已搜索现有问题,确认此问题尚未被提及
  • I have read the project documentation and confirmed this issue doesn't already exist | 我已阅读项目文档并确认此问题尚未存在
  • This issue is specific to MemOS and not a general software issue | 该问题是针对 MemOS 的,而不是一般软件问题

Bug Description | 问题描述

reported on 15 June 2026: https://github.com/MemTensor/MemOS/security/advisories/GHSA-9pw6-vmgx-qgwx

Summary

The MemOS authenticated server overlay (memos.api.server_api_ext:app, shipped as
docker/Dockerfile.krolik) protects its admin API-key management endpoints with the
verify_api_key dependency. Before validating a key, that dependency calls
is_internal_request() to allow trusted container-to-container calls. The header branch of
that function is:

internal_header = request.headers.get("X-Internal-Service")
return internal_header == os.getenv("INTERNAL_SERVICE_SECRET")

INTERNAL_SERVICE_SECRET has no default and is not set in any shipped configuration
(Dockerfile, docker-compose, or the Helm chart). When it is unset, os.getenv(...) returns
None. A normal external request that does not send the X-Internal-Service header has
request.headers.get("X-Internal-Service") == None. The comparison becomes None == None,
which is True, so the request is treated as an internal service and granted
scopes: ["all"] without any API key.

The result is that with AUTH_ENABLED=true set (the operator has explicitly turned on
authentication), an unauthenticated remote attacker is authorized as a fully privileged
internal principal and can reach the admin API-key endpoints (create keys, list keys, revoke
keys, generate a master key) as well as all data endpoints. The check fails open rather than
closed.

Affected component

  • File: src/memos/api/middleware/auth.py
  • Functions: is_internal_request() (line 154) and verify_api_key() (line 182)
  • Consumed by: src/memos/api/routers/admin_router.py (every /admin/* route via
    Depends(verify_api_key) / Depends(require_scope("admin")))
  • Deployment: src/memos/api/server_api_ext.py (docker/Dockerfile.krolik)

Impact

  • Complete bypass of API-key authentication on a deployment that has explicitly enabled it
    (AUTH_ENABLED=true) but has not set the undocumented, defaultless INTERNAL_SERVICE_SECRET.
  • The bypassed principal receives scopes: ["all"], which satisfies require_scope("admin").
  • An unauthenticated attacker can mint new API keys for any user, enumerate existing keys,
    revoke keys (denial of service), and generate a master key, then use those keys for
    persistent privileged access.
  • CWE-697 (Incorrect Comparison) / CWE-305 (Authentication Bypass by Primary Weakness),
    fail-open.

Root cause

is_internal_request() compares two values that are both None in the default deployment:

  1. request.headers.get("X-Internal-Service") is None because a normal request does not
    send that header.
  2. os.getenv("INTERNAL_SERVICE_SECRET") is None because the variable is never set.

None == None is True, so every header-less request is classified as internal. The safe
behavior is to treat an unset secret as "internal-via-header is disabled" and never match.

How to Reproduce | 如何重现

Proof of Concept

Prerequisites:

  • A clone of MemOS at v2.0.19.
  • Python 3.11+ with fastapi, starlette, pydantic available.
  • No special configuration. INTERNAL_SERVICE_SECRET is intentionally left unset, exactly as
    in every shipped Dockerfile / compose / Helm config. AUTH_ENABLED is set to true to show
    that authentication is enabled yet bypassed.

The PoC calls the real verify_api_key dependency the way FastAPI calls it for an admin
route. It simulates a remote client (source IP 203.0.113.9, which is not in the trusted
INTERNAL_SERVICE_IPS set) and sends no API key.

Save as poc.py inside the repository root and run with PYTHONPATH=src python3 poc.py:

import asyncio, os
# Operator turned authentication ON. INTERNAL_SERVICE_SECRET is left unset (the default).
os.environ["AUTH_ENABLED"] = "true"
os.environ.pop("INTERNAL_SERVICE_SECRET", None)

from starlette.requests import Request
from memos.api.middleware import auth as A

def make_request(headers):
    scope = {
        "type": "http",
        "method": "GET",
        "path": "/admin/keys",
        "client": ("203.0.113.9", 53124),  # external attacker, NOT an internal IP
        "headers": [(k.lower().encode(), v.encode()) for k, v in headers.items()],
    }
    return Request(scope)

async def main():
    print("AUTH_ENABLED          =", A.AUTH_ENABLED)
    print("INTERNAL_SERVICE_SECRET =", os.getenv("INTERNAL_SERVICE_SECRET"))

    # Attacker: external IP, no X-Internal-Service header, no API key.
    result = await A.verify_api_key(make_request({}), api_key=None)
    print("\n[A] no header, no key ->", result)

    # Control: same request but with a guessed header value -> correctly rejected.
    try:
        await A.verify_api_key(make_request({"X-Internal-Service": "guess"}), api_key=None)
        print("[B] wrong header value -> NOT rejected")
    except Exception as e:
        print("[B] wrong header value -> rejected:",
              getattr(e, "status_code", ""), getattr(e, "detail", e))

asyncio.run(main())

Observed output:

AUTH_ENABLED          = True
INTERNAL_SERVICE_SECRET = None

[A] no header, no key -> {'user_name': 'internal', 'scopes': ['all'], 'is_master_key': False, 'is_internal': True}
[B] wrong header value -> rejected: 401 Missing API key

Case [A] shows that an unauthenticated external request is granted scopes: ['all'] as the
internal principal, which satisfies require_scope("admin") and unlocks every /admin/*
endpoint. Case [B] confirms the gate works correctly whenever the two sides of the comparison
differ; the vulnerability is exactly the None == None fail-open.

Over HTTP against a running server_api_ext instance, the equivalent request is:

curl -s http://TARGET:8000/admin/keys

which passes the verify_api_key / require_scope("admin") dependencies as the internal
principal instead of returning 401.

Additional observation (secondary)

server_api_ext.py mounts server_router with
app.include_router(server_router) and no router-level auth dependency, so the /product/*
data endpoints (/product/search, /product/get_all, /product/delete_memory,
/product/chat/*, etc.) have no authentication even when AUTH_ENABLED=true. Only
/admin/* carries Depends(verify_api_key). The fail-open issue above is what additionally
exposes the admin key-management surface.

Suggested fix

  • In is_internal_request(), treat an unset secret as disabled and require a non-empty match:
secret = os.getenv("INTERNAL_SERVICE_SECRET")
internal_header = request.headers.get("X-Internal-Service")
if not secret or not internal_header:
    return False
return hmac.compare_digest(internal_header, secret)
  • Use a constant-time comparison (hmac.compare_digest) to avoid timing leaks.
  • Apply Depends(verify_api_key) (or an appropriate require_scope) to the server_router
    data endpoints so they are protected when AUTH_ENABLED=true.

Environment | 环境信息

docker

Additional Context | 其他信息

No response

Willingness to Implement | 实现意愿

  • I'm willing to implement this myself | 我愿意自己解决
  • I would like someone else to implement this | 我希望其他人来解决

Activity

  1. added
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    area:coreMOS 编排层 / 框架底座 / 跨模块问题
    status:in-progressSomeone or AI is working on it | 人工或 AI 正在处理
    types:bugSomething isn't working | 功能异常
    on Aug 16, 2026
  2. self-assigned this
    on Aug 16, 2026
  3. Memtensor-AI commented on Aug 16, 2026

    @Memtensor-AI
    Collaborator

    🤖 AutoDev has picked up this issue and started working on it.

    Task ID: 18aede470455b543
    Working branch: bugfix/autodev-2259-20260816223848617
    Target branch: latest dev* branch
    Workflow: opsp (analysis → coding → testing → PR)

    I will post the PR link here once done. If I need more information, I will ask in the comments.

  4. Memtensor-AI commented on Aug 16, 2026

    @Memtensor-AI
    Collaborator

    ✅ AutoDev task 18aede470455b543 completed.

    Summary: Closes the fail-open authentication bypass in memos.api.middleware.auth.is_internal_request() reported in GHSA-9pw6-vmgx-qgwx / issue #2259. In every shipped deployment INTERNAL_SERVICE_SECRET is unset, so os.getenv(...) returned None; a normal external request also produced None from request.headers.get("X-Internal-Service"), and None == None authorised anonymous callers as the internal principal with scopes: ["all"], satisfying require_scope("admin") on every /admin/* route.

    The fix reads the secret into a local variable, returns False when either the secret or the header is missing/empty, and compares the two non-empty strings with hmac.compare_digest for constant-time equality. The INTERNAL_SERVICE_IPS allowlist branch is unchanged. No public API, request/response model, config schema, Dockerfile, Helm chart or docs modified — deployments that never set the secret simply lose the never-safely-usable header path.

    Added tests/api/test_auth_internal_request.py with 12 regression cases: five reproducing the exact fail-open scenarios (all failed on the unpatched baseline as expected), one asserting hmac.compare_digest is invoked, plus positive-path and end-to-end coverage that runs the advisory PoC against verify_api_key. After the fix all 12 pass; the advisory PoC now returns 401 Missing API key for both header-less and wrong-header external requests. ruff check and ruff format are clean on both changed files. Neighbour tests (tests/api/test_lifecycle.py, test_product_models.py, test_thread_context.py) still green; the 4 pre-existing test_mcp_serve.py failures reproduce identically on the unpatched baseline (missing pytest-asyncio marker in this CI env) and are unrelated to this change.

    Scope note: the reporter also observed as a secondary issue that server_router mounted by server_api_ext.py has no auth dependency on /product/* data endpoints. That change touches public routes (AGENTS.md "Ask first" list) and is deferred to a separate issue — this PR intentionally keeps scope tight to the primary fail-open per issue #2259.

    Base branch: main
    Branch: bugfix/autodev-2259-20260816223848617
    Commit: 20dd858
    PR: #2260
    Assigned to: @syzsunshine219
    Reviewers: @WeiminLee

  5. added
    ai:testingAI agent is running tests | AI 正在运行测试
    ai:pr-readyAI-created PR is ready for review | AI 生成的 PR 已等待评审
    and removed
    ai:codingAI agent is coding | AI 正在编码
    ai:testingAI agent is running tests | AI 正在运行测试
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    on Aug 16, 2026
  6. github-actions commented on Sep 16, 2026

    @github-actions
    Contributor

    This issue has been automatically marked as stale due to inactivity.

  7. added
    status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.
    on Sep 16, 2026
  8. github-actions commented on Sep 23, 2026

    @github-actions
    Contributor

    This issue has been automatically closed due to inactivity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

ai:pr-readyAI-created PR is ready for review | AI 生成的 PR 已等待评审area:coreMOS 编排层 / 框架底座 / 跨模块问题status:in-progressSomeone or AI is working on it | 人工或 AI 正在处理status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.types:bugSomething isn't working | 功能异常

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions