RuntimeError: memory access out of bounds (Pyodide 128MB limit)

[deanflaviano]

I'm using py-wrangler to create a FastAPI to route requests to my AI backend, I'm currently streaming the results, but Pyodide keeps running into memory limits (128MB). Is there a way to configure the memory or are there other options for a python FastAPI deployment on Cloudflare using the edge deployments? I'm running into the memory limit when running "uv run pywrangler dev" locally.

[dom96]

Currently 128mb is the maximum a Worker can use. What are you doing in your Worker that is exceeding this? Can you share a minimal repro?

[deanflaviano]

tldr, I tried to create a proxy to route requests to my AI API backend with images 1-5MB in size. I tried most of the different ways of full loading, buffering, streaming, forwarding the requests and also testing request limits and different max connections keep alive connections etc, but I could never get the server to handle more than 5-10 request/second before running into memory limits. When I tried it on a JS worker it could handle 20-40/requests per second before running into issues.

In production, the workers will scale automatically, but imo 5-10 requests per seconds was too low to be reliable. My conclusion was that the Cloudflare python workers add too much bloat in the request handling.

I asked Claude to minimum viable example from my latest try, which gives a rough idea of what I was trying to achieve.

"""
Minimal Cloudflare Worker demonstrating memory issues with image streaming
"""

from fastapi import FastAPI, Request, HTTPException
from starlette.responses import StreamingResponse
from workers import WorkerEntrypoint
import httpx

app = FastAPI(title="CF Worker Memory Test")


async def stream_proxy(request: Request, backend_url: str) -> StreamingResponse:
    """Stream image data from client -> backend -> client without buffering"""
    
    # Prepare headers for backend request
    headers = {"Content-Type": request.headers.get("content-type", "application/json")}
    
    # Conservative connection limits
    limits = httpx.Limits(max_connections=30, max_keepalive_connections=30)
    
    client = httpx.AsyncClient(timeout=None, limits=limits, follow_redirects=False)
    
    # Stream request to backend
    cm = client.stream(
        "POST",
        backend_url,
        content=request.stream(),  # Stream upload without buffering
        headers=headers,
    )
    resp = await cm.__aenter__()
    
    # Handle errors
    if resp.status_code >= 300:
        err_bytes = await resp.aread()
        await cm.__aexit__(None, None, None)
        await client.aclose()
        raise HTTPException(
            status_code=resp.status_code,
            detail=err_bytes.decode("utf-8", errors="ignore") or "Backend error"
        )
    
    # Stream response back to client
    async def iter_response():
        try:
            async for chunk in resp.aiter_bytes():
                if chunk:
                    yield chunk
        finally:
            await cm.__aexit__(None, None, None)
            await client.aclose()
    
    return StreamingResponse(
        iter_response(),
        media_type=resp.headers.get("content-type", "application/octet-stream"),
        status_code=resp.status_code,
    )


@app.get("/")
async def root():
    """Health check"""
    return {"status": "ok", "message": "Memory test worker"}


@app.post("/process_image")
async def process_image(request: Request):
    """
    Accept image and proxy to backend
    
    Expected request body (multipart/form-data or JSON with base64):
    {
        "image": "base64_encoded_image_data" or binary file
    }
    """
    # Get backend URL from environment
    backend_url = request.scope.get('env', {}).get(
        'BACKEND_URL',
        'https://your-test-backend.example.com/process'
    )
    
    return await stream_proxy(request, backend_url)


class Default(WorkerEntrypoint):
    async def fetch(self, request):
        """Cloudflare Worker entry point"""
        import asgi
        return await asgi.fetch(app, request, self.env)

[dom96]

What issues did you run into with the JS worker? Was it memory problems too? If you're running into limitations with JS Workers then Python Workers are going to suffer from the same limits.

[deanflaviano]

Yes, both ran out of memory, but the JS workers requires a lot less memory per request, as mentioned the python workers can only handle 5-10 request/second and the JS workers could handle 20-40/requests. I believe the Python Workers is a good use-cases to use as proxy AI APIs, however, they memory handling makes the scaling of the workers in Prod a risk factor, while JS workers have more room to scale and avoid the memory limits.