feat: support user-impersonated backend job launches by rohan-uiuc · Pull Request #40 · Center-for-AI-Innovation/LLMHub

rohan-uiuc · 2026-07-16T14:07:31Z

Summary

port Center-for-AI-Innovation/llm-serving-backend#32 into the backend subtree
submit vec-inf jobs as the requesting cluster user
add impersonation setup scripts, configuration, and launch-flow tests
retain current monorepo Delta resource types while adding the incoming backend values

Impact

Backend deployments can launch under the requesting cluster user when impersonation mode is configured. Direct execution remains supported.

Validation

Python syntax compilation passed for app, tests, and the new scripts
git diff --check passed
targeted pytest execution is currently blocked because pyproject.toml permits Python 3.9 while vec-inf 0.9.0 requires Python 3.10 or newer

Source

Ports Center-for-AI-Innovation/llm-serving-backend#32.

…g-backend into job-as-user

joshfactorial · 2026-07-17T20:31:56Z


 # Environment variable that overrides auto-detection (e.g. INFRASTRUCTURE=delta-ai-ncsa)
 INFRASTRUCTURE_ENV_VAR = "INFRASTRUCTURE"
+_CLUSTER_USERNAME_RE = re.compile(r"^[A-Za-z][A-Za-z0-9._-]{0,63}$")


I'm wondering if the regex is too Delta-aware. Maybe it should be a configuration item?

joshfactorial · 2026-07-17T20:40:10Z

+    return str(Path(raw_root).expanduser() / username)


 def get_vec_inf_log_base_dir(infrastructure: Optional[str] = None) -> Optional[str]:


Same definition as above, maybe missed deleting this one?

joshfactorial · 2026-07-17T20:53:42Z

+            workspace_dir = _ensure_impersonated_workspace_dir(cluster_username)
+            _ensure_shared_cache_dir_access(cluster_username)
+            if workspace_dir is not None:
+                params.setdefault("work_dir", str(workspace_dir))


okay, weird issue. params is set as

params = deployment.model_dump(exclude={"modelName", "modelId", "userId"})

model_dump sets work_dir:

work_dir: Optional[str] = None # Optional working directory for vec-inf jobs

and setdefault silently refuses to override None:

>>>my_dict = {"work_dir": None} >>>my_dict.setdefault("work_dir", "test") >>>my_dict {'work_dir': None}

joshfactorial · 2026-07-17T21:02:02Z

-
+    VEC_INF_LOG_DIR: Optional[str] = None  # Shared vec-inf log directory override
+    VEC_INF_SHARED_WORK_ROOT: Optional[str] = None  # Parent directory for impersonated per-user logs/scripts
+    VEC_INF_EXECUTION_MODE: str = "direct"  # direct or impersonate


Can we restrict to "direct" or "impersonate" here?

joshfactorial · 2026-07-17T21:02:52Z

+    ]
+    for command in acl_commands:
+        try:
+            subprocess.run(command, text=True, capture_output=True, check=True)


Add a generous timeout?

joshfactorial · 2026-07-17T21:17:33Z

Claude logged some security concerns, but I didn't know how much of that to mention. I know we were saving a security pass for later. Here's what claude had. I can make tickets for them if desired

Shared model cache is writable by all impersonated users. _ensure_shared_cache_dir_access grants every user rwx + default ACLs on the shared HuggingFace/torch-inductor cache dirs. Any user can modify cached model artifacts that other users' jobs load — with pickle-bearing formats this is code execution as the victim. Already TODO'd in code as temporary. Fix: gate behind an explicit opt-in setting (default off), then remove once cache population is managed centrally.
(More of a documontation thing, but the kind of thing security might ask about) Impersonation wrapper is an unrestricted sudo trampoline. scripts/impersonate-wrapper.py runs any command as any user and preserves PYTHONPATH, so the feature's entire security boundary is the sudoers rule. Fix: narrow the sudoers grant (restrict target users/group and ideally pin the interpreter/command path); document the required sudoers shape in scripts/README.md.
Raw subprocess output leaks into API error responses. _parse_impersonated_response returns stderr/stdout verbatim in the error field; sudo failures expose host paths, env/config details. Fix: log raw output server-side, return a generic error to clients.

minump · 2026-08-06T14:42:43Z

@rohan-uiuc was wondering if you can resolve the merge conflicts and Josh's review.. Will get started on reviewing this soon.

Vismayak · 2026-08-06T20:57:37Z

@rohan-uiuc I have tried running this in the delta service machine as the service user svcdeltallmhub. In the backend, I have set these values to allow impersonation

VEC_INF_EXECUTION_MODE=impersonate
VEC_INF_IMPERSONATE_PYTHON="/u/svcdeltallmhub/mohanar2/LLMHub/backend/.venv/bin/python"

I understand this is strictly a backend PR only so to test it I need to run the deployments API

curl -X POST http://localhost:8002/api/models/deployments \
  -H 'Content-Type: application/json' \
  -d '{
    "modelName": "Qwen2.5-0.5B-Instruct",
    "userId": "0ea3686b-10b2-42f0-85ab-60cdadbb87e2",
    "clusterUsername": "svcllmhubmohanar2",
    "partition": "gpuA40x4",
    "time": "01:00:00",
    "resource_type": "nvidia_a40"
  }'

However, I am getting this error on running it

{"detail":"/var/tmp/scleylNJB: line 5: /u/svcdeltallmhub/mohanar2/LLMHub/backend/.venv/bin/python: Permission denied"}

Is this because the impersonated user svcllmhubmohanar2, does not have the permission to run the python command to launch the job? Do I need to set something else to circumvent this?

P.S I think we should make a seperate issue to update deployments API, though the json parameter is modelName we actually provide the model ID

Vismayak · 2026-08-06T21:17:30Z

@rohan-uiuc I have tried running this in the delta service machine as the service user svcdeltallmhub. In the backend, I have set these values to allow impersonation
VEC_INF_EXECUTION_MODE=impersonate
VEC_INF_IMPERSONATE_PYTHON="/u/svcdeltallmhub/mohanar2/LLMHub/backend/.venv/bin/python"
I understand this is strictly a backend PR only so to test it I need to run the deployments API
curl -X POST http://localhost:8002/api/models/deployments \
  -H 'Content-Type: application/json' \
  -d '{
    "modelName": "Qwen2.5-0.5B-Instruct",
    "userId": "0ea3686b-10b2-42f0-85ab-60cdadbb87e2",
    "clusterUsername": "svcllmhubmohanar2",
    "partition": "gpuA40x4",
    "time": "01:00:00",
    "resource_type": "nvidia_a40"
  }'
However, I am getting this error on running it
{"detail":"/var/tmp/scleylNJB: line 5: /u/svcdeltallmhub/mohanar2/LLMHub/backend/.venv/bin/python: Permission denied"}
Is this because the impersonated user svcllmhubmohanar2, does not have the permission to run the python command to launch the job? Do I need to set something else to circumvent this?

I moved the repo to the prohects folder, so the new path is /projects/bfmz/svcdeltallmhub/Delta-deployment/mohanar2-test/LLMHub/backend/ but still getting the error which makes sense as I am unable to reach this directory as the impersonated user, which directory can we install a python virtual environment that can be run by any impersonated user. The directory /projects/llmhub/ also does not seem to grant the necessary permission for impersonated users, they can only access their own directories

Because the requested Python version (>=3.9) does not satisfy Python>=3.10 and vec-inf==0.9.0 depends on Python>=3.10, we can conclude that vec-inf==0.9.0 cannot be used.

joshfactorial · 2026-08-13T15:29:19Z

@Vismayak and @rohan-uiuc I'm wondering if we're doing the impersonation on the wrong level. What if svc-account runs the back and then we save impersonation just for the sbatch and cancel portions:

sudo -u <user> sbatch <script>

We could do something similar for scancel as well.

rohan-uiuc added 10 commits April 16, 2026 15:47

chore: remove slurm log dir setting

afb457e

fix: read vec-inf logs from environment config

2563db5

fix: update delta vec-inf log path

7197971

chore: add comment to change log_dir based on env/user

933adfc

feat: support user-impersonated job launches

e456cf5

chore: update Delta vec-inf defaults

0f11df7

test: cover user-impersonated launch flow

6b77bd4

chore: add impersonation setup checks

e1f2627

Merge branch 'main' of github.com:Center-for-AI-Innovation/llm-servin…

cdf2c3b

…g-backend into job-as-user

feat: port backend user impersonation PR #32

84e2fbc

rohan-uiuc marked this pull request as ready for review July 16, 2026 14:08

rohan-uiuc self-assigned this Jul 16, 2026

rohan-uiuc requested review from Vismayak and joshfactorial July 16, 2026 18:47

joshfactorial reviewed Jul 17, 2026

View reviewed changes

minump self-requested a review July 30, 2026 16:34

update python to 3.10

c90f9f1

Because the requested Python version (>=3.9) does not satisfy Python>=3.10 and vec-inf==0.9.0 depends on Python>=3.10, we can conclude that vec-inf==0.9.0 cannot be used.

Navigation Menu

Uh oh!

feat: support user-impersonated backend job launches - #40

feat: support user-impersonated backend job launches#40
rohan-uiuc wants to merge 11 commits into
mainfrom
port/backend-pr-32

rohan-uiuc commented Jul 16, 2026

Uh oh!

joshfactorial Jul 17, 2026

Uh oh!

joshfactorial Jul 17, 2026

Uh oh!

joshfactorial Jul 17, 2026

Uh oh!

joshfactorial Jul 17, 2026

Uh oh!

joshfactorial Jul 17, 2026

Uh oh!

joshfactorial commented Jul 17, 2026

Uh oh!

minump commented Aug 6, 2026

Uh oh!

Vismayak commented Aug 6, 2026 •

edited

Loading

Uh oh!

Vismayak commented Aug 6, 2026 •

edited

Loading

Uh oh!

joshfactorial commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		return str(Path(raw_root).expanduser() / username)


		def get_vec_inf_log_base_dir(infrastructure: Optional[str] = None) -> Optional[str]:

Uh oh!

Conversation

rohan-uiuc commented Jul 16, 2026

Summary

Impact

Validation

Source

Uh oh!

joshfactorial Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

joshfactorial Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

joshfactorial Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

joshfactorial Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

joshfactorial Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

joshfactorial commented Jul 17, 2026

Uh oh!

minump commented Aug 6, 2026

Uh oh!

Vismayak commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Vismayak commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joshfactorial commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Vismayak commented Aug 6, 2026 •

edited

Loading

Vismayak commented Aug 6, 2026 •

edited

Loading