python_workflow_submitter

Python alternative to creating and running argo workflows in the Data Analysis Platform

This allows for the submission of yaml files to Diamond's GraphQL API. This automatically lints yaml files before submission.

Notably, this requires a .env file in your src/ folder.

What	Where
Source	https://github.com/DiamondLightSource/python-workflow-submitter
Docker	docker run ghcr.io/DiamondLightSource/python-workflow-submitter:latest
Releases	https://github.com/DiamondLightSource/python-workflow-submitter/releases

To create a helm template in your helm chart folder:

import asyncio
from python_workflow_submitter.create_helm_yaml import create_helm_yaml

create_helm_yaml("notebook.yaml", "/workspaces/your_path/helm", "values/values.yaml")

To lint a yaml file (automatically called upon submitting workflows):

import asyncio
from python_workflow_submitter.lintyaml import lint_yaml

lint_yaml("notebook.yaml")

To submit your workflow written in a notebook:

from python_workflow_submitter.submit_workflow import submit_workflow_yaml

await submit_workflow_yaml("example.yaml")

To submit a workflow script:

import asyncio
from python_workflow_submitter.submit_workflow import submit_workflow_yaml

asyncio.run(submit_workflow_yaml("example.yaml", visit="ks10000-3"))

To submit a generic workflow via a GraphQL mutation:

import asyncio
from python_workflow_submitter.submit_workflow import submit_workflow

asyncio.run(
    submit_workflow(
        "example-template",
        {"png": True, "jpg": False, "jpeg": True, "tif": True, "tiff": False},
        visit=str(os.environ.get("VISIT")),
    )
)

To list workflow templates available to run:

import asyncio
from python_workflow_submitter.list_workflows import list_workflows

asyncio.run(list_workflows(limit=5, filter="EXAMPLES"))

To list workflows in specific visit with optional filtering:

import asyncio
from python_workflow_submitter.list_workflows import list_workflows_in_visit

asyncio.run(list_workflows_in_visit(limit=5,filter=filter: {
    "creator": "gmg29649",
    "template": "example-template",
  	"workflowStatusFilter": {"succeeded": True}
  }))

To list information about a workflow in a visit with a specific name:

import asyncio
from python_workflow_submitter.list_workflows import info_about_workflow

asyncio.run(info_about_workflow(name="conditional-steps-tswxm", visit="ks10000-3"))