SK-2954: Split skyflow-python into common/v2/v3 build variants, add flowdb (v3) insert support

Restructures the repo into three build variants sharing a bundled common/
module (SK-2938 Option C): v2 (today's SDK, behavior-preserving) and a new
v3 built on the flowservice/flowdb API, insert-only this round.

common/
- Shared credential resolution, vault-URL resolution, and bearer-token
  fetch/cache/expiry logic (VaultController, BaseVaultClient), enums,
  errors, service_account, and generic validators.
- VaultController declares insert/get/update/delete/query/detokenize as
  abstract methods (Java-interface-style); v2 and v3 each provide their
  own concrete/stub implementations.

v2
- Relocated from the repo root via git mv; public API unchanged (same
  class names, signatures, import paths). Vault is now a backward-compatible
  alias for the internal PdbVaultController class.
- Fixed a latent bug where v2's own Env enum failed cross-class comparisons
  against common's Env; both now share one definition.

v3 (skyflow-flowvault, starting at 1.0.0)
- New InsertRequest/InsertRecord/Upsert/InsertResponse types, FlowVaultController,
  and VaultClient targeting the flowservice REST API.
- Insert validation ported from Java's v3 Validations.java: table/upsert
  must live in exactly one place (request-level or per-record, matching
  in both), 10k record cap, empty key/value checks.
- InsertResponse mirrors Java's v3 shape (summary/success/errors) as plain
  dicts, each result tagged with its index in the original record list
  (stable across batch boundaries).
- Structured per-record error parsing from the backend's actual error
  body, plus x-request-id propagation onto error entries.
- Batching via INSERT_BATCH_SIZE (default 50, max 1000), sequential,
  isolate-and-continue on a failing batch.
- Vault URL resolution uses v3's own skyvault.skyflowapis.* domain
  (confirmed against a live vault to differ from v2's vault.skyflowapis.*).

CI/CD
- shared-tests.yml and shared-build-and-deploy.yml now take a `variant`
  input and scope every step to v2/ or v3/ via working-directory.
- main.yml, ci.yml, beta-release.yml, internal-release.yml, and
  release.yml matrix over both variants. v3 releases are distinguished
  from v2's via a flowvault- tag/branch prefix (flowvault-1.0.0,
  flowvault-release/*) so a release trigger is never ambiguous between
  the two independently-versioned packages; v2's existing bare-semver
  tags are untouched.
- Fixed ruff.toml/.codespellrc still excluding a pre-split "skyflow/generated"
  path that no longer existed after the relocation.
- Fixed a bump_version.sh sed collision with a comment that happened to
  contain the literal text "__version__ = ...".
- Added a common/ test job to main.yml/ci.yml.

Tests: common 36, v3 62, v2 426 (2 pre-existing unrelated fixture
failures), tests/contract passing for both variants.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Author: saileshwar-skyflow

.codespellrc

@@ -3,7 +3,7 @@
 ignore-words-list = Skyflow,skyflow,skyflowapi,skyflowapis,deidentify,reidentify,detokenize,upsert,upserting,binlookup,byot,creds,fpe,devsecops,formdata,vaultid,dotenv,usecwd,runid,dateutil,Homogenous
 
 # Skip these files and folders
-skip = .git,.venv,venv,env,__pycache__,*.pyc,*.egg-info,dist,build,.idea,.vscode,*.log,requirements.txt,./skyflow/generated,setup.py
+skip = .git,.venv,venv,env,__pycache__,*.pyc,*.egg-info,dist,build,.idea,.vscode,*.log,requirements.txt,generated,setup.py
 
 # If you want to verify it is working, you can uncomment this line to see what files it checks
 # count = 

.github/workflows/beta-release.yml

@@ -4,15 +4,25 @@ on:
   push:
     tags: '*.*.*b*'
     paths-ignore:
-      - "setup.py"
+      - "*/setup.py"
       - "*.yml"
       - "*.md"
-      - "skyflow/utils/_version.py"
+      - "*/skyflow/utils/_version.py"
 
 jobs:
   build-and-deploy:
+    strategy:
+      matrix:
+        include:
+          - variant: v2
+            tag-prefix: ''
+          - variant: v3
+            tag-prefix: 'flowvault-'
+    if: (matrix.variant == 'v3' && startsWith(github.ref_name, 'flowvault-')) || (matrix.variant == 'v2' && !startsWith(github.ref_name, 'flowvault-'))
     uses: ./.github/workflows/shared-build-and-deploy.yml
     with:
       ref: ${{ github.ref_name }}
       tag: 'beta'
+      variant: ${{ matrix.variant }}
+      tag-prefix: ${{ matrix.tag-prefix }}
     secrets: inherit

.github/workflows/ci.yml

@@ -18,7 +18,27 @@ jobs:
           error: 'One of your your commit messages is not matching the format with JIRA ID Ex: ( SDK-123 commit message )'
 
   test:
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - variant: v2
+            coverage-omit: "skyflow/generated/*,skyflow/utils/validations/*,skyflow/vault/data/*,skyflow/vault/detect/*,skyflow/vault/tokens/*,skyflow/vault/connection/*,skyflow/error/*,skyflow/utils/enums/*,skyflow/vault/controller/_audit.py,skyflow/vault/controller/_bin_look_up.py"
+          - variant: v3
+            coverage-omit: "skyflow/generated/*"
     uses: ./.github/workflows/shared-tests.yml
     with:
       python-version: '3.9'
+      variant: ${{ matrix.variant }}
+      coverage-omit: ${{ matrix.coverage-omit }}
     secrets: inherit
+
+  test-common:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+        with:
+          python-version: '3.9'
+      - run: pip install -e ./common
+      - run: python -m unittest discover -s common/tests -t .

.github/workflows/internal-release.yml

@@ -5,19 +5,30 @@ on:
     tags-ignore:
       - '*.*'
     paths-ignore:
-      - "setup.py"
+      - "*/setup.py"
       - "*.yml"
       - "*.md"
-      - "skyflow/utils/_version.py"
+      - "*/skyflow/utils/_version.py"
       - "samples/**"
+      - "v3/samples/**"
     branches:
       - release/*
+      - flowvault-release/*
 
 jobs:
   build-and-deploy:
+    strategy:
+      matrix:
+        include:
+          - variant: v2
+            tag-prefix: ''
+          - variant: v3
+            tag-prefix: 'flowvault-'
+    if: (matrix.variant == 'v3' && startsWith(github.ref_name, 'flowvault-')) || (matrix.variant == 'v2' && !startsWith(github.ref_name, 'flowvault-'))
     uses: ./.github/workflows/shared-build-and-deploy.yml
     with:
       ref: ${{ github.ref_name }}
       tag: 'internal'
+      variant: ${{ matrix.variant }}
+      tag-prefix: ${{ matrix.tag-prefix }}
     secrets: inherit
-      

.github/workflows/main.yml

@@ -7,7 +7,27 @@ on:
 
 jobs:
   test:
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - variant: v2
+            coverage-omit: "skyflow/generated/*,skyflow/utils/validations/*,skyflow/vault/data/*,skyflow/vault/detect/*,skyflow/vault/tokens/*,skyflow/vault/connection/*,skyflow/error/*,skyflow/utils/enums/*,skyflow/vault/controller/_audit.py,skyflow/vault/controller/_bin_look_up.py"
+          - variant: v3
+            coverage-omit: "skyflow/generated/*"
     uses: ./.github/workflows/shared-tests.yml
     with:
       python-version: '3.9'
+      variant: ${{ matrix.variant }}
+      coverage-omit: ${{ matrix.coverage-omit }}
     secrets: inherit
+
+  test-common:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+        with:
+          python-version: '3.9'
+      - run: pip install -e ./common
+      - run: python -m unittest discover -s common/tests -t .

.github/workflows/release.yml

@@ -4,15 +4,25 @@ on:
   push:
     tags: "*.*.*"
     paths-ignore:
-      - "setup.py"
+      - "*/setup.py"
       - "*.yml"
       - "*.md"
-      - "skyflow/utils/_version.py"
+      - "*/skyflow/utils/_version.py"
 
 jobs:
   build-and-deploy:
+    strategy:
+      matrix:
+        include:
+          - variant: v2
+            tag-prefix: ''
+          - variant: v3
+            tag-prefix: 'flowvault-'
+    if: (matrix.variant == 'v3' && startsWith(github.ref_name, 'flowvault-')) || (matrix.variant == 'v2' && !startsWith(github.ref_name, 'flowvault-'))
     uses: ./.github/workflows/shared-build-and-deploy.yml
     with:
       ref: main
       tag: 'public'
+      variant: ${{ matrix.variant }}
+      tag-prefix: ${{ matrix.tag-prefix }}
     secrets: inherit

.github/workflows/shared-build-and-deploy.yml

@@ -7,12 +7,23 @@ on:
         description: 'Git reference to use (e.g., main or branch name)'
         required: true
         type: string
-      
+
       tag:
         description: 'Release Tag'
         required: true
         type: string
 
+      variant:
+        description: 'Build variant directory to release (v2 or v3)'
+        required: true
+        type: string
+
+      tag-prefix:
+        description: 'Prefix distinguishing this variant''s git tags from other variants'' (e.g. "flowvault-" for v3, empty for v2)'
+        required: false
+        type: string
+        default: ''
+
 jobs:
   build-and-deploy:
     runs-on: ubuntu-latest
@@ -29,7 +40,7 @@ jobs:
 
       - name: Resolve Branch for the Tagged Commit
         id: resolve-branch
-        if: ${{ inputs.tag == 'beta' || inputs.tag == 'public' }} 
+        if: ${{ inputs.tag == 'beta' || inputs.tag == 'public' }}
         run: |
           TAG_COMMIT=$(git rev-list -n 1 ${{ github.ref_name }})
 
@@ -47,18 +58,28 @@ jobs:
         id: previoustag
         uses: WyriHaximus/github-action-get-previous-tag@v1
         with:
-          fallback: 1.0.0
+          fallback: ${{ inputs.tag-prefix }}1.0.0
+          pattern: ${{ inputs.tag-prefix }}[0-9]*.[0-9]*.[0-9]*
+
+      - name: Resolve version number
+        id: version
+        run: |
+          TAG="${{ steps.previoustag.outputs.tag }}"
+          VERSION="${TAG#${{ inputs.tag-prefix }}}"
+          echo "version=$VERSION" >> $GITHUB_OUTPUT
 
       - name: Bump Version
+        working-directory: ${{ inputs.variant }}
         run: |
-          chmod +x ./ci-scripts/bump_version.sh
+          chmod +x ../ci-scripts/bump_version.sh
           if ${{ inputs.tag == 'internal' }}; then
-            ./ci-scripts/bump_version.sh "${{ steps.previoustag.outputs.tag }}" "$(git rev-parse --short "$GITHUB_SHA")"
+            ../ci-scripts/bump_version.sh "${{ steps.version.outputs.version }}" "$(git rev-parse --short "$GITHUB_SHA")"
           else
-            ./ci-scripts/bump_version.sh "${{ steps.previoustag.outputs.tag }}"
+            ../ci-scripts/bump_version.sh "${{ steps.version.outputs.version }}"
           fi
 
       - name: Commit changes
+        working-directory: ${{ inputs.variant }}
         run: |
           git config user.name "${{ github.actor }}"
           git config user.email "${{ github.actor }}@users.noreply.github.com"
@@ -71,24 +92,26 @@ jobs:
           git add skyflow/utils/_version.py
 
           if [[ "${{ inputs.tag }}" == "internal" ]]; then
-            VERSION="${{ steps.previoustag.outputs.tag }}.dev0+$(git rev-parse --short $GITHUB_SHA)"
-            COMMIT_MESSAGE="[AUTOMATED] Private Release $VERSION"
+            VERSION="${{ steps.version.outputs.version }}.dev0+$(git rev-parse --short $GITHUB_SHA)"
+            COMMIT_MESSAGE="[AUTOMATED] Private Release (${{ inputs.variant }}) $VERSION"
             git commit -m "$COMMIT_MESSAGE"
             git push origin ${{ github.ref_name }} -f
           fi
           if [[ "${{ inputs.tag }}" == "beta" || "${{ inputs.tag }}" == "public" ]]; then
-            COMMIT_MESSAGE="[AUTOMATED] Public Release - ${{ steps.previoustag.outputs.tag }}"
+            COMMIT_MESSAGE="[AUTOMATED] Public Release (${{ inputs.variant }}) - ${{ steps.previoustag.outputs.tag }}"
             git commit -m "$COMMIT_MESSAGE"
             git push origin ${{ env.branch_name }}
           fi
 
-      - name: Build and install skyflow package
+      - name: Build and install package
+        working-directory: ${{ inputs.variant }}
         run: |
           python setup.py sdist bdist_wheel
-          pip install dist/skyflow-*.whl
+          pip install dist/*.whl
 
       - name: Build and Publish Package
-        if: ${{ inputs.tag == 'beta' || inputs.tag == 'public' }} 
+        if: ${{ inputs.tag == 'beta' || inputs.tag == 'public' }}
+        working-directory: ${{ inputs.variant }}
         env:
           TWINE_USERNAME: __token__
           TWINE_PASSWORD: ${{ secrets.PYPI_PUBLISH_TOKEN }}
@@ -98,9 +121,10 @@ jobs:
 
       - name: Build and Publish to JFrog Artifactory
         if: ${{ inputs.tag == 'internal' }}
+        working-directory: ${{ inputs.variant }}
         env:
           TWINE_USERNAME: ${{ secrets.JFROG_USERNAME }}
           TWINE_PASSWORD: ${{ secrets.JFROG_PASSWORD }}
         run: |
           python setup.py sdist bdist_wheel
-          twine upload --repository-url https://prekarilabs.jfrog.io/artifactory/api/pypi/skyflow-python/ dist/*
+          twine upload --repository-url https://prekarilabs.jfrog.io/artifactory/api/pypi/skyflow-python/ dist/*

.github/workflows/shared-tests.yml

@@ -7,6 +7,15 @@ on:
         description: 'Python version to use'
         required: true
         type: string
+      variant:
+        description: 'Build variant directory to test (v2 or v3)'
+        required: true
+        type: string
+      coverage-omit:
+        description: 'Comma-separated coverage --omit patterns, relative to the variant directory'
+        required: false
+        type: string
+        default: 'skyflow/generated/*'
 
 jobs:
   run-tests:
@@ -23,32 +32,36 @@ jobs:
         with:
           name: "credentials.json"
           json: ${{ secrets.VALID_SKYFLOW_CREDS_TEST }}
+          dir: ${{ inputs.variant }}
 
-      - name: Build and install skyflow package
+      - name: Build and install package
+        working-directory: ${{ inputs.variant }}
         run: |
           pip install --upgrade pip setuptools wheel
           python setup.py sdist bdist_wheel
-          pip install dist/skyflow-*.whl
+          pip install dist/*.whl
           pip install ".[dev]"
 
       - name: Run Spell Check
         run: codespell
 
       - name: Run Linter Ruff
         run: ruff check . --output-format=github
-      
+
       - name: 'Run Tests'
+        working-directory: ${{ inputs.variant }}
         run: |
           pip install -r requirements.txt
-          python -m coverage run --source=skyflow --omit=skyflow/generated/*,skyflow/utils/validations/*,skyflow/vault/data/*,skyflow/vault/detect/*,skyflow/vault/tokens/*,skyflow/vault/connection/*,skyflow/error/*,skyflow/utils/enums/*,skyflow/vault/controller/_audit.py,skyflow/vault/controller/_bin_look_up.py -m unittest discover
+          python -m coverage run --source=skyflow --omit=${{ inputs.coverage-omit }} -m unittest discover
 
       - name: coverage
+        working-directory: ${{ inputs.variant }}
         run: coverage xml -o test-coverage.xml
 
       - name: Codecov
         uses: codecov/codecov-action@v2.1.0
         with:
           token: ${{ secrets.CODECOV_REPO_UPLOAD_TOKEN }}
-          files: test-coverage.xml
-          name: codecov-skyflow-python
+          files: ${{ inputs.variant }}/test-coverage.xml
+          name: codecov-skyflow-python-${{ inputs.variant }}
           verbose: true

.gitignore

@@ -104,6 +104,7 @@ celerybeat.pid
 # Environments
 .env
 .venv
+.venv-*
 env/
 venv/
 ENV/

common/__init__.py

@@ -0,0 +1,3 @@
+# common.utils and common.errors mutually depend on each other -- forcing utils to load first
+# here avoids the circular import (mirrors skyflow/__init__.py's own first line).
+from . import utils  # noqa: F401