Skip to content

ai: approval prediction — scheduled per-org retraining job #652

Description

@babltiga

Part of #645 (approval-outcome prediction epic — read it first for the big picture).

Goal

The clustered-safe scheduled job that retrains every organization's model daily and stores it with quality metrics. A model only gets serving=true when it clears the sample-count and AUC gates.

Depends on: trainer/evaluator, feature extractor, core.api history lookup, config properties.

Steps

  1. ai/internal/scheduled/ApprovalPredictionTrainingJob.java — follow the CLAUDE.md scheduled-job rules to the letter (copy BehaviorAnomalyDetectionJob as the template):
    • @Scheduled(fixedDelayString = "${accessflow.ai.approval-prediction.retrain-poll-interval:P1D}")
    • @SchedulerLock(name = "approvalPredictionTrainingJob", lockAtMostFor = "PT30M", lockAtLeastFor = "PT30S")
    • Thin: iterate orgs, delegate to ApprovalPredictionService.trainAll() / per-org train method, and swallow per-org RuntimeException with log.error so one bad org doesn't abort the batch.
  2. Per-org training logic (in DefaultApprovalPredictionService):
    • Fetch decided samples via ApprovalOutcomeHistoryLookupService (lookback = training-lookback, row cap ~20k).
    • Guards: total < min-training-samples (50) or either class < 10 → store/refresh the model row with serving=false and the sample counts (so admins can see why), skip training.
    • Deterministic holdout split: abs(queryId.hashCode()) % 10 < holdoutFraction*10 → holdout; rest → train. No RNG.
    • Train (LogisticRegressionTrainer), evaluate on holdout (ModelEvaluator), persist coefficients + training_samples/positive_samples/auc/accuracy, and set serving = (auc >= min-auc-to-serve).
  3. Docs: add the job to docs/05-backend.md → "Scheduled jobs and clustering" (job name, lock name, cadence property, default) — same row format as the existing jobs.

Acceptance criteria

  • ApprovalPredictionTrainingJobTest: delegates per org; one org throwing doesn't stop the others (verify subsequent org still processed).
  • Training-logic tests in DefaultApprovalPredictionServiceTest: cold-start guard (< 50), class-imbalance guard (< 10 negatives), AUC-below-threshold stores serving=false, happy path stores serving=true with metrics; holdout split determinism.
  • mvn test -Dtest=ApplicationModulesTest passes.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions