Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
104 commits
Select commit Hold shift + click to select a range
b327363
fix: handle trailing slashes in registry path (#1588) (#1592)
btjd Feb 19, 2026
14260a2
chore: update CI Docker image to Go 1.25 (#1593)
akurinnoy Feb 19, 2026
e5db1b6
Revert "chore: update CI Docker image to Go 1.25 (#1593)"
akurinnoy Feb 20, 2026
3c338a3
chore: install Go 1.25.7 (#1596)
akurinnoy Feb 23, 2026
dd71281
chore: update Go to 1.25.7 (#1574)
akurinnoy Feb 23, 2026
ba32f47
docs: add v0.40.0 changelog entries (#1597)
akurinnoy Feb 24, 2026
9b46583
chore: add v0.40.0 bundle to main branch (#1599)
akurinnoy Feb 26, 2026
b88e1d1
Backup - Unify annotation with default value
Allda Mar 2, 2026
316dd05
chore : regenerate CRDs after backup cronjob schedule's default value…
rohanKanojia Mar 2, 2026
4410b61
Merge pull request #1602 from devfile/backup-default
Allda Mar 2, 2026
9415b15
chore: release: bump to 0.41.0-dev in main (#1598)
che-incubator-bot Mar 6, 2026
cce29e8
feat: Mounting PVC into Workspace within Subpath (#1595)
tolusha Mar 6, 2026
6ddba09
docs : update outdated restore documentation with restore functionali…
rohanKanojia Mar 19, 2026
2d6966e
Update backup job's default backoff limit to 1 (#1605)
dkwon17 Mar 19, 2026
441bcb2
ci: Improve che-happy-path test reliability with retry logic and heal…
akurinnoy Mar 27, 2026
83ef0d1
chore(deps): bump google.golang.org/grpc from 1.72.1 to 1.79.3 (#1604)
dependabot[bot] Mar 30, 2026
66b4c5a
chore(deps): bump go.opentelemetry.io/otel/sdk from 1.34.0 to 1.40.0 …
dependabot[bot] Mar 31, 2026
68ce90d
Change project-backup image from ubi10 to ubi9 (#1610)
dkwon17 Apr 8, 2026
81f58db
chore(deps): bump github.com/cloudflare/circl from 1.6.1 to 1.6.3 (#1…
dependabot[bot] Apr 9, 2026
dfac21c
chore(deps): bump github.com/go-git/go-git/v5 from 5.16.5 to 5.17.1 (…
dependabot[bot] Apr 9, 2026
de4c0bf
chore (deps) : bump go version to v1.25.8 (#1611)
rohanKanojia Apr 9, 2026
fcee0cd
chore(deps): bump go.opentelemetry.io/otel/sdk from 1.40.0 to 1.43.0 …
dependabot[bot] Apr 9, 2026
5ffc09f
Add project clone retries functionality (#1613)
dkwon17 Apr 13, 2026
943ae8e
fix: fall back to canonical backup auth secret name on restore (#1614)
akurinnoy Apr 15, 2026
62b391c
chore(deps): bump go version to v1.25.9 (#1617)
rohanKanojia Apr 27, 2026
e1097f6
feat: Mounting resources should not cause workspace restart (#1533)
tolusha Apr 28, 2026
fa371eb
chore (deps) : update controller-runtime to v0.23.3 (#1568)
rohanKanojia May 5, 2026
3aa293d
feat: mount ConfigMaps/Secrets/PVCs only to a specific DevWorkspace (…
tolusha May 8, 2026
09af429
chore(deps): bump github.com/go-git/go-git/v5 from 5.17.1 to 5.18.0 (…
dependabot[bot] May 8, 2026
13a8966
chore(deps): bump github.com/go-git/go-git/v5 from 5.18.0 to 5.19.0 (…
dependabot[bot] May 11, 2026
c3feb2e
refactor (backup) : conditionally copy registry auth secret from oper…
rohanKanojia May 12, 2026
1e949fc
chore: add v0.40.1 bundle to main branch (#1624)
dkwon17 May 12, 2026
a6b7a01
chore: add v0.41.0 bundle to main branch (#1632)
dkwon17 May 15, 2026
13291ae
Add changelog entries for 0.40.1, and 0.41.0 (#1625)
dkwon17 May 15, 2026
842e945
chore: release: bump to 0.42.0-dev in main (#1629)
che-incubator-bot May 19, 2026
113b402
Set controller UsePriorityQueue to false. Field defaults to true star…
dkwon17 May 22, 2026
a266d16
Backup registry auth secret must not be owned by any workspace (#1631)
akurinnoy May 26, 2026
17aefae
chore(deps): bump go version to v1.26.2 (#1639)
rohanKanojia May 28, 2026
0f63810
chore(deps): bump github.com/go-git/go-git/v5 from 5.19.0 to 5.19.1 (…
dependabot[bot] May 28, 2026
ec6de18
fix (storage) : use operator podSecurityContext for PVC cleanup job o…
rohanKanojia Jun 3, 2026
7b6ff69
fix(rbac): implement registry rolebinding finalization for OpenShift
danielpenad Jun 8, 2026
32c3d0a
fix(config): bump default persistent home init container memory limit…
btjd Jun 12, 2026
a4b9afd
fix: initialization HTTP clients with merged DevWorkspaceOperatorConfig
tolusha Jun 18, 2026
cdc8050
Add user home mount for ephemeral storage when init-persistent-home i…
dkwon17 Jun 18, 2026
3bbd19a
fix: create parent directories for nested clonePath (#1652)
snecklifter Jun 22, 2026
e31ead1
feat: add configurable field-level restrictions for container and pod…
tolusha Jun 23, 2026
af782ad
fix: preserve default proxy settings (#1656)
tolusha Jun 23, 2026
e2c9d9b
chore: release: bump to 0.43.0-dev in main (#1657)
che-incubator-bot Jun 25, 2026
a15632e
chore: add v0.42.0 bundle to main branch (#1663)
akurinnoy Jun 25, 2026
efa4b67
docs: establish ADR framework with template and agent guidelines (#1633)
akurinnoy Jun 25, 2026
4ea3124
Update go toolchain, x/crypto, x/net (#1659)
dkwon17 Jun 25, 2026
6807a49
Create DWO 0.42.0 changelog (#1666)
dkwon17 Jun 26, 2026
855fdc3
fix: handle non-passphrase SSH prompts in askpass script (#1658)
vinokurig Jul 2, 2026
8501cef
chore (deps): bump go version to v1.26.4
rohanKanojia Jun 29, 2026
686bc5f
docs: add documentation for cross-namespace DevWorkspaceTemplate impo…
manweg Jul 14, 2026
1cd06ee
fix(backup): use configured imagePullPolicy for backup CronJob (#1654)
snecklifter Jul 14, 2026
5413a98
fix(backup): apply configured podSecurityContext to backup CronJob (#…
snecklifter Jul 15, 2026
8481361
chore (deps) : update controller-runtime to v0.24.1 (#1642)
rohanKanojia Jul 20, 2026
675d5e6
chore(deps): bump golang.org/x/crypto, x/net, and x/mod to latest (#1…
rohanKanojia Jul 23, 2026
1f86ade
chore(deps): bump go version to v1.26.5 (#1671)
rohanKanojia Jul 23, 2026
7cc5b95
chore(deps): bump google.golang.org/grpc from 1.79.3 to 1.82.1 (#1676)
dependabot[bot] Jul 24, 2026
f91eefe
chore: release: bump to 0.44.0-dev in main (#1678)
btjd Jul 24, 2026
f11feff
docs: create DWO 0.43.0 changelog
btjd Jul 28, 2026
7d48ec9
Merge pull request #1682 from devfile/changelog-v0.43.0
btjd Jul 28, 2026
22b8657
chore(deps): bump github.com/google/cel-go from 0.26.0 to 0.29.0 (#1679)
dependabot[bot] Jul 28, 2026
a9cb4de
chore: add v0.43.0 bundle to main branch (#1683)
btjd Jul 29, 2026
41f60d5
fix: add node affinity to backup Jobs to prevent Multi-Attach errors …
rohanKanojia Jul 29, 2026
9e3dcc0
docs: add PR #1658 to v0.43.0 release notes (#1687)
btjd Jul 30, 2026
65e0bea
Try updating happy path test install namespace to openshift-operators…
dkwon17 Aug 6, 2026
20ec7ee
Add codeql workflow (#1685)
dkwon17 Aug 7, 2026
2a7ec67
fix: add NetworkPolicy RBAC permissions (#1690)
rohanKanojia Aug 11, 2026
fc00100
chore(deps): bump github.com/go-git/go-git/v5 from 5.19.1 to 5.19.2 (…
dependabot[bot] Aug 11, 2026
5684f19
Fix issue #1668 about auto renew certificate on mutating / validating…
disaster37 Aug 18, 2026
a73ca37
fix: sanitize automount volume names to be DNS-1123 compliant (#1699)
oskutka Aug 27, 2026
970cecc
feat: Improve Webhook validations (#1693)
tolusha Sep 8, 2026
f0052ad
Bump ossf/scorecard-action to 2.4.4 (#1701)
dkwon17 Sep 9, 2026
ca957f2
chore(deps): bump google.golang.org/grpc from 1.82.1 to 1.83.1 (#1700)
dependabot[bot] Sep 9, 2026
82828bf
fix: webhook should use non cached client for reading certificates CM…
tolusha Sep 14, 2026
d22b15e
chore(deps): bump google.golang.org/grpc from 1.83.1 to 1.83.2 (#1702)
dependabot[bot] Sep 16, 2026
45cd9c3
chore: bump Go version to 1.26.7 (#1705)
rohanKanojia Sep 16, 2026
44faaff
Install go 1.26.7 in the development dockerfile (#1712)
dkwon17 Sep 25, 2026
3526ad3
Update ci image node version to 22.12.0 (#1713)
dkwon17 Sep 29, 2026
f8dcf37
Tolerate additional labels and annotations under deployment's spec.te…
dkwon17 Sep 30, 2026
46ea381
chore(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace (…
dependabot[bot] Sep 30, 2026
69738f0
Retrieve TLS config from apiserver for OpenShift (#1695)
dkwon17 Sep 30, 2026
20dbffb
chore: networkpolicy provisioning (#1710)
tolusha Sep 30, 2026
7281879
chore(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/o…
dependabot[bot] Oct 1, 2026
c87517f
docs: add v1.0.0 changelog entries (#1715)
btjd Oct 5, 2026
71d6fff
chore: release: bump to 1.1.0-dev in main (#1716)
che-incubator-bot Oct 6, 2026
e2e5fd2
chore: add v1.0.0 bundle to main branch (#1717)
btjd Oct 6, 2026
f6f09c0
chore(deps): update controller-runtime to v0.25.2 (#1706)
rohanKanojia Oct 7, 2026
e585656
chore: bump github.com/devfile/api/v2 v2.3.1-alpha.0.20261007143616-3…
tolusha Oct 7, 2026
5296313
chore: pass operator namespace to GetDefaultNetworkPolicy (#1719)
tolusha Oct 8, 2026
9e9bab4
refactor: extract shared rewriteContainerVolumeMounts logic between s…
SAY-5 Oct 8, 2026
70be7ac
fix: default init container imagePullPolicy to IfNotPresent (#1692)
dkwon17 Oct 9, 2026
cd25c62
fix: preserve deny-all NetworkPolicy config and stop mutating default…
tolusha Oct 13, 2026
49eaaab
chore: add graphify knowledge graph and IDE integrations (#1714)
akurinnoy Oct 20, 2026
b45bb50
fix: allow OpenShift console ingress in default NetworkPolicy for Web…
tolusha Oct 23, 2026
e0979a9
feat: implement Gateway API routing class and solver
btjd May 12, 2026
35ceb27
feat(routing): complete Gateway API solver integration, fix spec drif…
btjd May 21, 2026
fabc9f9
feat: implement gateway-api routing class and fix controller readines…
btjd Jun 18, 2026
bfb8b96
fix: address PR review feedback for Gateway API routing
btjd Aug 4, 2026
cfd3b2f
fix: address PR review feedback for Gateway API routing (round 2)
btjd Sep 8, 2026
74b40fa
fix: skip HTTPRoute sync when Gateway API is not installed
btjd Oct 28, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
302 changes: 302 additions & 0 deletions .ci/README-CHE-HAPPY-PATH.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,302 @@
# Che Happy-Path Test

**Script**: `.ci/oci-devworkspace-happy-path.sh`
**Purpose**: Integration test validating DevWorkspace Operator with Eclipse Che deployment

## Overview

This script deploys and validates the full DevWorkspace Operator + Eclipse Che stack on OpenShift, ensuring the happy-path user workflow succeeds. It's used in the `v14-che-happy-path` Prow CI test.

## Features

### Retry Logic
- **Che deployment**: 2 attempts with exponential backoff (60s base + jitter)
- **Cleanup**: Waits for CheCluster CR deletion before retry
- **Happy-path test retry**: 1 retry with 30s delay if Selenium test fails

### Health Checks
- **OLM**: Verifies `catalog-operator` and `olm-operator` are available before Che deployment (2-minute timeout each)
- **DWO**: Waits for `deployment condition=available` (5-minute timeout)
- **Che**: chectl's built-in readiness checks ensure deployment is healthy

### Artifact Collection
On each failure, collects:
- OLM diagnostics (Subscription, InstallPlan, CSV, CatalogSource)
- CatalogSource pod logs
- Che operator logs (last 1000 lines)
- CheCluster CR status (full YAML)
- All pod logs from Che namespace
- Kubernetes events
- chectl server logs

### Error Handling
- Graceful error handling with stage-specific messages
- Progress indicators: "Attempt 1/2", "Retrying in 71s..."
- No crash on failures

## Configuration

Environment variables (all optional):

| Variable | Default | Description |
|----------|---------|-------------|
| `CHE_NAMESPACE` | `eclipse-che` | Namespace for Che deployment |
| `MAX_RETRIES` | `2` | Maximum retry attempts |
| `BASE_DELAY` | `60` | Base delay in seconds for exponential backoff |
| `MAX_JITTER` | `15` | Maximum jitter in seconds |
| `ARTIFACT_DIR` | `/tmp/dwo-e2e-artifacts` | Directory for diagnostic artifacts |
| `DEVWORKSPACE_OPERATOR` | (required) | DWO image to deploy |

## Usage

### In Prow CI

The script is called automatically by the `v14-che-happy-path` Prow job. Prow sets `DEVWORKSPACE_OPERATOR` based on the context:

**For PR checks** (testing PR code):
```bash
export DEVWORKSPACE_OPERATOR="quay.io/devfile/devworkspace-controller:pr-${PR_NUMBER}-${COMMIT_SHA}"
./.ci/oci-devworkspace-happy-path.sh
```

**For periodic/nightly runs** (testing main branch):
```bash
export DEVWORKSPACE_OPERATOR="quay.io/devfile/devworkspace-controller:next"
./.ci/oci-devworkspace-happy-path.sh
```

### Local Testing
```bash
export DEVWORKSPACE_OPERATOR="quay.io/youruser/devworkspace-controller:your-tag"
export ARTIFACT_DIR="/tmp/my-test-artifacts"
./.ci/oci-devworkspace-happy-path.sh
```

## Test Flow

1. **Deploy DWO**
- Runs `make install`
- Waits for controller deployment to be available
- Collects artifacts if deployment fails

2. **Deploy Che** (with retry)
- Runs `chectl server:deploy` with extended timeouts (24h)
- chectl handles readiness checks internally
- Collects artifacts on failure
- Cleans up and retries if needed

3. **Run Happy-Path Test**
- Downloads test script from Eclipse Che repository
- Executes Che happy-path workflow
- Retries once after 30s if test fails
- Collects artifacts on failure

## Exit Codes

- `0`: Success - All stages completed
- `1`: Failure - Check `$ARTIFACT_DIR` for diagnostics

## Timeouts

| Component | Timeout | Purpose |
|-----------|---------|---------|
| DWO deployment | 5 minutes | Pod becomes available |
| chectl pod wait/ready | 24 hours | Generous for slow environments |

## Common Failures

### OLM Infrastructure Not Ready
**Symptoms**: "ERROR: OLM infrastructure is not healthy, cannot proceed with Che deployment"
**Check**: `$ARTIFACT_DIR/olm-diagnostics-olm-check.yaml`
**Common causes**:
- OLM operators not running (`catalog-operator`, `olm-operator`)
- Cluster provisioning issues during bootstrap
- Resource constraints preventing OLM operator scheduling
**Resolution**: This indicates a fundamental cluster infrastructure issue. Check cluster health and OLM operator logs before retrying.

### DWO Deployment Fails
**Symptoms**: "ERROR: DWO controller is not ready"
**Check**: `$ARTIFACT_DIR/devworkspace-controller-info/`
**Common causes**: Image pull errors, resource constraints, webhook conflicts

### Che Deployment Timeout
**Symptoms**: "ERROR: chectl server:deploy failed" with timeout-related messages
**Check**: `$ARTIFACT_DIR/che-operator-logs-attempt-*.log`, `$ARTIFACT_DIR/olm-diagnostics-attempt-*.yaml`, `$ARTIFACT_DIR/chectl-logs-attempt-*/`
**Common causes**:
- OLM subscription timeout (check `olm-diagnostics` for subscription state)
- Database connection issues
- Image pull failures
- Operator reconciliation errors
- chectl timeout waiting for pods/resources to become ready

### Pod CrashLoopBackOff
**Symptoms**: "ERROR: chectl server:deploy failed"
**Check**: `$ARTIFACT_DIR/eclipse-che-info/` for pod logs
**Common causes**: Configuration errors, resource limits, TLS certificate issues

### OLM Subscription Stuck
**Symptoms**: Subscription timeout after 120 seconds with no resources created
**Check**: `$ARTIFACT_DIR/olm-diagnostics-attempt-*.yaml`, `$ARTIFACT_DIR/catalogsource-logs-attempt-*.log`
**Common causes**:
- CatalogSource pod not pulling/running
- InstallPlan not created (subscription cannot resolve dependencies)
- Cluster resource exhaustion preventing operator pod scheduling
**Resolution**: Check OLM operator logs and CatalogSource pod status. See "Advanced Troubleshooting" section for monitoring and alternative deployment options.

## Artifact Locations

After a failed test run:
```
$ARTIFACT_DIR/
├── attempt-log.txt
├── failure-report.json
├── failure-report.md
├── devworkspace-controller-info/
│ ├── <pod-name>-<container>.log
│ └── events.log
├── eclipse-che-info/
│ ├── <pod-name>-<container>.log
│ └── events.log
├── che-operator-logs-attempt-1.log
├── che-operator-logs-attempt-2.log
├── checluster-status-attempt-1.yaml
├── checluster-status-attempt-2.yaml
├── olm-diagnostics-attempt-1.yaml
├── olm-diagnostics-attempt-2.yaml
├── catalogsource-logs-attempt-1.log
├── catalogsource-logs-attempt-2.log
├── chectl-logs-attempt-1/
└── chectl-logs-attempt-2/
```

## Dependencies

- `kubectl` - Kubernetes CLI
- `oc` - OpenShift CLI (for log collection)
- `chectl` - Eclipse Che CLI (v7.114.0+)
- `jq` - JSON processor (for chectl)

## Advanced Troubleshooting

### OLM Infrastructure Issues

If you experience persistent OLM subscription timeouts (see `olm-diagnostics-*.yaml` artifacts):

#### Option 1: OLM Health Check (Implemented)
The script now verifies OLM infrastructure health before deploying Che:
- Checks `catalog-operator` is available
- Checks `olm-operator` is available
- Verifies `openshift-marketplace` is accessible

If OLM is unhealthy, the test fails fast with diagnostic artifacts instead of waiting through timeouts.

#### Option 2: Monitor Subscription Progress (Advanced)
For debugging stuck subscriptions, you can add active monitoring to detect zero-progress scenarios earlier:

```bash
# Example: Monitor subscription state every 10 seconds
while [ $elapsed -lt 300 ]; do
state=$(kubectl get subscription eclipse-che -n eclipse-che \
-o jsonpath='{.status.state}' 2>/dev/null)
echo "[$elapsed/300s] Subscription state: ${state:-unknown}"
if [ "$state" = "AtLatestKnown" ]; then
break
fi
sleep 10
elapsed=$((elapsed + 10))
done
```

This helps identify whether subscriptions are progressing slowly vs. completely stuck.

#### Option 3: Skip OLM Installation (Alternative Approach)
For CI environments with persistent OLM issues, consider deploying Che operator directly instead of via OLM:

```bash
chectl server:deploy \
--installer=operator \ # Uses direct YAML deployment
-p openshift \
--batch \
--telemetry=off \
--skip-devworkspace-operator \
--chenamespace="$CHE_NAMESPACE"
```

**Trade-offs**:
- ✅ Bypasses OLM infrastructure entirely
- ✅ More reliable in resource-constrained CI environments
- ❌ Doesn't test OLM integration path (used by production OperatorHub)
- ❌ May miss OLM-specific issues

**When to use**: Temporary workaround for CI infrastructure issues while OLM problems are being resolved.

### Subscription Timeout Issues

If OLM subscriptions consistently timeout (visible in `olm-diagnostics-*.yaml`):

1. **Check OLM operator logs**:
```bash
kubectl logs -n openshift-operator-lifecycle-manager \
deployment/catalog-operator --tail=100
kubectl logs -n openshift-operator-lifecycle-manager \
deployment/olm-operator --tail=100
```

2. **Verify CatalogSource pod is running**:
```bash
kubectl get pods -n openshift-marketplace \
-l olm.catalogSource=eclipse-che
kubectl logs -n openshift-marketplace \
-l olm.catalogSource=eclipse-che
```

3. **Check InstallPlan creation**:
```bash
kubectl get installplan -n eclipse-che -o yaml
```
- If no InstallPlan exists, OLM couldn't resolve the subscription
- If InstallPlan exists but isn't complete, check its status conditions

## CI Failure Reports

The script automatically generates failure reports and posts them as PR comments after each run (both failures and successes with retries). **Do not delete these comments** — they are used to track flakiness patterns across PRs.

### What gets reported

Each report includes a table of all attempts with:
- **Attempt**: Which attempt number (e.g., `1/2`, `2/2`)
- **Stage**: Which function failed (`deployChe`, `runHappyPathTest`, etc.)
- **Result**: `PASSED` or `FAILED`
- **Reason**: Classified failure reason (e.g., "Che operator reconciliation failure")

### Failure categories

| Category | Meaning | Retryable? |
|----------|---------|------------|
| `INFRA` | Infrastructure issue (OLM, image pull, operator reconciliation) | Yes — `/retest` |
| `TEST` | Test execution issue (Dashboard UI timeout, workspace start) | Maybe |
| `MIXED` | Both infrastructure and test issues across attempts | Yes — `/retest` |
| `UNKNOWN` | Could not classify — check artifacts | Investigate |

### Report artifacts

Reports are always saved to `$ARTIFACT_DIR/` regardless of whether PR commenting succeeds:
- `failure-report.json` — structured data for programmatic analysis
- `failure-report.md` — human-readable markdown (same as the PR comment)
- `attempt-log.txt` — raw attempt tracking log

### Why these comments matter

Over time, these reports reveal:
- Which failure categories are most common
- Whether flakiness is improving or worsening
- Which infrastructure components are least reliable
- Whether retry logic is effective (passed-on-retry patterns)

## Related Documentation

- [Eclipse Che Documentation](https://eclipse.dev/che/docs/)
- [chectl GitHub Repository](https://github.com/che-incubator/chectl)
- [OLM Troubleshooting Guide](https://olm.operatorframework.io/docs/troubleshooting/)
- [DevWorkspace Operator README](../README.md)
- [Contributing Guidelines](../CONTRIBUTING.md)
Loading
Loading