feat(cmdb-import): read the archive sheet and let module and usage be archived

CHANGELOG.md

@@ -7,6 +7,15 @@
   sheets becomes or updates a module, its vendor, a usage for the municipality
   and a contact person for its owner; a repeat import matches on APPID, so it
   updates instead of duplicating. See `docs/features/cmdb-import.md`.
+- CMDB import, repeat imports: applications that left the CMDB sheets are
+  archived when the sheet "Gearchiveerde Applicaties" lists them and moved to
+  OpenRegister's trash when no sheet does; one that returns is unarchived or
+  restored, never duplicated. The new option `missingRecords` (`archive`, the
+  default, or `keep`) is a choice in the CMDB import section, and the report
+  counts archived, unarchived, deleted and restored applications. The module
+  (0.3.9) and usage (1.5.7) schemas opt into OpenRegister's archive state, and
+  the Applications and Applications in use pages get an "Archived" quick
+  filter. (cmdb-import-archive-reconciliation)
 
 ### Changed
 - EOL feed: the register and schema slugs the feature reads from are corrected.

docs/features/README.md

@@ -158,7 +158,7 @@ The ArchiMate integration maps GEMMA Softwarecatalogus objects to ArchiMate appl
 
 ## CMDB Import
 
-Import a TOPdesk CMDB export (`.xlsx`) for one municipality from the **CMDB import** section of the admin settings. Every application row of the two CMDB sheets ("Onbeh Applicaties CMDB" and "Beheerde Applicaties CMDB") becomes or updates a module, its vendor as a Supplier organisation, a usage that links it to the municipality and records whether maintenance is arranged, and a contact person for its owner (identity in Nextcloud Contacts, never public). A repeat import matches on APPID per municipality, so it updates instead of duplicating, and leaves applications missing from the newer export as they are. The column mapping is declarative JSON executed by OpenRegister's mapping engine.
+Import a TOPdesk CMDB export (`.xlsx`) for one municipality from the **CMDB import** section of the admin settings. Every application row of the two CMDB sheets ("Onbeh Applicaties CMDB" and "Beheerde Applicaties CMDB") becomes or updates a module, its vendor as a Supplier organisation, a usage that links it to the municipality and records whether maintenance is arranged, and a contact person for its owner (identity in Nextcloud Contacts, never public). A repeat import matches on APPID per municipality, so it updates instead of duplicating. By default it archives the applications that moved to the sheet "Gearchiveerde Applicaties" and moves those on no sheet any more to OpenRegister's trash, and brings either back when it returns; the admin can choose to keep them instead. The Applications and Applications in use pages list the archive under the quick filter "Archived". The column mapping is declarative JSON executed by OpenRegister's mapping engine.
 
 See [CMDB import](cmdb-import.md) for the steps, the expected file structure, the error codes and how to adjust the mapping.
 

lib/Controller/CmdbImportController.php

@@ -86,14 +86,16 @@ public function __construct(
 	 * Import a TOPdesk CMDB export for one municipality.
 	 *
 	 * Multipart fields: `cmdbFile`, `municipalityUuid` or `municipalityName`,
-	 * `updateExisting` (default true), `publish` (default true), `missingRecords` (only `keep`) and
+	 * `updateExisting` (default true), `publish` (default true), `missingRecords` (`keep` or `archive`,
+	 * default `archive`) and
 	 * `operationId` (pattern `cmdb-` plus 8 to 64 letters, digits or hyphens).
 	 *
 	 * @return JSONResponse The report (200), or an error envelope with the contract code.
 	 *
 	 * @auth admin-only the import writes with RBAC and multitenancy off, across tenants, so only a full Nextcloud admin may run it.
 	 *
 	 * @spec openspec/changes/cmdb-export-import/tasks.md#task-8
+	 * @spec openspec/changes/cmdb-import-archive-reconciliation/specs/cmdb-export-import/spec.md#requirement-records-missing-from-a-newer-export-shall-be-archived-by-default-or-kept-on-request-req-cmdb-012
 	 */
 	public function import(): JSONResponse {
 		try {
@@ -247,15 +249,21 @@ protected function iniBytes(string $name): ?int {
 	 * @return array{path: string, options: array<string, mixed>}|JSONResponse The import input, or the first error.
 	 *
 	 * @spec openspec/changes/cmdb-export-import/tasks.md#task-8
+	 * @spec openspec/changes/cmdb-import-archive-reconciliation/specs/cmdb-export-import/spec.md#requirement-records-missing-from-a-newer-export-shall-be-archived-by-default-or-kept-on-request-req-cmdb-012
 	 */
 	private function readOptions(string $path, string $fileName): array|JSONResponse {
-		$missingRecords = $this->stringParam(name: 'missingRecords', default: 'keep');
+		$missingRecords = $this->stringParam(name: 'missingRecords', default: CmdbExportImportService::DEFAULT_MISSING_RECORDS);
 		if ($missingRecords === null) {
 			return $this->invalidField(field: 'missingRecords');
 		}
 
-		if ($this->importService->supportsMissingRecords(mode: $missingRecords) === false) {
-			return $this->error(code: 'MISSING_RECORDS_UNSUPPORTED', status: Http::STATUS_UNPROCESSABLE_ENTITY, details: ['accepted' => ['keep']]);
+		$accepted = $this->importService->missingRecordsModes();
+		if (in_array($missingRecords, $accepted, true) === false) {
+			return $this->error(
+				code: 'MISSING_RECORDS_UNSUPPORTED',
+				status: Http::STATUS_UNPROCESSABLE_ENTITY,
+				details: ['accepted' => $accepted]
+			);
 		}
 
 		// An unrecognised value is refused rather than read as true: true is the mode that overwrites.
@@ -293,6 +301,7 @@ private function readOptions(string $path, string $fileName): array|JSONResponse
 				'municipalityName' => $municipalityName,
 				'updateExisting' => $updateExisting,
 				'publish' => $publish,
+				'missingRecords' => $missingRecords,
 				'operationId' => $this->request->getParam('operationId'),
 				'fileName' => basename(str_replace('\\', '/', $fileName)),
 			],
@@ -394,7 +403,7 @@ private function message(string $code, array $details): string {
 			'NO_FILE_UPLOADED' => $this->l10n->t('No file was uploaded.'),
 			'NOT_XLSX' => $this->l10n->t('The file is not an Excel workbook (.xlsx).'),
 			'FILE_TOO_LARGE' => $this->l10n->t('The file is larger than the maximum of %s MB.', [$megabytes]),
-			'MISSING_RECORDS_UNSUPPORTED' => $this->l10n->t('Only keeping records that are missing from the export is supported.'),
+			'MISSING_RECORDS_UNSUPPORTED' => $this->l10n->t('Applications missing from the export can only be archived or kept.'),
 			'FIELD_INVALID' => $this->fieldMessage(field: (string)($details['field'] ?? ''), accepted: $accepted),
 			'MUNICIPALITY_REQUIRED' => $this->l10n->t('Choose a municipality or enter the name of a new one.'),
 			'MUNICIPALITY_INVALID' => $this->l10n->t('The chosen organisation is not a municipality.'),
@@ -410,6 +419,9 @@ private function message(string $code, array $details): string {
 			'SCHEMA_OUTDATED' => $this->l10n->t('The stackiq register is out of date; import its configuration again.'),
 			'IMPORT_IN_PROGRESS' => $this->l10n->t('Another CMDB import is running; try again when it has finished.'),
 			'MUNICIPALITY_AMBIGUOUS' => $this->l10n->t('Several municipalities have this name; choose one from the list.'),
+			'ARCHIVE_UNAVAILABLE' => $this->l10n->t(
+				'Applications missing from the export cannot be archived: OpenRegister is missing or too old. Choose to keep them, or update OpenRegister.'
+			),
 			default => $this->l10n->t('The import failed. The details are in the Nextcloud log.'),
 		};
 	}//end message()

lib/Exception/CmdbImportException.php

@@ -50,6 +50,7 @@ class CmdbImportException extends RuntimeException {
 	public const SCHEMA_OUTDATED = 'SCHEMA_OUTDATED';
 	public const IMPORT_IN_PROGRESS = 'IMPORT_IN_PROGRESS';
 	public const MUNICIPALITY_AMBIGUOUS = 'MUNICIPALITY_AMBIGUOUS';
+	public const ARCHIVE_UNAVAILABLE = 'ARCHIVE_UNAVAILABLE';
 
 	/**
 	 * HTTP status per error code.
@@ -70,6 +71,7 @@ class CmdbImportException extends RuntimeException {
 		self::SCHEMA_OUTDATED => 503,
 		self::IMPORT_IN_PROGRESS => 409,
 		self::MUNICIPALITY_AMBIGUOUS => 422,
+		self::ARCHIVE_UNAVAILABLE => 503,
 	];
 
 	/**

lib/Service/Cmdb/CmdbImportProfile.php

@@ -332,6 +332,42 @@ public function sheetNames(): array {
 		return array_column($this->sheets(), 'name');
 	}//end sheetNames()
 
+	/**
+	 * The name of the archive sheet, or null when the profile names none.
+	 *
+	 * The archive sheet lists the applications the municipality archived; the
+	 * import reads only its key column (`archiveSheet.name` in the profile).
+	 *
+	 * @return string|null
+	 *
+	 * @spec openspec/changes/cmdb-import-archive-reconciliation/specs/cmdb-export-import/spec.md#requirement-an-application-missing-from-the-cmdb-sheets-shall-be-archived-when-the-archive-sheet-lists-it-and-soft-deleted-when-no-sheet-does-req-cmdb-015
+	 */
+	public function archiveSheetName(): ?string {
+		$name = $this->profile()['archiveSheet']['name'] ?? null;
+		if (is_string($name) === false || $name === '' || in_array($name, $this->sheetNames(), true) === true) {
+			return null;
+		}
+
+		return $name;
+	}//end archiveSheetName()
+
+	/**
+	 * Every sheet the reader may parse: the source sheets, then the archive sheet.
+	 *
+	 * @return array<int, string>
+	 *
+	 * @spec openspec/changes/cmdb-import-archive-reconciliation/specs/cmdb-export-import/spec.md#requirement-an-application-missing-from-the-cmdb-sheets-shall-be-archived-when-the-archive-sheet-lists-it-and-soft-deleted-when-no-sheet-does-req-cmdb-015
+	 */
+	public function readSheetNames(): array {
+		$names = $this->sheetNames();
+		$archive = $this->archiveSheetName();
+		if ($archive !== null) {
+			$names[] = $archive;
+		}
+
+		return $names;
+	}//end readSheetNames()
+
 	/**
 	 * The rank of a sheet when an APPID is on more than one: lower wins.
 	 *
@@ -574,19 +610,28 @@ public function neverWrittenOnUpdate(string $target): array {
 	}//end neverWrittenOnUpdate()
 
 	/**
-	 * The accepted values of the missingRecords option.
+	 * The accepted values of the missingRecords option, readable without validating the packs.
+	 *
+	 * The controller checks the option before the import runs, so, like
+	 * maxFileBytes(), this must not depend on OpenRegister being available.
 	 *
 	 * @return array<int, string>
 	 *
-	 * @spec openspec/changes/cmdb-export-import/tasks.md#task-8
+	 * @spec openspec/changes/cmdb-import-archive-reconciliation/specs/cmdb-export-import/spec.md#requirement-records-missing-from-a-newer-export-shall-be-archived-by-default-or-kept-on-request-req-cmdb-012
 	 */
 	public function missingRecordsModes(): array {
-		$modes = $this->stringList(key: 'missingRecords');
-		if ($modes === []) {
+		try {
+			$profile = $this->profile ?? $this->decodeFile(fileName: $this->profileFile);
+		} catch (CmdbImportException $e) {
+			return ['keep'];
+		}
+
+		$modes = $profile['missingRecords'] ?? [];
+		if (is_array($modes) === false || $modes === []) {
 			return ['keep'];
 		}
 
-		return $modes;
+		return array_values(array_map('strval', $modes));
 	}//end missingRecordsModes()
 
 	/**

lib/Service/Cmdb/CmdbImportReport.php

@@ -36,6 +36,10 @@ class CmdbImportReport {
 	public const UNCHANGED = 'unchanged';
 	public const SKIPPED = 'skipped';
 	public const FAILED = 'failed';
+	public const ARCHIVED = 'archived';
+	public const UNARCHIVED = 'unarchived';
+	public const DELETED = 'deleted';
+	public const RESTORED = 'restored';
 
 	/**
 	 * The row entries, in processing order.
@@ -90,7 +94,7 @@ public function __construct(
 	 * Add one row outcome.
 	 *
 	 * @param string $sheet The sheet name.
-	 * @param int $row The 1-based sheet row number.
+	 * @param int $row The 1-based sheet row number, or 0 for an application the reconciliation archived or deleted.
 	 * @param string $appId The APPID ('' when missing).
 	 * @param string $name The application name ('' when missing).
 	 * @param string $outcome One of the outcome constants.
@@ -180,36 +184,54 @@ public function markCancelled(): void {
 	}//end markCancelled()
 
 	/**
-	 * The number of rows processed so far.
+	 * Whether the run stopped on a cancel.
+	 *
+	 * @return bool
+	 *
+	 * @spec openspec/changes/cmdb-import-archive-reconciliation/specs/cmdb-export-import/spec.md#requirement-an-application-missing-from-the-cmdb-sheets-shall-be-archived-when-the-archive-sheet-lists-it-and-soft-deleted-when-no-sheet-does-req-cmdb-015
+	 */
+	public function isCancelled(): bool {
+		return $this->cancelled;
+	}//end isCancelled()
+
+	/**
+	 * The number of sheet rows processed so far; reconciliation entries (row 0) are not sheet rows.
 	 *
 	 * @return int
 	 *
 	 * @spec openspec/changes/cmdb-export-import/tasks.md#task-7
 	 */
 	public function processed(): int {
-		return count($this->rows);
+		return count(array_filter($this->rows, static fn (array $row): bool => $row['row'] > 0));
 	}//end processed()
 
 	/**
 	 * The summary counts.
 	 *
 	 * `unpublished` counts the modules created without a publication date;
 	 * a row whose module was created but whose usage then failed counts too.
+	 * `processed` counts sheet rows only; every outcome, also of a
+	 * reconciliation entry, is counted under its own key.
 	 *
 	 * @return array{rowsRead: int, processed: int, created: int, updated: int, unchanged: int, skipped: int, failed: int,
-	 *     warnings: int, unpublished: int}
+	 *     archived: int, unarchived: int, deleted: int, restored: int, warnings: int, unpublished: int}
 	 *
 	 * @spec openspec/changes/cmdb-export-import/tasks.md#task-7
+	 * @spec openspec/changes/cmdb-import-archive-reconciliation/specs/cmdb-export-import/spec.md#requirement-an-application-missing-from-the-cmdb-sheets-shall-be-archived-when-the-archive-sheet-lists-it-and-soft-deleted-when-no-sheet-does-req-cmdb-015
 	 */
 	public function summary(): array {
 		$summary = [
 			'rowsRead' => $this->rowsRead,
-			'processed' => count($this->rows),
+			'processed' => $this->processed(),
 			self::CREATED => 0,
 			self::UPDATED => 0,
 			self::UNCHANGED => 0,
 			self::SKIPPED => 0,
 			self::FAILED => 0,
+			self::ARCHIVED => 0,
+			self::UNARCHIVED => 0,
+			self::DELETED => 0,
+			self::RESTORED => 0,
 			'warnings' => 0,
 			'unpublished' => $this->unpublished,
 		];