[Fixes #14524] Remote-resource detection is inconsistent across harvesters by etj · Pull Request #14532 · GeoNode/geonode

etj · 2026-08-17T16:27:07Z

This PR is built upon #14404 since it needs its fixes.

Checklist

Reviewing is a process done by project maintainers, mostly on a volunteer basis. We try to keep the overhead as small as possible and appreciate if you help us to do so by completing the following items. Feel free to ask in a comment if you have troubles with any of them.

For all pull requests:

Confirm you have read the contribution guidelines
You have sent a Contribution Licence Agreement (CLA) as necessary (not required for small changes, e.g., fixing typos in the documentation)
Make sure the first PR targets the master branch, eventual backports will be managed later. This can be ignored if the PR is fixing an issue that only happens in a specific branch, but not in newer ones.

The following are required only for core and extension modules (they are welcomed, but not required, for contrib modules):

There is a ticket in https://github.com/GeoNode/geonode/issues describing the issue/improvement/feature (a notable exemption is, changes not visible to end-users)
The issue connected to the PR must have Labels and Milestone assigned
PR for bug fixes and small new features are presented as a single commit
PR title must be in the form "[Fixes #<issue_number>] Title of the PR"
New unit tests have been added covering the changes, unless there is an explanation on why the tests are not necessary/implemented

Submitting the PR does not require you to check all items, but by the time it gets merged, they should be either satisfied or inapplicable.

…ath in test_get_service_handler_wms

Resolves conflicts from PR #14328 (generalize remote service type registration via ServiceTypeRegistry) against the #14381 work (relax Service.base_url uniqueness).

…andler Leftover from the earlier manual "Align to master" commit; harmless but redundant.

_get_service_handler (views.py) and GeoNodeServiceHandler.parsed_service (wms.py) build the request auth via get_request_auth(), which returns a HashableAuthBase-wrapped requests.auth object, but only passed auth= to get_service_handler, not auth_config=. Since HashableAuthBase exposes no `.username` attribute, _build_auth_cache_fingerprint fell through to a fingerprint that was constant for every HashableAuthBase instance ("auth:HashableAuthBase"), so changing a service's stored credentials would not bust the cached service handler until it naturally expired. Fix both call sites to also pass auth_config=service.auth_config, consistent with every other get_service_handler/get_service_cache_key call site. Also harden _build_auth_cache_fingerprint itself to unwrap HashableAuthBase and fingerprint the full underlying credential dict, so the auth= path is correct on its own rather than relying solely on callers always supplying auth_config too.

parsed_service calls self.ows_endpoint() (a real HTTP request) before get_cleaned_url_params, which wasn't mocked, causing a real DNS lookup against the phony test hostname. Mock ows_endpoint too.

Copilot

Pull request overview

This PR addresses inconsistencies in how GeoNode detects and handles remote resources across harvesters and service handlers, especially after allowing multiple Service records to share the same endpoint URL. It standardizes “remote-ness” on sourcetype, ensures harvested datasets don’t claim a GeoServer store, and makes service-handler caching service- and auth-specific.

Changes:

Make remote-resource detection consistent by keying it on sourcetype (and backfilling existing harvested resources via migration).
Relax remote service endpoint uniqueness (Service.base_url no longer unique) and make handler caching discriminated by (service_id, auth) to avoid collisions.
Ensure remote/harvested datasets have store = NULL (and Dataset.store becomes nullable), preventing uniqueness conflicts and incorrect GeoServer assumptions.

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
geonode/utils.py	Passes `service_id` and `auth_config` to service handlers when creating default links.
geonode/services/views.py	Updates service-handler caching to use a service/auth-specific cache key; adjusts imports accordingly.
geonode/services/tests.py	Adds/updates tests for cache key stability, auth discrimination, and unique-name retry behavior.
geonode/services/serviceprocessors/wms.py	Ensures remote datasets have `store=None`, fixes auth resolution in `GeoNodeServiceHandler.parsed_service`, and uses unique-name creation helper.
geonode/services/serviceprocessors/cache.py	Introduces a dedicated cache wrapper for service handlers with auth-aware keying.
geonode/services/serviceprocessors/base.py	Adds helpers to generate unique names across `Service` and `Harvester`, retrying on `IntegrityError`.
geonode/services/serviceprocessors/arcgis.py	Aligns ArcGIS service creation with unique-name helper and sets remote dataset `store=None`.
geonode/services/serviceprocessors/init.py	Reworks handler caching to be keyed by service+auth and avoids caching when `service_id` is missing.
geonode/services/models.py	Removes uniqueness constraint from `Service.base_url`.
geonode/services/migrations/0061_alter_service_base_url.py	Migration to drop `unique=True` from `Service.base_url`.
geonode/services/forms.py	Removes duplicate-URL validation and updates Taggit field import usage.
geonode/security/tests.py	Adjusts expectations around metadata-only datasets; adds test ensuring remote resources skip GeoFence syncing.
geonode/resource/utils.py	Changes `is_remote_resource()` to rely on `sourcetype` (with fallback warning for legacy records).
geonode/layers/tests.py	Adds coverage for remote dataset uniqueness behavior and separates vector/raster link tests.
geonode/layers/models.py	Makes `Dataset.store` nullable/blank for remote datasets.
geonode/layers/migrations/0047_alter_dataset_store.py	Migration to make `Dataset.store` nullable.
geonode/layers/api/tests.py	Updates API listing/count expectations to exclude metadata-only datasets.
geonode/harvesting/tests/test_harvester_worker_wms.py	Adds test asserting harvested WMS resources have no `store`.
geonode/harvesting/migrations/0051_flag_harvested_resources_as_remote.py	Backfills harvested resources to `sourcetype=REMOTE` and sets `remote_service` links.
geonode/harvesting/harvesters/wms.py	Ensures harvested WMS datasets set `subtype` explicitly and `store=None`.
geonode/harvesting/harvesters/geonodeharvester.py	Ensures remote dataset descriptors/additional parameters use `store=None`.
geonode/harvesting/harvesters/base.py	Sets harvested resource `sourcetype=REMOTE` consistently (not conditional on bbox presence).
geonode/harvesting/harvesters/arcgis.py	Ensures harvested ArcGIS datasets set `subtype` and `store=None`.
geonode/geoserver/signals.py	Makes MapLayer→Dataset resolution tolerant of ambiguity when endpoints are shared across services.
geonode/geoserver/security.py	Skips GeoFence rule syncing for remote datasets and clears dirty state instead.
geonode/geoserver/manager.py	Uses `is_remote_resource()` for GeoServer existence checks rather than subtype heuristics.
geonode/base/api/tests.py	Updates base resource totals to match fixture/state changes.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

        dataset_queryset = Dataset.objects.filter(Q(alternate=instance.name) | Q(name=instance.name))
-        if instance.local and instance.store:
+        if instance.store:
            dataset_queryset = dataset_queryset.filter(store=instance.store)
        elif instance.ows_url:
            dataset_queryset = dataset_queryset.filter(remote_service__base_url=instance.ows_url)


Copilot

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated no new comments.

Suppressed comments (2)

geonode/services/serviceprocessors/arcgis.py:142

In ArcGIS service registration, the newly created Harvester is assigned to new_instance.harvester, but the Service instance is not saved afterward. Since Service.harvester is a DB field (FK), this assignment won't persist and the Service may end up without an attached harvester in the database.

                new_instance.harvester = new_harvester
                return new_instance

geonode/harvesting/migrations/0051_flag_harvested_resources_as_remote.py:3

This data migration imports SOURCE_TYPE_REMOTE from application code. Migrations should be self-contained so they keep working even if geonode.base.enumerations changes in the future; prefer inlining the current literal value instead.

from django.db import migrations

from geonode.base.enumerations import SOURCE_TYPE_REMOTE

etj · 2026-08-18T14:07:04Z

Superseded by #14536

etj added 17 commits July 2, 2026 16:33

[Fixes #1438] Relax Remote Service URL uniqueness

71572aa

Fixes

4b692b1

tests: restore ServiceTypeRegistry tests and reset guards; fix mock p…

88a3bcb

…ath in test_get_service_handler_wms

Align to master

a9ba4b7

Merge branch 'master' into 14381_url_unique

2142582

Resolves conflicts from PR #14328 (generalize remote service type registration via ServiceTypeRegistry) against the #14381 work (relax Service.base_url uniqueness).

Remove duplicate self.args/self.kwargs assignment in ArcImageServiceH…

966b8f5

…andler Leftover from the earlier manual "Align to master" commit; harmless but redundant.

Fix test_geonode_service_handler_parsed_service_passes_auth_config

66faea2

parsed_service calls self.ows_endpoint() (a real HTTP request) before get_cleaned_url_params, which wasn't mocked, causing a real DNS lookup against the phony test hostname. Mock ows_endpoint too.

Refact services cache

dd91d8b

Fixes according to review

83d98e9

Fixes according to review

d96a25c

Do not cache unpersisted services

fcac009

Fix test

d514862

Fix cache by adding auth

4d187f1

Nullable store for remote services

da14dbc

Merge remote-tracking branch 'origin/master' into 14381_url_unique

72dacf9

Fix tests

861c919

etj self-assigned this Aug 17, 2026

Copilot AI lite review requested due to automatic review settings August 17, 2026 16:27

cla-bot Bot added the cla-signed CLA Bot: community license agreement signed label Aug 17, 2026

Copilot started reviewing on behalf of etj August 17, 2026 16:27 View session

Copilot AI reviewed Aug 17, 2026

View reviewed changes

etj added 3 commits August 18, 2026 10:12

Prevent username leak in auth cache key

f37f4bb

Fix detection of remote resources

89532cb

Avoid handling GeoFence rules for remote datasets

246ecde

etj force-pushed the 14524_remove branch from abbdf10 to 246ecde Compare August 18, 2026 08:14

etj requested review from mattiagiupponi and a lite review from Copilot August 18, 2026 11:00

Copilot started reviewing on behalf of etj August 18, 2026 11:00 View session

Copilot AI reviewed Aug 18, 2026

View reviewed changes

Fixes

5c9ccae

etj mentioned this pull request Aug 18, 2026

[Fixes #14381] Service URL uniqueness [Fixes #14524] Remote-resource detection #14536

Open

8 tasks

etj closed this Aug 18, 2026

Navigation Menu

Uh oh!

[Fixes #14524] Remote-resource detection is inconsistent across harvesters - #14532

[Fixes #14524] Remote-resource detection is inconsistent across harvesters#14532
etj wants to merge 21 commits into
masterfrom
14524_remove

etj commented Aug 17, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

etj commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

etj commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

etj commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

etj commented Aug 17, 2026 •

edited

Loading