[Re] Invisible Farms: Reproducing and Stress-Testing Small-Scale Irrigation Detection for Zambia

[welu2027]

Original article: https://www.climatechange.ai/papers/neurips2025/114/paper.pdf

PDF URL: https://zenodo.org/records/21897060
Metadata URL: https://github.com/AIscend-Research/irrigation-repro/blob/main/metadata.yaml
Code URL: https://github.com/AIscend-Research/irrigation-repro

Scientific domain: Computational Sustainability
Programming language: Python
Suggested editor: @rougier

[rougier]

Thanks you for you submission, I'll handle it.

Since we're are a bit overwhelmed by submissions, can you, you and your co-authors review other submissions?
You'll find the list here. Just say you can review in the relevant issue.

Since you have several pending submissions, please review at least 5 submissions (excluding yours of course).

[tridev-tej]

@rougier will take this up

[tridev-tej]

Here is my review:

Overall

Well written, and I liked that you concluded the test set cannot answer the fairness question (§4.3) instead of reporting a gap. The Earth Engine → Planetary Computer swap is a genuinely useful contribution.

One major comment, then minors.

Major comment

I could not find the code behind Tables 3–9. The conclusion says "All code, the one-command Kaggle notebook, and the per-seed CSVs behind every table are released at the repository listed on the first page", but at the Code URL: the four experiments/ notebooks have no stored outputs, no result CSVs are tracked outside data/ and shiny_app/, and experiments/phase2_baseline/README.md says its figures "were produced by a Kaggle run that is not checked in". In the non-data paths and notebook sources I searched I found no implementation of the site-clustered permutation test, the clustered bootstrap, the reliability/Brier analysis (§4.2), or the synthetic-data fallback (§3.1).

The notebooks that are there do run — but they are a different experiment from the paper. The main one, phase3_extensions.ipynb, uses a single GroupShuffleSplit(test_size=0.3, random_state=42) at the default 0.5 threshold, not the KL holdout with five seeds that Tables 3–6 report; KL appears in it only as a dedupe priority. Its models also differ from Table 1 (RF-full = 400 trees with no depth cap vs "800 trees, max depth 25"; GB = 200 trees, depth 3 vs "100 trees, learning rate 0.1"). So I did not re-run anything, because the code that produced the paper's tables is not there to run. Could you push it, along with the per-seed outputs? The feature stage is not the obstacle — both Kaggle Sentinel-2 datasets are public.

Minor comments, in no particular order:

• The original article is public: CCAI NeurIPS 2025, Proposals track, https://www.climatechange.ai/papers/neurips2025/114 (PDF, slides and recorded talk). The metadata's "not yet public" TODO is out of date — please cite that URL in the replication entry. I confirmed from the original PDF that it reports no model metrics ("Precision, recall... and F1 score will be calculated"), so your framing in §2 is fair.

• The metadata abstract describes a different analysis from the paper: smallholder recall 0.23 against 0.32 and 0.30 as "the central failure", where the paper has 0.14 / 0.20 / 0.17 and p = 0.94, and four models at F1 0.49–0.53 where Table 3 has eight at 0.21–0.47 (default threshold). Notably the metadata numbers are what the repository notebooks produce — fairness_experiments.ipynb states the 0.23 / 0.32 / 0.30 result directly. So the metadata appears to describe the code that is in the repo, and the paper an analysis whose code is not.

• The repository is a fork of anna-boser/smallholder-irrigation-dataset, with that history and MIT license retained, but the README never names Boser or links the parent repository. Could you add a line saying what is inherited and what is yours? The paper itself credits the dataset properly.

• requirements.txt omits shap (imported by the Phase 3 notebook), pystac-client and planetary-computer (imported by both download notebooks), and imbalanced-learn (needed for the Table 5 SMOTE ablation), and specifies scikit-learn~=1.7.1 while §3.3 says 1.6.1.

• Table 6: for the > 10 ha row all 12 observations are positive, so AUPRC is undefined rather than degenerate; at n = 6 and n = 12 both strata are uninterpretable anyway. I would write n/a for both rather than a daggered number.

• §3.1 says 88 features. I get 72 from 12 months × 3 indices × 2 statistics — could you enumerate the other 16?

• §4.4 says 3.1% of test images share a site with training data. Since the labeler split stands in for spatially independent evaluation, could you report the numbers with those excluded?

• §4.6: the noise result reads as the same operating-point story rather than an independent robustness finding. Could the noise and cloud-gap sweeps be re-run at the tuned threshold?

• Table 9 ratios do not recompute from the printed values (319/1.6 = 199, not 204; 319/2.6 = 123, not 124). Unrounded timings? Worth a decimal.

• Metadata needs work regardless: code.doi empty, author emails empty, journal fields say volume 4 / issue 1 while the header says 12.1. You will also need the code archived on Software Heritage (https://www.softwareheritage.org/save-and-reference-research-software/) with the swh id in the metadata — the current template has a field for it.

Conclusion

Thoughtful analysis, but I cannot recommend acceptance until the code behind the tables is in the repository and the metadata matches the submitted paper. Happy to look again — I would like to re-run the baseline myself.

For the editor: the original is a Proposals-track submission reporting no model metrics, so this is arguably a new baseline rather than a reproduction of a published number. The authors are upfront about it (§2), but may be worth deciding explicitly.

P.S. em dashes are human generated :)