Add retries with backoff and increase timeout in statistics_poland download script

[abhishekjaisw]

Description

Fixes download script failure in statvar_imports/statistics_poland/download_input_data.py.

Root Cause

1. In Cloud Batch execution, requests to the external Statistics Poland API (https://bdl.stat.gov.pl/api/v1/) intermittently take > 20s, causing Read timed out failures when timeout=20.
2. When rapid requests were sent sequentially, the API began returning HTTP 429 Too Many Requests. The previous logic dropped non-200 responses without retrying.

Fix

1. HTTP Session Retries: Configured urllib3.util.Retry on requests.Session for connection drops and server 5xx errors.
2. Request-Level Backoff (make_request): Added 5 retries with exponential backoff on timeouts/connection errors and HTTP 429/5xx status codes (respecting Retry-After header).
3. Increased Timeout: Increased read timeout from 20s to 60s.
4. Pacing: Added pacing delay between variable requests.

Dev Cloud Batch Verification (us-west1)

• Job ID: statistics-poland-abhishekjaisw-20260813-130859 (UID: statistics-poland-5ae3a2da-5e3c-45c700)
• Status: SUCCESS (Stage: STAGING)
• Rows Processed: 42,228 (100% matched)
• Deleted Obs: 0 (0.0%)

Validation Check	Status	Details
check_deleted_records_percent	PASSED	0 deleted records (0.0%)
check_empty_import	PASSED	42,317 nodes, 42,228 rows
check_missing_refs_count	PASSED	0 missing refs
check_lint_error_count	PASSED	0 lint errors
check_goldens_summary_report	PASSED	108/108 variables matched
check_goldens_observations	PASSED	17/17 goldens matched, 42,228 inputs matched

[gemini-code-assist]

Code Review

This pull request introduces robust HTTP session management with automatic retries and exponential backoff for downloading data from Statistics Poland. It adds helper functions get_http_session and make_request and refactors fetch_variables and the main download loop to use this session. The review feedback points out a redundant double-retry loop in make_request because the session is already configured with an HTTPAdapter and a Retry strategy. It is recommended to simplify make_request to rely entirely on the session's built-in retry strategy and increase the retry count to guarantee download success.

[gemini-code-assist]

The current implementation of make_request introduces a redundant double-retry loop. Since the requests.Session is already configured with an HTTPAdapter and a Retry strategy, wrapping session.get in a manual retry loop will cause nested retries. We should simplify make_request to rely entirely on the session's built-in retry strategy. Additionally, when downloading data from unstable or unreliable external sources like Statistics Poland, ensure that the session's retry strategy is configured with a higher number of retries (e.g., 10 retries) to guarantee download success, even if it results in a long cumulative wait time.

def make_request(session, url, headers=None, params=None, timeout=60):
    # Ensure the session's retry strategy is configured with a high number of retries (e.g., 10)
    return session.get(url, headers=headers, params=params, timeout=timeout)

References

1. When downloading data from unstable or unreliable external sources, configure a higher number of retries (e.g., tries=10) to ensure download success, even if it results in a long cumulative wait time.

[abhishekjaisw]

Addressed in commit f905d4c. Simplified make_request to rely directly on the session's built-in HTTPAdapter with urllib3.util.Retry and configured 10 retries with exponential backoff on all status codes and timeouts.