diff --git a/.gitignore b/.gitignore index 380efe19e..622aebe01 100644 --- a/.gitignore +++ b/.gitignore @@ -60,3 +60,7 @@ frontend/.astro # Bundler/vendored dependencies /vendor/ + +# Local bundle builds +/dist/ + diff --git a/.rubocop.yml b/.rubocop.yml index ee5e2c04c..328acf9b4 100644 --- a/.rubocop.yml +++ b/.rubocop.yml @@ -5,6 +5,7 @@ plugins: - rubocop-thread_safety AllCops: + TargetRubyVersion: 4.0 DisplayCopNames: true NewCops: enable Exclude: @@ -38,6 +39,10 @@ Style/Documentation: AllowedConstants: - App +Style/ItBlockParameter: + Enabled: true + EnforcedStyle: allow_single_line + RSpec/SpecFilePathFormat: Exclude: - 'spec/html2rss/web/app/*_spec.rb' diff --git a/AGENTS.md b/AGENTS.md index 07b31e815..5a053e053 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -54,21 +54,63 @@ See [docs/design-system.md](docs/design-system.md) for visual rules. - **No host execution:** All commands MUST run inside the Dev Container via `make` or `bundle exec`. - **No skipped quality gate:** Opening a PR without a green Dev Container gate is forbidden. If the gate cannot run, do not open the PR; fix the environment or hand off with explicit blocker + next command for the user. +## Ruby 4 Style + +**Ruby 4.0+ only** (see `.tool-versions`). No Ruby 3.x backward-compat shims, guards, or dual-path APIs. + +### Baseline + +- `# frozen_string_literal: true` on every `.rb` file +- Plain Ruby — no ActiveSupport +- Keyword arguments for public multi-arg APIs +- Typed YARD on public methods in `app/` (`@param`, `@return`) — enforced by `make yard-verify-public-docs` + +### Modern syntax (prefer consistently) + +| Idiom | Use instead of | +| --- | --- | +| Leading `&&` / `\|\|` at line start (Ruby 4) | Trailing operators on long wrapped conditions | +| `it` in single-parameter blocks | `{ \|x\| x.foo }` when block has one arg only | +| Pattern matching (`in`, `case … in`) | Deep `if/elsif` chains on shape | +| `Data.define` | OpenStruct / hand-rolled structs | +| `filter_map`, `index_by`, `then`, `match?` | Verbose `map`/`compact`, nested `if`, `=~` | +| Endless `def` | One-line pure helpers when RuboCop allows | +| Core `Set` (no `require 'set'`) | Array membership/diff on growing collections | + +### Performance (agent defaults) + +- **Set** for catalog/diff/membership when sizes can grow +- **Memoize** repeated `ENV.fetch` / pure computations on hot paths +- **One owner** for duplicated helpers — dedupe before splitting into new files +- **Functional iterators** over imperative loops +- **No metric-driven micro-methods** whose only purpose is satisfying RuboCop metrics +- Do **not** document ZJIT/Ruby Box/Ractor as defaults + +### Web-specific deltas + +- Prefer `class << self` + `private` over `module_function` (see docs/README Architectural Constraints) +- Do not use `send(...)` to reach private APIs in app code or specs +- Specs: table-drive matrices; `:aggregate_failures` for discriminating multi-assert examples +- LOC: dedupe/unify before extracting — new files only when they buy a real seam or test surface + ## Config catalog API -Public feed-directory metadata for embedded and local configs. +Public feed-directory metadata from verified registry bundles and local `feeds.yml` entries. | Item | Detail | | --- | --- | | Endpoint | `GET /api/v1/configs` | | Flag | `CONFIG_CATALOG_ENABLED` (default `true`; set `false` to disable) | | Disabled response | `404` with `{ "error": "catalog_disabled" }` | -| Embedded entries | `Html2rss::Configs::Catalog.entries` — do not re-walk YAML in the handler | -| Local entries | `Catalog::Merge` includes `feeds.yml` feeds only when `directory.title` is set | +| Registry entries | `Registry::Index.current.catalog_rows` — loads signed bundles from `config/registries.yml`; adds `source: registry`, `registry: ` | +| Local entries | `Registry::Index` catalog rows include `feeds.yml` feeds only when `directory.title` is set (`source: local`) | +| Per-registry privacy | `catalog: false` in `registries.yml` omits that registry from the API (feeds still served) | | Starter feeds (UI) | Frontend `selectStarterFeeds` when feed creation is disabled; catalog find uses full catalog when enabled | | Catalog find | `findCatalogEntries` → multi-hit list under create URL; links via `catalogFeedHref` (path + defaults) | | CORS | Route-scoped on `/api/v1/configs` only (`GET`, `OPTIONS`) | -| Root metadata | `GET /api/v1/` exposes `instance.catalog: { enabled, url }` | +| Root metadata | `GET /api/v1/` exposes `instance.catalog: { enabled, url }` and `instance.registries` sync status | | Contract SSOT | Request specs under `spec/html2rss/web/api/v1_spec.rb` and generated `public/openapi.yaml` | +Registry sync: `bin/html2rss-web registry status`; embedded bundle + optional runtime sync via `Registry::Sync.boot!`. See [docs/README.md](docs/README.md#registry-sync-runbook). + After handler or envelope changes: `make openapi` and `make ci-ready`. diff --git a/CONTEXT.md b/CONTEXT.md index 81c2ec430..bd65d1bc3 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -54,3 +54,12 @@ Audit channel: snake_case `security_event` with IP / user-agent / token hash. Au ### LogEvent Shared emit plumbing for both channels (`RequestContext`, `LogSanitizer`, `AppLogger` / Sentry). Not a third public facade. + +### Registry Index +Backend merge owner for registry bundles and local `feeds.yml` feeds. Builds catalog wire rows (`Registry::Index::CatalogRow`), enforces load-time trust and channel-domain allowlists, and serves `config_for` / `catalog_rows` / `status`. + +### Registry Sync +Backend orchestration for fetch → verify → stage/promote of signed registry bundles. Owns boot initialization, background refresh, CLI exit codes, and catalog-change telemetry after promotion. + +### Sync Transport +Backend HTTPS fetch, sync URL resolution (GitHub releases and channel defaults), and manifest version gating used by `Registry::Sync` and parse-time config resolution. diff --git a/Dockerfile b/Dockerfile index d9bc8e0a5..0f16d5a53 100644 --- a/Dockerfile +++ b/Dockerfile @@ -38,7 +38,26 @@ RUN apk add --no-cache \ /usr/local/bundle/bundler/gems/*/.git \ /usr/local/bundle/cache/bundler/git -# Stage 3: Runtime +# Stage 3: Official Registry Artifact Builder & Verifier +FROM ${RUBY_BASE_IMAGE} AS registry-builder + +ARG REGISTRY_BUNDLE_URL="https://github.com/html2rss/html2rss-configs/releases/latest/download/registry-bundle.tar.gz" + +WORKDIR /app + +COPY --from=builder /usr/local/bundle /usr/local/bundle +COPY bin ./bin +COPY app ./app +COPY config ./config + +# hadolint ignore=DL3018 +RUN apk add --no-cache curl tar ca-certificates \ + && curl -fsSL -o registry-bundle.tar.gz "${REGISTRY_BUNDLE_URL}" \ + && mkdir -p /build/official \ + && tar -xzf registry-bundle.tar.gz -C /build/official \ + && bin/html2rss-web registry verify --registry official --dir /build/official + +# Stage 4: Runtime FROM ${RUBY_BASE_IMAGE} LABEL maintainer="Gil Desmarais " @@ -51,13 +70,14 @@ ARG GIT_SHA ENV PORT=4000 \ RACK_ENV=production \ RUBY_YJIT_ENABLE=1 \ + PATH="/app/bin:$PATH" \ BUILD_TAG=${BUILD_TAG} \ GIT_SHA=${GIT_SHA} EXPOSE $PORT HEALTHCHECK --interval=30m --timeout=60s --start-period=5s \ - CMD ["/app/bin/docker-healthcheck"] + CMD ["/app/bin/html2rss-web", "healthcheck"] ARG USER=html2rss ARG UID=991 @@ -78,6 +98,7 @@ RUN apk add --no-cache \ && mkdir -p /app \ && mkdir -p /app/tmp/rack-cache-body \ && mkdir -p /app/tmp/rack-cache-meta \ + && mkdir -p /app/data/registries \ && chown "$USER":"$USER" -R /app WORKDIR /app @@ -85,10 +106,11 @@ WORKDIR /app USER 991 COPY --from=builder /usr/local/bundle /usr/local/bundle -COPY --chown=$USER:$USER bin/docker-healthcheck ./bin/docker-healthcheck +COPY --chown=$USER:$USER bin ./bin COPY --chown=$USER:$USER Gemfile Gemfile.lock app.rb config.ru ./ COPY --chown=$USER:$USER app ./app COPY --chown=$USER:$USER config ./config +COPY --from=registry-builder --chown=$USER:$USER /build/official ./registries/official COPY --chown=$USER:$USER public ./public COPY --from=frontend-builder --chown=$USER:$USER /app/frontend/dist ./frontend/dist diff --git a/Gemfile b/Gemfile index 5816b002d..85e42a7e8 100644 --- a/Gemfile +++ b/Gemfile @@ -4,13 +4,8 @@ source 'https://rubygems.org' git_source(:github) { |repo_name| "https://github.com/#{repo_name}" } -gem 'html2rss', '~> 0.27' -# gem 'html2rss', github: 'html2rss/html2rss', branch: 'master' -gem 'html2rss-configs', github: 'html2rss/html2rss-configs' - -# Use these instead of the two above (uncomment them) when developing locally: -# gem 'html2rss', path: '../html2rss' -# gem 'html2rss-configs', path: '../html2rss-configs' +# Until rubygems 0.28.0: git branch; local monorepo: BUNDLE_LOCAL__HTML2RSS=/path/to/html2rss +gem 'html2rss', github: 'html2rss/html2rss', branch: 'feat/registry-v1' gem 'base64' gem 'rack-cache' diff --git a/Gemfile.lock b/Gemfile.lock index 9e2e0b168..9f4a6eb49 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1,9 +1,29 @@ GIT - remote: https://github.com/html2rss/html2rss-configs - revision: f888a8dc5bb260c998c5ae8f39bb7d45320638ee + remote: https://github.com/html2rss/html2rss + revision: 5044272b9dcf404a1ef550e0496508cd6ce5687e + branch: feat/registry-v1 specs: - html2rss-configs (0.2.0) - html2rss + html2rss (0.28.0) + addressable (~> 2.7) + brotli + dry-validation + faraday (> 2.0.1, < 3.0) + faraday-follow_redirects + faraday-gzip (~> 3) + kramdown + mcp (~> 1.2) + mime-types (> 3.0) + nokogiri (>= 1.10, < 2.0) + rack (~> 3.0) + rackup (~> 2.0) + regexp_parser + reverse_markdown (~> 3.0) + rss + sanitize + thor + tzinfo + webrick (~> 1.9) + zeitwerk GEM remote: https://rubygems.org/ @@ -103,27 +123,6 @@ GEM net-http (~> 0.5) hana (1.3.7) hashdiff (1.2.1) - html2rss (0.27.2) - addressable (~> 2.7) - brotli - dry-validation - faraday (> 2.0.1, < 3.0) - faraday-follow_redirects - faraday-gzip (~> 3) - kramdown - mcp (~> 1.2) - mime-types (> 3.0) - nokogiri (>= 1.10, < 2.0) - rack (~> 3.0) - rackup (~> 2.0) - regexp_parser - reverse_markdown (~> 3.0) - rss - sanitize - thor - tzinfo - webrick (~> 1.9) - zeitwerk i18n (1.15.2) concurrent-ruby (~> 1.0) io-console (0.9.2) @@ -319,8 +318,7 @@ PLATFORMS DEPENDENCIES base64 climate_control - html2rss (~> 0.27) - html2rss-configs! + html2rss! irb puma rack-cache @@ -377,8 +375,7 @@ CHECKSUMS faraday-net_http (3.4.4) sha256=0e78af151747ed1b00f33e25973b4bc220d7f16c00c39676817c8b12331eb588 hana (1.3.7) sha256=5425db42d651fea08859811c29d20446f16af196308162894db208cac5ce9b0d hashdiff (1.2.1) sha256=9c079dbc513dfc8833ab59c0c2d8f230fa28499cc5efb4b8dd276cf931457cd1 - html2rss (0.27.2) sha256=82b28308c023cb669704bf9b0612aa2a00730be4b53d88f957d456ba5e631ac8 - html2rss-configs (0.2.0) + html2rss (0.28.0) i18n (1.15.2) sha256=00f9eb62412fe593b2a65a97daa75300d37abb8f7202ec748e94b6d46a9dd1b5 io-console (0.9.2) sha256=efa74f891dd03c0939a931dfc6e74c2813d904763d456ea9762b0525e748db08 irb (1.18.0) sha256=de9454a0703a54704b9811a5ef31a60c86949fbf4013fcf244fabc7c775248e3 diff --git a/app/web/api/v1/configs.rb b/app/web/api/v1/configs.rb index 900b9b9b9..694b46252 100644 --- a/app/web/api/v1/configs.rb +++ b/app/web/api/v1/configs.rb @@ -17,7 +17,7 @@ def index(_router) entries, duration_ms = build_entries emit_success(entries.size, duration_ms) success_payload(entries) - rescue Html2rss::Configs::Catalog::MissingDirectoryTitle => error + rescue Html2rss::Registry::CatalogBuilder::MissingDirectoryTitle => error emit_failure(error) raise end @@ -25,12 +25,16 @@ def index(_router) private def build_entries - started = Process.clock_gettime(Process::CLOCK_MONOTONIC) - entries = Html2rss::Web::Catalog::Merge.call - duration_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started) * 1000).round + started = monotonic_now + entries = Registry::Index.current.catalog_rows + duration_ms = ((monotonic_now - started) * 1000).round [entries, duration_ms] end + def monotonic_now + Process.clock_gettime(Process::CLOCK_MONOTONIC) + end + def emit_success(count, duration_ms) Observability.emit( event_name: 'catalog.build', diff --git a/app/web/api/v1/root_metadata.rb b/app/web/api/v1/root_metadata.rb index 562f5a86b..e3d38096e 100644 --- a/app/web/api/v1/root_metadata.rb +++ b/app/web/api/v1/root_metadata.rb @@ -27,16 +27,41 @@ def build(router) # @return [Hash{Symbol=>Object}] def instance_payload(router) { - feed_creation: { - enabled: Flags.auto_source_enabled?, - access_token_required: Flags.auto_source_enabled? - }, - catalog: { - enabled: Flags.config_catalog_enabled?, - url: "#{router.base_url}/api/v1/configs" - } + feed_creation: feed_creation_payload, + catalog: catalog_payload(router), + registries: registry_status_rows + } + end + + # @return [Hash{Symbol => Object}] + def feed_creation_payload + auto_source = Flags.auto_source_enabled? + { + enabled: auto_source, + access_token_required: auto_source + } + end + + # @param router [Roda::RodaRequest] + # @return [Hash{Symbol => Object}] + def catalog_payload(router) + { + enabled: Flags.config_catalog_enabled?, + url: "#{router.base_url}/api/v1/configs" } end + + # @return [Array Object}>] + def registry_status_rows + Registry::Index.current.status.map do |entry| + { + id: entry.id, + version: entry.version, + updated_at: entry.updated_at&.utc&.iso8601, + sync_mode: entry.mode.to_s + } + end + end end end end diff --git a/app/web/boot.rb b/app/web/boot.rb index deed80ba6..ceba52f75 100644 --- a/app/web/boot.rb +++ b/app/web/boot.rb @@ -60,7 +60,10 @@ def build_loader def configure_loader(new_loader) new_loader.push_dir(app_root, namespace: Html2rss) collapsed_web_dirs.each { |path| new_loader.collapse(path) } - new_loader.inflector.inflect('api_v1' => 'ApiV1') + new_loader.inflector.inflect( + 'api_v1' => 'ApiV1', + 'cli' => 'CLI' + ) end # @return [Array] diff --git a/app/web/boot/setup.rb b/app/web/boot/setup.rb index 55ac05b34..16a112a2b 100644 --- a/app/web/boot/setup.rb +++ b/app/web/boot/setup.rb @@ -24,6 +24,7 @@ def call! configure_request_service! configure_runtime_logging! configure_gem_defaults! + configure_registry! log_startup! end @@ -32,9 +33,11 @@ def call! # @return [void] def configure_gem_defaults! global_config = LocalConfig.global + headers = global_config[:headers] + stylesheets = global_config[:stylesheets] Html2rss.configure do |config| - config.headers = global_config[:headers] if global_config[:headers] - config.stylesheets = global_config[:stylesheets] if global_config[:stylesheets] + config.headers = headers if headers + config.stylesheets = stylesheets if stylesheets end end @@ -76,6 +79,11 @@ def configure_runtime_logging! Rack::Timeout::Logger.logger = AppLogger.logger end + # @return [void] + def configure_registry! + Registry::Sync.boot! + end + # @return [void] def log_startup! AppLogger.logger.info( diff --git a/app/web/catalog/merge.rb b/app/web/catalog/merge.rb deleted file mode 100644 index d7e6b7734..000000000 --- a/app/web/catalog/merge.rb +++ /dev/null @@ -1,101 +0,0 @@ -# frozen_string_literal: true - -require 'html2rss/configs' - -module Html2rss - module Web - ## - # Merges embedded catalog entries with local feed configs for the public catalog API. - module Catalog - module Merge - STARTER_FEED_IDS = %w[ - microsoft.com/azure-products - phys.org/weekly - softwareleadweekly.com/issues - ].freeze - - module_function - - ## - # @return [Array Object}>] - def call - embedded = Html2rss::Configs::Catalog.entries.map(&:to_h) - local = local_entries - (embedded + local).sort_by { |entry| entry.fetch(:id) } - end - - ## - # @return [Array Object}>] - def starter_entries - entries = call - selected = STARTER_FEED_IDS.filter_map { |id| entries.find { |entry| entry.fetch(:id) == id } } - selected.empty? ? entries.first(3) : selected - end - - ## - # @return [Array Object}>] - def local_entries - LocalConfig.feeds.filter_map do |feed_name, feed_config| - build_local_entry(feed_name, feed_config) - end - end - - ## - # @param feed_name [String, Symbol] - # @param feed_config [Hash] - # @return [Hash{Symbol => Object}, nil] - def build_local_entry(feed_name, feed_config) - directory = feed_config[:directory] || {} - title = directory[:title] - return nil if title.to_s.strip.empty? - - id = feed_name.to_s - channel = feed_config[:channel] || {} - - local_entry(id, directory, title, channel) - end - - ## - # @param id [String] - # @param directory [Hash] - # @param title [String] - # @param channel [Hash] - # @return [Hash{Symbol => Object}] - def local_entry(id, directory, title, channel) - { - id:, - path: "/#{id}.rss", - source: 'local', - directory: local_directory(directory, title), - channel: local_channel(channel, title), - parameters: { schema: {}, defaults: {} } - } - end - - ## - # @param directory [Hash] - # @param title [String] - # @return [Hash{Symbol => Object}] - def local_directory(directory, title) - { - title: title.to_s, - summary: directory[:summary], - topics: Array(directory[:topics]) - }.compact - end - - ## - # @param channel [Hash] - # @param title [String] - # @return [Hash{Symbol => Object}] - def local_channel(channel, title) - { - url: channel.fetch(:url), - language: channel[:language], - title: channel[:title] || title.to_s - }.compact - end - end - end - end -end diff --git a/app/web/cli.rb b/app/web/cli.rb new file mode 100644 index 000000000..bca687f21 --- /dev/null +++ b/app/web/cli.rb @@ -0,0 +1,234 @@ +# frozen_string_literal: true + +require 'net/http' +require 'optparse' +require 'uri' + +module Html2rss + module Web + ## + # Unified command-line interface for operator workflows and container management. + module CLI # rubocop:disable Metrics/ModuleLength + STATUS_HEADERS = %w[registry mode version staged_version updated_at sync_url last_error].freeze + + module_function + + ## + # Dispatches command-line arguments to the appropriate handler. + # + # @param argv [Array] command-line arguments + # @param out [IO] standard output stream + # @param err [IO] standard error stream + # @return [Integer] process exit code (0 for success, 1 for failure) + def run(argv = ARGV, out: $stdout, err: $stderr) # rubocop:disable Metrics/MethodLength + case argv.first + when 'registry' then run_registry(argv[1..] || [], out:, err:) + when 'healthcheck' then run_healthcheck(out:, err:) + when 'version', '-v', '--version' then run_version(out:) + when '-h', '--help', 'help', nil + print_root_help(out:) + 0 + else + err.puts "Unknown command: #{argv.first.inspect}. Run with --help for usage." + 1 + end + end + + ## + # @param args [Array] + # @param out [IO] + # @param err [IO] + # @return [Integer] + def run_registry(args, out:, err:) # rubocop:disable Metrics/CyclomaticComplexity, Metrics/MethodLength + case args.first + when 'status' then registry_status(args[1..] || [], out:) + when 'sync' then registry_sync(args[1..] || []) + when 'promote' then registry_promote(args[1..] || []) + when 'verify' then registry_verify(args[1..] || [], out:, err:) + when '-h', '--help', 'help', nil + print_registry_help(out:) + 0 + else + err.puts "Unknown registry command: #{args.first.inspect}. Run with --help for usage." + 1 + end + end + + ## + # @param args [Array] + # @param out [IO] + # @return [Integer] + def registry_status(args, out:) + options = { registry_id: nil } + OptionParser.new do |opts| + opts.banner = 'Usage: html2rss-web registry status [options]' + opts.on('--registry ID', 'Inspect a single registry ID') { options[:registry_id] = it } + end.parse!(args) + + print_status_table(options[:registry_id], out:) + Registry::Sync.cli_exit_code + end + + ## + # @param args [Array] + # @return [Integer] + def registry_sync(args) + options = { registry_id: nil, dry_run: false } + OptionParser.new do |opts| + opts.banner = 'Usage: html2rss-web registry sync [options]' + opts.on('--registry ID', 'Sync a single registry ID') { options[:registry_id] = it } + opts.on('--dry-run', 'Fetch and verify without swapping active bundle') { options[:dry_run] = true } + end.parse!(args) + + target_registry_ids(options[:registry_id]).each do |registry_id| + Registry::Sync.run(registry_id:, dry_run: options[:dry_run]) + end + Registry::Sync.cli_exit_code + end + + ## + # @param args [Array] + # @return [Integer] + def registry_promote(args) + options = { registry_id: nil } + OptionParser.new do |opts| + opts.banner = 'Usage: html2rss-web registry promote [options]' + opts.on('--registry ID', 'Promote a single registry ID') { options[:registry_id] = it } + end.parse!(args) + + target_registry_ids(options[:registry_id]).each do |registry_id| + Registry::Sync.promote_staged!(registry_id:) + end + Registry::Sync.cli_exit_code + end + + ## + # @param args [Array] + # @param out [IO] + # @param err [IO] + # @return [Integer] + def registry_verify(args, out:, err:) # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity + options = { registry_id: 'official', dir: nil } + OptionParser.new do |opts| + opts.banner = 'Usage: html2rss-web registry verify [options]' + opts.on('--registry ID', 'Registry ID from config (default: official)') { options[:registry_id] = it } + opts.on('--dir PATH', 'Directory containing bundle to verify') { options[:dir] = it } + end.parse!(args) + + registry_id = options[:registry_id] + definition = Registry::Config.entry(registry_id) + raise Registry::Errors::ConfigError, "Unknown registry '#{registry_id}'" unless definition + + dir = options[:dir] || Registry::Store.active_dir(registry_id) || Registry::Store.embedded_dir(registry_id) + raise Registry::Errors::LoadError, "Bundle directory not found: #{dir}" unless dir && File.directory?(dir) + + trust = definition.mode == :path ? :integrity_only : :signed + public_keys = definition.mode == :path ? {} : { definition.public_key_id => definition.public_key }.compact + manifest = Html2rss::Registry::Verifier.verify!(dir, trust:, public_keys:) + out.puts "Verified registry bundle '#{registry_id}' (#{manifest.version}) at #{dir}" + 0 + rescue StandardError => error + err.puts "Registry verification failed: #{error.message}" + 1 + end + + ## + # @param out [IO] + # @param err [IO] + # @return [Integer] + def run_healthcheck(out:, err:) # rubocop:disable Metrics/MethodLength + port = ENV.fetch('PORT', 4000) + uri = URI.parse("http://127.0.0.1:#{port}/api/v1/health/live") + response = Net::HTTP.get_response(uri) + if response.is_a?(Net::HTTPSuccess) + out.puts 'OK' + 0 + else + err.puts "Healthcheck failed: HTTP #{response.code}" + 1 + end + rescue StandardError => error + err.puts "Healthcheck failed: #{error.message}" + 1 + end + + ## + # @param out [IO] + # @return [Integer] + def run_version(out:) + out.puts "html2rss-web build=#{RuntimeEnv.build_tag} sha=#{RuntimeEnv.git_sha} ruby=#{RUBY_VERSION}" + 0 + end + + ## + # @param out [IO] + # @return [void] + def print_root_help(out:) + out.puts <<~HELP + Usage: html2rss-web [command] [options] + + Commands: + registry status [--registry ID] Show status of configured registries + registry sync [--registry ID] [--dry-run] Fetch, verify, and sync registry bundles + registry promote [--registry ID] Promote verified staged bundle to active + registry verify [--registry ID] [--dir DIR] Verify registry bundle signature and integrity + healthcheck Verify container process liveness + version Print version and runtime information + HELP + end + + ## + # @param out [IO] + # @return [void] + def print_registry_help(out:) + out.puts <<~HELP + Usage: html2rss-web registry [subcommand] [options] + + Subcommands: + status Show registry sync status table + sync Fetch and verify remote registry bundle(s) + promote Promote staged verified bundle to active + verify Verify bundle signature and file integrity against pinned configuration + HELP + end + + ## + # @param registry_id [String, nil] + # @return [Array] + def target_registry_ids(registry_id) + return Array(registry_id) if registry_id + + Registry::Config.precedence + end + + ## + # @param registry_id [String, nil] + # @param out [IO] + # @return [void] + def print_status_table(registry_id, out:) # rubocop:disable Metrics/MethodLength + rows = Registry::Sync.status(registry_id:) + out.puts STATUS_HEADERS.join("\t") + rows.each do |row| + out.puts [ + row.id, + row.mode, + row.version || '-', + row.staged_version || '-', + format_time(row.updated_at), + row.sync_url || '-', + row.last_error || '-' + ].join("\t") + end + end + + ## + # @param value [Time, nil] + # @return [String] + def format_time(value) = value ? value.utc.iso8601 : '-' + + private_class_method :run_registry, :registry_status, :registry_sync, :registry_promote, + :registry_verify, :run_healthcheck, :run_version, :print_root_help, + :print_registry_help, :target_registry_ids, :print_status_table, :format_time + end + end +end diff --git a/app/web/config/config_snapshot.rb b/app/web/config/config_snapshot.rb index 0a8fc03c5..91e42b708 100644 --- a/app/web/config/config_snapshot.rb +++ b/app/web/config/config_snapshot.rb @@ -47,7 +47,8 @@ def normalize_feeds(raw_feeds) return {} unless raw_feeds.is_a?(Hash) raw_feeds.each_with_object({}) do |(name, config), memo| - memo[name.to_sym] = FeedConfig.new(name: name.to_sym, raw: deep_dup(config).freeze) + sym_name = name.to_sym + memo[sym_name] = FeedConfig.new(name: sym_name, raw: StructuredData.deep_dup(config).freeze) end end @@ -68,7 +69,7 @@ def normalize_accounts(raw_accounts) # @param accounts [Array] # @return [Hash{Symbol=>Object}] def normalized_global_hash(global_hash, accounts) - normalized = deep_dup(global_hash) + normalized = StructuredData.deep_dup(global_hash) return normalized unless normalized.key?(:auth) normalized[:auth] = normalized_auth_hash(normalized[:auth], accounts) @@ -85,35 +86,6 @@ def normalized_auth_hash(auth_hash, accounts) end auth end - - # @param value [Object] - # @return [Object] - def deep_dup(value) - case value - when Hash - deep_dup_hash(value) - when Array - deep_dup_array(value) - when String - value.dup - else - value - end - end - - # @param value [Hash] - # @return [Hash] - def deep_dup_hash(value) - value.each_with_object({}) do |(key, val), memo| - memo[key.is_a?(String) ? key.dup : key] = deep_dup(val) - end - end - - # @param value [Array] - # @return [Array] - def deep_dup_array(value) - value.map { |element| deep_dup(element) } - end end end end diff --git a/app/web/config/local_config.rb b/app/web/config/local_config.rb index fcf4f98d3..7326a052f 100644 --- a/app/web/config/local_config.rb +++ b/app/web/config/local_config.rb @@ -3,20 +3,12 @@ require 'erb' require 'yaml' require_relative 'runtime_env' -begin - require 'html2rss/configs' -rescue LoadError => error - warn "[html2rss-web] Failed to load 'html2rss/configs': #{error.message}" - raise -end +require_relative 'structured_data' module Html2rss module Web ## - # Loads and normalizes feed configuration from disk. - # - # Keeping lookup/defaulting here gives the rest of the app one predictable - # config shape instead of repeating file parsing and fallback logic. + # Loads and normalizes local feed configuration from disk. module LocalConfig @mutex = Mutex.new @snapshot = nil @@ -28,33 +20,32 @@ class NotFound < RuntimeError; end # raised when the local config shape is invalid class InvalidConfig < RuntimeError; end FEED_EXTENSION_PATTERN = /\.(json|rss|xml)\z/ - EMBEDDED_FEED_NAME_PATTERN = %r{\A[^/]+/.+\z} # Path to local feed configuration file. CONFIG_FILE = 'config/feeds.yml' class << self ## - # @param name [String, Symbol, #to_sym] - # @return [Hash] - def find(name) - normalized_name = normalize_name(name) - config_hash = local_feed_config(normalized_name) || embedded_feed_config(normalized_name) - raise NotFound, "Did not find local feed config at '#{normalized_name}'" unless config_hash - - config_hash + # @param feed_id [String, Symbol] + # @return [Hash{Symbol => Object}] + def find(feed_id) + normalized = feed_id.to_s.delete_prefix('/').sub(FEED_EXTENSION_PATTERN, '') + config = snapshot.feeds[normalized.to_sym] + raise NotFound, "Did not find local feed config at '#{normalized}'" unless config + + StructuredData.deep_dup(config.raw) end ## - # @return [Hash] + # @return [Hash{Symbol => Hash{Symbol => Object}}] def feeds - snapshot.feeds.transform_values { |feed| deep_dup(feed.raw) } + snapshot.feeds.transform_values { StructuredData.deep_dup(it.raw) } end ## - # @return [Hash] + # @return [Hash{Symbol => Object}] def global - deep_dup(snapshot.global) + StructuredData.deep_dup(snapshot.global) end ## @@ -66,11 +57,7 @@ def snapshot end ## - # Reparses the current config file without touching memoized runtime - # state. Health checks use this path so config drift shows up without - # forcing live request handlers onto a reload path. - # - # @return [Hash] + # @return [Hash{Symbol => Object}] def load_yaml template = File.read(CONFIG_FILE) YAML.safe_load(ERB.new(template, trim_mode: '-').result, symbolize_names: true).freeze @@ -79,9 +66,6 @@ def load_yaml end ## - # Reparses and normalizes the current config file without mutating the - # memoized runtime snapshot. - # # @return [Html2rss::Web::ConfigSnapshot::Snapshot] def load_snapshot ConfigSnapshot.load(load_yaml) @@ -94,6 +78,7 @@ def load_snapshot # @return [nil] def reload!(reason: 'manual') @mutex.synchronize { @snapshot = nil } + Registry::Index.reload! Observability.emit( event_name: 'cache.lifecycle', outcome: 'success', @@ -101,49 +86,6 @@ def reload!(reason: 'manual') ) nil end - - private - - # @param normalized_name [String] - # @return [Hash{Symbol=>Object}, nil] - def local_feed_config(normalized_name) - config = snapshot.feeds[normalized_name.to_sym] - return nil unless config - - deep_dup(config.raw) - end - - # @param normalized_name [String] - # @return [Hash{Symbol=>Object}, nil] - def embedded_feed_config(normalized_name) - return nil unless defined?(Html2rss::Configs) - return nil unless normalized_name.match?(EMBEDDED_FEED_NAME_PATTERN) - - deep_dup(Html2rss::Configs.find_by_name(normalized_name)) - rescue Html2rss::Configs::ConfigNotFound - nil - end - - # @param name [String, Symbol, #to_s] - # @return [String] path without feed extension for feed lookup. - def normalize_name(name) - name.to_s.delete_prefix('/').sub(FEED_EXTENSION_PATTERN, '') - end - - # Deep-duplicates nested config structures to avoid mutating shared data. - # - # @param value [Object] - # @return [Object] - def deep_dup(value) - case value - when Hash - value.transform_values { |val| deep_dup(val) } - when Array - value.map { |element| deep_dup(element) } - else - value - end - end end end end diff --git a/app/web/config/structured_data.rb b/app/web/config/structured_data.rb new file mode 100644 index 000000000..d829d6d5c --- /dev/null +++ b/app/web/config/structured_data.rb @@ -0,0 +1,29 @@ +# frozen_string_literal: true + +module Html2rss + module Web + ## + # Shared helpers for cloning nested Hash/Array config structures. + module StructuredData + module_function + + ## + # @param value [Object] + # @return [Object] + def deep_dup(value) # rubocop:disable Metrics/MethodLength + case value + when Hash + value.each_with_object({}) do |(key, val), memo| + memo[key.is_a?(String) ? key.dup : key] = deep_dup(val) + end + when Array + value.map { deep_dup(it) } + when String + value.dup + else + value + end + end + end + end +end diff --git a/app/web/feeds/source_resolver.rb b/app/web/feeds/source_resolver.rb index a90ae65ee..d15046b42 100644 --- a/app/web/feeds/source_resolver.rb +++ b/app/web/feeds/source_resolver.rb @@ -12,13 +12,14 @@ class << self # @param feed_request [Html2rss::Web::Feeds::Contracts::Request] # @return [Html2rss::Web::Feeds::Contracts::ResolvedSource] def call(feed_request) - case feed_request.target_kind + target_kind = feed_request.target_kind + case target_kind when :static resolve_static(feed_request) when :token resolve_token(feed_request) else - raise Html2rss::Web::BadRequestError, "Unsupported feed target: #{feed_request.target_kind}" + raise Html2rss::Web::BadRequestError, "Unsupported feed target: #{target_kind}" end end @@ -27,31 +28,31 @@ def call(feed_request) # @param feed_request [Html2rss::Web::Feeds::Contracts::Request] # @return [Html2rss::Web::Feeds::Contracts::ResolvedSource] def resolve_static(feed_request) - config = LocalConfig.find(feed_request.feed_name) - generator_input = static_generator_input(config, feed_request.params) + feed_name = feed_request.feed_name + params = feed_request.params + config = Registry::Index.current.config_for(feed_name) || raise(Html2rss::Web::NotFoundError) + generator_input = static_generator_input(config, params) resolved_source_for( source_kind: :static, - cache_identity: static_cache_identity(feed_request.feed_name, feed_request.params), - generator_input: generator_input, + cache_identity: static_cache_identity(feed_name, params), + generator_input:, ttl_seconds: Cache.seconds_from_minutes(generator_input.dig(:channel, :ttl)) ) - rescue Html2rss::Web::LocalConfig::NotFound - raise Html2rss::Web::NotFoundError end # @param feed_request [Html2rss::Web::Feeds::Contracts::Request] # @return [Html2rss::Web::Feeds::Contracts::ResolvedSource] def resolve_token(feed_request) ensure_auto_source_enabled! - feed_token = authorize_feed_token!(feed_request.token) - strategy = resolved_strategy(feed_token) - generator_input = token_generator_input(feed_token.url, strategy) + token = feed_request.token + feed_token = authorize_feed_token!(token) + generator_input = token_generator_input(feed_token.url, resolved_strategy(feed_token)) resolved_source_for( source_kind: :token, - cache_identity: token_cache_identity(feed_request.token), - generator_input: generator_input, + cache_identity: token_cache_identity(token), + generator_input:, ttl_seconds: Cache.seconds_from_minutes(generator_input.dig(:channel, :ttl), default: 300) ) end diff --git a/app/web/registry/channel_resolver.rb b/app/web/registry/channel_resolver.rb new file mode 100644 index 000000000..93d6b0dc4 --- /dev/null +++ b/app/web/registry/channel_resolver.rb @@ -0,0 +1,52 @@ +# frozen_string_literal: true + +require 'json' + +module Html2rss + module Web + module Registry + ## + # Resolves download URLs for official registry channels and GitHub release tags. + module ChannelResolver + OFFICIAL_GITHUB_RELEASES_API = 'https://api.github.com/repos/html2rss/html2rss-configs/releases/latest' + OFFICIAL_GITHUB_TAG_RELEASES_API = 'https://api.github.com/repos/html2rss/html2rss-configs/releases/tags/%s' + OFFICIAL_ASSET_NAME = 'registry-bundle.tar.gz' + + module_function + + ## + # @param definition [Definition] + # @return [String] download URL + def resolve(definition) + sync_url = definition.sync_url + return sync_url if sync_url && !sync_url.empty? + + if definition.sync_channel == Config::DEFAULT_OFFICIAL_SYNC_CHANNEL + pin = definition.sync_policy.pin_version + return Config::OFFICIAL_RELEASE_URL if pin.to_s.empty? + + api_url = format(OFFICIAL_GITHUB_TAG_RELEASES_API, tag: pin) + return resolve_github_asset(api_url, tag: pin) + end + + raise Errors::SyncError, "Registry '#{definition.id}' has no sync URL" + end + + ## + # @param api_url [String] + # @param tag [String, nil] + # @return [String] + def resolve_github_asset(api_url, tag: nil) + release = JSON.parse(HttpTransport.fetch!(api_url), symbolize_names: true) + url = Array(release[:assets]).find { it[:name] == OFFICIAL_ASSET_NAME }&.dig(:browser_download_url) + return url if url + + tag_suffix = " for #{tag}" if tag + raise Errors::SyncError, "Official asset '#{OFFICIAL_ASSET_NAME}' not found#{tag_suffix}" + rescue JSON::ParserError => error + raise Errors::SyncError, "Invalid GitHub release metadata: #{error.message}" + end + end + end + end +end diff --git a/app/web/registry/config.rb b/app/web/registry/config.rb new file mode 100644 index 000000000..1854faf23 --- /dev/null +++ b/app/web/registry/config.rb @@ -0,0 +1,196 @@ +# frozen_string_literal: true + +require 'yaml' +require 'openssl' + +module Html2rss + module Web + module Registry + ## + # Sync policy parsed from registry YAML. + SyncPolicy = Data.define(:pin_version, :max_version, :auto_promote) + + ## + # Registry definition parsed from {Config::REGISTRIES_FILE}. + Definition = Data.define( + :id, + :mode, + :path, + :sync_channel, + :sync_url, + :catalog, + :public_key_id, + :public_key, + :sync_policy, + :allowed_channel_domains + ) do + ## + # @return [Hash{String => OpenSSL::PKey::PKey}] + def public_keys + public_key ? { public_key_id => public_key } : {} + end + end + + ## + # Parses registry configuration and applies zero-config defaults. + module Config # rubocop:disable Metrics/ModuleLength + REGISTRIES_FILE = 'config/registries.yml' + DEFAULT_PRECEDENCE = %w[official].freeze + DEFAULT_OFFICIAL_SYNC_CHANNEL = 'html2rss-official' + OFFICIAL_RELEASE_URL = 'https://github.com/html2rss/html2rss-configs/releases/latest/download/registry-bundle.tar.gz' + DEFAULT_PUBLIC_KEY_PEM = <<~PEM + -----BEGIN PUBLIC KEY----- + MCowBQYDK2VwAyEAiMbg/04MyC5azBdM/aeY0mNuA8JbP5/jOiNRwJ2KJHE= + -----END PUBLIC KEY----- + PEM + + ## + # Registry configuration document containing precedence order and registry definitions. + Document = Data.define(:precedence, :entries) + + @mutex = Mutex.new + @current = nil + + class << self + ## + # @return [Array] + def precedence + current.precedence + end + + ## + # @param registry_id [String, Symbol] + # @return [Definition] + def entry(registry_id) + current.entries.fetch(registry_id.to_s) do + raise Errors::UnknownRegistry, "Unknown registry '#{registry_id}'" + end + end + + ## + # @param registry_id [String, Symbol] + # @return [Boolean] + def catalog_enabled?(registry_id) + entry(registry_id).catalog + end + + ## + # @return [Document] + def current + @mutex.synchronize { @current ||= parse_document } + end + + ## + # @return [nil] + def reload! + @mutex.synchronize { @current = nil } + nil + end + + private + + ## + # @return [Document] + def parse_document + raw_doc = load_yaml + precedence = parse_precedence(raw_doc[:precedence]) + registries = raw_doc[:registries] || {} + + entries = precedence.to_h do |id| + raw = registries[id.to_sym] || registries[id] || {} + [id, build_definition(id, raw)] + end + + Document.new(precedence:, entries:) + end + + def parse_precedence(raw_precedence) + list = Array(raw_precedence).map(&:to_s).reject(&:empty?) + list.empty? ? DEFAULT_PRECEDENCE : list + end + + ## + # @return [Hash{Symbol => Object}] + def load_yaml + path = ENV.fetch('REGISTRIES_CONFIG', REGISTRIES_FILE) + return default_document unless File.file?(path) + + YAML.safe_load_file(path, symbolize_names: true) || {} + rescue Psych::SyntaxError => error + raise Errors::ConfigError, "Invalid #{path}: #{error.message}" + end + + ## + # @return [Hash{Symbol => Object}] + def default_document # rubocop:disable Metrics/MethodLength + { + precedence: DEFAULT_PRECEDENCE, + registries: { + 'official' => { + sync: { channel: DEFAULT_OFFICIAL_SYNC_CHANNEL }, + catalog: true, + public_key_id: 'html2rss:registry:2026', + public_key: DEFAULT_PUBLIC_KEY_PEM + } + } + } + end + + ## + # @param id [String] + # @param raw [Hash{Symbol => Object}] + # @return [Definition] + def build_definition(id, raw) # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity + sync_raw = raw[:sync] + sync = sync_raw.is_a?(Hash) ? sync_raw : {} + path = raw[:path]&.to_s + mode = path && !path.empty? ? :path : :sync + public_key = parse_public_key(raw[:public_key]&.to_s) + + definition = Definition.new( + id:, + mode:, + path: mode == :path ? File.expand_path(path, Dir.pwd) : nil, + sync_channel: sync[:channel]&.to_s || DEFAULT_OFFICIAL_SYNC_CHANNEL, + sync_url: sync[:url]&.to_s, + catalog: raw.fetch(:catalog, true), + public_key_id: raw[:public_key_id]&.to_s || 'html2rss:registry:2026', + public_key:, + sync_policy: build_sync_policy(sync, raw), + allowed_channel_domains: Array(raw[:allowed_channel_domains]).map(&:to_s).reject(&:empty?) + ) + + if definition.mode == :sync && !definition.public_key + raise Errors::ConfigError, "Sync registry '#{id}' requires a pinned public_key" + end + + definition + end + + ## + # @param sync [Hash{Symbol => Object}] + # @param raw [Hash{Symbol => Object}] + # @return [SyncPolicy] + def build_sync_policy(sync, raw) + SyncPolicy.new( + pin_version: sync[:pin_version]&.to_s, + max_version: sync[:max_version]&.to_s, + auto_promote: raw[:auto_promote] == true + ) + end + + ## + # @param key_pem [String, nil] + # @return [OpenSSL::PKey::PKey, nil] + def parse_public_key(key_pem) + return nil if key_pem.to_s.strip.empty? + + OpenSSL::PKey.read(key_pem) + rescue OpenSSL::PKey::PKeyError => error + raise Errors::ConfigError, "Invalid registry public_key: #{error.message}" + end + end + end + end + end +end diff --git a/app/web/registry/errors.rb b/app/web/registry/errors.rb new file mode 100644 index 000000000..bb917c753 --- /dev/null +++ b/app/web/registry/errors.rb @@ -0,0 +1,26 @@ +# frozen_string_literal: true + +module Html2rss + module Web + module Registry + ## + # Actionable registry runtime errors for operators and API handlers. + module Errors + # Base error for web registry operations. + class Error < StandardError; end + + # Raised when registry configuration is invalid or incomplete. + class ConfigError < Error; end + + # Raised when a registry bundle cannot be loaded. + class LoadError < Error; end + + # Raised when registry synchronization fails. + class SyncError < Error; end + + # Raised when a registry id is unknown. + class UnknownRegistry < Error; end + end + end + end +end diff --git a/app/web/registry/http_transport.rb b/app/web/registry/http_transport.rb new file mode 100644 index 000000000..392cc8d14 --- /dev/null +++ b/app/web/registry/http_transport.rb @@ -0,0 +1,124 @@ +# frozen_string_literal: true + +require 'net/http' +require 'uri' + +module Html2rss + module Web + module Registry + ## + # HTTPS client for fetching remote registry bundles. + module HttpTransport + DEFAULT_ALLOWED_HOSTS = %w[ + api.github.com + github.com + objects.githubusercontent.com + release-assets.githubusercontent.com + ].freeze + OPEN_TIMEOUT_SECONDS = 10 + READ_TIMEOUT_SECONDS = 60 + MAX_RESPONSE_BYTES = 52_428_800 + DEFAULT_MAX_REDIRECTS = 5 + + module_function + + ## + # @param url [String] + # @param max_redirects [Integer] + # @param allowed_hosts [Array] + # @return [String] response body + def fetch!(url, max_redirects: DEFAULT_MAX_REDIRECTS, allowed_hosts: default_allowed_hosts) # rubocop:disable Metrics/MethodLength + uri = parse_https_uri!(url) + hops = 0 + + loop do + ensure_allowed_host!(uri.host, allowed_hosts) + response = perform_request(uri) + + case response + in Net::HTTPRedirection + hops += 1 + raise Errors::SyncError, 'Registry sync exceeded redirect limit' if hops > max_redirects + + uri = handle_redirect(uri, response['location']) + in Net::HTTPSuccess + return read_body(response) + else + raise Errors::SyncError, "Registry sync fetch failed with HTTP #{response.code}" + end + end + end + + ## + # @param current_uri [URI::HTTPS] + # @param location [String, nil] + # @return [URI::HTTPS] + def handle_redirect(current_uri, location) + raise Errors::SyncError, 'Registry sync redirect missing Location header' if location.to_s.empty? + + parse_https_uri!(URI.join(current_uri, location).to_s) + end + + ## + # @return [Array] + def default_allowed_hosts + extra = ENV.fetch('REGISTRY_SYNC_ALLOWED_HOSTS', '').split(',').map(&:strip).reject(&:empty?) + DEFAULT_ALLOWED_HOSTS + extra + end + + ## + # @param url [String] + # @return [URI::HTTPS] + def parse_https_uri!(url) + uri = URI(url) + unless uri.is_a?(URI::HTTPS) + raise Errors::SyncError, "Registry sync requires HTTPS URLs (got #{uri.scheme.inspect})" + end + + uri + end + + ## + # @param host [String] + # @param allowed_hosts [Array] + # @return [void] + def ensure_allowed_host!(host, allowed_hosts) + return if allowed_hosts.include?(host) + + raise Errors::SyncError, "Registry sync host not allowed: #{host}" + end + + ## + # @param uri [URI::HTTPS] + # @return [Net::HTTPResponse] + def perform_request(uri) # rubocop:disable Metrics/MethodLength + Net::HTTP.start( + uri.host, + uri.port, + use_ssl: true, + open_timeout: OPEN_TIMEOUT_SECONDS, + read_timeout: READ_TIMEOUT_SECONDS + ) do |http| + request = Net::HTTP::Get.new(uri) + request['Accept'] = 'application/octet-stream' + request['User-Agent'] = 'html2rss-web/registry-sync' + http.request(request) + end + end + + ## + # @param response [Net::HTTPResponse] + # @return [String] + def read_body(response) + body = response.body.to_s + if body.bytesize > MAX_RESPONSE_BYTES + raise Errors::SyncError, + "Response exceeds max bytes (#{MAX_RESPONSE_BYTES})" + end + + body + end + end + end + end +end diff --git a/app/web/registry/index.rb b/app/web/registry/index.rb new file mode 100644 index 000000000..c442c802f --- /dev/null +++ b/app/web/registry/index.rb @@ -0,0 +1,271 @@ +# frozen_string_literal: true + +require 'uri' +require_relative '../config/structured_data' + +module Html2rss + module Web + module Registry + ## + # Unified status model for registry state across API, CLI, and index. + Status = Data.define( + :id, + :mode, + :version, + :staged_version, + :updated_at, + :sync_url, + :last_error + ) do + ## + # @param id [String, Symbol] + # @param mode [Symbol] + # @param version [String, nil] + # @param staged_version [String, nil] + # @param updated_at [Time, nil] + # @param sync_url [String, nil] + # @param last_error [String, nil] + def initialize(id:, mode:, version: nil, staged_version: nil, updated_at: nil, sync_url: nil, last_error: nil) + super + end + end + + ## + # Sole feed repository combining registry bundles and local feeds. + class Index # rubocop:disable Metrics/ClassLength + ## + # Immutable container holding a loaded registry bundle's manifest, configs, and catalog entries. + RegistryBundle = Data.define(:registry_id, :manifest, :configs, :catalog_entries) + + @mutex = Mutex.new + @current = nil + + class << self + ## + # @return [Index] + def current + @mutex.synchronize do + if @current.nil? || @current.stale? + @current = new + else + @current + end + end + end + + ## + # @return [nil] + def reload! + @mutex.synchronize { @current = nil } + Config.reload! + nil + end + end + + def initialize + @loaded_bundles = Config.precedence.to_h { [it, load_bundle(it)] }.compact + @manifest_mtimes = current_manifest_mtimes + end + + ## + # @return [Boolean] + def stale? + @manifest_mtimes != current_manifest_mtimes + end + + ## + # @param feed_id [String, Symbol] + # @return [Hash{Symbol => Object}, nil] + def config_for(feed_id) + id = normalize_feed_id(feed_id) + local = local_config_for(id) + return local if local + + loaded_bundles.each_value do |bundle| + config = bundle.configs[id] + return StructuredData.deep_dup(config) if config + end + + nil + end + + ## + # @return [Array] + def catalog_entries + registry_entries = build_registry_catalog_entries + local_entries = build_local_catalog_entries + registry_entries.merge(local_entries).values.sort_by(&:id) + end + + ## + # Wire rows for the catalog API (+source+ / +registry+ stamped here). + # + # @return [Array Object}>] + def catalog_rows + registry_rows = build_registry_catalog_rows + local_rows = build_local_catalog_rows + registry_rows.merge(local_rows).values.sort_by { it.fetch(:id) } + end + + ## + # @param registry_id [String, Symbol] + # @return [RegistryBundle, nil] + def bundle_for(registry_id) + loaded_bundles[registry_id.to_s] + end + + ## + # @return [Array] + def status # rubocop:disable Metrics/MethodLength + Config.precedence.map do |registry_id| + definition = Config.entry(registry_id) + bundle = loaded_bundles[registry_id] + sync_state = Store.sync_state(registry_id) + mode = definition.mode + sync_url = mode == :sync ? ChannelResolver.resolve(definition) : nil + + Status.new( + id: registry_id, + mode:, + version: bundle&.manifest&.version, + staged_version: Store.staged_version(registry_id), + updated_at: Store.manifest_mtime(bundle_directory(definition)), + sync_url:, + last_error: sync_state.last_error + ) + end + end + + ## + # @param registry_id [String, Symbol] + # @return [Status, nil] + def status_entry_for(registry_id) + status.find { it.id == registry_id.to_s } + end + + private + + attr_reader :loaded_bundles, :manifest_mtimes + + def current_manifest_mtimes + Config.precedence.to_h do |registry_id| + definition = Config.entry(registry_id) + directory = bundle_directory(definition) + [registry_id, Store.manifest_mtime(directory)] + end + end + + def build_registry_catalog_entries + merge_registry_catalog { |entry, _registry_id| entry } + end + + def build_registry_catalog_rows + merge_registry_catalog do |entry, registry_id| + entry.to_h.merge(source: 'registry', registry: registry_id) + end + end + + def merge_registry_catalog + Config.precedence.each_with_object({}) do |registry_id, rows| + next unless Config.catalog_enabled?(registry_id) + + bundle = loaded_bundles[registry_id] + next unless bundle + + bundle.catalog_entries.each do |entry| + rows[entry.id] ||= yield(entry, registry_id) + end + end + end + + def build_local_catalog_entries + LocalConfig.feeds.filter_map do |feed_name, feed_config| + entry = build_local_catalog_entry(feed_name, feed_config) + [entry.id, entry] if entry + end.to_h + end + + def build_local_catalog_rows + build_local_catalog_entries.transform_values { |entry| entry.to_h.merge(source: 'local') } + end + + def load_bundle(registry_id) # rubocop:disable Metrics/MethodLength + definition = Config.entry(registry_id) + directory = bundle_directory(definition) + return nil unless directory && File.directory?(directory) && Store.bundle_present_at?(directory) + + trust_opts = if definition.mode == :path + { trust: :integrity_only } + else + { trust: :signed, public_keys: definition.public_keys } + end + bundle_data = Html2rss::Registry::Bundle.load(directory, **trust_opts) + bundle = RegistryBundle.new( + registry_id:, + manifest: bundle_data.manifest, + configs: bundle_data.configs, + catalog_entries: bundle_data.catalog_entries + ) + enforce_scrape_policy!(definition, bundle) + bundle + rescue Html2rss::Registry::Error => error + raise Errors::LoadError, "Failed to load registry '#{registry_id}': #{error.message}" + end + + def enforce_scrape_policy!(definition, bundle) + allowed = definition.allowed_channel_domains + return if allowed.empty? + + bundle.configs.each do |feed_id, config| + next if host_allowed?(config.dig(:channel, :url), allowed) + + raise Errors::LoadError, "Registry '#{definition.id}' config '#{feed_id}' host not allowed" + end + end + + def host_allowed?(url, allowed) + host = URI.parse(url.to_s).host&.downcase rescue nil # rubocop:disable Style/RescueModifier + return false unless host + + allowed.any? { |domain| domain_matches?(host, domain.downcase) } + end + + def domain_matches?(host, domain) + host == domain || host.end_with?(".#{domain}") + end + + def bundle_directory(definition) + definition.mode == :path ? definition.path : Store.active_dir(definition.id) + end + + def normalize_feed_id(feed_id) + feed_id.to_s.delete_prefix('/').sub(LocalConfig::FEED_EXTENSION_PATTERN, '') + end + + def local_config_for(feed_id) + feeds = LocalConfig.feeds + feed = feeds[feed_id.to_sym] || feeds[feed_id] + feed ? StructuredData.deep_dup(feed) : nil + rescue StandardError + nil + end + + def build_local_catalog_entry(feed_name, feed_config) # rubocop:disable Metrics/MethodLength + directory = feed_config[:directory] || {} + title = directory[:title]&.to_s + return nil if title.to_s.strip.empty? + + id = feed_name.to_s + channel = feed_config[:channel] || {} + Html2rss::Registry::CatalogEntry.new( + id:, + path: "/#{id}.rss", + directory: Html2rss::Registry::CatalogBuilder.directory_payload(directory, title), + channel: Html2rss::Registry::CatalogBuilder.channel_payload(channel, title), + parameters: { schema: {}, defaults: {} } + ) + end + end + end + end +end diff --git a/app/web/registry/store.rb b/app/web/registry/store.rb new file mode 100644 index 000000000..1a181a890 --- /dev/null +++ b/app/web/registry/store.rb @@ -0,0 +1,232 @@ +# frozen_string_literal: true + +require 'fileutils' +require 'json' + +module Html2rss + module Web + module Registry + ## + # Transactional storage engine for registry bundles and sync state. + module Store # rubocop:disable Metrics/ModuleLength + DEFAULT_DATA_ROOT = 'tmp/registry-data' + DEFAULT_EMBEDDED_ROOT = '/app/registries' + SYNC_STATE_FILE = '.sync-state.json' + + ## + # Immutable registry sync state. + SyncState = Data.define(:last_error, :last_sync_at) + + class << self # rubocop:disable Metrics/ClassLength + ## + # @return [String] + def data_root + File.expand_path(ENV.fetch('REGISTRY_DATA_ROOT', DEFAULT_DATA_ROOT)) + end + + ## + # @return [String] + def embedded_root + File.expand_path(ENV.fetch('REGISTRY_EMBEDDED_ROOT', DEFAULT_EMBEDDED_ROOT)) + end + + ## + # @param registry_id [String, Symbol] + # @return [String] + def registry_dir(registry_id) + File.join(data_root, registry_id.to_s) + end + + ## + # @param registry_id [String, Symbol] + # @return [String] + def embedded_dir(registry_id) + File.join(embedded_root, registry_id.to_s) + end + + ## + # Resolves the active bundle directory, preferring synced runtime data + # and falling back to the baked image bundle. + # + # @param registry_id [String, Symbol] + # @return [String, nil] + def active_dir(registry_id) + synced = registry_dir(registry_id) + return synced if bundle_present_at?(synced) + + embedded = embedded_dir(registry_id) + return embedded if bundle_present_at?(embedded) + + nil + end + + ## + # @param registry_id [String, Symbol] + # @return [String] + def staging_dir(registry_id) + File.join(registry_dir(registry_id), '.staging') + end + + ## + # @param registry_id [String, Symbol] + # @return [Boolean] + def bundle_present?(registry_id) + bundle_present_at?(registry_dir(registry_id)) || bundle_present_at?(embedded_dir(registry_id)) + end + + ## + # @param path [String, nil] + # @return [Boolean] + def bundle_present_at?(path) + return false unless path && File.directory?(path) + + File.file?(File.join(path, Html2rss::Registry::Manifest::MANIFEST_FILE)) + end + + ## + # @param registry_id [String, Symbol] + # @param staged_dir [String] + # @return [String] + def stage_bundle!(registry_id, staged_dir) + target = staging_dir(registry_id) + FileUtils.mkdir_p(registry_dir(registry_id)) + FileUtils.rm_rf(target) + FileUtils.cp_r(staged_dir, target) + target + end + + ## + # @param registry_id [String, Symbol] + # @return [Boolean] + def staged_present?(registry_id) + bundle_present_at?(staging_dir(registry_id)) + end + + ## + # @param registry_id [String, Symbol] + # @return [String, nil] + def staged_version(registry_id) + manifest_version_at(staging_dir(registry_id)) + end + + ## + # Atomically swaps a directory into the active registry directory. + # + # @param registry_id [String, Symbol] + # @param source_dir [String] + # @return [String] active directory + def swap!(registry_id, source_dir) + target = registry_dir(registry_id) + FileUtils.mkdir_p(File.dirname(target)) + + temp_target = staging_swap_dir(registry_id) + FileUtils.rm_rf(temp_target) + FileUtils.cp_r(source_dir, temp_target) + + replace_directory!(temp_target, target) + target + ensure + FileUtils.rm_rf(temp_target) + end + + ## + # Promotes staging directory to active directory. + # + # @param registry_id [String, Symbol] + # @return [String] active directory + def promote_staged!(registry_id) + staged = staging_dir(registry_id) + raise Errors::LoadError, "No staged bundle for '#{registry_id}'" unless staged_present?(registry_id) + + temp_root = Dir.mktmpdir('registry-promote-') + temp_staged = File.join(temp_root, 'bundle') + FileUtils.mv(staged, temp_staged) + + swap!(registry_id, temp_staged) + registry_dir(registry_id) + ensure + FileUtils.rm_rf(temp_root) if temp_root + end + + ## + # @param registry_id [String, Symbol] + # @return [SyncState] + def sync_state(registry_id) + raw = read_sync_state.fetch(registry_id.to_s, {}) + last_sync = raw['last_sync_at'] + SyncState.new( + last_error: raw['last_error'], + last_sync_at: last_sync ? Time.parse(last_sync) : nil + ) + end + + ## + # @param registry_id [String, Symbol] + # @param last_error [String, nil] + # @param last_sync_at [Time, nil] + # @return [void] + def write_sync_state!(registry_id, last_error:, last_sync_at: Time.now.utc) + state = read_sync_state + state[registry_id.to_s] = { + 'last_error' => last_error, + 'last_sync_at' => last_sync_at&.iso8601 + }.compact + FileUtils.mkdir_p(data_root) + File.write(sync_state_path, JSON.generate(state)) + end + + ## + # @param path [String, nil] + # @return [Time, nil] + def manifest_mtime(path) + return nil unless path && File.directory?(path) + + manifest_file = File.join(path, Html2rss::Registry::Manifest::MANIFEST_FILE) + File.file?(manifest_file) ? File.mtime(manifest_file) : nil + end + + private + + def sync_state_path + File.join(data_root, SYNC_STATE_FILE) + end + + def read_sync_state + return {} unless File.file?(sync_state_path) + + JSON.parse(File.read(sync_state_path)) + rescue JSON::ParserError + {} + end + + def manifest_version_at(path) + manifest_file = File.join(path, Html2rss::Registry::Manifest::MANIFEST_FILE) + return nil unless File.file?(manifest_file) + + Html2rss::Registry::Manifest.parse(File.read(manifest_file)).version + rescue Html2rss::Registry::ManifestError + nil + end + + def replace_directory!(source, target) + backup = "#{target}.backup.#{Process.pid}" + FileUtils.mv(target, backup) if File.exist?(target) + begin + File.rename(source, target) + rescue StandardError + FileUtils.mv(backup, target) if File.exist?(backup) && !File.exist?(target) + raise + ensure + FileUtils.rm_rf(backup) + end + end + + def staging_swap_dir(registry_id) + stamp = Process.clock_gettime(Process::CLOCK_MONOTONIC_RAW, :nanosecond) + File.join(data_root, ".tmp-swap-#{registry_id}-#{Process.pid}-#{stamp}") + end + end + end + end + end +end diff --git a/app/web/registry/sync.rb b/app/web/registry/sync.rb new file mode 100644 index 000000000..af16fcb84 --- /dev/null +++ b/app/web/registry/sync.rb @@ -0,0 +1,205 @@ +# frozen_string_literal: true + +require 'fileutils' + +module Html2rss + module Web + module Registry + ## + # Fetches, verifies, and stores signed registry bundles. + module Sync # rubocop:disable Metrics/ModuleLength + @boot_mutex = Mutex.new + @boot_started = false + @timer_started = false + REGISTRY_MUTEXES = Hash.new { |mutexes, registry_id| mutexes[registry_id] = Mutex.new } + + class << self # rubocop:disable Metrics/ClassLength + ## + # @param registry_id [String, Symbol] + # @return [String] + def sync_url_for(registry_id) + definition = Config.entry(registry_id) + raise Errors::SyncError, "Registry '#{registry_id}' is path mode" if definition.mode == :path + + ChannelResolver.resolve(definition) + end + + ## + # @param registry_id [String, Symbol] + # @param dry_run [Boolean] + # @return [Status] + def run(registry_id:, dry_run: false) + REGISTRY_MUTEXES[registry_id.to_s].synchronize { run_sync!(registry_id:, dry_run:) } + end + + ## + # @param registry_id [String, Symbol] + # @return [Status] + def promote_staged!(registry_id:) # rubocop:disable Metrics/MethodLength + REGISTRY_MUTEXES[registry_id.to_s].synchronize do + definition = Config.entry(registry_id) + raise Errors::SyncError, "Registry '#{registry_id}' is path mode" if definition.mode == :path + unless Store.staged_present?(registry_id) + raise Errors::SyncError, "Registry '#{registry_id}' has no staged bundle" + end + + previous_bundle = Index.current.bundle_for(registry_id) + Store.promote_staged!(registry_id) + Index.reload! + report_catalog_change!(registry_id, previous_bundle) + Store.write_sync_state!(registry_id, last_error: nil) + Index.current.status_entry_for(registry_id) + end + end + + ## + # @param registry_id [String, Symbol] + # @return [Array] + def status(registry_id: nil) + rows = Index.current.status + registry_id ? rows.select { it.id == registry_id.to_s } : rows + end + + ## + # Returns process exit code for CLI invocations: 0 on success, 1 on failure. + # + # @return [Integer] + def cli_exit_code + unusable = Index.current.status.any? { it.last_error || (it.mode == :sync && it.version.nil?) } + unusable ? 1 : 0 + end + + ## + # @return [void] + def boot! # rubocop:disable Metrics/MethodLength + @boot_mutex.synchronize do + return if @boot_started + + @boot_started = true + end + return if ENV.fetch('RACK_ENV', 'development') == 'test' + + if ENV.fetch('REGISTRY_SYNC_ON_BOOT', 'false') == 'true' + Config.precedence.each do |id| + definition = Config.entry(id) + next unless definition.mode == :sync + + Thread.new { run(registry_id: id) } # rubocop:disable ThreadSafety/NewThread + end + end + + start_background_timer! + end + + ## + # @return [void] + def start_background_timer! # rubocop:disable Metrics/MethodLength + interval = Integer(ENV.fetch('REGISTRY_SYNC_INTERVAL_HOURS', '24')) + return if interval <= 0 + + @boot_mutex.synchronize do + return if @timer_started + + @timer_started = true + end + + Thread.new do # rubocop:disable ThreadSafety/NewThread + loop do + sleep(interval * 3600) + Config.precedence.each { |id| run(registry_id: id) if Config.entry(id).mode == :sync rescue nil } # rubocop:disable Style/RescueModifier + end + end + end + + private + + def run_sync!(registry_id:, dry_run:) # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity + definition = Config.entry(registry_id) + raise Errors::SyncError, "Registry '#{registry_id}' is path mode" if definition.mode == :path + + staging_root = Dir.mktmpdir('registry-sync-') + staged_dir = File.join(staging_root, 'bundle') + FileUtils.mkdir_p(staged_dir) + + download_url = ChannelResolver.resolve(definition) + tarball = HttpTransport.fetch!(download_url) + tarball_path = File.join(staging_root, 'download.tar.gz') + File.binwrite(tarball_path, tarball) + + File.open(tarball_path, 'rb') do |io| + Html2rss::Registry::Archive.extract!(io, into: staged_dir) + end + + manifest = Html2rss::Registry::Verifier.verify!(staged_dir, trust: :signed, + public_keys: definition.public_keys) + enforce_max_version!(definition, manifest) + + unless dry_run + if definition.sync_policy.auto_promote + previous = Index.current.bundle_for(registry_id) + Store.swap!(registry_id, staged_dir) + Index.reload! + report_catalog_change!(registry_id, previous) + else + Store.stage_bundle!(registry_id, staged_dir) + end + Store.write_sync_state!(registry_id, last_error: nil) + end + + Index.current.status_entry_for(registry_id) + rescue Html2rss::Registry::Error => error + msg = error.message + log_signature_failure!(registry_id, msg) if msg.match?(/signature|public_key_id/i) + Store.write_sync_state!(registry_id, last_error: msg) unless dry_run + raise Errors::SyncError, msg + rescue StandardError => error + Store.write_sync_state!(registry_id, last_error: error.message) unless dry_run + raise + ensure + FileUtils.rm_rf(staging_root) if staging_root + end + + def log_signature_failure!(registry_id, message) + SecurityLogger.log_registry_signature_failure(registry_id, message) + rescue StandardError + nil + end + + def enforce_max_version!(definition, manifest) + max = definition.sync_policy.max_version + return if max.to_s.empty? + + version = manifest.version + return unless Html2rss::Registry::Manifest.exceeds_max?(version, max) + + raise Errors::SyncError, "Manifest version '#{version}' exceeds max_version '#{max}'" + end + + def report_catalog_change!(registry_id, previous) # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity + current = Index.current.bundle_for(registry_id) + return unless current + + current_version = current.manifest.version + prev_ids = previous ? Set.new(previous.catalog_entries.map(&:id)) : Set.new + curr_ids = Set.new(current.catalog_entries.map(&:id)) + added = (curr_ids - prev_ids).to_a.sort + removed = (prev_ids - curr_ids).to_a.sort + return if added.empty? && removed.empty? && previous&.manifest&.version == current_version + + Observability.emit( + event_name: 'registry.catalog_changed', + outcome: 'success', + level: :warn, + details: { + registry_id:, + version: current_version, + added_count: added.size, + removed_count: removed.size + } + ) + end + end + end + end + end +end diff --git a/app/web/security/security_logger.rb b/app/web/security/security_logger.rb index 1349e8155..894345e81 100644 --- a/app/web/security/security_logger.rb +++ b/app/web/security/security_logger.rb @@ -69,6 +69,20 @@ def log_blocked_request(ip, reason, endpoint) log_event('blocked_request', { ip:, reason:, endpoint: }, severity: :warn) end + # @param registry_id [String] + # @param reason [String] + # @return [void] + def log_registry_signature_failure(registry_id, reason) + log_event('registry_signature_failure', { registry_id:, reason: }, severity: :warn) + end + + # @param registry_id [String] + # @param details [Hash{Symbol => Object}] + # @return [void] + def log_registry_catalog_changed(registry_id, details) + log_event('registry_catalog_changed', { registry_id:, **details }, severity: :warn) + end + private # @param event_type [String] diff --git a/bin/docker-build b/bin/docker-build index 7b4e5f9d7..b88cbc487 100755 --- a/bin/docker-build +++ b/bin/docker-build @@ -2,3 +2,4 @@ set -eux docker build --no-cache -t html2rss/web -f Dockerfile . + diff --git a/bin/docker-healthcheck b/bin/docker-healthcheck deleted file mode 100755 index 617c99ceb..000000000 --- a/bin/docker-healthcheck +++ /dev/null @@ -1,20 +0,0 @@ -#!/usr/bin/env ruby -# frozen_string_literal: true - -require 'uri' -require 'net/http' - -port = ENV.fetch('PORT', '4000') -token = ENV.fetch('HEALTH_CHECK_TOKEN', nil) -token = 'CHANGE_ME_HEALTH_CHECK_TOKEN' if token.nil? || token.empty? - -uri = URI("http://localhost:#{port}/api/v1/health") -request = Net::HTTP::Get.new(uri) -request['Authorization'] = "Bearer #{token}" - -begin - response = Net::HTTP.start(uri.hostname, uri.port) { |http| http.request(request) } - exit(response.is_a?(Net::HTTPSuccess) ? 0 : 1) -rescue StandardError - exit(1) -end diff --git a/bin/html2rss-web b/bin/html2rss-web new file mode 100755 index 000000000..aad85bd4f --- /dev/null +++ b/bin/html2rss-web @@ -0,0 +1,11 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +require 'bundler/setup' + +ENV['RACK_ENV'] ||= 'development' + +require_relative '../app' +require_relative '../app/web/cli' + +exit Html2rss::Web::CLI.run(ARGV) if $PROGRAM_NAME == __FILE__ diff --git a/bin/prepare-registry-seed b/bin/prepare-registry-seed new file mode 100755 index 000000000..acbd8b74b --- /dev/null +++ b/bin/prepare-registry-seed @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +# frozen_string_literal: true + +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "$0")/.." && pwd)" +CONFIGS_REPO="${HTML2RSS_CONFIGS_ROOT:-$ROOT_DIR/../html2rss-configs}" +OUTPUT_TAR="${1:-$ROOT_DIR/dist/registry-bundle.tar.gz}" + +if [[ ! -d "$CONFIGS_REPO/configs" ]]; then + echo "Missing configs tree at $CONFIGS_REPO/configs" >&2 + exit 1 +fi + +mkdir -p "$(dirname "$OUTPUT_TAR")" +echo "Building registry bundle from $CONFIGS_REPO into $OUTPUT_TAR" +( + cd "$CONFIGS_REPO" + BUNDLE_GEMFILE=tool/Gemfile bundle exec ruby tool/registry-build --output "$OUTPUT_TAR" +) + +echo "Registry bundle ready at $OUTPUT_TAR" + diff --git a/config/registries.yml b/config/registries.yml new file mode 100644 index 000000000..2bbd9b557 --- /dev/null +++ b/config/registries.yml @@ -0,0 +1,22 @@ +precedence: + - official + +registries: + official: + sync: + channel: html2rss-official + catalog: true + public_key_id: html2rss:registry:2026 + public_key: | + -----BEGIN PUBLIC KEY----- + MCowBQYDK2VwAyEAiMbg/04MyC5azBdM/aeY0mNuA8JbP5/jOiNRwJ2KJHE= + -----END PUBLIC KEY----- + # Pin the publisher Ed25519 public key for signed network sync and embedded bundle verification. + # Production: keep auto_promote false and promote manually after review. + # auto_promote: false + # sync: + # pin_version: v2026.08.22 + # max_version: v2026.08.22 + # allowed_channel_domains: + # - anthropic.com + diff --git a/docker-compose.yml b/docker-compose.yml index c098fb039..d81ee451e 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -24,6 +24,7 @@ services: RACK_TIMEOUT_SERVICE_TIMEOUT: 55 BOTASAURUS_SCRAPE_TIMEOUT_SECONDS: 45 BOTASAURUS_SCRAPE_WORK_TIMEOUT_SECONDS: 30 + REGISTRY_DATA_ROOT: /app/data/registries BOTASAURUS_SCRAPER_URL: http://botasaurus:4010 # Trial runs use the image's bundled config/feeds.yml. # Uncomment the block below when you want to replace it with your own file. @@ -32,14 +33,8 @@ services: # source: ./config/feeds.yml # target: /app/config/feeds.yml # read_only: true - - watchtower: - image: containrrr/watchtower - restart: unless-stopped volumes: - - /var/run/docker.sock:/var/run/docker.sock - - "${HOME}/.docker/config.json:/config.json" - command: --cleanup --interval 7200 + - registry-data:/app/data/registries botasaurus: image: html2rss/botasaurus-scrape-api:latest @@ -53,3 +48,19 @@ services: SCRAPE_WORK_TIMEOUT_SECONDS: 30 ports: - "127.0.0.1:4010:4010" + + registry-sync: + image: html2rss/web + restart: unless-stopped + command: ["sh", "-c", "while true; do html2rss-web registry sync; sleep $${REGISTRY_SYNC_INTERVAL_SECONDS:-86400}; done"] + env_file: + - path: .env + required: false + environment: + REGISTRY_DATA_ROOT: /app/data/registries + REGISTRY_SYNC_INTERVAL_SECONDS: ${REGISTRY_SYNC_INTERVAL_SECONDS:-86400} + volumes: + - registry-data:/app/data/registries + +volumes: + registry-data: diff --git a/docs/README.md b/docs/README.md index 480eb526b..3082556f7 100644 --- a/docs/README.md +++ b/docs/README.md @@ -26,7 +26,7 @@ Welcome! This is the canonical source of truth for contributing to `html2rss-web - **Runtime behavior**: Application code plus tests. - **HTTP contract**: Request specs plus generated OpenAPI. -- **Config catalog API**: `GET /api/v1/configs` — embedded data from `Html2rss::Configs::Catalog`, merged with local `feeds.yml` entries that include `directory.title`. Disabled when `CONFIG_CATALOG_ENABLED=false` (`404`, `catalog_disabled`). CORS is enabled on this route only. +- **Config catalog API**: `GET /api/v1/configs` — catalog rows from `Registry::Index` (verified registry bundles per `config/registries.yml`, merged with local `feeds.yml` entries that include `directory.title`). Disabled when `CONFIG_CATALOG_ENABLED=false` (`404`, `catalog_disabled`). CORS is enabled on this route only. - **This file**: Contributor conventions and current project rules. --- @@ -167,12 +167,15 @@ Search these pages for examples, plugins, and configuration options: ## Architectural Constraints - **No Persistence**: Do not add databases, ORMs, or background job systems. -- **Backend Style**: +- **Backend Style** (Ruby **4.0+** only — see [AGENTS.md](../AGENTS.md#ruby-4-style)): - Keep the main `app.rb` thin; organize routes in `Html2rss::Web::Routes::*`. - For helpers, use `class << self` and `private` methods. Avoid `module_function`. - Use YARD doc comments for all public methods in `app/`. - Add `# frozen_string_literal: true` to all Ruby files. - Do not use `send(...)` to reach into private APIs; expose what is needed at the module level. + - Prefer leading `&&` / `||` at line start for wrapped conditions; `it` in single-parameter blocks; pattern matching over deep `if/elsif` chains. + - Prefer `Data.define`, `filter_map`, `index_by`, `then`, `match?`, and core `Set` (no `require 'set'`) over OpenStruct, verbose `map`/`compact`, nested `if`, `=~`, and array membership on growing collections. + - Dedupe helpers before extracting new files; use `Set` and memoization on hot paths; table-drive specs with `:aggregate_failures` for multi-assert outcomes. - **Frontend Style**: - Follow visual and CSS rules in [design-system.md](design-system.md). - Use Preact components in `frontend/src/`. @@ -284,6 +287,107 @@ Tune alert thresholds from sustained `request.error` or `feed.render` failure sp --- +## Registry sync runbook + +For end-to-end release and deployment steps (maintainers and operators), see [registry-go-live.md](registry-go-live.md). + +Signed feed registries replace the embedded `html2rss-configs` gem. Each registry is defined in `config/registries.yml` (override path with `REGISTRIES_CONFIG`). + +### Check sync status + +Inside the Dev Container or Docker container: + +```bash +bin/html2rss-web registry status +``` + +Columns: `registry`, `mode`, `version`, `staged_version`, `updated_at`, `sync_url`, `last_error`. Exit code is non-zero when any sync-mode registry lacks a usable on-disk bundle. + +Sync, dry-run, or promote a staged bundle: + +```bash +bin/html2rss-web registry sync --registry official +bin/html2rss-web registry sync --registry official --dry-run +bin/html2rss-web registry promote --registry official +``` + +Production recommendation: keep `auto_promote: false` (default), pin `sync.pin_version` to the approved configs tag, run sync to stage a verified bundle, then promote manually after review. Use `sync.max_version` as an incident freeze cap. + +In Docker Compose, the dedicated `registry-sync` service runs periodic background updates cleanly outside Puma/Ruby. + +Optional hardening: `allowed_channel_domains` suffix-matches every registry config `channel.url` host at bundle load time. + +### Boot behavior + +`Registry::Sync.boot!` runs during app boot (see `app/web/boot/setup.rb`): + +1. **Instant index** — loads the embedded official bundle (140+ feeds) directly from `/app/registries/official` (or `REGISTRY_DATA_ROOT/official` if an updated synced bundle exists). Zero copying, zero startup network traffic. +2. **Sync on boot** — when `REGISTRY_SYNC_ON_BOOT=true`, triggers an asynchronous background sync. +3. **Background refresh** — in standalone single-process mode, `REGISTRY_SYNC_INTERVAL_HOURS` (default `24`) re-syncs on a periodic timer. Set to `0` when using the `registry-sync` Compose service or in static/offline environments. + +Network sync and embedded bundle verification verify Ed25519 signatures using the `public_key` pinned in `registries.yml`. Local `path:` mounts use integrity-only verification. + +### Add a corporate registry + +```yaml +precedence: + - official + - corp + +registries: + official: + sync: + channel: html2rss-official + pin_version: v2026.08.22 # optional + max_version: v2026.08.22 # optional incident freeze + auto_promote: false # default; verified bundles stage until --promote + catalog: true + public_key_id: html2rss:registry:2026 + public_key: | + -----BEGIN PUBLIC KEY----- + ... + -----END PUBLIC KEY----- + + corp: + sync: + url: https://registry.example.com/registry-bundle.tar.gz + catalog: false # feeds served; omitted from GET /api/v1/configs + public_key_id: corp:registry:2026 + public_key: | + -----BEGIN PUBLIC KEY----- + ... + -----END PUBLIC KEY----- +``` + +- **`precedence`** — merge order for feed lookup; first match wins. +- **`sync.url`** — direct tarball URL, or use `sync.channel: html2rss-official` for the default GitHub release asset. +- **`sync.pin_version` / `sync.max_version`** — fetch a specific tag or reject manifests newer than the cap. +- **`auto_promote: false`** — default; verified bundles land in `REGISTRY_DATA_ROOT//.staging/` until `bin/html2rss-web registry promote`. +- **`allowed_channel_domains`** — optional suffix allowlist enforced when the bundle loads. +- **`catalog: false`** — private registry: configs are served at `/{registry.id}.rss` but excluded from the public catalog API (privacy for internal feeds). +- Restrict outbound hosts with `REGISTRY_SYNC_ALLOWED_HOSTS` (comma-separated hostnames). + +### Air-gapped / offline path mount + +For environments without outbound network access, mount a verified bundle directory: + +```yaml +registries: + official: + path: /opt/html2rss/registry/official + catalog: true +``` + +The directory must contain `manifest.json`, optional `manifest.sig`, and `configs/`. Path mode skips network sync; run `bin/html2rss-web registry status` to confirm `mode` is `path`. + +### Key rotation + +1. Publish a new bundle signed with the new key (`public_key_id` in `manifest.json`). +2. Update `public_key_id` and `public_key` in `registries.yml` on every instance **before** or together with the first release that requires the new key. +3. Re-sync (`bin/html2rss-web registry sync`) and promote when `auto_promote: false` (`bin/html2rss-web registry promote`). Failed verification leaves the previous bundle active and records `last_error`. + +--- + ## Documentation Policy - Prefer deleting stale docs over archiving them in-place. diff --git a/docs/architecture.md b/docs/architecture.md index 19530d7c9..f819e0582 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -58,3 +58,23 @@ Strategies are defined by the `html2rss` gem but can be configured here. - **Botasaurus**: Used for JavaScript-heavy websites or anti-bot protected pages (`BOTASAURUS_SCRAPER_URL`). To add or configure strategies, see `app/web/feeds/source_resolver.rb` and the `html2rss` gem documentation. + +## Feed Registries & Storage Architecture + +Feed configs are dynamically resolved from curated registry bundles and local feeds via `Registry::Index`: + +```mermaid +flowchart TD + req["Request (/{feed_id}.rss or /api/v1/configs)"] --> index["Registry::Index.current"] + index --> store["Registry::Store.active_dir(id)"] + store --> synced{"Synced Bundle in data_root?"} + synced -- Yes --> data["/app/data/registries/"] + synced -- No --> embedded["/app/registries/ (Baked Image Bundle)"] + data --> bundle["Html2rss::Registry::Bundle.load"] + embedded --> bundle + bundle --> config["Feed Config"] +``` + +- **Baked Embedded Bundles:** Immutable image layers at `/app/registries/official` provide offline, instant-boot feeds without startup copying. +- **Runtime Synced Bundles:** Periodic synchronization writes verified bundles to `/app/data/registries/`, dynamically taking precedence over the baked image bundle. + diff --git a/docs/registry-go-live.md b/docs/registry-go-live.md new file mode 100644 index 000000000..cfa7dd835 --- /dev/null +++ b/docs/registry-go-live.md @@ -0,0 +1,505 @@ +# Registry go-live manual + +Step-by-step guide to ship and operate signed `registry.v1` bundles across the html2rss org repos. + +**Related docs** + +- Bundle format: [`html2rss/lib/html2rss/registry/README.md`](../../html2rss/lib/html2rss/registry/README.md) +- Operator runbook (sync flags, custom registries): [README.md — Registry sync runbook](./README.md#registry-sync-runbook) + +--- + +## 1. Maintainers — merge and release order + +Work in this order when a change touches registry contracts or feed configs. + +### 1.1 `html2rss` (core gem) + +Source of truth for `registry.v1` schema, `Html2rss::Registry::Verifier`, `CatalogBuilder`, and archive limits. + +1. Merge registry-related changes to `main`. +2. Regenerate schema if config schema changed: + ```bash + cd html2rss + mise exec -- make ready + ``` +3. Tag/release the gem when the contract is stable (web depends on this). + +### 1.2 `html2rss-configs` (signed bundle publisher) + +Source of truth for feed YAML and signed release artifacts. + +1. Merge config changes to `master`. +2. Run the configs quality gate: + ```bash + cd html2rss-configs + make ready + ``` +3. Tag the release (tag name becomes `manifest.json` `version` in CI via `REGISTRY_VERSION: ${{ github.ref_name }}`): + ```bash + git tag v2026.08.22 # example; use your release tag + git push origin v2026.08.22 + ``` +4. Tag push triggers [`.github/workflows/release.yml`](../../html2rss-configs/.github/workflows/release.yml): + - `make registry-build -- --sign` + - uploads `dist/registry-bundle.tar.gz` to a **draft** GitHub Release (human publishes after review) + +**Draft → publish:** CI creates a draft release. Maintainers review the asset, release notes, and tag diff, then click **Publish release** in GitHub. Draft releases are invisible to `/releases/latest` until published. + +### 1.3 `html2rss-web` (runtime + Docker image) + +Consumes verified bundles; bakes the official release bundle into the Docker image at build time. + +1. Bump the `html2rss` gem dependency if the core contract changed. +2. Update `config/registries.yml` when the signing key or sync channel changes (see sections 2 and 5). +3. Build the image with the official release artifact baked (section 3). +4. Merge to `main`, wait for CI (`ci` workflow) to pass. +5. Publish a GitHub Release on `html2rss-web` — [`.github/workflows/release.yml`](../../html2rss-web/.github/workflows/release.yml) builds and pushes `html2rss/web` tags (`latest`, semver, major, commit SHA). + +**Rule of thumb:** core contract → configs signed release → web image that embeds/verifies that release. + +--- + +## 2. First signed release — signing key, tag, verify artifact + +### 2.1 Generate an Ed25519 key pair + +Use OpenSSL (same algorithm as `tool/registry-build` and `Html2rss::Registry::Verifier`): + +```bash +openssl genpkey -algorithm ED25519 -out registry-signing.pem +openssl pkey -in registry-signing.pem -pubout -out registry-signing.pub +``` + +Keep the private key offline except where signing happens. + +### 2.2 Configure `html2rss-configs` CI secrets + +In the `html2rss-configs` GitHub repo, add secrets on the **`registry-release`** environment: + +| Secret | Purpose | +| --- | --- | +| `REGISTRY_SIGNING_KEY` | Full PEM contents of `registry-signing.pem` (private key; used by `make registry-build -- --sign`) | +| `REGISTRY_PUBLIC_KEY_PEM` | Full PEM contents of `registry-signing.pub` (public key; used by the release verify step — not a repo fixture) | + +Release CI reads them here: + +```yaml +env: + REGISTRY_VERSION: ${{ github.ref_name }} + REGISTRY_SIGNING_KEY: ${{ secrets.REGISTRY_SIGNING_KEY }} +run: make registry-build -- --sign +``` + +The verify step loads the public key from `REGISTRY_PUBLIC_KEY_PEM` in-process (`OpenSSL::PKey.read(ENV.fetch('REGISTRY_PUBLIC_KEY_PEM'))`); no committed PEM file in the configs repo. + +At go-live, the same public key must appear in `html2rss-web/config/registries.yml` (`public_key` / `public_key_id`) so runtime sync verification matches CI. + +### 2.3 Pin the public key in `html2rss-web` + +Add the public key to `config/registries.yml` before instances need to sync signed bundles over the network: + +```yaml +registries: + official: + sync: + channel: html2rss-official + pin_version: v2026.08.22 # optional: fetch this tag only + max_version: v2026.08.22 # optional: reject newer manifests (incident freeze) + auto_promote: false # default false; verified bundles stage until manual promote + catalog: true + public_key_id: html2rss:registry:2026 + public_key: | + -----BEGIN PUBLIC KEY----- + ... + -----END PUBLIC KEY----- + allowed_channel_domains: # optional suffix allowlist for channel.url domains + - anthropic.com + - github.com +``` + +- `public_key_id` must match the value in signed `manifest.json` (default in `tool/registry-build`: `html2rss:registry:2026`). +- Network sync and embedded bundle loading use `:signed` trust and require this pin. Local `path:` mounts use `:integrity_only` trust (no signature check on load). +- **`auto_promote: false`** (default) writes verified bundles to `REGISTRY_DATA_ROOT//.staging/` without changing the active catalog. Promote after review with `bin/html2rss-web registry promote --registry official`. +- **`pin_version`** resolves the GitHub tag release API instead of `/releases/latest`. +- **`max_version`** rejects sync when the verified manifest version is newer than the cap (incident freeze). +- **`allowed_channel_domains`** rejects bundle load when any config `channel.url` host is outside the suffix allowlist. + +### 2.4 Tag and publish the configs release + +```bash +cd html2rss-configs +git tag # e.g. 2026.08.22 +git push origin +``` + +Wait for the Release workflow to finish. The asset name is always **`registry-bundle.tar.gz`**. + +### 2.5 Verify the release artifact + +Download the asset from the GitHub Release, then verify locally: + +```bash +mkdir -p /tmp/registry-verify && tar -xzf registry-bundle.tar.gz -C /tmp/registry-verify + +# Inspect manifest +jq . /tmp/registry-verify/manifest.json +# Confirm: format=registry.v1, registry_id=official, version=, public_key_id matches pin + +# Confirm signature file exists +test -f /tmp/registry-verify/manifest.sig + +# Optional: verify with core gem (from html2rss checkout) +cd html2rss +mise exec -- bundle exec ruby -rhtml2rss -ropenssl -e " + pk = OpenSSL::PKey.read(File.read('path/to/registry-signing.pub')) + Html2rss::Registry::Verifier.verify!( + '/tmp/registry-verify', + trust: :signed, + public_keys: { 'html2rss:registry:2026' => pk } + ) + puts 'OK' +" +``` + +Unsigned local builds (no `--sign`) are valid for integrity-only use only: + +```bash +cd html2rss-configs +make registry-build # writes dist/registry-bundle.tar.gz without manifest.sig +``` + +--- + +## 3. Web image build — artifact baking and Docker + +### 3.1 How the official registry gets into the image + +| Step | What happens | +| --- | --- | +| `Dockerfile` (`registry-builder`) | Downloads the official release `registry-bundle.tar.gz`, extracts it, and verifies its Ed25519 signature & file digests against `config/registries.yml` | +| `Dockerfile` (Runtime) | `COPY --from=registry-builder /build/official /app/registries/official` | +| Boot | `Registry::Index.current` loads `/app/registries/official` directly. Zero copying, zero network calls. | + +### 3.2 Build locally + +To build a production image fetching the latest official release: + +```bash +cd html2rss-web +docker build -t html2rss/web -f Dockerfile . +``` + +To build pinning a specific configs release tag: + +```bash +docker build \ + --build-arg REGISTRY_RELEASE_TAG=v2026.08.22 \ + --build-arg REGISTRY_BUNDLE_URL=https://github.com/html2rss/html2rss-configs/releases/download/v2026.08.22/registry-bundle.tar.gz \ + --build-arg BUILD_TAG=1.2.3 \ + --build-arg GIT_SHA="$(git rev-parse HEAD)" \ + -t html2rss/web \ + -f Dockerfile . +``` + +### 3.3 What operators get in the image + +- **`/app/registries/official/`** — Pre-verified official release bundle baked at build time (immutable image layer). +- **`/app/config/registries.yml`** — Default official registry configuration with pinned public key. +- **`/app/data/registries`** — Empty at build; used at runtime for network synchronization. + +--- + +## 4. Operators — default path (official registry) + +### 4.1 Pull a new image (simplest update path) + +`docker-compose.yml` defaults: + +- Image: `html2rss/web` +- Volume: `registry-data:/app/data/registries` (`REGISTRY_DATA_ROOT`) + +```bash +docker compose pull html2rss-web +docker compose up -d html2rss-web +``` + +A new image updates the **embedded bundle** inside the container layers. + +### 4.2 Zero-config first boot + +With the stock `config/registries.yml`, no extra registry env vars are required. + +On boot (`Registry::Sync.boot!`): + +1. **Instant index** — loads directly from `/app/registries/official` (or `REGISTRY_DATA_ROOT/official` if an updated synced bundle exists). +2. **Sync on boot** — runs only when `REGISTRY_SYNC_ON_BOOT=true`. +3. **Background refresh** — when `REGISTRY_SYNC_INTERVAL_HOURS` > 0 (default **24**), re-syncs on a periodic timer. Set to `0` to disable. + +Official sync URL (from `config/registries.yml` + `Registry::Config`): + +`https://github.com/html2rss/html2rss-configs/releases/latest/download/registry-bundle.tar.gz` + +Allowed outbound hosts (built-in): `api.github.com`, `github.com`, `objects.githubusercontent.com`. + +### 4.3 Existing instance — volume retained + +The named volume preserves runtime-synced bundles across container restarts. When present and valid, `Store.active_dir` prefers the volume copy over the embedded image copy. + +- Old bundle stays active until a successful sync swaps it. +- Use `bin/html2rss-web registry sync` or wait for background refresh to pick up a new configs release without rebuilding the image (section 7). + +--- + +## 5. Operators — custom corporate registry + +Override or extend `config/registries.yml` (or set `REGISTRIES_CONFIG` to an alternate file path). + +Example from the [registry sync runbook](./README.md#add-a-corporate-registry): + +```yaml +precedence: + - official + - corp + +registries: + official: + sync: + channel: html2rss-official + catalog: true + public_key_id: html2rss:registry:2026 + public_key: | + -----BEGIN PUBLIC KEY----- + ... + -----END PUBLIC KEY----- + + corp: + sync: + url: https://registry.example.com/registry-bundle.tar.gz + catalog: false + public_key_id: corp:registry:2026 + public_key: | + -----BEGIN PUBLIC KEY----- + ... + -----END PUBLIC KEY----- +``` + +| Field | Purpose | +| --- | --- | +| `precedence` | Feed lookup merge order; first match wins | +| `sync.url` | Direct HTTPS tarball URL for network sync | +| `sync.channel: html2rss-official` | Resolves to the official GitHub release asset | +| `catalog: false` | Feeds served at `/{feed_id}.rss`; **omitted** from `GET /api/v1/configs` | +| `public_key` / `public_key_id` | Required for `:signed` network sync verification | + +For hosts outside the default GitHub allowlist: + +```bash +REGISTRY_SYNC_ALLOWED_HOSTS=registry.example.com,cdn.example.com +``` + +Mount a custom registries file in Compose: + +```yaml +volumes: + - ./config/registries.yml:/app/config/registries.yml:ro +``` + +--- + +## 6. Verify live + +Run these checks after deploy or sync. + +### 6.1 Registry sync status (CLI) + +Inside the running container (or Dev Container): + +```bash +bin/html2rss-web registry status +``` + +Tab-separated columns: `registry`, `mode`, `version`, `staged_version`, `updated_at`, `sync_url`, `last_error`. + +- Exit code **0** — all sync-mode registries have a usable on-disk bundle. +- Exit code **1** — at least one sync registry lacks a bundle (see `Registry::Sync.unusable_sync_registries`). + +Single registry, dry-run, or promote staged bundle: + +```bash +bin/html2rss-web registry sync --registry official +bin/html2rss-web registry sync --registry official --dry-run +bin/html2rss-web registry promote --registry official +``` + +### 6.2 Instance metadata API + +```bash +curl -sS http://127.0.0.1:4000/api/v1/ | jq '.data.instance' +``` + +Confirm: + +- **`instance.registries`** — array with `id`, `version`, `updated_at`, `sync_mode` per configured registry +- **`instance.catalog`** — `{ "enabled": true, "url": ".../api/v1/configs" }` (unless `CONFIG_CATALOG_ENABLED=false`) + +### 6.3 Catalog API + +```bash +curl -sS http://127.0.0.1:4000/api/v1/configs | jq '.data.configs[0]' +``` + +Expect rows with `source: "registry"`, `registry: "official"`, and a `path` like `/anthropic.com/news.rss`. + +When `CONFIG_CATALOG_ENABLED=false`, expect `404` with `{ "error": "catalog_disabled" }`. + +### 6.4 Sample static feed + +Pick a feed id from the catalog (`id` field) or from `configs/.yml` in the bundle. Request: + +```bash +curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:4000/anthropic.com/news.rss +``` + +Expect HTTP **200** and valid RSS XML. + +--- + +## 7. Update configs without pulling a new Docker image + +Configs releases are independent of web image releases. To refresh feeds on a running instance: + +### 7.1 Manual sync and promote + +Production recommendation: keep `auto_promote: false`, set `sync.pin_version` to the approved tag, fetch with sync, then promote manually after review. + +```bash +bin/html2rss-web registry sync --registry official +bin/html2rss-web registry status # staged_version shows verified bundle +bin/html2rss-web registry promote --registry official +``` + +Fetches the pinned or latest signed tarball, verifies signature + digests, and either stages (`auto_promote: false`) or atomically swaps the active bundle (`auto_promote: true`). + +### 7.2 Incident freeze + +To block uptake of a newer configs release without disabling sync entirely: + +```yaml +registries: + official: + sync: + channel: html2rss-official + max_version: v2026.08.21 + auto_promote: false +``` + +Set `REGISTRY_SYNC_INTERVAL_HOURS=0` to pause background refresh while investigating. + +### 7.3 Automatic refresh via Docker Compose + +In production Compose environments, registry synchronization is handled cleanly outside the Ruby/Puma process lifecycle via the dedicated `registry-sync` Compose service in `docker-compose.yml`: + +```yaml + registry-sync: + image: html2rss/web + restart: unless-stopped + command: ["sh", "-c", "while true; do html2rss-web registry sync; sleep $${REGISTRY_SYNC_INTERVAL_SECONDS:-86400}; done"] + environment: + REGISTRY_DATA_ROOT: /app/data/registries + REGISTRY_SYNC_INTERVAL_SECONDS: 86400 + volumes: + - registry-data:/app/data/registries +``` + +When new bundles are synced to disk, the `html2rss-web` server detects the updated `manifest.json` mtimes automatically on subsequent requests. + +To run an on-demand sync or check status with Docker Compose: + +```bash +docker compose exec html2rss-web html2rss-web registry status +docker compose exec html2rss-web html2rss-web registry sync --registry official +``` + +--- + +## 8. Troubleshooting + +### 8.1 Signature verification failure + +Symptoms in `bin/html2rss-web registry status`: + +- `last_error` contains `Unknown public_key_id`, `Invalid manifest signature`, or `Missing manifest.sig` + +Checks: + +1. `public_key_id` and `public_key` in `registries.yml` match the signed release. +2. Deploy web config **before** or **with** the first bundle signed by a new key (key rotation). +3. Downloaded asset is `registry-bundle.tar.gz` from the expected release (not an unsigned local build). + +Signature-related failures are logged via `SecurityLogger.log_registry_signature_failure`. + +Verify without swapping the active bundle: + +```bash +bin/html2rss-web registry sync --registry official --dry-run +``` + +### 8.2 Sync failure keeps the old bundle + +By design: + +- `Registry::Sync.run` only calls `Store.swap!` after fetch **and** verification succeed. +- On failure, `last_error` is recorded; the previous bundle under `REGISTRY_DATA_ROOT//` remains served. +- `Store.promote_bundle!` rolls back on swap failure. + +If sync fails on first boot with no prior bundle in the volume, the instance serves the embedded official bundle (140+ feeds) directly from the container image. + +### 8.3 Network / host errors + +| Error pattern | Likely cause | +| --- | --- | +| `Registry sync host not allowed` | Add host to `REGISTRY_SYNC_ALLOWED_HOSTS` | +| `Registry sync rejects HTTP redirects` | Publish a direct HTTPS asset URL (`sync.url`) | +| `Registry sync fetch failed with HTTP …` | Release missing, URL wrong, or GitHub outage | +| `Registry sync requires HTTPS URLs` | Use `https://` in `sync.url` | + +Default allowed hosts cover official GitHub release downloads (`api.github.com`, `github.com`, `objects.githubusercontent.com`, `release-assets.githubusercontent.com`). + +### 8.4 Air-gapped / offline (`path` mode) + +For environments without outbound sync, mount a verified bundle directory and skip network sync: + +```yaml +registries: + official: + path: /opt/html2rss/registry/official + catalog: true +``` + +Requirements: + +- Directory contains `manifest.json`, `configs/`, and optionally `manifest.sig` +- `bin/html2rss-web registry sync` is not applicable (`path mode; sync is not applicable`) +- `bin/html2rss-web registry status` should show `mode: path` + +Load path uses `:integrity_only` trust (disk/image trust boundary). + +In Docker Compose, bind-mount the bundle and optionally set `REGISTRIES_CONFIG`: + +```yaml +volumes: + - /opt/html2rss/registry/official:/opt/html2rss/registry/official:ro +environment: + REGISTRIES_CONFIG: /app/config/registries.yml +``` + +### 8.5 CLI exit code non-zero with empty `last_error` + +`bin/html2rss-web registry status` exits **1** when a sync-mode registry has **no usable bundle** (neither in the volume nor embedded in the image). Run a sync or verify the embedded bundle is present: + +```bash +bin/html2rss-web registry sync --registry official +bin/html2rss-web registry status +``` + diff --git a/frontend/src/api/generated/types.gen.ts b/frontend/src/api/generated/types.gen.ts index 2939a0d7b..8a300e216 100644 --- a/frontend/src/api/generated/types.gen.ts +++ b/frontend/src/api/generated/types.gen.ts @@ -13,7 +13,7 @@ export type GetApiMetadataData = { export type GetApiMetadataResponses = { /** - * returns catalog pointer metadata + * returns registry status metadata */ 200: { data: { @@ -31,6 +31,12 @@ export type GetApiMetadataResponses = { access_token_required: boolean; enabled: boolean; }; + registries: Array<{ + id: string; + sync_mode: string; + updated_at?: string; + version?: string; + }>; }; }; success: boolean; @@ -77,39 +83,14 @@ export type GetConfigCatalogResponses = { id: string; parameters: { defaults: { - blog?: string | null; - id?: string | null; - region?: string | null; - repository?: string | null; - section?: string | null; - user_id?: string | null; - username?: string | null; + [key: string]: unknown; }; schema: { - blog?: { - type: string; - } | null; - id?: { - type: string; - } | null; - region?: { - type: string; - } | null; - repository?: { - type: string; - } | null; - section?: { - type: string; - } | null; - user_id?: { - type: string; - } | null; - username?: { - type: string; - } | null; + [key: string]: unknown; }; }; path: string; + registry?: string; source: string; }>; }; diff --git a/public/openapi.yaml b/public/openapi.yaml index 863713c24..f87837ca3 100644 --- a/public/openapi.yaml +++ b/public/openapi.yaml @@ -64,9 +64,28 @@ paths: - enabled - access_token_required type: object + registries: + items: + properties: + id: + type: string + sync_mode: + type: string + updated_at: + type: string + nullable: true + version: + type: string + nullable: true + required: + - id + - sync_mode + type: object + type: array required: - feed_creation - catalog + - registries type: object required: - api @@ -78,7 +97,7 @@ paths: - success - data type: object - description: returns catalog pointer metadata + description: returns registry status metadata security: - {} summary: API metadata @@ -132,101 +151,10 @@ paths: parameters: properties: defaults: - properties: - blog: - type: - - string - - 'null' - id: - type: - - string - - 'null' - region: - type: - - string - - 'null' - repository: - type: - - string - - 'null' - section: - type: - - string - - 'null' - user_id: - type: - - string - - 'null' - username: - type: - - string - - 'null' + properties: {} type: object schema: - properties: - blog: - properties: - type: - type: string - required: - - type - type: - - object - - 'null' - id: - properties: - type: - type: string - required: - - type - type: - - object - - 'null' - region: - properties: - type: - type: string - required: - - type - type: - - object - - 'null' - repository: - properties: - type: - type: string - required: - - type - type: - - object - - 'null' - section: - properties: - type: - type: string - required: - - type - type: - - object - - 'null' - user_id: - properties: - type: - type: string - required: - - type - type: - - object - - 'null' - username: - properties: - type: - type: string - required: - - type - type: - - object - - 'null' + properties: {} type: object required: - schema @@ -234,15 +162,17 @@ paths: type: object path: type: string + registry: + type: string source: type: string required: - id - path - - source - directory - channel - parameters + - source type: object type: array required: diff --git a/spec/fixtures/registries/keys/test-key.pem b/spec/fixtures/registries/keys/test-key.pem new file mode 100644 index 000000000..d1583ef45 --- /dev/null +++ b/spec/fixtures/registries/keys/test-key.pem @@ -0,0 +1,3 @@ +-----BEGIN PRIVATE KEY----- +MC4CAQAwBQYDK2VwBCIEIKWA7CdQvmMCa06H6jojjHE30xRV9Ps823kjgqWKvfSE +-----END PRIVATE KEY----- diff --git a/spec/fixtures/registries/keys/test-key.pub b/spec/fixtures/registries/keys/test-key.pub new file mode 100644 index 000000000..236aab386 --- /dev/null +++ b/spec/fixtures/registries/keys/test-key.pub @@ -0,0 +1,3 @@ +-----BEGIN PUBLIC KEY----- +MCowBQYDK2VwAyEAiMbg/04MyC5azBdM/aeY0mNuA8JbP5/jOiNRwJ2KJHE= +-----END PUBLIC KEY----- diff --git a/spec/fixtures/registries/official/configs/anthropic.com/news.yml b/spec/fixtures/registries/official/configs/anthropic.com/news.yml new file mode 100644 index 000000000..af1cbc812 --- /dev/null +++ b/spec/fixtures/registries/official/configs/anthropic.com/news.yml @@ -0,0 +1,23 @@ +# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json +registry: + id: anthropic.com/news +directory: + topics: + - tech + - research + title: "Anthropic — News" + summary: "Product and research announcements from Anthropic." +channel: + title: "Anthropic — News" + url: https://www.anthropic.com/news + language: en + time_zone: UTC + ttl: 360 +selectors: + items: + selector: 'ul[class*="PublicationList"] > li > a[href^="/news/"]' + enhance: false + title: + selector: '[class*="__title"]' + url: + extractor: href diff --git a/spec/fixtures/registries/official/configs/deepmind.google/blog.yml b/spec/fixtures/registries/official/configs/deepmind.google/blog.yml new file mode 100644 index 000000000..6d3733c1e --- /dev/null +++ b/spec/fixtures/registries/official/configs/deepmind.google/blog.yml @@ -0,0 +1,27 @@ +# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json +registry: + id: deepmind.google/blog +directory: + topics: + - science + - research + title: "Google DeepMind — Blog" + summary: "Research and product posts from Google DeepMind." + +channel: + title: "Google DeepMind — Blog" + url: https://deepmind.google/blog/ + language: en + time_zone: UTC + ttl: 360 +selectors: + items: + selector: .card__inner + enhance: false + title: + selector: h3 + url: + selector: .card__overlay-link + extractor: href + published_at: + selector: time diff --git a/spec/fixtures/registries/official/manifest.json b/spec/fixtures/registries/official/manifest.json new file mode 100644 index 000000000..82f32f845 --- /dev/null +++ b/spec/fixtures/registries/official/manifest.json @@ -0,0 +1,11 @@ +{ + "format": "registry.v1", + "registry_id": "official", + "version": "test-fixture", + "public_key_id": "test", + "files": { + "configs/anthropic.com/news.yml": "679b3c28b6dd7e08bbfdaf8564e077e7c2ac4140eacbc9543756591aaf0621f5", + "configs/deepmind.google/blog.yml": "3dbc30f109bb169f1de06372236a2d352584808d11be8243ba93befcd94a7b41" + } +} + diff --git a/spec/fixtures/registries/private/configs/secret.example/private.yml b/spec/fixtures/registries/private/configs/secret.example/private.yml new file mode 100644 index 000000000..14a119e7f --- /dev/null +++ b/spec/fixtures/registries/private/configs/secret.example/private.yml @@ -0,0 +1,17 @@ +registry: + id: secret.example/private +directory: + title: "Private Corp Feed" + summary: "Hidden from catalog." +channel: + title: "Private Corp Feed" + url: https://secret.example/private + ttl: 60 +selectors: + items: + selector: article + title: + selector: h1 + url: + selector: a + extractor: href diff --git a/spec/fixtures/registries/private/manifest.json b/spec/fixtures/registries/private/manifest.json new file mode 100644 index 000000000..6d901dbbf --- /dev/null +++ b/spec/fixtures/registries/private/manifest.json @@ -0,0 +1,9 @@ +{ + "format": "registry.v1", + "registry_id": "private", + "version": "test-fixture", + "public_key_id": "test", + "files": { + "configs/secret.example/private.yml": "f3b4f6d567e44960be89c171a915a5451266998090ca9ffdb394050bcbbd0dd2" + } +} diff --git a/spec/fixtures/registries/registries.yml b/spec/fixtures/registries/registries.yml new file mode 100644 index 000000000..b505e5794 --- /dev/null +++ b/spec/fixtures/registries/registries.yml @@ -0,0 +1,11 @@ +precedence: + - official + - private + +registries: + official: + path: spec/fixtures/registries/official + catalog: true + private: + path: spec/fixtures/registries/private + catalog: false diff --git a/spec/fixtures/registries/sync/bundle/manifest.json b/spec/fixtures/registries/sync/bundle/manifest.json new file mode 100644 index 000000000..8a0bf8a6d --- /dev/null +++ b/spec/fixtures/registries/sync/bundle/manifest.json @@ -0,0 +1,11 @@ +{ + "format": "registry.v1", + "registry_id": "official", + "version": "test-fixture", + "public_key_id": "test-key", + "files": { + "configs/anthropic.com/news.yml": "679b3c28b6dd7e08bbfdaf8564e077e7c2ac4140eacbc9543756591aaf0621f5", + "configs/deepmind.google/blog.yml": "3dbc30f109bb169f1de06372236a2d352584808d11be8243ba93befcd94a7b41" + } +} + diff --git a/spec/fixtures/registries/sync/registries.yml b/spec/fixtures/registries/sync/registries.yml new file mode 100644 index 000000000..4f1ab3dff --- /dev/null +++ b/spec/fixtures/registries/sync/registries.yml @@ -0,0 +1,14 @@ +precedence: + - official + +registries: + official: + sync: + url: https://registry.test.example/registry-bundle.tar.gz + auto_promote: true + catalog: true + public_key_id: test-key + public_key: | + -----BEGIN PUBLIC KEY----- + MCowBQYDK2VwAyEAiMbg/04MyC5azBdM/aeY0mNuA8JbP5/jOiNRwJ2KJHE= + -----END PUBLIC KEY----- diff --git a/spec/html2rss/web/api/v1_spec.rb b/spec/html2rss/web/api/v1_spec.rb index e037715f7..5eb375b23 100644 --- a/spec/html2rss/web/api/v1_spec.rb +++ b/spec/html2rss/web/api/v1_spec.rb @@ -196,6 +196,21 @@ def relative_feed_link_header(token) ) end + it 'returns registry status metadata', :aggregate_failures do + get '/api/v1' + + expect(last_response.status).to eq(200) + json = expect_success_response(last_response) + official = json.dig('data', 'instance', 'registries').find { |row| row['id'] == 'official' } + + expect(official).to include( + 'id' => 'official', + 'version' => 'test-fixture', + 'sync_mode' => 'path' + ) + expect(official['updated_at']).to be_a(String) + end + it 'returns API information with trailing slash', :aggregate_failures do get '/api/v1/' @@ -221,7 +236,10 @@ def relative_feed_link_header(token) json = expect_success_response(last_response) expect(json.dig('meta', 'catalog_version')).to eq(1) expect(json.dig('data', 'configs')).to be_an(Array) - expect(json.dig('data', 'configs').first).to include('id', 'path', 'source', 'directory', 'channel', 'parameters') + first = json.dig('data', 'configs').first + expect(first).to include('id', 'path', 'source', 'directory', 'channel', 'parameters') + expect(first['source']).to eq('registry') + expect(first).to include('registry' => 'official') end it 'returns 404 when the catalog is disabled', :aggregate_failures do diff --git a/spec/html2rss/web/app_spec.rb b/spec/html2rss/web/app_spec.rb index ead4b1827..294b3ebed 100644 --- a/spec/html2rss/web/app_spec.rb +++ b/spec/html2rss/web/app_spec.rb @@ -24,7 +24,7 @@ def static_feed_json end def stub_static_feed(rss_body: '', json_body: static_feed_json, ttl: 180) - allow(Html2rss::Web::LocalConfig).to receive(:find).and_return({ channel: { ttl: ttl } }) + allow(Html2rss::Web::Registry::Index.current).to receive(:config_for).and_return({ channel: { ttl: ttl } }) stub_static_renderers(static_feed_result(ttl:), rss_body:, json_body:) end @@ -59,8 +59,8 @@ def static_service_error_result end def stub_static_service_error(feed_name) - allow(Html2rss::Web::LocalConfig) - .to receive(:find) + allow(Html2rss::Web::Registry::Index.current) + .to receive(:config_for) .with(feed_name) .and_return({ channel: { ttl: 180 } }) allow(Html2rss::Web::Feeds::Service).to receive(:call).and_return(static_service_error_result) @@ -145,7 +145,8 @@ def app = described_class end it 'serves nested static feed routes' do - allow(Html2rss::Web::LocalConfig).to receive(:find).with('team/releases').and_return({ channel: { ttl: 180 } }) + allow(Html2rss::Web::Registry::Index.current) + .to receive(:config_for).with('team/releases').and_return({ channel: { ttl: 180 } }) stub_static_renderers(static_feed_result(ttl: 180), rss_body: '', json_body: static_feed_json) get '/team/releases.xml' @@ -157,7 +158,8 @@ def app = described_class it 'serves HEAD requests for static feed routes with negotiated headers only' do feed_name = "legacy-head-#{SecureRandom.hex(4)}" - allow(Html2rss::Web::LocalConfig).to receive(:find).with(feed_name).and_return({ channel: { ttl: 180 } }) + allow(Html2rss::Web::Registry::Index.current) + .to receive(:config_for).with(feed_name).and_return({ channel: { ttl: 180 } }) stub_static_renderers(static_feed_result(ttl: 180), rss_body: '', json_body: static_feed_json) head "/#{feed_name}" @@ -176,7 +178,8 @@ def app = described_class it 'coerces string ttl values before cache expiry math' do feed_name = "legacy-ttl-#{SecureRandom.hex(4)}" - allow(Html2rss::Web::LocalConfig).to receive(:find).with(feed_name).and_return({ channel: { ttl: '180' } }) + allow(Html2rss::Web::Registry::Index.current) + .to receive(:config_for).with(feed_name).and_return({ channel: { ttl: '180' } }) stub_static_renderers(static_feed_result(ttl: '180'), rss_body: '', json_body: static_feed_json) get "/#{feed_name}" diff --git a/spec/html2rss/web/boot/setup_spec.rb b/spec/html2rss/web/boot/setup_spec.rb index 116b60811..d98558843 100644 --- a/spec/html2rss/web/boot/setup_spec.rb +++ b/spec/html2rss/web/boot/setup_spec.rb @@ -21,6 +21,7 @@ before do allow(Html2rss::Web::Flags).to receive(:validate!) allow(Html2rss::Web::Boot::Sentry).to receive(:configure!) + allow(Html2rss::Web::Registry::Sync).to receive(:boot!) end describe '.call!' do @@ -33,6 +34,7 @@ expect(Html2rss::Web::EnvironmentValidator).to have_received(:validate_environment!).once expect(Html2rss::Web::EnvironmentValidator).to have_received(:validate_production_security!).once expect(Html2rss::Web::Flags).to have_received(:validate!).once + expect(Html2rss::Web::Registry::Sync).to have_received(:boot!).once end it 'routes rack-timeout logs through the shared app logger' do diff --git a/spec/html2rss/web/cli_spec.rb b/spec/html2rss/web/cli_spec.rb new file mode 100644 index 000000000..094e1fd45 --- /dev/null +++ b/spec/html2rss/web/cli_spec.rb @@ -0,0 +1,125 @@ +# frozen_string_literal: true + +require 'stringio' +require_relative '../../../app' +require_relative '../../../app/web/cli' + +RSpec.describe Html2rss::Web::CLI do + describe '.run' do + let(:stdout) { StringIO.new } + let(:stderr) { StringIO.new } + + context 'with root help and version commands' do + it 'prints root help for help options', :aggregate_failures do + exit_code = described_class.run(['--help'], out: stdout, err: stderr) + expect(exit_code).to eq(0) + expect(stdout.string).to include('Usage: html2rss-web [command] [options]') + expect(stdout.string).to include('registry status') + end + + it 'prints root help when argv is empty', :aggregate_failures do + exit_code = described_class.run([], out: stdout, err: stderr) + expect(exit_code).to eq(0) + expect(stdout.string).to include('Usage: html2rss-web [command] [options]') + end + + it 'prints version and runtime info', :aggregate_failures do + exit_code = described_class.run(['version'], out: stdout, err: stderr) + expect(exit_code).to eq(0) + expect(stdout.string).to match(/html2rss-web build=.* sha=.* ruby=.*/) + end + + it 'handles unknown commands', :aggregate_failures do + exit_code = described_class.run(['invalid-cmd'], out: stdout, err: stderr) + expect(exit_code).to eq(1) + expect(stderr.string).to include('Unknown command: "invalid-cmd"') + end + end + + context 'with healthcheck command' do + let(:success_response) { instance_double(Net::HTTPSuccess, is_a?: true) } + let(:error_response) { instance_double(Net::HTTPInternalServerError, is_a?: false, code: '500') } + + it 'returns 0 when HTTP healthcheck succeeds', :aggregate_failures do + allow(Net::HTTP).to receive(:get_response).and_return(success_response) + + exit_code = described_class.run(['healthcheck'], out: stdout, err: stderr) + expect(exit_code).to eq(0) + expect(stdout.string).to include('OK') + end + + it 'returns 1 when HTTP healthcheck returns non-200', :aggregate_failures do + allow(Net::HTTP).to receive(:get_response).and_return(error_response) + + exit_code = described_class.run(['healthcheck'], out: stdout, err: stderr) + expect(exit_code).to eq(1) + expect(stderr.string).to include('Healthcheck failed: HTTP 500') + end + + it 'returns 1 when HTTP connection fails', :aggregate_failures do + allow(Net::HTTP).to receive(:get_response).and_raise(Errno::ECONNREFUSED) + + exit_code = described_class.run(['healthcheck'], out: stdout, err: stderr) + expect(exit_code).to eq(1) + expect(stderr.string).to include('Healthcheck failed') + end + end + + context 'with registry subcommands' do + it 'prints registry help for --help', :aggregate_failures do + exit_code = described_class.run(%w[registry --help], out: stdout, err: stderr) + expect(exit_code).to eq(0) + expect(stdout.string).to include('Usage: html2rss-web registry [subcommand]') + end + + it 'prints registry status table', :aggregate_failures do + exit_code = described_class.run(%w[registry status], out: stdout, err: stderr) + expect(exit_code).to eq(0) + expect(stdout.string).to include("registry\tmode\tversion\tstaged_version\tupdated_at\tsync_url\tlast_error") + expect(stdout.string).to include('official') + end + + it 'runs registry sync', :aggregate_failures do + allow(Html2rss::Web::Registry::Sync).to receive_messages(run: nil, cli_exit_code: 0) + + exit_code = described_class.run(%w[registry sync --registry official --dry-run], out: stdout, err: stderr) + expect(exit_code).to eq(0) + expect(Html2rss::Web::Registry::Sync).to have_received(:run).with(registry_id: 'official', dry_run: true) + end + + it 'runs registry promote', :aggregate_failures do + allow(Html2rss::Web::Registry::Sync).to receive_messages(promote_staged!: nil, cli_exit_code: 0) + + exit_code = described_class.run(%w[registry promote --registry official], out: stdout, err: stderr) + expect(exit_code).to eq(0) + expect(Html2rss::Web::Registry::Sync).to have_received(:promote_staged!).with(registry_id: 'official') + end + + it 'verifies valid registry bundles', :aggregate_failures do + exit_code = described_class.run( + ['registry', 'verify', '--registry', 'official', '--dir', 'spec/fixtures/registries/official'], + out: stdout, + err: stderr + ) + expect(exit_code).to eq(0) + expect(stdout.string).to include("Verified registry bundle 'official' (test-fixture)") + end + + it 'reports failure on invalid registry verification', :aggregate_failures do + exit_code = described_class.run( + ['registry', 'verify', '--registry', 'official', '--dir', 'tmp/non-existent'], + out: stdout, + err: stderr + ) + expect(exit_code).to eq(1) + expect(stderr.string).to include('Registry verification failed') + end + + it 'handles unknown registry subcommands', :aggregate_failures do + exit_code = described_class.run(%w[registry invalid-sub], out: stdout, err: stderr) + expect(exit_code).to eq(1) + expect(stderr.string).to include('Unknown registry command: "invalid-sub"') + end + end + end +end diff --git a/spec/html2rss/web/feeds/responder_spec.rb b/spec/html2rss/web/feeds/responder_spec.rb index 1d8e1a9d5..0ebd91e54 100644 --- a/spec/html2rss/web/feeds/responder_spec.rb +++ b/spec/html2rss/web/feeds/responder_spec.rb @@ -25,7 +25,7 @@ end before do - allow(Html2rss::Web::LocalConfig).to receive(:find).with('example').and_return(static_config) + allow(Html2rss::Web::Registry::Index.current).to receive(:config_for).with('example').and_return(static_config) allow(Html2rss::Web::Observability).to receive(:emit) allow(Html2rss::Web::SentryOps).to receive(:emit_failure_telemetry) end diff --git a/spec/html2rss/web/feeds/source_resolver_spec.rb b/spec/html2rss/web/feeds/source_resolver_spec.rb index fcb43545f..a17725228 100644 --- a/spec/html2rss/web/feeds/source_resolver_spec.rb +++ b/spec/html2rss/web/feeds/source_resolver_spec.rb @@ -27,7 +27,7 @@ def resolved_tuple(resolved) end before do - allow(Html2rss::Web::LocalConfig).to receive(:find).with('legacy').and_return(config) + allow(Html2rss::Web::Registry::Index.current).to receive(:config_for).with('legacy').and_return(config) end it 'normalizes the static source into shared generator input without forcing a default strategy', diff --git a/spec/html2rss/web/local_config_spec.rb b/spec/html2rss/web/local_config_spec.rb index aa0a10d35..8eaaa60c6 100644 --- a/spec/html2rss/web/local_config_spec.rb +++ b/spec/html2rss/web/local_config_spec.rb @@ -51,33 +51,11 @@ def account_token(snapshot) expect(titles_for('example.json', 'example.rss', 'example.xml')).to eq(%w[Example Example Example]) end - it 'falls back to embedded configs when the feed is not in local yaml' do - stub_const('Html2rss::Configs', Module.new do - def self.find_by_name(_name); end - end) - stub_const('Html2rss::Configs::ConfigNotFound', Class.new(StandardError)) - allow(Html2rss::Configs) - .to receive(:find_by_name) - .with('support.apple.com/en_gb_ht201222') - .and_return({ channel: { title: 'Apple security releases' } }) - allow(described_class).to receive(:snapshot).and_return(empty_snapshot) - - config = described_class.find('support.apple.com/en_gb_ht201222.rss') - - expect(config).to include(channel: { title: 'Apple security releases' }) - end - - it 'returns not found for malformed embedded config paths instead of depending on gem error messages' do - stub_const('Html2rss::Configs', Module.new do - def self.find_by_name(_name); end - end) - stub_const('Html2rss::Configs::ConfigNotFound', Class.new(StandardError)) - allow(Html2rss::Configs).to receive(:find_by_name) + it 'returns not found for unknown feed ids' do allow(described_class).to receive(:snapshot).and_return(empty_snapshot) expect { described_class.find('/broken-name.rss') } .to raise_error(described_class::NotFound, "Did not find local feed config at 'broken-name'") - expect(Html2rss::Configs).not_to have_received(:find_by_name) end end diff --git a/spec/html2rss/web/registry/config_spec.rb b/spec/html2rss/web/registry/config_spec.rb new file mode 100644 index 000000000..e5edd9788 --- /dev/null +++ b/spec/html2rss/web/registry/config_spec.rb @@ -0,0 +1,93 @@ +# frozen_string_literal: true + +require 'fileutils' +require 'climate_control' +require 'spec_helper' + +require_relative '../../../../app' + +RSpec.describe Html2rss::Web::Registry::Config do + describe '.entry' do + let(:config_path) { File.join(Dir.pwd, 'tmp', 'missing-public-key-registries.yml') } + + before do + FileUtils.mkdir_p(File.dirname(config_path)) + File.write(config_path, <<~YAML) + precedence: + - official + registries: + official: + sync: + url: https://registry.test.example/registry-bundle.tar.gz + catalog: true + YAML + end + + after do + FileUtils.rm_f(config_path) + end + + it 'requires a pinned public key for sync-mode registries' do + ClimateControl.modify('REGISTRIES_CONFIG' => config_path) do + expect do + described_class.reload! + described_class.entry('official') + end.to raise_error(Html2rss::Web::Registry::Errors::ConfigError, /requires a pinned public_key/) + end + end + end + + describe 'sync policy parsing' do + let(:config_path) { File.join(Dir.pwd, 'tmp', 'sync-policy-registries.yml') } + + before do + FileUtils.mkdir_p(File.dirname(config_path)) + File.write(config_path, <<~YAML) + precedence: + - official + registries: + official: + sync: + channel: html2rss-official + pin_version: v2026.08.22 + max_version: v2026.08.21 + auto_promote: true + allowed_channel_domains: + - phys.org + catalog: true + public_key_id: html2rss:registry:2026 + public_key: | + -----BEGIN PUBLIC KEY----- + MCowBQYDK2VwAyEAiMbg/04MyC5azBdM/aeY0mNuA8JbP5/jOiNRwJ2KJHE= + -----END PUBLIC KEY----- + YAML + end + + after do + FileUtils.rm_f(config_path) + end + + it 'parses sync policy and domain allowlist fields', :aggregate_failures do + ClimateControl.modify('REGISTRIES_CONFIG' => config_path) do + described_class.reload! + entry = described_class.entry('official') + + expect(entry.sync_policy).to have_attributes( + pin_version: 'v2026.08.22', + max_version: 'v2026.08.21', + auto_promote: true + ) + expect(entry.allowed_channel_domains).to eq(['phys.org']) + end + end + + it 'defaults auto_promote to false for security' do + ClimateControl.modify('REGISTRIES_CONFIG' => nil) do + described_class.reload! + entry = described_class.entry('official') + + expect(entry.sync_policy.auto_promote).to be(false) + end + end + end +end diff --git a/spec/html2rss/web/registry/index_spec.rb b/spec/html2rss/web/registry/index_spec.rb new file mode 100644 index 000000000..a4a600058 --- /dev/null +++ b/spec/html2rss/web/registry/index_spec.rb @@ -0,0 +1,215 @@ +# frozen_string_literal: true + +require 'fileutils' +require 'spec_helper' + +require_relative '../../../../app' + +RSpec.describe Html2rss::Web::Registry::Index do + describe '#config_for' do + it 'returns registry configs by feed id' do + config = described_class.current.config_for('anthropic.com/news') + + expect(config).to include(channel: hash_including(title: 'Anthropic — News')) + end + + it 'returns nil for unknown ids' do + expect(described_class.current.config_for('missing.example/feed')).to be_nil + end + + it 'resolves feeds by alias' do # rubocop:disable RSpec/ExampleLength + fake_bundle = described_class::RegistryBundle.new( + registry_id: 'official', + manifest: nil, + configs: { + 'anthropic.com/news' => { channel: { title: 'Anthropic — News' } }, + 'anthropic.com/legacy-news' => { channel: { title: 'Anthropic — News' } } + }, + catalog_entries: [] + ) + allow(described_class.current).to receive(:loaded_bundles).and_return('official' => fake_bundle) + + expect(described_class.current.config_for('anthropic.com/legacy-news')).to include( + channel: hash_including(title: 'Anthropic — News') + ) + end + + it 'still resolves catalog-disabled registry feeds by id' do + config = described_class.current.config_for('secret.example/private') + + expect(config).to include(channel: hash_including(url: 'https://secret.example/private')) + end + end + + describe '.current' do + it 'reloads when manifest mtime changes' do + initial_index = described_class.current + allow(Html2rss::Web::Registry::Store).to receive(:manifest_mtime).and_return('2099-01-01T00:00:00Z') + + expect(described_class.current).not_to be(initial_index) + end + end + + describe '#catalog_rows' do + it 'includes registry rows with source and registry fields' do + rows = described_class.current.catalog_rows + anthropic = rows.find { it.fetch(:id) == 'anthropic.com/news' } + + expect(anthropic).to include( + source: 'registry', + registry: 'official', + path: '/anthropic.com/news.rss' + ) + end + + it 'omits catalog-disabled registries from the catalog API rows' do + rows = described_class.current.catalog_rows + + expect(rows.map { it.fetch(:id) }).not_to include('secret.example/private') + end + + it 'prefers the first registry in precedence for duplicate feed ids' do + rows = described_class.current.catalog_rows + deepmind = rows.find { it.fetch(:id) == 'deepmind.google/blog' } + + expect(deepmind.fetch(:registry)).to eq('official') + end + + it 'merges local feeds.yml rows after registry rows', :aggregate_failures do # rubocop:disable RSpec/ExampleLength + allow(Html2rss::Web::LocalConfig).to receive(:feeds).and_return( + 'team/releases' => { + directory: { title: 'Team Releases', summary: 'Internal release notes' }, + channel: { url: 'https://team.example/releases', title: 'Team Releases' } + } + ) + + rows = described_class.current.catalog_rows + local = rows.find { it.fetch(:id) == 'team/releases' } + + expect(local).to include( + source: 'local', + path: '/team/releases.rss', + directory: hash_including(title: 'Team Releases'), + channel: hash_including(url: 'https://team.example/releases') + ) + expect(local).not_to have_key(:registry) + end + end + + describe '#status' do + it 'reports loaded registry metadata' do + status = described_class.current.status + official = status.find { it.id == 'official' } + + expect(official).to have_attributes( + version: 'test-fixture', + mode: :path + ) + end + end + + describe 'allowed_channel_domains' do + let(:config_path) { File.join(Dir.pwd, 'tmp', 'domain-allowlist-registries.yml') } + + after do + FileUtils.rm_f(config_path) + end + + it 'allows suffix-matching channel domains via config load' do # rubocop:disable RSpec/ExampleLength + FileUtils.mkdir_p(File.dirname(config_path)) + File.write(config_path, <<~YAML) + precedence: + - official + registries: + official: + path: spec/fixtures/registries/official + catalog: true + allowed_channel_domains: + - anthropic.com + - deepmind.google + YAML + ENV['REGISTRIES_CONFIG'] = config_path + described_class.reload! + + expect(described_class.current.config_for('anthropic.com/news')).to include( + channel: hash_including(url: 'https://www.anthropic.com/news') + ) + end + + it 'rejects bundles with channel URLs outside the allowlist' do # rubocop:disable RSpec/ExampleLength + FileUtils.mkdir_p(File.dirname(config_path)) + File.write(config_path, <<~YAML) + precedence: + - official + registries: + official: + path: spec/fixtures/registries/official + catalog: true + allowed_channel_domains: + - blocked.example + YAML + ENV['REGISTRIES_CONFIG'] = config_path + described_class.reload! + + expect { described_class.current.config_for('anthropic.com/news') } + .to raise_error(Html2rss::Web::Registry::Errors::LoadError, /anthropic\.com/) + end + end + + describe 'embedded bundle resolution' do + let(:embedded_config_path) { File.join(Dir.pwd, 'tmp', 'embedded-registries.yml') } + let(:embedded_root) { File.join(Dir.pwd, 'tmp', 'embedded-root') } + + before do + FileUtils.mkdir_p(File.dirname(embedded_config_path)) + File.write( + embedded_config_path, + YAML.dump( + 'precedence' => ['official'], + 'registries' => { + 'official' => { + 'sync' => { 'channel' => 'html2rss-official' }, + 'catalog' => true, + 'public_key_id' => 'test-key', + 'public_key' => RegistrySyncTestHelpers::TEST_PUBLIC_KEY + } + } + ) + ) + ENV['REGISTRIES_CONFIG'] = embedded_config_path + ENV['REGISTRY_DATA_ROOT'] = File.join(Dir.pwd, 'tmp', 'empty-data-root') + ENV['REGISTRY_EMBEDDED_ROOT'] = embedded_root + FileUtils.rm_rf(ENV.fetch('REGISTRY_DATA_ROOT')) + FileUtils.rm_rf(embedded_root) + + embedded_official = File.join(embedded_root, 'official') + FileUtils.mkdir_p(embedded_official) + FileUtils.cp_r(File.join(RegistryTestHelpers::FIXTURES_ROOT, 'official', 'configs'), embedded_official) + FileUtils.cp( + File.join(RegistrySyncTestHelpers::SYNC_FIXTURES_ROOT, 'bundle', Html2rss::Registry::Manifest::MANIFEST_FILE), + File.join(embedded_official, Html2rss::Registry::Manifest::MANIFEST_FILE) + ) + manifest = Html2rss::Registry::Manifest.parse( + File.read(File.join(embedded_official, Html2rss::Registry::Manifest::MANIFEST_FILE)) + ) + Html2rss::Registry::Signer.sign!( + manifest, + key_pem: RegistrySyncTestHelpers::TEST_PRIVATE_KEY, + bundle_dir: embedded_official + ) + described_class.reload! + end + + after do + FileUtils.rm_f(embedded_config_path) + FileUtils.rm_rf(embedded_root) + FileUtils.rm_rf(ENV.fetch('REGISTRY_DATA_ROOT', nil)) + end + + it 'loads configs directly from the embedded root when no runtime sync data exists' do + expect(described_class.current.config_for('anthropic.com/news')).to include( + channel: hash_including(title: 'Anthropic — News') + ) + end + end +end diff --git a/spec/html2rss/web/registry/store_spec.rb b/spec/html2rss/web/registry/store_spec.rb new file mode 100644 index 000000000..d6cd8a77f --- /dev/null +++ b/spec/html2rss/web/registry/store_spec.rb @@ -0,0 +1,115 @@ +# frozen_string_literal: true + +require 'fileutils' +require 'json' +require 'digest' +require 'spec_helper' + +require_relative '../../../../app' + +RSpec.describe Html2rss::Web::Registry::Store do + let(:registry_id) { 'store-test' } + let(:data_root) { File.join(Dir.pwd, 'tmp', 'store-spec-data') } + + let(:embedded_root) { File.join(Dir.pwd, 'tmp', 'store-spec-embedded') } + + before do + ENV['REGISTRY_DATA_ROOT'] = data_root + ENV['REGISTRY_EMBEDDED_ROOT'] = embedded_root + FileUtils.rm_rf(data_root) + FileUtils.rm_rf(embedded_root) + end + + describe '.stage_bundle! and .promote_staged!' do + it 'stages a verified bundle and promotes it to active', :aggregate_failures do + source = build_bundle_dir('stage-source') + active_before = build_bundle_dir('active-before') + + FileUtils.mkdir_p(File.dirname(described_class.registry_dir(registry_id))) + FileUtils.cp_r(active_before, described_class.registry_dir(registry_id)) + + described_class.stage_bundle!(registry_id, source) + + expect(described_class.staged_present?(registry_id)).to be(true) + expect(described_class.staged_version(registry_id)).to eq('stage-source') + expect(described_class.bundle_present?(registry_id)).to be(true) + expect(read_manifest_version(described_class.registry_dir(registry_id))).to eq('active-before') + + described_class.promote_staged!(registry_id) + + expect(described_class.staged_present?(registry_id)).to be(false) + expect(read_manifest_version(described_class.registry_dir(registry_id))).to eq('stage-source') + end + end + + describe '.bundle_present?' do + it 'requires manifest.json instead of any non-empty directory' do + path = described_class.registry_dir(registry_id) + + FileUtils.rm_rf(path) + FileUtils.mkdir_p(path) + File.write(File.join(path, 'placeholder.txt'), 'content') + + expect(described_class.bundle_present?(registry_id)).to be(false) + + File.write(File.join(path, Html2rss::Registry::Manifest::MANIFEST_FILE), '{}') + expect(described_class.bundle_present?(registry_id)).to be(true) + end + end + + describe '.active_dir' do + let(:embedded_root) { File.join(Dir.pwd, 'tmp', 'store-spec-embedded') } + + before do + ENV['REGISTRY_EMBEDDED_ROOT'] = embedded_root + FileUtils.rm_rf(embedded_root) + end + + it 'returns nil when neither synced nor embedded bundle exists' do + expect(described_class.active_dir(registry_id)).to be_nil + end + + it 'falls back to embedded bundle directory when synced bundle is absent' do + embedded_dir = described_class.embedded_dir(registry_id) + FileUtils.mkdir_p(embedded_dir) + File.write(File.join(embedded_dir, Html2rss::Registry::Manifest::MANIFEST_FILE), '{}') + + expect(described_class.active_dir(registry_id)).to eq(embedded_dir) + end + + it 'prefers synced runtime bundle over embedded bundle' do + embedded_dir = described_class.embedded_dir(registry_id) + FileUtils.mkdir_p(embedded_dir) + File.write(File.join(embedded_dir, Html2rss::Registry::Manifest::MANIFEST_FILE), '{}') + + synced_dir = described_class.registry_dir(registry_id) + FileUtils.mkdir_p(synced_dir) + File.write(File.join(synced_dir, Html2rss::Registry::Manifest::MANIFEST_FILE), '{}') + + expect(described_class.active_dir(registry_id)).to eq(synced_dir) + end + end + + def build_bundle_dir(version) # rubocop:disable Metrics/MethodLength + dir = Dir.mktmpdir("registry-store-#{version}") + config_path = File.join(dir, 'configs', 'example.com', 'feed.yml') + FileUtils.mkdir_p(File.dirname(config_path)) + File.write(config_path, "channel:\n url: https://example.com/\n") + digest = Digest::SHA256.file(config_path).hexdigest + File.write( + File.join(dir, Html2rss::Registry::Manifest::MANIFEST_FILE), + { + format: 'registry.v1', + registry_id: 'store-test', + version:, + public_key_id: 'test', + files: { 'configs/example.com/feed.yml' => digest } + }.to_json + ) + dir + end + + def read_manifest_version(path) + JSON.parse(File.read(File.join(path, Html2rss::Registry::Manifest::MANIFEST_FILE))).fetch('version') + end +end diff --git a/spec/html2rss/web/registry/sync_spec.rb b/spec/html2rss/web/registry/sync_spec.rb new file mode 100644 index 000000000..d0ac20012 --- /dev/null +++ b/spec/html2rss/web/registry/sync_spec.rb @@ -0,0 +1,259 @@ +# frozen_string_literal: true + +require 'fileutils' +require 'climate_control' +require 'spec_helper' +require 'webmock/rspec' + +require_relative '../../../../app' + +RSpec.describe Html2rss::Web::Registry::Sync do + describe '.sync_url_for' do + it 'resolves the official release URL for sync-mode defaults' do + ClimateControl.modify('REGISTRIES_CONFIG' => nil) do + Html2rss::Web::Registry::Index.reload! + + expect(described_class.sync_url_for('official')).to eq( + Html2rss::Web::Registry::Config::OFFICIAL_RELEASE_URL + ) + end + end + end + + describe '.cli_exit_code', :registry_sync do + let(:download_url) { 'https://registry.test.example/registry-bundle.tar.gz' } + let(:tarball) { RegistrySyncTestHelpers.build_signed_tarball } + + before do + stub_request(:get, download_url) + .to_return(status: 200, body: tarball, headers: { 'Content-Type' => 'application/octet-stream' }) + end + + it 'returns 1 when a sync registry is not yet synced' do + expect(described_class.cli_exit_code).to eq(1) + end + + it 'returns 0 when all registries are synced and healthy' do + described_class.run(registry_id: 'official') + expect(described_class.cli_exit_code).to eq(0) + end + + it 'returns 1 when any registry has last_error' do + described_class.run(registry_id: 'official') + allow(Html2rss::Web::Registry::Store).to receive(:sync_state).and_return( + Html2rss::Web::Registry::Store::SyncState.new(last_error: 'Boom', last_sync_at: nil) + ) + + expect(described_class.cli_exit_code).to eq(1) + end + end + + describe '.run', :registry_sync do + let(:download_url) { 'https://registry.test.example/registry-bundle.tar.gz' } + let(:tarball) { RegistrySyncTestHelpers.build_signed_tarball } + + before do + stub_request(:get, download_url) + .to_return(status: 200, body: tarball, headers: { 'Content-Type' => 'application/octet-stream' }) + end + + it 'fetches, verifies, and stores a signed bundle', :aggregate_failures do + status = described_class.run(registry_id: 'official') + + expect(status.version).to eq('test-fixture') + expect(Html2rss::Web::Registry::Store.bundle_present?('official')).to be(true) + expect(Html2rss::Web::Registry::Index.current.config_for('anthropic.com/news')).to include( + channel: hash_including(title: 'Anthropic — News') + ) + end + + it 'supports dry-run verification without swapping the active bundle' do + expect do + described_class.run(registry_id: 'official', dry_run: true) + end.not_to(change { Html2rss::Web::Registry::Store.bundle_present?('official') }) + end + + it 'follows bounded redirects to allowed CDN hosts', :aggregate_failures do + cdn_url = 'https://release-assets.githubusercontent.com/registry-bundle.tar.gz' + stub_request(:get, download_url) + .to_return(status: 302, headers: { 'Location' => cdn_url }) + stub_request(:get, cdn_url) + .to_return(status: 200, body: tarball, headers: { 'Content-Type' => 'application/octet-stream' }) + + status = described_class.run(registry_id: 'official') + + expect(status.version).to eq('test-fixture') + expect(Html2rss::Web::Registry::Store.bundle_present?('official')).to be(true) + end + + it 'rejects redirects to disallowed hosts' do + stub_request(:get, download_url) + .to_return(status: 302, headers: { 'Location' => 'https://evil.example/bundle.tar.gz' }) + + expect { described_class.run(registry_id: 'official') } + .to raise_error(Html2rss::Web::Registry::Errors::SyncError, /host not allowed/i) + end + + it 'rejects excessive redirect chains' do + (1..6).each do |hop| + from = hop == 1 ? download_url : "#{download_url}?hop=#{hop - 1}" + to = "#{download_url}?hop=#{hop}" + stub_request(:get, from).to_return(status: 302, headers: { 'Location' => to }) + end + + expect { described_class.run(registry_id: 'official') } + .to raise_error(Html2rss::Web::Registry::Errors::SyncError, /redirect limit/i) + end + + it 'logs signature verification failures to the security logger' do + allow(Html2rss::Web::SecurityLogger).to receive(:log_registry_signature_failure) + stub_request(:get, download_url).to_return(status: 200, body: 'not-a-tarball') + + expect { described_class.run(registry_id: 'official') } + .to raise_error(Html2rss::Web::Registry::Errors::SyncError) + + expect(Html2rss::Web::SecurityLogger).not_to have_received(:log_registry_signature_failure) + end + + it 'keeps the previous bundle when sync fails' do + described_class.run(registry_id: 'official') + stub_request(:get, download_url).to_return(status: 500, body: 'fail') + + expect { described_class.run(registry_id: 'official') } + .to raise_error(Html2rss::Web::Registry::Errors::SyncError, /HTTP 500/) + + expect(Html2rss::Web::Registry::Store.bundle_present?('official')).to be(true) + end + end + + describe '.run' do + it 'rejects path-mode registries' do + expect { described_class.run(registry_id: 'official') } + .to raise_error(Html2rss::Web::Registry::Errors::SyncError, /path mode/) + end + end + + describe '.status', :registry_sync do + it 'includes sync metadata and last error state' do + row = described_class.status(registry_id: 'official').first + + expect(row).to have_attributes( + id: 'official', + mode: :sync, + sync_url: 'https://registry.test.example/registry-bundle.tar.gz', + staged_version: nil + ) + end + end + + describe 'sync policy', :registry_sync do + let(:download_url) { 'https://registry.test.example/registry-bundle.tar.gz' } + let(:tarball) { RegistrySyncTestHelpers.build_signed_tarball } + let(:policy_config_path) { File.join(Dir.pwd, 'tmp', 'sync-policy-official.yml') } + + before do + FileUtils.mkdir_p(File.dirname(policy_config_path)) + stub_request(:get, download_url) + .to_return(status: 200, body: tarball, headers: { 'Content-Type' => 'application/octet-stream' }) + end + + after do + FileUtils.rm_f(policy_config_path) + end + + def write_policy_config(yaml) + File.write(policy_config_path, yaml) + ENV['REGISTRIES_CONFIG'] = policy_config_path + Html2rss::Web::Registry::Index.reload! + end + + it 'stages verified bundles when auto_promote is false', :aggregate_failures do + write_policy_config( + RegistrySyncTestHelpers.policy_registry_yaml( + download_url:, + auto_promote: false, + sync_extra: { max_version: 'test-fixture' } + ) + ) + + status = described_class.run(registry_id: 'official') + + expect(Html2rss::Web::Registry::Store.staged_present?('official')).to be(true) + expect(Html2rss::Web::Registry::Store.bundle_present?('official')).to be(false) + expect(status.staged_version).to eq('test-fixture') + end + + it 'promotes staged bundles and emits catalog change telemetry', :aggregate_failures do + allow(Html2rss::Web::Observability).to receive(:emit) + + write_policy_config( + RegistrySyncTestHelpers.policy_registry_yaml(download_url:, auto_promote: false) + ) + + described_class.run(registry_id: 'official') + status = described_class.promote_staged!(registry_id: 'official') + + expect(status.version).to eq('test-fixture') + expect(Html2rss::Web::Registry::Store.staged_present?('official')).to be(false) + expect(Html2rss::Web::Observability).to have_received(:emit).with( + hash_including(event_name: 'registry.catalog_changed', outcome: 'success') + ) + end + + it 'rejects manifests newer than max_version' do + write_policy_config( + RegistrySyncTestHelpers.policy_registry_yaml( + download_url:, + auto_promote: true, + sync_extra: { max_version: '0.0.1' } + ) + ) + + expect { described_class.run(registry_id: 'official') } + .to raise_error(Html2rss::Web::Registry::Errors::SyncError, /exceeds max_version/) + end + end + + describe '.boot!' do + it 'does not run in the test environment' do + allow(described_class).to receive(:start_background_timer!) + + described_class.boot! + + expect(described_class).not_to have_received(:start_background_timer!) + end + end + + describe Html2rss::Web::Registry::ChannelResolver do + describe '.resolve' do + it 'uses the GitHub tag release API when pin_version is set', :aggregate_failures do # rubocop:disable RSpec/ExampleLength + tag_api = format( + described_class::OFFICIAL_GITHUB_TAG_RELEASES_API, + tag: 'v2026.08.22' + ) + download_url = 'https://release-assets.githubusercontent.com/registry-bundle.tar.gz' + stub_request(:get, tag_api).to_return( + status: 200, + body: { + assets: [{ name: described_class::OFFICIAL_ASSET_NAME, browser_download_url: download_url }] + }.to_json + ) + + definition = Html2rss::Web::Registry::Definition.new( + id: 'official', + mode: :sync, + path: nil, + sync_channel: Html2rss::Web::Registry::Config::DEFAULT_OFFICIAL_SYNC_CHANNEL, + sync_url: nil, + catalog: true, + public_key_id: 'html2rss:registry:2026', + public_key: nil, + sync_policy: Html2rss::Web::Registry::SyncPolicy.new('v2026.08.22', nil, false), + allowed_channel_domains: [] + ) + + expect(described_class.resolve(definition)).to eq(download_url) + end + end + end +end diff --git a/spec/smoke/docker_spec.rb b/spec/smoke/docker_spec.rb index 115d2b76b..c056a2624 100644 --- a/spec/smoke/docker_spec.rb +++ b/spec/smoke/docker_spec.rb @@ -112,4 +112,13 @@ def expect_json_feed_response(path) expect(body.dig('error', 'code')).to eq('FORBIDDEN') expect(body.dig('error', 'message')).to eq('Auto source feature is disabled') end + + it 'exposes the config catalog without authentication', :aggregate_failures do + response, payload = get_json('/api/v1/configs') + + expect(response).to be_a(Net::HTTPOK) + expect(payload.fetch('success')).to be(true) + expect(payload.dig('data', 'configs')).to be_an(Array) + expect(payload.dig('meta', 'catalog_version')).to eq(1) + end end diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 4a7811d74..25b4b1d90 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -8,13 +8,17 @@ require 'simplecov' SimpleCov.start do - add_filter '/spec/' - add_filter '/config/' + enable_coverage :branch + primary_coverage :branch - track_files '**/*.rb' + add_group 'App', 'app' + add_group 'Config', 'config' - minimum_coverage 80 unless ENV['OPENAPI'] - maximum_coverage_drop 5 unless ENV['OPENAPI'] + add_filter %r{/spec/} + add_filter %r{/config/} + + minimum_coverage line: 80, branch: 70 unless ENV['OPENAPI'] + maximum_coverage_drop line: 5, branch: 5 unless ENV['OPENAPI'] end end diff --git a/spec/support/openapi.rb b/spec/support/openapi.rb index 30ea6e7e1..766d9159a 100644 --- a/spec/support/openapi.rb +++ b/spec/support/openapi.rb @@ -233,5 +233,21 @@ else spec[:tags] = tags end + + paths = spec['paths'] || spec[:paths] + catalog_configs_required = paths&.dig('/configs', 'get', 'responses', '200', 'content', + 'application/json', 'schema', 'properties', 'data', 'properties', 'configs', + 'items', 'required') + catalog_configs_required&.delete('registry') if catalog_configs_required.is_a?(Array) + + root_registries = paths&.dig('/', 'get', 'responses', '200', 'content', + 'application/json', 'schema', 'properties', 'data', 'properties', 'instance', + 'properties', 'registries', 'items') + if root_registries.is_a?(Hash) + root_registries['properties']&.fetch('version', {})&.merge!('nullable' => true) + root_registries['properties']&.fetch('updated_at', {})&.merge!('nullable' => true) + root_registries['required']&.delete('version') + root_registries['required']&.delete('updated_at') + end end end diff --git a/spec/support/registry_sync.rb b/spec/support/registry_sync.rb new file mode 100644 index 000000000..3959ff761 --- /dev/null +++ b/spec/support/registry_sync.rb @@ -0,0 +1,100 @@ +# frozen_string_literal: true + +require 'fileutils' +require 'stringio' +require 'yaml' + +module RegistryTestHelpers + FIXTURES_ROOT = File.expand_path('../fixtures/registries', __dir__) + DEFAULT_REGISTRIES_CONFIG = File.join(FIXTURES_ROOT, 'registries.yml') + + module_function + + def configure_registry_fixtures! + ENV['REGISTRIES_CONFIG'] = DEFAULT_REGISTRIES_CONFIG + ENV['REGISTRY_DATA_ROOT'] = File.join(Dir.pwd, 'tmp', 'test-registry-data') + FileUtils.rm_rf(ENV.fetch('REGISTRY_DATA_ROOT', nil)) + end + + def reset_registry! + Html2rss::Web::Registry::Index.reload! + end +end + +module RegistrySyncTestHelpers + FIXTURE_KEYS_ROOT = File.expand_path('../fixtures/registries/keys', __dir__) + TEST_PUBLIC_KEY = File.read(File.join(FIXTURE_KEYS_ROOT, 'test-key.pub')) + TEST_PRIVATE_KEY = File.read(File.join(FIXTURE_KEYS_ROOT, 'test-key.pem')) + SYNC_FIXTURES_ROOT = File.expand_path('../fixtures/registries/sync', __dir__) + OFFICIAL_FIXTURES_ROOT = File.join(RegistryTestHelpers::FIXTURES_ROOT, 'official') + SYNC_REGISTRIES_CONFIG = File.join(SYNC_FIXTURES_ROOT, 'registries.yml') + SYNC_DATA_ROOT = File.join(Dir.pwd, 'tmp', 'sync-registry-data') + + module_function + + def configure_sync_registry! + ENV['REGISTRIES_CONFIG'] = SYNC_REGISTRIES_CONFIG + ENV['REGISTRY_DATA_ROOT'] = SYNC_DATA_ROOT + ENV['REGISTRY_SYNC_ALLOWED_HOSTS'] = 'registry.test.example,release-assets.githubusercontent.com' + FileUtils.rm_rf(SYNC_DATA_ROOT) + Html2rss::Web::Registry::Index.reload! + end + + def build_signed_tarball # rubocop:disable Metrics/MethodLength + bundle_dir = Dir.mktmpdir('signed-registry-bundle') + manifest_target = File.join(bundle_dir, Html2rss::Registry::Manifest::MANIFEST_FILE) + FileUtils.cp_r(File.join(OFFICIAL_FIXTURES_ROOT, 'configs'), File.join(bundle_dir, 'configs')) + FileUtils.cp( + File.join(SYNC_FIXTURES_ROOT, 'bundle', Html2rss::Registry::Manifest::MANIFEST_FILE), + manifest_target + ) + manifest = Html2rss::Registry::Manifest.parse(File.read(manifest_target)) + Html2rss::Registry::Signer.sign!(manifest, key_pem: TEST_PRIVATE_KEY, bundle_dir:) + + pack_bundle_dir(bundle_dir) + ensure + FileUtils.rm_rf(bundle_dir) + end + + def pack_bundle_dir(bundle_dir) + dir = File.dirname(tarball_path = File.join(Dir.mktmpdir('registry-sync-tarball'), 'bundle.tar.gz')) + env = { 'COPYFILE_DISABLE' => '1' } + success = system(env, 'tar', '--format=ustar', '-czf', tarball_path, '-C', bundle_dir, '.', exception: false) + raise "Failed to pack registry test bundle from #{bundle_dir}" unless success + + File.binread(tarball_path) + ensure + FileUtils.rm_rf(dir) if dir + end + + def policy_registry_yaml(download_url:, auto_promote:, sync_extra: {}) # rubocop:disable Metrics/MethodLength + sync = { 'url' => download_url }.merge(sync_extra.transform_keys(&:to_s)) + YAML.dump( + { + 'precedence' => ['official'], + 'registries' => { + 'official' => { + 'sync' => sync, + 'auto_promote' => auto_promote, + 'catalog' => true, + 'public_key_id' => 'test-key', + 'public_key' => TEST_PUBLIC_KEY + } + } + } + ) + end +end + +RSpec.configure do |config| + config.before do + RegistryTestHelpers.configure_registry_fixtures! + RegistryTestHelpers.reset_registry! + end + + config.before do |example| + next unless example.metadata[:registry_sync] + + RegistrySyncTestHelpers.configure_sync_registry! + end +end