Fix S3/SFTP job contracts, the JSON content type check and dead bulk_root_url

AGENTS.md

@@ -43,9 +43,12 @@ CI runs `bundle exec rspec` only. `rake default` also runs RuboCop, which report
 ## Rules
 
 - **Only endpoints documented at https://www.zerobounce.net/docs/.** Do not add a method for an endpoint the public docs do not describe, even if it exists on the server. Check before concluding something is undocumented: the S3 and SFTP bulk endpoints live under https://www.zerobounce.net/docs/api-integrations, not the main API reference.
-- **Four base URLs.** `api_root_url`, `bulk_api_root_url` (ends in `/v2/`), `bulk_root_url` (the S3/SFTP jobs, **no** `/v2`), and `list_evaluator_root_url` / `inbox_placement_root_url`. The last three are separate values rather than paths built from `bulk_api_root_url`, so overriding that alone leaves a test pointing at the live API.
-- **Where the live API and the docs disagree, follow the API and say so in a comment at the call site.** The ones found so far: `validatebatch` takes `timeout` as a JSON *string*; the Inbox Placement Tester authenticates with `Authorization: Bearer`, not `api_key`; its listing is `GET /tests`, not `/test`; `send-evaluated-file` takes `file_id` as a form field, not a file upload; the S3 and SFTP jobs require `return_url` and `file_name`, `validate_sftp_file` included; `/filters/*` answers a *success* with a bare `{"Message": ...}` body; the Warmup `check-domain-dmarc` takes an email address, not a domain; and a failed SMTP login in `test-smtp-configuration` comes back as an HTTP 500.
-- **`export_sftp_file` sends `export_path`**, not the `export_bucket` that `export_s3_file` sends.
+- **Four base URLs.** `api_root_url`, `bulk_api_root_url` (ends in `/v2/`), `bulk_root_url` (the S3/SFTP jobs, **no** `/v2`), and `list_evaluator_root_url` / `inbox_placement_root_url`. `list_evaluator_root_url` and `inbox_placement_root_url` are separate values rather than paths built from `bulk_api_root_url`, so overriding that alone leaves a test pointing at the live API. `bulk_root_url` is the exception: it defaults to `nil` and `BaseRequest.bulk_api_root_without_v2` then derives it from `bulk_api_root_url` with the `/v2` stripped. Set it (or `ZEROBOUNCE_BULK_ROOT_URL`) to move only the S3/SFTP jobs.
+- **Where the live API and the docs disagree, follow the API and say so in a comment at the call site.** The ones found so far: `validatebatch` takes `timeout` as a JSON *string*; the Inbox Placement Tester authenticates with `Authorization: Bearer`, not `api_key`; its listing is `GET /tests`, not `/test`, and that listing's page size parameter is `page_size`, not the camelCase `pageSize` the rest of that API uses (`pageSize` is accepted and ignored); `send-evaluated-file` takes `file_id` as a form field, not a file upload; the S3 and SFTP jobs require `return_url` and `file_name`, `validate_sftp_file` included; `/filters/*` answers a *success* with a bare `{"Message": ...}` body; the Warmup `check-domain-dmarc` takes an email address, not a domain; and a failed SMTP login in `test-smtp-configuration` comes back as an HTTP 500.
+- **`export_sftp_file` sends `export_path`**, not the `export_bucket` that `export_s3_file` sends. Both are optional; omitted, the export goes to the account's configured destination.
+- **The eight S3/SFTP file jobs go through `Priority3#bulk_root_file_job`.** It is the one place that enforces `file_name` and `return_url` and turns booleans into the strings `true`/`false`. `validate_s3_file` and `score_s3_file` use it too, so do not rebuild the parameter hash per endpoint.
+- **`listS3files` is on the bulkapi root, not `/v2`.** It is the only S3 call that is a GET, which is how it ended up on `bulk_get` (and a 404) once before.
+- **A JSON content type is matched by media type, not `==`.** The API types some responses `application/json; charset=utf-8`; `BaseRequest.__json_content_type__` is the only thing that should decide whether a body is parsed.
 - **`{"success": false}` on an HTTP 200 is a failure.** The API answers an unknown bulk file id that way; both `bulk_get` and `__parse_json_response__` raise on it.
 - **A body that is nothing but `Message` means different things per endpoint.** On the validate endpoints it is a rejection; on `/filters/add`, `/filters/delete` and `/filters/list` it is the success. `Request::FILTER_PATHS` is the exception list.
 - **`Gemfile.lock` pins the gem's own version.** A version bump must update both `lib/zerobounce/version.rb` and `Gemfile.lock`, or `bundle install --deployment` in CI fails.

CHANGELOG.md

@@ -3,7 +3,7 @@
 * `validate` accepts `domain_info`, `greylist_processing` and `callback_url`. An option left out is not sent; booleans go on the wire as the strings `true`/`false`. `callback_url` is where the greylist retry posts its result, so it is rejected without `greylist_processing`, and when blank, before any request is sent.
 * Add the rest of the documented bulk surface: `email_finder_s3_file`, `domain_search_s3_file`, and the SFTP/FTP jobs `validate_sftp_file`, `score_sftp_file`, `email_finder_sftp_file`, `domain_search_sftp_file` and `export_sftp_file`. These are on the bulkapi host **without** `/v2`. `validate_sftp_file`'s file name is optional - passing `nil` uses the input bucket from your ZeroBounce settings - and `export_sftp_file` takes `export_path`, not the `export_bucket` `export_s3_file` sends.
 * Add the Inbox Placement Tester: the API key goes on the query string, bodies are camelCase JSON, and unset fields are left out.
-* `Configuration` gains `bulk_root_url`, `list_evaluator_root_url` and `inbox_placement_root_url`. These are separate base URLs rather than paths built from `bulk_api_root_url`, so overriding that alone leaves them pointing at the live API.
+* `Configuration` gains `bulk_root_url`, `list_evaluator_root_url` and `inbox_placement_root_url`. `list_evaluator_root_url` and `inbox_placement_root_url` are separate base URLs rather than paths built from `bulk_api_root_url`, so overriding that alone leaves them pointing at the live API. `bulk_root_url` (`ZEROBOUNCE_BULK_ROOT_URL`) now actually reaches the request layer: set, it is the root for the S3 and SFTP/FTP jobs; unset, that root is still derived from `bulk_api_root_url` with the `/v2` stripped. It was previously written and never read, so the environment variable looked supported and did nothing.
 * `get_list_evaluator_result` uses the list evaluator's own base URL, encodes the file id as a single path segment, and rejects a blank id before a request is sent.
 * An error body carrying a `detail` key (the Inbox Placement Tester is a Django REST framework app) is now raised with that text.
 * Add `Zerobounce::Warmup`, a client for the Warmup API. It is a separate product with its own host (`https://warmup-api.zerobounce.net`), its own team-scoped key sent in an `API-Key` header, and all 26 documented endpoints, including the seed-list and statistics CSV downloads.
@@ -14,6 +14,14 @@
   * The Inbox Placement Tester authenticates with an `Authorization: Bearer` header, not an `api_key` parameter, so the key no longer reaches a query string that gets logged. Its listing is `GET /tests`; `/test` answers `Method "GET" not allowed`.
   * `send_evaluated_file` takes the list evaluator's `file_id` as a form field, not a file upload, so its argument is now a file id.
   * The S3 and SFTP jobs require `file_name` and `return_url`, `validate_sftp_file` included, although the docs list `file_name` as optional for that one.
+* Fixed: `list_s3_files` asked for `bulkapi/v2/listS3files`. Like every other S3 and SFTP/FTP call it lives on the bulkapi **root**, so the request 404ed.
+* Fixed: a response typed `application/json; charset=utf-8` skipped JSON parsing, because the content type was compared with `==` against the bare media type. The raw body came back as a `String`, so an `{"error": ...}` payload read as a success instead of raising. The media type is now matched on its own, ignoring the parameters after the semicolon. Affects the bulk file status, get and delete calls, the AI scoring, email finder and domain search bulk jobs, and `list_s3_files`.
+* Fixed: `validate_s3_file` and `score_s3_file` took `return_url` through `**options` and never required it, so the live API rejected the job. They now go through the same helper as the other six S3/SFTP jobs, which requires `file_name` and `return_url` and sends `has_header_row` and `remove_duplicates` in the documented shape. `has_header_row` is now a named keyword and defaults to `false`.
+* Fixed: `export_s3_file` demanded `export_bucket`. It is optional - omitted, the export goes to the bucket in your ZeroBounce settings - the way `export_sftp_file`'s `export_path` already was.
+* Fixed: `list_inbox_placement_tests` sent `pageSize`, which the API accepts and then ignores; the parameter it honours is `page_size`. It also accepts `created_after` / `created_before` (sent as `createdAfter` / `createdBefore`), and rejects a page size outside 1-100 before a request is sent.
+* Fixed: `send_email_finder_file` now requires `first_name_column` or `full_name_column`. With neither, the finder has no name to build a local part from.
+* Fixed: `create_inbox_placement_test` checks the documented bounds before spending inbox-test credits: `name` at most 64 characters, `test_type` one of 1, 2 or 3, and `sender_email` required when `test_type` is 1.
+* Fixed: `find_email`'s `first_name` is optional. `/guessformat` only requires a domain or a company name; without a name it answers with the domain's format. It is still the first positional argument, so existing calls are unchanged, and `find_domain` and the deprecated `guessformat` keep their own signatures.
 * Fixed: `add_filter`, `delete_filter` and `list_filters` raised on success. They answer with a bare `{"Message": "..."}` body, which everywhere else means the request was rejected.
 * Fixed: `{"success": false}` on an HTTP 200 raises instead of being returned as a result. The API answers an unknown bulk file id that way.
 * Security: a Warmup API key with surrounding whitespace or control characters is rejected, because the transport would otherwise quote it unredacted in its own error; a Warmup base URL must be `https://`; and the Warmup SMTP test masks the username and password out of every error it raises.
@@ -34,6 +42,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `score_email` for `GET` and `POST /scoring`.
 - Bulk email finder, bulk domain search, list evaluator, filters, and S3 (`validate_s3_file`, `score_s3_file`, `export_s3_file`, `list_s3_files`).
 - `Zerobounce::Client` so each API key stays on its own thread. Module methods still share `Zerobounce.config`.
+- `Configuration#bulk_root_url` / `ZEROBOUNCE_BULK_ROOT_URL` is read by the request layer and overrides the root used by the S3 and SFTP/FTP jobs.
 
 ### Changed
 
@@ -49,6 +58,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Empty and non-JSON bodies are a short error. A repeated `Error:` / `[Error]:` prefix is removed.
 - `validate_batch` raises for an account-level failure (HTTP 200, error for address `all`, such as an invalid API key or no credits).
 - Bulk email finder and domain search send `has_header_row` when it is false. A column index of `0` is sent when the caller passes it.
+- `list_s3_files` is sent to the bulkapi root instead of `/v2`, where it 404ed.
+- A `Content-Type` of `application/json; charset=utf-8` is parsed as JSON, so an error payload raises instead of being returned as a raw `String`.
+- `validate_s3_file` and `score_s3_file` require `return_url`, which the live API rejects the job without.
+- `export_s3_file`'s `export_bucket` is optional and falls back to the account's settings.
+- `list_inbox_placement_tests` sends `page_size` (the API ignores `pageSize`), accepts `created_after` / `created_before`, and bounds the page size to 1-100.
+- `send_email_finder_file` requires `first_name_column` or `full_name_column`.
+- `create_inbox_placement_test` enforces the documented `name`, `test_type` and `sender_email` rules.
+- `find_email`'s `first_name` is optional.
 - An empty API key is rejected before a request is sent.
 - `validate` timeout must be 3–60 seconds. `validate_batch` timeout must be 10–120 seconds.
 

README.md

@@ -41,6 +41,9 @@ Zerobounce.configure do |config|
   # Optional: Set custom API URLs
   config.api_root_url = 'https://api.zerobounce.net/v2'
   config.bulk_api_root_url = 'https://bulkapi.zerobounce.net/v2'
+  # Optional: the root the S3 and SFTP/FTP jobs use (the bulk host without
+  # /v2). Left unset it is derived from bulk_api_root_url.
+  config.bulk_root_url = 'https://bulkapi.zerobounce.net'
 end
 ```
 
@@ -82,6 +85,8 @@ Create a `.env` file in your project root:
 ZEROBOUNCE_API_KEY=your_api_key_here
 ZEROBOUNCE_API_URL=https://api.zerobounce.net/v2
 ZEROBOUNCE_BULK_API_URL=https://bulkapi.zerobounce.net/v2
+# Optional. Only needed to point the S3 and SFTP/FTP jobs somewhere of their own.
+ZEROBOUNCE_BULK_ROOT_URL=https://bulkapi.zerobounce.net
 ```
 
 The gem will automatically load these environment variables when initialized. No additional configuration needed in your code.
@@ -92,6 +97,8 @@ Set environment variables in your system:
 export ZEROBOUNCE_API_KEY=your_api_key_here
 export ZEROBOUNCE_API_URL=https://api.zerobounce.net/v2
 export ZEROBOUNCE_BULK_API_URL=https://bulkapi.zerobounce.net/v2
+# Optional. Only needed to point the S3 and SFTP/FTP jobs somewhere of their own.
+export ZEROBOUNCE_BULK_ROOT_URL=https://bulkapi.zerobounce.net
 ```
 
 
@@ -101,6 +108,9 @@ export ZEROBOUNCE_BULK_API_URL=https://bulkapi.zerobounce.net/v2
 If you don't specify `api_root_url` or `bulk_api_root_url`, the gem will use the following defaults:
 - **Main API**: `https://api.zerobounce.net/v2`
 - **Bulk API**: `https://bulkapi.zerobounce.net/v2`
+- **S3 and SFTP/FTP jobs**: `https://bulkapi.zerobounce.net` — the bulk host *without* `/v2`
+
+`bulk_root_url` (`ZEROBOUNCE_BULK_ROOT_URL`) is the one base URL that is derived rather than defaulted: left unset, it is `bulk_api_root_url` with the trailing `/v2` stripped, so pointing `bulk_api_root_url` at a mock server moves the S3 and SFTP/FTP endpoints with it. Set it to send those jobs somewhere on their own.
 
 These defaults are defined in the `Zerobounce::ApiUrls` class constants:
 - `Zerobounce::ApiUrls::DEFAULT_URL` - Main API URL (Global)
@@ -149,8 +159,8 @@ end
 #### Environment Variable Priority
 The configuration follows this priority order (highest to lowest):
 1. Explicitly set in code (`config.api_root_url = '...'`)
-2. Environment variables (`ZEROBOUNCE_API_URL`, `ZEROBOUNCE_BULK_API_URL`)
-3. Default constants (`ApiUrls::DEFAULT_URL`, `ApiUrls::BULK_DEFAULT_URL`)
+2. Environment variables (`ZEROBOUNCE_API_URL`, `ZEROBOUNCE_BULK_API_URL`, `ZEROBOUNCE_BULK_ROOT_URL`)
+3. Default constants (`ApiUrls::DEFAULT_URL`, `ApiUrls::BULK_DEFAULT_URL`); `bulk_root_url` falls back to `bulk_api_root_url` without `/v2`
 
 Credits
 ```ruby
@@ -645,6 +655,10 @@ Zerobounce.find_email("John", company_name: "Zero Bounce")
 
 # With additional name information for better accuracy
 Zerobounce.find_email("John", domain: "zerobounce.net", middle_name: "Deere", last_name: "Doe")
+
+# The first name is optional. Without one the call answers with the domain's
+# address format rather than one person's address.
+Zerobounce.find_email(domain: "zerobounce.net")
 ```
 
 Find Domain Information

lib/zerobounce.rb

@@ -588,7 +588,8 @@ def guessformat(domain, first_name: '', middle_name: '', last_name: '')
     #
     # @option [String] domain Domain to search within (e.g. example.com). Required if company_name is not provided.
     # @option [String] company_name Company name to search within (e.g. Example). Required if domain is not provided.
-    # @option [String] first_name First name of target.
+    # @param [String] first_name First name of target. Optional: left out, the
+    #   call answers with the domain's format rather than one person's address.
     # @option [String] middle_name Middle name of target.
     # @option [String] last_name Last name of target.
     #
@@ -601,15 +602,19 @@ def guessformat(domain, first_name: '', middle_name: '', last_name: '')
     #   "did_you_mean": "",
     #   "failure_reason": ""
     # }
-    def find_email(first_name, domain: '', company_name: '', middle_name: '', last_name: '')
+    # first_name is optional: /guessformat only requires a domain or a company
+    # name, and answers the domain's format when no name is given. It stays the
+    # first positional argument, so every existing call keeps working.
+    def find_email(first_name = nil, domain: '', company_name: '', middle_name: '', last_name: '')
       # Validate that exactly one of domain or company_name is provided
       if (domain.nil? || domain.empty?) && (company_name.nil? || company_name.empty?)
         raise ArgumentError, "Either domain or company_name must be provided"
       elsif !(domain.nil? || domain.empty?) && !(company_name.nil? || company_name.empty?)
         raise ArgumentError, "Only one of domain or company_name can be provided"
       end
 
-      params = { first_name: first_name }
+      params = {}
+      params[:first_name] = first_name unless first_name.nil? || first_name.to_s.empty?
       params[:domain] = domain unless domain.nil? || domain.empty?
       params[:company_name] = company_name unless company_name.nil? || company_name.empty?
       params[:middle_name] = middle_name unless middle_name.nil? || middle_name.empty?

lib/zerobounce/base_request.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require 'json'
+require 'zerobounce/api_urls'
 require 'zerobounce/response_error'
 
 # https://github.com/rest-client/rest-client
@@ -177,9 +178,31 @@ def self.ipt_request(method, path, params = {}, json: nil)
       __parse_json_response__(response)
     end
 
+    # Whether a Content-Type names a JSON body.
+    #
+    # The API types most responses 'application/json' but some
+    # 'application/json; charset=utf-8'. Compared with ==, the second kind
+    # skipped JSON parsing entirely, so an {"error": ...} payload came back as
+    # a raw String and read as a success. Only the media type decides; the
+    # parameters after the semicolon are ignored.
+    def self.__json_content_type__(content_type)
+      media_type = content_type.to_s.split(';', 2).first.to_s.strip.downcase
+      media_type == 'application/json' || media_type.end_with?('+json')
+    end
+
+    # The root for the S3 and SFTP/FTP jobs: the bulkapi host WITHOUT /v2.
+    #
+    # +bulk_root_url+ (ZEROBOUNCE_BULK_ROOT_URL) wins when it is set, so these
+    # endpoints can be pointed at a mock server on their own. Left unset it is
+    # derived from +bulk_api_root_url+ with the /v2 stripped, so overriding
+    # that one alone still moves these endpoints with it.
     def self.bulk_api_root_without_v2
-      root = Zerobounce.configuration.bulk_api_root_url.to_s.sub(%r{/v2/?$}, '')
-      __root_without_trailing_slashes__(root)
+      configured = Zerobounce.configuration.bulk_root_url.to_s.strip
+      return __root_without_trailing_slashes__(configured) unless configured.empty?
+
+      derived = Zerobounce.configuration.bulk_api_root_url.to_s.sub(%r{/v2/?$}, '')
+      derived = ApiUrls::BULK_ROOT_URL if derived.strip.empty?
+      __root_without_trailing_slashes__(derived)
     end
 
     # Whether a "success" field means failure. The API sends it as a JSON