v2.11.0 Fixes & Features

docs/HA_SYNC.md

@@ -0,0 +1,74 @@
+# HA Sync
+
+The REST API package supports High Availability (HA) synchronization, allowing you to automatically replicate your
+REST API settings from one pfSense host to one or more remote peers. This is especially useful in HA deployments where
+you want all nodes to share the same REST API configuration without having to configure each host manually.
+
+!!! Important
+    HA sync only synchronizes the **REST API package settings** (such as authentication methods, allowed interfaces,
+    JWT expiration, and other options found under the REST API settings). It does not synchronize the rest of the
+    pfSense configuration. Use pfSense's built-in HA/XMLRPC configuration sync for the base system configuration.
+
+## How It Works
+
+When HA sync is enabled and one or more sync hosts are defined, the source host will push its REST API settings to each
+configured peer whenever those settings change. Each peer receives the sync request and applies the incoming settings
+to its own REST API configuration.
+
+## Requirements
+
+Before enabling HA sync, ensure the following:
+
+- The REST API package is installed on every host that will participate in synchronization.
+- A dedicated user exists on **every** peer defined in the sync hosts list. This user must hold the `page-all`
+  privilege and must use the same username and password across all hosts.
+- Each peer is reachable from the source host over HTTPS (the webConfigurator port).
+- If certificate validation is enabled, each peer's webConfigurator must present a certificate trusted by the source
+  host.
+
+!!! Warning
+    It is expected that the source host and each of its peers are as identical as possible. Changes between REST API 
+    versions or pfSense versions may make the synced configurations incompatible with the remote peers. Additionally, 
+    some settings (such as the allowed interfaces) are configuration-specific, and it is expected that those interfaces 
+    also exist on each remote peer.
+
+## Configuring HA Sync on the Source Host
+
+Follow these steps on the host that will send its REST API settings to peers:
+
+1. In the webConfigurator, navigate to **System > REST API > Settings**.
+2. Enable the **HA Sync** toggle. This allows the host to both send sync requests to peers and accept incoming sync
+   requests from peers.
+3. In the **HA Sync Hosts** field, enter the IP address or hostname of each remote peer that should receive this host's
+   REST API settings. You may define up to 10 hosts.
+4. Set the **HA Sync Username** to the username of the account used to authenticate sync requests. This user must exist on
+   every peer and hold the `page-all` privilege.
+5. Set the **HA Sync Password** to the password for the HA username. This must match on all peers.
+6. (Recommended) Leave **HA Sync Validate Certs** enabled to validate each peer's webConfigurator certificate. This
+   helps mitigate Man-in-the-Middle attacks. If you disable it, certificate validation will be skipped.
+7. Save your changes.
+
+## Configuring HA Sync on Remote Peers
+
+Repeat the following on **every** remote peer listed in the source host's **HA Sync Hosts** field:
+
+1. In the webConfigurator, navigate to **System > REST API > Settings**.
+2. Enable the **HA Sync** toggle. A peer will reject incoming sync requests by default unless the HA Sync toggle is enabled on that
+   peer.
+3. Ensure the HA sync user (with the `page-all` privilege) exists on the peer and uses the same username and password
+   as configured on the source host.
+4. Save your changes.
+
+Once HA Sync is enabled on both the source host and all peers, any change to the source host's REST API settings will
+automatically be synchronized to each configured peer.
+
+## Configuring HA Sync via the REST API
+
+All HA sync options are also available through the REST API settings endpoint, so you can automate HA sync configuration
+as part of your provisioning workflow. The relevant settings include:
+
+- `ha_sync` — Enables or disables HA sync on the host. Must be enabled on both the sending and receiving hosts.
+- `ha_sync_validate_certs` — Enables or disables certificate validation when syncing to peers.
+- `ha_sync_hosts` — A list of IP addresses or hostnames to sync REST API settings to.
+- `ha_sync_username` — The username used to authenticate sync requests. Must hold the `page-all` privilege on all peers.
+- `ha_sync_password` — The password for the HA sync user. Must match across all peers.

mkdocs.yml

@@ -13,6 +13,7 @@ nav:
       - Working with HATEOAS: WORKING_WITH_HATEOAS.md
       - Common Control Parameters: COMMON_CONTROL_PARAMETERS.md
       - GraphQL: GRAPHQL.md
+      - HA Sync: HA_SYNC.md
       - Securing API Access: SECURING_API_ACCESS.md
       - Limitations & FAQs: LIMITATIONS_AND_FAQS.md
       - API Reference: https://pfrest.org/api-docs/

pfSense-pkg-RESTAPI/files/usr/local/pkg/RESTAPI/Caches/PrivilegesCache.inc

@@ -17,7 +17,13 @@ class PrivilegesCache extends Cache {
      */
     public function get_data_to_cache(): array {
         # Use a variable to keep track of the privileges associated with this endpoint
-        $privs = [];
+        $privs = [
+            'page-system-restapi-documentation' => [
+                'name' => 'WebCfg - System:  RESTAPI Documentation',
+                'descr' => 'Allow access to the /api/v2/documentation page.',
+                'match' => ['api/v2/documentation*'],
+            ],
+        ];
 
         # Obtain privileges for each Endpoint class
         foreach (get_classes_from_namespace('\\RESTAPI\\Endpoints\\') as $endpoint_class) {

pfSense-pkg-RESTAPI/files/usr/local/pkg/RESTAPI/Core/Field.inc

@@ -83,6 +83,9 @@ class Field {
      *      Make this a required field. If false, a valid $default, $default_callable or $allow_null must be set.
      * @param bool $unique
      *      If true, $value must be unique among all parent Model objects (only for parent Models with $many set to true).
+     * @param bool $skip_unique_validation
+     *      If true, fields marked as unique will skip enforcement of the unique constraint. This is necessary for
+     *      fields that use $unique for schema or indexing purposes, but do not need the constraint enforced.
      * @param mixed $default
      *      The default $value for this field if none is assigned. The default specified here should be static. If the
      *      default for this field is dynamic in nature, use a $default_callable instead.
@@ -136,6 +139,7 @@ class Field {
         public string $type = '',
         public bool $required = false,
         public bool $unique = false,
+        public bool $skip_unique_validation = false,
         public mixed $default = null,
         public string $default_callable = '',
         public array $choices = [],
@@ -725,8 +729,8 @@ class Field {
      * @throws ServerError When $unique is enabled, but the parent model $context does not have $many enabled
      */
     private function check_field_unique(mixed $value, ?ModelSet $modelset = null): void {
-        # Only check this if `unique` is enabled
-        if ($this->unique) {
+        # Only check this if `unique` is enabled and the unique constraint is enforced
+        if ($this->unique and !$this->skip_unique_validation) {
             # Throw an error if unique is set but the model context is not a 'many' model.
             if (!$this->context->many) {
                 throw new ServerError(

pfSense-pkg-RESTAPI/files/usr/local/pkg/RESTAPI/Core/Model.inc

@@ -1870,6 +1870,17 @@ class Model {
         $this->apply();
     }
 
+    /**
+     * Prepares representation data before validating a `replace_all()` request, including dry runs.
+     * Override this method to supply operation-specific defaults without changing other CRUD actions.
+     * @param array $data The requested Model representations.
+     * @param ModelSet $initial_objects The existing Model objects before replacement.
+     * @return array The prepared Model representations to validate and replace.
+     */
+    protected function pre_validate_replace_all(array $data, ModelSet $initial_objects): array {
+        return $data;
+    }
+
     /**
      * Performs steps and processes required to apply changes after the `replace_all()` method runs. This can be
      * overridden to add custom steps to apply after replace all actions only. Defaults to simply calling `apply()`.
@@ -2446,6 +2457,9 @@ class Model {
         # if needed.
         $initial_objects = $this->query(parent_id: $this->parent_id);
 
+        # Prepare operation-specific defaults before validation or any configuration changes
+        $data = $this->pre_validate_replace_all($data, $initial_objects);
+
         # Obtain any Models that are deemed protected to ensure they do not removed in the next step.
         $new_objects = new ModelSet();
         if ($this->protected_model_query) {
@@ -2456,21 +2470,28 @@ class Model {
         # do not mutate or persist the configuration. This still allows clients to detect duplicate or invalid
         # entries before applying the replacement.
         if ($this->dry_run) {
-            foreach ($data as $representation_object) {
-                unset($representation_object['id']);
-                $model_object = new $this(data: $representation_object, skip_init: true, client: $this->client);
-                # Assign a synthetic ID for uniqueness comparisons during validation
-                $model_object->id = count($new_objects->model_objects);
-                $model_object->validate(modelset: $new_objects);
-                $new_objects->model_objects[] = $model_object;
+            # Use the root signature so self-references resolve against the simulated replacement set
+            $new_objects = new ModelSet(model_objects: $new_objects->model_objects, signature: $this->get_class_fqn());
 
-                # In the event that the model can reference itself (e.g. FirewallAlias), ensure the model cache is
-                # updated so ForeignModelField can accurately find referenced objects.
-                self::get_model_cache()::cache_modelset($new_objects);
-            }
+            try {
+                foreach ($data as $representation_object) {
+                    # Discard stale queries and indexes before caching the growing replacement set
+                    self::clear_model_cache();
+                    self::get_model_cache()::cache_modelset($new_objects);
+
+                    unset($representation_object['id']);
+                    $model_object = new $this(data: $representation_object, skip_init: true, client: $this->client);
+                    # Assign a synthetic ID for uniqueness comparisons during validation
+                    $model_object->id = count($new_objects->model_objects);
+                    $model_object->validate(modelset: $new_objects);
+                    $new_objects->model_objects[] = $model_object;
+                }
 
-            self::clear_model_cache();
-            return $new_objects;
+                return $new_objects;
+            } finally {
+                # Never retain simulated objects, including when validation throws an error
+                self::clear_model_cache();
+            }
         }
 
         # Before beginning validation, empty the stored values in the config path. Retain protected Models (if any).

pfSense-pkg-RESTAPI/files/usr/local/pkg/RESTAPI/Fields/Base64Field.inc

@@ -18,6 +18,7 @@ class Base64Field extends Field {
      * @param bool $required If `true`, this field is required to have a value at all times.
      * @param bool $unique If `true`, this field must be unique from all other parent model objects. Enabling this
      * option requires the Model $context to be set AND the Model $context must have a `config_path` set.
+     * @param bool $skip_unique_validation If `true`, skip uniqueness enforcement while retaining the `unique` metadata.
      * @param mixed|null $default Assign a default string value to assign this Field if no value is present.
      * @param string $default_callable Defines a callable method that should be called to populate the default value
      * for this field. It is strongly encouraged to use a default callable when the default is variable and may change
@@ -72,6 +73,7 @@ class Base64Field extends Field {
     public function __construct(
         bool $required = false,
         bool $unique = false,
+        bool $skip_unique_validation = false,
         mixed $default = null,
         string $default_callable = '',
         array $choices = [],
@@ -101,6 +103,7 @@ class Base64Field extends Field {
             type: 'string',
             required: $required,
             unique: $unique,
+            skip_unique_validation: $skip_unique_validation,
             default: $default,
             default_callable: $default_callable,
             choices: $choices,

pfSense-pkg-RESTAPI/files/usr/local/pkg/RESTAPI/Fields/BooleanField.inc

@@ -27,6 +27,7 @@ class BooleanField extends RESTAPI\Core\Field {
      * @param bool $required If `true`, this field is required to have a value at all times.
      * @param bool $unique If `true`, this field must be unique from all other parent model objects. Enabling this
      * option requires the Model $context to be set AND the Model $context must have a `config_path` set.
+     * @param bool $skip_unique_validation If `true`, skip uniqueness enforcement while retaining the `unique` metadata.
      * @param mixed|null $default Assign a default string value to assign this Field if no value is present.
      * @param string $default_callable Defines a callable method that should be called to populate the default value
      * for this field. It is strongly encouraged to use a default callable when the default is variable and may change
@@ -70,6 +71,7 @@ class BooleanField extends RESTAPI\Core\Field {
     public function __construct(
         bool $required = false,
         bool $unique = false,
+        bool $skip_unique_validation = false,
         mixed $default = null,
         string $default_callable = '',
         bool $allow_empty = false,
@@ -95,6 +97,7 @@ class BooleanField extends RESTAPI\Core\Field {
             type: 'boolean',
             required: $required,
             unique: $unique,
+            skip_unique_validation: $skip_unique_validation,
             default: $default,
             default_callable: $default_callable,
             allow_empty: $allow_empty,

pfSense-pkg-RESTAPI/files/usr/local/pkg/RESTAPI/Fields/DateTimeField.inc

@@ -19,6 +19,7 @@ class DateTimeField extends Field {
      * @param bool $required If `true`, this field is required to have a value at all times.
      * @param bool $unique If `true`, this field must be unique from all other parent model objects. Enabling this
      * option requires the Model $context to be set AND the Model $context must have a `config_path` set.
+     * @param bool $skip_unique_validation If `true`, skip uniqueness enforcement while retaining the `unique` metadata.
      * @param mixed|null $default Assign a default string value to assign this Field if no value is present.
      * @param string $default_callable Defines a callable method that should be called to populate the default value
      * for this field. It is strongly encouraged to use a default callable when the default is variable and may change
@@ -78,6 +79,7 @@ class DateTimeField extends Field {
     public function __construct(
         bool $required = false,
         bool $unique = false,
+        bool $skip_unique_validation = false,
         mixed $default = null,
         string $default_callable = '',
         array $choices = [],
@@ -117,6 +119,7 @@ class DateTimeField extends Field {
             type: 'string',
             required: $required,
             unique: $unique,
+            skip_unique_validation: $skip_unique_validation,
             default: $default,
             default_callable: $default_callable,
             choices: $choices,

pfSense-pkg-RESTAPI/files/usr/local/pkg/RESTAPI/Fields/FilterAddressField.inc

@@ -22,6 +22,7 @@ class FilterAddressField extends InterfaceField {
      * @param bool $required If `true`, this field is required to have a value at all times.
      * @param bool $unique If `true`, this field must be unique from all other parent model objects. Enabling this
      * option requires the Model $context to be set AND the Model $context must have a `config_path` set.
+     * @param bool $skip_unique_validation If `true`, skip uniqueness enforcement while retaining the `unique` metadata.
      * @param mixed|null $default Assign a default string value to assign this Field if no value is present.
      * @param string $default_callable Defines a callable method that should be called to populate the default value
      * for this field. It is strongly encouraged to use a default callable when the default is variable and may change
@@ -90,6 +91,7 @@ class FilterAddressField extends InterfaceField {
     public function __construct(
         bool $required = false,
         bool $unique = false,
+        bool $skip_unique_validation = false,
         mixed $default = null,
         string $default_callable = '',
         public bool $allow_ipaddr = true,
@@ -128,6 +130,7 @@ class FilterAddressField extends InterfaceField {
         parent::__construct(
             required: $required,
             unique: $unique,
+            skip_unique_validation: $skip_unique_validation,
             default: $default,
             default_callable: $default_callable,
             interface_query: $interface_query,

pfSense-pkg-RESTAPI/files/usr/local/pkg/RESTAPI/Fields/FloatField.inc

@@ -16,6 +16,7 @@ class FloatField extends Field {
      * @param bool $required If `true`, this field is required to have a value at all times.
      * @param bool $unique If `true`, this field must be unique from all other parent model objects. Enabling this
      * option requires the Model $context to be set AND the Model $context must have a `config_path` set.
+     * @param bool $skip_unique_validation If `true`, skip uniqueness enforcement while retaining the `unique` metadata.
      * @param mixed|null $default Assign a default string value to assign this Field if no value is present.
      * @param string $default_callable Defines a callable method that should be called to populate the default value
      * for this field. It is strongly encouraged to use a default callable when the default is variable and may change
@@ -70,6 +71,7 @@ class FloatField extends Field {
     public function __construct(
         bool $required = false,
         bool $unique = false,
+        bool $skip_unique_validation = false,
         mixed $default = null,
         string $default_callable = '',
         array $choices = [],
@@ -98,6 +100,7 @@ class FloatField extends Field {
             type: 'double', // gettype() identifies floats as doubles
             required: $required,
             unique: $unique,
+            skip_unique_validation: $skip_unique_validation,
             default: $default,
             default_callable: $default_callable,
             choices: $choices,

pfSense-pkg-RESTAPI/files/usr/local/pkg/RESTAPI/Fields/ForeignModelField.inc

@@ -47,6 +47,7 @@ class ForeignModelField extends Field {
      * @param bool $required If `true`, this field is required to have a value at all times.
      * @param bool $unique If `true`, this field must be unique from all other parent model objects. Enabling this
      * option requires the Model $context to be set AND the Model $context must have a `config_path` set.
+     * @param bool $skip_unique_validation If `true`, skip uniqueness enforcement while retaining the `unique` metadata.
      * @param mixed|null $default Assign a default string value to assign this Field if no value is present.
      * @param string $default_callable Defines a callable method that should be called to populate the default value
      * for this field. It is strongly encouraged to use a default callable when the default is variable and may change
@@ -106,6 +107,7 @@ class ForeignModelField extends Field {
         public array $allowed_keywords = [],
         bool $required = false,
         bool $unique = false,
+        bool $skip_unique_validation = false,
         mixed $default = null,
         string $default_callable = '',
         array $choices = [],
@@ -144,6 +146,7 @@ class ForeignModelField extends Field {
             type: $model_field === 'id' ? 'integer' : $this->models[0]->$model_field->type,
             required: $required,
             unique: $unique,
+            skip_unique_validation: $skip_unique_validation,
             default: $default,
             default_callable: $default_callable,
             choices: $choices,