diff --git a/.agents/skills/netifyd/references/nethsecurity-wiring.md b/.agents/skills/netifyd/references/nethsecurity-wiring.md index 1876e961f..9c0870344 100644 --- a/.agents/skills/netifyd/references/nethsecurity-wiring.md +++ b/.agents/skills/netifyd/references/nethsecurity-wiring.md @@ -83,29 +83,44 @@ lists. `procd_set_param file /etc/netifyd.conf` means procd restarts the service `/usr/sbin/dpi` is the apply path, in order: -1. `/usr/sbin/dpi-config` — reads `/etc/config/dpi`, writes `/etc/netifyd/netify-proc-flow-actions.json` -2. `/usr/sbin/dpi-nft` — writes the `dpi_actions` / `dpi_dummy` chains into +1. `/usr/libexec/ns-dpi/dpi-migrate` — converts a `/etc/config/dpi` still on the pre-appgroup schema + (`dpi.migrate_schema`, a no-op when there is nothing to migrate). It also runs from the + `21_dpi_migrate` uci-default; running it here covers restored backups and configs synced by an HA + peer, where uci-defaults don't run +2. `/usr/sbin/dpi-config` — reads `/etc/config/dpi`, writes `/etc/netifyd/netify-proc-flow-actions.json` +3. `/usr/sbin/dpi-nft` — writes the `dpi_actions` / `dpi_dummy` chains into `/usr/share/nftables.d/table-pre/` -3. `/etc/init.d/netifyd reload` +4. `/etc/init.d/netifyd reload` + +Steps 1 and 2 run under `/var/lock/dpi.lock`, shared with the iface hotplug, so the migration and the +generation never run twice at once. Current generator behaviour worth knowing before touching it: -- `valid_actions` is `block`, `bulk`, `best_effort`, `video`, `voice`; anything else, or a disabled rule, - is skipped. -- A rule's raw `criteria` takes precedence over generated source/application/protocol/category matching. - Double quotes in it are rewritten to single quotes. **No `;` is appended to raw criteria** — generated - criteria get one, raw ones don't. Same for entries in the global `exemptions` array. -- A `device` naming a VLAN device causes `vlan_id == N && ` to be prepended — including to raw criteria, - so a rule carrying both its own `criteria` and a `device` gets the term whether or not it already has one. -- A hidden `analyzed` action (`detection_guessed || detection_complete;`) is appended last, labelling every - analysed flow. -- `dpi.config.firewall_exemption` optionally pre-fills `exemptions` with every firewall interface IP; - `exemption` sections add more, and an exemption whose criteria is an object ID is expanded to its IPs. +- The first action is a hidden `analyzed` one, generated by `dpi-config` and never stored in UCI: + priority `0`, `halt_on_match: false`, criteria `detection_guessed || detection_complete;`. It labels every + analysed flow and lets the evaluation continue to the user rules. +- User rules are emitted with their UCI `priority` (dense, from `1`) and `halt_on_match: true`: the first + matching rule wins. Actions are `block` and `allow`; `allow` sets a label no nft rule acts on, so it only + shields its flows from the block rules below it. A rule with no `enabled` option is skipped. +- Rules without a raw `criteria` (the ones created by the API) get it from `dpi.build_rule_criteria`: + `source` ORed, ANDed with the expanded `appgroup` members; `ns_match_all` with no source becomes `*`. +- Rules with a raw `criteria` (hand-written, or frozen by the migration) have double quotes rewritten to + single quotes and `;` appended when missing: an unterminated expression is a syntax error. +- A rule that would emit an empty criteria is skipped, because an empty criteria takes the agent down. +- A rule still on the pre-appgroup schema (`device`, `application`, `protocol` or `category` set) is + skipped with a warning, never emitted: read as a current rule, its `source` alone would block every flow + of those hosts. The migration in step 1 converts it first. +- The global `exemptions` are always pre-filled with every firewall interface IP, so DPI rules can not + lock the administrator out; `/etc/hotplug.d/iface/90-dpi-exemption` re-runs `dpi-config` on `ifup` / + address `ifupdate` and does `netifyd reload` only if the JSON changed and the agent is running (so HA + backup nodes are left alone). They are the only entries in `exemptions`. - `dpi.config.log_blocked` drives the nft log rule in `dpi-nft`, not any netify logging target. Enforcement is nft reading conntrack labels, not netifyd acting directly: netifyd classifies and sets a -label via its `ctlabel` target, and `dpi_actions` (hook prerouting, `filter + 10`) rejects labelled -traffic (`netify-blocked`) or DSCP-marks it (the QoS labels — the DPI→qosify bridge). +label via its `ctlabel` target, and `dpi_actions` (hook prerouting, `filter + 10`) rejects the flows +labelled `netify-blocked`. The chain still DSCP-marks the QoS labels, which no DPI action sets any more: +they are the DPI→qosify bridge, left in place. ## Conntrack labels @@ -121,6 +136,7 @@ across an upgrade keep their labels, so **never reassign an existing bit** — t | 4 | `best_effort` | | 5 | `video` | | 6 | `voice` | +| 7 | `netify-allowed` | `netify-analyzed` is consumed by no nft rule and read only by `ns.flows` for display — it is a diagnostic that tells support whether netifyd actually analysed a flow. "Nothing reads it in code" is not a reason to diff --git a/packages/ns-api/README.md b/packages/ns-api/README.md index 382effb66..0c5ea2b03 100644 --- a/packages/ns-api/README.md +++ b/packages/ns-api/README.md @@ -3548,104 +3548,118 @@ Response example: ## ns.dpi -Manage netifyd DPI engine. +Manage DPI rules and application groups. Rules are evaluated in priority order and the first rule +matching a flow wins. -### list-applications +### list-application-catalog -List application and protocols: +Return the raw application catalog, `/etc/netifyd/netify-application-catalog.json`: ```bash -api-cli ns.dpi list-applications +api-cli ns.dpi list-application-catalog ``` -Filtering is provided out of the box, searching in both name and category: +The response is `{"values": [...]}`, with the catalog entries as downloaded; `{}` if the catalog is +missing. + +### list-protocol-catalog + +Same as `list-application-catalog`, for `/etc/netifyd/netify-protocol-catalog.json`: ```bash -api-cli ns.dpi list-applications --data '{"search": "apple"}' +api-cli ns.dpi list-protocol-catalog ``` -Data can be limited and paginated by using the `limit` and `page` parameters: +### list-appgroup-catalog + +List the applications and protocols an application group can be built with, grouped by category: ```bash -api-cli ns.dpi list-applications --data '{"limit": 10, "page": 3}' +api-cli ns.dpi list-appgroup-catalog ``` -**PLEASE NOTE**: `category` field can be missing in some applications/protocols. - Example response: + ```json { "values": { - "data": [ - { - "id": 10392, - "name": "netify.apple-siri", - "type": "application", - "category": { - "name": "business" - } - }, + "applications": [ { - "id": 10706, - "name": "netify.apple-id", - "type": "application" - }, - { - "id": 10152, - "name": "netify.appnexus", - "type": "application", - "category": { - "name": "advertiser" - } - }, - { - "id": 142, - "name": "WhatsApp", - "type": "protocol", - "category": { - "name": "messaging" - } - }, - { - "id": 238, - "name": "Apple/Push", - "type": "protocol" + "tag": "portal", + "label": "Portal", + "selectable": true, + "items": [ + { + "id": "netify.qq", + "label": "Tencent QQ", + "selectable": true, + "logo": "https://static.netify.ai/logos/q/q//dd/icon.png?v=4" + }, + { + "id": "netify.premium-only", + "label": "Premium Only", + "selectable": false + } + ] }, { - "id": 246, - "name": "WhatsApp/Call", - "type": "protocol", - "category": { - "name": "voip" - } + "tag": "", + "label": "", + "selectable": false, + "items": [ + { + "id": "netify.brand-new", + "label": "netify.brand-new", + "selectable": true + } + ] } ], - "meta": { - "last_page": 2, - "total": 12 - } + "protocols": [ + { + "tag": "file-server", + "label": "File Server", + "selectable": true, + "items": [ + { + "id": "FTP", + "label": "FTP Control", + "selectable": true + } + ] + } + ] } } ``` -### list-popular +- item `selectable`: the engine has the signature loaded, so the item can be used in a group. Without a + subscription most applications are not selectable +- item `id`: the value to pass to `add-appgroup`; only valid when `selectable` is `true` +- group `selectable`: the category can be used as a group member (`application_categories` or + `protocol_categories`). The group with an empty `tag` collects uncategorised items and is never + selectable as a whole +- `logo` is omitted when the catalog has no icon +- groups and items are sorted by label, the uncategorised group last -List popular applications and protocols: +Inactive applications, deprecated protocols and the `unknown` protocol are left out. -```bash -api-cli ns.dpi list-popular -``` +The call fails when the engine is not running or a catalog has not been downloaded yet. -Data can be limited and paginated by using the `limit` and `page` parameters: +### list-appgroups + +List the application groups, ordered by name: ```bash -api-cli ns.dpi list-popular --data '{"limit": 3, "page": 2}' +api-cli ns.dpi list-appgroups ``` -**PLEASE NOTE**: +Optional parameters: `search` (matched against the name), `limit` and `page`. Without `limit` every group +is returned: -- `category` field can be missing in some applications/protocols. -- `missing` field is true when the application/protocol is not available in the current netifyd database. +```bash +api-cli ns.dpi list-appgroups --data '{"search": "business", "limit": 10, "page": 2}' +``` Example response: @@ -3654,116 +3668,84 @@ Example response: "values": { "data": [ { - "id": 10392, - "name": "netify.apple-siri", - "type": "application", - "category": { - "name": "business" - }, - "missing": false - }, - { - "id": 142, - "name": "WhatsApp", - "type": "protocol", - "category": { - "name": "messaging" - }, - "missing": false - }, - { - "name": "whatsapp", - "missing": true - }, - { - "id": 238, - "name": "Apple/Push", - "type": "protocol", - "missing": false + "id": "ns_1a2b3c4d5", + "name": "Business and technology services", + "applications": ["netify.amazon", "netify.apple"], + "application_categories": ["cybersecurity"], + "protocols": ["HTTP/Connect"], + "protocol_categories": ["games"], + "used": true, + "matches": ["dpi/ns_9f8e7d6c5"] } ], "meta": { - "last_page": 3, - "total": 8 + "last_page": 1, + "total": 1 } } } ``` +`used` and `matches` list the rules referencing the group. -### list-rules +### add-appgroup -List created rules: +Create an application group: ```bash -api-cli ns.dpi list-rules +api-cli ns.dpi add-appgroup --data '{ + "name": "Business and technology services", + "applications": ["netify.amazon"], + "application_categories": ["cybersecurity"], + "protocols": ["HTTP/Connect"], + "protocol_categories": ["games"] +}' ``` +- `name`: mandatory, up to 64 characters, unique among the groups regardless of case +- `applications`, `protocols`: names as reported by the engine, i.e. the `id` of a selectable item of + `list-appgroup-catalog` (`HTTP/Connect`, not the catalog tag `http-connect`) +- `application_categories`, `protocol_categories`: category tags +- the four lists are optional, but the group must have at least one member + +Members are checked against the engine and the downloaded catalogs; the check is skipped when neither is +available. + Example response: ```json { - "values": [ - { - "config-name": "ns_3869dc35", - "enabled": true, - "device": "eth4", - "interface": "GREEN_1", - "action": "block", - "criteria": [ - { - "id": 156, - "name": "netify.spotify", - "type": "application", - "category": { - "name": "streaming-media" - } - }, - { - "id": 10119, - "name": "netify.adobe", - "type": "application", - "category": { - "name": "business" - } - } - ] - }, - { - "config-name": "ns_f1c6e9e0", - "enabled": false, - "interface": "eth4", - "action": "block", - "criteria": [ - { - "id": 196, - "name": "HTTP/S", - "type": "protocol", - "category": { - "name": "web" - } - } - ] - } - ] + "id": "ns_1a2b3c4d5" } ``` -### add-rule +### edit-appgroup -Add DPI rule: +Same payload as `add-appgroup`, plus the `id` of the group. Lists left out are emptied: ```bash -api-cli ns.dpi add-rule --data '{"enabled": false, "device": "eth4", "applications": [], "protocols": ["HTTP/S"]}' +api-cli ns.dpi edit-appgroup --data '{ + "id": "ns_1a2b3c4d5", + "name": "Business", + "applications": ["netify.amazon", "netify.apple"] +}' +``` + +Example response: + +```json +{ + "id": "ns_1a2b3c4d5" +} ``` -Rundown of required parameters: +### delete-appgroup -- `enabled`: `true` or `false` -- `device`: device name, e.g. `eth4` -- `applications`: list of application names, e.g. `["netify.spotify", "netify.adobe"]`, refer to `list-applications` - api. -- `protocols`: list of protocol names, e.g. `["HTTP/S"]`, refer to `list-applications` api. +Delete an application group: + +```bash +api-cli ns.dpi delete-appgroup --data '{"id": "ns_1a2b3c4d5"}' +``` Example response: @@ -3773,110 +3755,157 @@ Example response: } ``` -### delete-rule +A group referenced by a rule can't be deleted. -Delete DPI rule: +### list-rules + +List the rules in priority order: ```bash -api-cli nd.dpi delete-rule --data '{"config-name": "ns_f1c6e9e0"}' +api-cli ns.dpi list-rules ``` -Required parameters: +Rules with `ns_visible '0'` are left out. -- `config-name`: rule name, refer to `list-rules` api. +- `managed`: the rule was created through the API. Unmanaged rules, e.g. the ones migrated from the + previous schema, carry their raw `criteria` and can be renamed, enabled, disabled, reordered and + deleted, but not edited +- `match_all`: the rule matches every flow instead of naming application groups +- `index`: position of the rule in the list Example response: ```json { - "message": "success" + "values": [ + { + "id": "ns_3869dc35", + "name": "Block streaming", + "enabled": true, + "action": "block", + "source": ["192.168.1.0/24"], + "appgroups": [ + { + "id": "ns_1a2b3c4d5", + "name": "Streaming services" + } + ], + "match_all": false, + "managed": true, + "index": 0 + }, + { + "id": "ns_9d40be71", + "name": "Allow everything", + "enabled": true, + "action": "allow", + "source": [], + "appgroups": [], + "match_all": true, + "managed": true, + "index": 1 + }, + { + "id": "ns_f1c6e9e0", + "name": "Migrated rule 1", + "enabled": true, + "action": "block", + "source": [], + "appgroups": [], + "match_all": false, + "managed": false, + "criteria": "local_ip == 192.168.100.22 && app == 'netify.facebook';", + "index": 2 + } + ] } ``` -### edit-rule +### add-rule -Edit DPI rule: +Add a rule: ```bash -api-cli ns.dpi edit-rule --data '{"config-name": "ns_f1c6e9e0", "enabled": true, "device": "eth4", "applications": ["netify.spotify", "netify.adobe"], "protocols": []}' +api-cli ns.dpi add-rule --data '{ + "name": "Block streaming", + "enabled": true, + "action": "block", + "source": ["192.168.1.0/24"], + "appgroups": ["ns_1a2b3c4d5"], + "match_all": false, + "position": "bottom" +}' ``` -Rundown of required parameters: -- `config-name`: rule name, refer to `list-rules` api. -- `enabled`: `true` or `false` -- `device`: device name, e.g. `eth4` -- `applications`: list of application names, e.g. `["netify.spotify", "netify.adobe"]`, refer to `list-applications` - api. -- `protocols`: list of protocol names, e.g. `["HTTP/S"]`, refer to `list-applications` api. +- `name`: mandatory, up to 64 characters +- `enabled`: mandatory, `true` or `false` +- `action`: mandatory, `block` or `allow`. `allow` stops the evaluation, so the rules below it don't apply + to the matching flows +- `source`: addresses, networks or ranges, IPv4 or IPv6, e.g. + `["192.168.1.1", "192.168.1.0/24", "10.0.0.10-10.0.0.20"]`. Empty or omitted matches every host +- `appgroups`: ids of the application groups to match, see `list-appgroups`. Mandatory unless + `match_all` is `true` +- `match_all`: match every flow, optionally narrowed by `source`. `appgroups` must be empty. Defaults to + `false` +- `position`: `top` or `bottom` of the list. Defaults to `bottom` Example response: ```json { - "message": "success" + "id": "ns_3869dc35" } ``` -### list-devices +### edit-rule -List available devices to be added to DPI rules: +Same payload as `add-rule` without `position`, plus the `id` of the rule. The priority is not changed, +use `order-rules` to move the rule. Only managed rules can be edited: ```bash -api-cli ns.dpi list-devices +api-cli ns.dpi edit-rule --data '{ + "id": "ns_3869dc35", + "name": "Block streaming", + "enabled": false, + "action": "block", + "source": [], + "appgroups": ["ns_1a2b3c4d5"], + "match_all": false +}' ``` Example response: ```json { - "values": [ - { - "interface": "GREEN_1", - "device": "eth0" - }, - { - "interface": "GREEN_2", - "device": "eth4" - } - ] + "id": "ns_3869dc35" } ``` -### list-exemptions +### delete-rule -List configured global exemptions: +Delete a rule. The remaining rules are renumbered: ```bash -api-cli ns.dpi list-exemptions +api-cli ns.dpi delete-rule --data '{"id": "ns_f1c6e9e0"}' ``` Example response: + ```json { - "values": [ - { - "config-name": "cfg024ffe", - "enabled": true, - "criteria": "192.168.1.1", - "description": "my ex" - } - ] + "message": "success" } ``` -### add-exemption -Add global exemption: +### rename-rule + +Rename a rule, managed or not: ```bash -api-cli ns.dpi add-rule --data '{"criteria": "192.168.1.1", "description": "my host", "enabled": true}' +api-cli ns.dpi rename-rule --data '{"id": "ns_f1c6e9e0", "name": "Legacy rule"}' ``` -Rundown of required parameters: - -- `enabled`: `true` or `false` -- `critera`: an IP address like `192.168.1.1` -- `description`: an optional description - Example response: ```json @@ -3885,34 +3914,30 @@ Example response: } ``` -It can raise a validation error if the criteria is duplicated. Example: +### enable-rule + +Enable a rule, managed or not: + +```bash +api-cli ns.dpi enable-rule --data '{"id": "ns_f1c6e9e0"}' +``` + +Example response: ```json { - "validation": { - "errors": [ - { - "parameter": "criteria", - "message": "criteria_already_exists", - "value": "192.168.1.3" - } - ] - } + "message": "success" } ``` -### delete-exemption +### disable-rule -Delete global exemption rule: +Disable a rule, managed or not: ```bash -api-cli nd.dpi delete-exemption --data '{"config-name": "ns_f1c6e9e0"}' +api-cli ns.dpi disable-rule --data '{"id": "ns_f1c6e9e0"}' ``` -Required parameters: - -- `config-name`: exemption name, refer to `list-exemptions` api. - Example response: ```json @@ -3921,25 +3946,21 @@ Example response: } ``` -### edit-rule +### order-rules -Edit global exemption: +Set the priority of the rules from the given order: ```bash -api-cli ns.dpi edit-rule --data '{"config-name": "ns_f1c6e9e0", "criteria": "192.168.1.1", "description": "my host", "enabled": true}' +api-cli ns.dpi order-rules --data '{"order": ["ns_f1c6e9e0", "ns_3869dc35"]}' ``` -Rundown of required parameters: -- `config-name`: rule name, refer to `list-rules` api. -- `enabled`: `true` or `false` -- `critera`: an IP address like `192.168.1.1` -- `description`: an optional description +`order` must list every rule exactly once, including the ones with `ns_visible '0'`. Example response: ```json { - "message": "success" + "values": ["ns_f1c6e9e0", "ns_3869dc35"] } ``` diff --git a/packages/ns-api/files/ns.dpi b/packages/ns-api/files/ns.dpi index 84c3dadc5..aa6a6d002 100755 --- a/packages/ns-api/files/ns.dpi +++ b/packages/ns-api/files/ns.dpi @@ -1,7 +1,7 @@ #!/usr/bin/python3 # -# Copyright (C) 2023 Nethesis S.r.l. +# Copyright (C) 2026 Nethesis S.r.l. # SPDX-License-Identifier: GPL-2.0-only # @@ -15,78 +15,79 @@ cmd = sys.argv[1] if cmd == 'list': print(json.dumps({ - 'list-applications': { - 'search': 'str', - 'limit': 32, - 'page': 32 - }, 'list-rules': {}, 'add-rule': { + 'name': 'str', 'enabled': False, - 'device': 'str', - 'applications': 'str', - 'protocols': 'str' - }, - 'delete-rule': { - 'config-name': 'str' + 'action': 'str', + 'source': [], + 'appgroups': [], + 'match_all': False, + 'position': 'str' }, 'edit-rule': { - 'config-name': 'str', + 'id': 'str', + 'name': 'str', 'enabled': False, - 'device': 'str', - 'applications': 'str', - 'protocols': 'str' + 'action': 'str', + 'source': [], + 'appgroups': [], + 'match_all': False + }, + 'delete-rule': { + 'id': 'str' }, - 'add-exemption': { - 'criteria': 'str', - 'description': 'str', - 'enabled': False + 'rename-rule': { + 'id': 'str', + 'name': 'str' }, - 'delete-exemption': { - 'config-name': 'str' + 'enable-rule': { + 'id': 'str' }, - 'edit-exemption': { - 'config-name': 'str', - 'criteria': 'str', - 'description': 'str', - 'enabled': False + 'disable-rule': { + 'id': 'str' }, - 'list-exemptions': {}, - 'list-devices': {}, - 'list-popular': { + 'order-rules': { + 'order': ['ns_1234', 'ns_5678'] + }, + 'list-application-catalog': {}, + 'list-protocol-catalog': {}, + 'list-appgroup-catalog': {}, + 'list-appgroups': { + 'search': 'str', 'limit': 32, 'page': 32 }, - 'list-application-categories': {}, - 'list-application-catalog': {}, - 'list-protocol-categories': {}, - 'list-protocol-catalog': {} + 'add-appgroup': { + 'name': 'str', + 'applications': [], + 'application_categories': [], + 'protocols': [], + 'protocol_categories': [] + }, + 'edit-appgroup': { + 'id': 'str', + 'name': 'str', + 'applications': [], + 'application_categories': [], + 'protocols': [], + 'protocol_categories': [] + }, + 'delete-appgroup': { + 'id': 'str' + } })) elif cmd == 'call': action = sys.argv[2] e_uci = EUci() try: - if action == 'list-application-categories': - try: - with open('/etc/netifyd/netify-application-categories.json', 'r') as f: - content = json.load(f) - print(json.dumps({'values': content})) - except Exception: - print(json.dumps({})) - elif action == 'list-application-catalog': + if action == 'list-application-catalog': try: with open('/etc/netifyd/netify-application-catalog.json', 'r') as f: content = json.load(f) print(json.dumps({'values': content})) except Exception: print(json.dumps({})) - elif action == 'list-protocol-categories': - try: - with open('/etc/netifyd/netify-protocol-categories.json', 'r') as f: - content = json.load(f) - print(json.dumps({'values': content})) - except Exception: - print(json.dumps({})) elif action == 'list-protocol-catalog': try: with open('/etc/netifyd/netify-protocol-catalog.json', 'r') as f: @@ -94,45 +95,65 @@ elif cmd == 'call': print(json.dumps({'values': content})) except Exception: print(json.dumps({})) - elif action == 'list-applications': - data = json.JSONDecoder().decode(sys.stdin.read()) - result = dpi.list_applications(data.get('search', None), data.get('limit', None), data.get('page', 1)) - print(json.dumps({'values': result})) + elif action == 'list-appgroup-catalog': + try: + print(json.dumps({'values': dpi.list_appgroup_catalog()})) + except Exception: + print(json.dumps(utils.generic_error('catalog_not_available'))) elif action == 'list-rules': print(json.dumps({'values': dpi.list_rules(e_uci)})) elif action == 'add-rule': - data = json.JSONDecoder().decode(sys.stdin.read()) - dpi.add_rule(e_uci, data['enabled'], data['device'], 'block', data['applications'], data['protocols']) - print(json.dumps({'message': 'success'})) - elif action == 'delete-rule': - data = json.JSONDecoder().decode(sys.stdin.read()) - dpi.delete_rule(e_uci, data['config-name']) - print(json.dumps({'message': 'success'})) + data = json.load(sys.stdin) + config_name = dpi.add_rule(e_uci, data['name'], data['enabled'], data['action'], + data.get('source', []), data.get('appgroups', []), + data.get('position', 'bottom'), data.get('match_all', False)) + print(json.dumps({'id': config_name})) elif action == 'edit-rule': - data = json.JSONDecoder().decode(sys.stdin.read()) - dpi.edit_rule(e_uci, data['config-name'], data['enabled'], data['device'], 'block', data['applications'], - data['protocols']) + data = json.load(sys.stdin) + config_name = dpi.edit_rule(e_uci, data['id'], data['name'], data['enabled'], data['action'], + data.get('source', []), data.get('appgroups', []), + data.get('match_all', False)) + print(json.dumps({'id': config_name})) + elif action == 'delete-rule': + data = json.load(sys.stdin) + dpi.delete_rule(e_uci, data['id']) print(json.dumps({'message': 'success'})) - elif action == 'add-exemption': - data = json.JSONDecoder().decode(sys.stdin.read()) - dpi.add_exemption(e_uci, data['criteria'], data['description'], data['enabled']) + elif action == 'rename-rule': + data = json.load(sys.stdin) + dpi.rename_rule(e_uci, data['id'], data['name']) print(json.dumps({'message': 'success'})) - elif action == 'delete-exemption': - data = json.JSONDecoder().decode(sys.stdin.read()) - dpi.delete_exemption(e_uci, data['config-name']) + elif action == 'enable-rule': + data = json.load(sys.stdin) + dpi.enable_rule(e_uci, data['id']) print(json.dumps({'message': 'success'})) - elif action == 'edit-exemption': - data = json.JSONDecoder().decode(sys.stdin.read()) - dpi.edit_exemption(e_uci, data['config-name'], data['criteria'], data['description'], data['enabled']) + elif action == 'disable-rule': + data = json.load(sys.stdin) + dpi.disable_rule(e_uci, data['id']) print(json.dumps({'message': 'success'})) - elif action == 'list-exemptions': - print(json.dumps({'values': dpi.list_exemptions(e_uci)})) - elif action == 'list-devices': - print(json.dumps({'values': dpi.list_devices(e_uci)})) - elif action == 'list-popular': - data = json.JSONDecoder().decode(sys.stdin.read()) - result = dpi.list_popular(e_uci, data.get('limit', None), data.get('page', 1)) + elif action == 'order-rules': + data = json.load(sys.stdin) + print(json.dumps({'values': dpi.order_rules(e_uci, data['order'])})) + elif action == 'list-appgroups': + data = json.load(sys.stdin) + result = dpi.list_appgroups(e_uci, data.get('search', None), data.get('limit', None), + data.get('page', 1)) print(json.dumps({'values': result})) + elif action == 'add-appgroup': + data = json.load(sys.stdin) + config_name = dpi.add_appgroup(e_uci, data['name'], data.get('applications', []), + data.get('application_categories', []), data.get('protocols', []), + data.get('protocol_categories', [])) + print(json.dumps({'id': config_name})) + elif action == 'edit-appgroup': + data = json.load(sys.stdin) + config_name = dpi.edit_appgroup(e_uci, data['id'], data['name'], data.get('applications', []), + data.get('application_categories', []), data.get('protocols', []), + data.get('protocol_categories', [])) + print(json.dumps({'id': config_name})) + elif action == 'delete-appgroup': + data = json.load(sys.stdin) + dpi.delete_appgroup(e_uci, data['id']) + print(json.dumps({'message': 'success'})) except KeyError as e: print(json.dumps(utils.validation_error(e.args[0], 'required'))) except json.JSONDecodeError: diff --git a/packages/ns-dpi/Makefile b/packages/ns-dpi/Makefile index dc28ddbb3..134747449 100644 --- a/packages/ns-dpi/Makefile +++ b/packages/ns-dpi/Makefile @@ -62,6 +62,7 @@ endef define Package/ns-dpi/install $(INSTALL_DIR) $(1)/usr/sbin + $(INSTALL_DIR) $(1)/usr/libexec/ns-dpi $(INSTALL_DIR) $(1)/etc/init.d $(INSTALL_DIR) $(1)/etc/uci-defaults $(INSTALL_DIR) $(1)/usr/share/ns-plug/hooks/unregister @@ -78,7 +79,10 @@ define Package/ns-dpi/install $(INSTALL_BIN) ./files/dpi-license-update.py $(1)/usr/sbin/dpi-license-update $(INSTALL_BIN) ./files/dpi-data-update.py $(1)/usr/sbin/dpi-data-update $(INSTALL_BIN) ./files/dpi-update.py $(1)/usr/sbin/dpi-update + $(INSTALL_BIN) ./files/dpi-migrate.py $(1)/usr/libexec/ns-dpi/dpi-migrate $(INSTALL_CONF) ./files/20_dpi $(1)/etc/uci-defaults + $(INSTALL_BIN) ./files/21_dpi_migrate $(1)/etc/uci-defaults/21_dpi_migrate + $(INSTALL_BIN) ./files/23_dpi_drop_options $(1)/etc/uci-defaults/23_dpi_drop_options $(INSTALL_BIN) ./files/99-dpi-license-update-cron.uci-defaults $(1)/etc/uci-defaults/99-dpi-license-update-cron $(INSTALL_BIN) ./files/99-dpi-data-update-cron.uci-defaults $(1)/etc/uci-defaults/99-dpi-data-update-cron $(INSTALL_DIR) $(1)/usr/share/ns-plug/hooks/register diff --git a/packages/ns-dpi/README.md b/packages/ns-dpi/README.md index ff267ad6b..e5bbbe5ae 100644 --- a/packages/ns-dpi/README.md +++ b/packages/ns-dpi/README.md @@ -1,183 +1,112 @@ # ns-dpi -Manage network traffic using DPI on network flows. +Block traffic by application and protocol using the netifyd DPI engine. How it works: -- Netify flow actions plugin adds a label to matching connections -- nft rules can block or change priority (`dscp`) to connections with labels - -To enable traffic processing: -- configure `dpi` UCI database (see below for an example) -- enable DPI service: - ``` - uci set dpi.config.enabled=1 - uci commit dpi - /etc/init.d/dpi restart - service netifyd reload - ``` - -Global options: - -- `enabled`: can be `0` or `1`, if set to `1` enable the service -- `log_blocked`: can be `0` or `1`, if set to `1` blocked connections will be logged inside `/var/log/messages` -- `firewall_exemption`: can be `0` or `1`, if set to `1` all firewall IP addresses will be - added to global exemption list and will not match DPI rules -- `popular_filters`: list of filters that will be returned to from `api-cli ns.dpi list-popular` call. -- `ns_exclude`: list of network interface exclusions in Netifyd that will be returned by `uci show netifyd.@netifyd[0].ns_exclude` - -Rule options: - -- `criteria`: DPI expression to match the traffic - - the criteria must terminate with `;` when using complex expressions - - use the `"` symbol to enclose strings, double-qoutes will be then translated to `'` inside the plugin configuration file (`/etc/netifyd/netify-flow-actions.json`) -- `source`: match all traffic from the given address, it accepts also an object like `/`; when using an object make sure to not use complex criteria -- `action`: valid actions are: - - `block`: matching traffic will be blocked - - `bulk`: matching traffic will be moved to low priority QoS class named `Bulk` - - `best_effort`: matching traffic will be moved to average priority QoS class named `Best Effort` - - `video`: matching traffic will be moved to high priority QoS class named `Video` - - `voice`: matching traffic will be moved to very high priority QoS class named `Voice` -- `description`: an optional rule description -- `device`: optional device name, if set the rule will be applied only to the given device, example `br-lan` -- `application`: list of applications to match, the list can contain application names like `netify.amazon-prime` -- `enabled`: can be `0` or `1`, if set to `1` the rule will be enabled -- `log`: can be set to `1` to log matching connections, gives an improved visibility on matched connections; - logs are stored in `/var/run/netifyd/dpi-actions-*.json` - -Global exemptions options: - -- `criteria`: global exemption criteria, usually it's an IP address; it can also be an object like `/` -- `enabled`: can be `0` or `1`, if set to `1` enable the exemption -- `description`: an optional exemption description - -All enabled rules are always evaluated by netifyd, the rule order doesn't matter. - -Example of `/etc/config/dpi`: -``` -config main 'config' - option log_blocked '1' - option enabled '1' - option firewall_exemption '1' - list popular_filters 'netify.netflix' - list popular_filters 'netify.telegram' - list popular_filters 'DoT' - list popular_filters 'netify.twitch' - list popular_filters 'netify.teamviewer' - list popular_filters 'DoH' - -config rule - option action 'bulk' - option criteria 'local_ip == 192.168.100.22 && application == "netify.facebook";' - option description 'Low priority for 192.168.100.22 when accessing Facebook' - option enabled 1 - -config rule 'ns_e775b8a7' - option enabled '1' - option device 'br-lan' - option action 'block' - list application 'netify.amazon-prime' - option description 'Block Amazon Prime for everyone' - -config rule - option action 'block' - list application 'netify.twitter - list application 'netify.instagram' - list source 'objects/ns_hostset_1' - list source 'dhcp/ns_reservation_1' - option description 'Block Twitter and Instagrm for some hosts' - option enabled 1 - -config exemption - option criteria '192.168.1.22' - option description 'Important host' - option enabled '1' +- `dpi-config` turns `/etc/config/dpi` into the netifyd flow actions config, + `/etc/netifyd/netify-proc-flow-actions.json` +- netifyd sets a conntrack label on the flows matching a rule +- the `dpi_actions` nft chain, written by `dpi-nft` in `/usr/share/nftables.d/table-pre/`, rejects the + flows labelled `netify-blocked` -config exemption - option criteria 'dhcp/ns_271ca281' - option description 'Important host with a reservation' - option enabled '1' -``` - -QoS rules do not have any effect if qosify is not enabled. -To enable qosify use: +netifyd runs all the time: traffic is filtered as soon as one rule is enabled. +Flows seen on the WAN interfaces are excluded from the DPI rules (global `iface == 'wan'` exemption, as +in the netifyd `10-nfqueue.conf`). +Rules and application groups are managed by the `ns.dpi` API, see `packages/ns-api/README.md`. +After editing `/etc/config/dpi` by hand, apply the changes with: ``` -uci set qosify.wan.disabled=0 -uci commit qosify -/etc/init.d/qosify restart +uci commit dpi +/etc/init.d/dpi reload ``` -To inspect qosify status use: -``` -qosify-status -``` +## Configuration -Check if traffic is matching: -``` -nft list table inet dpi -``` +Global options, in the `config` section: -List connections with `block` labels: -``` -conntrack -L -o label -l block -``` +- `log_blocked`: `1` logs the blocked connections in `/var/log/messages`, with the `DPI block: ` prefix -If needed, start netifyd in debug mode: -``` -netifyd -R -d -I br-lan -E eth1 -``` +Application group (`appgroup` section), a named set of members: -## Supplementary signatures +- `ns_name`: name of the group +- `app`: list of application names, as reported by the engine (e.g. `netify.netflix`) +- `app_category`: list of application category tags (e.g. `games`) +- `proto`: list of protocol names, as reported by the engine (e.g. `HTTP/Connect`) +- `proto_category`: list of protocol category tags -By default, netifyd is equipped to detect around 430 protocols and applications. With the inclusion of -supplementary signatures, netifyd can extend its recognition capabilities to encompass over 1600 protocols and applications. +Rule (`rule` section): -Extra signatures are accessible only from a machine with a valid subscription. -A cron job will update DPI signatures during the night and upon machine registration. -The download will be authenticated using a Nethesis proxy. +- `ns_name`: name of the rule +- `enabled`: `0` or `1` +- `action`: `block` or `allow` +- `priority`: evaluation order, starting at `1`; the first rule matching a flow wins +- `appgroup`: list of application groups to match +- `source`: list of addresses, networks or ranges to narrow the rule to; empty means every host +- `ns_match_all`: `1` matches every flow instead of naming application groups +- `ns_managed`: `1` for the rules created by the API +- `criteria`: raw netifyd expression, used by the rules not created by the API; the API can't edit them -To force the update execute: -``` -dpi-update +Example: ``` +config main 'config' + option log_blocked '1' -You can use a different proxy by overriding the `HOST` env variable. -If the proxy is authenticated, use the `__USER__` and `__PASSWORD__` placeholders. -The placeholders will be replaced with systemd id and secret from `ns-plug`. +config appgroup 'ns_1a2b3c4d' + option ns_name 'Streaming and games' + list app 'netify.netflix' + list app_category 'games' -Example: -``` -HOST=http://__USER__:__PASSWORD__@sp.gs.nethserver.net dpi-update +config rule 'ns_3869dc35' + option ns_name 'Allow the office' + option ns_managed '1' + option enabled '1' + option action 'allow' + option priority '1' + option ns_match_all '1' + list source '192.168.1.10' + +config rule 'ns_9d40be71' + option ns_name 'Block streaming and games' + option ns_managed '1' + option enabled '1' + option action 'block' + option priority '2' + list appgroup 'ns_1a2b3c4d' + list source '192.168.1.0/24' ``` -## Managing Interface Exclusions in Netifyd +## Migration from the previous schema -By default, Netifyd monitors all interfaces. To exclude specific interfaces, you can define an exclusion list. Below are commands to add, modify, or remove excluded interfaces. +On upgrade, and every time the DPI service starts, `/usr/libexec/ns-dpi/dpi-migrate` converts the +configuration written before application groups existed, e.g. after restoring an old backup: -- Add interfaces to exclusion list -``` -uci add_list netifyd.@netifyd[0].ns_exclude='eth1' -uci add_list netifyd.@netifyd[0].ns_exclude='tun*' -uci add_list netifyd.@netifyd[0].ns_exclude='wg*' -uci commit netifyd -echo '{"changes": {"network": {}}}' | /usr/libexec/rpcd/ns.commit call commit -``` +- every rule with `device`, `application`, `protocol` or `category` becomes an unmanaged rule, with its + behaviour kept in `criteria`. Rules with an action other than `block`, such as the QoS ones (`bulk`, + `best_effort`, `video`, `voice`), and rules matching nothing are dropped. The per-rule `exemption` and + `log` options are removed +- every `exemption` section becomes an `allow` rule at the top of the list, named after the exemption + description +- `popular_filters` is removed -- Modify exclusion list -``` -uci delete netifyd.@netifyd[0].ns_exclude='eth1' -uci add_list netifyd.@netifyd[0].ns_exclude='eth2' -uci commit netifyd -echo '{"changes": {"network": {}}}' | /usr/libexec/rpcd/ns.commit call commit -``` +Dropped rules and removed options are logged in `/var/log/messages`. The `enabled` and `firewall_exemption` +global options, which no longer switch anything, are removed on upgrade as well. + +## Troubleshooting -- Clear exclusion list +Check the generated rules: ``` -uci delete netifyd.@netifyd[0].ns_exclude -uci commit netifyd -echo '{"changes": {"network": {}}}' | /usr/libexec/rpcd/ns.commit call commit +cat /etc/netifyd/netify-proc-flow-actions.json +nft list chain inet fw4 dpi_actions ``` -- Return the exclusion list +List the blocked connections: ``` -uci show netifyd.@netifyd[0].ns_exclude +conntrack -L -o label -l netify-blocked ``` + +## Signatures and catalogs + +- `dpi-update` downloads the extra signatures, available only with a valid subscription, every night; + run it by hand to force an update. On unregistration the extra signatures are removed and the ones + shipped with the image are restored +- `dpi-data-update` downloads the application and protocol catalogs (labels, categories, icons) into + `/etc/netifyd`, every night diff --git a/packages/ns-dpi/files/20_dpi b/packages/ns-dpi/files/20_dpi index f5e6adaf2..3db4b3df6 100644 --- a/packages/ns-dpi/files/20_dpi +++ b/packages/ns-dpi/files/20_dpi @@ -1,29 +1,14 @@ #!/bin/sh +# TODO: ship /etc/config/dpi as a conffile instead of creating it here. Before doing so: +# - 19-ns-api.wizard detects a fresh install by the absence of /etc/config/dpi, it needs another marker +# - the dpi-update cron entry below must move to its own uci-default +# - fix the conffiles define in the Makefile, it is declared for ns-plug instead of ns-dpi + if ! uci -q get dpi.config > /dev/null; then uci -q import dpi << EOI config main 'config' option log_blocked '0' - option enabled '0' - option firewall_exemption '0' - list popular_filters 'netify.facebook' - list popular_filters 'netify.amazon-prime' - list popular_filters 'netify.whatsapp' - list popular_filters 'netify.instagram' - list popular_filters 'netify.netflix' - list popular_filters 'netify.telegram' - list popular_filters 'netify.tiktok' - list popular_filters 'netify.youtube' - list popular_filters 'netify.facebook-messenger' - list popular_filters 'netify.twitter' - list popular_filters 'netify.vimeo' - list popular_filters 'netify.snapchat' - list popular_filters 'netify.pinterest' - list popular_filters 'netify.nordvpn' - list popular_filters 'DoT' - list popular_filters 'netify.twitch' - list popular_filters 'netify.teamviewer' - list popular_filters 'DoH' EOI fi diff --git a/packages/ns-dpi/files/21_dpi_migrate b/packages/ns-dpi/files/21_dpi_migrate new file mode 100755 index 000000000..839b49e22 --- /dev/null +++ b/packages/ns-dpi/files/21_dpi_migrate @@ -0,0 +1,13 @@ +#!/bin/sh + +# +# Copyright (C) 2026 Nethesis S.r.l. +# SPDX-License-Identifier: GPL-2.0-only +# + +# Migrate /etc/config/dpi from the schema used before application groups existed. Safe to run on every +# boot: dpi-migrate is a no-op once there is nothing left in the old schema. + +set -e + +/usr/libexec/ns-dpi/dpi-migrate diff --git a/packages/ns-dpi/files/23_dpi_drop_options b/packages/ns-dpi/files/23_dpi_drop_options new file mode 100755 index 000000000..fa23094c3 --- /dev/null +++ b/packages/ns-dpi/files/23_dpi_drop_options @@ -0,0 +1,21 @@ +#!/bin/sh + +# +# Copyright (C) 2026 Nethesis S.r.l. +# SPDX-License-Identifier: GPL-2.0-only +# + +# Remove the global options that no longer switch anything: +# - enabled: netifyd runs all the time, DPI filters traffic as soon as one rule is enabled +# - firewall_exemption: the flows seen on the WAN interfaces are always excluded from the DPI rules + +changed=0 +for option in enabled firewall_exemption; do + if uci -q get dpi.config.$option > /dev/null; then + uci -q delete dpi.config.$option + changed=1 + fi +done + +[ "$changed" = "1" ] && uci commit dpi +exit 0 diff --git a/packages/ns-dpi/files/connlabel.conf b/packages/ns-dpi/files/connlabel.conf index 784b84ec4..6131ac74c 100644 --- a/packages/ns-dpi/files/connlabel.conf +++ b/packages/ns-dpi/files/connlabel.conf @@ -4,4 +4,5 @@ 3 bulk 4 best_effort 5 video -6 voice \ No newline at end of file +6 voice +7 netify-allowed \ No newline at end of file diff --git a/packages/ns-dpi/files/dpi b/packages/ns-dpi/files/dpi index 267e6d6fe..228d33859 100755 --- a/packages/ns-dpi/files/dpi +++ b/packages/ns-dpi/files/dpi @@ -5,6 +5,9 @@ # SPDX-License-Identifier: GPL-2.0-only # +# Convert a configuration still on the previous schema, e.g. restored from an old backup or synced +# by an HA peer: a no-op when there is nothing to migrate +/usr/libexec/ns-dpi/dpi-migrate # Generate netify flow actions config file /usr/sbin/dpi-config # Apply nftables rules diff --git a/packages/ns-dpi/files/dpi-config b/packages/ns-dpi/files/dpi-config index 969796d0a..fa3748b29 100755 --- a/packages/ns-dpi/files/dpi-config +++ b/packages/ns-dpi/files/dpi-config @@ -1,31 +1,15 @@ #!/usr/bin/python3 # -# Copyright (C) 2025 Nethesis S.r.l. +# Copyright (C) 2026 Nethesis S.r.l. # SPDX-License-Identifier: GPL-2.0-only # from euci import EUci import json -import subprocess import os -from nethsec import utils, objects - -def get_interface_ips(interface): - ret = list() - try: - proc = subprocess.run(['/sbin/ifstatus', section], check=True, capture_output=True) - info = json.loads(proc.stdout) - except: - return ret - - for ip_type in ('ipv4-address', 'ipv6-address'): - if ip_type in info: - for addr in info[ip_type]: - if 'address' in addr: - ret.append(addr['address']) - - return ret +import syslog +from nethsec import dpi cfg_file = "/etc/netifyd/netify-proc-flow-actions.json" config = { @@ -43,12 +27,12 @@ config = { "netify-blocked" ] }, - "log": { - "target_type": "log", + "allow": { + "target_type": "ctlabel", "target_enabled": True, - "interval": 60, - "path": "/var/run/netifyd/", - "prefix": "dpi-actions", + "labels": [ + "netify-allowed" + ] }, "analyzed": { "target_type": "ctlabel", @@ -58,104 +42,72 @@ config = { ] } }, - "actions": {}, - "exemptions": [] + # analyzed comes first and lets the evaluation continue, so the label is never lost to a rule + # that halts on match + "actions": { + "analyzed": { + "enabled": True, + "priority": 0, + "halt_on_match": False, + "criteria": 'detection_guessed || detection_complete;', + "targets": ["analyzed"], + "exemptions": [] + } + }, + # based on 10-nfqueue.conf in netifyd + "exemptions": [ + "iface == 'wan';" + ] } -# Add firewall IP address as global exemptions u = EUci() -if u.get('dpi', 'config', 'firewall_exemption', default=0) == '1': - for section in u.get_all('network'): - if u.get('network', section) == 'interface': - ip_addrs = get_interface_ips(section) - if ip_addrs: - config["exemptions"] = config["exemptions"] + ip_addrs - -# Add custom global exemptions -exemptions = utils.get_all_by_type(u, 'dpi', 'exemption') -for ex_name in exemptions: - exemption = exemptions[ex_name] - criteria = exemption.get('criteria', '') - enabled = exemption.get('enabled', '1') - if enabled == '1' and criteria: - if objects.is_object_id(criteria): - for ip in objects.get_object_ips(u, criteria): - config["exemptions"].append(ip) - else: - config["exemptions"].append(criteria) -rcount = 0 -valid_actions = ['block', 'bulk', 'best_effort', 'video', 'voice'] +# a rule from the schema used before application groups existed carries at least one of these fields; +# dpi.migrate_schema converts it, so one still here was not migrated yet +legacy_fields = ('device', 'application', 'protocol', 'category') for section in u.get_all('dpi'): if u.get('dpi', section) != 'rule': continue rule = u.get_all('dpi', section) - if rule['action'] not in valid_actions or not rule['enabled']: + if not rule.get('enabled'): continue - device = rule.get('device', '*') + if any(field in rule for field in legacy_fields): + # never emit it as a current rule: its source alone would block every flow of those hosts + syslog.syslog(syslog.LOG_WARNING, f"dpi-config: skipping {section}, not migrated to the current schema") + continue + if rule.get('action') not in dpi.DPI_RULE_ACTIONS: + syslog.syslog(syslog.LOG_WARNING, f"dpi-config: skipping {section}, unknown action {rule.get('action')}") + continue + try: + priority = int(rule.get('priority', 1)) + except ValueError: + syslog.syslog(syslog.LOG_WARNING, f"dpi-config: skipping {section}, invalid priority") + continue if 'criteria' in rule: - # criteria has precedence over sources, protocol, category and application - criteria = rule['criteria'].replace('"',"'") + # raw criteria: hand-written, or migrated from the old schema with its behaviour frozen + # (dpi.migrate_schema). Double quotes become single ones, and the terminator is added when + # missing: an unterminated expression is a syntax error + criteria = rule['criteria'].replace('"', "'").strip() + if criteria and not criteria.endswith(';'): + criteria = f'{criteria};' else: - # generate criteria from source, protocol, category and application - sources = [] - for source in rule.get('source', []): - if objects.is_object_id(source): - for ip in objects.get_object_ips(u, source): - sources.append(f'local_ip == {ip}') - else: # custom local IPs - sources.append(f'local_ip == {source}') - - applications = [] - for app in rule.get('application', []): - applications.append(f"app == '{app}'") - for proto in rule.get('protocol', []): - applications.append(f"proto == '{proto.lower()}'") - for cat in rule.get('category', []): - applications.append(f"category == '{cat.lower()}'") - - sources_s = ' or '.join(sources) - applications_s = ' or '.join(applications) - criteria = f'(iface_nfq_src == \'{device}\' or iface_nfq_dst == \'{device}\') && ' - if len(sources) < 1: - criteria += f'({applications_s}) ;' - elif len(applications) < 1: - criteria += f'({sources_s}) ;' - else: - criteria += f'({sources_s}) && ({applications_s}) ;' - - vlan_id = None - base_if = None - for item in utils.get_all_by_type(u, 'network', 'device').values(): - if item.get('vid', None) is not None and item.get('name', '') == device: - vlan_id = item.get('vid', None) - base_if = item.get('ifname', None) - break - - if vlan_id is not None and base_if is not None: - criteria = f'vlan_id == {vlan_id} && {criteria}' - device = base_if + # application groups, optionally narrowed to a source, or a match-all rule + criteria = dpi.build_rule_criteria(u, rule) + if not criteria: + # never emit an empty criteria: it is a parse error, and a parse error kills the agent + syslog.syslog(syslog.LOG_WARNING, f"dpi-config: skipping {section}, it matches nothing") + continue - targets = [rule['action']] - if 'log' in rule and rule['log'] == '1': - targets.append('log') - config["actions"][f"rule{rcount}"] = { + config["actions"][section] = { "enabled": rule['enabled'] == '1', + "priority": priority, + "halt_on_match": True, "criteria": criteria, - "targets": targets, - "exemptions": rule.get('exemption', []) + "targets": [rule['action']], + "exemptions": [] } - rcount = rcount + 1 - -# add global analyzed for all flows that are not blocked -config["actions"]["analyzed"] = { - "enabled": True, - "criteria": 'detection_guessed || detection_complete;', - "targets": ["analyzed"], - "exemptions": [] -} # if directories do not exist, create them if not os.path.exists(os.path.dirname(cfg_file)): diff --git a/packages/ns-dpi/files/dpi-migrate.py b/packages/ns-dpi/files/dpi-migrate.py new file mode 100755 index 000000000..1a3b32d15 --- /dev/null +++ b/packages/ns-dpi/files/dpi-migrate.py @@ -0,0 +1,13 @@ +#!/usr/bin/env python3 + +# +# Copyright (C) 2026 Nethesis S.r.l. +# SPDX-License-Identifier: GPL-2.0-only +# + +# this script is supposed to be run by the 21_dpi_migrate uci-defaults + +from euci import EUci +from nethsec import dpi + +dpi.migrate_schema(EUci()) diff --git a/packages/ns-ui/Makefile b/packages/ns-ui/Makefile index 7245172c5..4d6ac49e0 100644 --- a/packages/ns-ui/Makefile +++ b/packages/ns-ui/Makefile @@ -7,7 +7,7 @@ include $(TOPDIR)/rules.mk PKG_NAME:=ns-ui # renovate: datasource=github-releases depName=NethServer/nethsecurity-ui -PKG_VERSION:=2.26.1 +PKG_VERSION:=2.27.0 PKG_RELEASE:=1 PKG_SOURCE_PROTO:=git diff --git a/packages/python3-nethsec/src/nethsec/dpi/__init__.py b/packages/python3-nethsec/src/nethsec/dpi/__init__.py index 5de35668e..7c640ca98 100644 --- a/packages/python3-nethsec/src/nethsec/dpi/__init__.py +++ b/packages/python3-nethsec/src/nethsec/dpi/__init__.py @@ -1,7 +1,7 @@ #!/usr/bin/python3 # -# Copyright (C) 2023 Nethesis S.r.l. +# Copyright (C) 2026 Nethesis S.r.l. # SPDX-License-Identifier: GPL-2.0-only # @@ -9,64 +9,42 @@ Library that handles the DPI rules. """ +import ipaddress import json import subprocess -from fnmatch import fnmatch +import syslog import math from euci import EUci -from nethsec import utils, firewall +from nethsec import objects, utils from nethsec.utils import ValidationError -def __load_applications() -> dict[int, str]: +def load_applications() -> dict[int, str]: """ - Reads the applications from the netify-apps.conf file. + Reads the applications loaded in memory by the engine, using the netifyd --dump-apps command. Returns: - dict of applications, each dict contains the property "id" and "name" + dict of applications, the key is the application id and the value is the application name """ + result = subprocess.run(['netifyd', '--dump-apps'], check=True, capture_output=True) applications = dict[int, str]() - with open('/etc/netifyd/netify-apps.conf', 'r') as file: - for line in file.readlines(): - if line.startswith('app'): - line_split = line.strip().removesuffix('\n').removeprefix('app:').split(":") - applications[int(line_split[0])] = line_split[1] - return applications - - -def __load_application_categories() -> dict[int, dict[str]]: - """ - Reads the application categories from the netify-categories.json file. - - Returns: - dict of application categories, each dict contains the property "id" and "name" - """ - categories = dict[int, dict[str]]() - with open('/etc/netifyd/netify-categories.json', 'r') as file: - categories_file = json.load(file) - - categories_names = dict[int, str]() - if 'application_tag_index' not in categories_file: - for category_name, applications in categories_file['application_index'].items(): - for application in applications: - categories[application] = { - 'name': category_name - } - else: - categories_application_tag_index: dict[str, int] = categories_file['application_tag_index'] - for category_name, category_id in categories_application_tag_index.items(): - categories_names[category_id] = category_name - - categories_application_index: list[int, list[int]] = categories_file['application_index'] - for category_id, applications_id in categories_application_index: - for application_id in applications_id: - categories[application_id] = { - 'name': categories_names[category_id] - } + for line in result.stdout.decode().splitlines(): + # lines can be empty + if len(line.strip()) < 1: + continue + line_split = line.split(":", 1) + if len(line_split) < 2: + continue + try: + application_id = int(line_split[0].strip()) + except ValueError: + # skip lines not in the "id: name" format, like headers + continue + applications[application_id] = line_split[1].strip() - return categories + return applications def load_protocols() -> dict[int, str]: @@ -88,132 +66,349 @@ def load_protocols() -> dict[int, str]: return protocols -def __load_protocol_categories() -> dict[int, dict[str]]: +# Application groups: a NethSecurity abstraction, the netifyd plugin has no group primitive. +# A group is a named set of match members, expanded into an inline expression at generation time. + +NETIFYD_DATA_DIR = '/etc/netifyd' + +# UCI list name of every member kind, mapped to the API parameter it is fed by +APPGROUP_MEMBERS = { + 'app': 'applications', + 'app_category': 'application_categories', + 'proto': 'protocols', + 'proto_category': 'protocol_categories' +} + +# validation error message of every member kind +__APPGROUP_MEMBER_ERRORS = { + 'app': 'invalid_application', + 'app_category': 'invalid_application_category', + 'proto': 'invalid_protocol', + 'proto_category': 'invalid_protocol_category' +} + +# characters that would let a member value escape its quoted literal, or terminate the expression +__APPGROUP_FORBIDDEN_CHARS = '\'"\\;()\n\r\t' + +APPGROUP_NAME_MAX_LENGTH = 64 + + +def load_categories() -> dict[str, dict[int, str]]: """ - Reads the protocol categories from the netify-categories.json file. + Reads the categories loaded in memory by the engine, using the netifyd --dump-categories command. + + The dump reports one category per line, as ": : ". Returns: - dict of protocol categories, each dict contains the property "id" and "name" + dict of categories indexed by type, e.g. `{'application': {1: 'adult'}, 'protocol': {2: 'database'}}`. + Ids are the ones the agent uses internally: they do not match the ids of the downloaded catalogs, + so categories must always be crossed by tag. """ - categories = dict[int, dict[str]]() - with open('/etc/netifyd/netify-categories.json', 'r') as file: - categories_file = json.load(file) + result = subprocess.run(['netifyd', '--dump-categories'], check=True, capture_output=True) + categories = dict[str, dict[int, str]]() + for line in result.stdout.decode().splitlines(): + line_split = line.split(":", 2) + if len(line_split) < 3: + continue + try: + category_id = int(line_split[0].strip()) + except ValueError: + # skip lines not in the "id: type: tag" format, like headers + continue + category_type = line_split[1].strip() + categories.setdefault(category_type, dict[int, str]())[category_id] = line_split[2].strip() - categories_names = dict[int, str]() + return categories - if 'protocol_tag_index' not in categories_file: - for category_name, protocols in categories_file['protocol_index'].items(): - for protocol in protocols: - categories[protocol] = { - 'name': category_name - } - else: - categories_protocol_tag_index: dict[str, int] = categories_file['protocol_tag_index'] - for category_name, category_id in categories_protocol_tag_index.items(): - categories_names[category_id] = category_name - categories_protocol_index: list[int, list[int]] = categories_file['protocol_index'] - for category_id, protocol_ids in categories_protocol_index: - for protocol_id in protocol_ids: - categories[protocol_id] = { - 'name': categories_names[category_id] - } +def __load_catalog(filename: str) -> list[dict]: + """ + Reads a downloaded catalog, one of the `/etc/netifyd/netify-*.json` files. - return categories + Args: + - filename: name of the catalog file inside the netifyd data directory + Returns: + list of catalog entries + """ + with open(f'{NETIFYD_DATA_DIR}/{filename}', 'r') as file: + return json.load(file) -def __load_blocklist() -> list[dict[str]]: + +def __appgroup_vocabulary(kind: str) -> set[str] | None: """ - Format the applications and protocols into a list of dicts. + Values a member of the given kind can take, lowercased. + + The engine is the authority: only what it has loaded can be matched. The downloaded catalog backs + applications up, so that a group stays editable on a machine that lost its premium signatures. + + Args: + - kind: one of the keys of `APPGROUP_MEMBERS` Returns: - list of dicts, each dict contains the property "id", "name", "type" and "category" + set of accepted values, or None when no source is available and the values cannot be checked """ - result = list[dict[str]]() - applications = __load_applications() - application_categories = __load_application_categories() - - for application_id, application_name in applications.items(): - result_application = { - 'id': application_id, - 'name': application_name, - 'type': 'application' - } - if application_id in application_categories: - result_application['category'] = application_categories[application_id] - result.append(result_application) - - protocols = load_protocols() - protocol_categories = __load_protocol_categories() - - for protocol_id, protocol_name in protocols.items(): - result_protocol = { - 'id': protocol_id, - 'name': protocol_name, - 'type': 'protocol' + vocabulary = set[str]() + try: + if kind == 'app': + vocabulary.update({value.lower() for value in load_applications().values()}) + elif kind == 'proto': + vocabulary.update({value.lower() for value in load_protocols().values()}) + else: + category_type = 'application' if kind == 'app_category' else 'protocol' + vocabulary.update({value.lower() for value in load_categories().get(category_type, {}).values()}) + except Exception: + # the engine can't be queried, fall back to the downloaded catalogs below + pass + + if kind != 'proto': + # protocol names come from the agent itself and have no counterpart in the catalog, where the + # same protocol is listed under a different tag: the catalog would accept unmatchable values + catalogs = { + 'app': 'netify-application-catalog.json', + 'app_category': 'netify-application-categories.json', + 'proto_category': 'netify-protocol-categories.json' } - if protocol_id in protocol_categories: - result_protocol['category'] = protocol_categories[protocol_id] - result.append(result_protocol) + try: + vocabulary.update({entry['tag'].lower() for entry in __load_catalog(catalogs[kind]) if entry.get('tag')}) + except Exception: + pass + + # an unknown value never matches, so a missing vocabulary is a reason to skip the check, not to refuse + return vocabulary if vocabulary else None + +APPGROUP_CATALOG_FILES = { + 'applications': 'netify-application-catalog.json', + 'protocols': 'netify-protocol-catalog.json' +} + +UNCATEGORIZED_TAG = '' + + +def list_appgroup_catalog() -> dict[str, list[dict]]: + """ + List every application and protocol a group can be built with, grouped by category. + + Every source has to answer: the engine says what is matchable, the catalogs say how it reads, and a + partial answer would silently offer a smaller product than the machine has. So this raises instead + of returning what it managed to collect. + + The id of an item is what a group stores, and it is the engine name whenever there is one: the + catalog tag is only a fallback for an entry that cannot be matched anyway, so it is never a value a + client submits. A group is selectable when the category itself can be a group member, which every + real category can be and the uncategorised bucket never can. Groups and items are sorted by label, + with the uncategorised bucket last; a client translating the category labels has to sort them again. + + Returns: + dict with the keys "applications" and "protocols", each a list of category groups + + Raises: + - Exception: if the engine cannot be queried or a catalog is missing + """ + loaded = {'applications': load_applications(), 'protocols': load_protocols()} + result = {} + for kind, filename in APPGROUP_CATALOG_FILES.items(): + catalog = __load_catalog(filename) + # (category tag, category label, item), in catalog order + items = [] + for entry in catalog: + # an application marked inactive, or a protocol marked deprecated, is gone from the product + if kind == 'applications' and entry.get('active') is False: + continue + if kind == 'protocols' and entry.get('deprecated'): + continue + label = entry.get('label', '') + # "unknown" is the engine's catch-all for what it could not classify, not a group member + if kind == 'protocols' and label.lower() == 'unknown': + continue + engine_name = loaded[kind].get(entry.get('id')) + category = entry.get('category') or {} + item = {'id': engine_name or entry.get('tag', ''), 'label': label, 'selectable': engine_name is not None} + if entry.get('icon'): + item['logo'] = entry.get('icon') + items.append((category.get('tag', UNCATEGORIZED_TAG), category.get('label', ''), item)) + + # what the engine loaded but the catalog does not list is still matchable: kept, named after the + # engine and left uncategorised + known = {entry.get('id') for entry in catalog} + for entry_id, name in loaded[kind].items(): + if entry_id in known: + continue + if kind == 'protocols' and name.lower() == 'unknown': + continue + items.append((UNCATEGORIZED_TAG, '', {'id': name, 'label': name, 'selectable': True})) + + groups = dict[str, dict]() + for tag, label, item in items: + group = groups.get(tag) + if group is None: + group = groups[tag] = {'tag': tag, 'label': label, 'selectable': tag != UNCATEGORIZED_TAG, 'items': []} + group['items'].append(item) + + for group in groups.values(): + group['items'].sort(key=lambda item: item['label'].lower()) + + result[kind] = sorted(groups.values(), key=lambda group: (group['tag'] == UNCATEGORIZED_TAG, + group['label'].lower())) return result -def list_devices(e_uci: EUci): +def __validate_appgroup_members(kind: str, values: list[str]) -> list[str]: """ - List device-interface available for filtering. + Validate the members of one kind, stripping and deduplicating them. + + Args: + - kind: one of the keys of `APPGROUP_MEMBERS` + - values: list of member values Returns: - list of dicts, each dict contains the property "interface" and "device" + the sanitized list of members + + Raises: + - ValidationError: if a member is empty, holds a character that would break the expression, or is + unknown to both the engine and the catalog """ - instance_name = list(e_uci.get('netifyd').keys())[0] - devices = e_uci.get('netifyd', instance_name, 'internal_if', default=[], list=True) - for zone in firewall.list_zones(e_uci).values(): - if zone['name'] == 'wan': - continue - network_devices = utils.get_all_devices_by_zone(e_uci, zone['name']) - devices = list(set(list(devices) + network_devices)) - ret = [] - for item in devices: - interface_name = utils.get_interface_from_device(e_uci, item) - ret.append({ - 'interface': interface_name if interface_name is not None else item, - 'device': item - }) - return ret + parameter = APPGROUP_MEMBERS[kind] + message = __APPGROUP_MEMBER_ERRORS[kind] + if not values: + return [] + + vocabulary = __appgroup_vocabulary(kind) + members = [] + seen = set[str]() + for value in values: + if not isinstance(value, str): + raise ValidationError(parameter, message, value) + member = value.strip() + if not member or any(char in member for char in __APPGROUP_FORBIDDEN_CHARS): + raise ValidationError(parameter, message, value) + if vocabulary is not None and member.lower() not in vocabulary: + raise ValidationError(parameter, message, member) + if member.lower() not in seen: + seen.add(member.lower()) + members.append(member) + + return members + + +def __validate_appgroup(e_uci: EUci, name: str, applications: list[str], application_categories: list[str], + protocols: list[str], protocol_categories: list[str], + config_name: str = None) -> tuple[str, dict[str, list[str]]]: + """ + Validate a group before it is stored. + + Args: + - e_uci: euci instance + - name: name of the group + - applications: list of application names + - application_categories: list of application category tags + - protocols: list of protocol names + - protocol_categories: list of protocol category tags + - config_name: config name of the group being edited, excluded from the name uniqueness check + + Returns: + a tuple with the sanitized name and the sanitized members, indexed by UCI list name -def list_applications(search: str = None, limit: int = None, page: int = 1) -> dict: + Raises: + - ValidationError: if the name or any member is invalid, or if the group has no members """ - List applications available for filtering. + if not isinstance(name, str) or not name.strip(): + raise ValidationError('name', 'name_required', name) + name = name.strip() + if len(name) > APPGROUP_NAME_MAX_LENGTH: + raise ValidationError('name', 'name_too_long', name) + for section, group in (utils.get_all_by_type(e_uci, 'dpi', 'appgroup') or {}).items(): + if section != config_name and group.get('ns_name', '').strip().lower() == name.lower(): + raise ValidationError('name', 'name_already_exists', name) + + members = { + 'app': __validate_appgroup_members('app', applications), + 'app_category': __validate_appgroup_members('app_category', application_categories), + 'proto': __validate_appgroup_members('proto', protocols), + 'proto_category': __validate_appgroup_members('proto_category', protocol_categories) + } + if not any(members.values()): + # an empty group would expand to an empty criteria, which takes the agent down + raise ValidationError('members', 'appgroup_is_empty', '') + + return name, members + + +def __save_appgroup_data(e_uci: EUci, config_name: str, name: str, members: dict[str, list[str]]): + e_uci.set('dpi', config_name, 'ns_name', name) + for uci_list, values in members.items(): + if values: + e_uci.set('dpi', config_name, uci_list, values) + else: + # an emptied list must be removed, not stored empty + e_uci.delete('dpi', config_name, uci_list) + + +def is_used_appgroup(e_uci: EUci, config_name: str) -> tuple[bool, list[str]]: + """ + Check if an application group is referenced by a rule. Args: - - search: search string - - limit: limit the number of results + - e_uci: euci instance + - config_name: config name of the group + + Returns: + A tuple with: + - True if the group is referenced by at least one rule, False otherwise + - a list of the rule sections referencing it + """ + matches = [] + for section, rule in (utils.get_all_by_type(e_uci, 'dpi', 'rule') or {}).items(): + if config_name in rule.get('appgroup', []): + matches.append(f'dpi/{section}') + return len(matches) > 0, matches + + +def list_appgroups(e_uci: EUci, search: str = None, limit: int = None, page: int = 1, + used_info: bool = True) -> dict: + """ + List the application groups, ordered by name. + + Args: + - e_uci: euci instance + - search: search string, matched against the group name + - limit: limit the number of results, all of them if not given - page: page number + - used_info: include the used and matches info Returns: - list of dicts, each dict contains the property "code" and "name" + dict with the "data" list and the "meta" pagination info """ - result = __load_blocklist() + groups = [] + for section, group in (utils.get_all_by_type(e_uci, 'dpi', 'appgroup') or {}).items(): + data_group = {'id': section, 'name': group.get('ns_name', '')} + for uci_list, parameter in APPGROUP_MEMBERS.items(): + data_group[parameter] = list(group.get(uci_list, [])) + groups.append(data_group) - if search is not None: - # lower string so we can do a case-insensitive search + # pagination needs a deterministic order, the UCI one is not + groups.sort(key=lambda group: group['name'].lower()) + + if search: search = search.lower() - # I'm aware it's far from a readable code, but list comprehension is the fastest way to filter. - result = [item for item in result if - item.get('name', '').lower().find(search) != -1 or - item.get('category', {}).get('name', '').lower().find(search) != -1] + groups = [group for group in groups if search in group['name'].lower()] - total = len(result) + total = len(groups) if limit is not None: - result = result[limit * (page - 1):limit * page] + groups = groups[limit * (page - 1):limit * page] last_page = math.ceil(total / limit) else: last_page = 1 + if used_info: + # only for the returned page: every check walks the rules + for group in groups: + group['used'], group['matches'] = is_used_appgroup(e_uci, group['id']) + return { - 'data': result, + 'data': groups, 'meta': { 'last_page': last_page, 'total': total, @@ -221,276 +416,711 @@ def list_applications(search: str = None, limit: int = None, page: int = 1) -> d } -def list_popular(e_uci: EUci, limit: int = None, page: int = 1) -> dict: +def add_appgroup(e_uci: EUci, name: str, applications: list[str] = None, + application_categories: list[str] = None, protocols: list[str] = None, + protocol_categories: list[str] = None) -> str: """ - List popular applications available for filtering. + Store a new application group. Args: - - limit: limit the number of results - - page: page number + - e_uci: euci instance + - name: name of the group + - applications: list of application names, as the engine reports them + - application_categories: list of application category tags + - protocols: list of protocol names, as the engine reports them + - protocol_categories: list of protocol category tags Returns: - list of dicts, each dict contains the property "id", "name", "type" and "category" + config name of the group created + + Raises: + - ValidationError: if the name or any member is invalid, or if the group has no members """ - popular_filters = e_uci.get('dpi', 'config', 'popular_filters', default=[], list=True) - block_list = {block['name']: block for block in __load_blocklist()} - result = [] + name, members = __validate_appgroup(e_uci, name, applications, application_categories, protocols, + protocol_categories) + config_name = utils.get_random_id() + e_uci.set('dpi', config_name, 'appgroup') + __save_appgroup_data(e_uci, config_name, name, members) + e_uci.save('dpi') + return config_name + + +def edit_appgroup(e_uci: EUci, config_name: str, name: str, applications: list[str] = None, + application_categories: list[str] = None, protocols: list[str] = None, + protocol_categories: list[str] = None) -> str: + """ + Edit an application group. + + Args: + - e_uci: euci instance + - config_name: config name of the group to edit + - name: name of the group + - applications: list of application names, as the engine reports them + - application_categories: list of application category tags + - protocols: list of protocol names, as the engine reports them + - protocol_categories: list of protocol category tags + + Returns: + config name of the group edited + + Raises: + - ValidationError: if the group does not exist, or if the name or any member is invalid + """ + if e_uci.get('dpi', config_name, default=None) != 'appgroup': + raise ValidationError('id', 'appgroup_does_not_exists', config_name) + + name, members = __validate_appgroup(e_uci, name, applications, application_categories, protocols, + protocol_categories, config_name) + __save_appgroup_data(e_uci, config_name, name, members) + e_uci.save('dpi') + return config_name + + +def delete_appgroup(e_uci: EUci, config_name: str) -> str: + """ + Delete an application group. + + Args: + - e_uci: euci instance + - config_name: config name of the group to delete + + Returns: + config name of the group deleted + + Raises: + - ValidationError: if the group does not exist or is referenced by a rule + """ + if e_uci.get('dpi', config_name, default=None) != 'appgroup': + raise ValidationError('id', 'appgroup_does_not_exists', config_name) + + used, matches = is_used_appgroup(e_uci, config_name) + if used: + raise ValidationError('id', 'appgroup_is_used', matches) + + e_uci.delete('dpi', config_name) + e_uci.save('dpi') + return config_name + + +def expand_appgroups(e_uci: EUci, config_names: list[str]) -> str: + """ + Expand application groups into the parenthesised OR block of an action criteria. + + Members of several groups are merged and deduplicated; categories are never expanded into their + applications, so a category keeps following the signature updates. Groups that do not exist are + skipped. + + Args: + - e_uci: euci instance + - config_names: config names of the groups to expand + + Returns: + the expression, e.g. `(app == 'netify.amazon' || proto == 'http/connect')`, or an empty string + if the groups hold no member + """ + terms = [] + seen = set[str]() + for uci_list in APPGROUP_MEMBERS: + for config_name in config_names: + if e_uci.get('dpi', config_name, default=None) != 'appgroup': + continue + for value in e_uci.get('dpi', config_name, uci_list, list=True, default=[]): + # protocol and category names are matched lowercased, application tags already are + value = value if uci_list == 'app' else value.lower() + term = f"{uci_list} == '{value}'" + if term not in seen: + seen.add(term) + terms.append(term) + + if not terms: + return '' + return f"({' || '.join(terms)})" - for popular_filter in popular_filters: - if popular_filter in block_list.keys(): - result.append(block_list[popular_filter] | {'missing': False}) + +# Rules: ordered by priority, first match wins. A rule matches application groups only, optionally +# narrowed to a source; what it does is decided by its action. + +# 'allow' attaches a label no nft rule matches: it exists so a rule can win the first-match race and +# shield its traffic from the block rules below it +DPI_RULE_ACTIONS = ('block', 'allow') + +DPI_RULE_NAME_MAX_LENGTH = 64 + +DPI_MATCH_ALL_CRITERIA = '*' + +DPI_RULE_POSITIONS = ('top', 'bottom') + + +def __sorted_rules(e_uci: EUci) -> list[tuple[str, dict]]: + """ + Rule sections in priority order. Rules with no priority, i.e. written by hand, go last keeping + their config order. + """ + def sort_key(item: tuple[str, dict]): + try: + return 0, int(item[1].get('priority')) + except (TypeError, ValueError): + return 1, 0 + + return sorted((utils.get_all_by_type(e_uci, 'dpi', 'rule') or {}).items(), key=sort_key) + + +def __renumber_rules(e_uci: EUci, order: list[str] = None): + """ + Give every rule a dense priority, starting at 1. The generator keeps 0 for its own action, so no + user rule can be evaluated before it. + """ + if order is None: + order = [section for section, _ in __sorted_rules(e_uci)] + for priority, section in enumerate(order, start=1): + e_uci.set('dpi', section, 'priority', priority) + + +def __is_rule(e_uci: EUci, config_name: str) -> bool: + return e_uci.get('dpi', config_name, default=None) == 'rule' + + +def __validate_rule_name(name: str) -> str: + if not isinstance(name, str) or not name.strip(): + raise ValidationError('name', 'name_required', name) + name = name.strip() + if len(name) > DPI_RULE_NAME_MAX_LENGTH: + raise ValidationError('name', 'name_too_long', name) + return name + + +def __validate_source(values: list[str]) -> list[str]: + """ + Validate the source of a rule: addresses, networks and ranges, IPv4 and IPv6 alike. An empty + source means the rule matches every host. + """ + sources = [] + for value in values or []: + if not isinstance(value, str) or not value.strip(): + raise ValidationError('source', 'invalid_source', value) + source = value.strip() + try: + if '-' in source: + first, last = (ipaddress.ip_address(part.strip()) for part in source.split('-', 1)) + if first.version != last.version or first > last: + raise ValueError(source) + else: + ipaddress.ip_network(source, strict=False) + except ValueError: + raise ValidationError('source', 'invalid_source', value) + if source not in sources: + sources.append(source) + return sources + + +def __validate_appgroups(e_uci: EUci, config_names: list[str], match_all: bool = False) -> list[str]: + """ + Validate the groups a rule matches. At least one is required: a rule with a source and no group + would be an IP-level rule, which the firewall does better, and a rule with neither would expand to + an empty criteria. + + `match_all` is the way out of that requirement, and the only one: a rule that matches every flow + names no group by definition, so naming one at the same time is a contradiction, not a narrowing. + """ + appgroups = [] + for config_name in config_names or []: + if not isinstance(config_name, str) or e_uci.get('dpi', config_name, default=None) != 'appgroup': + raise ValidationError('appgroups', 'appgroup_does_not_exists', config_name) + if config_name not in appgroups: + appgroups.append(config_name) + if match_all and appgroups: + raise ValidationError('appgroups', 'appgroups_not_allowed_with_match_all', appgroups) + if not match_all and not appgroups: + raise ValidationError('appgroups', 'appgroups_required', appgroups) + return appgroups + + +def expand_source(values: list[str]) -> list[str]: + """ + Expand the source of a rule into values a criteria can match: ranges become CIDR blocks, addresses + and networks are left as they are. + + Args: + - values: source entries of the rule + + Returns: + list of addresses and CIDR blocks + """ + expanded = [] + for value in values: + if '-' in value: + first, last = (ipaddress.ip_address(part.strip()) for part in value.split('-', 1)) + expanded.extend(str(network) for network in ipaddress.summarize_address_range(first, last)) else: - result.append({ - 'name': popular_filter, - 'missing': True - }) + expanded.append(value) + return expanded - total = len(result) - if limit is not None: - result = result[limit * (page - 1):limit * page] - last_page = math.ceil(total / limit) - else: - last_page = 1 +def build_rule_criteria(e_uci: EUci, rule: dict) -> str: + """ + Build the criteria of a managed rule: the source, when set, ANDed with the union of its groups. - return { - 'data': result, - 'meta': { - 'last_page': last_page, - 'total': total, - } - } + Args: + - e_uci: euci instance + - rule: the rule section, as returned by `uci.get_all` + + Returns: + the expression, or an empty string when the rule matches nothing and must not be emitted + """ + match = expand_appgroups(e_uci, list(rule.get('appgroup', []))) + sources = [f'local_ip == {source}' for source in expand_source(list(rule.get('source', [])))] + + if not match: + if not sources: + return DPI_MATCH_ALL_CRITERIA if rule.get('ns_match_all', '0') == '1' else '' + return f"({' || '.join(sources)});" + if sources: + return f"({' || '.join(sources)}) && {match};" + return f'{match};' -def list_rules(e_uci: EUci) -> list[dict[str]]: + +def list_rules(e_uci: EUci) -> list[dict]: """ - Index all rules + List the rules in priority order, the order they are evaluated in. + + Rules hidden with `ns_visible '0'` are left out: they are system entries the user must not touch. Args: - e_uci: euci instance Returns: - list of dicts, each dict contains the property "config-name", "description", "enabled", "interface" and "blocks" - """ - rules = list[dict[str]]() - fetch_rules = utils.get_all_by_type(e_uci, 'dpi', 'rule') - - if not fetch_rules: - return rules - - for rule_name in fetch_rules.keys(): - # skipping rules with criteria, must be custom entries - if e_uci.get('dpi', rule_name, 'criteria', default=None) is None: - # load blocklist of applications and protocols - blocklist = __load_blocklist() - # get content of rule - rule = fetch_rules[rule_name] - # prepare the data to append to rules - data_rule = dict[str]() - data_rule['config-name'] = rule_name - data_rule['enabled'] = rule.get('enabled', '1') == '1' - data_rule['device'] = rule.get('device', '*') - # from device, get the interface - interface = utils.get_interface_from_device(e_uci, data_rule['device']) - if interface is not None: - data_rule['interface'] = interface - data_rule['action'] = rule.get('action') - # get the blocked applications/protocols - data_rule['criteria'] = list[dict[str]]() - - # filter by application - application_blocklist = [item for item in blocklist if item['type'] == 'application'] - for application in rule.get('application', []): - found_app = [item for item in application_blocklist if item['name'] == application] - # there's a possibility of not finding the application due to manual edit of the config - if len(found_app) > 0: - data_rule['criteria'].append(found_app[0]) - - # filter by protocol - protocol_blocklist = [item for item in blocklist if item['type'] == 'protocol'] - for protocol in rule.get('protocol', []): - found_protocol = [item for item in protocol_blocklist if item['name'] == protocol] - # there's a possibility of not finding the protocol due to manual edit of the config - if len(found_protocol) > 0: - data_rule['criteria'].append(found_protocol[0]) - - # append rule - rules.append(data_rule) + list of dicts, each dict contains the property "id", "name", "enabled", "action", "source", + "appgroups", "match_all", "managed" and "index", plus "criteria" for the rules the API did not + create + """ + groups = {section: group.get('ns_name', '') + for section, group in (utils.get_all_by_type(e_uci, 'dpi', 'appgroup') or {}).items()} + rules = [] + for section, rule in __sorted_rules(e_uci): + if rule.get('ns_visible', '1') == '0': + continue + data_rule = { + 'id': section, + 'name': rule.get('ns_name', ''), + 'enabled': rule.get('enabled', '1') == '1', + 'action': rule.get('action', ''), + 'source': list(rule.get('source', [])), + 'appgroups': [{'id': group, 'name': groups.get(group, group)} + for group in rule.get('appgroup', [])], + 'match_all': rule.get('ns_match_all', '0') == '1', + # a rule the UI did not create can be renamed, toggled, reordered and deleted, but not edited + 'managed': rule.get('ns_managed', '0') == '1', + 'index': len(rules) + } + if 'criteria' in rule: + data_rule['criteria'] = rule.get('criteria') + rules.append(data_rule) return rules -def __save_rule_data(e_uci: EUci, config_name: str, enabled: bool, device: str, action: str, applications: list[str], - protocols: list[str]): +def __save_rule_data(e_uci: EUci, config_name: str, name: str, enabled: bool, action: str, + source: list[str], appgroups: list[str], match_all: bool = False): + e_uci.set('dpi', config_name, 'ns_name', name) + e_uci.set('dpi', config_name, 'ns_managed', '1') e_uci.set('dpi', config_name, 'enabled', enabled) - e_uci.set('dpi', config_name, 'device', device) e_uci.set('dpi', config_name, 'action', action) - e_uci.set('dpi', config_name, 'application', applications) - e_uci.set('dpi', config_name, 'protocol', protocols) + if match_all: + e_uci.set('dpi', config_name, 'ns_match_all', '1') + else: + e_uci.delete('dpi', config_name, 'ns_match_all') + if appgroups: + e_uci.set('dpi', config_name, 'appgroup', appgroups) + else: + # a match-all rule or a rule migrated from a global exemption has no group: the option must go, not stay empty + e_uci.delete('dpi', config_name, 'appgroup') + if source: + e_uci.set('dpi', config_name, 'source', source) + else: + # no source means every host: the option must go, not stay empty + e_uci.delete('dpi', config_name, 'source') -def __save_exemption_data(e_uci: EUci, config_name: str, criteria: str, description: str, enabled: bool): - e_uci.set('dpi', config_name, 'enabled', enabled) - e_uci.set('dpi', config_name, 'criteria', criteria) - e_uci.set('dpi', config_name, 'description', description) -def __toggle_engine(e_uci: EUci): - count_enabled = 0 - for section in e_uci.get_all('dpi'): - if e_uci.get('dpi', section, default="") == "rule" and e_uci.get('dpi', section, 'enabled', default="0") == "1": - count_enabled = count_enabled + 1 +def add_rule(e_uci: EUci, name: str, enabled: bool, action: str, source: list[str], + appgroups: list[str], position: str = 'bottom', match_all: bool = False) -> str: + """ + Store a new rule. - if count_enabled > 0: - e_uci.set('dpi', 'config', 'enabled', '1') - else: - e_uci.set('dpi', 'config', 'enabled', '0') + Args: + - e_uci: euci instance + - name: name of the rule + - enabled: enable the rule + - action: 'block' or 'allow' + - source: list of addresses, networks or ranges, empty to match every host + - appgroups: config names of the application groups the rule matches, at least one unless + `match_all` is set + - position: 'top' to evaluate the rule before every other one, 'bottom' after them + - match_all: the rule matches every flow, narrowed to `source` when one is given. Mutually + exclusive with `appgroups` + + Returns: + config name of the rule created + + Raises: + - ValidationError: if any argument is invalid + """ + name = __validate_rule_name(name) + if action not in DPI_RULE_ACTIONS: + raise ValidationError('action', 'invalid_action', action) + if position not in DPI_RULE_POSITIONS: + raise ValidationError('position', 'invalid_position', position) + source = __validate_source(source) + appgroups = __validate_appgroups(e_uci, appgroups, match_all) + + config_name = utils.get_random_id() + e_uci.set('dpi', config_name, 'rule') + __save_rule_data(e_uci, config_name, name, enabled, action, source, appgroups, match_all) + + order = [section for section, _ in __sorted_rules(e_uci) if section != config_name] + order.insert(0, config_name) if position == 'top' else order.append(config_name) + __renumber_rules(e_uci, order) + + e_uci.save('dpi') + return config_name -def add_rule(e_uci: EUci, enabled: bool, device: str, action: str, applications: list[str], - protocols: list[str]) -> str: + +def edit_rule(e_uci: EUci, config_name: str, name: str, enabled: bool, action: str, source: list[str], + appgroups: list[str], match_all: bool = False) -> str: """ - Store a new rule + Edit a rule. Only rules created through the API can be edited: a rule carrying a hand-written + criteria has no source and no group to fill the form with, and rewriting it would change what it + matches. Args: - e_uci: euci instance - - description: description of the rule + - config_name: config name of the rule to edit + - name: name of the rule - enabled: enable the rule - - action: apply specific action to rule, can be 'block', 'bulk', 'best_effort', 'video' or 'voice' - - device: device to listen and apply the rule on - - applications: list of applications to block - - protocols: list of protocols to block + - action: 'block' or 'allow' + - source: list of addresses, networks or ranges, empty to match every host + - appgroups: config names of the application groups the rule matches, at least one unless + `match_all` is set + - match_all: the rule matches every flow, narrowed to `source` when one is given. Mutually + exclusive with `appgroups` Returns: - config name of the rule created + config name of the rule edited + + Raises: + - ValidationError: if the rule does not exist, is not managed, or any argument is invalid """ - rule_name = utils.get_random_id() - e_uci.set('dpi', rule_name, 'rule') - __save_rule_data(e_uci, rule_name, enabled, device, action, applications, protocols) - __toggle_engine(e_uci) + if not __is_rule(e_uci, config_name): + raise ValidationError('id', 'rule_not_found', config_name) + if e_uci.get('dpi', config_name, 'ns_managed', default='0') != '1': + raise ValidationError('id', 'rule_not_managed', config_name) + + name = __validate_rule_name(name) + if action not in DPI_RULE_ACTIONS: + raise ValidationError('action', 'invalid_action', action) + source = __validate_source(source) + appgroups = __validate_appgroups(e_uci, appgroups, match_all) + + __save_rule_data(e_uci, config_name, name, enabled, action, source, appgroups, match_all) e_uci.save('dpi') - return rule_name + return config_name -def delete_rule(e_uci: EUci, config_name: str): +def delete_rule(e_uci: EUci, config_name: str) -> str: """ - Delete a rule + Delete a rule and close the gap its priority leaves behind. Args: - e_uci: euci instance - config_name: config name of the rule to delete + + Returns: + config name of the rule deleted + + Raises: + - ValidationError: if the rule does not exist """ + if not __is_rule(e_uci, config_name): + raise ValidationError('id', 'rule_not_found', config_name) + e_uci.delete('dpi', config_name) - __toggle_engine(e_uci) + __renumber_rules(e_uci) e_uci.save('dpi') + return config_name -def edit_rule(e_uci: EUci, config_name: str, enabled: bool, device: str, action: str, applications: list[str], - protocols: list[str]): +def rename_rule(e_uci: EUci, config_name: str, name: str) -> str: """ - Edit a rule + Rename a rule, managed or not. Args: - e_uci: euci instance - - config_name: rule to change - - enabled: enable the rule - - device: device to listen and apply the rule on - - action: apply specific action to rule, can be 'block', 'bulk', 'best_effort', 'video' or 'voice' - - applications: array of applications to block - - protocols: array of protocols to block + - config_name: config name of the rule to rename + - name: new name of the rule - Raises - - ValidationError: if the config name is invalid - """ - if e_uci.get('dpi', config_name, default=None) is None: - raise ValidationError('config-name', 'invalid', config_name) + Returns: + config name of the rule renamed - __save_rule_data(e_uci, config_name, enabled, device, action, applications, protocols) - __toggle_engine(e_uci) + Raises: + - ValidationError: if the rule does not exist or the name is invalid + """ + if not __is_rule(e_uci, config_name): + raise ValidationError('id', 'rule_not_found', config_name) + e_uci.set('dpi', config_name, 'ns_name', __validate_rule_name(name)) e_uci.save('dpi') + return config_name + -def list_exemptions(e_uci: EUci) -> list[dict[str]]: +def enable_rule(e_uci: EUci, config_name: str) -> str: """ - Index all global exemptions + Enable a rule, managed or not. Args: - e_uci: euci instance + - config_name: config name of the rule to enable Returns: - list of dicts, each dict contains the property "config-name", "description", "enabled", "criteria" - """ - exemptions = list[dict[str]]() - fetch_ex = utils.get_all_by_type(e_uci, 'dpi', 'exemption') + config name of the rule enabled - if not fetch_ex: - return exemptions - for ex_name in fetch_ex.keys(): - # get content of exemption - ex = fetch_ex[ex_name] - # prepare the data to append to rules - data_ex = dict[str]() - data_ex['config-name'] = ex_name - data_ex['enabled'] = ex.get('enabled', '1') == '1' - data_ex['criteria'] = ex.get('criteria', '') - data_ex['description'] = ex.get('description', '') - # append exemption - exemptions.append(data_ex) + Raises: + - ValidationError: if the rule does not exist + """ + if not __is_rule(e_uci, config_name): + raise ValidationError('id', 'rule_not_found', config_name) - return exemptions + e_uci.set('dpi', config_name, 'enabled', '1') + e_uci.save('dpi') + return config_name -def add_exemption(e_uci: EUci, criteria: str, description: str, enabled: bool): +def disable_rule(e_uci: EUci, config_name: str) -> str: """ - Store a new global exemption + Disable a rule, managed or not. Args: - e_uci: euci instance - - criteria: exemption criteria, usually it's an IP address - - description: description of the rule - - enabled: enable the exemption + - config_name: config name of the rule to disable Returns: - config name of the exemption created + config name of the rule disabled + + Raises: + - ValidationError: if the rule does not exist """ - ex_list = utils.get_all_by_type(e_uci, 'dpi', 'exemption') - for ex_name in ex_list: - ex = ex_list[ex_name] - if ex.get('criteria', '') == criteria: - raise ValidationError('criteria', 'criteria_already_exists', criteria) + if not __is_rule(e_uci, config_name): + raise ValidationError('id', 'rule_not_found', config_name) - ex_name = utils.get_random_id() - e_uci.set('dpi', ex_name, 'exemption') - __save_exemption_data(e_uci, ex_name, criteria, description, enabled) + e_uci.set('dpi', config_name, 'enabled', '0') e_uci.save('dpi') - return ex_name + return config_name -def delete_exemption(e_uci: EUci, config_name: str): +def order_rules(e_uci: EUci, order: list[str]) -> list[str]: """ - Delete a global exemption + Reorder every rule, renumbering the priorities densely from the given order. + + The whole set must be listed, hidden rules included: a partial order would silently move the rules + left out, which with a paginated list is a rule the caller never saw. Args: - e_uci: euci instance - - config_name: config name of the rule to delete + - order: config names of every rule, in the order they must be evaluated + + Returns: + the list of the ordered rules + + Raises: + - ValidationError: if the order does not name every rule exactly once """ - e_uci.delete('dpi', config_name) + rules = [section for section, _ in __sorted_rules(e_uci)] + for section in order: + if section not in rules: + raise ValidationError('order', 'rule_not_found', section) + if len(order) != len(set(order)) or len(order) != len(rules): + raise ValidationError('order', 'invalid_order', order) + + __renumber_rules(e_uci, order) e_uci.save('dpi') + return order + +# Migration: one-time, in-place bump of /etc/config/dpi from the schema used before application groups +# existed. A rule from that schema carries `device`, `application`, `protocol` or `category` — fields +# only that API ever wrote, so their presence is a permanent, unambiguous marker regardless of whether +# `priority` was since assigned to it by an unrelated add/edit/delete. A global exemption is identified +# by its section type, which this migration removes once every exemption has become a rule. -def edit_exemption(e_uci: EUci, config_name: str, criteria: str, description: str, enabled: bool): +def freeze_legacy_criteria(e_uci: EUci, rule: dict) -> str: """ - Edit a global exemption + Build the criteria of a rule written before application groups existed, the way `dpi-config` has + always generated it: source, application, protocol and category turned into an expression narrowed + to the rule's device, with the VLAN rewrite applied when the device names a VLAN. Used by the + migration to freeze a rule's behaviour verbatim before clearing those fields. Args: - e_uci: euci instance - - config_name: rule to change - - criteria: exemption criteria, usually it's an IP address - - description: description of the rule - - enabled: enable the exemption + - rule: the rule section, as returned by `uci.get_all` - Raises - - ValidationError: if the config name is invalid + Returns: + the semicolon-terminated expression + """ + device = rule.get('device', '*') + + if 'criteria' in rule: + # criteria has precedence over source, protocol, category and application + criteria = rule['criteria'].replace('"', "'") + else: + sources = [] + for source in rule.get('source', []): + if objects.is_object_id(source): + for ip in objects.get_object_ips(e_uci, source): + sources.append(f'local_ip == {ip}') + else: + sources.append(f'local_ip == {source}') + + applications = [] + for app in rule.get('application', []): + applications.append(f"app == '{app}'") + for proto in rule.get('protocol', []): + applications.append(f"proto == '{proto.lower()}'") + for cat in rule.get('category', []): + applications.append(f"category == '{cat.lower()}'") + + sources_s = ' or '.join(sources) + applications_s = ' or '.join(applications) + criteria = f"(iface_nfq_src == '{device}' or iface_nfq_dst == '{device}') && " + if len(sources) < 1: + criteria += f'({applications_s}) ;' + elif len(applications) < 1: + criteria += f'({sources_s}) ;' + else: + criteria += f'({sources_s}) && ({applications_s}) ;' + + vlan_id = None + base_if = None + for item in utils.get_all_by_type(e_uci, 'network', 'device').values(): + if item.get('vid', None) is not None and item.get('name', '') == device: + vlan_id = item.get('vid', None) + base_if = item.get('ifname', None) + break + + if vlan_id is not None and base_if is not None: + criteria = f'vlan_id == {vlan_id} && {criteria}' + + return criteria + + +def __exemption_source(e_uci: EUci, criteria: str) -> list[str] | None: + """ + Turn a global exemption's criteria into a rule source. A firewall object is expanded into its + member addresses; a plain address, network or range is kept as is. + + Returns: + the source list, or None when the criteria is empty or does not resolve to at least one + address, telling the caller to fall back to an unmanaged rule carrying it verbatim + """ + if not criteria: + return None + if objects.is_object_id(criteria): + return objects.get_object_ips(e_uci, criteria) or None + try: + return __validate_source([criteria]) + except ValidationError: + return None + + +def migrate_schema(e_uci: EUci): """ - if e_uci.get('dpi', config_name, default=None) is None: - raise ValidationError('config-name', 'invalid', config_name) + Migrate every rule and exemption from the schema used before application groups existed. + + Each legacy rule (`device`, `application`, `protocol` or `category` set) is turned into an unmanaged + rule with its behaviour frozen verbatim into `criteria` — see `freeze_legacy_criteria`. A rule whose + action is a retired QoS value, or that ends up matching nothing, is dropped instead, with a log line + naming it. A per-rule exemption or `log` (fields no recent API ever wrote) is removed from the rule, + with a log line as well. + + Each global exemption becomes an Allow rule at the top of the list: a managed match-all rule + narrowed to a source when its criteria is a plain address, CIDR or firewall object; an unmanaged + one carrying the criteria verbatim otherwise. Disabled exemptions become disabled rules. The rule + is named after the exemption description, cut to the rule name limit, or `Migrated exception N` + when it has none. + The `exemption` section type and `popular_filters` are then removed. + + The changes are committed and the dpi service reloaded right away. + + Safe to call unconditionally on every boot: a box with nothing left in the old schema returns + without touching UCI. + + Args: + - e_uci: euci instance + """ + exemptions = utils.get_all_by_type(e_uci, 'dpi', 'exemption') or {} + rules = utils.get_all_by_type(e_uci, 'dpi', 'rule') or {} + legacy_fields = ('device', 'application', 'protocol', 'category') + legacy_rules = {section for section, rule in rules.items() + if any(field in rule for field in legacy_fields)} + + if not exemptions and not legacy_rules: + return + + order = [] + + exemption_count = 0 + for section, exemption in exemptions.items(): + exemption_count += 1 + # keep the name the user gave the exemption, cut to what a rule name allows + name = exemption.get('description', '').strip()[:DPI_RULE_NAME_MAX_LENGTH].strip() + if not name: + name = f'Migrated exception {exemption_count}' + enabled = exemption.get('enabled', '1') == '1' + criteria = exemption.get('criteria', '').strip() + source = __exemption_source(e_uci, criteria) + + e_uci.delete('dpi', section) + if source is not None: + rule_id = add_rule(e_uci, name, enabled, 'allow', source, [], match_all=True) + else: + rule_id = utils.get_random_id() + e_uci.set('dpi', rule_id, 'rule') + e_uci.set('dpi', rule_id, 'ns_name', name) + e_uci.set('dpi', rule_id, 'enabled', '1' if enabled else '0') + e_uci.set('dpi', rule_id, 'action', 'allow') + e_uci.set('dpi', rule_id, 'criteria', criteria if criteria.endswith(';') else f'{criteria};') + order.append(rule_id) + + rule_count = 0 + for section, rule in rules.items(): + if section not in legacy_rules: + order.append(section) + continue + + action = rule.get('action') + if action != 'block': + syslog.syslog(syslog.LOG_WARNING, f"dpi migration: dropping {section}, unsupported action {action!r}") + e_uci.delete('dpi', section) + continue + if rule.get('exemption'): + syslog.syslog(syslog.LOG_WARNING, + f"dpi migration: dropping the per-rule exemption on {section}, no longer supported") + if rule.get('log') == '1': + syslog.syslog(syslog.LOG_WARNING, + f"dpi migration: dropping the per-rule log on {section}, no longer supported") + if not any(rule.get(field) for field in ('criteria', 'source', 'application', 'protocol', 'category')): + syslog.syslog(syslog.LOG_WARNING, f"dpi migration: dropping {section}, it matches nothing") + e_uci.delete('dpi', section) + continue + + rule_count += 1 + criteria = freeze_legacy_criteria(e_uci, rule) + for field in (*legacy_fields, 'source', 'exemption', 'log'): + e_uci.delete('dpi', section, field) + e_uci.set('dpi', section, 'criteria', criteria) + e_uci.set('dpi', section, 'ns_name', f'Migrated rule {rule_count}') + order.append(section) - __save_exemption_data(e_uci, config_name, criteria, description, enabled) + __renumber_rules(e_uci, order) + e_uci.delete('dpi', 'config', 'popular_filters') e_uci.save('dpi') + e_uci.commit('dpi') + subprocess.run(["/etc/init.d/dpi", "reload"], check=True) diff --git a/packages/python3-nethsec/src/nethsec/inventory/__init__.py b/packages/python3-nethsec/src/nethsec/inventory/__init__.py index 413c2d6c9..cf22273a7 100644 --- a/packages/python3-nethsec/src/nethsec/inventory/__init__.py +++ b/packages/python3-nethsec/src/nethsec/inventory/__init__.py @@ -641,11 +641,43 @@ def fact_proxy_pass(uci: EUci): return ret def fact_dpi(uci: EUci): - ret = {"enabled": False, "rules": 0} - ret["enabled"] = uci.get('dpi', 'config', 'enabled', default='0') == '1' - for rule in utils.get_all_by_type(uci, 'dpi', 'rule'): - if uci.get('dpi', rule, 'enabled', default='0') == '1': - ret["rules"] += 1 + ret = { + "total": 0, + "enabled": 0, + "action": {"block": 0, "allow": 0}, + "source": {"all": 0, "ip": 0}, + "match": {"all": 0, "appgroup": 0}, + "kind": {"managed": 0, "unmanaged": 0}, + "appgroups": {"total": 0, "used": 0, "app": 0, "app_category": 0, "proto": 0, "proto_category": 0}, + } + used_appgroups = set() + for rule in (utils.get_all_by_type(uci, 'dpi', 'rule') or {}).values(): + ret["total"] += 1 + if rule.get('enabled', '0') == '1': + ret["enabled"] += 1 + if rule.get('action') in ret["action"]: + ret["action"][rule.get('action')] += 1 + # raw criteria is emitted as is, source and appgroup don't apply + if not rule.get('criteria'): + if rule.get('source'): + ret["source"]["ip"] += 1 + else: + ret["source"]["all"] += 1 + if rule.get('ns_match_all', '0') == '1': + ret["match"]["all"] += 1 + elif rule.get('appgroup'): + ret["match"]["appgroup"] += 1 + if rule.get('ns_managed', '0') == '1': + ret["kind"]["managed"] += 1 + else: + ret["kind"]["unmanaged"] += 1 + used_appgroups.update(rule.get('appgroup', [])) + for name, appgroup in (utils.get_all_by_type(uci, 'dpi', 'appgroup') or {}).items(): + ret["appgroups"]["total"] += 1 + if name in used_appgroups: + ret["appgroups"]["used"] += 1 + for member in ('app', 'app_category', 'proto', 'proto_category'): + ret["appgroups"][member] += len(appgroup.get(member, [])) return ret def fact_extra_packages(uci: EUci): diff --git a/packages/python3-nethsec/tests/test_dpi.py b/packages/python3-nethsec/tests/test_dpi.py index 2e049dba2..739585915 100644 --- a/packages/python3-nethsec/tests/test_dpi.py +++ b/packages/python3-nethsec/tests/test_dpi.py @@ -1,33 +1,13 @@ import pathlib +import subprocess import pytest from euci import EUci from pytest_mock import MockFixture -from nethsec import dpi +from nethsec import dpi, utils from nethsec.utils import ValidationError -applications_file = """ -# Comment to avoid - -# another comment to avoid - -app:133:netify.netflix -app:10119:netify.linkedin -app:10552:netify.tesla -app:10195:netify.avira -app:10194:netify.sophos -app:10244:netify.bbc -app:10362:netify.hulu -app:10118:netify.lets-encrypt -app:199:netify.snapchat -dom:133:dualstack.apiproxy-device-prod-nlb-1-4d12762d4ba53e45.elb.eu-west-1.amazonaws.com -net:10852:208.118.237.0/24 -net:124:208.117.254.0/24 -nsd:-1:39:YXBwID09ICduZXRpZnkuc2lnbmFsJyAmJiBwcm90b2NvbF9pZCA9PSA3OCAmJiAob3RoZXJfcG9ydCA9PSA0NDMgfHwgbG9jYWxfcG9ydCA9PSA0NDMpOw== -nsd:-1:67:YXBwID09ICduZXRpZnkuZ29vZ2xlLWNoYXQnICYmIHByb3RvY29sX2lkICE9IDY3ICYmIChvdGhlcl9wb3J0ID09IDUyMjggfHwgbG9jYWxfcG9ydCA9PSA1MjI4KTs= -""" - applications = { 133: 'netify.netflix', 10119: 'netify.linkedin', @@ -50,153 +30,18 @@ 130: 'HTTP/Connect' } -categories_file = """ -{ - "application_index": [ - [ - 3, - [ - 10362, - 10194, - 10552 - ] - ], - [ - 33, - [ - 133, - 10119, - 10195, - 10244 - ] - ], - [ - 20, - [ - 199 - ] - ] - ], - "application_tag_index": { - "first-category": 3, - "unknown": 33, - "last": 20 - }, - "protocol_index": [ - [ - 1, - [ - 121, - 127, - 128 - ] - ], - [ - 2, - [ - 116 - ] - ], - [ - 4, - [ - 129, - 130 - ] - ] - ], - "protocol_tag_index": { - "base": 1, - "games": 2, - "low": 4 - } -} -""" - -categories_file_old = """ -{ - "last_update": 1696950440, - "application_index": { - "first-category": [ - 10362, - 10194, - 10552 - ], - "unknown": [ - 133, - 10119, - 10195, - 10244 - ], - "last": [ - 199 - ] - }, - "protocol_index": { - "base": [ - 121, - 127, - 128 - ], - "games": [ - 116 - ], - "low": [ - 129, - 130 - ] - } -} +application_output = """ + 133: netify.netflix + 10119: netify.linkedin + 10552: netify.tesla + 10195: netify.avira + 10194: netify.sophos + 10244: netify.bbc + 10362: netify.hulu + 10118: netify.lets-encrypt + 199: netify.snapchat """ -application_categories = { - 10362: { - 'name': 'first-category' - }, - 10194: { - 'name': 'first-category' - }, - 10552: { - 'name': 'first-category' - }, - 133: { - 'name': 'unknown' - }, - 10119: { - 'name': 'unknown' - }, - 10195: { - 'name': 'unknown' - }, - 10244: { - 'name': 'unknown' - }, - 199: { - 'name': 'last' - } -} - -protocol_categories = { - 121: { - 'name': 'base' - }, - 127: { - 'name': 'base' - }, - 128: { - 'name': 'base' - }, - 116: { - 'name': 'games' - }, - 129: { - 'name': 'low' - }, - 130: { - 'name': 'low' - } -} - protocol_output = """ 116: Warcraft3 117: LotusNotes @@ -210,15 +55,11 @@ dpi_minimal_db = """ config main 'config' option log_blocked '0' - option firewall_exemption '0' - option enabled '0' """ dpi_db = """ config main 'config' option log_blocked '0' - option firewall_exemption '0' - option enabled '0' list popular_filters 'netify.netflix' list popular_filters 'netify.hulu' list popular_filters 'netify.whatsapp' @@ -257,11 +98,6 @@ list protocol 'HTTP/Connect' option device 'eth1' option enabled 1 - -config exemption exemp1 - option criteria '192.168.1.1' - option description 'my host' - option enabled 1 """ netifyd_config = """ @@ -349,6 +185,33 @@ """ +DPI_RELOAD = ["/etc/init.d/dpi", "reload"] + + +@pytest.fixture(autouse=True) +def mock_reload(mocker: MockFixture): + """ + The schema migration reloads the dpi service right away; tests only care about the resulting uci + state, not about actually reloading a system service. Every other command runs as usual. + """ + run = subprocess.run + return mocker.patch('nethsec.dpi.subprocess.run', + side_effect=lambda args, *a, **kw: None if args == DPI_RELOAD else run(args, *a, **kw)) + + +def reloaded(mock_reload) -> bool: + return any(call.args and call.args[0] == DPI_RELOAD for call in mock_reload.call_args_list) + + +def has_pending_changes(e_uci: EUci) -> bool: + delta = pathlib.Path(e_uci.savedir()).joinpath('dpi') + return delta.exists() and delta.stat().st_size > 0 + + +def dpi_config_file(e_uci: EUci) -> str: + return pathlib.Path(e_uci.confdir()).joinpath('dpi').read_text() + + @pytest.fixture def e_uci(tmp_path: pathlib.Path) -> EUci: conf_dir = tmp_path.joinpath('conf') @@ -373,695 +236,1023 @@ def e_uci_with_data(e_uci: EUci): return e_uci -@pytest.fixture -def mock_load(mocker): - mocker.patch('nethsec.dpi.__load_applications', return_value=applications) - mocker.patch('nethsec.dpi.__load_application_categories', return_value=application_categories) - mocker.patch('nethsec.dpi.load_protocols', return_value=protocols) - mocker.patch('nethsec.dpi.__load_protocol_categories', return_value=protocol_categories) +def test_load_applications_from_engine(mocker: MockFixture): + process_result = mocker.stub('subprocess_return') + process_result.stdout = bytes(application_output, 'utf-8') + mocker.patch('subprocess.run', return_value=process_result) + assert dpi.load_applications() == applications -def test_load_applications(mocker: MockFixture): - mocker.patch('builtins.open', mocker.mock_open(read_data=applications_file)) - assert dpi.__load_applications() == applications +def test_load_protocols(mocker: MockFixture): + process_result = mocker.stub('subprocess_return') + process_result.stdout = bytes(protocol_output, 'utf-8') + mocker.patch('subprocess.run', return_value=process_result) + assert dpi.load_protocols() == protocols + +# application groups -@pytest.mark.parametrize('data', [categories_file, categories_file_old]) -def test_load_application_categories(mocker: MockFixture, data: str): - mocker.patch('builtins.open', mocker.mock_open(read_data=data)) - assert dpi.__load_application_categories() == application_categories +category_output = """ + 1: application: adult + 5: application: cybersecurity + 12: application: games + 5: protocol: games + 18: protocol: web +""" + +categories = { + 'application': {1: 'adult', 5: 'cybersecurity', 12: 'games'}, + 'protocol': {5: 'games', 18: 'web'} +} -@pytest.mark.parametrize('data', [categories_file, categories_file_old]) -def test_load_protocol_categories(mocker: MockFixture, data: str): - mocker.patch('builtins.open', mocker.mock_open(read_data=data)) - assert dpi.__load_protocol_categories() == protocol_categories +@pytest.fixture +def mock_vocabularies(mocker): + """Every member kind resolves against the engine, the catalogs are never reached.""" + mocker.patch('nethsec.dpi.load_applications', return_value=applications) + mocker.patch('nethsec.dpi.load_protocols', return_value=protocols) + mocker.patch('nethsec.dpi.load_categories', return_value=categories) -def test_load_protocols(mocker: MockFixture): +@pytest.fixture +def e_uci_appgroups(e_uci_with_data, mock_vocabularies): + return e_uci_with_data + + +def test_load_categories(mocker: MockFixture): process_result = mocker.stub('subprocess_return') - process_result.stdout = bytes(protocol_output, 'utf-8') + process_result.stdout = bytes(category_output, 'utf-8') mocker.patch('subprocess.run', return_value=process_result) - assert dpi.load_protocols() == protocols + assert dpi.load_categories() == categories -@pytest.mark.parametrize('search', [None, '']) -def test_index_applications(mock_load, search): - assert dpi.list_applications(search) == { - 'data': [ - { - 'id': 133, - 'name': 'netify.netflix', - 'type': 'application', - 'category': { - 'name': 'unknown' - } - }, - { - 'id': 10119, - 'name': 'netify.linkedin', - 'type': 'application', - 'category': { - 'name': 'unknown' - } - }, - { - 'id': 10552, - 'name': 'netify.tesla', - 'type': 'application', - 'category': { - 'name': 'first-category' - } - }, - { - 'id': 10195, - 'name': 'netify.avira', - 'type': 'application', - 'category': { - 'name': 'unknown' - } - }, - { - 'id': 10194, - 'name': 'netify.sophos', - 'type': 'application', - 'category': { - 'name': 'first-category' - } - }, - { - 'id': 10244, - 'name': 'netify.bbc', - 'type': 'application', - 'category': { - 'name': 'unknown' - } - }, - { - 'id': 10362, - 'name': 'netify.hulu', - 'type': 'application', - 'category': { - 'name': 'first-category' - } - }, - { - 'id': 10118, - 'name': 'netify.lets-encrypt', - 'type': 'application' - }, - { - 'id': 199, - 'name': 'netify.snapchat', - 'type': 'application', - 'category': { - 'name': 'last' - } - }, - { - 'id': 116, - 'name': 'Warcraft3', - 'type': 'protocol', - 'category': { - 'name': 'games' - } - }, - { - 'id': 117, - 'name': 'LotusNotes', - 'type': 'protocol' - }, - { - 'id': 121, - 'name': 'Dropbox', - 'type': 'protocol', - 'category': { - 'name': 'base' - } - }, - { - 'id': 127, - 'name': 'RPC', - 'type': 'protocol', - 'category': { - 'name': 'base' - } - }, - { - 'id': 128, - 'name': 'NetFlow', - 'type': 'protocol', - 'category': { - 'name': 'base' - } - }, - { - 'id': 129, - 'name': 'SFlow', - 'type': 'protocol', - 'category': { - 'name': 'low' - } - }, - { - 'id': 130, - 'name': 'HTTP/Connect', - 'type': 'protocol', - 'category': { - 'name': 'low' - } - } - ], - 'meta': { - 'last_page': 1, - 'total': 16 - } - } +def test_add_appgroup(e_uci_appgroups): + group_id = dpi.add_appgroup(e_uci_appgroups, 'Group one', ['netify.netflix'], ['cybersecurity'], + ['Dropbox'], ['web']) + assert e_uci_appgroups.get('dpi', group_id) == 'appgroup' + assert e_uci_appgroups.get('dpi', group_id, 'ns_name') == 'Group one' + assert e_uci_appgroups.get('dpi', group_id, 'app', list=True) == ('netify.netflix',) + assert e_uci_appgroups.get('dpi', group_id, 'app_category', list=True) == ('cybersecurity',) + assert e_uci_appgroups.get('dpi', group_id, 'proto', list=True) == ('Dropbox',) + assert e_uci_appgroups.get('dpi', group_id, 'proto_category', list=True) == ('web',) + + +def test_add_appgroup_strips_and_deduplicates(e_uci_appgroups): + group_id = dpi.add_appgroup(e_uci_appgroups, ' Group one ', + [' netify.netflix ', 'netify.netflix', 'NETIFY.NETFLIX'], [], [], []) + assert e_uci_appgroups.get('dpi', group_id, 'ns_name') == 'Group one' + assert e_uci_appgroups.get('dpi', group_id, 'app', list=True) == ('netify.netflix',) + + +def test_add_appgroup_with_one_kind_only(e_uci_appgroups): + group_id = dpi.add_appgroup(e_uci_appgroups, 'Only protocols', protocols=['HTTP/Connect']) + assert e_uci_appgroups.get('dpi', group_id, 'proto', list=True) == ('HTTP/Connect',) + assert e_uci_appgroups.get('dpi', group_id, 'app', default=None) is None + + +def test_add_appgroup_requires_a_name(e_uci_appgroups): + with pytest.raises(ValidationError) as err: + dpi.add_appgroup(e_uci_appgroups, ' ', ['netify.netflix']) + assert err.value.args[0] == 'name' + assert err.value.args[1] == 'name_required' + + +def test_add_appgroup_refuses_a_long_name(e_uci_appgroups): + with pytest.raises(ValidationError) as err: + dpi.add_appgroup(e_uci_appgroups, 'x' * 65, ['netify.netflix']) + assert err.value.args[1] == 'name_too_long' + + +def test_add_appgroup_refuses_a_duplicated_name(e_uci_appgroups): + dpi.add_appgroup(e_uci_appgroups, 'Group one', ['netify.netflix']) + with pytest.raises(ValidationError) as err: + dpi.add_appgroup(e_uci_appgroups, 'group ONE', ['netify.tesla']) + assert err.value.args[1] == 'name_already_exists' + + +def test_add_appgroup_refuses_an_empty_group(e_uci_appgroups): + with pytest.raises(ValidationError) as err: + dpi.add_appgroup(e_uci_appgroups, 'Empty', [], [], [], []) + assert err.value.args[0] == 'members' + assert err.value.args[1] == 'appgroup_is_empty' + + +@pytest.mark.parametrize('value', ["netify.netflix'", 'netify.netflix;', 'netify.(netflix)', + 'netify.netflix\\', 'netify\nnetify.netflix', ' ']) +def test_add_appgroup_refuses_expression_breaking_members(e_uci_appgroups, value): + with pytest.raises(ValidationError) as err: + dpi.add_appgroup(e_uci_appgroups, 'Group one', [value]) + assert err.value.args[0] == 'applications' + assert err.value.args[1] == 'invalid_application' + + +def test_add_appgroup_refuses_unknown_members(e_uci_appgroups): + with pytest.raises(ValidationError) as err: + dpi.add_appgroup(e_uci_appgroups, 'Group one', ['netify.nonexistent']) + assert err.value.args[1] == 'invalid_application' + with pytest.raises(ValidationError) as err: + dpi.add_appgroup(e_uci_appgroups, 'Group one', protocols=['nonexistent']) + assert err.value.args[1] == 'invalid_protocol' + with pytest.raises(ValidationError) as err: + dpi.add_appgroup(e_uci_appgroups, 'Group one', application_categories=['web']) + assert err.value.args[1] == 'invalid_application_category' + with pytest.raises(ValidationError) as err: + dpi.add_appgroup(e_uci_appgroups, 'Group one', protocol_categories=['cybersecurity']) + assert err.value.args[1] == 'invalid_protocol_category' + + +def test_add_appgroup_refuses_a_protocol_named_by_its_catalog_tag(e_uci_appgroups): + # the catalog calls it "http-connect", but only the name the engine reports can be matched + with pytest.raises(ValidationError) as err: + dpi.add_appgroup(e_uci_appgroups, 'Group one', protocols=['http-connect']) + assert err.value.args[1] == 'invalid_protocol' + + +def test_add_appgroup_matches_members_case_insensitively(e_uci_appgroups): + group_id = dpi.add_appgroup(e_uci_appgroups, 'Group one', protocols=['http/connect']) + assert e_uci_appgroups.get('dpi', group_id, 'proto', list=True) == ('http/connect',) + + +def test_add_appgroup_skips_validation_without_a_vocabulary(e_uci_with_data, mocker): + mocker.patch('nethsec.dpi.load_applications', side_effect=FileNotFoundError) + mocker.patch('nethsec.dpi.__load_catalog', side_effect=FileNotFoundError) + group_id = dpi.add_appgroup(e_uci_with_data, 'Group one', ['netify.unknown-to-everything']) + assert e_uci_with_data.get('dpi', group_id, 'app', list=True) == ('netify.unknown-to-everything',) + + +def test_edit_appgroup(e_uci_appgroups): + group_id = dpi.add_appgroup(e_uci_appgroups, 'Group one', ['netify.netflix'], ['cybersecurity']) + dpi.edit_appgroup(e_uci_appgroups, group_id, 'Group two', ['netify.tesla']) + assert e_uci_appgroups.get('dpi', group_id, 'ns_name') == 'Group two' + assert e_uci_appgroups.get('dpi', group_id, 'app', list=True) == ('netify.tesla',) + # an emptied list is removed, not stored empty + assert e_uci_appgroups.get('dpi', group_id, 'app_category', default=None) is None + + +def test_edit_appgroup_keeps_its_own_name(e_uci_appgroups): + group_id = dpi.add_appgroup(e_uci_appgroups, 'Group one', ['netify.netflix']) + dpi.edit_appgroup(e_uci_appgroups, group_id, 'Group one', ['netify.tesla']) + assert e_uci_appgroups.get('dpi', group_id, 'app', list=True) == ('netify.tesla',) + + +def test_edit_appgroup_of_unknown_id(e_uci_appgroups): + with pytest.raises(ValidationError) as err: + dpi.edit_appgroup(e_uci_appgroups, 'ns_nonexistent', 'Group one', ['netify.netflix']) + assert err.value.args[1] == 'appgroup_does_not_exists' + + +def test_edit_appgroup_refuses_a_rule_id(e_uci_appgroups): + with pytest.raises(ValidationError) as err: + dpi.edit_appgroup(e_uci_appgroups, 'rule0', 'Group one', ['netify.netflix']) + assert err.value.args[1] == 'appgroup_does_not_exists' + + +def test_delete_appgroup(e_uci_appgroups): + group_id = dpi.add_appgroup(e_uci_appgroups, 'Group one', ['netify.netflix']) + dpi.delete_appgroup(e_uci_appgroups, group_id) + assert e_uci_appgroups.get('dpi', group_id, default=None) is None + + +def test_add_appgroup_leaves_the_change_pending(e_uci_appgroups, mock_reload): + dpi.add_appgroup(e_uci_appgroups, 'Group one', ['netify.netflix']) + assert has_pending_changes(e_uci_appgroups) + assert not reloaded(mock_reload) -def test_index_applications_search(mock_load): - assert dpi.list_applications(search='l') == { +def test_delete_appgroup_leaves_the_change_pending(e_uci_appgroups, mock_reload): + group_id = dpi.add_appgroup(e_uci_appgroups, 'Group one', ['netify.netflix']) + dpi.delete_appgroup(e_uci_appgroups, group_id) + assert has_pending_changes(e_uci_appgroups) + assert not reloaded(mock_reload) + + +def test_delete_appgroup_of_unknown_id(e_uci_appgroups): + with pytest.raises(ValidationError) as err: + dpi.delete_appgroup(e_uci_appgroups, 'ns_nonexistent') + assert err.value.args[1] == 'appgroup_does_not_exists' + + +def test_delete_appgroup_in_use(e_uci_appgroups): + group_id = dpi.add_appgroup(e_uci_appgroups, 'Group one', ['netify.netflix']) + e_uci_appgroups.set('dpi', 'rule0', 'appgroup', [group_id]) + with pytest.raises(ValidationError) as err: + dpi.delete_appgroup(e_uci_appgroups, group_id) + assert err.value.args[1] == 'appgroup_is_used' + assert err.value.args[2] == ['dpi/rule0'] + + +def test_is_used_appgroup(e_uci_appgroups): + group_id = dpi.add_appgroup(e_uci_appgroups, 'Group one', ['netify.netflix']) + assert dpi.is_used_appgroup(e_uci_appgroups, group_id) == (False, []) + e_uci_appgroups.set('dpi', 'rule0', 'appgroup', [group_id]) + e_uci_appgroups.set('dpi', 'rule1', 'appgroup', [group_id]) + used, matches = dpi.is_used_appgroup(e_uci_appgroups, group_id) + assert used + assert sorted(matches) == ['dpi/rule0', 'dpi/rule1'] + + +def test_list_appgroups(e_uci_appgroups): + group_id = dpi.add_appgroup(e_uci_appgroups, 'Group one', ['netify.netflix'], ['cybersecurity'], + ['Dropbox'], ['web']) + assert dpi.list_appgroups(e_uci_appgroups) == { 'data': [ { - 'id': 133, - 'name': 'netify.netflix', - 'type': 'application', - 'category': { - 'name': 'unknown' - } - }, - { - 'id': 10119, - 'name': 'netify.linkedin', - 'type': 'application', - 'category': { - 'name': 'unknown' - } - }, - { - 'id': 10552, - 'name': 'netify.tesla', - 'type': 'application', - 'category': { - 'name': 'first-category' - } - }, - { - 'id': 10362, - 'name': 'netify.hulu', - 'type': 'application', - 'category': { - 'name': 'first-category' - } - }, - { - 'id': 10118, - 'name': 'netify.lets-encrypt', - 'type': 'application' - }, - { - 'id': 199, - 'name': 'netify.snapchat', - 'type': 'application', - 'category': { - 'name': 'last' - } - }, - { - 'id': 117, - 'name': 'LotusNotes', - 'type': 'protocol' - }, - { - 'id': 128, - 'name': 'NetFlow', - 'type': 'protocol', - 'category': { - 'name': 'base' - } - }, - { - 'id': 129, - 'name': 'SFlow', - 'type': 'protocol', - 'category': { - 'name': 'low' - } - }, - { - 'id': 130, - 'name': 'HTTP/Connect', - 'type': 'protocol', - 'category': { - 'name': 'low' - } + 'id': group_id, + 'name': 'Group one', + 'applications': ['netify.netflix'], + 'application_categories': ['cybersecurity'], + 'protocols': ['Dropbox'], + 'protocol_categories': ['web'], + 'used': False, + 'matches': [] } ], 'meta': { 'last_page': 1, - 'total': 10 + 'total': 1 } } -def test_index_applications_paginate(mock_load): - assert dpi.list_applications(limit=2, page=2) == { - 'data': [ - { - 'id': 10552, - 'name': 'netify.tesla', - 'type': 'application', - 'category': { - 'name': 'first-category' - } - }, - { - 'id': 10195, - 'name': 'netify.avira', - 'type': 'application', - 'category': { - 'name': 'unknown' - } - } - ], - 'meta': { - 'last_page': 8, - 'total': 16 - } - } - assert dpi.list_applications(limit=2, page=3) == { - 'data': [ - { - 'id': 10194, - 'name': 'netify.sophos', - 'type': 'application', - 'category': { - 'name': 'first-category' - } - }, - { - 'id': 10244, - 'name': 'netify.bbc', - 'type': 'application', - 'category': { - 'name': 'unknown' - } - } - ], - 'meta': { - 'last_page': 8, - 'total': 16 - } - } +def test_list_appgroups_when_empty(e_uci_appgroups): + assert dpi.list_appgroups(e_uci_appgroups) == {'data': [], 'meta': {'last_page': 1, 'total': 0}} + + +def test_list_appgroups_is_ordered_by_name(e_uci_appgroups): + dpi.add_appgroup(e_uci_appgroups, 'zeta', ['netify.netflix']) + dpi.add_appgroup(e_uci_appgroups, 'Alpha', ['netify.tesla']) + assert [group['name'] for group in dpi.list_appgroups(e_uci_appgroups)['data']] == ['Alpha', 'zeta'] + + +def test_list_appgroups_pagination(e_uci_appgroups): + for index in range(5): + dpi.add_appgroup(e_uci_appgroups, f'Group {index}', ['netify.netflix']) + result = dpi.list_appgroups(e_uci_appgroups, limit=2, page=2) + assert [group['name'] for group in result['data']] == ['Group 2', 'Group 3'] + assert result['meta'] == {'last_page': 3, 'total': 5} + + +def test_list_appgroups_search(e_uci_appgroups): + dpi.add_appgroup(e_uci_appgroups, 'Streaming', ['netify.netflix']) + dpi.add_appgroup(e_uci_appgroups, 'Business', ['netify.tesla']) + result = dpi.list_appgroups(e_uci_appgroups, search='STREAM') + assert [group['name'] for group in result['data']] == ['Streaming'] + assert result['meta'] == {'last_page': 1, 'total': 1} + + +def test_list_appgroups_reports_usage(e_uci_appgroups): + group_id = dpi.add_appgroup(e_uci_appgroups, 'Group one', ['netify.netflix']) + e_uci_appgroups.set('dpi', 'rule0', 'appgroup', [group_id]) + group = dpi.list_appgroups(e_uci_appgroups)['data'][0] + assert group['used'] + assert group['matches'] == ['dpi/rule0'] + + +def test_expand_appgroups(e_uci_appgroups): + group_id = dpi.add_appgroup(e_uci_appgroups, 'Group one', ['netify.netflix'], ['cybersecurity'], + ['HTTP/Connect'], ['web']) + assert dpi.expand_appgroups(e_uci_appgroups, [group_id]) == ( + "(app == 'netify.netflix' || app_category == 'cybersecurity'" + " || proto == 'http/connect' || proto_category == 'web')" + ) + + +def test_expand_appgroups_merges_and_deduplicates(e_uci_appgroups): + first = dpi.add_appgroup(e_uci_appgroups, 'Group one', ['netify.netflix'], [], ['Dropbox']) + second = dpi.add_appgroup(e_uci_appgroups, 'Group two', ['netify.netflix', 'netify.tesla']) + assert dpi.expand_appgroups(e_uci_appgroups, [first, second]) == ( + "(app == 'netify.netflix' || app == 'netify.tesla' || proto == 'dropbox')" + ) + + +def test_expand_appgroups_skips_unknown_groups(e_uci_appgroups): + group_id = dpi.add_appgroup(e_uci_appgroups, 'Group one', ['netify.netflix']) + assert dpi.expand_appgroups(e_uci_appgroups, [group_id, 'ns_nonexistent', 'rule0']) == ( + "(app == 'netify.netflix')" + ) + + +def test_expand_appgroups_without_groups(e_uci_appgroups): + assert dpi.expand_appgroups(e_uci_appgroups, []) == '' + + +# appgroup catalog + +BUSINESS = {'id': 5, 'tag': 'business', 'label': 'Business'} +PORTAL = {'id': 19, 'tag': 'portal', 'label': 'Portal'} +FILE_SERVER = {'id': 4, 'tag': 'file-server', 'label': 'File Server'} + +application_catalog = [ + {'id': 133, 'tag': 'netify.netflix', 'label': 'Netflix', 'category': PORTAL, 'active': True, + 'icon': 'https://static.netify.ai/logos/n/netflix/icon.png'}, + {'id': 10552, 'tag': 'netify.tesla', 'label': 'Tesla', 'category': BUSINESS}, + # not loaded by the engine: listed, named after the catalog, not selectable + {'id': 90001, 'tag': 'netify.premium-only', 'label': 'Premium Only', 'category': BUSINESS}, + # retired from the product + {'id': 90002, 'tag': 'netify.retired', 'label': 'Retired', 'category': BUSINESS, 'active': False}, + # no category at all + {'id': 10119, 'tag': 'netify.linkedin', 'label': 'LinkedIn'} +] + +protocol_catalog = [ + {'id': 1, 'tag': 'ftp-control', 'label': 'FTP Control', 'category': FILE_SERVER, 'deprecated': ''}, + {'id': 130, 'tag': 'http-connect', 'label': 'HTTP/Connect', 'category': FILE_SERVER}, + {'id': 90003, 'tag': 'gone', 'label': 'Gone', 'category': FILE_SERVER, 'deprecated': '5.0'}, + {'id': 90004, 'tag': 'unknown', 'label': 'Unknown', 'category': FILE_SERVER} +] + + +@pytest.fixture +def mock_catalogs(mocker): + """Both catalogs are on disk and the engine has loaded part of them.""" + mocker.patch('nethsec.dpi.load_applications', return_value={ + 133: 'netify.netflix', 10552: 'netify.tesla', 10119: 'netify.linkedin', + # loaded but absent from the catalog + 99999: 'netify.brand-new' + }) + mocker.patch('nethsec.dpi.load_protocols', return_value={ + 1: 'FTP', 130: 'HTTP/Connect', 99998: 'Unknown' + }) + mocker.patch('nethsec.dpi.__load_catalog', side_effect=lambda filename: ( + application_catalog if 'application' in filename else protocol_catalog + )) + + +def group_by_tag(groups, tag): + return next(group for group in groups if group['tag'] == tag) + + +def test_list_appgroup_catalog_groups_by_category(mock_catalogs): + catalog = dpi.list_appgroup_catalog() + assert [group['tag'] for group in catalog['applications']] == ['business', 'portal', ''] + assert group_by_tag(catalog['applications'], 'portal')['label'] == 'Portal' + + +def test_list_appgroup_catalog_marks_every_real_category_selectable(mock_catalogs): + catalog = dpi.list_appgroup_catalog() + assert group_by_tag(catalog['applications'], 'business')['selectable'] is True + # the uncategorised bucket is not a category: no criteria can match on it + assert group_by_tag(catalog['applications'], '')['selectable'] is False + + +def test_list_appgroup_catalog_item_of_a_loaded_application(mock_catalogs): + items = group_by_tag(dpi.list_appgroup_catalog()['applications'], 'portal')['items'] + assert items == [{'id': 'netify.netflix', 'label': 'Netflix', 'selectable': True, + 'logo': 'https://static.netify.ai/logos/n/netflix/icon.png'}] + + +def test_list_appgroup_catalog_item_the_engine_did_not_load(mock_catalogs): + items = group_by_tag(dpi.list_appgroup_catalog()['applications'], 'business')['items'] + premium = next(item for item in items if item['label'] == 'Premium Only') + # no engine name to store, so the catalog tag stands in and the item cannot be picked + assert premium == {'id': 'netify.premium-only', 'label': 'Premium Only', 'selectable': False} + + +def test_list_appgroup_catalog_hides_an_inactive_application(mock_catalogs): + catalog = dpi.list_appgroup_catalog() + labels = [item['label'] for group in catalog['applications'] for item in group['items']] + assert 'Retired' not in labels + + +def test_list_appgroup_catalog_hides_a_deprecated_protocol(mock_catalogs): + catalog = dpi.list_appgroup_catalog() + labels = [item['label'] for group in catalog['protocols'] for item in group['items']] + assert 'Gone' not in labels + +def test_list_appgroup_catalog_hides_the_unknown_protocol(mock_catalogs): + catalog = dpi.list_appgroup_catalog() + ids = [item['id'] for group in catalog['protocols'] for item in group['items']] + # dropped both from the catalog and from what the engine loaded + assert 'unknown' not in ids and 'Unknown' not in ids -def test_list_empty_rules(e_uci, mock_load): - assert dpi.list_rules(e_uci) == [] +def test_list_appgroup_catalog_keeps_what_the_catalog_does_not_list(mock_catalogs): + items = group_by_tag(dpi.list_appgroup_catalog()['applications'], '')['items'] + assert {'id': 'netify.brand-new', 'label': 'netify.brand-new', 'selectable': True} in items -def test_list_rules(e_uci_with_data, mock_load): - assert dpi.list_rules(e_uci_with_data) == [ + +def test_list_appgroup_catalog_uses_the_engine_name_as_id(mock_catalogs): + # the catalog calls it 'ftp-control', the engine calls it 'FTP': a group stores the engine name + items = group_by_tag(dpi.list_appgroup_catalog()['protocols'], 'file-server')['items'] + assert next(item for item in items if item['label'] == 'FTP Control')['id'] == 'FTP' + + +def test_list_appgroup_catalog_omits_the_logo_when_there_is_none(mock_catalogs): + items = group_by_tag(dpi.list_appgroup_catalog()['applications'], 'business')['items'] + assert 'logo' not in next(item for item in items if item['label'] == 'Tesla') + + +def test_list_appgroup_catalog_sorts_items_by_label(mock_catalogs): + items = group_by_tag(dpi.list_appgroup_catalog()['applications'], 'business')['items'] + assert [item['label'] for item in items] == ['Premium Only', 'Tesla'] + + +def test_list_appgroup_catalog_raises_when_the_engine_is_down(mocker): + mocker.patch('nethsec.dpi.load_applications', side_effect=Exception('netifyd is not running')) + with pytest.raises(Exception): + dpi.list_appgroup_catalog() + + +def test_list_appgroup_catalog_raises_when_a_catalog_is_missing(mocker): + mocker.patch('nethsec.dpi.load_applications', return_value={}) + mocker.patch('nethsec.dpi.load_protocols', return_value={}) + mocker.patch('nethsec.dpi.__load_catalog', side_effect=FileNotFoundError) + with pytest.raises(FileNotFoundError): + dpi.list_appgroup_catalog() + + +# rules + +@pytest.fixture +def e_uci_rules(e_uci, mock_vocabularies): + """A clean dpi config with one application group to build rules on.""" + with pathlib.Path(e_uci.confdir()).joinpath('dpi').open('w') as fp: + fp.write(dpi_minimal_db) + return e_uci + + +def group_of(e_uci, name='Group one'): + return dpi.add_appgroup(e_uci, name, ['netify.netflix']) + + +def test_list_rules_when_empty(e_uci_rules): + assert dpi.list_rules(e_uci_rules) == [] + + +def test_add_rule(e_uci_rules): + group = group_of(e_uci_rules) + rule_id = dpi.add_rule(e_uci_rules, 'Block streaming', True, 'block', ['192.168.1.0/24'], [group]) + assert e_uci_rules.get('dpi', rule_id) == 'rule' + assert e_uci_rules.get('dpi', rule_id, 'ns_name') == 'Block streaming' + assert e_uci_rules.get('dpi', rule_id, 'ns_managed') == '1' + assert e_uci_rules.get('dpi', rule_id, 'enabled') == '1' + assert e_uci_rules.get('dpi', rule_id, 'action') == 'block' + assert e_uci_rules.get('dpi', rule_id, 'priority') == '1' + assert e_uci_rules.get('dpi', rule_id, 'source', list=True) == ('192.168.1.0/24',) + assert e_uci_rules.get('dpi', rule_id, 'appgroup', list=True) == (group,) + + +def test_add_rule_without_source(e_uci_rules): + group = group_of(e_uci_rules) + rule_id = dpi.add_rule(e_uci_rules, 'Block everywhere', True, 'block', [], [group]) + assert e_uci_rules.get('dpi', rule_id, 'source', default=None) is None + + +def test_add_rule_at_the_top(e_uci_rules): + group = group_of(e_uci_rules) + first = dpi.add_rule(e_uci_rules, 'First', True, 'block', [], [group]) + second = dpi.add_rule(e_uci_rules, 'Second', True, 'allow', [], [group], position='top') + assert [rule['id'] for rule in dpi.list_rules(e_uci_rules)] == [second, first] + assert e_uci_rules.get('dpi', second, 'priority') == '1' + assert e_uci_rules.get('dpi', first, 'priority') == '2' + + +def test_add_rule_requires_a_group(e_uci_rules): + with pytest.raises(ValidationError) as err: + dpi.add_rule(e_uci_rules, 'No match', True, 'block', ['192.168.1.1'], []) + assert err.value.args[0] == 'appgroups' + assert err.value.args[1] == 'appgroups_required' + + +def test_add_rule_refuses_an_unknown_group(e_uci_rules): + with pytest.raises(ValidationError) as err: + dpi.add_rule(e_uci_rules, 'Bad group', True, 'block', [], ['ns_nonexistent']) + assert err.value.args[1] == 'appgroup_does_not_exists' + + +def test_add_rule_refuses_an_unknown_action(e_uci_rules): + group = group_of(e_uci_rules) + with pytest.raises(ValidationError) as err: + dpi.add_rule(e_uci_rules, 'Bad action', True, 'bulk', [], [group]) + assert err.value.args[0] == 'action' + assert err.value.args[1] == 'invalid_action' + + +def test_add_rule_requires_a_name(e_uci_rules): + group = group_of(e_uci_rules) + with pytest.raises(ValidationError) as err: + dpi.add_rule(e_uci_rules, ' ', True, 'block', [], [group]) + assert err.value.args[1] == 'name_required' + + +@pytest.mark.parametrize('source', ['192.168.1.1', '192.168.1.0/24', '192.168.1.10-192.168.1.20', + '2001:db8::1', '2001:db8::/64']) +def test_add_rule_accepts_every_source_shape(e_uci_rules, source): + group = group_of(e_uci_rules) + rule_id = dpi.add_rule(e_uci_rules, f'Rule {source}', True, 'block', [source], [group]) + assert e_uci_rules.get('dpi', rule_id, 'source', list=True) == (source,) + + +@pytest.mark.parametrize('source', ['not-an-ip', '192.168.1.300', '192.168.1.20-192.168.1.10', + '192.168.1.1-2001:db8::1', '']) +def test_add_rule_refuses_an_invalid_source(e_uci_rules, source): + group = group_of(e_uci_rules) + with pytest.raises(ValidationError) as err: + dpi.add_rule(e_uci_rules, 'Bad source', True, 'block', [source], [group]) + assert err.value.args[0] == 'source' + assert err.value.args[1] == 'invalid_source' + + +def test_edit_rule(e_uci_rules): + group = group_of(e_uci_rules) + other = group_of(e_uci_rules, 'Group two') + rule_id = dpi.add_rule(e_uci_rules, 'Block streaming', True, 'block', ['192.168.1.1'], [group]) + dpi.edit_rule(e_uci_rules, rule_id, 'Allow streaming', False, 'allow', [], [other]) + assert e_uci_rules.get('dpi', rule_id, 'ns_name') == 'Allow streaming' + assert e_uci_rules.get('dpi', rule_id, 'enabled') == '0' + assert e_uci_rules.get('dpi', rule_id, 'action') == 'allow' + assert e_uci_rules.get('dpi', rule_id, 'appgroup', list=True) == (other,) + # an emptied source is removed, the rule matches every host again + assert e_uci_rules.get('dpi', rule_id, 'source', default=None) is None + + +def test_edit_rule_keeps_the_priority(e_uci_rules): + group = group_of(e_uci_rules) + first = dpi.add_rule(e_uci_rules, 'First', True, 'block', [], [group]) + second = dpi.add_rule(e_uci_rules, 'Second', True, 'block', [], [group]) + dpi.edit_rule(e_uci_rules, first, 'First edited', True, 'block', [], [group]) + assert [rule['id'] for rule in dpi.list_rules(e_uci_rules)] == [first, second] + + +def test_edit_rule_of_unknown_id(e_uci_rules): + group = group_of(e_uci_rules) + with pytest.raises(ValidationError) as err: + dpi.edit_rule(e_uci_rules, 'ns_nonexistent', 'Rule', True, 'block', [], [group]) + assert err.value.args[1] == 'rule_not_found' + + +def test_edit_rule_refuses_an_unmanaged_rule(e_uci_with_data, mock_vocabularies): + group = dpi.add_appgroup(e_uci_with_data, 'Group one', ['netify.netflix']) + with pytest.raises(ValidationError) as err: + dpi.edit_rule(e_uci_with_data, 'rule0', 'Rule', True, 'block', [], [group]) + assert err.value.args[1] == 'rule_not_managed' + + +def test_delete_rule_closes_the_gap(e_uci_rules): + group = group_of(e_uci_rules) + first = dpi.add_rule(e_uci_rules, 'First', True, 'block', [], [group]) + second = dpi.add_rule(e_uci_rules, 'Second', True, 'block', [], [group]) + third = dpi.add_rule(e_uci_rules, 'Third', True, 'block', [], [group]) + dpi.delete_rule(e_uci_rules, second) + assert e_uci_rules.get('dpi', first, 'priority') == '1' + assert e_uci_rules.get('dpi', third, 'priority') == '2' + + +def test_add_rule_leaves_the_change_pending(e_uci_rules, mock_reload): + group = group_of(e_uci_rules) + dpi.add_rule(e_uci_rules, 'Block streaming', True, 'block', [], [group]) + assert has_pending_changes(e_uci_rules) + assert not reloaded(mock_reload) + + +def test_delete_rule_leaves_the_change_pending(e_uci_rules, mock_reload): + group = group_of(e_uci_rules) + rule_id = dpi.add_rule(e_uci_rules, 'Block streaming', True, 'block', [], [group]) + dpi.delete_rule(e_uci_rules, rule_id) + assert has_pending_changes(e_uci_rules) + assert not reloaded(mock_reload) + + +def test_delete_rule_of_unknown_id(e_uci_rules): + with pytest.raises(ValidationError) as err: + dpi.delete_rule(e_uci_rules, 'ns_nonexistent') + assert err.value.args[1] == 'rule_not_found' + + +def test_rename_and_toggle_rule(e_uci_rules): + group = group_of(e_uci_rules) + rule_id = dpi.add_rule(e_uci_rules, 'Block streaming', True, 'block', [], [group]) + dpi.rename_rule(e_uci_rules, rule_id, 'Renamed') + assert e_uci_rules.get('dpi', rule_id, 'ns_name') == 'Renamed' + dpi.disable_rule(e_uci_rules, rule_id) + assert e_uci_rules.get('dpi', rule_id, 'enabled') == '0' + dpi.enable_rule(e_uci_rules, rule_id) + assert e_uci_rules.get('dpi', rule_id, 'enabled') == '1' + + +def test_rename_and_toggle_work_on_unmanaged_rules(e_uci_with_data): + dpi.rename_rule(e_uci_with_data, 'rule0', 'Migrated rule') + assert e_uci_with_data.get('dpi', 'rule0', 'ns_name') == 'Migrated rule' + dpi.disable_rule(e_uci_with_data, 'rule0') + assert e_uci_with_data.get('dpi', 'rule0', 'enabled') == '0' + + +def test_order_rules(e_uci_rules): + group = group_of(e_uci_rules) + first = dpi.add_rule(e_uci_rules, 'First', True, 'block', [], [group]) + second = dpi.add_rule(e_uci_rules, 'Second', True, 'block', [], [group]) + third = dpi.add_rule(e_uci_rules, 'Third', True, 'block', [], [group]) + dpi.order_rules(e_uci_rules, [third, first, second]) + assert [rule['id'] for rule in dpi.list_rules(e_uci_rules)] == [third, first, second] + assert e_uci_rules.get('dpi', third, 'priority') == '1' + + +def test_order_rules_refuses_a_partial_order(e_uci_rules): + group = group_of(e_uci_rules) + first = dpi.add_rule(e_uci_rules, 'First', True, 'block', [], [group]) + dpi.add_rule(e_uci_rules, 'Second', True, 'block', [], [group]) + with pytest.raises(ValidationError) as err: + dpi.order_rules(e_uci_rules, [first]) + assert err.value.args[1] == 'invalid_order' + + +def test_order_rules_refuses_an_unknown_rule(e_uci_rules): + group = group_of(e_uci_rules) + first = dpi.add_rule(e_uci_rules, 'First', True, 'block', [], [group]) + with pytest.raises(ValidationError) as err: + dpi.order_rules(e_uci_rules, [first, 'ns_nonexistent']) + assert err.value.args[1] == 'rule_not_found' + + +def test_list_rules(e_uci_rules): + group = group_of(e_uci_rules) + rule_id = dpi.add_rule(e_uci_rules, 'Block streaming', True, 'block', ['192.168.1.0/24'], [group]) + assert dpi.list_rules(e_uci_rules) == [ { - 'config-name': 'rule0', + 'id': rule_id, + 'name': 'Block streaming', 'enabled': True, - 'interface': 'GREEN_1', - 'device': 'eth0', 'action': 'block', - 'criteria': [ - { - 'id': 10119, - 'name': 'netify.linkedin', - 'type': 'application', - 'category': { - 'name': 'unknown' - } - }, - { - 'id': 199, - 'name': 'netify.snapchat', - 'type': 'application', - 'category': { - 'name': 'last' - } - }, - { - 'id': 130, - 'name': 'HTTP/Connect', - 'type': 'protocol', - 'category': { - 'name': 'low' - } - } - ] - }, - { - 'config-name': 'rule1', - 'enabled': False, - 'interface': 'GREEN_2', - 'device': 'eth4', - 'action': 'block', - 'criteria': [ - { - 'id': 10552, - 'name': 'netify.tesla', - 'type': 'application', - 'category': { - 'name': 'first-category' - } - } - ] - }, - { - 'config-name': 'rule3', - 'enabled': True, - 'interface': 'RED_1', - 'device': 'eth1', - 'action': 'video', - 'criteria': [ - { - 'id': 130, - 'name': 'HTTP/Connect', - 'type': 'protocol', - 'category': { - 'name': 'low' - } - } - ] + 'source': ['192.168.1.0/24'], + 'appgroups': [{'id': group, 'name': 'Group one'}], + 'match_all': False, + 'managed': True, + 'index': 0 } ] -def test_store_rule(e_uci, mock_load): - rule_created = dpi.add_rule(e_uci, True, 'eth0', 'best_effort', - ['netify.linkedin', 'netify.avira', 'netify.netflix'], ['LotusNotes', 'SFlow']) - assert dpi.list_rules(e_uci) == [ - { - 'config-name': rule_created, - 'enabled': True, - 'interface': 'GREEN_1', - 'device': 'eth0', - 'action': 'best_effort', - 'criteria': [ - { - 'id': 10119, - 'name': 'netify.linkedin', - 'type': 'application', - 'category': { - 'name': 'unknown' - } - }, - { - 'id': 10195, - 'name': 'netify.avira', - 'type': 'application', - 'category': { - 'name': 'unknown' - } - }, - { - 'id': 133, - 'name': 'netify.netflix', - 'type': 'application', - 'category': { - 'name': 'unknown' - } - }, - { - 'id': 117, - 'name': 'LotusNotes', - 'type': 'protocol', - }, - { - 'id': 129, - 'name': 'SFlow', - 'type': 'protocol', - 'category': { - 'name': 'low' - } - } - ] - } - ] - assert(e_uci.get("dpi", "config", "enabled") == "1") +def test_list_rules_reports_unmanaged_rules(e_uci_with_data): + rules = {rule['id']: rule for rule in dpi.list_rules(e_uci_with_data)} + assert rules['rule0']['managed'] is False + assert rules['rule2']['criteria'] == 'local_ip == 192.168.100.22 && application == "netify.facebook";' + assert 'criteria' not in rules['rule3'] -def test_delete_rule(e_uci_with_data, mock_load): - dpi.delete_rule(e_uci_with_data, 'rule1') - dpi.delete_rule(e_uci_with_data, 'rule0') - assert dpi.list_rules(e_uci_with_data) == [ - { - 'config-name': 'rule3', - 'enabled': True, - 'interface': 'RED_1', - 'device': 'eth1', - 'action': 'video', - 'criteria': [ - { - 'id': 130, - 'name': 'HTTP/Connect', - 'type': 'protocol', - 'category': { - 'name': 'low' - } - } - ] - } - ] - dpi.delete_rule(e_uci_with_data, 'rule2') - dpi.delete_rule(e_uci_with_data, 'rule3') - assert(e_uci_with_data.get("dpi", "config", "enabled") == "0") +def test_list_rules_hides_invisible_ones(e_uci_rules): + group = group_of(e_uci_rules) + rule_id = dpi.add_rule(e_uci_rules, 'Hidden', True, 'block', [], [group]) + e_uci_rules.set('dpi', rule_id, 'ns_visible', '0') + assert dpi.list_rules(e_uci_rules) == [] -def test_edit_rule(e_uci_with_data, mock_load): - dpi.edit_rule(e_uci_with_data, 'rule0', False, 'eth0', 'voice', [], - ['HTTP/Connect', 'LotusNotes']) - assert dpi.list_rules(e_uci_with_data) == [ - { - 'config-name': 'rule0', - 'enabled': False, - 'interface': 'GREEN_1', - 'device': 'eth0', - 'action': 'voice', - 'criteria': [ - { - 'id': 130, - 'name': 'HTTP/Connect', - 'type': 'protocol', - 'category': { - 'name': 'low' - } - }, - { - 'id': 117, - 'name': 'LotusNotes', - 'type': 'protocol', - } - ] - }, - { - 'config-name': 'rule1', - 'enabled': False, - 'interface': 'GREEN_2', - 'device': 'eth4', - 'action': 'block', - 'criteria': [ - { - 'id': 10552, - 'name': 'netify.tesla', - 'type': 'application', - 'category': { - 'name': 'first-category' - } - } - ] - }, - { - 'config-name': 'rule3', - 'enabled': True, - 'interface': 'RED_1', - 'device': 'eth1', - 'action': 'video', - 'criteria': [ - { - 'id': 130, - 'name': 'HTTP/Connect', - 'type': 'protocol', - 'category': { - 'name': 'low' - } - } - ] - } - ] +def test_list_rules_is_complete_and_indexed(e_uci_rules): + group = group_of(e_uci_rules) + for index in range(5): + dpi.add_rule(e_uci_rules, f'Rule {index}', True, 'block', [], [group]) + rules = dpi.list_rules(e_uci_rules) + assert [rule['name'] for rule in rules] == ['Rule 0', 'Rule 1', 'Rule 2', 'Rule 3', 'Rule 4'] + assert [rule['index'] for rule in rules] == [0, 1, 2, 3, 4] + + +def test_expand_source(): + assert dpi.expand_source(['192.168.1.1', '192.168.1.0/24']) == ['192.168.1.1', '192.168.1.0/24'] + assert dpi.expand_source(['192.168.1.10-192.168.1.11']) == ['192.168.1.10/31'] + assert dpi.expand_source(['192.168.1.1-192.168.1.4']) == ['192.168.1.1/32', '192.168.1.2/31', + '192.168.1.4/32'] + assert dpi.expand_source(['10.0.0.1-10.0.0.2']) == ['10.0.0.1/32', '10.0.0.2/32'] + + +def test_build_rule_criteria(e_uci_rules): + group = group_of(e_uci_rules) + rule_id = dpi.add_rule(e_uci_rules, 'Block streaming', True, 'block', + ['192.168.1.0/24', '10.0.0.1-10.0.0.2'], [group]) + rule = e_uci_rules.get_all('dpi', rule_id) + assert dpi.build_rule_criteria(e_uci_rules, rule) == ( + "(local_ip == 192.168.1.0/24 || local_ip == 10.0.0.1/32 || local_ip == 10.0.0.2/32)" + " && (app == 'netify.netflix');" + ) + + +def test_build_rule_criteria_without_source(e_uci_rules): + group = group_of(e_uci_rules) + rule_id = dpi.add_rule(e_uci_rules, 'Block streaming', True, 'block', [], [group]) + rule = e_uci_rules.get_all('dpi', rule_id) + assert dpi.build_rule_criteria(e_uci_rules, rule) == "(app == 'netify.netflix');" -def test_edit_rule_with_missing_rule(e_uci): +def test_build_rule_criteria_of_a_rule_matching_nothing(e_uci_rules): + assert dpi.build_rule_criteria(e_uci_rules, {}) == '' + + +def test_build_rule_criteria_of_a_match_all_rule(e_uci_rules): + # the wildcard stands alone and carries no terminator + assert dpi.build_rule_criteria(e_uci_rules, {'ns_match_all': '1'}) == '*' + + +def test_build_rule_criteria_of_a_match_all_rule_narrowed_to_a_source(e_uci_rules): + # the wildcard cannot be ANDed with anything: a source expresses the same thing on its own + rule = {'ns_match_all': '1', 'source': ['192.168.1.1', '192.168.1.0/24']} + assert dpi.build_rule_criteria(e_uci_rules, rule) == ( + '(local_ip == 192.168.1.1 || local_ip == 192.168.1.0/24);' + ) + + +def test_build_rule_criteria_source_only_without_the_flag(e_uci_rules): + # a rule migrated before ns_match_all existed carries the same shape and must keep working + rule = {'source': ['192.168.1.1']} + assert dpi.build_rule_criteria(e_uci_rules, rule) == '(local_ip == 192.168.1.1);' + + +def test_add_rule_without_appgroup_requires_match_all(e_uci_rules): with pytest.raises(ValidationError) as err: - dpi.edit_rule(e_uci, 'rule0', False, 'eth0', 'block', [], []) - - assert err.value.args[0] == 'config-name' - assert err.value.args[1] == 'invalid' - assert err.value.args[2] == 'rule0' - - -def test_list_interfaces(e_uci_with_data): - assert dpi.list_devices(e_uci_with_data).index({ - 'interface': 'GREEN_1', - 'device': 'eth0' - }) != -1 - assert dpi.list_devices(e_uci_with_data).index({ - 'interface': 'GREEN_2', - 'device': 'eth4' - }) != -1 - with pytest.raises(ValueError): - assert dpi.list_devices(e_uci_with_data).index({ - 'interface': 'RED_1', - 'device': 'eth1' - }) - - -def test_list_popular(e_uci_with_data, mock_load): - assert dpi.list_popular(e_uci_with_data) == { - 'data': [ - { - 'id': 133, - 'name': 'netify.netflix', - 'type': 'application', - 'category': { - 'name': 'unknown' - }, - 'missing': False - }, - { - 'id': 10362, - 'name': 'netify.hulu', - 'type': 'application', - 'category': { - 'name': 'first-category' - }, - 'missing': False - }, - { - 'name': 'netify.whatsapp', - 'missing': True - }, - { - 'name': 'netify.facebook', - 'missing': True - }, - { - 'id': 10194, - 'name': 'netify.sophos', - 'type': 'application', - 'category': { - 'name': 'first-category' - }, - 'missing': False - }, - { - 'id': 130, - 'name': 'HTTP/Connect', - 'type': 'protocol', - 'category': { - 'name': 'low' - }, - 'missing': False - }, - { - 'id': 121, - 'name': 'Dropbox', - 'type': 'protocol', - 'category': { - 'name': 'base' - }, - 'missing': False - } - ], - 'meta': { - 'last_page': 1, - 'total': 7 - } - } + dpi.add_rule(e_uci_rules, 'No group', True, 'allow', ['192.168.1.1'], []) + assert err.value.args[1] == 'appgroups_required' -def test_list_popular_with_limits(e_uci_with_data, mock_load): - assert dpi.list_popular(e_uci_with_data, limit=2, page=2) == { - 'data': [ - { - 'name': 'netify.whatsapp', - 'missing': True - }, - { - 'name': 'netify.facebook', - 'missing': True - } - ], - 'meta': { - 'last_page': 4, - 'total': 7 - } - } - assert dpi.list_popular(e_uci_with_data, limit=2, page=3) == { - 'data': [ - { - 'id': 10194, - 'name': 'netify.sophos', - 'type': 'application', - 'category': { - 'name': 'first-category' - }, - 'missing': False - }, - { - 'id': 130, - 'name': 'HTTP/Connect', - 'type': 'protocol', - 'category': { - 'name': 'low' - }, - 'missing': False - } - ], - 'meta': { - 'last_page': 4, - 'total': 7 - } - } +def test_add_match_all_rule(e_uci_rules): + rule_id = dpi.add_rule(e_uci_rules, 'Allow everything', True, 'allow', [], [], match_all=True) + assert e_uci_rules.get('dpi', rule_id, 'ns_managed') == '1' + assert e_uci_rules.get('dpi', rule_id, 'ns_match_all') == '1' + assert e_uci_rules.get('dpi', rule_id, 'appgroup', default=None) is None + assert e_uci_rules.get('dpi', rule_id, 'source', default=None) is None + assert dpi.build_rule_criteria(e_uci_rules, e_uci_rules.get_all('dpi', rule_id)) == '*' -def test_list_exemptions(e_uci_with_data): - assert dpi.list_exemptions(e_uci_with_data) == [ - { - 'config-name': 'exemp1', - 'enabled': True, - 'criteria': '192.168.1.1', - 'description': 'my host', - } - ] -def test_add_exemption(e_uci): - ex_created = dpi.add_exemption(e_uci, "192.168.2.2", 'my host2', True) - assert dpi.list_exemptions(e_uci) == [ - { - 'config-name': ex_created, - 'enabled': True, - 'criteria': "192.168.2.2", - 'description': 'my host2', - } - ] +def test_add_match_all_rule_narrowed_to_a_source(e_uci_rules): + rule_id = dpi.add_rule(e_uci_rules, 'Allow the office', True, 'allow', ['192.168.1.1'], [], + match_all=True) + assert e_uci_rules.get('dpi', rule_id, 'ns_match_all') == '1' + assert e_uci_rules.get('dpi', rule_id, 'source', list=True) == ('192.168.1.1',) + assert e_uci_rules.get('dpi', rule_id, 'appgroup', default=None) is None + + +def test_add_match_all_rule_refuses_an_appgroup(e_uci_rules): + group = group_of(e_uci_rules) with pytest.raises(ValidationError) as err: - dpi.add_exemption(e_uci, "192.168.2.2", 'duplicated', True) + dpi.add_rule(e_uci_rules, 'Contradiction', True, 'allow', [], [group], match_all=True) + assert err.value.args[1] == 'appgroups_not_allowed_with_match_all' + + +def test_list_rules_reports_match_all(e_uci_rules): + group = group_of(e_uci_rules) + dpi.add_rule(e_uci_rules, 'Allow everything', True, 'allow', [], [], match_all=True) + dpi.add_rule(e_uci_rules, 'Block streaming', True, 'block', [], [group]) + assert [rule['match_all'] for rule in dpi.list_rules(e_uci_rules)] == [True, False] + + +def test_edit_rule_turns_match_all_on(e_uci_rules): + group = group_of(e_uci_rules) + rule_id = dpi.add_rule(e_uci_rules, 'Block streaming', True, 'block', [], [group]) + dpi.edit_rule(e_uci_rules, rule_id, 'Allow everything', True, 'allow', [], [], match_all=True) + assert e_uci_rules.get('dpi', rule_id, 'ns_match_all') == '1' + assert e_uci_rules.get('dpi', rule_id, 'appgroup', default=None) is None + + +def test_edit_rule_turns_match_all_off(e_uci_rules): + group = group_of(e_uci_rules) + rule_id = dpi.add_rule(e_uci_rules, 'Allow everything', True, 'allow', [], [], match_all=True) + dpi.edit_rule(e_uci_rules, rule_id, 'Block streaming', True, 'block', [], [group]) + # the flag is removed, not stored as '0' + assert e_uci_rules.get('dpi', rule_id, 'ns_match_all', default=None) is None + assert e_uci_rules.get('dpi', rule_id, 'appgroup', list=True) == (group,) + + +# Migration from the schema used before application groups existed + +def test_freeze_legacy_criteria_from_application_and_device(e_uci: EUci): + rule = {'device': 'eth0', 'application': ['netify.amazon-prime', 'netify.netflix']} + assert dpi.freeze_legacy_criteria(e_uci, rule) == ( + "(iface_nfq_src == 'eth0' or iface_nfq_dst == 'eth0') && " + "(app == 'netify.amazon-prime' or app == 'netify.netflix') ;" + ) + + +def test_freeze_legacy_criteria_prefers_hand_written_criteria(e_uci: EUci): + rule = {'device': 'eth0', 'criteria': 'app == "netify.netflix";', 'application': ['netify.amazon-prime']} + assert dpi.freeze_legacy_criteria(e_uci, rule) == "app == 'netify.netflix';" + + +def test_freeze_legacy_criteria_applies_vlan_rewrite(e_uci: EUci): + with pathlib.Path(e_uci.confdir()).joinpath('network').open('a') as fp: + fp.write(""" +config device + option name 'eth0.10' + option ifname 'eth0' + option type '8021q' + option vid '10' +""") + rule = {'device': 'eth0.10', 'application': ['netify.netflix']} + assert dpi.freeze_legacy_criteria(e_uci, rule) == ( + "vlan_id == 10 && (iface_nfq_src == 'eth0.10' or iface_nfq_dst == 'eth0.10') && " + "(app == 'netify.netflix') ;" + ) + + +legacy_dpi_db = """ +config main 'config' + option log_blocked '0' + list popular_filters 'netify.facebook' + +config rule 'ns_b01a0e73' + option enabled '1' + option device 'eth0' + option action 'block' + list application 'netify.amazon-prime' + list application 'netify.netflix' + +config rule 'ns_qos1' + option enabled '1' + option device 'eth1' + option action 'bulk' + list application 'netify.dropbox' + +config rule 'ns_empty1' + option enabled '1' + option device 'eth2' + option action 'block' + +config rule 'ns_perrule_exempt' + option enabled '1' + option device 'eth3' + option action 'block' + list application 'netify.tesla' + list exemption '192.168.100.3' + +config exemption 'ns_2127b876' + option enabled '1' + option criteria '192.168.122.47' + option description 'my exception' + +config exemption 'ns_disabled_exempt' + option enabled '0' + option criteria '10.0.0.5' + option description 'disabled one' + +config exemption 'ns_bad_exempt' + option enabled '1' + option criteria 'not-an-address' + option description '' +""" + + +@pytest.fixture +def e_uci_legacy(e_uci: EUci): + with pathlib.Path(e_uci.confdir()).joinpath('dpi').open('w') as fp: + fp.write(legacy_dpi_db) + return e_uci + + +def test_migrate_schema_noop_when_nothing_to_migrate(e_uci, mock_reload): + config_before = dpi_config_file(e_uci) + dpi.migrate_schema(e_uci) + assert dpi_config_file(e_uci) == config_before + assert not has_pending_changes(e_uci) + assert not reloaded(mock_reload) + + +def test_migrate_schema_freezes_legacy_rule(e_uci_legacy): + dpi.migrate_schema(e_uci_legacy) + rule = e_uci_legacy.get_all('dpi', 'ns_b01a0e73') + assert rule['criteria'] == ( + "(iface_nfq_src == 'eth0' or iface_nfq_dst == 'eth0') && " + "(app == 'netify.amazon-prime' or app == 'netify.netflix') ;" + ) + assert 'device' not in rule + assert 'application' not in rule + assert 'ns_managed' not in rule + assert rule['action'] == 'block' + assert 'priority' in rule + + +def test_migrate_schema_names_migrated_rules_and_exemptions_in_order(e_uci_legacy): + dpi.migrate_schema(e_uci_legacy) + assert e_uci_legacy.get('dpi', 'ns_b01a0e73', 'ns_name') == 'Migrated rule 1' + # ns_qos1 is dropped (QoS), ns_empty1 is dropped (matches nothing): ns_perrule_exempt is next + perrule_ids = [s for s, r in utils.get_all_by_type(e_uci_legacy, 'dpi', 'rule').items() + if r.get('ns_name') == 'Migrated rule 2'] + assert len(perrule_ids) == 1 + + +def test_migrate_schema_drops_qos_rule(e_uci_legacy): + dpi.migrate_schema(e_uci_legacy) + assert e_uci_legacy.get('dpi', 'ns_qos1', default=None) is None + + +def test_migrate_schema_drops_rule_matching_nothing(e_uci_legacy): + dpi.migrate_schema(e_uci_legacy) + assert e_uci_legacy.get('dpi', 'ns_empty1', default=None) is None + + +def test_migrate_schema_drops_per_rule_exemption_but_keeps_the_rule(e_uci_legacy): + dpi.migrate_schema(e_uci_legacy) + rule = e_uci_legacy.get_all('dpi', 'ns_perrule_exempt') + assert 'exemption' not in rule + assert rule['action'] == 'block' + assert 'criteria' in rule + + +def test_migrate_schema_drops_per_rule_log_but_keeps_the_rule(e_uci, mocker): + syslog = mocker.patch('nethsec.dpi.syslog.syslog') + with pathlib.Path(e_uci.confdir()).joinpath('dpi').open('w') as fp: + fp.write(""" +config main 'config' + +config rule 'ns_logged' + option enabled '1' + option device 'eth0' + option action 'block' + option log '1' + list application 'netify.instagram' +""") + dpi.migrate_schema(e_uci) + rule = e_uci.get_all('dpi', 'ns_logged') + assert 'log' not in rule + assert rule['action'] == 'block' + assert 'criteria' in rule + assert any('per-rule log on ns_logged' in call.args[1] for call in syslog.call_args_list) + + +def test_migrate_schema_converts_address_exemption_to_managed_allow_rule(e_uci_legacy): + dpi.migrate_schema(e_uci_legacy) + rules = dpi.list_rules(e_uci_legacy) + exception1 = next(r for r in rules if r['name'] == 'my exception') + assert exception1['managed'] is True + assert exception1['action'] == 'allow' + assert exception1['source'] == ['192.168.122.47'] + assert exception1['appgroups'] == [] + assert exception1['match_all'] is True + assert exception1['enabled'] is True + + +def test_migrate_schema_keeps_a_disabled_exemption_disabled(e_uci_legacy): + dpi.migrate_schema(e_uci_legacy) + rules = dpi.list_rules(e_uci_legacy) + exception2 = next(r for r in rules if r['name'] == 'disabled one') + assert exception2['enabled'] is False + + +def test_migrate_schema_converts_non_address_exemption_to_unmanaged_allow_rule(e_uci_legacy): + dpi.migrate_schema(e_uci_legacy) + rules = dpi.list_rules(e_uci_legacy) + # no description: the name falls back to its position + exception3 = next(r for r in rules if r['name'] == 'Migrated exception 3') + assert exception3['managed'] is False + assert exception3['action'] == 'allow' + assert exception3.get('criteria') == 'not-an-address;' + + +def test_migrate_schema_converts_object_exemption_via_expanded_addresses(e_uci, mocker): + with pathlib.Path(e_uci.confdir()).joinpath('dpi').open('w') as fp: + fp.write(""" +config main 'config' + +config exemption 'ns_objex' + option enabled '1' + option criteria 'objects/ns_obj1' + option description 'office subnet' +""") + mocker.patch('nethsec.dpi.objects.is_object_id', return_value=True) + mocker.patch('nethsec.dpi.objects.get_object_ips', return_value=['192.168.50.0/24']) + dpi.migrate_schema(e_uci) + rules = dpi.list_rules(e_uci) + assert len(rules) == 1 + assert rules[0]['managed'] is True + assert rules[0]['source'] == ['192.168.50.0/24'] + + +def test_migrate_schema_cuts_a_long_exemption_description_to_the_rule_name_limit(e_uci): + description = 'x' * (dpi.DPI_RULE_NAME_MAX_LENGTH + 10) + with pathlib.Path(e_uci.confdir()).joinpath('dpi').open('w') as fp: + fp.write(f""" +config main 'config' + +config exemption 'ns_long' + option enabled '1' + option criteria '192.168.1.1' + option description '{description}' +""") + dpi.migrate_schema(e_uci) + rules = dpi.list_rules(e_uci) + assert rules[0]['name'] == description[:dpi.DPI_RULE_NAME_MAX_LENGTH] + assert rules[0]['managed'] is True + + +def test_migrate_schema_removes_exemption_sections_and_legacy_globals(e_uci_legacy): + dpi.migrate_schema(e_uci_legacy) + assert utils.get_all_by_type(e_uci_legacy, 'dpi', 'exemption') == {} + assert e_uci_legacy.get('dpi', 'config', 'popular_filters', list=True, default=None) is None + + +def test_migrate_schema_puts_exemptions_before_migrated_rules(e_uci_legacy): + dpi.migrate_schema(e_uci_legacy) + rules = dpi.list_rules(e_uci_legacy) + names_in_order = [r['name'] for r in rules] + assert names_in_order.index('my exception') < names_in_order.index('Migrated rule 1') + + +def test_migrate_schema_applies_immediately(e_uci_legacy, mock_reload): + dpi.migrate_schema(e_uci_legacy) + assert not has_pending_changes(e_uci_legacy) + assert reloaded(mock_reload) -def test_edit_exemption(e_uci_with_data): - dpi.edit_exemption(e_uci_with_data, 'exemp1', '192.168.1.3', 'my host 3', False) - assert dpi.list_exemptions(e_uci_with_data) == [ - { - 'config-name': 'exemp1', - 'enabled': False, - 'criteria': "192.168.1.3", - 'description': 'my host 3', - } - ] -def test_delete_exemption(e_uci): - dpi.delete_exemption(e_uci, 'exemp1') - assert dpi.list_exemptions(e_uci) == [] +def test_migrate_schema_is_idempotent(e_uci_legacy, mock_reload): + dpi.migrate_schema(e_uci_legacy) + config_after_first_run = dpi_config_file(e_uci_legacy) + mock_reload.reset_mock() + dpi.migrate_schema(e_uci_legacy) + assert dpi_config_file(e_uci_legacy) == config_after_first_run + assert not has_pending_changes(e_uci_legacy) + assert not reloaded(mock_reload) diff --git a/packages/python3-nethsec/tests/test_inventory.py b/packages/python3-nethsec/tests/test_inventory.py index beca36850..4654c5b17 100644 --- a/packages/python3-nethsec/tests/test_inventory.py +++ b/packages/python3-nethsec/tests/test_inventory.py @@ -561,14 +561,41 @@ dpi_db = """ config main 'config' option log_blocked '0' - option enabled '1' - option firewall_exemption '0' + +config appgroup 'ns_1a2b3c4d' + option ns_name 'Streaming and games' + list app 'netify.netflix' + list app 'netify.youtube' + list app_category 'games' + +config appgroup 'ns_5e6f7a8b' + option ns_name 'Unused' + list proto 'HTTP/Connect' + list proto_category 'vpn' + +config rule 'ns_3869dc35' + option ns_name 'Allow the office' + option ns_managed '1' + option enabled '0' + option action 'allow' + option priority '1' + option ns_match_all '1' + list source '192.168.1.10' config rule 'ns_2b170d05' + option ns_name 'Block streaming and games' + option ns_managed '1' option enabled '1' - option device 'br-lan' option action 'block' - list application 'netify.facebook' + option priority '2' + list appgroup 'ns_1a2b3c4d' + +config rule 'ns_9d40be71' + option ns_name 'Custom' + option enabled '1' + option action 'block' + option priority '3' + option criteria 'local_ip == 192.168.1.20;' """ dhcp_db = """ @@ -927,7 +954,15 @@ def test_fact_ipsec(tmp_path): def test_fact_dpi(tmp_path): u = _setup_db(tmp_path) - assert inventory.fact_dpi(u) == {"enabled": True, "rules": 1} + assert inventory.fact_dpi(u) == { + "total": 3, + "enabled": 2, + "action": {"block": 2, "allow": 1}, + "source": {"all": 1, "ip": 1}, + "match": {"all": 1, "appgroup": 1}, + "kind": {"managed": 2, "unmanaged": 1}, + "appgroups": {"total": 2, "used": 1, "app": 2, "app_category": 1, "proto": 1, "proto_category": 1}, + } def test_fact_dhcp_server(tmp_path): u = _setup_db(tmp_path)