diff --git a/.github/ISSUE_TEMPLATE/01_bug_report.md b/.github/ISSUE_TEMPLATE/01_bug_report.md new file mode 100644 index 00000000..8495c6cb --- /dev/null +++ b/.github/ISSUE_TEMPLATE/01_bug_report.md @@ -0,0 +1,21 @@ +--- +name: 🐜 Bug report +about: If something isn't working 🔧 +--- + +### Subject of the issue +Describe your issue here. + +### Your environment +* Version of detectmate +* Version of python +* Docker or manual installation? + +### Steps to reproduce +Tell us how to reproduce this issue. + +### Expected behaviour +Tell us what should happen + +### Actual behaviour +Tell us what happens instead diff --git a/.github/ISSUE_TEMPLATE/02_feature_request.md b/.github/ISSUE_TEMPLATE/02_feature_request.md new file mode 100644 index 00000000..442b05cd --- /dev/null +++ b/.github/ISSUE_TEMPLATE/02_feature_request.md @@ -0,0 +1,20 @@ +--- +name: 🚀 Feature request +about: If you have a feature request 💡 +--- + +**Context** + +What are you trying to do and how would you want to do it differently? Is it something you currently you cannot do? Is this related to an issue/problem? + +**Alternatives** + +Can you achieve the same result doing it in an alternative way? Is the alternative considerable? + +**Has the feature been requested before?** + +Please provide a link to the issue. + +**If the feature request is approved, would you be willing to submit a PR?** + +Yes / No _(Help can be provided if you need assistance submitting a PR)_ diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 00000000..3ba13e0c --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1 @@ +blank_issues_enabled: false diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 00000000..53dec300 --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,19 @@ +# Task + + +# Description + + + + +# How Has This Been Tested? + + +# Checklist + + +- [ ] This Pull-Request goes to the **development** branch. +- [ ] I have successfully run prek locally. +- [ ] I have added tests to cover my changes. +- [ ] I have linked the issue-id to the task-description. +- [ ] I have performed a self-review of my own code. diff --git a/.github/workflows/python-publish.yml b/.github/workflows/python-publish.yml new file mode 100644 index 00000000..c4f6d829 --- /dev/null +++ b/.github/workflows/python-publish.yml @@ -0,0 +1,26 @@ +name: Upload Python Package + +on: + release: + types: [published] + +jobs: + deploy: + + runs-on: ubuntu-latest + permissions: + id-token: write + steps: + - uses: actions/checkout@v3 + - name: Set up Python + uses: actions/setup-python@v3 + with: + python-version: '3.x' + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install build + - name: Build package + run: python -m build + - name: Publish package distributions to PyPI + uses: pypa/gh-action-pypi-publish@release/v1 diff --git a/.gitignore b/.gitignore index 3113ce1a..f9e5cbf7 100644 --- a/.gitignore +++ b/.gitignore @@ -7,8 +7,6 @@ __pycache__/ # C extensions *.so -dummy* - # Distribution / packaging .Python build/ @@ -199,3 +197,9 @@ cython_debug/ local/ test.ipynb test.py + +# claude code +CLAUDE.md +docs/superpowers/ +docs/design/ +.claude/ diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index b651b0ed..50901a6f 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -26,7 +26,7 @@ repos: hooks: - id: mypy args: [--strict, --ignore-missing-imports] - exclude: ^(docs/|tests/) + exclude: ^(docs/|detectmatelibrary_tests/) additional_dependencies: - pydantic - types-PyYAML @@ -58,7 +58,7 @@ repos: rev: 1.8.6 hooks: - id: bandit - exclude: (^tests/|.*/test_.*|^test_.*) + exclude: (^detectmatelibrary_tests/|.*/test_.*|^test_.*) # Unused code detection - repo: https://github.com/jendrikseipp/vulture diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..b13c3dcb --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,199 @@ +# AGENTS.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +DetectMateLibrary is a Python library for log processing and anomaly detection. It provides composable, stream-friendly components (parsers and detectors) that communicate via Protobuf-based schemas. The library is designed for both single-process and microservice deployments. + +## Development Commands + +```bash +# Install dependencies and pre-commit hooks +uv sync --dev +uv run prek install + +# Run tests +uv run pytest -q +uv run pytest -s # verbose with stdout +uv run pytest --cov=. --cov-report=term-missing # with coverage +uv run pytest detectmatelibrary_tests/test_foo.py # single test file + +# Run linting/formatting (all pre-commit hooks) +uv run prek run -a + +# Recompile Protobuf (only if schemas.proto is modified) +protoc --proto_path=src/detectmatelibrary/schemas/ \ + --python_out=src/detectmatelibrary/schemas/ \ + src/detectmatelibrary/schemas/schemas.proto + +# Scaffold a new component workspace +mate create --type --name --dir +``` + +## Architecture + +### Data Flow + +``` +Raw Logs → Parser → ParserSchema → Detector → DetectorSchema (Alerts) +``` + +All data flows through typed Protobuf-backed schema objects. Components are stateful and support an optional training phase before detection. + +### Core Abstractions (`src/detectmatelibrary/common/`) + +- **`CoreComponent`** — base class managing buffering, ID generation, and training state + - **`CoreParser(CoreComponent)`** — parse raw logs into `ParserSchema` + - **`CoreDetector(CoreComponent)`** — detect anomalies in `ParserSchema`, emit `DetectorSchema` +- **`CoreConfig`** / **`CoreParserConfig`** / **`CoreDetectorConfig`** — Pydantic-based configuration hierarchy (`extra="forbid"`) + +#### Component Lifecycle (FitLogic) + +Each `process()` call passes through a state machine controlled by config fields: + +| Config field | Meaning | +|---|---| +| `data_use_configure` | How many items to run through `configure()` before training. `None` = skip. | +| `data_use_training` | How many items to run through `train()` before detection. `None` = skip. | + +Phases in order: **CONFIGURE → TRAIN → DETECT** (phases are skipped when the corresponding field is `None`). + +State control enums allow overriding automatic transitions: `TrainState.KEEP_TRAINING` / `STOP_TRAINING` and `ConfigState.KEEP_CONFIGURE` / `STOP_CONFIGURE` on the component's `fit_logic`. + +#### CoreDetectorConfig: EventsConfig + +Detectors use a nested `events` structure to select which variables to track per event ID, plus `global_instances` for event-ID-independent variables (e.g., hostname, level): + +```yaml +detectors: + MyDetector: + method_type: new_value_detector + auto_config: false # true = auto-discover variables from training data + persist: # optional — omit to disable state saving + path: ./state # base path; detector name is appended automatically + interval_seconds: 300 # save every N seconds + events_until_save: null # also save after N ingested events (null = disabled) + auto_load: false # restore saved state on construction + storage_options: {} # fsspec credentials (S3, Azure, GCS, etc.) + events: + login_failure: # named event ID (string) or integer EventID + instance_label: # arbitrary instance name + params: {} + variables: + - pos: pid # named wildcard label OR integer position in ParserSchema.variables[] + name: pid + header_variables: + - pos: Type # key in ParserSchema.logFormatVariables{} + params: {} + global: # event-ID-independent instances (GLOBAL_EVENT_ID = "*") + global_monitor: + header_variables: + - pos: Level + params: {} +``` + +Named event IDs (strings) and named variable positions require templates loaded from a CSV with an `EventId` column. Call `TemplateMatcher.compile_detector_config(config)` to resolve names to integers at setup time. + +Use `generate_detector_config()` (`src/detectmatelibrary/common/_config/_compile.py`) to build this programmatically: + +```python +from detectmatelibrary.common._config import generate_detector_config + +config = generate_detector_config( + variable_selection={1: ["var_0", "var_1"]}, + detector_name="MyDetector", + method_type="new_value_detector", +) +``` + +Load/save configs via `BasicConfig.from_dict(d, method_id=...)` and `.to_dict(method_id=...)` for YAML round-trip compatibility. + +After training completes, detectors with `auto_config=False` automatically call `validate_config_coverage()` (`src/detectmatelibrary/common/detector.py`), which logs warnings when configured EventIDs or variable positions were never observed in training data. This catches config/data mismatches early — check logs after the training phase when adding new detector configs. + +### Schema System (`src/detectmatelibrary/schemas/`) + +- `BaseSchema` wraps generated Protobuf messages with dict-like access (`schema["field"]`) +- Key schemas: `LogSchema`, `ParserSchema`, `DetectorSchema` +- Support serialization to/from bytes for transport and persistence + +### Buffering Modes (`src/detectmatelibrary/utils/data_buffer.py`) + +Three modes via `ArgsBuffer` config: +- **NO_BUF** — one item at a time (default) +- **BATCH** — accumulate N items, process as batch +- **WINDOW** — sliding window of size N + +### Implementations + +- **Parsers** (`src/detectmatelibrary/parsers/`): `JsonParser`, `LogBatcherParser`, `DummyParser`, `MatcherParser` (Drain3 template mining; supports named wildcards `` alongside positional `<*>`) +- **Detectors** (`src/detectmatelibrary/detectors/`): `NewValueDetector`, `NewValueComboDetector`, `RandomDetector`, `DummyDetector` +- **Utilities** (`src/detectmatelibrary/utils/`): `DataBuffer`, `EventPersistency`, `KeyExtractor`, `TimeFormatHandler`, `IdGenerator` +- Uses the `regex` package (not stdlib `re`) — relevant when writing type annotations or imports involving patterns + +## Extending the Library + +Implement a custom detector by subclassing `CoreDetector`: + +```python +class MyDetectorConfig(CoreDetectorConfig): + method_type: str = "my_detector" + my_param: int = 10 + +class MyDetector(CoreDetector): + def __init__(self, name="MyDetector", config=MyDetectorConfig()): + super().__init__(name=name, config=config) + + def train(self, input_: ParserSchema) -> None: + pass # optional + + def detect(self, input_: ParserSchema, output_: DetectorSchema) -> bool: + output_["detectorID"] = self.name + output_["score"] = 0.0 + return False # True = anomaly detected +``` + +Same pattern applies for `CoreParser` — implement `parse(input_: LogSchema, output_: ParserSchema) -> bool`. + +### Wiring persist support into a new detector + +Detectors that maintain an `EventPersistency` instance must do two things to support the `persist:` config block: + +**1. Call `_register_persistency()` at the end of `__init__`:** + +```python +def __init__(self, name="MyDetector", config=MyDetectorConfig()): + super().__init__(name=name, config=config) + self.persistency = EventPersistency(event_data_class=EventStabilityTracker) + self._register_persistency(self.persistency) # must be last +``` + +**2. Preserve `config.persist` across `set_configuration()` rebuilds:** + +`set_configuration()` replaces `self.config` via `from_dict()`, which produces a config with no `persist` key — silently dropping the user's persist settings. Save and restore it: + +```python +def set_configuration(self) -> None: + old_persist = self.config.persist + # ... build config_dict, call from_dict() ... + self.config = MyDetectorConfig.from_dict(config_dict, self.name) + self.config.persist = old_persist +``` + +Omitting either step means a `persist:` block in the YAML is silently ignored with no error. + +## Code Quality + +Pre-commit hooks enforce: +- **mypy** strict mode +- **flake8** linting, **autopep8** formatting (max line 110) +- **bandit** security checks, **vulture** dead-code detection (70% threshold) +- **docformatter** docstring style + +Python 3.12 is required (see `.python-version`). + + +# Git +NEVER include "Co-Authored-By ..." in your commit or PR messages. + +Design documents (files under `docs/design/`) must NEVER be committed to the repository. diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index 95c372c9..00000000 --- a/CLAUDE.md +++ /dev/null @@ -1,102 +0,0 @@ -# CLAUDE.md - -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. - -## Project Overview - -DetectMateLibrary is a Python library for log processing and anomaly detection. It provides composable, stream-friendly components (parsers and detectors) that communicate via Protobuf-based schemas. The library is designed for both single-process and microservice deployments. - -## Development Commands - -```bash -# Install dependencies and pre-commit hooks -uv sync --dev -uv run prek install - -# Run tests -uv run pytest -q -uv run pytest -s # verbose with stdout -uv run pytest --cov=. --cov-report=term-missing # with coverage -uv run pytest tests/test_foo.py # single test file - -# Run linting/formatting (all pre-commit hooks) -uv run prek run -a - -# Recompile Protobuf (only if schemas.proto is modified) -protoc --proto_path=src/detectmatelibrary/schemas/ \ - --python_out=src/detectmatelibrary/schemas/ \ - src/detectmatelibrary/schemas/schemas.proto - -# Scaffold a new component workspace -mate create --type --name --dir -``` - -## Architecture - -### Data Flow - -``` -Raw Logs → Parser → ParserSchema → Detector → DetectorSchema (Alerts) -``` - -All data flows through typed Protobuf-backed schema objects. Components are stateful and support an optional training phase before detection. - -### Core Abstractions (`src/detectmatelibrary/common/`) - -- **`CoreComponent`** — base class managing buffering, ID generation, and training state - - **`CoreParser(CoreComponent)`** — parse raw logs into `ParserSchema` - - **`CoreDetector(CoreComponent)`** — detect anomalies in `ParserSchema`, emit `DetectorSchema` -- **`CoreConfig`** / **`CoreParserConfig`** / **`CoreDetectorConfig`** — Pydantic-based configuration hierarchy - -### Schema System (`src/detectmatelibrary/schemas/`) - -- `BaseSchema` wraps generated Protobuf messages with dict-like access (`schema["field"]`) -- Key schemas: `LogSchema`, `ParserSchema`, `DetectorSchema` -- Support serialization to/from bytes for transport and persistence - -### Buffering Modes (`src/detectmatelibrary/utils/data_buffer.py`) - -Three modes via `ArgsBuffer` config: -- **NO_BUF** — one item at a time (default) -- **BATCH** — accumulate N items, process as batch -- **WINDOW** — sliding window of size N - -### Implementations - -- **Parsers** (`src/detectmatelibrary/parsers/`): `JsonParser`, `DummyParser`, `TemplateMatcherParser` (uses Drain3 for template mining) -- **Detectors** (`src/detectmatelibrary/detectors/`): `NewValueDetector`, `NewValueComboDetector`, `RandomDetector`, `DummyDetector` -- **Utilities** (`src/detectmatelibrary/utils/`): `DataBuffer`, `EventPersistency`, `KeyExtractor`, `TimeFormatHandler`, `IdGenerator` - -## Extending the Library - -Implement a custom detector by subclassing `CoreDetector`: - -```python -class MyDetectorConfig(CoreDetectorConfig): - method_type: str = "my_detector" - my_param: int = 10 - -class MyDetector(CoreDetector): - def __init__(self, name="MyDetector", config=MyDetectorConfig()): - super().__init__(name=name, config=config) - - def train(self, input_: ParserSchema) -> None: - pass # optional - - def detect(self, input_: ParserSchema, output_: DetectorSchema) -> bool: - output_["detectorID"] = self.name - output_["score"] = 0.0 - return False # True = anomaly detected -``` - -Same pattern applies for `CoreParser` — implement `parse(input_: LogSchema, output_: ParserSchema) -> bool`. - -## Code Quality - -Pre-commit hooks enforce: -- **mypy** strict mode -- **flake8** linting, **autopep8** formatting (max line 110) -- **bandit** security checks, **vulture** dead-code detection (70% threshold) -- **docformatter** docstring style - -Python 3.12 is required (see `.python-version`). diff --git a/README.md b/README.md index 75442afa..c9ccb822 100644 --- a/README.md +++ b/README.md @@ -96,3 +96,19 @@ workspaces/custom_parser/ # workspace root ├── pyproject.toml # minimal project + dev extras └── README.md # setup instructions ``` + +## Documentation + +- [Project Documentation](https://ait-detectmate.github.io/DetectMateLibrary/latest/) + +## Contribution + +We're happily taking patches and other contributions. Please see the following links for how to get started: + +- [GitHub Workflow](https://ait-detectmate.github.io/DetectMateLibrary/latest/contribution/) + +If you encounter any bugs, please create an issue on [Github](https://github.com/ait-detectmate/DetectMateLibrary/issues). + +## License + +[EUPL-1.2](https://github.com/ait-detectmate/DetectMateLibrary/blob/main/LICENSE.md) diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 00000000..e7cbac59 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,32 @@ +# Security Policy + +## Supported Versions + +| Version | Supported | +| ------- | ------------------ | +| 1.x.x | :white_check_mark: | +| < 1.0.0 | :x: | + +> [!IMPORTANT] +> Currently DetectMateService is a work in progress and heavily under development. Possible vulnerabilities will not be treated any special and can be issued using [GitHub-Issues](https://github.com/ait-detectmate/DetectMateService/issues) + +## Reporting a Vulnerability + +Please email reports about any security related issues you find to aecid@ait.ac.at. This mail is delivered to a small developer team. Your email will be acknowledged within one business day, and you'll receive a more detailed response to your email within 7 days indicating the next steps in handling your report. + +Please use a descriptive subject line for your report email. After the initial reply to your report, our team will endeavor to keep you informed of the progress being made towards a fix and announcement. + +In addition, please include the following information along with your report: + +* Your name and affiliation (if any). +* A description of the technical details of the vulnerabilities. It is very important to let us know how we can reproduce your findings. +* An explanation who can exploit this vulnerability, and what they gain when doing so -- write an attack scenario. This will help us evaluate your report quickly, especially if the issue is complex. +* Whether this vulnerability public or known to third parties. If it is, please provide details. +* Whether we could mention your name in the changelogs. + +Once an issue is reported we use the following disclosure process: + +* When a report is received, we confirm the issue and determine its severity. +* If we know of specific third-party services or software based on DetectMateService that require mitigation before publication, those projects will be notified. +* Fixes are prepared for the last minor release of the latest major release. +* Patch releases are published for all fixed released versions. diff --git a/config/pipeline_config_default.yaml b/config/pipeline_config_default.yaml index 04754958..c9888068 100644 --- a/config/pipeline_config_default.yaml +++ b/config/pipeline_config_default.yaml @@ -2,13 +2,13 @@ parsers: MatcherParser: method_type: matcher_parser auto_config: False - log_format: "type= msg=audit(