diff --git a/README.md b/README.md index 4b150020..de570e59 100644 --- a/README.md +++ b/README.md @@ -22,27 +22,36 @@ executable, including the React client. ![Fanout architecture](docs/diagrams/architecture.svg) -Telemetry lands over OTLP/gRPC or OTLP/HTTP, is batched into DuckLake/Parquet, -and is read back through a DuckDB query kernel that also maintains service, -endpoint, and edge rollups. The browser client, an in-process agent, and any +Telemetry lands over OTLP/gRPC or OTLP/HTTP. Concurrent small requests may +share a group-commit batch, while up to four workers independently encode and +durably publish atomic Parquet directories with persistent trace indexes. +Targeted trace reads go through those indexes; DuckDB scans the same Parquet +for SQL and maintains rebuildable service, endpoint, and edge rollups. The +browser client, an in-process agent, and any external MCP host all reach the same typed observability contract rather than issuing raw SQL. -Every write to the telemetry catalog — ingest flush, rollups, and background -maintenance alike — passes through a single write gate that holds one catalog -write in flight at a time: +Parquet is the telemetry source of truth, DuckDB query state is disposable, +and SQLite is reserved for transactional product state. Native compaction +prepares replacements while reads continue and briefly gates readers only for +the crash-safe namespace swap: ![Fanout persistence](docs/diagrams/persistence.svg) Application state (users, sessions, dashboards, alert rules, agent threads) -lives in a separate SQLite database and never sits on the telemetry write path. +lives in that separate SQLite database and never sits on the telemetry write +path. There is no Iceberg, DuckLake, external catalog, or telemetry server +database. ## Performance The bundled [`cmd/bench`](cmd/bench) driver measures authenticated ingest and -optional dashboard read load against your hardware. Fanout does not publish a -throughput headline until the raw reports and exact driver revision can ship -with it; see [the benchmark publication standard](docs/benchmarking.md). +optional dashboard read load against your hardware. Ingest, DuckDB queries, +and native Parquet maintenance have separate coordination paths but still +compete for the same CPU, memory bandwidth, filesystem cache, and disk. Fanout +does not publish a throughput headline until the raw reports and exact driver +revision can ship with it; see [the benchmark publication +standard](docs/benchmarking.md). ## How it compares @@ -53,7 +62,7 @@ what separates it from its neighbours. | If you use | Where Fanout differs | | --- | --- | | **Grafana with Loki, Tempo, and Mimir** | That stack keeps a service and a query language per signal, plus object storage underneath. Fanout keeps one process, one data directory, and one typed contract across all three signals, at the cost of the horizontal scale those components are built for. | -| **SigNoz** | Both are OTLP-native and self-hosted. SigNoz composes a collector, ClickHouse, and query services; Fanout compiles ingest, storage, query, alerting, and the browser client into one binary, with DuckLake/Parquet on local disk instead of a database cluster. | +| **SigNoz** | Both are OTLP-native and self-hosted. SigNoz composes a collector, ClickHouse, and query services; Fanout compiles ingest, indexed storage, DuckDB queries, alerting, and the browser client into one binary, with open Parquet on local disk instead of a database cluster. | | **Jaeger** | Jaeger covers traces and expects a storage backend you run separately. Fanout ingests traces, logs, and metrics into the same store, with nothing else to deploy. | | **Prometheus with Grafana** | Prometheus pulls metrics and is excellent at them. Fanout accepts pushed OTLP for all three signals and is built around investigating a specific incident rather than maintaining long-range metric series. | | **Datadog**, **Honeycomb**, **Grafana Cloud** | Those are managed services: someone else runs the storage, the scaling, and the upgrades, and your telemetry leaves your network to get there. Fanout is a binary you run, on data that stays on your disk. | diff --git a/THIRD_PARTY_NOTICES b/THIRD_PARTY_NOTICES index e318bb06..be80c728 100644 --- a/THIRD_PARTY_NOTICES +++ b/THIRD_PARTY_NOTICES @@ -17,6 +17,7 @@ COMPONENT INVENTORY - Go: github.com/ag-ui-protocol/ag-ui/sdks/community/go v0.0.0-20260826145851-49e71f2b2d21 - Go: github.com/agext/levenshtein v1.2.3 - Go: github.com/alexedwards/scs/v2 v2.9.0 +- Go: github.com/andybalholm/brotli v1.2.2 - Go: github.com/antlr4-go/antlr/v4 v4.13.1 - Go: github.com/apparentlymart/go-textseg/v15 v15.0.0 - Go: github.com/apparentlymart/go-textseg/v17 v17.0.1 @@ -40,6 +41,8 @@ COMPONENT INVENTORY - Go: github.com/google/uuid v1.6.0 - Go: github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 - Go: github.com/hashicorp/hcl/v2 v2.24.0 +- Go: github.com/klauspost/compress v1.19.2 +- Go: github.com/klauspost/cpuid/v2 v2.4.0 - Go: github.com/knadh/koanf/maps v0.1.3 - Go: github.com/knadh/koanf/parsers/yaml v1.1.1 - Go: github.com/knadh/koanf/providers/confmap v1.0.1 @@ -53,6 +56,10 @@ COMPONENT INVENTORY - Go: github.com/modelcontextprotocol/go-sdk v1.7.0 - Go: github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 - Go: github.com/ncruces/go-strftime v1.0.0 +- Go: github.com/parquet-go/bitpack v1.0.0 +- Go: github.com/parquet-go/jsonlite v1.0.0 +- Go: github.com/parquet-go/parquet-go v0.32.0 +- Go: github.com/pierrec/lz4/v4 v4.1.29 - Go: github.com/prometheus/client_golang v1.24.1 - Go: github.com/prometheus/client_model v0.6.2 - Go: github.com/prometheus/common v0.70.1 @@ -61,10 +68,12 @@ COMPONENT INVENTORY - Go: github.com/segmentio/asm v1.2.1 - Go: github.com/segmentio/encoding v0.5.4 - Go: github.com/sirupsen/logrus v1.10.2 +- Go: github.com/twpayne/go-geom v1.6.1 - Go: github.com/wneessen/go-mail v0.8.1 - Go: github.com/yosida95/uritemplate/v3 v3.0.2 - Go: github.com/zclconf/go-cty v1.19.0 - Go: github.com/zclconf/go-cty-yaml v1.2.0 +- Go: github.com/zeebo/xxh3 v1.1.0 - Go: go.opentelemetry.io/proto/otlp v1.11.0 - Go: go.yaml.in/yaml/v3 v3.0.5 - Go: golang.org/x/crypto v0.55.0 @@ -869,6 +878,30 @@ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +--- Applies to ------------------------------------------------------------- +- Go: github.com/andybalholm/brotli v1.2.2 / LICENSE +---------------------------------------------------------------------------- + +Copyright (c) 2009, 2010, 2013-2016 by the Brotli Authors. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + --- Applies to ------------------------------------------------------------- - Go: github.com/antlr4-go/antlr/v4 v4.13.1 / LICENSE ---------------------------------------------------------------------------- @@ -1951,113 +1984,41 @@ Exhibit B - “Incompatible With Secondary Licenses” Notice the Mozilla Public License, v. 2.0. --- Applies to ------------------------------------------------------------- -- Go: github.com/knadh/koanf/maps v0.1.3 / LICENSE -- Go: github.com/knadh/koanf/parsers/yaml v1.1.1 / LICENSE -- Go: github.com/knadh/koanf/providers/confmap v1.0.1 / LICENSE -- Go: github.com/knadh/koanf/providers/file v1.2.1 / LICENSE -- Go: github.com/knadh/koanf/v2 v2.3.6 / LICENSE ----------------------------------------------------------------------------- - -The MIT License - -Copyright (c) 2019, Kailash Nadh. https://github.com/knadh - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in -all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN -THE SOFTWARE. - ---- Applies to ------------------------------------------------------------- -- Go: github.com/labstack/echo/v5 v5.3.1 / LICENSE ----------------------------------------------------------------------------- - -The MIT License (MIT) - -Copyright (c) 2022 LabStack - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. - ---- Applies to ------------------------------------------------------------- -- Go: github.com/mattn/go-isatty v0.0.24 / LICENSE ----------------------------------------------------------------------------- - -Copyright (c) Yasuhiro MATSUMOTO - -MIT License (Expat) - -Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - ---- Applies to ------------------------------------------------------------- -- Go: github.com/mitchellh/copystructure v1.2.0 / LICENSE -- Go: github.com/mitchellh/go-wordwrap v1.0.1 / LICENSE.md +- Go: github.com/klauspost/compress v1.19.2 / LICENSE ---------------------------------------------------------------------------- -The MIT License (MIT) - -Copyright (c) 2014 Mitchell Hashimoto - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in -all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN -THE SOFTWARE. +Copyright (c) 2012 The Go Authors. All rights reserved. +Copyright (c) 2019 Klaus Post. All rights reserved. ---- Applies to ------------------------------------------------------------- -- Go: github.com/modelcontextprotocol/go-sdk v1.7.0 / LICENSE ----------------------------------------------------------------------------- +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: -The MCP project is undergoing a licensing transition from the MIT License to the Apache License, Version 2.0 ("Apache-2.0"). All new code and specification contributions to the project are licensed under Apache-2.0. Documentation contributions (excluding specifications) are licensed under CC-BY-4.0. + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. -Contributions for which relicensing consent has been obtained are licensed under Apache-2.0. Contributions made by authors who originally licensed their work under the MIT License and who have not yet granted explicit permission to relicense remain licensed under the MIT License. +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -No rights beyond those granted by the applicable original license are conveyed for such contributions. +------------------ ---- +Files: gzhttp/* Apache License Version 2.0, January 2004 @@ -2109,9 +2070,9 @@ No rights beyond those granted by the applicable original license are conveyed f "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright - owner or by an individual or Legal Entity authorized to submit on behalf - of the copyright owner. For the purposes of this definition, "submitted" + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, @@ -2236,11 +2197,38 @@ No rights beyond those granted by the applicable original license are conveyed f END OF TERMS AND CONDITIONS ---- + APPENDIX: How to apply the Apache License to your work. -MIT License + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. -Copyright (c) 2024-2025 Model Context Protocol a Series of LF Projects, LLC. + Copyright 2016-2017 The New York Times Company + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +------------------ + +Files: s2/cmd/internal/readahead/* + +The MIT License (MIT) + +Copyright (c) 2015 Klaus Post Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal @@ -2260,43 +2248,31 @@ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ---- - -Creative Commons Attribution 4.0 International (CC-BY-4.0) - -Documentation in this project (excluding specifications) is licensed under -CC-BY-4.0. See https://creativecommons.org/licenses/by/4.0/legalcode for -the full license text. - ---- Applies to ------------------------------------------------------------- -- Go: github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 / LICENSE ----------------------------------------------------------------------------- +--------------------- +Files: snappy/* +Files: internal/snapref/* -Copyright (c) 2011, Open Knowledge Foundation Ltd. -All rights reserved. +Copyright (c) 2011 The Snappy-Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: - Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - - Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - - Neither the name of the Open Knowledge Foundation Ltd. nor the - names of its contributors may be used to endorse or promote - products derived from this software without specific prior written - permission. + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY @@ -2304,13 +2280,25 @@ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +----------------- + +Files: s2/cmd/internal/filepathx/* + +Copyright 2016 The filepathx Authors + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + --- Applies to ------------------------------------------------------------- -- Go: github.com/ncruces/go-strftime v1.0.0 / LICENSE +- Go: github.com/klauspost/cpuid/v2 v2.4.0 / LICENSE ---------------------------------------------------------------------------- -MIT License +The MIT License (MIT) -Copyright (c) 2022 Nuno Cruces +Copyright (c) 2015 Klaus Post Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal @@ -2331,24 +2319,885 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. --- Applies to ------------------------------------------------------------- -- Go: github.com/prometheus/client_golang v1.24.1 / NOTICE +- Go: github.com/knadh/koanf/maps v0.1.3 / LICENSE +- Go: github.com/knadh/koanf/parsers/yaml v1.1.1 / LICENSE +- Go: github.com/knadh/koanf/providers/confmap v1.0.1 / LICENSE +- Go: github.com/knadh/koanf/providers/file v1.2.1 / LICENSE +- Go: github.com/knadh/koanf/v2 v2.3.6 / LICENSE ---------------------------------------------------------------------------- -Prometheus instrumentation library for Go applications -Copyright 2012-2015 The Prometheus Authors - -This product includes software developed at -SoundCloud Ltd. (http://soundcloud.com/). +The MIT License +Copyright (c) 2019, Kailash Nadh. https://github.com/knadh -The following components are included in this product: +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: -perks - a fork of https://github.com/bmizerany/perks -https://github.com/beorn7/perks -Copyright 2013-2015 Blake Mizerany, Björn Rabenstein -See https://github.com/beorn7/perks/blob/master/README.md for license details. +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. -Go support for Protocol Buffers - Google's data interchange format +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +--- Applies to ------------------------------------------------------------- +- Go: github.com/labstack/echo/v5 v5.3.1 / LICENSE +---------------------------------------------------------------------------- + +The MIT License (MIT) + +Copyright (c) 2022 LabStack + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +--- Applies to ------------------------------------------------------------- +- Go: github.com/mattn/go-isatty v0.0.24 / LICENSE +---------------------------------------------------------------------------- + +Copyright (c) Yasuhiro MATSUMOTO + +MIT License (Expat) + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +--- Applies to ------------------------------------------------------------- +- Go: github.com/mitchellh/copystructure v1.2.0 / LICENSE +- Go: github.com/mitchellh/go-wordwrap v1.0.1 / LICENSE.md +---------------------------------------------------------------------------- + +The MIT License (MIT) + +Copyright (c) 2014 Mitchell Hashimoto + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +--- Applies to ------------------------------------------------------------- +- Go: github.com/modelcontextprotocol/go-sdk v1.7.0 / LICENSE +---------------------------------------------------------------------------- + +The MCP project is undergoing a licensing transition from the MIT License to the Apache License, Version 2.0 ("Apache-2.0"). All new code and specification contributions to the project are licensed under Apache-2.0. Documentation contributions (excluding specifications) are licensed under CC-BY-4.0. + +Contributions for which relicensing consent has been obtained are licensed under Apache-2.0. Contributions made by authors who originally licensed their work under the MIT License and who have not yet granted explicit permission to relicense remain licensed under the MIT License. + +No rights beyond those granted by the applicable original license are conveyed for such contributions. + +--- + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright + owner or by an individual or Legal Entity authorized to submit on behalf + of the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + +--- + +MIT License + +Copyright (c) 2024-2025 Model Context Protocol a Series of LF Projects, LLC. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +--- + +Creative Commons Attribution 4.0 International (CC-BY-4.0) + +Documentation in this project (excluding specifications) is licensed under +CC-BY-4.0. See https://creativecommons.org/licenses/by/4.0/legalcode for +the full license text. + +--- Applies to ------------------------------------------------------------- +- Go: github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 / LICENSE +---------------------------------------------------------------------------- + +Copyright (c) 2011, Open Knowledge Foundation Ltd. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + + Neither the name of the Open Knowledge Foundation Ltd. nor the + names of its contributors may be used to endorse or promote + products derived from this software without specific prior written + permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +--- Applies to ------------------------------------------------------------- +- Go: github.com/ncruces/go-strftime v1.0.0 / LICENSE +---------------------------------------------------------------------------- + +MIT License + +Copyright (c) 2022 Nuno Cruces + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +--- Applies to ------------------------------------------------------------- +- Go: github.com/parquet-go/bitpack v1.0.0 / LICENSE +---------------------------------------------------------------------------- + +Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2025 Achille Roussel, Filip Petkovski + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +--- Applies to ------------------------------------------------------------- +- Go: github.com/parquet-go/jsonlite v1.0.0 / LICENSE +---------------------------------------------------------------------------- + +MIT License + +Copyright (c) 2025 parquet-go + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +--- Applies to ------------------------------------------------------------- +- Go: github.com/parquet-go/parquet-go v0.32.0 / LICENSE +---------------------------------------------------------------------------- + +Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2023 Twilio, Inc. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +-------------------------------------------------------------------------------- + +This product includes code from Apache Parquet. + +* deprecated/parquet.go is based on Apache Parquet's thrift file +* format/parquet.go is based on Apache Parquet's thrift file + +Copyright: 2014 The Apache Software Foundation. +Home page: https://github.com/apache/parquet-format +License: http://www.apache.org/licenses/LICENSE-2.0 + +--- Applies to ------------------------------------------------------------- +- Go: github.com/pierrec/lz4/v4 v4.1.29 / LICENSE +---------------------------------------------------------------------------- + +Copyright (c) 2015, Pierre Curto +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of xxHash nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +--- Applies to ------------------------------------------------------------- +- Go: github.com/prometheus/client_golang v1.24.1 / NOTICE +---------------------------------------------------------------------------- + +Prometheus instrumentation library for Go applications +Copyright 2012-2015 The Prometheus Authors + +This product includes software developed at +SoundCloud Ltd. (http://soundcloud.com/). + + +The following components are included in this product: + +perks - a fork of https://github.com/bmizerany/perks +https://github.com/beorn7/perks +Copyright 2013-2015 Blake Mizerany, Björn Rabenstein +See https://github.com/beorn7/perks/blob/master/README.md for license details. + +Go support for Protocol Buffers - Google's data interchange format http://github.com/golang/protobuf/ Copyright 2010 The Go Authors See source code for license details. @@ -2490,6 +3339,34 @@ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +--- Applies to ------------------------------------------------------------- +- Go: github.com/twpayne/go-geom v1.6.1 / LICENSE +---------------------------------------------------------------------------- + +Copyright (c) 2013, Tom Payne +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + + Redistributions in binary form must reproduce the above copyright notice, this + list of conditions and the following disclaimer in the documentation and/or + other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + --- Applies to ------------------------------------------------------------- - Go: github.com/wneessen/go-mail v0.8.1 / LICENSE ---------------------------------------------------------------------------- @@ -2633,6 +3510,39 @@ license. See LICENSE.libyaml for more information. Modifications for cty interfacing copyright 2019 Martin Atkins, and distributed under the same license terms. +--- Applies to ------------------------------------------------------------- +- Go: github.com/zeebo/xxh3 v1.1.0 / LICENSE +---------------------------------------------------------------------------- + +BSD 2-Clause License + +Copyright (c) 2012-2014, Yann Collet +Copyright (c) 2019, Jeff Wendling +All rights reserved. + +xxHash Library + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, this + list of conditions and the following disclaimer in the documentation and/or + other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + --- Applies to ------------------------------------------------------------- - Go: go.yaml.in/yaml/v3 v3.0.5 / LICENSE - Go: gopkg.in/yaml.v3 v3.0.1 / LICENSE diff --git a/cmd/bench/main.go b/cmd/bench/main.go index 882e9427..fbf9e538 100644 --- a/cmd/bench/main.go +++ b/cmd/bench/main.go @@ -81,8 +81,7 @@ type config struct { maxQueryP95 float64 // backfillHours, when >0, spreads each event's timestamp uniformly over the // last N hours (instead of "now"). Used to PRE-SEED a multi-hour dataset so - // the lake spans several hour partitions — required to exercise within-day - // (hour-partition) pruning, which a same-hour run can't. + // time-window queries and rollups cover more than the current hour. backfillHours float64 // seed makes the synthetic workload reproducible: same seed, same services, // endpoints, attributes, and error placement. Two runs are only comparable if @@ -557,7 +556,7 @@ func (g *generator) outCtx(ctx context.Context) context.Context { // eventTime returns the timestamp for an emitted event: now(), or — when // backfillHours>0 — a time spread uniformly over the last N hours so a pre-seed -// run populates multiple hour partitions (to exercise within-day pruning). +// run exercises multi-hour time-window queries and rollups. func (g *generator) eventTime(rng *rand.Rand) time.Time { if g.cfg.backfillHours <= 0 { return time.Now() @@ -910,43 +909,43 @@ type latencyReport struct { } type serverReport struct { - BaselineAvailable bool `json:"baseline_available"` - ProcessStartTime float64 `json:"process_start_time_seconds"` - ProcessRestarted bool `json:"process_restarted"` - IngestRowsStart float64 `json:"ingest_rows_start"` - IngestRowsEnd float64 `json:"ingest_rows_end"` - IngestRowsDelta float64 `json:"ingest_rows_delta"` - RowsDroppedStart float64 `json:"rows_dropped_start"` - RowsDroppedEnd float64 `json:"rows_dropped_end"` - RowsDroppedDelta float64 `json:"rows_dropped_delta"` - LakePartitionsStart float64 `json:"lake_partitions_start"` - LakePartitions float64 `json:"lake_partitions"` - LakePartitionsDelta float64 `json:"lake_partitions_delta"` - LakeSizeBytesStart float64 `json:"lake_size_bytes_start"` - LakeSizeBytes float64 `json:"lake_size_bytes"` - LakeSizeBytesDelta float64 `json:"lake_size_bytes_delta"` - LakeGrowthBytesPerSec float64 `json:"lake_growth_bytes_per_sec"` - IngestQueueDepth float64 `json:"ingest_queue_depth"` - AvgRollupMs float64 `json:"avg_rollup_ms"` - AvgFlushMs float64 `json:"avg_flush_ms"` - AvgQueryMs float64 `json:"avg_query_ms"` - CPUSecondsStart float64 `json:"cpu_seconds_start"` - CPUSecondsEnd float64 `json:"cpu_seconds_end"` - CPUSecondsDelta float64 `json:"cpu_seconds_delta"` - CPUCores float64 `json:"cpu_cores"` - RSSBytes float64 `json:"rss_bytes"` - HeapAllocBytes float64 `json:"heap_alloc_bytes"` - AllocBytesStart float64 `json:"alloc_bytes_start"` - AllocBytesEnd float64 `json:"alloc_bytes_end"` - AllocBytesDelta float64 `json:"alloc_bytes_delta"` - AllocBytesPerSec float64 `json:"alloc_bytes_per_sec"` - GCPauseSecondsStart float64 `json:"gc_pause_seconds_start"` - GCPauseSecondsEnd float64 `json:"gc_pause_seconds_end"` - GCPauseSecondsDelta float64 `json:"gc_pause_seconds_delta"` - WriteGateWaitMs map[string]distributionReport `json:"write_gate_wait_ms,omitempty"` - WriteGateHoldMs map[string]distributionReport `json:"write_gate_hold_ms,omitempty"` - DuckLakeOperations map[string]backgroundOperationReport `json:"ducklake_operations,omitempty"` - Rollups map[string]rollupReport `json:"rollups,omitempty"` + BaselineAvailable bool `json:"baseline_available"` + ProcessStartTime float64 `json:"process_start_time_seconds"` + ProcessRestarted bool `json:"process_restarted"` + IngestRowsStart float64 `json:"ingest_rows_start"` + IngestRowsEnd float64 `json:"ingest_rows_end"` + IngestRowsDelta float64 `json:"ingest_rows_delta"` + RowsDroppedStart float64 `json:"rows_dropped_start"` + RowsDroppedEnd float64 `json:"rows_dropped_end"` + RowsDroppedDelta float64 `json:"rows_dropped_delta"` + ParquetFilesStart float64 `json:"parquet_files_start"` + ParquetFiles float64 `json:"parquet_files"` + ParquetFilesDelta float64 `json:"parquet_files_delta"` + ParquetSizeBytesStart float64 `json:"parquet_size_bytes_start"` + ParquetSizeBytes float64 `json:"parquet_size_bytes"` + ParquetSizeBytesDelta float64 `json:"parquet_size_bytes_delta"` + ParquetGrowthBytesPerSec float64 `json:"parquet_growth_bytes_per_sec"` + IngestQueueDepth float64 `json:"ingest_queue_depth"` + AvgRollupMs float64 `json:"avg_rollup_ms"` + AvgFlushMs float64 `json:"avg_flush_ms"` + AvgQueryMs float64 `json:"avg_query_ms"` + CPUSecondsStart float64 `json:"cpu_seconds_start"` + CPUSecondsEnd float64 `json:"cpu_seconds_end"` + CPUSecondsDelta float64 `json:"cpu_seconds_delta"` + CPUCores float64 `json:"cpu_cores"` + RSSBytes float64 `json:"rss_bytes"` + HeapAllocBytes float64 `json:"heap_alloc_bytes"` + AllocBytesStart float64 `json:"alloc_bytes_start"` + AllocBytesEnd float64 `json:"alloc_bytes_end"` + AllocBytesDelta float64 `json:"alloc_bytes_delta"` + AllocBytesPerSec float64 `json:"alloc_bytes_per_sec"` + GCPauseSecondsStart float64 `json:"gc_pause_seconds_start"` + GCPauseSecondsEnd float64 `json:"gc_pause_seconds_end"` + GCPauseSecondsDelta float64 `json:"gc_pause_seconds_delta"` + WriteGateWaitMs map[string]distributionReport `json:"write_gate_wait_ms,omitempty"` + WriteGateHoldMs map[string]distributionReport `json:"write_gate_hold_ms,omitempty"` + TelemetryOperations map[string]backgroundOperationReport `json:"telemetry_operations,omitempty"` + Rollups map[string]rollupReport `json:"rollups,omitempty"` } func printReport(r report) { @@ -982,11 +981,11 @@ func printReport(r report) { s := r.Server fmt.Printf("server (Δ over run):\n") fmt.Printf(" rows accepted=%.0f dropped=%.0f\n", s.IngestRowsDelta, s.RowsDroppedDelta) - fmt.Printf(" lake_partitions=%.0f lake_size=%.1fMB ingest_queue_depth=%.0f\n", - s.LakePartitions, s.LakeSizeBytes/(1<<20), s.IngestQueueDepth) + fmt.Printf(" parquet_files=%.0f parquet_size=%.1fMB ingest_queue_depth=%.0f\n", + s.ParquetFiles, s.ParquetSizeBytes/(1<<20), s.IngestQueueDepth) fmt.Printf(" avg rollup=%.1fms flush=%.1fms query=%.1fms\n", s.AvgRollupMs, s.AvgFlushMs, s.AvgQueryMs) - fmt.Printf(" cpu=%.2f core(s) rss=%.1fMB alloc=%.1fMB/s lake_growth=%.1fMB\n", - s.CPUCores, s.RSSBytes/(1<<20), s.AllocBytesPerSec/(1<<20), s.LakeSizeBytesDelta/(1<<20)) + fmt.Printf(" cpu=%.2f core(s) rss=%.1fMB alloc=%.1fMB/s parquet_growth=%.1fMB\n", + s.CPUCores, s.RSSBytes/(1<<20), s.AllocBytesPerSec/(1<<20), s.ParquetSizeBytesDelta/(1<<20)) } if r.Passed { fmt.Printf("verdict PASS\n") diff --git a/cmd/bench/metrics_report.go b/cmd/bench/metrics_report.go index 7763548b..936ecae1 100644 --- a/cmd/bench/metrics_report.go +++ b/cmd/bench/metrics_report.go @@ -292,8 +292,8 @@ func serverDelta(base, final *metricSnapshot, durationSeconds float64) *serverRe } return round2(value / durationSeconds) } - lakePartitionsStart := base.total("fanout_lake_partitions") - lakeSizeStart := base.total("fanout_lake_size_bytes") + parquetFilesStart := base.total("fanout_parquet_files") + parquetSizeStart := base.total("fanout_parquet_size_bytes") cpuSeconds := delta("process_cpu_seconds_total") allocBytes := delta("go_memstats_alloc_bytes_total") // process_start_time_seconds is constant for the life of a process, so a @@ -302,43 +302,43 @@ func serverDelta(base, final *metricSnapshot, durationSeconds float64) *serverRe startTimeBefore := base.total("process_start_time_seconds") startTimeAfter := final.total("process_start_time_seconds") return &serverReport{ - BaselineAvailable: baselineAvailable, - ProcessStartTime: startTimeAfter, - ProcessRestarted: baselineAvailable && startTimeBefore > 0 && startTimeAfter != startTimeBefore, - IngestRowsStart: base.total("fanout_ingest_rows_total"), - IngestRowsEnd: final.total("fanout_ingest_rows_total"), - IngestRowsDelta: delta("fanout_ingest_rows_total"), - RowsDroppedStart: base.total("fanout_rows_dropped_total"), - RowsDroppedEnd: final.total("fanout_rows_dropped_total"), - RowsDroppedDelta: delta("fanout_rows_dropped_total"), - LakePartitionsStart: lakePartitionsStart, - LakePartitions: final.total("fanout_lake_partitions"), - LakePartitionsDelta: final.total("fanout_lake_partitions") - lakePartitionsStart, - LakeSizeBytesStart: lakeSizeStart, - LakeSizeBytes: final.total("fanout_lake_size_bytes"), - LakeSizeBytesDelta: final.total("fanout_lake_size_bytes") - lakeSizeStart, - LakeGrowthBytesPerSec: rate(final.total("fanout_lake_size_bytes") - lakeSizeStart), - IngestQueueDepth: final.total("fanout_ingest_queue_depth"), - AvgRollupMs: averageDurationMs(base, final, "fanout_rollup_duration_seconds"), - AvgFlushMs: averageDurationMs(base, final, "fanout_flush_duration_seconds"), - AvgQueryMs: averageDurationMs(base, final, "fanout_query_duration_seconds"), - CPUSecondsStart: round4(base.total("process_cpu_seconds_total")), - CPUSecondsEnd: round4(final.total("process_cpu_seconds_total")), - CPUSecondsDelta: round4(cpuSeconds), - CPUCores: perSecond(cpuSeconds, durationSeconds), - RSSBytes: final.total("process_resident_memory_bytes"), - HeapAllocBytes: final.total("go_memstats_heap_alloc_bytes"), - AllocBytesStart: base.total("go_memstats_alloc_bytes_total"), - AllocBytesEnd: final.total("go_memstats_alloc_bytes_total"), - AllocBytesDelta: allocBytes, - AllocBytesPerSec: rate(allocBytes), - GCPauseSecondsStart: round4(base.total("go_gc_duration_seconds_sum")), - GCPauseSecondsEnd: round4(final.total("go_gc_duration_seconds_sum")), - GCPauseSecondsDelta: round4(delta("go_gc_duration_seconds_sum")), - WriteGateWaitMs: histogramReports(base, final, "fanout_write_gate_wait_seconds", "operation"), - WriteGateHoldMs: histogramReports(base, final, "fanout_write_gate_hold_seconds", "operation"), - DuckLakeOperations: backgroundReports(base, final), - Rollups: rollupReports(base, final), + BaselineAvailable: baselineAvailable, + ProcessStartTime: startTimeAfter, + ProcessRestarted: baselineAvailable && startTimeBefore > 0 && startTimeAfter != startTimeBefore, + IngestRowsStart: base.total("fanout_ingest_rows_total"), + IngestRowsEnd: final.total("fanout_ingest_rows_total"), + IngestRowsDelta: delta("fanout_ingest_rows_total"), + RowsDroppedStart: base.total("fanout_rows_dropped_total"), + RowsDroppedEnd: final.total("fanout_rows_dropped_total"), + RowsDroppedDelta: delta("fanout_rows_dropped_total"), + ParquetFilesStart: parquetFilesStart, + ParquetFiles: final.total("fanout_parquet_files"), + ParquetFilesDelta: final.total("fanout_parquet_files") - parquetFilesStart, + ParquetSizeBytesStart: parquetSizeStart, + ParquetSizeBytes: final.total("fanout_parquet_size_bytes"), + ParquetSizeBytesDelta: final.total("fanout_parquet_size_bytes") - parquetSizeStart, + ParquetGrowthBytesPerSec: rate(final.total("fanout_parquet_size_bytes") - parquetSizeStart), + IngestQueueDepth: final.total("fanout_ingest_queue_depth"), + AvgRollupMs: averageDurationMs(base, final, "fanout_rollup_duration_seconds"), + AvgFlushMs: averageDurationMs(base, final, "fanout_flush_duration_seconds"), + AvgQueryMs: averageDurationMs(base, final, "fanout_query_duration_seconds"), + CPUSecondsStart: round4(base.total("process_cpu_seconds_total")), + CPUSecondsEnd: round4(final.total("process_cpu_seconds_total")), + CPUSecondsDelta: round4(cpuSeconds), + CPUCores: perSecond(cpuSeconds, durationSeconds), + RSSBytes: final.total("process_resident_memory_bytes"), + HeapAllocBytes: final.total("go_memstats_heap_alloc_bytes"), + AllocBytesStart: base.total("go_memstats_alloc_bytes_total"), + AllocBytesEnd: final.total("go_memstats_alloc_bytes_total"), + AllocBytesDelta: allocBytes, + AllocBytesPerSec: rate(allocBytes), + GCPauseSecondsStart: round4(base.total("go_gc_duration_seconds_sum")), + GCPauseSecondsEnd: round4(final.total("go_gc_duration_seconds_sum")), + GCPauseSecondsDelta: round4(delta("go_gc_duration_seconds_sum")), + WriteGateWaitMs: histogramReports(base, final, "fanout_write_gate_wait_seconds", "operation"), + WriteGateHoldMs: histogramReports(base, final, "fanout_write_gate_hold_seconds", "operation"), + TelemetryOperations: backgroundReports(base, final), + Rollups: rollupReports(base, final), } } @@ -425,10 +425,10 @@ func histogramDelta(base, final *metricSnapshot, name string, filters map[string func backgroundReports(base, final *metricSnapshot) map[string]backgroundOperationReport { operations := unionStrings( - base.labelValues("fanout_ducklake_operation_total", "operation", nil), - final.labelValues("fanout_ducklake_operation_total", "operation", nil), - base.labelValues("fanout_ducklake_operation_duration_seconds_count", "operation", nil), - final.labelValues("fanout_ducklake_operation_duration_seconds_count", "operation", nil), + base.labelValues("fanout_telemetry_operation_total", "operation", nil), + final.labelValues("fanout_telemetry_operation_total", "operation", nil), + base.labelValues("fanout_telemetry_operation_duration_seconds_count", "operation", nil), + final.labelValues("fanout_telemetry_operation_duration_seconds_count", "operation", nil), ) if len(operations) == 0 { return nil @@ -437,8 +437,8 @@ func backgroundReports(base, final *metricSnapshot) map[string]backgroundOperati for _, operation := range operations { filters := map[string]string{"operation": operation} reports[operation] = backgroundOperationReport{ - DurationMs: histogramDelta(base, final, "fanout_ducklake_operation_duration_seconds", filters), - Outcomes: counterOutcomes(base, final, "fanout_ducklake_operation_total", "result", filters), + DurationMs: histogramDelta(base, final, "fanout_telemetry_operation_duration_seconds", filters), + Outcomes: counterOutcomes(base, final, "fanout_telemetry_operation_total", "result", filters), } } return reports diff --git a/cmd/bench/metrics_report_test.go b/cmd/bench/metrics_report_test.go index b117d5e0..3074dd94 100644 --- a/cmd/bench/metrics_report_test.go +++ b/cmd/bench/metrics_report_test.go @@ -9,14 +9,14 @@ import ( func TestParseMetricSnapshotPreservesLabelsAndTotals(t *testing.T) { snapshot := mustMetricSnapshot(t, ` # HELP fanout_test_total test metric -fanout_test_total{operation="merge",detail="quoted\" value\\path"} 2 +fanout_test_total{operation="compaction",detail="quoted\" value\\path"} 2 fanout_test_total{operation="maintenance",detail="plain"} 3 `) if got := snapshot.total("fanout_test_total"); got != 5 { t.Fatalf("total = %v, want 5", got) } - if got := snapshot.filteredTotal("fanout_test_total", map[string]string{"operation": "merge"}); got != 2 { + if got := snapshot.filteredTotal("fanout_test_total", map[string]string{"operation": "compaction"}); got != 2 { t.Fatalf("merge total = %v, want 2", got) } if got := snapshot.Samples[0].Labels["detail"]; got != "quoted\" value\\path" { @@ -29,10 +29,10 @@ func TestServerDeltaReportsOperationAndRuntimeDistributions(t *testing.T) { fanout_ingest_rows_total{signal="spans"} 10 fanout_ingest_rows_total{signal="logs"} 5 fanout_rows_dropped_total{signal="spans"} 0 -fanout_lake_partitions{signal="spans"} 2 -fanout_lake_partitions{signal="logs"} 1 -fanout_lake_size_bytes{signal="spans"} 60 -fanout_lake_size_bytes{signal="logs"} 40 +fanout_parquet_files{signal="spans"} 2 +fanout_parquet_files{signal="logs"} 1 +fanout_parquet_size_bytes{signal="spans"} 60 +fanout_parquet_size_bytes{signal="logs"} 40 fanout_ingest_queue_depth{signal="spans"} 2 fanout_ingest_queue_depth{signal="logs"} 1 process_cpu_seconds_total 10 @@ -57,14 +57,14 @@ fanout_write_gate_hold_seconds_bucket{operation="ingest_spans",le="0.100"} 1 fanout_write_gate_hold_seconds_bucket{operation="ingest_spans",le="+Inf"} 1 fanout_write_gate_hold_seconds_sum{operation="ingest_spans"} 0.005 fanout_write_gate_hold_seconds_count{operation="ingest_spans"} 1 -fanout_ducklake_operation_duration_seconds_bucket{operation="merge",le="0.010"} 1 -fanout_ducklake_operation_duration_seconds_bucket{operation="merge",le="0.100"} 1 -fanout_ducklake_operation_duration_seconds_bucket{operation="merge",le="1.000"} 1 -fanout_ducklake_operation_duration_seconds_bucket{operation="merge",le="+Inf"} 1 -fanout_ducklake_operation_duration_seconds_sum{operation="merge"} 0.005 -fanout_ducklake_operation_duration_seconds_count{operation="merge"} 1 -fanout_ducklake_operation_total{operation="merge",result="success"} 1 -fanout_ducklake_operation_total{operation="merge",result="throttled"} 2 +fanout_telemetry_operation_duration_seconds_bucket{operation="compaction",le="0.010"} 1 +fanout_telemetry_operation_duration_seconds_bucket{operation="compaction",le="0.100"} 1 +fanout_telemetry_operation_duration_seconds_bucket{operation="compaction",le="1.000"} 1 +fanout_telemetry_operation_duration_seconds_bucket{operation="compaction",le="+Inf"} 1 +fanout_telemetry_operation_duration_seconds_sum{operation="compaction"} 0.005 +fanout_telemetry_operation_duration_seconds_count{operation="compaction"} 1 +fanout_telemetry_operation_total{operation="compaction",result="success"} 1 +fanout_telemetry_operation_total{operation="compaction",result="throttled"} 2 fanout_rollup_enabled{rollup="service"} 1 fanout_rollup_watermark_timestamp_seconds{rollup="service"} 100 fanout_rollup_source_timestamp_seconds{rollup="service"} 105 @@ -82,10 +82,10 @@ fanout_rollup_component_total{rollup="service",result="success"} 1 fanout_ingest_rows_total{signal="spans"} 30 fanout_ingest_rows_total{signal="logs"} 15 fanout_rows_dropped_total{signal="spans"} 0 -fanout_lake_partitions{signal="spans"} 3 -fanout_lake_partitions{signal="logs"} 2 -fanout_lake_size_bytes{signal="spans"} 100 -fanout_lake_size_bytes{signal="logs"} 80 +fanout_parquet_files{signal="spans"} 3 +fanout_parquet_files{signal="logs"} 2 +fanout_parquet_size_bytes{signal="spans"} 100 +fanout_parquet_size_bytes{signal="logs"} 80 fanout_ingest_queue_depth{signal="spans"} 0 fanout_ingest_queue_depth{signal="logs"} 1 process_cpu_seconds_total 14 @@ -110,15 +110,15 @@ fanout_write_gate_hold_seconds_bucket{operation="ingest_spans",le="0.100"} 3 fanout_write_gate_hold_seconds_bucket{operation="ingest_spans",le="+Inf"} 3 fanout_write_gate_hold_seconds_sum{operation="ingest_spans"} 0.035 fanout_write_gate_hold_seconds_count{operation="ingest_spans"} 3 -fanout_ducklake_operation_duration_seconds_bucket{operation="merge",le="0.010"} 1 -fanout_ducklake_operation_duration_seconds_bucket{operation="merge",le="0.100"} 2 -fanout_ducklake_operation_duration_seconds_bucket{operation="merge",le="1.000"} 3 -fanout_ducklake_operation_duration_seconds_bucket{operation="merge",le="+Inf"} 3 -fanout_ducklake_operation_duration_seconds_sum{operation="merge"} 0.125 -fanout_ducklake_operation_duration_seconds_count{operation="merge"} 3 -fanout_ducklake_operation_total{operation="merge",result="success"} 3 -fanout_ducklake_operation_total{operation="merge",result="throttled"} 5 -fanout_ducklake_operation_total{operation="merge",result="error"} 1 +fanout_telemetry_operation_duration_seconds_bucket{operation="compaction",le="0.010"} 1 +fanout_telemetry_operation_duration_seconds_bucket{operation="compaction",le="0.100"} 2 +fanout_telemetry_operation_duration_seconds_bucket{operation="compaction",le="1.000"} 3 +fanout_telemetry_operation_duration_seconds_bucket{operation="compaction",le="+Inf"} 3 +fanout_telemetry_operation_duration_seconds_sum{operation="compaction"} 0.125 +fanout_telemetry_operation_duration_seconds_count{operation="compaction"} 3 +fanout_telemetry_operation_total{operation="compaction",result="success"} 3 +fanout_telemetry_operation_total{operation="compaction",result="throttled"} 5 +fanout_telemetry_operation_total{operation="compaction",result="error"} 1 fanout_rollup_enabled{rollup="service"} 1 fanout_rollup_watermark_timestamp_seconds{rollup="service"} 200 fanout_rollup_source_timestamp_seconds{rollup="service"} 212 @@ -141,9 +141,9 @@ fanout_rollup_component_total{rollup="service",result="noop"} 2 assertFloat(t, "ingest rows start", report.IngestRowsStart, 15) assertFloat(t, "ingest rows end", report.IngestRowsEnd, 45) assertFloat(t, "ingest rows", report.IngestRowsDelta, 30) - assertFloat(t, "lake partitions delta", report.LakePartitionsDelta, 2) - assertFloat(t, "lake size delta", report.LakeSizeBytesDelta, 80) - assertFloat(t, "lake growth rate", report.LakeGrowthBytesPerSec, 10) + assertFloat(t, "Parquet files delta", report.ParquetFilesDelta, 2) + assertFloat(t, "Parquet size delta", report.ParquetSizeBytesDelta, 80) + assertFloat(t, "Parquet growth rate", report.ParquetGrowthBytesPerSec, 10) assertFloat(t, "average rollup", report.AvgRollupMs, 750) assertFloat(t, "average flush", report.AvgFlushMs, 200) assertFloat(t, "average query", report.AvgQueryMs, 500) @@ -170,7 +170,7 @@ fanout_rollup_component_total{rollup="service",result="noop"} 2 assertFloat(t, "wait p95", wait.P95Ms, 100) assertFloat(t, "hold mean", report.WriteGateHoldMs["ingest_spans"].MeanMs, 15) - merge := report.DuckLakeOperations["merge"] + merge := report.TelemetryOperations["compaction"] if merge.DurationMs.Count != 2 { t.Fatalf("merge duration count = %d, want 2", merge.DurationMs.Count) } diff --git a/cmd/bench/verdict.go b/cmd/bench/verdict.go index 2168016d..0f069f27 100644 --- a/cmd/bench/verdict.go +++ b/cmd/bench/verdict.go @@ -85,9 +85,9 @@ func evaluateReport(cfg config, report report, infrastructureFailures []string) fails = append(fails, fmt.Sprintf("%s rollup errors=%.0f", name, errors)) } } - for _, name := range sortedMapKeys(report.Server.DuckLakeOperations) { - if errors := report.Server.DuckLakeOperations[name].Outcomes["error"]; errors > 0 { - fails = append(fails, fmt.Sprintf("ducklake %s errors=%.0f", name, errors)) + for _, name := range sortedMapKeys(report.Server.TelemetryOperations) { + if errors := report.Server.TelemetryOperations[name].Outcomes["error"]; errors > 0 { + fails = append(fails, fmt.Sprintf("telemetry %s errors=%.0f", name, errors)) } } } diff --git a/cmd/bench/verdict_test.go b/cmd/bench/verdict_test.go index cfa29c53..d67ef9ed 100644 --- a/cmd/bench/verdict_test.go +++ b/cmd/bench/verdict_test.go @@ -152,18 +152,18 @@ func TestEvaluateReportFailsOnBackgroundWorkErrors(t *testing.T) { "service": {Outcomes: map[string]float64{"success": 30}}, "edge": {Outcomes: map[string]float64{"error": 12}}, }, - DuckLakeOperations: map[string]backgroundOperationReport{ + TelemetryOperations: map[string]backgroundOperationReport{ "maintenance": {Outcomes: map[string]float64{"error": 3}}, - "merge": {Outcomes: map[string]float64{"success": 9}}, + "compaction": {Outcomes: map[string]float64{"success": 9}}, }, }, }, nil) - for _, want := range []string{"edge rollup errors=12", "ducklake maintenance errors=3"} { + for _, want := range []string{"edge rollup errors=12", "telemetry maintenance errors=3"} { if !hasFailure(failures, want) { t.Fatalf("failures %q do not contain %q", failures, want) } } - if hasFailure(failures, "service rollup") || hasFailure(failures, "ducklake merge") { + if hasFailure(failures, "service rollup") || hasFailure(failures, "telemetry merge") { t.Errorf("healthy components reported as failures: %q", failures) } } diff --git a/cmd/fanout/main.go b/cmd/fanout/main.go index dca92d8f..dfcb1aeb 100644 --- a/cmd/fanout/main.go +++ b/cmd/fanout/main.go @@ -35,13 +35,13 @@ import ( "github.com/labstack/fanout/internal/dashboard" "github.com/labstack/fanout/internal/ingest" "github.com/labstack/fanout/internal/intelligence" - "github.com/labstack/fanout/internal/lake" "github.com/labstack/fanout/internal/mcp" appmetrics "github.com/labstack/fanout/internal/metrics" "github.com/labstack/fanout/internal/observability" "github.com/labstack/fanout/internal/query" "github.com/labstack/fanout/internal/settings" "github.com/labstack/fanout/internal/store" + telemetrystore "github.com/labstack/fanout/internal/telemetry/store" "github.com/labstack/fanout/internal/ui" ) @@ -50,8 +50,13 @@ var tokenRedactRe = regexp.MustCompile(`token=[^&]+`) // version is set at build time via -ldflags "-X main.version=...". var version = "dev" +type repairCommand struct { + action string + batch string +} + func main() { - configPath, showVersion, loginEmail, healthURL, err := parseCommandLine(os.Args[1:], os.Stderr) + configPath, showVersion, loginEmail, healthURL, repair, err := parseCommandLine(os.Args[1:], os.Stderr) if errors.Is(err, flag.ErrHelp) { return } @@ -77,6 +82,13 @@ func main() { slog.Error("invalid configuration", "err", err) os.Exit(1) } + if repair != nil { + if err := runRepair(cfg.TelemetryDir(), *repair, os.Stdout); err != nil { + fmt.Fprintln(os.Stderr, "fanout repair:", err) + os.Exit(1) + } + return + } if loginEmail != "" { if err := createLoginLink(cfg, loginEmail, os.Stderr); err != nil { slog.Error("create login link failed", "err", err) @@ -95,11 +107,6 @@ func main() { os.Exit(1) } - // Channels for ingest → lake writer - chSpans := make(chan lake.SpanRow, 10000) - chLogs := make(chan lake.LogRow, 10000) - chMetrics := make(chan lake.MetricRow, 10000) - ctx, cancel := context.WithCancel(context.Background()) defer cancel() @@ -109,27 +116,31 @@ func main() { // Error channel for goroutine failures errCh := make(chan error, 4) - // Start DuckDB + rollups - q, err := query.NewDuck(ctx, cfg) + repository, err := telemetrystore.Open(cfg.TelemetryDir()) + if err != nil { + slog.Error("telemetry store init failed", "err", err) + os.Exit(1) + } + defer repository.Close() + + // DuckDB is the query facade over open Parquet and its indexed trace sidecars. + q, err := query.NewDuck(ctx, cfg, repository) if err != nil { slog.Error("duckdb init failed", "err", err) os.Exit(1) } defer q.Close() - // Start Lake Writer. Share the query layer's write gate so appender flushes - // serialize with rollup/maintenance commits when the pool holds >1 connection. - writer := lake.NewWriter(cfg, q.DB, chSpans, chLogs, chMetrics) - writer.UseWriteGate(q.WriteGate()) + writer := telemetrystore.NewWriter(repository, cfg.IngestBatchSize) writerResult := make(chan error, 1) go func() { err := writer.Run(ctx) // Publish the result before notifying the process-wide error channel. The - // shutdown path waits on writerResult after Writer.Wait, so a final-flush + // shutdown path waits on writerResult after Writer.Wait, so a final-commit // failure cannot be lost in the close(done) -> goroutine-send scheduling gap. writerResult <- err if err != nil { - errCh <- fmt.Errorf("lake writer: %w", err) + errCh <- fmt.Errorf("telemetry writer: %w", err) } }() @@ -194,7 +205,7 @@ func main() { os.Exit(1) } grpcSrv := grpc.NewServer(grpcOpts...) - ing := ingest.NewServer(cfg, chSpans, chLogs, chMetrics) + ing := ingest.NewServer(cfg, writer) ingest.RegisterOTLP(grpcSrv, ing) otlpHTTPLis, err := net.Listen("tcp", cfg.OTLPHTTPAddr) if err != nil { @@ -291,8 +302,8 @@ func main() { // Fanout owns telemetry semantics; agents and web clients consume this one // typed query kernel through deterministic HTTP or standard MCP tools. // Route both HTTP and MCP reads through Duck's retrying adapter. Passing the - // raw *sql.DB here bypassed the DuckLake maintenance-race protection. - queries := observability.New(q) + // raw *sql.DB here bypassed the Telemetry maintenance-race protection. + queries := observability.New(q, q) api.NewObservabilityHandler(queries).Register(e.Group("/api/observability", api.RequireCapability(api.ReadTelemetry))) api.RegisterIntelligenceRoutes(e, detector) dashboards := dashboard.New(sqlite.DB) @@ -452,20 +463,20 @@ func main() { cancel() writer.Wait() if err := <-writerResult; err != nil { - slog.Error("lake writer stopped with unwritten telemetry", "err", err) + slog.Error("telemetry writer stopped with unwritten telemetry", "err", err) } httpCancel() // triggers graceful HTTP shutdown (5s timeout) } -func parseCommandLine(args []string, output io.Writer) (configPath string, showVersion bool, loginEmail, healthURL string, err error) { +func parseCommandLine(args []string, output io.Writer) (configPath string, showVersion bool, loginEmail, healthURL string, repair *repairCommand, err error) { if len(args) == 1 && args[0] == "version" { - return "", true, "", "", nil + return "", true, "", "", nil, nil } if len(args) >= 1 && len(args) <= 2 && args[0] == "healthcheck" { if len(args) == 2 { - return "", false, "", args[1], nil + return "", false, "", args[1], nil, nil } - return "", false, "", "http://127.0.0.1:7520/healthz", nil + return "", false, "", "http://127.0.0.1:7520/healthz", nil, nil } flags := flag.NewFlagSet("fanout", flag.ContinueOnError) @@ -474,6 +485,8 @@ func parseCommandLine(args []string, output io.Writer) (configPath string, showV fmt.Fprintln(output, "Usage of fanout:") fmt.Fprintln(output, " fanout [flags]") fmt.Fprintln(output, " fanout [--config path] login-link ") + fmt.Fprintln(output, " fanout [--config path] repair verify") + fmt.Fprintln(output, " fanout [--config path] repair quarantine --batch ") fmt.Fprintln(output, " fanout healthcheck [url]") fmt.Fprintln(output, " fanout version") fmt.Fprintln(output, "Flags:") @@ -483,18 +496,71 @@ func parseCommandLine(args []string, output io.Writer) (configPath string, showV flags.BoolVar(&showVersion, "version", false, "print the Fanout version") flags.BoolVar(&showVersion, "v", false, "print the Fanout version") if err := flags.Parse(args); err != nil { - return "", false, "", "", err + return "", false, "", "", nil, err } if flags.NArg() == 2 && flags.Arg(0) == "login-link" { - return configPath, false, flags.Arg(1), "", nil + return configPath, false, flags.Arg(1), "", nil, nil + } + if flags.NArg() > 0 && flags.Arg(0) == "repair" { + repair, err := parseRepairCommand(flags.Args()[1:], output) + return configPath, false, "", "", repair, err } if flags.NArg() != 0 { err := fmt.Errorf("unexpected arguments: %s", strings.Join(flags.Args(), " ")) fmt.Fprintln(output, err) flags.Usage() - return "", false, "", "", err + return "", false, "", "", nil, err + } + return configPath, showVersion, "", "", nil, nil +} + +func parseRepairCommand(args []string, output io.Writer) (*repairCommand, error) { + if len(args) == 1 && args[0] == "verify" { + return &repairCommand{action: "verify"}, nil + } + if len(args) == 0 || args[0] != "quarantine" { + return nil, errors.New("repair requires 'verify' or 'quarantine --batch '") + } + flags := flag.NewFlagSet("fanout repair quarantine", flag.ContinueOnError) + flags.SetOutput(output) + batch := flags.String("batch", "", "unreadable authoritative batch ID to set aside") + if err := flags.Parse(args[1:]); err != nil { + return nil, err + } + if strings.TrimSpace(*batch) == "" || flags.NArg() != 0 { + return nil, errors.New("repair quarantine requires exactly --batch ") + } + return &repairCommand{action: "quarantine", batch: *batch}, nil +} + +func runRepair(root string, command repairCommand, output io.Writer) error { + switch command.action { + case "verify": + issues, err := telemetrystore.VerifyBatches(root) + if err != nil { + return err + } + if len(issues) == 0 { + fmt.Fprintln(output, "all authoritative telemetry batches passed validation") + return nil + } + for _, issue := range issues { + fmt.Fprintf(output, "%s: %v\n", issue.ID, issue.Err) + } + return fmt.Errorf("%d unreadable authoritative telemetry batch(es)", len(issues)) + case "quarantine": + destination, err := telemetrystore.QuarantineBatch(root, command.batch) + if destination != "" { + fmt.Fprintf(output, "batch %s set aside at %s\n", command.batch, destination) + } + if err != nil { + return err + } + fmt.Fprintln(output, "the quarantined telemetry is no longer queryable; preserve it for recovery from backup") + return nil + default: + return fmt.Errorf("unknown repair action %q", command.action) } - return configPath, showVersion, "", "", nil } func checkHealth(healthURL string) error { diff --git a/cmd/fanout/main_test.go b/cmd/fanout/main_test.go index 70e64387..5e0482e6 100644 --- a/cmd/fanout/main_test.go +++ b/cmd/fanout/main_test.go @@ -2,6 +2,7 @@ package main import ( "bytes" + "context" "errors" "flag" "io" @@ -9,12 +10,15 @@ import ( "net/http/httptest" "net/url" "os" + "path/filepath" "strings" "testing" "github.com/labstack/fanout/internal/auth" "github.com/labstack/fanout/internal/config" "github.com/labstack/fanout/internal/store" + "github.com/labstack/fanout/internal/telemetry" + telemetrystore "github.com/labstack/fanout/internal/telemetry/store" ) func TestParseCommandLine(t *testing.T) { @@ -25,6 +29,7 @@ func TestParseCommandLine(t *testing.T) { wantVersion bool wantEmail string wantHealth string + wantRepair *repairCommand wantErr bool }{ {name: "server defaults", args: nil}, @@ -36,25 +41,32 @@ func TestParseCommandLine(t *testing.T) { {name: "login link with config", args: []string{"--config", "/etc/fanout.yaml", "login-link", "admin@example.com"}, wantPath: "/etc/fanout.yaml", wantEmail: "admin@example.com"}, {name: "default healthcheck", args: []string{"healthcheck"}, wantHealth: "http://127.0.0.1:7520/healthz"}, {name: "custom healthcheck", args: []string{"healthcheck", "http://fanout:8080/healthz"}, wantHealth: "http://fanout:8080/healthz"}, + {name: "repair verify", args: []string{"repair", "verify"}, wantRepair: &repairCommand{action: "verify"}}, + {name: "repair quarantine with config", args: []string{"--config", "/etc/fanout.yaml", "repair", "quarantine", "--batch", "broken"}, wantPath: "/etc/fanout.yaml", wantRepair: &repairCommand{action: "quarantine", batch: "broken"}}, + {name: "repair quarantine missing batch", args: []string{"repair", "quarantine"}, wantErr: true}, + {name: "unknown repair action", args: []string{"repair", "delete"}, wantErr: true}, {name: "login link missing email", args: []string{"login-link"}, wantErr: true}, {name: "unexpected argument", args: []string{"serve"}, wantErr: true}, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { - path, showVersion, email, healthURL, err := parseCommandLine(test.args, io.Discard) + path, showVersion, email, healthURL, repair, err := parseCommandLine(test.args, io.Discard) if (err != nil) != test.wantErr { t.Fatalf("error = %v, wantErr %v", err, test.wantErr) } if path != test.wantPath || showVersion != test.wantVersion || email != test.wantEmail || healthURL != test.wantHealth { t.Fatalf("result = (%q, %v, %q, %q), want (%q, %v, %q, %q)", path, showVersion, email, healthURL, test.wantPath, test.wantVersion, test.wantEmail, test.wantHealth) } + if repair == nil != (test.wantRepair == nil) || repair != nil && *repair != *test.wantRepair { + t.Fatalf("repair = %#v, want %#v", repair, test.wantRepair) + } }) } } func TestParseCommandLineHelp(t *testing.T) { - _, _, _, _, err := parseCommandLine([]string{"--help"}, io.Discard) + _, _, _, _, _, err := parseCommandLine([]string{"--help"}, io.Discard) if !errors.Is(err, flag.ErrHelp) { t.Fatalf("error = %v, want flag.ErrHelp", err) } @@ -62,7 +74,7 @@ func TestParseCommandLineHelp(t *testing.T) { func TestParseCommandLinePrintsOneErrorAndUsage(t *testing.T) { var output bytes.Buffer - _, _, _, _, err := parseCommandLine([]string{"serve"}, &output) + _, _, _, _, _, err := parseCommandLine([]string{"serve"}, &output) if err == nil { t.Fatal("expected unexpected-argument error") } @@ -78,6 +90,9 @@ func TestParseCommandLinePrintsOneErrorAndUsage(t *testing.T) { if !strings.Contains(output.String(), "healthcheck [url]") { t.Fatalf("output does not include healthcheck command: %q", output.String()) } + if !strings.Contains(output.String(), "repair quarantine --batch ") { + t.Fatalf("output does not include repair command: %q", output.String()) + } } func TestCheckHealth(t *testing.T) { @@ -102,6 +117,41 @@ func TestCheckHealth(t *testing.T) { } } +func TestRunRepairVerifiesAndQuarantinesUnreadableBatch(t *testing.T) { + root := t.TempDir() + repository, err := telemetrystore.Open(root) + if err != nil { + t.Fatal(err) + } + if err := repository.Commit(context.Background(), telemetrystore.Batch{ + ID: "broken", Spans: []telemetry.Span{{TraceID: "trace", SpanID: "span"}}, + }); err != nil { + t.Fatal(err) + } + if err := repository.Close(); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "parquet", "batches", "broken.batch", "trace.fidx"), []byte("bad"), 0o644); err != nil { + t.Fatal(err) + } + + var output bytes.Buffer + if err := runRepair(root, repairCommand{action: "verify"}, &output); err == nil || !strings.Contains(output.String(), "broken:") { + t.Fatalf("verify = %v, output %q", err, output.String()) + } + output.Reset() + if err := runRepair(root, repairCommand{action: "quarantine", batch: "broken"}, &output); err != nil { + t.Fatal(err) + } + if !strings.Contains(output.String(), "no longer queryable") { + t.Fatalf("quarantine output = %q", output.String()) + } + output.Reset() + if err := runRepair(root, repairCommand{action: "verify"}, &output); err != nil { + t.Fatal(err) + } +} + func TestCreateLoginLink(t *testing.T) { cfg := config.Config{ AuthMode: "local", diff --git a/docs/diagrams/architecture.d2 b/docs/diagrams/architecture.d2 index 7932131b..d2b34aaa 100644 --- a/docs/diagrams/architecture.d2 +++ b/docs/diagrams/architecture.d2 @@ -26,12 +26,12 @@ fanout: "fanout — one Go process" { agent: "Agent runtime\nmodel + tool loop" mcp: "MCP server\n5 typed + 4 dashboard tools" obs: "Typed observability contract" - lake: "Lake writer\nbatched flush" + commit: "Telemetry commit workers\natomic Parquet batches" query: "Query kernel\nDuckDB + rollups" - gate: "Write gate\none catalog write in flight" {style.stroke-width: 2} + maint: "Storage maintenance\nretention + native compaction" alert: "Alert engine\nrule evaluation + webhooks" - ingest -> lake + ingest -> commit http -> ui: serves embedded assets http -> agent: AG-UI stream http -> mcp @@ -40,13 +40,11 @@ fanout: "fanout — one Go process" { http -> obs: typed HTTP API obs -> query alert -> query: evaluates rollups - lake -> gate - query -> gate: "rollups, merge, maintenance" } store: Storage { style.fill: transparent - telemetry: "DuckLake + Parquet\nstorage.data_dir/telemetry" {shape: cylinder} + telemetry: "Parquet batches + trace indexes\nstorage.data_dir/telemetry" {shape: cylinder} qstate: "Query catalog\nstorage.data_dir/query" {shape: cylinder} control: "Control SQLite\nstorage.data_dir/control" {shape: cylinder} } @@ -57,8 +55,10 @@ clients.collector -> fanout.ingest: "OTLP/gRPC or OTLP/HTTP" clients.browser -> fanout.http: HTTPS clients.ext -> fanout.http: "/mcp — OAuth" -fanout.gate -> store.telemetry +fanout.commit -> store.telemetry +fanout.query -> store.telemetry: "indexed traces + SQL scans" fanout.query -> store.qstate +fanout.maint -> store.telemetry: "atomic replacement" fanout.http -> store.control: "users, sessions, settings, dashboards, threads" fanout.alert -> store.control: "rules and fired alerts" fanout.agent -> model: HTTPS diff --git a/docs/diagrams/architecture.svg b/docs/diagrams/architecture.svg index 4e266d4f..6f054670 100644 --- a/docs/diagrams/architecture.svg +++ b/docs/diagrams/architecture.svg @@ -1,24 +1,24 @@ -Clientsfanout — one Go processStorageAnthropic or OpenAIOTLP collector or SDKBrowserExternal MCP hostHTTP :7520routes + auth middlewareIngestOTLP gRPC :4317 + HTTP :4318Embedded React assetsAgent runtimemodel + tool loopMCP server5 typed + 4 dashboard toolsTyped observability contractLake writerbatched flushQuery kernelDuckDB + rollupsWrite gateone catalog write in flightAlert enginerule evaluation + webhooksDuckLake + Parquetstorage.data_dir/telemetryQuery catalogstorage.data_dir/queryControl SQLitestorage.data_dir/control serves embedded assetsAG-UI streamin-memory transporttyped HTTP APIevaluates rollupsrollups, merge, maintenanceOTLP/gRPC or OTLP/HTTPHTTPS/mcp — OAuthusers, sessions, settings, dashboards, threadsrules and fired alertsHTTPS - - - - - - - + .d2-634443295 .fill-N1{fill:#0A0F25;} + .d2-634443295 .fill-N2{fill:#676C7E;} + .d2-634443295 .fill-N3{fill:#9499AB;} + .d2-634443295 .fill-N4{fill:#CFD2DD;} + .d2-634443295 .fill-N5{fill:#DEE1EB;} + .d2-634443295 .fill-N6{fill:#EEF1F8;} + .d2-634443295 .fill-N7{fill:#FFFFFF;} + .d2-634443295 .fill-B1{fill:#000536;} + .d2-634443295 .fill-B2{fill:#0F66B7;} + .d2-634443295 .fill-B3{fill:#4393DD;} + .d2-634443295 .fill-B4{fill:#87BFF3;} + .d2-634443295 .fill-B5{fill:#BCDDFB;} + .d2-634443295 .fill-B6{fill:#E5F3FF;} + .d2-634443295 .fill-AA2{fill:#7639C5;} + .d2-634443295 .fill-AA4{fill:#C1A2F3;} + .d2-634443295 .fill-AA5{fill:#DACEFB;} + .d2-634443295 .fill-AB4{fill:#EA99C6;} + .d2-634443295 .fill-AB5{fill:#FFDEF1;} + .d2-634443295 .stroke-N1{stroke:#0A0F25;} + .d2-634443295 .stroke-N2{stroke:#676C7E;} + .d2-634443295 .stroke-N3{stroke:#9499AB;} + .d2-634443295 .stroke-N4{stroke:#CFD2DD;} + .d2-634443295 .stroke-N5{stroke:#DEE1EB;} + .d2-634443295 .stroke-N6{stroke:#EEF1F8;} + .d2-634443295 .stroke-N7{stroke:#FFFFFF;} + .d2-634443295 .stroke-B1{stroke:#000536;} + .d2-634443295 .stroke-B2{stroke:#0F66B7;} + .d2-634443295 .stroke-B3{stroke:#4393DD;} + .d2-634443295 .stroke-B4{stroke:#87BFF3;} + .d2-634443295 .stroke-B5{stroke:#BCDDFB;} + .d2-634443295 .stroke-B6{stroke:#E5F3FF;} + .d2-634443295 .stroke-AA2{stroke:#7639C5;} + .d2-634443295 .stroke-AA4{stroke:#C1A2F3;} + .d2-634443295 .stroke-AA5{stroke:#DACEFB;} + .d2-634443295 .stroke-AB4{stroke:#EA99C6;} + .d2-634443295 .stroke-AB5{stroke:#FFDEF1;} + .d2-634443295 .background-color-N1{background-color:#0A0F25;} + .d2-634443295 .background-color-N2{background-color:#676C7E;} + .d2-634443295 .background-color-N3{background-color:#9499AB;} + .d2-634443295 .background-color-N4{background-color:#CFD2DD;} + .d2-634443295 .background-color-N5{background-color:#DEE1EB;} + .d2-634443295 .background-color-N6{background-color:#EEF1F8;} + .d2-634443295 .background-color-N7{background-color:#FFFFFF;} + .d2-634443295 .background-color-B1{background-color:#000536;} + .d2-634443295 .background-color-B2{background-color:#0F66B7;} + .d2-634443295 .background-color-B3{background-color:#4393DD;} + .d2-634443295 .background-color-B4{background-color:#87BFF3;} + .d2-634443295 .background-color-B5{background-color:#BCDDFB;} + .d2-634443295 .background-color-B6{background-color:#E5F3FF;} + .d2-634443295 .background-color-AA2{background-color:#7639C5;} + .d2-634443295 .background-color-AA4{background-color:#C1A2F3;} + .d2-634443295 .background-color-AA5{background-color:#DACEFB;} + .d2-634443295 .background-color-AB4{background-color:#EA99C6;} + .d2-634443295 .background-color-AB5{background-color:#FFDEF1;} + .d2-634443295 .color-N1{color:#0A0F25;} + .d2-634443295 .color-N2{color:#676C7E;} + .d2-634443295 .color-N3{color:#9499AB;} + .d2-634443295 .color-N4{color:#CFD2DD;} + .d2-634443295 .color-N5{color:#DEE1EB;} + .d2-634443295 .color-N6{color:#EEF1F8;} + .d2-634443295 .color-N7{color:#FFFFFF;} + .d2-634443295 .color-B1{color:#000536;} + .d2-634443295 .color-B2{color:#0F66B7;} + .d2-634443295 .color-B3{color:#4393DD;} + .d2-634443295 .color-B4{color:#87BFF3;} + .d2-634443295 .color-B5{color:#BCDDFB;} + .d2-634443295 .color-B6{color:#E5F3FF;} + .d2-634443295 .color-AA2{color:#7639C5;} + .d2-634443295 .color-AA4{color:#C1A2F3;} + .d2-634443295 .color-AA5{color:#DACEFB;} + .d2-634443295 .color-AB4{color:#EA99C6;} + .d2-634443295 .color-AB5{color:#FFDEF1;}.appendix text.text{fill:#0A0F25}.md{--color-fg-default:#0A0F25;--color-fg-muted:#676C7E;--color-fg-subtle:#9499AB;--color-canvas-default:#FFFFFF;--color-canvas-subtle:#EEF1F8;--color-border-default:#000536;--color-border-muted:#0F66B7;--color-neutral-muted:#EEF1F8;--color-accent-fg:#0F66B7;--color-accent-emphasis:#0F66B7;--color-attention-subtle:#676C7E;--color-danger-fg:red;}.sketch-overlay-B1{fill:url(#streaks-darker-d2-634443295);mix-blend-mode:lighten}.sketch-overlay-B2{fill:url(#streaks-dark-d2-634443295);mix-blend-mode:overlay}.sketch-overlay-B3{fill:url(#streaks-dark-d2-634443295);mix-blend-mode:overlay}.sketch-overlay-B4{fill:url(#streaks-normal-d2-634443295);mix-blend-mode:color-burn}.sketch-overlay-B5{fill:url(#streaks-normal-d2-634443295);mix-blend-mode:color-burn}.sketch-overlay-B6{fill:url(#streaks-bright-d2-634443295);mix-blend-mode:darken}.sketch-overlay-AA2{fill:url(#streaks-dark-d2-634443295);mix-blend-mode:overlay}.sketch-overlay-AA4{fill:url(#streaks-normal-d2-634443295);mix-blend-mode:color-burn}.sketch-overlay-AA5{fill:url(#streaks-normal-d2-634443295);mix-blend-mode:color-burn}.sketch-overlay-AB4{fill:url(#streaks-normal-d2-634443295);mix-blend-mode:color-burn}.sketch-overlay-AB5{fill:url(#streaks-bright-d2-634443295);mix-blend-mode:darken}.sketch-overlay-N1{fill:url(#streaks-darker-d2-634443295);mix-blend-mode:lighten}.sketch-overlay-N2{fill:url(#streaks-dark-d2-634443295);mix-blend-mode:overlay}.sketch-overlay-N3{fill:url(#streaks-normal-d2-634443295);mix-blend-mode:color-burn}.sketch-overlay-N4{fill:url(#streaks-normal-d2-634443295);mix-blend-mode:color-burn}.sketch-overlay-N5{fill:url(#streaks-bright-d2-634443295);mix-blend-mode:darken}.sketch-overlay-N6{fill:url(#streaks-bright-d2-634443295);mix-blend-mode:darken}.sketch-overlay-N7{fill:url(#streaks-bright-d2-634443295);mix-blend-mode:darken}.light-code{display: block}.dark-code{display: none}]]>Clientsfanout — one Go processStorageAnthropic or OpenAIOTLP collector or SDKBrowserExternal MCP hostHTTP :7520routes + auth middlewareIngestOTLP gRPC :4317 + HTTP :4318Embedded React assetsAgent runtimemodel + tool loopMCP server5 typed + 4 dashboard toolsTyped observability contractTelemetry commit workersatomic Parquet batchesQuery kernelDuckDB + rollupsStorage maintenanceretention + native compactionAlert enginerule evaluation + webhooksParquet batches + trace indexesstorage.data_dir/telemetryQuery catalogstorage.data_dir/queryControl SQLitestorage.data_dir/control serves embedded assetsAG-UI streamin-memory transporttyped HTTP APIevaluates rollupsOTLP/gRPC or OTLP/HTTPHTTPS/mcp — OAuthindexed traces + SQL scansatomic replacementusers, sessions, settings, dashboards, threadsrules and fired alertsHTTPS + + + + + + - - - - - + + + + + + + diff --git a/docs/diagrams/persistence.d2 b/docs/diagrams/persistence.d2 index 2345f6cb..33472831 100644 --- a/docs/diagrams/persistence.d2 +++ b/docs/diagrams/persistence.d2 @@ -1,4 +1,4 @@ -# Where Fanout keeps state, and what serializes writes to it. +# How telemetry becomes durable, queryable, and compacted. # Render with `just diagrams`; commit the generated SVG beside this file. direction: down @@ -10,44 +10,54 @@ vars: { } } -writers: Writers { - style.fill: transparent - ingest: "Ingest flush\nspans, logs, metrics" {shape: rectangle} - rollup: "Rollups\nservice, endpoint, edge" {shape: rectangle} - maint: "Merge and maintenance" {shape: rectangle} +ingest: "Authenticated OTLP requests\nspans, logs, metrics" { + shape: rectangle + style.stroke-dash: 3 } -gate: "Write gate — internal/lake/writegate\none catalog write in flight at a time" { - shape: rectangle - style: { - stroke-width: 2 - bold: true - } +writers: "Durable write path" { + style.fill: transparent + admission: "Bounded group admission\nconfigured row target + 20 ms window" {shape: rectangle} + workers: "Up to four commit workers\nindependent batches encode in parallel" {shape: rectangle} + stage: "Staging batch directory\nmetadata + signal Parquet + trace.fidx for spans" {shape: rectangle} + publish: "Durable publication\nfsync → atomic rename → fsync parent" {shape: rectangle} + + admission -> workers + workers -> stage + stage -> publish } -telemetry: "DuckLake catalog + Parquet\nstorage.data_dir/telemetry" { +authority: "Authoritative telemetry\nimmutable *.batch directories" { shape: cylinder - tooltip: Partitioned telemetry. Written only through the gate. + tooltip: Ordinary Parquet files are the source of truth; the filesystem is the catalog. } -querystate: "DuckDB query state\nstorage.data_dir/query" { - shape: cylinder - tooltip: Catalog attachment and temp spill. Not the telemetry itself. +readers: Readers { + style.fill: transparent + trace: "Indexed trace read\nbinary-search .fidx → Parquet rows" {shape: rectangle} + sql: "DuckDB SQL\nfilters, ordering, broad scans" {shape: rectangle} + rollups: "DuckDB rollups\nrebuildable analytical cache" {shape: rectangle} } -control: "Control SQLite\nstorage.data_dir/control/fanout.sqlite" { - shape: cylinder - tooltip: Application state. Never on the telemetry write path. +maintenance: "Bounded maintenance" { + style.fill: transparent + choose: "Retention + compaction admission\nbounded batch and row groups" {shape: rectangle} + merge: "Native Parquet merge\nprepared outside the reader gate" {shape: rectangle} + swap: "Crash-safe namespace swap\nmarker + bounded renames + fsync" {shape: rectangle} + + choose -> merge + merge -> swap } -control_tables: "users, user_identities, verifications, sessions, auth_audit_events\noauth_clients, oauth_tokens, oauth_authorization_codes\ndashboards, dashboard_widgets, dashboard_state\nagui_threads, agui_runs, alert_rules, alerts, settings" { - shape: text - style.font-size: 13 +querystate: "Rebuildable query state\nstorage.data_dir/query" { + shape: cylinder + tooltip: DuckDB rollups and temporary spill. Not authoritative telemetry. } -writers.ingest -> gate -writers.rollup -> gate -writers.maint -> gate -gate -> telemetry: "wait and hold measured per operation" -telemetry <- querystate: attached -control -> control_tables: {style.stroke-dash: 3} +ingest -> writers.admission +writers.publish -> authority: "acknowledge only after publication" +authority -> readers.trace +authority -> readers.sql +readers.sql -> readers.rollups +readers.rollups -> querystate +authority -> maintenance.choose diff --git a/docs/diagrams/persistence.svg b/docs/diagrams/persistence.svg index 9ccf978c..e1c636dc 100644 --- a/docs/diagrams/persistence.svg +++ b/docs/diagrams/persistence.svg @@ -1,27 +1,27 @@ -WritersWrite gate — internal/lake/writegateone catalog write in flight at a timeDuckLake catalog + Parquetstorage.data_dir/telemetryPartitioned telemetry. Written only through the gate.DuckDB query statestorage.data_dir/queryCatalog attachment and temp spill. Not the telemetry itself.Control SQLitestorage.data_dir/control/fanout.sqliteApplication state. Never on the telemetry write path.users, user_identities, verifications, sessions, auth_audit_eventsoauth_clients, oauth_tokens, oauth_authorization_codesdashboards, dashboard_widgets, dashboard_stateagui_threads, agui_runs, alert_rules, alerts, settingsIngest flushspans, logs, metricsRollupsservice, endpoint, edgeMerge and maintenance wait and hold measured per operation attached Partitioned telemetry. Written only through the gate. - + .d2-2487203756 .fill-N1{fill:#0A0F25;} + .d2-2487203756 .fill-N2{fill:#676C7E;} + .d2-2487203756 .fill-N3{fill:#9499AB;} + .d2-2487203756 .fill-N4{fill:#CFD2DD;} + .d2-2487203756 .fill-N5{fill:#DEE1EB;} + .d2-2487203756 .fill-N6{fill:#EEF1F8;} + .d2-2487203756 .fill-N7{fill:#FFFFFF;} + .d2-2487203756 .fill-B1{fill:#000536;} + .d2-2487203756 .fill-B2{fill:#0F66B7;} + .d2-2487203756 .fill-B3{fill:#4393DD;} + .d2-2487203756 .fill-B4{fill:#87BFF3;} + .d2-2487203756 .fill-B5{fill:#BCDDFB;} + .d2-2487203756 .fill-B6{fill:#E5F3FF;} + .d2-2487203756 .fill-AA2{fill:#7639C5;} + .d2-2487203756 .fill-AA4{fill:#C1A2F3;} + .d2-2487203756 .fill-AA5{fill:#DACEFB;} + .d2-2487203756 .fill-AB4{fill:#EA99C6;} + .d2-2487203756 .fill-AB5{fill:#FFDEF1;} + .d2-2487203756 .stroke-N1{stroke:#0A0F25;} + .d2-2487203756 .stroke-N2{stroke:#676C7E;} + .d2-2487203756 .stroke-N3{stroke:#9499AB;} + .d2-2487203756 .stroke-N4{stroke:#CFD2DD;} + .d2-2487203756 .stroke-N5{stroke:#DEE1EB;} + .d2-2487203756 .stroke-N6{stroke:#EEF1F8;} + .d2-2487203756 .stroke-N7{stroke:#FFFFFF;} + .d2-2487203756 .stroke-B1{stroke:#000536;} + .d2-2487203756 .stroke-B2{stroke:#0F66B7;} + .d2-2487203756 .stroke-B3{stroke:#4393DD;} + .d2-2487203756 .stroke-B4{stroke:#87BFF3;} + .d2-2487203756 .stroke-B5{stroke:#BCDDFB;} + .d2-2487203756 .stroke-B6{stroke:#E5F3FF;} + .d2-2487203756 .stroke-AA2{stroke:#7639C5;} + .d2-2487203756 .stroke-AA4{stroke:#C1A2F3;} + .d2-2487203756 .stroke-AA5{stroke:#DACEFB;} + .d2-2487203756 .stroke-AB4{stroke:#EA99C6;} + .d2-2487203756 .stroke-AB5{stroke:#FFDEF1;} + .d2-2487203756 .background-color-N1{background-color:#0A0F25;} + .d2-2487203756 .background-color-N2{background-color:#676C7E;} + .d2-2487203756 .background-color-N3{background-color:#9499AB;} + .d2-2487203756 .background-color-N4{background-color:#CFD2DD;} + .d2-2487203756 .background-color-N5{background-color:#DEE1EB;} + .d2-2487203756 .background-color-N6{background-color:#EEF1F8;} + .d2-2487203756 .background-color-N7{background-color:#FFFFFF;} + .d2-2487203756 .background-color-B1{background-color:#000536;} + .d2-2487203756 .background-color-B2{background-color:#0F66B7;} + .d2-2487203756 .background-color-B3{background-color:#4393DD;} + .d2-2487203756 .background-color-B4{background-color:#87BFF3;} + .d2-2487203756 .background-color-B5{background-color:#BCDDFB;} + .d2-2487203756 .background-color-B6{background-color:#E5F3FF;} + .d2-2487203756 .background-color-AA2{background-color:#7639C5;} + .d2-2487203756 .background-color-AA4{background-color:#C1A2F3;} + .d2-2487203756 .background-color-AA5{background-color:#DACEFB;} + .d2-2487203756 .background-color-AB4{background-color:#EA99C6;} + .d2-2487203756 .background-color-AB5{background-color:#FFDEF1;} + .d2-2487203756 .color-N1{color:#0A0F25;} + .d2-2487203756 .color-N2{color:#676C7E;} + .d2-2487203756 .color-N3{color:#9499AB;} + .d2-2487203756 .color-N4{color:#CFD2DD;} + .d2-2487203756 .color-N5{color:#DEE1EB;} + .d2-2487203756 .color-N6{color:#EEF1F8;} + .d2-2487203756 .color-N7{color:#FFFFFF;} + .d2-2487203756 .color-B1{color:#000536;} + .d2-2487203756 .color-B2{color:#0F66B7;} + .d2-2487203756 .color-B3{color:#4393DD;} + .d2-2487203756 .color-B4{color:#87BFF3;} + .d2-2487203756 .color-B5{color:#BCDDFB;} + .d2-2487203756 .color-B6{color:#E5F3FF;} + .d2-2487203756 .color-AA2{color:#7639C5;} + .d2-2487203756 .color-AA4{color:#C1A2F3;} + .d2-2487203756 .color-AA5{color:#DACEFB;} + .d2-2487203756 .color-AB4{color:#EA99C6;} + .d2-2487203756 .color-AB5{color:#FFDEF1;}.appendix text.text{fill:#0A0F25}.md{--color-fg-default:#0A0F25;--color-fg-muted:#676C7E;--color-fg-subtle:#9499AB;--color-canvas-default:#FFFFFF;--color-canvas-subtle:#EEF1F8;--color-border-default:#000536;--color-border-muted:#0F66B7;--color-neutral-muted:#EEF1F8;--color-accent-fg:#0F66B7;--color-accent-emphasis:#0F66B7;--color-attention-subtle:#676C7E;--color-danger-fg:red;}.sketch-overlay-B1{fill:url(#streaks-darker-d2-2487203756);mix-blend-mode:lighten}.sketch-overlay-B2{fill:url(#streaks-dark-d2-2487203756);mix-blend-mode:overlay}.sketch-overlay-B3{fill:url(#streaks-dark-d2-2487203756);mix-blend-mode:overlay}.sketch-overlay-B4{fill:url(#streaks-normal-d2-2487203756);mix-blend-mode:color-burn}.sketch-overlay-B5{fill:url(#streaks-normal-d2-2487203756);mix-blend-mode:color-burn}.sketch-overlay-B6{fill:url(#streaks-bright-d2-2487203756);mix-blend-mode:darken}.sketch-overlay-AA2{fill:url(#streaks-dark-d2-2487203756);mix-blend-mode:overlay}.sketch-overlay-AA4{fill:url(#streaks-normal-d2-2487203756);mix-blend-mode:color-burn}.sketch-overlay-AA5{fill:url(#streaks-normal-d2-2487203756);mix-blend-mode:color-burn}.sketch-overlay-AB4{fill:url(#streaks-normal-d2-2487203756);mix-blend-mode:color-burn}.sketch-overlay-AB5{fill:url(#streaks-bright-d2-2487203756);mix-blend-mode:darken}.sketch-overlay-N1{fill:url(#streaks-darker-d2-2487203756);mix-blend-mode:lighten}.sketch-overlay-N2{fill:url(#streaks-dark-d2-2487203756);mix-blend-mode:overlay}.sketch-overlay-N3{fill:url(#streaks-normal-d2-2487203756);mix-blend-mode:color-burn}.sketch-overlay-N4{fill:url(#streaks-normal-d2-2487203756);mix-blend-mode:color-burn}.sketch-overlay-N5{fill:url(#streaks-bright-d2-2487203756);mix-blend-mode:darken}.sketch-overlay-N6{fill:url(#streaks-bright-d2-2487203756);mix-blend-mode:darken}.sketch-overlay-N7{fill:url(#streaks-bright-d2-2487203756);mix-blend-mode:darken}.light-code{display: block}.dark-code{display: none}]]>Authenticated OTLP requestsspans, logs, metricsDurable write pathAuthoritative telemetryimmutable *.batch directoriesOrdinary Parquet files are the source of truthReadersBounded maintenanceRebuildable query statestorage.data_dir/queryDuckDB rollups and temporary spill. Not authoritative telemetry.Bounded group admissionconfigured row target + 20 ms windowUp to four commit workersindependent batches encode in parallelStaging batch directorymetadata + signal Parquet + trace.fidx for spansDurable publicationfsync → atomic rename → fsync parentthe filesystem is the catalogIndexed trace readbinary-search .fidx → Parquet rowsDuckDB SQLfilters, ordering, broad scansDuckDB rollupsrebuildable analytical cacheRetention + compaction admissionbounded batch and row groupsNative Parquet mergeprepared outside the reader gateCrash-safe namespace swapmarker + bounded renames + fsync acknowledge only after publicationOrdinary Parquet files are the source of truth + - + -Catalog attachment and temp spill. Not the telemetry itself. - +DuckDB rollups and temporary spill. Not authoritative telemetry. + - + -Application state. Never on the telemetry write path. - - - - - - - - - - - - - - - - + + + diff --git a/docs/operations.md b/docs/operations.md index 79a6c80a..49599efd 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -67,14 +67,14 @@ its configured cycle rather than immediately when the setting changes. ## Backup The supported portable baseline is a **cold backup** of the complete -`FANOUT_DATA_DIR`. It contains the telemetry catalog and Parquet files, query -state, and the control SQLite database; copying only one subdirectory does not -produce a recoverable installation. +`FANOUT_DATA_DIR`. It contains atomic telemetry Parquet batches, query state, +and the control SQLite database; copying only one +subdirectory does not produce a recoverable installation. 1. Record the running Fanout version and configuration, excluding secrets from ordinary logs or tickets. 2. Stop Fanout cleanly and wait for the process to exit. Shutdown stops both - OTLP listeners before draining the lake writer. + OTLP listeners before draining the telemetry commit workers. 3. Snapshot or copy the complete data directory with ownership and permissions preserved. 4. Restart Fanout and confirm `/readyz`. diff --git a/fanout.example.yaml b/fanout.example.yaml index 029df643..c8c6eb17 100644 --- a/fanout.example.yaml +++ b/fanout.example.yaml @@ -21,8 +21,7 @@ ingest: otlp_http_addr: "127.0.0.1:4318" # FANOUT_OTLP_HTTP_ADDR advertised_endpoint: "" # FANOUT_INGEST_ADVERTISED_ENDPOINT default_namespace: default # FANOUT_DEFAULT_NAMESPACE - flush_interval: 15s # FANOUT_FLUSH_INTERVAL - flush_batch_size: 50000 # FANOUT_FLUSH_BATCH_SIZE + batch_size: 50000 # FANOUT_INGEST_BATCH_SIZE storage: data_dir: ./data # FANOUT_DATA_DIR @@ -30,7 +29,6 @@ storage: rollup_interval: 1m # FANOUT_ROLLUP_INTERVAL rollup_skip_to_latest: false # FANOUT_ROLLUP_SKIP_TO_LATEST maintenance_interval: 1h # FANOUT_MAINTENANCE_INTERVAL - merge_interval: 1m # FANOUT_MERGE_INTERVAL duckdb: # Empty/zero values let Fanout size these from the machine. memory: "" # FANOUT_DUCKDB_MEMORY diff --git a/go.mod b/go.mod index 7a011b13..d81ba590 100644 --- a/go.mod +++ b/go.mod @@ -19,11 +19,14 @@ require ( github.com/knadh/koanf/v2 v2.3.6 github.com/labstack/echo/v5 v5.3.1 github.com/modelcontextprotocol/go-sdk v1.7.0 + github.com/parquet-go/parquet-go v0.32.0 github.com/prometheus/client_golang v1.24.1 github.com/prometheus/client_model v0.6.2 github.com/wneessen/go-mail v0.8.1 + github.com/zeebo/xxh3 v1.1.0 go.opentelemetry.io/proto/otlp v1.11.0 golang.org/x/oauth2 v0.36.0 + golang.org/x/sync v0.22.0 golang.org/x/sys v0.47.0 golang.org/x/time v0.15.0 google.golang.org/grpc v1.83.2 @@ -34,8 +37,10 @@ require ( require ( cel.dev/expr v0.25.3 // indirect github.com/agext/levenshtein v1.2.3 // indirect + github.com/andybalholm/brotli v1.2.2 // indirect github.com/antlr4-go/antlr/v4 v4.13.1 // indirect - github.com/apache/arrow-go/v18 v18.7.0 // indirect + github.com/apache/arrow-go/v18 v18.5.1 // indirect + github.com/apache/thrift v0.24.0 // indirect github.com/apparentlymart/go-textseg/v15 v15.0.0 // indirect github.com/apparentlymart/go-textseg/v17 v17.0.1 // indirect github.com/beorn7/perks v1.0.1 // indirect @@ -67,6 +72,8 @@ require ( github.com/mitchellh/reflectwalk v1.0.2 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect github.com/ncruces/go-strftime v1.0.0 // indirect + github.com/parquet-go/bitpack v1.0.0 // indirect + github.com/parquet-go/jsonlite v1.0.0 // indirect github.com/pierrec/lz4/v4 v4.1.29 // indirect github.com/prometheus/common v0.70.1 // indirect github.com/prometheus/procfs v0.21.1 // indirect @@ -74,18 +81,19 @@ require ( github.com/segmentio/asm v1.2.1 // indirect github.com/segmentio/encoding v0.5.4 // indirect github.com/sirupsen/logrus v1.10.2 // indirect + github.com/twpayne/go-geom v1.6.1 // indirect github.com/yosida95/uritemplate/v3 v3.0.2 // indirect github.com/zclconf/go-cty v1.19.0 // indirect github.com/zclconf/go-cty-yaml v1.2.0 // indirect - github.com/zeebo/xxh3 v1.1.0 // indirect go.yaml.in/yaml/v3 v3.0.5 // indirect golang.org/x/crypto v0.55.0 // indirect golang.org/x/exp v0.0.0-20260824195058-e88cd73687aa // indirect golang.org/x/mod v0.40.0 // indirect golang.org/x/net v0.58.0 // indirect - golang.org/x/sync v0.22.0 // indirect + golang.org/x/telemetry v0.0.0-20260811182544-a038080d80e5 // indirect golang.org/x/text v0.41.0 // indirect golang.org/x/tools v0.49.0 // indirect + golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect google.golang.org/genproto/googleapis/api v0.0.0-20260825221802-da73d73af1c5 // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20260825221802-da73d73af1c5 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect diff --git a/go.sum b/go.sum index 409875ed..d10fac7c 100644 --- a/go.sum +++ b/go.sum @@ -10,14 +10,18 @@ github.com/ag-ui-protocol/ag-ui/sdks/community/go v0.0.0-20260826145851-49e71f2b github.com/ag-ui-protocol/ag-ui/sdks/community/go v0.0.0-20260826145851-49e71f2b2d21/go.mod h1:ERAMOexUee4AIuoxksuuGoEcHl3aqLwaazjGwlR9ZCI= github.com/agext/levenshtein v1.2.3 h1:YB2fHEn0UJagG8T1rrWknE3ZQzWM06O8AMAatNn7lmo= github.com/agext/levenshtein v1.2.3/go.mod h1:JEDfjyjHDjOF/1e4FlBE/PkbqA9OfWu2ki2W0IB5558= +github.com/alecthomas/assert/v2 v2.10.0 h1:jjRCHsj6hBJhkmhznrCzoNpbA3zqy0fYiUcYZP/GkPY= +github.com/alecthomas/assert/v2 v2.10.0/go.mod h1:Bze95FyfUr7x34QZrjL+XP+0qgp/zg8yS+TtBj1WA3k= +github.com/alecthomas/repr v0.4.0 h1:GhI2A8MACjfegCPVq9f1FLvIBS+DrQ2KQBFZP1iFzXc= +github.com/alecthomas/repr v0.4.0/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4= github.com/alexedwards/scs/v2 v2.9.0 h1:xa05mVpwTBm1iLeTMNFfAWpKUm4fXAW7CeAViqBVS90= github.com/alexedwards/scs/v2 v2.9.0/go.mod h1:ToaROZxyKukJKT/xLcVQAChi5k6+Pn1Gvmdl7h3RRj8= github.com/andybalholm/brotli v1.2.2 h1:HzTuoo2ErYQqf5qvcJInB8uvqSVxRttzkFexPWtnceM= github.com/andybalholm/brotli v1.2.2/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY= github.com/antlr4-go/antlr/v4 v4.13.1 h1:SqQKkuVZ+zWkMMNkjy5FZe5mr5WURWnlpmOuzYWrPrQ= github.com/antlr4-go/antlr/v4 v4.13.1/go.mod h1:GKmUxMtwp6ZgGwZSva4eWPC5mS6vUAmOABFgjdkM7Nw= -github.com/apache/arrow-go/v18 v18.7.0 h1:Vw/i+cJyebUofT7JlqFpe65LrmwxULn166jjwStM4HY= -github.com/apache/arrow-go/v18 v18.7.0/go.mod h1:PM6IigLJkdMwIpeHXnymo+xZ52f42a9EYiLtRel4p/A= +github.com/apache/arrow-go/v18 v18.5.1 h1:yaQ6zxMGgf9YCYw4/oaeOU3AULySDlAYDOcnr4LdHdI= +github.com/apache/arrow-go/v18 v18.5.1/go.mod h1:OCCJsmdq8AsRm8FkBSSmYTwL/s4zHW9CqxeBxEytkNE= github.com/apache/thrift v0.24.0 h1:zy31L1a49QTNB2bG1BBfMXol3yJrTH975G3pPubQVLQ= github.com/apache/thrift v0.24.0/go.mod h1:zPt6WxgvTOM6hF92y8C+MkEM5LMxZuk4JcQOiU4Esvs= github.com/apparentlymart/go-textseg/v15 v15.0.0 h1:uYvfpb3DyLSCGWnctWKGj857c6ew1u1fNQOlOtuGxQY= @@ -70,6 +74,8 @@ github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63Y github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE= github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= +github.com/golang/snappy v1.0.0 h1:Oy607GVXHs7RtbggtPBnr2RmDArIsAefDwvrdWvRhGs= +github.com/golang/snappy v1.0.0/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= github.com/google/flatbuffers v25.12.19+incompatible h1:haMV2JRRJCe1998HeW/p0X9UaMTK6SDo0ffLn2+DbLs= github.com/google/flatbuffers v25.12.19+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8= github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= @@ -86,7 +92,11 @@ github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM= github.com/hashicorp/hcl/v2 v2.24.0 h1:2QJdZ454DSsYGoaE6QheQZjtKZSUs9Nh2izTWiwQxvE= github.com/hashicorp/hcl/v2 v2.24.0/go.mod h1:oGoO1FIQYfn/AgyOhlg9qLC6/nOJPX3qGbkZpYAcqfM= +github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM= +github.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg= github.com/kisielk/sqlstruct v0.0.0-20201105191214-5f3e10d3ab46/go.mod h1:yyMNCyc/Ib3bDTKd379tNMpB/7/H5TjM2Y9QJ5THLbE= +github.com/klauspost/asmfmt v1.3.2 h1:4Ri7ox3EwapiOjCki+hw14RyKk201CN4rzyCJRFLpK4= +github.com/klauspost/asmfmt v1.3.2/go.mod h1:AG8TuvYojzulgDAMCnYn50l/5QV3Bs/tp6j0HLHbNSE= github.com/klauspost/compress v1.19.2 h1:hMRETovs/pu/dVWN7zIT1PGG8t509MwT6bO7XSi26R8= github.com/klauspost/compress v1.19.2/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ= github.com/klauspost/cpuid/v2 v2.4.0 h1:S6Hrbc7+ywsr0r+RLapfGBHfyefhCTwEh3A0tV913Dw= @@ -111,6 +121,10 @@ github.com/labstack/echo/v5 v5.3.1 h1:75maCxkQVGualckLc/5s/ihgpH1a1Dc6AuGWNVNs6b github.com/labstack/echo/v5 v5.3.1/go.mod h1:4iEGNQiPPZnkfYpNR/L6fINd3NLiGWUD5+eBotFALas= github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI= github.com/mattn/go-isatty v0.0.24/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A= +github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8 h1:AMFGa4R4MiIpspGNG7Z948v4n35fFGB3RR3G/ry4FWs= +github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8/go.mod h1:mC1jAcsrzbxHt8iiaC+zU4b1ylILSosueou12R++wfY= +github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3 h1:+n/aFZefKZp7spd8DFdX7uMikMLXX4oubIzJF4kv/wI= +github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3/go.mod h1:RagcQ7I8IeTMnF8JTXieKnO4Z6JCsikNEzj0DwauVzE= github.com/mitchellh/copystructure v1.2.0 h1:vpKXTN4ewci03Vljg/q9QvCGUDttBOGBIa15WveJJGw= github.com/mitchellh/copystructure v1.2.0/go.mod h1:qLl+cE2AmVv+CoeAwDPye/v+N2HKCj9FbZEVFJRxO9s= github.com/mitchellh/go-wordwrap v1.0.1 h1:TLuKupo69TCn6TQSyGxwI1EblZZEsQ0vMlAFQflz0v0= @@ -123,6 +137,12 @@ github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w= github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls= +github.com/parquet-go/bitpack v1.0.0 h1:AUqzlKzPPXf2bCdjfj4sTeacrUwsT7NlcYDMUQxPcQA= +github.com/parquet-go/bitpack v1.0.0/go.mod h1:XnVk9TH+O40eOOmvpAVZ7K2ocQFrQwysLMnc6M/8lgs= +github.com/parquet-go/jsonlite v1.0.0 h1:87QNdi56wOfsE5bdgas0vRzHPxfJgzrXGml1zZdd7VU= +github.com/parquet-go/jsonlite v1.0.0/go.mod h1:nDjpkpL4EOtqs6NQugUsi0Rleq9sW/OtC1NnZEnxzF0= +github.com/parquet-go/parquet-go v0.32.0 h1:NWDqTUHfrCS4cJP/Fj2HlxvqsrVedWG3sayMkf+znzM= +github.com/parquet-go/parquet-go v0.32.0/go.mod h1:navtkAYr2LGoJVp141oXPlO/sxLvaOe3la2JEoD8+rg= github.com/pierrec/lz4/v4 v4.1.29 h1:CDQY6qZOLI4DW0Nx6R1vRrifrCeQHnNXkMb0hZWXFjg= github.com/pierrec/lz4/v4 v4.1.29/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4= github.com/prometheus/client_golang v1.24.1 h1:JnJkREXzWxUdCuPFpIWZiPispT9xVV59uiuyR2bPlnU= @@ -143,8 +163,12 @@ github.com/sirupsen/logrus v1.10.2 h1:G2SED73/qrAu6YwbdxOD6peLkCBI3z7L+ykJFTXJBB github.com/sirupsen/logrus v1.10.2/go.mod h1:SLEg8TqYulVKKfIGHldVp2K2aYz2DKSVBq4g/H5bR7Q= github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE= github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg= +github.com/twpayne/go-geom v1.6.1 h1:iLE+Opv0Ihm/ABIcvQFGIiFBXd76oBIar9drAwHFhR4= +github.com/twpayne/go-geom v1.6.1/go.mod h1:Kr+Nly6BswFsKM5sd31YaoWS5PeDDH2NftJTK7Gd028= github.com/wneessen/go-mail v0.8.1 h1:tVcncj02/QySVFw3zr/kXOzZcuFQqBNT6K+Rbgm/pcM= github.com/wneessen/go-mail v0.8.1/go.mod h1:dWZ61zadzCIyvB4y1/YzC5O7MrbbzBfPkARmbosdf8w= +github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU= +github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E= github.com/yosida95/uritemplate/v3 v3.0.2 h1:Ed3Oyj9yrmi9087+NczuL5BwkIc4wvTb5zIM+UJPGz4= github.com/yosida95/uritemplate/v3 v3.0.2/go.mod h1:ILOh0sOhIJR3+L/8afwt/kE++YT040gmv5BQTMR2HP4= github.com/zclconf/go-cty v1.19.0 h1:IV8WdqYZc2c5rLX9bEoLNXKojBAp0MZPBHMIrCoa/s4= @@ -191,12 +215,16 @@ golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek= golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/telemetry v0.0.0-20260811182544-a038080d80e5 h1:ZUSxONxc981v7AW7QUg+I9WwZzSTTJ019ENBYr5pV/Q= +golang.org/x/telemetry v0.0.0-20260811182544-a038080d80e5/go.mod h1:LVehoXe41cL5SCVQilsV7Gg6BNG+Js6P9PhSbYTIUkQ= golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8= golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M= golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U= golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno= golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI= golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo= +golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da h1:noIWHXmPHxILtqtCOPIhSt0ABwskkZKjD3bXGnZGpNY= +golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da/go.mod h1:NDW/Ps6MPRej6fsCIbMTohpP40sJ/P/vI1MoTEGwX90= gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= google.golang.org/genproto/googleapis/api v0.0.0-20260825221802-da73d73af1c5 h1:izFU9hz7aeLI/Mi1J0991ae+xcwRLr7hTqWnB/9aIIU= diff --git a/internal/api/health.go b/internal/api/health.go index 86e0d478..06466f77 100644 --- a/internal/api/health.go +++ b/internal/api/health.go @@ -3,6 +3,7 @@ package api import ( "context" "database/sql" + "errors" "fmt" "log/slog" "net/http" @@ -59,7 +60,7 @@ func (h *HealthHandler) Readiness(c *echo.Context) error { // Check DuckDB resp.Checks["duckdb"] = h.checkDuckDB() - resp.Checks["ducklake"] = h.checkDuckLake() + resp.Checks["telemetry"] = h.checkTelemetry() // Check data directory resp.Checks["data"] = h.checkDataDir() @@ -117,8 +118,8 @@ func (h *HealthHandler) checkDuckDB() CheckResult { } // diskDegradedPct is the free-space fraction below which the data dir reports -// degraded. A full disk silently fails Parquet flushes (the writer drops rows -// once its retry buffer overflows), so we surface pressure before that point. +// degraded. A full disk fails Parquet flushes and stops ingest, so we surface +// pressure before that point. const diskDegradedPct = 10.0 func (h *HealthHandler) checkDataDir() CheckResult { @@ -161,7 +162,7 @@ func diskSpaceResult(freeBytes, totalBytes uint64) CheckResult { } if freePct < diskDegradedPct { res.Status = "degraded" - res.Error = fmt.Sprintf("low disk space: %.1f%% free — ingest flushes drop rows when the disk fills", freePct) + res.Error = fmt.Sprintf("low disk space: %.1f%% free — ingest stops when telemetry cannot commit", freePct) } return res } @@ -177,7 +178,9 @@ func maintenanceStaleThreshold(maintEvery time.Duration) time.Duration { return stale } -func (h *HealthHandler) checkDuckLake() CheckResult { +var telemetryReadinessTimeout = 5 * time.Second + +func (h *HealthHandler) checkTelemetry() CheckResult { if h.duck == nil { return CheckResult{ Status: "unhealthy", @@ -186,11 +189,20 @@ func (h *HealthHandler) checkDuckLake() CheckResult { } start := time.Now() - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + ctx, cancel := context.WithTimeout(context.Background(), telemetryReadinessTimeout) defer cancel() var one int - err := h.duck.DB.QueryRowContext(ctx, "SELECT 1 FROM lake.spans LIMIT 1").Scan(&one) + // Use the same snapshot gate as public reads so a short compaction/retention + // publication cannot race the Parquet scan and report a false outage. + err := h.duck.QueryRowScan(ctx, []any{&one}, "SELECT 1 FROM telemetry.spans LIMIT 1") + if errors.Is(err, query.ErrParquetReadWait) { + return CheckResult{ + Status: "degraded", + LatencyMs: time.Since(start).Milliseconds(), + Error: err.Error(), + } + } if err != nil && err != sql.ErrNoRows { return CheckResult{ Status: "unhealthy", @@ -204,13 +216,14 @@ func (h *HealthHandler) checkDuckLake() CheckResult { LatencyMs: time.Since(start).Milliseconds(), } if err == sql.ErrNoRows { - res.Detail = "telemetry attached, no spans yet" + res.Detail = "telemetry repository ready, no spans yet" } return res } -// checkMaintenance surfaces the maintenance loop's own health (retention + -// DuckLake compaction). A failing pass reports "degraded", not "unhealthy": +// checkMaintenance surfaces the maintenance loop's own health (retention, +// compaction, and cache checkpointing). A failing pass reports "degraded", not +// "unhealthy": // ingest and queries still work while maintenance fails, and a restart // wouldn't fix it — pulling the instance from rotation would only hide the // signal that storage growth is no longer being reclaimed. A maintenance pass @@ -226,8 +239,8 @@ func (h *HealthHandler) checkMaintenance() CheckResult { } } - lastOK, lastAt, lastErr := h.duck.MaintenanceHealth() - return maintenanceResult(lastOK, lastAt, lastErr, h.started, + lastOK, lastAt, failures, lastErr := h.duck.MaintenanceHealth() + return maintenanceResult(lastOK, lastAt, lastErr, failures, h.started, h.cfg.RollupInterval, h.cfg.MaintenanceInterval, time.Now()) @@ -236,7 +249,7 @@ func (h *HealthHandler) checkMaintenance() CheckResult { // maintenanceResult classifies the maintenance loop's health. now/started are // injected so every branch — a failing pass, never-ran-past-grace, and a loop // that stalled after a clean pass — is unit-testable without a running Duck. -func maintenanceResult(lastOK, lastAt time.Time, lastErr error, started time.Time, rollupEvery, maintEvery time.Duration, now time.Time) CheckResult { +func maintenanceResult(lastOK, lastAt time.Time, lastErr error, consecutiveFailures int, started time.Time, rollupEvery, maintEvery time.Duration, now time.Time) CheckResult { res := CheckResult{Status: "ok"} if !lastAt.IsZero() { res.UpdatedAt = lastAt.UTC().Format(time.RFC3339) @@ -244,8 +257,9 @@ func maintenanceResult(lastOK, lastAt time.Time, lastErr error, started time.Tim if lastErr != nil { res.Status = "degraded" res.Error = lastErr.Error() + res.Detail = fmt.Sprintf("%d consecutive failed passes", consecutiveFailures) if !lastOK.IsZero() { - res.Detail = "last clean pass: " + lastOK.UTC().Format(time.RFC3339) + res.Detail += "; last clean pass: " + lastOK.UTC().Format(time.RFC3339) } return res } diff --git a/internal/api/health_test.go b/internal/api/health_test.go index ca3049c3..e50a5ef7 100644 --- a/internal/api/health_test.go +++ b/internal/api/health_test.go @@ -1,6 +1,7 @@ package api import ( + "context" "encoding/json" "errors" "net/http" @@ -111,7 +112,7 @@ func TestReadinessReportsSizingResolvedByLoader(t *testing.T) { } } -func TestReadiness_HealthyDuckLakeAndRollups(t *testing.T) { +func TestReadiness_HealthyTelemetryAndRollups(t *testing.T) { db, mock, err := sqlmock.New() if err != nil { t.Fatalf("sqlmock.New: %v", err) @@ -125,7 +126,7 @@ func TestReadiness_HealthyDuckLakeAndRollups(t *testing.T) { mock.ExpectQuery("SELECT 1"). WillReturnRows(sqlmock.NewRows([]string{"1"}).AddRow(1)) - mock.ExpectQuery("SELECT 1 FROM lake.spans LIMIT 1"). + mock.ExpectQuery("SELECT 1 FROM telemetry.spans LIMIT 1"). WillReturnRows(sqlmock.NewRows([]string{"1"})) mock.ExpectQuery("SELECT\\s+MAX\\(updated_at\\),\\s+COUNT\\(\\*\\),\\s+COALESCE\\(date_diff\\('second', MAX\\(updated_at\\), now\\(\\)\\), 0\\)"). WillReturnRows(sqlmock.NewRows([]string{"max", "count", "age_seconds"}).AddRow(time.Now().UTC(), 2, int64(30))) @@ -151,12 +152,12 @@ func TestReadiness_HealthyDuckLakeAndRollups(t *testing.T) { } // The "data" check reads real host free space, which can legitimately be - // degraded on a low-disk CI/dev box. This test is about ducklake+rollups, so + // degraded on a low-disk CI/dev box. This test is about telemetry+rollups, so // tolerate a data-only degradation but require the actual subjects to be ok. if resp.Status != "ready" && resp.Status != "degraded" { t.Fatalf("status = %q, want ready or degraded", resp.Status) } - for _, key := range []string{"duckdb", "ducklake", "data", "rollups", "maintenance"} { + for _, key := range []string{"duckdb", "telemetry", "data", "rollups", "maintenance"} { if _, ok := resp.Checks[key]; !ok { t.Fatalf("missing %s check", key) } @@ -167,7 +168,7 @@ func TestReadiness_HealthyDuckLakeAndRollups(t *testing.T) { if resp.RuntimeSizing.GOMAXPROCS <= 0 { t.Fatalf("runtime sizing GOMAXPROCS = %d, want positive", resp.RuntimeSizing.GOMAXPROCS) } - for _, key := range []string{"duckdb", "ducklake", "rollups", "maintenance"} { + for _, key := range []string{"duckdb", "telemetry", "rollups", "maintenance"} { if got := resp.Checks[key].Status; got != "ok" { t.Fatalf("%s check = %q, want ok", key, got) } @@ -182,6 +183,33 @@ func TestReadiness_HealthyDuckLakeAndRollups(t *testing.T) { } } +func TestTelemetryReadinessReportsPublicationContentionAsDegraded(t *testing.T) { + duck := &query.Duck{} + entered := make(chan struct{}) + release := make(chan struct{}) + done := make(chan error, 1) + go func() { + done <- duck.PublishParquet(context.Background(), func(context.Context) error { + close(entered) + <-release + return nil + }) + }() + <-entered + original := telemetryReadinessTimeout + telemetryReadinessTimeout = 20 * time.Millisecond + t.Cleanup(func() { telemetryReadinessTimeout = original }) + + result := NewHealthHandler(duck, config.Config{}).checkTelemetry() + if result.Status != "degraded" { + t.Fatalf("telemetry status = %q, want degraded: %+v", result.Status, result) + } + close(release) + if err := <-done; err != nil { + t.Fatal(err) + } +} + // A maintenance pass that never executes must degrade once the startup grace // period elapses — the wedged-first-rollup case the check exists to expose. func TestCheckMaintenance_DegradedWhenNeverRanPastGrace(t *testing.T) { @@ -266,23 +294,25 @@ func TestMaintenanceResult(t *testing.T) { name string lastOK, lastAt time.Time lastErr error + failures int started time.Time maintEvery time.Duration wantStatus string }{ - {"clean recent pass", now.Add(-10 * time.Minute), now.Add(-10 * time.Minute), nil, now.Add(-2 * time.Hour), time.Hour, "ok"}, - {"failing pass", now.Add(-3 * time.Hour), now.Add(-time.Hour), errors.New("boom"), now.Add(-4 * time.Hour), time.Hour, "degraded"}, - {"never ran, past grace", time.Time{}, time.Time{}, nil, now.Add(-10 * time.Minute), time.Hour, "degraded"}, - {"never ran, within grace", time.Time{}, time.Time{}, nil, now.Add(-time.Minute), time.Hour, "ok"}, - {"stalled after clean pass", now.Add(-5 * time.Hour), now.Add(-5 * time.Hour), nil, now.Add(-6 * time.Hour), time.Hour, "degraded"}, + {"clean recent pass", now.Add(-10 * time.Minute), now.Add(-10 * time.Minute), nil, 0, now.Add(-2 * time.Hour), time.Hour, "ok"}, + {"failing pass", now.Add(-3 * time.Hour), now.Add(-time.Hour), errors.New("boom"), 1, now.Add(-4 * time.Hour), time.Hour, "degraded"}, + {"repeated failure remains routable", now.Add(-4 * time.Hour), now.Add(-time.Hour), errors.New("boom"), 3, now.Add(-5 * time.Hour), time.Hour, "degraded"}, + {"never ran, past grace", time.Time{}, time.Time{}, nil, 0, now.Add(-10 * time.Minute), time.Hour, "degraded"}, + {"never ran, within grace", time.Time{}, time.Time{}, nil, 0, now.Add(-time.Minute), time.Hour, "ok"}, + {"stalled after clean pass", now.Add(-5 * time.Hour), now.Add(-5 * time.Hour), nil, 0, now.Add(-6 * time.Hour), time.Hour, "degraded"}, // maintEvery=0 must still detect staleness: the loop floors 0→1h and keeps // running, so the check mirrors that (2h stale > 1h floored threshold). - {"stalled, interval unset (floored)", now.Add(-3 * time.Hour), now.Add(-3 * time.Hour), nil, now.Add(-4 * time.Hour), 0, "degraded"}, - {"recent pass, interval unset", now.Add(-10 * time.Minute), now.Add(-10 * time.Minute), nil, now.Add(-4 * time.Hour), 0, "ok"}, + {"stalled, interval unset (floored)", now.Add(-3 * time.Hour), now.Add(-3 * time.Hour), nil, 0, now.Add(-4 * time.Hour), 0, "degraded"}, + {"recent pass, interval unset", now.Add(-10 * time.Minute), now.Add(-10 * time.Minute), nil, 0, now.Add(-4 * time.Hour), 0, "ok"}, } for _, c := range cases { t.Run(c.name, func(t *testing.T) { - res := maintenanceResult(c.lastOK, c.lastAt, c.lastErr, c.started, rollupEvery, c.maintEvery, now) + res := maintenanceResult(c.lastOK, c.lastAt, c.lastErr, c.failures, c.started, rollupEvery, c.maintEvery, now) if res.Status != c.wantStatus { t.Errorf("status = %q, want %q (detail=%q err=%q)", res.Status, c.wantStatus, res.Detail, res.Error) } diff --git a/internal/config/config.go b/internal/config/config.go index 9eeb8f9a..78f27664 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -12,6 +12,8 @@ import ( appauth "github.com/labstack/fanout/internal/auth" ) +const maxIngestBatchSize = 50_000 + // Config is Fanout's canonical configuration schema. Public names use // FANOUT_ plus the shortest stable term that is clear in a docker run, and the // same terminology across YAML, environment variables, Go, logs, and docs. @@ -27,23 +29,13 @@ type Config struct { // derive host:port from the browser request and OTLPGRPCAddr as a best effort. IngestAdvertisedEndpoint string `koanf:"ingest.advertised_endpoint" env:"FANOUT_INGEST_ADVERTISED_ENDPOINT"` DataDir string `koanf:"storage.data_dir" env:"FANOUT_DATA_DIR" default:"./data"` - FlushInterval time.Duration `koanf:"ingest.flush_interval" env:"FANOUT_FLUSH_INTERVAL" default:"15s"` - FlushBatchSize int `koanf:"ingest.flush_batch_size" env:"FANOUT_FLUSH_BATCH_SIZE" default:"50000"` + IngestBatchSize int `koanf:"ingest.batch_size" env:"FANOUT_INGEST_BATCH_SIZE" default:"50000"` RollupInterval time.Duration `koanf:"storage.rollup_interval" env:"FANOUT_ROLLUP_INTERVAL" default:"1m"` MCPEnabled bool `koanf:"mcp.enabled" env:"FANOUT_MCP_ENABLED" default:"true"` RetentionDays int `koanf:"storage.retention_days" env:"FANOUT_RETENTION_DAYS" default:"30"` - // MaintenanceInterval throttles the DuckLake maintenance cycle (retention - // deletes + compaction). Default 1h. Lower it to compact more - // aggressively, or for soak tests that need to observe file-count staying - // bounded within minutes rather than hours. + // MaintenanceInterval controls Parquet retention and compaction, and + // query-cache checkpointing. MaintenanceInterval time.Duration `koanf:"storage.maintenance_interval" env:"FANOUT_MAINTENANCE_INTERVAL" default:"1h"` - // MergeInterval is the cadence for the cheap, frequent DuckLake file - // compaction pass (ducklake_merge_adjacent_files only — it consolidates the - // newest small parquet files and deletes nothing). Run often (default 1m) it - // keeps the queryable file count continuously low, which is what bounds - // rollup/query scan latency — WITHOUT the churn, deletion race, or catalog - // cost of the full hourly maintenance pass (expire + cleanup). 0 disables it. - MergeInterval time.Duration `koanf:"storage.merge_interval" env:"FANOUT_MERGE_INTERVAL" default:"1m"` // RollupSkipToLatest, set once at boot, advances every rollup watermark to the // current max ingested timestamp so existing data is treated as already-rolled-up // instead of aggregated as a backlog. Stands up a large pre-seeded historical @@ -59,9 +51,9 @@ type Config struct { // values validated on the reference deployment target, a small shared VM // (Hetzner CPX32: 4 vCPU, 8 GB RAM, 160 GB disk). There the self-sizing // resolves to a ~6.4 GB memory cap and 4 query threads (deterministic from - // 8 GB / 4 vCPU). For a current throughput figure run `just stress hetzner` - // rather than trusting a number here — as of 2026-06 it handled ~55k rows/s - // with 0 drops and ~0.4 GB RSS, but that will drift with the ingest path. + // 8 GB / 4 vCPU). Measure the target host with cmd/bench before setting + // production limits; throughput and memory use vary with CPU, disk, and the + // telemetry mix. // // Kept free-floating, separated from the field below by a blank line, so it // stays context for the group rather than becoming DuckDBMemory's own doc @@ -78,15 +70,8 @@ type Config struct { // DuckDB's own default in place (one worker per core). Set it to leave // cores free for ingest on a query-heavy co-tenant host. DuckDBThreads int `koanf:"storage.duckdb.threads" env:"FANOUT_DUCKDB_THREADS"` - // DuckDBMaxConns caps the DuckDB connection pool. A value of 1 serializes - // everything through one handle; the machine-sized default lets read queries - // run concurrently with each other and with ingest flushes. Two things make >1 - // safe: the DuckLake SQLite catalog is opened in WAL mode (enableCatalogWAL), - // so readers don't collide with the single writer and a crashed writer can't - // leave the catalog permanently locked; and write commits are serialized by - // the shared write gate (Duck.WriteGate, wired into the writer via - // UseWriteGate in cmd/fanout/main.go, enforced at startup). Without the WAL - // mode, pool >1 fails with "database is locked". + // DuckDBMaxConns caps the DuckDB connection pool. Reads scan immutable Parquet + // concurrently; rollup-cache writes are serialized by the query write gate. // Zero means "size it from the machine" — the same spelling DuckDBThreads // uses for deferring to a default. Resolution happens in resolveSizing and // is reported in the startup configuration log. @@ -154,10 +139,6 @@ func (c Config) TelemetryParquetDir() string { return filepath.Join(c.TelemetryDir(), "parquet") } -func (c Config) TelemetryDuckLakePath() string { - return filepath.Join(c.TelemetryDir(), "ducklake.sqlite") -} - func (c Config) QueryDir() string { return filepath.Join(c.DataDir, "query") } @@ -199,11 +180,8 @@ func (c Config) Validate() error { if strings.TrimSpace(c.DataDir) == "" { return fmt.Errorf("storage.data_dir must not be empty") } - if c.FlushInterval < time.Second { - return fmt.Errorf("ingest.flush_interval must be at least 1s, got %s", c.FlushInterval) - } - if c.FlushBatchSize <= 0 { - return fmt.Errorf("ingest.flush_batch_size must be > 0, got %d", c.FlushBatchSize) + if c.IngestBatchSize <= 0 || c.IngestBatchSize > maxIngestBatchSize { + return fmt.Errorf("ingest.batch_size must be between 1 and %d, got %d", maxIngestBatchSize, c.IngestBatchSize) } if c.RollupInterval < time.Second { return fmt.Errorf("storage.rollup_interval must be at least 1s, got %s", c.RollupInterval) @@ -214,9 +192,6 @@ func (c Config) Validate() error { if c.MaintenanceInterval < time.Second { return fmt.Errorf("storage.maintenance_interval must be at least 1s, got %s", c.MaintenanceInterval) } - if c.MergeInterval < 0 || (c.MergeInterval > 0 && c.MergeInterval < time.Second) { - return fmt.Errorf("storage.merge_interval must be 0s or at least 1s, got %s", c.MergeInterval) - } if c.DuckDBThreads < 0 { return fmt.Errorf("storage.duckdb.threads must be >= 0, got %d", c.DuckDBThreads) } diff --git a/internal/config/config_test.go b/internal/config/config_test.go index 77e63974..bcd16947 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -42,11 +42,8 @@ func TestLoadReturnsDefaults(t *testing.T) { if cfg.DataDir != "./data" { t.Errorf("DataDir = %q, want %q", cfg.DataDir, "./data") } - if cfg.FlushInterval != 15*time.Second { - t.Errorf("FlushInterval = %s, want %s", cfg.FlushInterval, 15*time.Second) - } - if cfg.FlushBatchSize != 50000 { - t.Errorf("FlushBatchSize = %d, want %d", cfg.FlushBatchSize, 50000) + if cfg.IngestBatchSize != 50000 { + t.Errorf("IngestBatchSize = %d, want %d", cfg.IngestBatchSize, 50000) } if cfg.RollupInterval != time.Minute { t.Errorf("RollupInterval = %s, want %s", cfg.RollupInterval, time.Minute) @@ -77,8 +74,6 @@ func TestLoadLayering(t *testing.T) { path := filepath.Join(t.TempDir(), "fanout.yaml") if err := os.WriteFile(path, []byte(`server: http_addr: ":1111" -storage: - merge_interval: 75s mcp: enabled: false metrics: @@ -100,7 +95,7 @@ auth: if cfg.HTTPAddr != ":2222" { t.Fatalf("HTTPAddr = %q, want environment override", cfg.HTTPAddr) } - if cfg.MergeInterval != 75*time.Second || cfg.MCPEnabled || !cfg.MetricsPublic || cfg.SessionIdleTTL != 10*time.Hour || !cfg.SelfSignup { + if cfg.MCPEnabled || !cfg.MetricsPublic || cfg.SessionIdleTTL != 10*time.Hour || !cfg.SelfSignup { t.Fatalf("YAML values were not merged: %+v", cfg) } } @@ -109,14 +104,13 @@ func TestLoadTypedEnvironmentValues(t *testing.T) { cfg, err := Load(LoadOptions{Environ: append(validEnvironment(), "FANOUT_MCP_ENABLED=false", "FANOUT_METRICS_PUBLIC=true", - "FANOUT_MERGE_INTERVAL=75s", "FANOUT_SESSION_IDLE_TTL=10h", "FANOUT_SELF_SIGNUP=true", )}) if err != nil { t.Fatalf("Load: %v", err) } - if cfg.MCPEnabled || !cfg.MetricsPublic || cfg.MergeInterval != 75*time.Second || cfg.SessionIdleTTL != 10*time.Hour || !cfg.SelfSignup { + if cfg.MCPEnabled || !cfg.MetricsPublic || cfg.SessionIdleTTL != 10*time.Hour || !cfg.SelfSignup { t.Fatalf("environment values were not decoded: %+v", cfg) } } @@ -274,7 +268,7 @@ func TestConfigurationSchemaUsesUnitBearingDurations(t *testing.T) { strings.Contains(env, "_seconds") || strings.Contains(env, "_ms") { t.Errorf("elapsed-time setting uses a unit suffix: %s / %s", spec.key, spec.env) } - if spec.typ == durationType && !strings.HasSuffix(key, "_interval") && !strings.HasSuffix(key, "_ttl") { + if spec.typ == durationType && !strings.HasSuffix(key, "_interval") && !strings.HasSuffix(key, "_ttl") && !strings.HasSuffix(key, "_retention") { t.Errorf("duration setting %s must end in _interval or _ttl", spec.key) } } @@ -283,11 +277,8 @@ func TestConfigurationSchemaUsesUnitBearingDurations(t *testing.T) { func TestLoadDurationIntervals(t *testing.T) { t.Run("YAML", func(t *testing.T) { path := filepath.Join(t.TempDir(), "fanout.yaml") - document := `ingest: - flush_interval: 30s -storage: + document := `storage: rollup_interval: 5m - merge_interval: 0s maintenance_interval: 1h alerts: evaluation_interval: 45s @@ -299,8 +290,8 @@ alerts: if err != nil { t.Fatalf("Load: %v", err) } - if cfg.FlushInterval != 30*time.Second || cfg.RollupInterval != 5*time.Minute || - cfg.MergeInterval != 0 || cfg.MaintenanceInterval != time.Hour || + if cfg.RollupInterval != 5*time.Minute || + cfg.MaintenanceInterval != time.Hour || cfg.AlertEvaluationInterval != 45*time.Second { t.Fatalf("duration values were not decoded: %+v", cfg) } @@ -308,17 +299,15 @@ alerts: t.Run("environment", func(t *testing.T) { cfg, err := Load(LoadOptions{Environ: append(validEnvironment(), - "FANOUT_FLUSH_INTERVAL=30s", "FANOUT_ROLLUP_INTERVAL=5m", - "FANOUT_MERGE_INTERVAL=0s", "FANOUT_MAINTENANCE_INTERVAL=1h", "FANOUT_ALERTS_EVALUATION_INTERVAL=45s", )}) if err != nil { t.Fatalf("Load: %v", err) } - if cfg.FlushInterval != 30*time.Second || cfg.RollupInterval != 5*time.Minute || - cfg.MergeInterval != 0 || cfg.MaintenanceInterval != time.Hour || + if cfg.RollupInterval != 5*time.Minute || + cfg.MaintenanceInterval != time.Hour || cfg.AlertEvaluationInterval != 45*time.Second { t.Fatalf("duration values were not decoded: %+v", cfg) } @@ -403,9 +392,9 @@ func TestLoadRejectsUnknownInputs(t *testing.T) { t.Run("removed environment variables", func(t *testing.T) { for _, name := range []string{ - "FANOUT_FLUSH_SECONDS", "FANOUT_ROLLUP_EVERY_SECONDS", "FANOUT_MERGE_EVERY_SECONDS", + "FANOUT_MERGE_INTERVAL", "FANOUT_MAINTENANCE_EVERY_SECONDS", "FANOUT_ALERTS_EVALUATION_INTERVAL_SECONDS", "FANOUT_MCP_PUBLIC_URL", @@ -503,7 +492,7 @@ func TestLoadRejectsInvalidFilesAndValues(t *testing.T) { for _, test := range []struct { name, document string }{ - {"invalid duration", "ingest:\n flush_interval: never\n"}, + {"invalid duration", "storage:\n rollup_interval: never\n"}, {"fractional integer", "smtp:\n port: 25.9\n"}, {"integer as boolean", "mcp:\n enabled: 2\n"}, {"YAML keyword as boolean", "alerts:\n enabled: off\n"}, @@ -521,8 +510,8 @@ func TestLoadRejectsInvalidFilesAndValues(t *testing.T) { } t.Run("invalid merged config", func(t *testing.T) { - _, err := Load(LoadOptions{Environ: append(validEnvironment(), "FANOUT_FLUSH_INTERVAL=0s")}) - if err == nil || !strings.Contains(err.Error(), "flush") { + _, err := Load(LoadOptions{Environ: append(validEnvironment(), "FANOUT_INGEST_BATCH_SIZE=0")}) + if err == nil || !strings.Contains(err.Error(), "batch_size") { t.Fatalf("error = %v, want validation error", err) } }) @@ -543,7 +532,6 @@ func TestLoadRejectsInvalidFilesAndValues(t *testing.T) { }{ {"negative max connections is not auto", "FANOUT_DUCKDB_MAX_CONNECTIONS=-3", "max_connections"}, {"zero alert interval", "FANOUT_ALERTS_EVALUATION_INTERVAL=0s", "evaluation_interval"}, - {"negative merge interval", "FANOUT_MERGE_INTERVAL=-5s", "merge_interval"}, {"negative maintenance interval", "FANOUT_MAINTENANCE_INTERVAL=-5s", "maintenance_interval"}, {"negative DuckDB threads", "FANOUT_DUCKDB_THREADS=-4", "duckdb.threads"}, } { @@ -558,9 +546,7 @@ func TestLoadRejectsInvalidFilesAndValues(t *testing.T) { for _, test := range []struct { name, assignment, key string }{ - {"subsecond flush interval", "FANOUT_FLUSH_INTERVAL=999ms", "flush_interval"}, {"subsecond rollup interval", "FANOUT_ROLLUP_INTERVAL=999ms", "rollup_interval"}, - {"subsecond merge interval", "FANOUT_MERGE_INTERVAL=1ns", "merge_interval"}, {"subsecond maintenance interval", "FANOUT_MAINTENANCE_INTERVAL=500ms", "maintenance_interval"}, {"subsecond alert interval", "FANOUT_ALERTS_EVALUATION_INTERVAL=999ms", "evaluation_interval"}, } { @@ -622,12 +608,10 @@ func TestValidate(t *testing.T) { OTLPGRPCAddr: "127.0.0.1:4317", OTLPHTTPAddr: "127.0.0.1:4318", DataDir: "./data", - FlushInterval: 15 * time.Second, - FlushBatchSize: 50000, + IngestBatchSize: 50000, RollupInterval: time.Minute, RetentionDays: 30, MaintenanceInterval: time.Hour, - MergeInterval: time.Minute, DuckDBMaxConns: 4, AlertEvaluationInterval: 30 * time.Second, AlertHistoryDays: 7, @@ -651,10 +635,9 @@ func TestValidate(t *testing.T) { name string modify func(*Config) }{ - {"FlushInterval=0", func(c *Config) { c.FlushInterval = 0 }}, - {"FlushInterval=999ms", func(c *Config) { c.FlushInterval = 999 * time.Millisecond }}, - {"FlushBatchSize=0", func(c *Config) { c.FlushBatchSize = 0 }}, - {"FlushBatchSize=-1", func(c *Config) { c.FlushBatchSize = -1 }}, + {"IngestBatchSize=0", func(c *Config) { c.IngestBatchSize = 0 }}, + {"IngestBatchSize=-1", func(c *Config) { c.IngestBatchSize = -1 }}, + {"IngestBatchSize=50001", func(c *Config) { c.IngestBatchSize = 50001 }}, {"RollupInterval=0", func(c *Config) { c.RollupInterval = 0 }}, {"RollupInterval=999ms", func(c *Config) { c.RollupInterval = 999 * time.Millisecond }}, {"RetentionDays=-1", func(c *Config) { c.RetentionDays = -1 }}, @@ -664,8 +647,6 @@ func TestValidate(t *testing.T) { {"DataDir empty", func(c *Config) { c.DataDir = "" }}, {"MaintenanceInterval=0", func(c *Config) { c.MaintenanceInterval = 0 }}, {"MaintenanceInterval=999ms", func(c *Config) { c.MaintenanceInterval = 999 * time.Millisecond }}, - {"MergeInterval=-1s", func(c *Config) { c.MergeInterval = -time.Second }}, - {"MergeInterval=1ns", func(c *Config) { c.MergeInterval = time.Nanosecond }}, {"DuckDBThreads=-1", func(c *Config) { c.DuckDBThreads = -1 }}, {"DuckDBMaxConns=0", func(c *Config) { c.DuckDBMaxConns = 0 }}, {"AlertEvaluationInterval=0", func(c *Config) { c.AlertEvaluationInterval = 0 }}, @@ -716,14 +697,6 @@ func TestValidate(t *testing.T) { } }) - t.Run("MergeInterval=0_valid", func(t *testing.T) { - c := valid - c.MergeInterval = 0 - if err := c.Validate(); err != nil { - t.Errorf("MergeInterval=0 should disable the merge pass: %v", err) - } - }) - t.Run("local mode allows absent SMTP and agent", func(t *testing.T) { c := valid c.SMTPHost, c.SMTPUser, c.SMTPPass, c.SMTPFrom = "", "", "", "" diff --git a/internal/config/sizing.go b/internal/config/sizing.go index 162a95ea..22c78fc5 100644 --- a/internal/config/sizing.go +++ b/internal/config/sizing.go @@ -37,9 +37,7 @@ const ( // exceed available parallelism they add contention rather than concurrency. maxAutoDuckDBConns = 16 - // minDuckDBConns preserves an invariant rather than a preference: - // internal/lake rejects max_connections > 1 without a shared write gate, - // and a pool of 1 serializes reads behind writes. + // minDuckDBConns keeps one long scan from serializing all other reads. minDuckDBConns = 2 ) diff --git a/internal/config/sizing_test.go b/internal/config/sizing_test.go index c39324a2..95a6ff4f 100644 --- a/internal/config/sizing_test.go +++ b/internal/config/sizing_test.go @@ -52,10 +52,8 @@ func TestResolveDuckDBMemoryDeclinesForAbsurdlySmallMachines(t *testing.T) { } } -func TestResolveDuckDBMaxConnsKeepsTheWriteGateInvariant(t *testing.T) { - // internal/lake refuses to start when max_connections > 1 without a write - // gate; at or below 1 it serializes everything through one handle. The - // floor is an invariant, not a preference. +func TestResolveDuckDBMaxConnsKeepsReadConcurrency(t *testing.T) { + // At or below 1 every query is serialized through one handle. for _, cores := range []int{0, 1, 2} { if got := resolveDuckDBMaxConns(cores); got < 2 { t.Fatalf("resolveDuckDBMaxConns(%d) = %d, want at least 2", cores, got) diff --git a/internal/ingest/attrs_test.go b/internal/ingest/attrs_test.go index 381ce2f6..465f9022 100644 --- a/internal/ingest/attrs_test.go +++ b/internal/ingest/attrs_test.go @@ -163,8 +163,8 @@ func TestSpanDurationMs(t *testing.T) { } for _, c := range cases { t.Run(c.name, func(t *testing.T) { - if got := spanDurationMs(c.start, c.end); got != c.want { - t.Errorf("spanDurationMs(%d, %d) = %v, want %v", c.start, c.end, got, c.want) + if got := spanDurationMS(c.start, c.end); got != c.want { + t.Errorf("spanDurationMS(%d, %d) = %v, want %v", c.start, c.end, got, c.want) } }) } diff --git a/internal/ingest/http_test.go b/internal/ingest/http_test.go index e0bb6eba..dd24d440 100644 --- a/internal/ingest/http_test.go +++ b/internal/ingest/http_test.go @@ -22,17 +22,17 @@ import ( "google.golang.org/protobuf/proto" "github.com/labstack/fanout/internal/config" - "github.com/labstack/fanout/internal/lake" "github.com/labstack/fanout/internal/settings" + "github.com/labstack/fanout/internal/telemetry" ) type httpIngestFixture struct { handler http.Handler token string store *settings.Store - spans chan lake.SpanRow - logs chan lake.LogRow - metrics chan lake.MetricRow + spans chan telemetry.Span + logs chan telemetry.Log + metrics chan telemetry.Metric } func newHTTPIngestFixture(t *testing.T, configured bool) *httpIngestFixture { @@ -50,10 +50,10 @@ func newHTTPIngestFixture(t *testing.T, configured bool) *httpIngestFixture { t.Fatalf("SetIngest: %v", err) } } - spans := make(chan lake.SpanRow, 8) - logs := make(chan lake.LogRow, 8) - metrics := make(chan lake.MetricRow, 8) - srv := NewServer(config.Config{DefaultNamespace: "default"}, spans, logs, metrics) + spans := make(chan telemetry.Span, 8) + logs := make(chan telemetry.Log, 8) + metrics := make(chan telemetry.Metric, 8) + srv := NewServer(config.Config{DefaultNamespace: "default"}, newTestSubmitter(spans, logs, metrics)) return &httpIngestFixture{ handler: NewHTTPHandler(srv, store), token: token, @@ -267,8 +267,8 @@ func TestReadOTLPHTTPBodyLimitsDecompressedSize(t *testing.T) { func TestHTTPAndGRPCPathsProduceEquivalentRows(t *testing.T) { t.Run("traces", func(t *testing.T) { request := testTraceRequest() - grpcRows := make(chan lake.SpanRow, 1) - grpcSrv := NewServer(config.Config{DefaultNamespace: "default"}, grpcRows, make(chan lake.LogRow, 1), make(chan lake.MetricRow, 1)) + grpcRows := make(chan telemetry.Span, 1) + grpcSrv := NewServer(config.Config{DefaultNamespace: "default"}, newTestSubmitter(grpcRows, make(chan telemetry.Log, 1), make(chan telemetry.Metric, 1))) if _, err := grpcSrv.exportTraces(context.Background(), request); err != nil { t.Fatal(err) } @@ -284,8 +284,8 @@ func TestHTTPAndGRPCPathsProduceEquivalentRows(t *testing.T) { t.Run("logs", func(t *testing.T) { request := testLogsRequest() - grpcRows := make(chan lake.LogRow, 1) - grpcSrv := NewServer(config.Config{DefaultNamespace: "default"}, make(chan lake.SpanRow, 1), grpcRows, make(chan lake.MetricRow, 1)) + grpcRows := make(chan telemetry.Log, 1) + grpcSrv := NewServer(config.Config{DefaultNamespace: "default"}, newTestSubmitter(make(chan telemetry.Span, 1), grpcRows, make(chan telemetry.Metric, 1))) if _, err := grpcSrv.exportLogs(context.Background(), request); err != nil { t.Fatal(err) } @@ -301,8 +301,8 @@ func TestHTTPAndGRPCPathsProduceEquivalentRows(t *testing.T) { t.Run("metrics", func(t *testing.T) { request := testMetricsRequest() - grpcRows := make(chan lake.MetricRow, 1) - grpcSrv := NewServer(config.Config{DefaultNamespace: "default"}, make(chan lake.SpanRow, 1), make(chan lake.LogRow, 1), grpcRows) + grpcRows := make(chan telemetry.Metric, 1) + grpcSrv := NewServer(config.Config{DefaultNamespace: "default"}, newTestSubmitter(make(chan telemetry.Span, 1), make(chan telemetry.Log, 1), grpcRows)) if _, err := grpcSrv.exportMetrics(context.Background(), request); err != nil { t.Fatal(err) } diff --git a/internal/ingest/server.go b/internal/ingest/server.go index 4379fe01..3dbdf4b8 100644 --- a/internal/ingest/server.go +++ b/internal/ingest/server.go @@ -24,14 +24,17 @@ import ( "google.golang.org/grpc" "github.com/labstack/fanout/internal/config" - "github.com/labstack/fanout/internal/lake" + "github.com/labstack/fanout/internal/telemetry" + telemetrystore "github.com/labstack/fanout/internal/telemetry/store" ) +type batchSubmitter interface { + Submit(context.Context, telemetrystore.Batch) error +} + type Server struct { - cfg config.Config - outSpans chan<- lake.SpanRow - outLogs chan<- lake.LogRow - outMetrics chan<- lake.MetricRow + cfg config.Config + submitter batchSubmitter } type traceService struct { @@ -49,8 +52,8 @@ type metricsService struct { srv *Server } -func NewServer(cfg config.Config, spans chan<- lake.SpanRow, logs chan<- lake.LogRow, metrics chan<- lake.MetricRow) *Server { - return &Server{cfg: cfg, outSpans: spans, outLogs: logs, outMetrics: metrics} +func NewServer(cfg config.Config, submitter batchSubmitter) *Server { + return &Server{cfg: cfg, submitter: submitter} } func RegisterOTLP(s grpc.ServiceRegistrar, srv *Server) { @@ -68,6 +71,7 @@ func (ts *traceService) Export(ctx context.Context, req *collectortrace.ExportTr func (s *Server) exportTraces(ctx context.Context, req *collectortrace.ExportTraceServiceRequest) (*collectortrace.ExportTraceServiceResponse, error) { cfg := s.cfg now := time.Now().UnixNano() + batch := telemetrystore.Batch{} for _, rs := range req.ResourceSpans { resourceJSON := resourceAttrsJSON(rs.Resource) svc := getServiceName(rs.Resource) @@ -78,7 +82,7 @@ func (s *Server) exportTraces(ctx context.Context, req *collectortrace.ExportTra for _, ss := range rs.ScopeSpans { scopeName, scopeVer := scopeInfo(ss.Scope) for _, sp := range ss.Spans { - row := lake.SpanRow{ + row := telemetry.Span{ Namespace: namespace, TraceID: fmt.Sprintf("%x", sp.TraceId), SpanID: fmt.Sprintf("%x", sp.SpanId), @@ -88,7 +92,7 @@ func (s *Server) exportTraces(ctx context.Context, req *collectortrace.ExportTra Kind: sp.Kind.String(), StartUnixNanos: int64(sp.StartTimeUnixNano), EndUnixNanos: int64(sp.EndTimeUnixNano), - DurationMs: spanDurationMs(sp.StartTimeUnixNano, sp.EndTimeUnixNano), + DurationMS: spanDurationMS(sp.StartTimeUnixNano, sp.EndTimeUnixNano), StatusCode: sp.Status.Code.String(), StatusMsg: sp.Status.Message, ResourceJSON: resourceJSON, @@ -115,15 +119,13 @@ func (s *Server) exportTraces(ctx context.Context, req *collectortrace.ExportTra excType, excMsg := extractException(sp.Events) row.ExceptionType = excType row.ExceptionMessage = excMsg - // Partial ingest is acceptable: OTLP clients retry the full batch on error. - select { - case s.outSpans <- row: - case <-ctx.Done(): - return nil, ctx.Err() - } + batch.Spans = append(batch.Spans, row) } } } + if err := s.submitter.Submit(ctx, batch); err != nil { + return nil, err + } return &collectortrace.ExportTraceServiceResponse{}, nil } @@ -136,6 +138,7 @@ func (ls *logsService) Export(ctx context.Context, req *collectorlogs.ExportLogs func (s *Server) exportLogs(ctx context.Context, req *collectorlogs.ExportLogsServiceRequest) (*collectorlogs.ExportLogsServiceResponse, error) { cfg := s.cfg now := time.Now().UnixNano() + batch := telemetrystore.Batch{} for _, rl := range req.ResourceLogs { resourceJSON := resourceAttrsJSON(rl.Resource) svc := getServiceName(rl.Resource) @@ -148,7 +151,7 @@ func (s *Server) exportLogs(ctx context.Context, req *collectorlogs.ExportLogsSe for _, lr := range sl.LogRecords { body := bodyString(lr.Body) tmpl := safeNormalizeTemplate(body) - row := lake.LogRow{ + row := telemetry.Log{ Namespace: namespace, TimeUnixNanos: int64(lr.TimeUnixNano), ObservedTimeNanos: int64(lr.ObservedTimeUnixNano), @@ -166,14 +169,13 @@ func (s *Server) exportLogs(ctx context.Context, req *collectorlogs.ExportLogsSe ScopeVersion: scopeVer, IngestedAt: now, } - select { - case s.outLogs <- row: - case <-ctx.Done(): - return nil, ctx.Err() - } + batch.Logs = append(batch.Logs, row) } } } + if err := s.submitter.Submit(ctx, batch); err != nil { + return nil, err + } return &collectorlogs.ExportLogsServiceResponse{}, nil } @@ -186,6 +188,7 @@ func (ms *metricsService) Export(ctx context.Context, req *collectormetrics.Expo func (s *Server) exportMetrics(ctx context.Context, req *collectormetrics.ExportMetricsServiceRequest) (*collectormetrics.ExportMetricsServiceResponse, error) { cfg := s.cfg now := time.Now().UnixNano() + batch := telemetrystore.Batch{} for _, rm := range req.ResourceMetrics { resourceJSON := resourceAttrsJSON(rm.Resource) svc := getServiceName(rm.Resource) @@ -199,13 +202,13 @@ func (s *Server) exportMetrics(ctx context.Context, req *collectormetrics.Export switch d := m.Data.(type) { case *metricspb.Metric_Gauge: for _, dp := range d.Gauge.DataPoints { - row := lake.MetricRow{ + row := telemetry.Metric{ Namespace: namespace, TimeUnixNanos: int64(dp.TimeUnixNano), Name: m.Name, Description: m.Description, Unit: m.Unit, - MType: "gauge", + Type: "gauge", ServiceName: svc, Value: number(dp.Value), ExemplarsJSON: exemplarsToJSON(dp.Exemplars), @@ -215,11 +218,7 @@ func (s *Server) exportMetrics(ctx context.Context, req *collectormetrics.Export ScopeVersion: scopeVer, IngestedAt: now, } - select { - case s.outMetrics <- row: - case <-ctx.Done(): - return nil, ctx.Err() - } + batch.Metrics = append(batch.Metrics, row) } case *metricspb.Metric_Sum: kind := "sum" @@ -227,13 +226,13 @@ func (s *Server) exportMetrics(ctx context.Context, req *collectormetrics.Export kind = "sum_delta" } for _, dp := range d.Sum.DataPoints { - row := lake.MetricRow{ + row := telemetry.Metric{ Namespace: namespace, TimeUnixNanos: int64(dp.TimeUnixNano), Name: m.Name, Description: m.Description, Unit: m.Unit, - MType: kind, + Type: kind, ServiceName: svc, Value: number(dp.Value), ExemplarsJSON: exemplarsToJSON(dp.Exemplars), @@ -243,11 +242,7 @@ func (s *Server) exportMetrics(ctx context.Context, req *collectormetrics.Export ScopeVersion: scopeVer, IngestedAt: now, } - select { - case s.outMetrics <- row: - case <-ctx.Done(): - return nil, ctx.Err() - } + batch.Metrics = append(batch.Metrics, row) } case *metricspb.Metric_Histogram: for _, dp := range d.Histogram.DataPoints { @@ -255,13 +250,13 @@ func (s *Server) exportMetrics(ctx context.Context, req *collectormetrics.Export if dp.Sum != nil { histSum = *dp.Sum } - row := lake.MetricRow{ + row := telemetry.Metric{ Namespace: namespace, TimeUnixNanos: int64(dp.TimeUnixNano), Name: m.Name, Description: m.Description, Unit: m.Unit, - MType: "histogram", + Type: "histogram", ServiceName: svc, HistBoundsJSON: toJSON(dp.ExplicitBounds), HistCountsJSON: toJSON(dp.BucketCounts), @@ -274,11 +269,7 @@ func (s *Server) exportMetrics(ctx context.Context, req *collectormetrics.Export ScopeVersion: scopeVer, IngestedAt: now, } - select { - case s.outMetrics <- row: - case <-ctx.Done(): - return nil, ctx.Err() - } + batch.Metrics = append(batch.Metrics, row) } case *metricspb.Metric_ExponentialHistogram: for _, dp := range d.ExponentialHistogram.DataPoints { @@ -286,13 +277,13 @@ func (s *Server) exportMetrics(ctx context.Context, req *collectormetrics.Export if dp.Sum != nil { histSum = *dp.Sum } - row := lake.MetricRow{ + row := telemetry.Metric{ Namespace: namespace, TimeUnixNanos: int64(dp.TimeUnixNano), Name: m.Name, Description: m.Description, Unit: m.Unit, - MType: "exp_histogram", + Type: "exp_histogram", ServiceName: svc, HistBoundsJSON: toJSON(expHistBuckets(dp)), HistCountsJSON: toJSON(expHistCounts(dp)), @@ -305,21 +296,17 @@ func (s *Server) exportMetrics(ctx context.Context, req *collectormetrics.Export ScopeVersion: scopeVer, IngestedAt: now, } - select { - case s.outMetrics <- row: - case <-ctx.Done(): - return nil, ctx.Err() - } + batch.Metrics = append(batch.Metrics, row) } case *metricspb.Metric_Summary: for _, dp := range d.Summary.DataPoints { - row := lake.MetricRow{ + row := telemetry.Metric{ Namespace: namespace, TimeUnixNanos: int64(dp.TimeUnixNano), Name: m.Name, Description: m.Description, Unit: m.Unit, - MType: "summary", + Type: "summary", ServiceName: svc, HistBoundsJSON: toJSON(summaryQuantiles(dp)), HistCountsJSON: toJSON(summaryValues(dp)), @@ -331,26 +318,25 @@ func (s *Server) exportMetrics(ctx context.Context, req *collectormetrics.Export ScopeVersion: scopeVer, IngestedAt: now, } - select { - case s.outMetrics <- row: - case <-ctx.Done(): - return nil, ctx.Err() - } + batch.Metrics = append(batch.Metrics, row) } } } } } + if err := s.submitter.Submit(ctx, batch); err != nil { + return nil, err + } return &collectormetrics.ExportMetricsServiceResponse{}, nil } // ---- helpers ---- -// spanDurationMs computes a span's duration in milliseconds, guarding against +// spanDurationMS computes a span's duration in milliseconds, guarding against // the unsigned underflow that a malformed span (end before start) would produce: // uint64 subtraction wraps to a huge positive value, which would otherwise be // stored as a multi-century duration. Such spans are clamped to 0. -func spanDurationMs(startNano, endNano uint64) float64 { +func spanDurationMS(startNano, endNano uint64) float64 { if endNano < startNano { return 0 } diff --git a/internal/ingest/server_test.go b/internal/ingest/server_test.go index 7d52a632..2894cd2f 100644 --- a/internal/ingest/server_test.go +++ b/internal/ingest/server_test.go @@ -12,7 +12,7 @@ import ( tracepb "go.opentelemetry.io/proto/otlp/trace/v1" "github.com/labstack/fanout/internal/config" - "github.com/labstack/fanout/internal/lake" + "github.com/labstack/fanout/internal/telemetry" ) func TestToJSON(t *testing.T) { @@ -507,11 +507,11 @@ func TestExtractException(t *testing.T) { func TestTraceExportContextCancellation(t *testing.T) { // Use an unbuffered channel so the send blocks - spans := make(chan lake.SpanRow) - logs := make(chan lake.LogRow, 1) - metrics := make(chan lake.MetricRow, 1) + spans := make(chan telemetry.Span) + logs := make(chan telemetry.Log, 1) + metrics := make(chan telemetry.Metric, 1) - srv := NewServer(config.Config{}, spans, logs, metrics) + srv := NewServer(config.Config{}, newTestSubmitter(spans, logs, metrics)) ts := &traceService{srv: srv} // Cancel the context before calling Export diff --git a/internal/ingest/submitter_test.go b/internal/ingest/submitter_test.go new file mode 100644 index 00000000..6e6eaeb2 --- /dev/null +++ b/internal/ingest/submitter_test.go @@ -0,0 +1,43 @@ +package ingest + +import ( + "context" + + "github.com/labstack/fanout/internal/telemetry" + telemetrystore "github.com/labstack/fanout/internal/telemetry/store" +) + +type channelSubmitter struct { + spans chan<- telemetry.Span + logs chan<- telemetry.Log + metrics chan<- telemetry.Metric +} + +func newTestSubmitter(spans chan<- telemetry.Span, logs chan<- telemetry.Log, metrics chan<- telemetry.Metric) *channelSubmitter { + return &channelSubmitter{spans: spans, logs: logs, metrics: metrics} +} + +func (s *channelSubmitter) Submit(ctx context.Context, batch telemetrystore.Batch) error { + for _, row := range batch.Spans { + select { + case s.spans <- row: + case <-ctx.Done(): + return ctx.Err() + } + } + for _, row := range batch.Logs { + select { + case s.logs <- row: + case <-ctx.Done(): + return ctx.Err() + } + } + for _, row := range batch.Metrics { + select { + case s.metrics <- row: + case <-ctx.Done(): + return ctx.Err() + } + } + return nil +} diff --git a/internal/lake/writer.go b/internal/lake/writer.go deleted file mode 100644 index e8fe3f68..00000000 --- a/internal/lake/writer.go +++ /dev/null @@ -1,615 +0,0 @@ -package lake - -import ( - "context" - "database/sql" - "database/sql/driver" - "errors" - "fmt" - "log/slog" - "time" - - "github.com/duckdb/duckdb-go/v2" - "github.com/labstack/fanout/internal/config" - "github.com/labstack/fanout/internal/lake/writegate" - "github.com/labstack/fanout/internal/metrics" -) - -type SpanRow struct { - Namespace string - TraceID string - SpanID string - ParentSpanID string - ServiceName string - Name string - Kind string - StartUnixNanos int64 - EndUnixNanos int64 - DurationMs float64 - StatusCode string - StatusMsg string - ResourceJSON []byte - AttributesJSON []byte - EventsJSON []byte - LinksJSON []byte - TraceState string - Flags uint32 - ScopeName string - ScopeVersion string - IngestedAt int64 - HTTPMethod string - HTTPStatusCode string - HTTPRoute string - DBSystem string - RPCMethod string - RPCService string - PeerService string - ServiceVersion string - DeploymentEnv string - ExceptionType string - ExceptionMessage string -} - -type LogRow struct { - Namespace string - TimeUnixNanos int64 - ObservedTimeNanos int64 - Severity string - SeverityNumber int32 - Body string - ServiceName string - TraceID string - SpanID string - Flags uint32 - ResourceJSON []byte - AttributesJSON []byte - ScopeName string - ScopeVersion string - IngestedAt int64 - BodyTemplate string -} - -type MetricRow struct { - Namespace string - TimeUnixNanos int64 - Name string - Description string - Unit string - MType string - ServiceName string - Value float64 - HistBoundsJSON []byte - HistCountsJSON []byte - HistCount int64 - HistSum float64 - ExemplarsJSON []byte - AttributesJSON []byte - ResourceJSON []byte - ScopeName string - ScopeVersion string - IngestedAt int64 -} - -// flushQueueDepth bounds how many filled batches can be queued for the flush -// worker before the receive loop blocks. Blocking here is the intended -// backpressure: it propagates to the ingest channels rather than letting memory -// grow unbounded when the database can't keep up. -const flushQueueDepth = 4 - -type Writer struct { - cfg config.Config - db *sql.DB - chSpans <-chan SpanRow - chLogs <-chan LogRow - chMetrics <-chan MetricRow - // Buffers are owned exclusively by the Run goroutine — no mutex needed. On - // flush their contents are copied into a detached batch and the buffers are - // truncated-and-retained, so the hot receive-loop append never reallocates. - bufSpans []SpanRow - bufLogs []LogRow - bufMetrics []MetricRow - done chan struct{} - // writeGate, when set, serializes appender flushes against the query layer's - // rollup/maintenance commits so two connections never commit to the DuckLake - // catalog at once on a multi-connection pool. Nil is fine (single-connection - // pools already serialize through one handle). - writeGate *writegate.WriteGate -} - -// UseWriteGate shares the query layer's write-serialization gate with the -// writer. Call before Run when the DuckDB pool may hold more than one connection. -func (w *Writer) UseWriteGate(gate *writegate.WriteGate) { w.writeGate = gate } - -// flushBatch is a detached set of rows handed to the flush worker. The worker -// owns the slices once sent. -type flushBatch struct { - spans []SpanRow - logs []LogRow - metrics []MetricRow -} - -func NewWriter(cfg config.Config, db *sql.DB, spans <-chan SpanRow, logs <-chan LogRow, metricsCh <-chan MetricRow) *Writer { - return &Writer{ - cfg: cfg, - db: db, - chSpans: spans, - chLogs: logs, - chMetrics: metricsCh, - done: make(chan struct{}), - } -} - -// Wait blocks until Run() has returned (final flush complete). -func (w *Writer) Wait() { - <-w.done -} - -func (w *Writer) Run(ctx context.Context) error { - defer close(w.done) - - // A multi-connection pool requires the shared write gate so appender flushes - // don't commit concurrently with rollups. Fail loudly rather than silently - // running unserialized writes (which surface only as catalog-lock errors - // under load). Single-connection pools serialize through the one handle, so a - // nil gate is fine there. - if w.cfg.DuckDBMaxConns > 1 && w.writeGate == nil { - return fmt.Errorf("lake writer: storage.duckdb.max_connections=%d requires a shared write gate; call UseWriteGate before Run", w.cfg.DuckDBMaxConns) - } - - // Flushes run on a dedicated worker so a slow database insert never stalls the - // receive loop below (which would stop draining the ingest channels and apply - // backpressure all the way to the gRPC handlers). Run detaches a filled batch - // and hands it off; the worker serializes the actual writes and retries. - flushCh := make(chan flushBatch, flushQueueDepth) - workerDone := make(chan error, 1) - go w.flushWorker(flushCh, workerDone) - - ticker := time.NewTicker(w.cfg.FlushInterval) - defer ticker.Stop() - - spansCh := w.chSpans - logsCh := w.chLogs - metricsCh := w.chMetrics - - finish := func() error { - w.drainChannels(&spansCh, &logsCh, &metricsCh) - w.flush(flushCh) - close(flushCh) - return <-workerDone - } - - for { - select { - case r, ok := <-spansCh: - if !ok { - spansCh = nil - continue - } - w.bufSpans = append(w.bufSpans, r) - metrics.RecordIngest("spans", 1) - metrics.UpdateQueueDepth("spans", len(spansCh)) - if w.shouldFlush() { - w.flush(flushCh) - } - case r, ok := <-logsCh: - if !ok { - logsCh = nil - continue - } - w.bufLogs = append(w.bufLogs, r) - metrics.RecordIngest("logs", 1) - metrics.UpdateQueueDepth("logs", len(logsCh)) - if w.shouldFlush() { - w.flush(flushCh) - } - case r, ok := <-metricsCh: - if !ok { - metricsCh = nil - continue - } - w.bufMetrics = append(w.bufMetrics, r) - metrics.RecordIngest("metrics", 1) - metrics.UpdateQueueDepth("metrics", len(metricsCh)) - if w.shouldFlush() { - w.flush(flushCh) - } - case <-ticker.C: - w.flush(flushCh) - case <-ctx.Done(): - return finish() - } - - if spansCh == nil && logsCh == nil && metricsCh == nil { - return finish() - } - } -} - -// shouldFlush reports whether any buffer has reached the configured batch size. -func (w *Writer) shouldFlush() bool { - total := len(w.bufSpans) + len(w.bufLogs) + len(w.bufMetrics) - return len(w.bufSpans) >= w.cfg.FlushBatchSize || - len(w.bufLogs) >= w.cfg.FlushBatchSize || - len(w.bufMetrics) >= w.cfg.FlushBatchSize || - total >= w.cfg.FlushBatchSize -} - -// flush detaches the current buffers and hands them to the flush worker. Sending -// on flushCh blocks if the worker is behind, which is the intended backpressure. -func (w *Writer) flush(flushCh chan<- flushBatch) { - // Copy the filled rows into a freshly-sized batch and RETAIN the receive - // buffers (truncate, keep the backing array). The per-row append in the hot - // receive loop then never reallocates after warmup — it was the top ingest - // allocator (profiled ~11.5GB / 23%), and a sync.Pool didn't help because - // the GC-heavy workload evicts pooled buffers every cycle. The cost moves to - // one exact-sized copy per flush (infrequent) instead of a per-row regrow. - batch := flushBatch{} - if len(w.bufSpans) > 0 { - batch.spans = make([]SpanRow, len(w.bufSpans)) - copy(batch.spans, w.bufSpans) - w.bufSpans = w.bufSpans[:0] - } - if len(w.bufLogs) > 0 { - batch.logs = make([]LogRow, len(w.bufLogs)) - copy(batch.logs, w.bufLogs) - w.bufLogs = w.bufLogs[:0] - } - if len(w.bufMetrics) > 0 { - batch.metrics = make([]MetricRow, len(w.bufMetrics)) - copy(batch.metrics, w.bufMetrics) - w.bufMetrics = w.bufMetrics[:0] - } - if batch.spans == nil && batch.logs == nil && batch.metrics == nil { - return - } - flushCh <- batch -} - -// flushWorker serializes all database writes. It carries rows that failed to -// insert forward and prepends them to the next batch so a transient error -// doesn't drop data (until the retry buffer cap is exceeded — see retainRows). -// When the input closes, it retries the final carry and reports any rows that -// remain unwritten to Run. -func (w *Writer) flushWorker(flushCh <-chan flushBatch, workerDone chan<- error) { - var carry flushBatch - var spanErr, logErr, metricErr error - for batch := range flushCh { - // Common path (no retry leftover): adopt the batch slice directly — no - // copy. Only when carry holds un-written rows from a failed flush do we - // append (carry's backing array is reused across flushes). - if len(carry.spans) == 0 { - carry.spans = batch.spans - } else { - carry.spans = append(carry.spans, batch.spans...) - } - if len(carry.logs) == 0 { - carry.logs = batch.logs - } else { - carry.logs = append(carry.logs, batch.logs...) - } - if len(carry.metrics) == 0 { - carry.metrics = batch.metrics - } else { - carry.metrics = append(carry.metrics, batch.metrics...) - } - carry.spans, spanErr = writeRows(carry.spans, "spans", w.insertSpans, w.retryCap()) - carry.logs, logErr = writeRows(carry.logs, "logs", w.insertLogs, w.retryCap()) - carry.metrics, metricErr = writeRows(carry.metrics, "metrics", w.insertMetrics, w.retryCap()) - } - - // A normal batch already had one attempt above. Give the final carry two - // additional attempts with a short backoff, then surface a hard error to Run. - // Successful signals are cleared independently so one failing table cannot - // cause already-committed rows from another table to be duplicated. - for attempt := 1; hasRows(carry) && attempt <= 2; attempt++ { - time.Sleep(time.Duration(attempt) * 100 * time.Millisecond) - carry.spans, spanErr = writeRows(carry.spans, "spans", w.insertSpans, w.retryCap()) - carry.logs, logErr = writeRows(carry.logs, "logs", w.insertLogs, w.retryCap()) - carry.metrics, metricErr = writeRows(carry.metrics, "metrics", w.insertMetrics, w.retryCap()) - } - - var errs []error - if len(carry.spans) > 0 { - errs = append(errs, fmt.Errorf("spans final flush (%d rows): %w", len(carry.spans), spanErr)) - } - if len(carry.logs) > 0 { - errs = append(errs, fmt.Errorf("logs final flush (%d rows): %w", len(carry.logs), logErr)) - } - if len(carry.metrics) > 0 { - errs = append(errs, fmt.Errorf("metrics final flush (%d rows): %w", len(carry.metrics), metricErr)) - } - workerDone <- errors.Join(errs...) -} - -func hasRows(batch flushBatch) bool { - return len(batch.spans) > 0 || len(batch.logs) > 0 || len(batch.metrics) > 0 -} - -// writeRows inserts a batch and returns the rows to carry forward plus the -// insertion error: empty/nil on success, or the retry-capped remainder/error -// on failure. -func writeRows[T any](rows []T, signal string, insert func([]T) error, retryCap int) ([]T, error) { - if len(rows) == 0 { - return rows[:0], nil - } - start := time.Now() - if err := insert(rows); err != nil { - slog.Error("write failed", "signal", signal, "err", err) - metrics.FlushErrors.WithLabelValues(signal).Inc() - return retainRows(rows, retryCap, signal), err - } - metrics.RecordFlush(signal, 0, time.Since(start).Seconds()) - return rows[:0], nil -} - -// drainChannels non-blockingly pulls any buffered rows into the local buffers -// during shutdown. It honors the ok flag so a closed channel is retired (set to -// nil) instead of spinning on zero values. -func (w *Writer) drainChannels(spansCh *<-chan SpanRow, logsCh *<-chan LogRow, metricsCh *<-chan MetricRow) { - for { - drained := false - - select { - case r, ok := <-*spansCh: - if ok { - w.bufSpans = append(w.bufSpans, r) - drained = true - } else { - *spansCh = nil - } - default: - } - select { - case r, ok := <-*logsCh: - if ok { - w.bufLogs = append(w.bufLogs, r) - drained = true - } else { - *logsCh = nil - } - default: - } - select { - case r, ok := <-*metricsCh: - if ok { - w.bufMetrics = append(w.bufMetrics, r) - drained = true - } else { - *metricsCh = nil - } - default: - } - - if !drained { - return - } - } -} - -func (w *Writer) retryCap() int { - maxRetry := w.cfg.FlushBatchSize * 3 - if maxRetry < 0 { - return 0 - } - return maxRetry -} - -func (w *Writer) insertSpans(rows []SpanRow) error { - ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) - defer cancel() - return withAppender(ctx, w.db, w.writeGate, writegate.WriteIngestSpans, "spans", func(a *duckdb.Appender) error { - for _, row := range rows { - namespace := normalizeNamespace(row.Namespace) - if err := a.AppendRow( - namespace, - row.TraceID, - row.SpanID, - optionalString(row.ParentSpanID), - row.ServiceName, - row.Name, - row.Kind, - eventTime(row.StartUnixNanos, 0, row.IngestedAt), - optionalTime(row.EndUnixNanos), - row.StartUnixNanos, - row.EndUnixNanos, - row.DurationMs, - row.StatusCode, - optionalString(row.StatusMsg), - optionalJSON(row.ResourceJSON), - optionalJSON(row.AttributesJSON), - optionalJSON(row.EventsJSON), - optionalJSON(row.LinksJSON), - optionalString(row.TraceState), - int64(row.Flags), - optionalString(row.ScopeName), - optionalString(row.ScopeVersion), - optionalTime(row.IngestedAt), - row.IngestedAt, - optionalString(row.HTTPMethod), - optionalString(row.HTTPStatusCode), - optionalString(row.HTTPRoute), - optionalString(row.DBSystem), - optionalString(row.RPCMethod), - optionalString(row.RPCService), - optionalString(row.PeerService), - optionalString(row.ServiceVersion), - optionalString(row.DeploymentEnv), - optionalString(row.ExceptionType), - optionalString(row.ExceptionMessage), - ); err != nil { - // Skip the malformed row rather than aborting the batch: the - // appender flushes rows already added on Close, so returning here - // would commit the prefix and then re-append it on retry (a - // duplicate), and a permanently-bad row would poison every flush. - slog.Error("skip malformed span row", "err", err) - metrics.RowsDropped.WithLabelValues("spans").Inc() - continue - } - } - return nil - }) -} - -func (w *Writer) insertLogs(rows []LogRow) error { - ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) - defer cancel() - return withAppender(ctx, w.db, w.writeGate, writegate.WriteIngestLogs, "logs", func(a *duckdb.Appender) error { - for _, row := range rows { - namespace := normalizeNamespace(row.Namespace) - if err := a.AppendRow( - namespace, - eventTime(row.TimeUnixNanos, row.ObservedTimeNanos, row.IngestedAt), - eventTime(row.ObservedTimeNanos, row.TimeUnixNanos, row.IngestedAt), - row.TimeUnixNanos, - optionalInt64(row.ObservedTimeNanos), - row.Severity, - int64(row.SeverityNumber), - row.Body, - optionalString(row.ServiceName), - optionalString(row.TraceID), - optionalString(row.SpanID), - int64(row.Flags), - optionalJSON(row.ResourceJSON), - optionalJSON(row.AttributesJSON), - optionalString(row.ScopeName), - optionalString(row.ScopeVersion), - optionalTime(row.IngestedAt), - row.IngestedAt, - optionalString(row.BodyTemplate), - ); err != nil { - slog.Error("skip malformed log row", "err", err) - metrics.RowsDropped.WithLabelValues("logs").Inc() - continue - } - } - return nil - }) -} - -func (w *Writer) insertMetrics(rows []MetricRow) error { - ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) - defer cancel() - return withAppender(ctx, w.db, w.writeGate, writegate.WriteIngestMetrics, "metrics", func(a *duckdb.Appender) error { - for _, row := range rows { - namespace := normalizeNamespace(row.Namespace) - if err := a.AppendRow( - namespace, - eventTime(row.TimeUnixNanos, 0, row.IngestedAt), - row.TimeUnixNanos, - row.Name, - optionalString(row.Description), - optionalString(row.Unit), - row.MType, - optionalString(row.ServiceName), - row.Value, - optionalJSON(row.HistBoundsJSON), - optionalJSON(row.HistCountsJSON), - optionalInt64(row.HistCount), - row.HistSum, - optionalJSON(row.ExemplarsJSON), - optionalJSON(row.AttributesJSON), - optionalJSON(row.ResourceJSON), - optionalString(row.ScopeName), - optionalString(row.ScopeVersion), - optionalTime(row.IngestedAt), - row.IngestedAt, - ); err != nil { - slog.Error("skip malformed metric row", "err", err) - metrics.RowsDropped.WithLabelValues("metrics").Inc() - continue - } - } - return nil - }) -} - -func normalizeNamespace(namespace string) string { - if namespace == "" { - namespace = "default" - } - return namespace -} - -// eventTime keeps query event-time columns populated when a producer omits its -// primary OTLP timestamp. The fallback order is primary, secondary (for logs), -// then ingest time. Raw *_unix_nano fields remain unchanged and therefore -// preserve which timestamps the producer actually supplied. -func eventTime(primary, secondary, ingested int64) any { - for _, nanos := range []int64{primary, secondary, ingested} { - if nanos > 0 { - return time.Unix(0, nanos).UTC() - } - } - return nil -} - -func optionalString(v string) any { - if v == "" { - return nil - } - return v -} - -func optionalJSON(v []byte) any { - if len(v) == 0 { - return nil - } - return string(v) -} - -func optionalTime(unixNano int64) any { - if unixNano <= 0 { - return nil - } - return time.Unix(0, unixNano).UTC() -} - -func optionalInt64(v int64) any { - if v == 0 { - return nil - } - return v -} - -func withAppender(ctx context.Context, db *sql.DB, gate *writegate.WriteGate, operation writegate.WriteOperation, table string, fn func(a *duckdb.Appender) error) error { - // Acquire the write gate before the connection so lock ordering matches the - // query layer (write gate → conn); the two write paths therefore can't deadlock - // against each other. (A writer holding the gate can still wait on a connection - // behind long-running readers on a small pool — that's throughput, not a - // deadlock, since readers never take the write gate.) - if gate != nil { - unlock := gate.Lock(operation) - defer unlock() - } - conn, err := db.Conn(ctx) - if err != nil { - return err - } - defer conn.Close() - - return conn.Raw(func(raw any) error { - driverConn, ok := raw.(driver.Conn) - if !ok { - return fmt.Errorf("unexpected driver connection type %T", raw) - } - appender, err := duckdb.NewAppender(driverConn, "lake", "", table) - if err != nil { - return err - } - if err := fn(appender); err != nil { - _ = appender.Close() - return err - } - return appender.Close() - }) -} - -func retainRows[T any](rows []T, maxRetry int, signal string) []T { - if len(rows) <= maxRetry { - return rows - } - dropped := len(rows) - maxRetry - slog.Error("data loss: retry buffer full, dropping rows", "signal", signal, "buffered", len(rows), "dropped", dropped) - metrics.RowsDropped.WithLabelValues(signal).Add(float64(dropped)) - return rows[:maxRetry] -} diff --git a/internal/lake/writer_test.go b/internal/lake/writer_test.go deleted file mode 100644 index f17e68b3..00000000 --- a/internal/lake/writer_test.go +++ /dev/null @@ -1,171 +0,0 @@ -package lake - -import ( - "context" - "database/sql" - "testing" - "time" - - _ "github.com/duckdb/duckdb-go/v2" - - "github.com/labstack/fanout/internal/config" - "github.com/labstack/fanout/internal/query" -) - -func openWriterTestDB(t *testing.T) *sql.DB { - t.Helper() - db, err := sql.Open("duckdb", "") - if err != nil { - t.Fatalf("open duckdb: %v", err) - } - if _, err := db.Exec(`ATTACH ':memory:' AS lake`); err != nil { - t.Fatalf("attach lake catalog: %v", err) - } - if err := query.CreateTables(db); err != nil { - t.Fatalf("CreateTables failed: %v", err) - } - t.Cleanup(func() { _ = db.Close() }) - return db -} - -func TestWriterFlushBatchSize(t *testing.T) { - db := openWriterTestDB(t) - - chSpans := make(chan SpanRow, 10) - chLogs := make(chan LogRow, 10) - chMetrics := make(chan MetricRow, 10) - - w := NewWriter(config.Config{ - FlushInterval: time.Minute, - FlushBatchSize: 2, - DefaultNamespace: "default", - }, db, chSpans, chLogs, chMetrics) - - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - go func() { _ = w.Run(ctx) }() - - now := time.Now().UnixNano() - chSpans <- SpanRow{ - Namespace: "ns-a", - TraceID: "trace-1", - SpanID: "span-1", - ServiceName: "svc", - Name: "op", - StartUnixNanos: now, - EndUnixNanos: now + int64(time.Millisecond), - DurationMs: 1, - IngestedAt: now, - } - chSpans <- SpanRow{ - Namespace: "ns-a", - TraceID: "trace-1", - SpanID: "span-2", - ServiceName: "svc", - Name: "op", - StartUnixNanos: now, - EndUnixNanos: now + int64(time.Millisecond), - DurationMs: 1, - IngestedAt: now, - } - - requireCount(t, db, `SELECT count(*) FROM lake.spans`, 2) -} - -func TestWriterFlushesRemainderOnShutdown(t *testing.T) { - db := openWriterTestDB(t) - - chSpans := make(chan SpanRow, 10) - chLogs := make(chan LogRow, 10) - chMetrics := make(chan MetricRow, 10) - - w := NewWriter(config.Config{ - FlushInterval: time.Hour, // long, so only shutdown triggers the flush - FlushBatchSize: 1000, // large, so the single row never hits a size flush - DefaultNamespace: "default", - }, db, chSpans, chLogs, chMetrics) - - ctx, cancel := context.WithCancel(context.Background()) - go func() { _ = w.Run(ctx) }() - - now := time.Now().UnixNano() - chSpans <- SpanRow{ - Namespace: "ns-a", TraceID: "t", SpanID: "s", ServiceName: "svc", Name: "op", - StartUnixNanos: now, EndUnixNanos: now + 1, DurationMs: 1, IngestedAt: now, - } - - // Give the row time to be received into the buffer, then shut down. The - // remaining buffered row must be drained and flushed before Wait() returns. - time.Sleep(100 * time.Millisecond) - cancel() - w.Wait() - - var got int - if err := db.QueryRow(`SELECT count(*) FROM lake.spans`).Scan(&got); err != nil { - t.Fatalf("count query failed: %v", err) - } - if got != 1 { - t.Fatalf("count = %d, want 1 (row should be flushed on shutdown)", got) - } -} - -func TestEventTimeFallsBackWithoutSchemaDuplication(t *testing.T) { - ingested := time.Now().UTC().Truncate(time.Microsecond) - got, ok := eventTime(0, 0, ingested.UnixNano()).(time.Time) - if !ok { - t.Fatalf("eventTime() type = %T, want time.Time", got) - } - if !got.Equal(ingested) { - t.Fatalf("eventTime() = %s, want %s", got, ingested) - } - - observed := ingested.Add(-time.Second) - got, ok = eventTime(0, observed.UnixNano(), ingested.UnixNano()).(time.Time) - if !ok || !got.Equal(observed) { - t.Fatalf("eventTime() secondary fallback = %v, want %s", got, observed) - } - - primary := observed.Add(-time.Second) - got, ok = eventTime(primary.UnixNano(), observed.UnixNano(), ingested.UnixNano()).(time.Time) - if !ok || !got.Equal(primary) { - t.Fatalf("eventTime() primary timestamp = %v, want %s", got, primary) - } -} - -func TestFlushWorkerReportsUnwrittenFinalCarry(t *testing.T) { - db, err := sql.Open("duckdb", "") - if err != nil { - t.Fatalf("open duckdb: %v", err) - } - if err := db.Close(); err != nil { - t.Fatalf("close duckdb: %v", err) - } - - w := &Writer{db: db, cfg: config.Config{FlushBatchSize: 10}} - flushCh := make(chan flushBatch, 1) - done := make(chan error, 1) - flushCh <- flushBatch{spans: []SpanRow{{TraceID: "t", SpanID: "s"}}} - close(flushCh) - w.flushWorker(flushCh, done) - if err := <-done; err == nil { - t.Fatal("flushWorker() error = nil, want final-flush error") - } -} - -func requireCount(t *testing.T, db *sql.DB, q string, want int) { - t.Helper() - deadline := time.Now().Add(2 * time.Second) - for time.Now().Before(deadline) { - var got int - if err := db.QueryRow(q).Scan(&got); err == nil && got == want { - return - } - time.Sleep(25 * time.Millisecond) - } - - var got int - if err := db.QueryRow(q).Scan(&got); err != nil { - t.Fatalf("count query failed: %v", err) - } - t.Fatalf("count = %d, want %d", got, want) -} diff --git a/internal/metrics/metrics.go b/internal/metrics/metrics.go index 859f7099..07e1c85a 100644 --- a/internal/metrics/metrics.go +++ b/internal/metrics/metrics.go @@ -24,20 +24,21 @@ const ( RollupDisabled RollupResult = "disabled" ) -type DuckLakeOperation string +type TelemetryOperation string const ( - DuckLakeMerge DuckLakeOperation = "merge" - DuckLakeMaintenance DuckLakeOperation = "maintenance" + TelemetryCompaction TelemetryOperation = "compaction" + TelemetryMaintenance TelemetryOperation = "maintenance" ) -type DuckLakeResult string +type TelemetryResult string const ( - DuckLakeSuccess DuckLakeResult = "success" - DuckLakeError DuckLakeResult = "error" - DuckLakeDisabled DuckLakeResult = "disabled" - DuckLakeThrottled DuckLakeResult = "throttled" + TelemetrySuccess TelemetryResult = "success" + TelemetryError TelemetryResult = "error" + TelemetryDisabled TelemetryResult = "disabled" + TelemetryThrottled TelemetryResult = "throttled" + TelemetryNoop TelemetryResult = "noop" ) var ( @@ -49,7 +50,7 @@ var ( IngestQueueDepth = promauto.NewGaugeVec(prometheus.GaugeOpts{ Name: "fanout_ingest_queue_depth", - Help: "Current queue depth per signal", + Help: "Current ingest submission queue depth", }, []string{"signal"}) FlushTotal = promauto.NewCounterVec(prometheus.CounterOpts{ @@ -57,11 +58,6 @@ var ( Help: "Total flush operations", }, []string{"signal"}) - FlushBytes = promauto.NewCounterVec(prometheus.CounterOpts{ - Name: "fanout_flush_bytes_total", - Help: "Total bytes flushed", - }, []string{"signal"}) - FlushDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{ Name: "fanout_flush_duration_seconds", Help: "Flush duration in seconds", @@ -75,18 +71,18 @@ var ( RowsDropped = promauto.NewCounterVec(prometheus.CounterOpts{ Name: "fanout_rows_dropped_total", - Help: "Total rows dropped due to retry buffer overflow", + Help: "Total rows left uncommitted after a permanent flush failure", }, []string{"signal"}) WriteGateWait = promauto.NewHistogramVec(prometheus.HistogramOpts{ Name: "fanout_write_gate_wait_seconds", - Help: "Time spent waiting to enter the DuckLake catalog write critical section", + Help: "Time spent waiting to enter the Telemetry catalog write critical section", Buckets: []float64{.0001, .0005, .001, .005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10, 30, 60}, }, []string{"operation"}) WriteGateHold = promauto.NewHistogramVec(prometheus.HistogramOpts{ Name: "fanout_write_gate_hold_seconds", - Help: "Time spent inside the DuckLake catalog write critical section", + Help: "Time spent inside the Telemetry catalog write critical section", Buckets: []float64{.0001, .0005, .001, .005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10, 30, 60}, }, []string{"operation"}) @@ -170,28 +166,44 @@ var ( Help: "Estimated bounded catch-up chunks remaining for the rollup", }, []string{"rollup"}) - DuckLakeOperationTotal = promauto.NewCounterVec(prometheus.CounterOpts{ - Name: "fanout_ducklake_operation_total", - Help: "DuckLake merge and maintenance calls by bounded outcome", + TelemetryOperationTotal = promauto.NewCounterVec(prometheus.CounterOpts{ + Name: "fanout_telemetry_operation_total", + Help: "Telemetry compaction and maintenance calls by bounded outcome", }, []string{"operation", "result"}) - DuckLakeOperationDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{ - Name: "fanout_ducklake_operation_duration_seconds", - Help: "Executed DuckLake merge and maintenance duration in seconds, including write-gate wait", + TelemetryOperationDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{ + Name: "fanout_telemetry_operation_duration_seconds", + Help: "Executed telemetry compaction and maintenance duration in seconds", Buckets: []float64{.001, .005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10, 30, 60, 120, 300}, }, []string{"operation"}) // Storage metrics - LakeSize = promauto.NewGaugeVec(prometheus.GaugeOpts{ - Name: "fanout_lake_size_bytes", - Help: "Total size of lake data in bytes", + ParquetSize = promauto.NewGaugeVec(prometheus.GaugeOpts{ + Name: "fanout_parquet_size_bytes", + Help: "Total size of telemetry Parquet files in bytes", }, []string{"signal"}) - LakePartitions = promauto.NewGaugeVec(prometheus.GaugeOpts{ - Name: "fanout_lake_partitions", - Help: "Number of partitions per signal", + ParquetFiles = promauto.NewGaugeVec(prometheus.GaugeOpts{ + Name: "fanout_parquet_files", + Help: "Number of telemetry Parquet files per signal", }, []string{"signal"}) + ParquetPublishWaiters = promauto.NewGauge(prometheus.GaugeOpts{ + Name: "fanout_parquet_publish_waiters", + Help: "Current Parquet publications waiting for active readers", + }) + + ParquetPublishWait = promauto.NewHistogram(prometheus.HistogramOpts{ + Name: "fanout_parquet_publish_wait_seconds", + Help: "Time spent waiting to publish a new Parquet file set", + Buckets: []float64{.0001, .0005, .001, .005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10, 30, 60}, + }) + + ParquetPublishTimeouts = promauto.NewCounter(prometheus.CounterOpts{ + Name: "fanout_parquet_publish_timeouts_total", + Help: "Parquet publications abandoned after reader wait timeout", + }) + // HTTP metrics HTTPRequestsTotal = promauto.NewCounterVec(prometheus.CounterOpts{ Name: "fanout_http_requests_total", @@ -222,13 +234,12 @@ func RecordIngest(signal string, count int) { } // RecordFlush records a flush event -func RecordFlush(signal string, bytes int64, durationSec float64) { +func RecordFlush(signal string, durationSec float64) { FlushTotal.WithLabelValues(signal).Inc() - FlushBytes.WithLabelValues(signal).Add(float64(bytes)) FlushDuration.WithLabelValues(signal).Observe(durationSec) } -// RecordWriteGate records one complete DuckLake catalog write critical section. +// RecordWriteGate records one complete Telemetry catalog write critical section. // Callers constrain operation to the fixed writegate.WriteOperation set so this // metric cannot grow with tenant or telemetry cardinality. func RecordWriteGate(operation string, waitSec, holdSec float64) { @@ -294,21 +305,21 @@ func UpdateRollupProgress(component RollupComponent, enabled bool, watermarkNano RollupBacklogChunks.WithLabelValues(string(component)).Set(float64(backlogChunks)) } -// RecordDuckLakeOperation records an outcome for merge or maintenance. Skipped +// RecordTelemetryOperation records an outcome for compaction or maintenance. Skipped // calls have no duration sample so throttle ticks cannot distort execution p95. -func RecordDuckLakeOperation(operation DuckLakeOperation, result DuckLakeResult, durationSec float64) { - validateDuckLakeOperation(operation) - validateDuckLakeResult(result) - DuckLakeOperationTotal.WithLabelValues(string(operation), string(result)).Inc() - if result == DuckLakeSuccess || result == DuckLakeError { - DuckLakeOperationDuration.WithLabelValues(string(operation)).Observe(math.Max(durationSec, 0)) +func RecordTelemetryOperation(operation TelemetryOperation, result TelemetryResult, durationSec float64) { + validateTelemetryOperation(operation) + validateTelemetryResult(result) + TelemetryOperationTotal.WithLabelValues(string(operation), string(result)).Inc() + if result == TelemetrySuccess || result == TelemetryError { + TelemetryOperationDuration.WithLabelValues(string(operation)).Observe(math.Max(durationSec, 0)) } } -// UpdateLakeStats updates lake storage metrics -func UpdateLakeStats(signal string, bytes int64, partitions int) { - LakeSize.WithLabelValues(signal).Set(float64(bytes)) - LakePartitions.WithLabelValues(signal).Set(float64(partitions)) +// UpdateParquetStats updates open Parquet storage metrics. +func UpdateParquetStats(signal string, bytes int64, partitions int) { + ParquetSize.WithLabelValues(signal).Set(float64(bytes)) + ParquetFiles.WithLabelValues(signal).Set(float64(partitions)) } // UpdateQueueDepth updates queue depth metric @@ -334,20 +345,20 @@ func validateRollupResult(result RollupResult) { } } -func validateDuckLakeOperation(operation DuckLakeOperation) { +func validateTelemetryOperation(operation TelemetryOperation) { switch operation { - case DuckLakeMerge, DuckLakeMaintenance: + case TelemetryCompaction, TelemetryMaintenance: return default: - panic("metrics: invalid DuckLake operation: " + string(operation)) + panic("metrics: invalid Telemetry operation: " + string(operation)) } } -func validateDuckLakeResult(result DuckLakeResult) { +func validateTelemetryResult(result TelemetryResult) { switch result { - case DuckLakeSuccess, DuckLakeError, DuckLakeDisabled, DuckLakeThrottled: + case TelemetrySuccess, TelemetryError, TelemetryDisabled, TelemetryThrottled, TelemetryNoop: return default: - panic("metrics: invalid DuckLake result: " + string(result)) + panic("metrics: invalid Telemetry result: " + string(result)) } } diff --git a/internal/metrics/metrics_test.go b/internal/metrics/metrics_test.go index 2acba47d..c6e71e74 100644 --- a/internal/metrics/metrics_test.go +++ b/internal/metrics/metrics_test.go @@ -31,22 +31,15 @@ func TestRecordIngest(t *testing.T) { func TestRecordFlush(t *testing.T) { // Reset metrics FlushTotal.Reset() - FlushBytes.Reset() - RecordFlush("spans", 1024, 0.5) - RecordFlush("spans", 2048, 0.3) + RecordFlush("spans", 0.5) + RecordFlush("spans", 0.3) // Check counter incremented flushCount := testutil.ToFloat64(FlushTotal.WithLabelValues("spans")) if flushCount != 2 { t.Errorf("FlushTotal[spans] = %f, want 2", flushCount) } - - // Check bytes accumulated - bytesCount := testutil.ToFloat64(FlushBytes.WithLabelValues("spans")) - if bytesCount != 3072 { - t.Errorf("FlushBytes[spans] = %f, want 3072", bytesCount) - } } func TestRecordQuery(t *testing.T) { @@ -136,42 +129,42 @@ func TestRecordRollupComponentAndProgress(t *testing.T) { } } -func TestRecordDuckLakeOperationOutcomes(t *testing.T) { - DuckLakeOperationTotal.Reset() - DuckLakeOperationDuration.Reset() +func TestRecordTelemetryOperationOutcomes(t *testing.T) { + TelemetryOperationTotal.Reset() + TelemetryOperationDuration.Reset() - RecordDuckLakeOperation(DuckLakeMerge, DuckLakeSuccess, 0.2) - RecordDuckLakeOperation(DuckLakeMerge, DuckLakeThrottled, 0) - RecordDuckLakeOperation(DuckLakeMerge, DuckLakeDisabled, 0) - RecordDuckLakeOperation(DuckLakeMaintenance, DuckLakeError, 1.5) + RecordTelemetryOperation(TelemetryCompaction, TelemetrySuccess, 0.2) + RecordTelemetryOperation(TelemetryCompaction, TelemetryThrottled, 0) + RecordTelemetryOperation(TelemetryCompaction, TelemetryDisabled, 0) + RecordTelemetryOperation(TelemetryMaintenance, TelemetryError, 1.5) for _, tc := range []struct { operation string result string }{ - {"merge", "success"}, - {"merge", "throttled"}, - {"merge", "disabled"}, + {"compaction", "success"}, + {"compaction", "throttled"}, + {"compaction", "disabled"}, {"maintenance", "error"}, } { - if got := testutil.ToFloat64(DuckLakeOperationTotal.WithLabelValues(tc.operation, tc.result)); got != 1 { - t.Errorf("DuckLakeOperationTotal[%s,%s] = %f, want 1", tc.operation, tc.result, got) + if got := testutil.ToFloat64(TelemetryOperationTotal.WithLabelValues(tc.operation, tc.result)); got != 1 { + t.Errorf("TelemetryOperationTotal[%s,%s] = %f, want 1", tc.operation, tc.result, got) } } - if got := histogramSampleCount(t, "fanout_ducklake_operation_duration_seconds", "operation", "merge"); got != 1 { + if got := histogramSampleCount(t, "fanout_telemetry_operation_duration_seconds", "operation", "compaction"); got != 1 { t.Errorf("merge duration samples = %d, want 1 (skipped outcomes must not distort duration)", got) } - if got := histogramSampleCount(t, "fanout_ducklake_operation_duration_seconds", "operation", "maintenance"); got != 1 { + if got := histogramSampleCount(t, "fanout_telemetry_operation_duration_seconds", "operation", "maintenance"); got != 1 { t.Errorf("maintenance duration samples = %d, want 1", got) } } func TestBoundedMetricLabelsRejectUnknownValues(t *testing.T) { for name, call := range map[string]func(){ - "rollup component": func() { RecordRollupComponent(RollupComponent("tenant"), RollupSuccess, 0, 0) }, - "rollup result": func() { RecordRollupComponent(RollupService, RollupResult("unknown"), 0, 0) }, - "lake operation": func() { RecordDuckLakeOperation(DuckLakeOperation("query"), DuckLakeSuccess, 0) }, - "lake result": func() { RecordDuckLakeOperation(DuckLakeMerge, DuckLakeResult("unknown"), 0) }, + "rollup component": func() { RecordRollupComponent(RollupComponent("tenant"), RollupSuccess, 0, 0) }, + "rollup result": func() { RecordRollupComponent(RollupService, RollupResult("unknown"), 0, 0) }, + "telemetry operation": func() { RecordTelemetryOperation(TelemetryOperation("query"), TelemetrySuccess, 0) }, + "telemetry result": func() { RecordTelemetryOperation(TelemetryCompaction, TelemetryResult("unknown"), 0) }, } { t.Run(name, func(t *testing.T) { defer func() { @@ -184,30 +177,30 @@ func TestBoundedMetricLabelsRejectUnknownValues(t *testing.T) { } } -func TestUpdateLakeStats(t *testing.T) { +func TestUpdateParquetStats(t *testing.T) { // Reset metrics - LakeSize.Reset() - LakePartitions.Reset() + ParquetSize.Reset() + ParquetFiles.Reset() - UpdateLakeStats("spans", 1024*1024, 10) - UpdateLakeStats("logs", 512*1024, 5) + UpdateParquetStats("spans", 1024*1024, 10) + UpdateParquetStats("logs", 512*1024, 5) // Check spans size - spansSize := testutil.ToFloat64(LakeSize.WithLabelValues("spans")) + spansSize := testutil.ToFloat64(ParquetSize.WithLabelValues("spans")) if spansSize != 1024*1024 { - t.Errorf("LakeSize[spans] = %f, want %d", spansSize, 1024*1024) + t.Errorf("ParquetSize[spans] = %f, want %d", spansSize, 1024*1024) } // Check spans partitions - spansPartitions := testutil.ToFloat64(LakePartitions.WithLabelValues("spans")) + spansPartitions := testutil.ToFloat64(ParquetFiles.WithLabelValues("spans")) if spansPartitions != 10 { - t.Errorf("LakePartitions[spans] = %f, want 10", spansPartitions) + t.Errorf("ParquetFiles[spans] = %f, want 10", spansPartitions) } // Check logs size - logsSize := testutil.ToFloat64(LakeSize.WithLabelValues("logs")) + logsSize := testutil.ToFloat64(ParquetSize.WithLabelValues("logs")) if logsSize != 512*1024 { - t.Errorf("LakeSize[logs] = %f, want %d", logsSize, 512*1024) + t.Errorf("ParquetSize[logs] = %f, want %d", logsSize, 512*1024) } } diff --git a/internal/observability/logs.go b/internal/observability/logs.go index e4d2300c..216ee973 100644 --- a/internal/observability/logs.go +++ b/internal/observability/logs.go @@ -6,36 +6,30 @@ import ( "strings" ) -// The user-supplied search filter must compare against the REDACTED body, -// not the raw column: a telemetry viewer could otherwise confirm a secret's -// presence by probing search= even though the display shows -// [REDACTED]. redactedBodySQL replays the exact -// Go-side patterns inside DuckDB (same RE2 engine; parity pinned by -// TestRedactSQLMatchesGo), and both the row query and the histogram query -// use it so their counts can never disagree and leak the same signal. -var redactedBodySQL = redactLogBodySQL("COALESCE(body, '')") - -var logsEntriesQuery = ` -SELECT time, COALESCE(severity, ''), COALESCE(service, ''), COALESCE(body, ''), - COALESCE(trace_id, ''), COALESCE(span_id, '') -FROM logs -WHERE time >= ? AND time < ? AND (? = '' OR namespace = ?) +var logFilters = ` +WHERE time >= ? AND time < ? + AND (? = '' OR namespace = ?) AND (? = '' OR service = ?) - AND (? = '' OR upper(severity) = upper(?)) - AND (? = '' OR ` + redactedBodySQL + ` ILIKE ?) + AND (? = '' OR lower(severity) = lower(?)) + AND (? = '' OR contains(lower(` + redactLogBodySQL("body") + `), lower(?)))` + +// DuckDB answers the two questions the API asks: the newest `limit` entries +// and per-bucket counts. LIMIT bounds the entry stream and GROUP BY bounds the +// histogram stream regardless of the retained Parquet row count. +var logEntriesQuery = ` +SELECT time, severity, coalesce(service, ''), ` + redactLogBodySQL("body") + `, + coalesce(trace_id, ''), coalesce(span_id, '') + FROM logs` + logFilters + ` ORDER BY time DESC LIMIT ?` -var logsBucketsQuery = ` +var logBucketsQuery = ` SELECT time_bucket(INTERVAL '5 minutes', time) AS point_time, - COALESCE(NULLIF(upper(severity), ''), 'UNSPECIFIED'), CAST(COUNT(*) AS BIGINT) -FROM logs -WHERE time >= ? AND time < ? AND (? = '' OR namespace = ?) - AND (? = '' OR service = ?) - AND (? = '' OR upper(severity) = upper(?)) - AND (? = '' OR ` + redactedBodySQL + ` ILIKE ?) -GROUP BY point_time, severity -ORDER BY point_time ASC, severity ASC` + coalesce(nullif(upper(severity), ''), 'UNSPECIFIED') AS bucket_severity, + CAST(count(*) AS BIGINT) + FROM logs` + logFilters + ` +GROUP BY point_time, bucket_severity +ORDER BY point_time ASC, bucket_severity ASC` func (s *Service) Logs(ctx context.Context, scope Scope, service, severity, search string, limit int) (Result[Logs], error) { scope, err := s.normalizeScope(scope) @@ -46,16 +40,11 @@ func (s *Service) Logs(ctx context.Context, scope Scope, service, severity, sear if err != nil { return Result[Logs]{}, err } - service = strings.TrimSpace(service) - severity = strings.TrimSpace(severity) - search = strings.TrimSpace(search) - pattern := search - if pattern != "" { - pattern = "%" + pattern + "%" - } - + service, severity, search = strings.TrimSpace(service), strings.TrimSpace(severity), strings.TrimSpace(search) + search = strings.ToLower(search) data := Logs{Entries: []LogEntry{}, Buckets: []LogBucket{}} - rows, err := s.db.QueryContext(ctx, logsEntriesQuery, scope.Start, scope.End, scope.Namespace, scope.Namespace, service, service, severity, severity, search, pattern, limit) + filters := []any{scope.Start, scope.End, scope.Namespace, scope.Namespace, service, service, severity, severity, search, search} + rows, err := s.db.QueryContext(ctx, logEntriesQuery, append(append([]any{}, filters...), limit)...) if err != nil { return Result[Logs]{}, fmt.Errorf("query logs: %w", err) } @@ -65,6 +54,8 @@ func (s *Service) Logs(ctx context.Context, scope Scope, service, severity, sear rows.Close() return Result[Logs]{}, fmt.Errorf("scan log: %w", err) } + // Keep Go redaction as a defense-in-depth boundary even though DuckDB + // applies the equivalent expression before filtering and transfer. entry.Body = redactLogBody(entry.Body) data.Entries = append(data.Entries, entry) } @@ -74,28 +65,28 @@ func (s *Service) Logs(ctx context.Context, scope Scope, service, severity, sear } rows.Close() - rows, err = s.db.QueryContext(ctx, logsBucketsQuery, scope.Start, scope.End, scope.Namespace, scope.Namespace, service, service, severity, severity, search, pattern) + matched := 0 + bucketRows, err := s.db.QueryContext(ctx, logBucketsQuery, filters...) if err != nil { return Result[Logs]{}, fmt.Errorf("query log histogram: %w", err) } - for rows.Next() { + for bucketRows.Next() { var bucket LogBucket - if err := rows.Scan(&bucket.Time, &bucket.Severity, &bucket.Count); err != nil { - rows.Close() - return Result[Logs]{}, fmt.Errorf("scan log histogram: %w", err) + if err := bucketRows.Scan(&bucket.Time, &bucket.Severity, &bucket.Count); err != nil { + bucketRows.Close() + return Result[Logs]{}, fmt.Errorf("scan log bucket: %w", err) } + bucket.Time = bucket.Time.UTC() + matched += int(bucket.Count) data.Buckets = append(data.Buckets, bucket) } - if err := rows.Err(); err != nil { - rows.Close() + if err := bucketRows.Err(); err != nil { + bucketRows.Close() return Result[Logs]{}, fmt.Errorf("iterate log histogram: %w", err) } - rows.Close() - + bucketRows.Close() return Result[Logs]{ - Schema: LogsSchema, - Summary: fmt.Sprintf("%d logs matched the selected telemetry window", len(data.Entries)), - Data: data, - Provenance: s.provenanceFor(scope, "logs"), + Schema: LogsSchema, Summary: fmt.Sprintf("%d logs matched the selected telemetry window", matched), + Data: data, Provenance: s.provenanceFor(scope, "parquet"), }, nil } diff --git a/internal/observability/namespace_test.go b/internal/observability/namespace_test.go index 5a62d33b..db74c7a0 100644 --- a/internal/observability/namespace_test.go +++ b/internal/observability/namespace_test.go @@ -38,7 +38,7 @@ CREATE TABLE service_rollup ( t.Fatalf("insert service rollups: %v", err) } - svc := New(db) + svc := New(SQLDB(db), newTestRepository(t).Parquet) result, err := svc.Overview(context.Background(), Scope{Start: stamp.Add(-time.Minute), End: stamp.Add(time.Minute)}, 100) if err != nil { t.Fatalf("Overview: %v", err) diff --git a/internal/observability/performance_benchmark_test.go b/internal/observability/performance_benchmark_test.go index 57fc1a56..b1ab6b26 100644 --- a/internal/observability/performance_benchmark_test.go +++ b/internal/observability/performance_benchmark_test.go @@ -19,9 +19,6 @@ func BenchmarkEndpointQueries24Hours(b *testing.B) { if _, err := db.Exec(`SET TimeZone='UTC'; SET threads=4`); err != nil { b.Fatal(err) } - if _, err := db.Exec(`ATTACH ':memory:' AS lake`); err != nil { - b.Fatal(err) - } if err := query.CreateTables(db); err != nil { b.Fatal(err) } @@ -29,7 +26,7 @@ func BenchmarkEndpointQueries24Hours(b *testing.B) { b.Fatal(err) } start := time.Date(2026, 7, 20, 0, 0, 30, 0, time.UTC) - if _, err := db.Exec(`INSERT INTO lake.spans ( + if _, err := db.Exec(`INSERT INTO telemetry.spans ( namespace, service, start_time, duration_ms, status, http_method, http_route, operation ) SELECT diff --git a/internal/observability/performance_rollup_test.go b/internal/observability/performance_rollup_test.go index 79a61ccf..5d13e3e6 100644 --- a/internal/observability/performance_rollup_test.go +++ b/internal/observability/performance_rollup_test.go @@ -21,9 +21,6 @@ func TestEndpointRollupQueryMergesBucketsAndExactBoundaries(t *testing.T) { if _, err := db.Exec(`SET TimeZone='UTC'`); err != nil { t.Fatalf("set timezone: %v", err) } - if _, err := db.Exec(`ATTACH ':memory:' AS lake`); err != nil { - t.Fatalf("attach lake catalog: %v", err) - } if err := query.CreateTables(db); err != nil { t.Fatalf("CreateTables: %v", err) } @@ -65,7 +62,7 @@ FROM (VALUES ` + seed.values + `) t(ms)` // Include raw rows for every minute. The query must use raw rows for the two // partial boundaries and for complete minutes newer than the watermark, while // excluding raw rows already represented by mature cached buckets. - if _, err := db.Exec(`INSERT INTO lake.spans ( + if _, err := db.Exec(`INSERT INTO telemetry.spans ( namespace, service, start_time, duration_ms, status, http_method, http_route, operation ) VALUES ('prod','checkout',?,5.0,'OK','GET','/pay','GET /pay'), @@ -93,7 +90,7 @@ FROM (VALUES ` + seed.values + `) t(ms)` t.Fatalf("seed endpoint rollup state: %v", err) } - svc := New(db) + svc := New(SQLDB(db), newTestRepository(t).Parquet) svc.endpointMature.Store(true) var cachedCalls, totalCachedCalls int64 var minBucket, maxBucket time.Time diff --git a/internal/observability/service.go b/internal/observability/service.go index 6ee193b8..efd1aa15 100644 --- a/internal/observability/service.go +++ b/internal/observability/service.go @@ -2,7 +2,6 @@ package observability import ( "context" - "database/sql" "errors" "fmt" "strings" @@ -10,6 +9,8 @@ import ( "time" appid "github.com/labstack/fanout/internal/id" + "github.com/labstack/fanout/internal/queryrows" + "github.com/labstack/fanout/internal/telemetry" ) const ( @@ -24,22 +25,30 @@ var ( ErrInvalidLimit = errors.New("invalid observability query limit") ) -// DB is the narrow database surface used by the query kernel. *sql.DB -// satisfies it, while tests can use sqlmock without a storage-specific fake. -type DB interface { - QueryContext(ctx context.Context, query string, args ...any) (*sql.Rows, error) +type DB = queryrows.Queryer + +type traceReader interface { + Trace(context.Context, telemetry.TraceQuery) ([]telemetry.IndexedSpan, error) } type Service struct { db DB + repository traceReader now func() time.Time endpointMature atomic.Bool } -func New(db DB) *Service { - return &Service{db: db, now: time.Now} +func New(db DB, repository traceReader) *Service { + if db == nil || repository == nil { + panic("observability requires query engine and indexed trace reader") + } + return &Service{db: db, repository: repository, now: time.Now} } +// SQLDB adapts a standard database/sql queryer for tests and callers that do +// not need storage-engine row-lifetime hooks. +func SQLDB(db queryrows.SQLQueryer) DB { return queryrows.SQLAdapter{DB: db} } + func (s *Service) normalizeScope(scope Scope) (Scope, error) { now := s.now().UTC() if scope.End.IsZero() { diff --git a/internal/observability/service_test.go b/internal/observability/service_test.go index 011b538f..b397f502 100644 --- a/internal/observability/service_test.go +++ b/internal/observability/service_test.go @@ -2,29 +2,43 @@ package observability import ( "context" - "database/sql" "errors" "regexp" + "strings" "testing" "time" "github.com/DATA-DOG/go-sqlmock" + "github.com/labstack/fanout/internal/queryrows" + "github.com/labstack/fanout/internal/telemetry" + telemetrystore "github.com/labstack/fanout/internal/telemetry/store" ) -func newMockService(t *testing.T) (*Service, sqlmock.Sqlmock) { +func newTestRepository(t *testing.T) *telemetrystore.Repository { + t.Helper() + repository, err := telemetrystore.Open(t.TempDir()) + if err != nil { + t.Fatalf("open telemetry repository: %v", err) + } + t.Cleanup(func() { _ = repository.Close() }) + return repository +} + +func newMockService(t *testing.T) (*Service, sqlmock.Sqlmock, *telemetrystore.Repository) { t.Helper() db, mock, err := sqlmock.New() if err != nil { t.Fatalf("sqlmock.New: %v", err) } t.Cleanup(func() { _ = db.Close() }) - svc := New(db) + repository := newTestRepository(t) + svc := New(SQLDB(db), repository.Parquet) svc.now = func() time.Time { return time.Date(2026, 7, 20, 12, 0, 0, 0, time.UTC) } - return svc, mock + return svc, mock, repository } func TestNormalizeScopeDefaultsAndBounds(t *testing.T) { - svc, _ := newMockService(t) + svc, _, _ := newMockService(t) scope, err := svc.normalizeScope(Scope{}) if err != nil { t.Fatalf("normalizeScope: %v", err) @@ -43,7 +57,7 @@ func TestNormalizeScopeDefaultsAndBounds(t *testing.T) { } func TestOverviewReturnsCanonicalEnvelope(t *testing.T) { - svc, mock := newMockService(t) + svc, mock, _ := newMockService(t) start := time.Date(2026, 7, 20, 11, 0, 0, 0, time.UTC) end := start.Add(time.Hour) rows := sqlmock.NewRows([]string{"service", "spans", "error_rate", "p50_ms", "p95_ms", "log_count", "metric_count"}). @@ -75,7 +89,7 @@ func TestOverviewReturnsCanonicalEnvelope(t *testing.T) { } func TestTopologyUsesSharedNodesAndTypedEdges(t *testing.T) { - svc, mock := newMockService(t) + svc, mock, _ := newMockService(t) start := time.Date(2026, 7, 20, 11, 0, 0, 0, time.UTC) end := start.Add(time.Hour) mock.ExpectQuery(regexp.QuoteMeta(overviewQuery)). @@ -104,7 +118,7 @@ func TestTopologyUsesSharedNodesAndTypedEdges(t *testing.T) { } func TestPerformanceReturnsAllVisualizationDatasets(t *testing.T) { - svc, mock := newMockService(t) + svc, mock, _ := newMockService(t) svc.endpointMature.Store(true) start := time.Date(2026, 7, 20, 11, 0, 0, 0, time.UTC) end := start.Add(time.Hour) @@ -150,7 +164,7 @@ func TestPerformanceReturnsAllVisualizationDatasets(t *testing.T) { } func TestQueryEndpointsFallsBackToRawUntilBackfillReady(t *testing.T) { - svc, mock := newMockService(t) + svc, mock, _ := newMockService(t) start := time.Date(2026, 7, 20, 11, 0, 0, 0, time.UTC) end := start.Add(time.Hour) @@ -174,7 +188,7 @@ func TestQueryEndpointsFallsBackToRawUntilBackfillReady(t *testing.T) { } func TestQueryEndpointsFallsBackToRawWhenRollupProbeFails(t *testing.T) { - svc, mock := newMockService(t) + svc, mock, _ := newMockService(t) start := time.Date(2026, 7, 20, 11, 0, 0, 0, time.UTC) end := start.Add(time.Hour) @@ -198,19 +212,23 @@ func TestQueryEndpointsFallsBackToRawWhenRollupProbeFails(t *testing.T) { } func TestTraceSelectsRecentErrorAndCorrelatesLogs(t *testing.T) { - svc, mock := newMockService(t) + svc, mock, repository := newMockService(t) start := time.Date(2026, 7, 20, 11, 0, 0, 0, time.UTC) end := start.Add(time.Hour) mock.ExpectQuery(regexp.QuoteMeta(recentTraceQuery)). WithArgs(start, end, "prod", "prod", "checkout", "checkout"). WillReturnRows(sqlmock.NewRows([]string{"trace_id"}).AddRow("trace-1")) - mock.ExpectQuery(regexp.QuoteMeta(traceSpansQuery)). - WithArgs(start, end, "prod", "prod", "trace-1", 20). - WillReturnRows(sqlmock.NewRows([]string{"span_id", "parent_span_id", "service", "operation", "kind", "start_time", "duration_ms", "status", "status_message"}). - AddRow("root", "", "checkout", "POST /pay", "SERVER", start, 200.0, "ERROR", "declined"). - AddRow("child", "root", "payments", "charge", "CLIENT", start.Add(20*time.Millisecond), 80.0, "OK", "")) + if err := repository.Commit(context.Background(), telemetrystore.Batch{ID: "trace-fixture", Spans: []telemetry.Span{ + {Namespace: "prod", TraceID: "trace-1", SpanID: "root", ServiceName: "checkout", Name: "POST /pay", Kind: "SERVER", StartUnixNanos: start.UnixNano(), DurationMS: 200, StatusCode: "ERROR", StatusMsg: "declined"}, + {Namespace: "prod", TraceID: "trace-1", SpanID: "child", ParentSpanID: "root", ServiceName: "payments", Name: "charge", Kind: "CLIENT", StartUnixNanos: start.Add(20 * time.Millisecond).UnixNano(), DurationMS: 80, StatusCode: "OK"}, + }, Logs: []telemetry.Log{ + {Namespace: "prod", TimeUnixNanos: start.Add(150 * time.Millisecond).UnixNano(), Severity: "ERROR", ServiceName: "checkout", Body: "payment declined", TraceID: "trace-1", SpanID: "root"}, + {Namespace: "prod", TimeUnixNanos: start.Add(160 * time.Millisecond).UnixNano(), Severity: "ERROR", ServiceName: "payments", Body: `charge failed: token=abc123 {"client_secret":"cs_live_9"}`, TraceID: "trace-1", SpanID: "child"}, + }}); err != nil { + t.Fatalf("commit trace fixture: %v", err) + } mock.ExpectQuery(regexp.QuoteMeta(traceLogsQuery)). - WithArgs(start, end, "prod", "prod", "trace-1", 20). + WithArgs("trace-1", start, end, "prod", "prod", 20). WillReturnRows(sqlmock.NewRows([]string{"time", "severity", "service", "body", "trace_id", "span_id"}). AddRow(start.Add(150*time.Millisecond), "ERROR", "checkout", "payment declined", "trace-1", "root"). AddRow(start.Add(160*time.Millisecond), "ERROR", "payments", `charge failed: token=abc123 {"client_secret":"cs_live_9"}`, "trace-1", "child")) @@ -234,19 +252,27 @@ func TestTraceSelectsRecentErrorAndCorrelatesLogs(t *testing.T) { } func TestLogsAppliesFiltersAndBuildsHistogram(t *testing.T) { - svc, mock := newMockService(t) + svc, mock, repository := newMockService(t) start := time.Date(2026, 7, 20, 11, 0, 0, 0, time.UTC) end := start.Add(time.Hour) - mock.ExpectQuery(regexp.QuoteMeta(logsEntriesQuery)). - WithArgs(start, end, "prod", "prod", "checkout", "checkout", "error", "error", "declined", "%declined%", 10). + if err := repository.Commit(context.Background(), telemetrystore.Batch{ID: "logs-fixture", Logs: []telemetry.Log{ + {Namespace: "prod", TimeUnixNanos: start.UnixNano(), Severity: "ERROR", ServiceName: "checkout", Body: "payment declined", TraceID: "trace-1", SpanID: "root"}, + {Namespace: "prod", TimeUnixNanos: start.Add(time.Millisecond).UnixNano(), Severity: "ERROR", ServiceName: "checkout", Body: "card declined: token=abc123", TraceID: "trace-2", SpanID: "root2"}, + {Namespace: "prod", TimeUnixNanos: start.Add(2 * time.Millisecond).UnixNano(), Severity: "ERROR", ServiceName: "checkout", Body: `auth declined: {"password":"hunter2"}`, TraceID: "trace-3", SpanID: "root3"}, + }}); err != nil { + t.Fatalf("commit logs fixture: %v", err) + } + mock.ExpectQuery(regexp.QuoteMeta(logEntriesQuery)). + WithArgs(start, end, "prod", "prod", "checkout", "checkout", "error", "error", "declined", "declined", 10). WillReturnRows(sqlmock.NewRows([]string{"time", "severity", "service", "body", "trace_id", "span_id"}). - AddRow(start, "ERROR", "checkout", "payment declined", "trace-1", "root"). - AddRow(start, "ERROR", "checkout", "card declined: token=abc123", "trace-2", "root2"). - AddRow(start, "ERROR", "checkout", `auth declined: {"password":"hunter2"}`, "trace-3", "root3")) - mock.ExpectQuery(regexp.QuoteMeta(logsBucketsQuery)). - WithArgs(start, end, "prod", "prod", "checkout", "checkout", "error", "error", "declined", "%declined%"). - WillReturnRows(sqlmock.NewRows([]string{"point_time", "severity", "count"}). - AddRow(start, "ERROR", int64(3))) + // Return raw values to prove the Go boundary still redacts even if the + // SQL expression and driver ever diverge. + AddRow(start.Add(2*time.Millisecond), "ERROR", "checkout", `auth declined: {"password":"hunter2"}`, "trace-3", "root3"). + AddRow(start.Add(time.Millisecond), "ERROR", "checkout", "card declined: token=abc123", "trace-2", "root2"). + AddRow(start, "ERROR", "checkout", "payment declined", "trace-1", "root")) + mock.ExpectQuery(regexp.QuoteMeta(logBucketsQuery)). + WithArgs(start, end, "prod", "prod", "checkout", "checkout", "error", "error", "declined", "declined"). + WillReturnRows(sqlmock.NewRows([]string{"point_time", "severity", "count"}).AddRow(start, "ERROR", int64(3))) result, err := svc.Logs(context.Background(), Scope{Namespace: "prod", Start: start, End: end}, "checkout", "error", "declined", 10) if err != nil { @@ -258,12 +284,297 @@ func TestLogsAppliesFiltersAndBuildsHistogram(t *testing.T) { if want := "card declined: token=[REDACTED]"; result.Data.Entries[1].Body != want { t.Fatalf("log body = %q, want %q (redaction bypassed)", result.Data.Entries[1].Body, want) } - if want := `auth declined: {"password":"[REDACTED]"}`; result.Data.Entries[2].Body != want { - t.Fatalf("log body = %q, want %q (redaction bypassed)", result.Data.Entries[2].Body, want) + if want := `auth declined: {"password":"[REDACTED]"}`; result.Data.Entries[0].Body != want { + t.Fatalf("log body = %q, want %q (redaction bypassed)", result.Data.Entries[0].Body, want) } if err := mock.ExpectationsWereMet(); err != nil { t.Fatal(err) } } -var _ DB = (*sql.DB)(nil) +func TestLogsRetainsOnlyNewestLimit(t *testing.T) { + svc, mock, repository := newMockService(t) + start := time.Date(2026, 7, 20, 11, 0, 0, 0, time.UTC) + logs := make([]telemetry.Log, 100) + for i := range logs { + logs[i] = telemetry.Log{Namespace: "prod", TimeUnixNanos: start.Add(time.Duration(i) * time.Millisecond).UnixNano(), Severity: "INFO", Body: "entry"} + } + if err := repository.Commit(context.Background(), telemetrystore.Batch{ID: "bounded-logs", Logs: logs}); err != nil { + t.Fatal(err) + } + entryRows := sqlmock.NewRows([]string{"time", "severity", "service", "body", "trace_id", "span_id"}) + for i := 99; i >= 95; i-- { + entryRows.AddRow(start.Add(time.Duration(i)*time.Millisecond), "INFO", "", "entry", "", "") + } + mock.ExpectQuery(regexp.QuoteMeta(logEntriesQuery)). + WithArgs(start, start.Add(time.Hour), "prod", "prod", "", "", "", "", "", "", 5). + WillReturnRows(entryRows) + mock.ExpectQuery(regexp.QuoteMeta(logBucketsQuery)). + WithArgs(start, start.Add(time.Hour), "prod", "prod", "", "", "", "", "", ""). + WillReturnRows(sqlmock.NewRows([]string{"point_time", "severity", "count"}).AddRow(start, "INFO", int64(100))) + result, err := svc.Logs(context.Background(), Scope{Namespace: "prod", Start: start, End: start.Add(time.Hour)}, "", "", "", 5) + if err != nil { + t.Fatal(err) + } + if len(result.Data.Entries) != 5 || !result.Data.Entries[0].Time.Equal(start.Add(99*time.Millisecond)) || !result.Data.Entries[4].Time.Equal(start.Add(95*time.Millisecond)) { + t.Fatalf("newest bounded entries = %#v", result.Data.Entries) + } +} + +func TestLogsAlwaysUseAuthoritativeParquet(t *testing.T) { + svc, mock, repository := newMockService(t) + start := time.Date(2026, 7, 20, 11, 0, 0, 0, time.UTC) + end := start.Add(time.Hour) + if err := repository.Commit(context.Background(), telemetrystore.Batch{ID: "parquet-logs", Logs: []telemetry.Log{{ + Namespace: "prod", TimeUnixNanos: start.Add(time.Minute).UnixNano(), Severity: "ERROR", + ServiceName: "checkout", Body: "token=secret", TraceID: "trace-parquet", + }}}); err != nil { + t.Fatal(err) + } + mock.ExpectQuery(regexp.QuoteMeta(logEntriesQuery)). + WithArgs(start, end, "prod", "prod", "checkout", "checkout", "error", "error", "", "", 10). + WillReturnRows(sqlmock.NewRows([]string{"time", "severity", "service", "body", "trace_id", "span_id"}). + AddRow(start.Add(time.Minute), "ERROR", "checkout", "token=[REDACTED]", "trace-parquet", "")) + mock.ExpectQuery(regexp.QuoteMeta(logBucketsQuery)). + WithArgs(start, end, "prod", "prod", "checkout", "checkout", "error", "error", "", ""). + WillReturnRows(sqlmock.NewRows([]string{"point_time", "severity", "count"}). + AddRow(start, "ERROR", int64(1))) + result, err := svc.Logs(context.Background(), Scope{Namespace: "prod", Start: start, End: end}, "checkout", "error", "", 10) + if err != nil { + t.Fatal(err) + } + if len(result.Data.Entries) != 1 || result.Data.Entries[0].Body != "token=[REDACTED]" || result.Provenance.DataSource != "parquet" { + t.Fatalf("Parquet logs result = %#v", result) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatal(err) + } +} + +func TestLogsQueryParquetAcrossBatches(t *testing.T) { + svc, mock, repository := newMockService(t) + start := time.Date(2026, 7, 20, 11, 0, 0, 0, time.UTC) + end := start.Add(time.Second) + if err := repository.Commit(context.Background(), telemetrystore.Batch{ID: "overlap-newer", Logs: []telemetry.Log{ + {Namespace: "prod", TimeUnixNanos: start.Add(150 * time.Millisecond).UnixNano(), Body: "newer-old", Severity: "INFO"}, + {Namespace: "prod", TimeUnixNanos: start.Add(300 * time.Millisecond).UnixNano(), Body: "newer-batch", Severity: "INFO"}, + }}); err != nil { + t.Fatal(err) + } + if err := repository.Commit(context.Background(), telemetrystore.Batch{ID: "overlap-late", Logs: []telemetry.Log{ + {Namespace: "prod", TimeUnixNanos: start.Add(100 * time.Millisecond).UnixNano(), Body: "late-old", Severity: "INFO"}, + {Namespace: "prod", TimeUnixNanos: start.Add(200 * time.Millisecond).UnixNano(), Body: "late-boundary", Severity: "INFO"}, + }}); err != nil { + t.Fatal(err) + } + mock.ExpectQuery(regexp.QuoteMeta(logEntriesQuery)). + WithArgs(start, end, "prod", "prod", "", "", "", "", "", "", 10). + WillReturnRows(sqlmock.NewRows([]string{"time", "severity", "service", "body", "trace_id", "span_id"}). + AddRow(start.Add(300*time.Millisecond), "INFO", "", "newer-batch", "", ""). + AddRow(start.Add(200*time.Millisecond), "INFO", "", "late-boundary", "", ""). + AddRow(start.Add(150*time.Millisecond), "INFO", "", "newer-old", "", ""). + AddRow(start.Add(100*time.Millisecond), "INFO", "", "late-old", "", "")) + mock.ExpectQuery(regexp.QuoteMeta(logBucketsQuery)). + WithArgs(start, end, "prod", "prod", "", "", "", "", "", ""). + WillReturnRows(sqlmock.NewRows([]string{"point_time", "severity", "count"}). + AddRow(start, "INFO", int64(4))) + result, err := svc.Logs(context.Background(), Scope{Namespace: "prod", Start: start, End: end}, "", "", "", 10) + if err != nil { + t.Fatal(err) + } + if len(result.Data.Entries) != 4 || result.Summary != "4 logs matched the selected telemetry window" { + t.Fatalf("boundary logs = %#v", result) + } + if result.Data.Entries[0].Body != "newer-batch" || result.Provenance.DataSource != "parquet" { + t.Fatalf("boundary result = %#v", result) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatal(err) + } +} + +func TestTraceLogsUseFullScopeEventTimeAcrossBatches(t *testing.T) { + svc, mock, repository := newMockService(t) + start := time.Date(2026, 7, 20, 11, 0, 0, 0, time.UTC) + batches := []telemetrystore.Batch{ + {ID: "trace-latest", Spans: []telemetry.Span{{Namespace: "prod", TraceID: "trace-order", SpanID: "root", StartUnixNanos: start.UnixNano(), DurationMS: 1}}, Logs: []telemetry.Log{{Namespace: "prod", TraceID: "trace-order", TimeUnixNanos: start.Add(30 * time.Millisecond).UnixNano(), Body: "latest"}}}, + {ID: "trace-earliest", Logs: []telemetry.Log{{Namespace: "prod", TraceID: "trace-order", TimeUnixNanos: start.Add(10 * time.Millisecond).UnixNano(), Body: "earliest"}}}, + {ID: "trace-middle", Logs: []telemetry.Log{{Namespace: "prod", TraceID: "trace-order", TimeUnixNanos: start.Add(20 * time.Millisecond).UnixNano(), Body: "middle"}}}, + {ID: "trace-outside-span", Logs: []telemetry.Log{{Namespace: "prod", TraceID: "trace-order", TimeUnixNanos: start.Add(2 * time.Minute).UnixNano(), Body: "unrelated later event"}}}, + } + for _, batch := range batches { + if err := repository.Commit(context.Background(), batch); err != nil { + t.Fatal(err) + } + } + mock.ExpectQuery(regexp.QuoteMeta(traceLogsQuery)). + WithArgs("trace-order", start, start.Add(time.Hour), "prod", "prod", 10). + WillReturnRows(sqlmock.NewRows([]string{"time", "severity", "service", "body", "trace_id", "span_id"}). + AddRow(start.Add(10*time.Millisecond), "", "", "earliest", "trace-order", ""). + AddRow(start.Add(20*time.Millisecond), "", "", "middle", "trace-order", ""). + AddRow(start.Add(30*time.Millisecond), "", "", "latest", "trace-order", ""). + AddRow(start.Add(2*time.Minute), "", "", "unrelated later event", "trace-order", "")) + result, err := svc.Trace(context.Background(), Scope{Namespace: "prod", Start: start, End: start.Add(time.Hour)}, "trace-order", "", 10) + if err != nil { + t.Fatal(err) + } + if len(result.Data.Logs) != 4 || result.Data.Logs[0].Body != "earliest" || result.Data.Logs[3].Body != "unrelated later event" { + t.Fatalf("trace logs = %#v", result.Data.Logs) + } +} + +func TestTraceUsesIndexedParquet(t *testing.T) { + svc, mock, repository := newMockService(t) + start := time.Date(2026, 7, 20, 11, 0, 0, 0, time.UTC) + end := start.Add(time.Hour) + if err := repository.Commit(context.Background(), telemetrystore.Batch{ID: "indexed-trace", Spans: []telemetry.Span{{ + Namespace: "prod", TraceID: "parquet-trace", SpanID: "root", ServiceName: "checkout", Name: "pay", + Kind: "SERVER", StartUnixNanos: start.UnixNano(), DurationMS: 25, StatusCode: "ERROR", StatusMsg: "declined", + }}}); err != nil { + t.Fatal(err) + } + mock.ExpectQuery(regexp.QuoteMeta(traceLogsQuery)). + WithArgs("parquet-trace", start, end, "prod", "prod", 10). + WillReturnRows(sqlmock.NewRows([]string{"time", "severity", "service", "body", "trace_id", "span_id"}). + AddRow(start.Add(time.Millisecond), "ERROR", "checkout", "token=secret", "parquet-trace", "root")) + result, err := svc.Trace(context.Background(), Scope{Namespace: "prod", Start: start, End: end}, "parquet-trace", "", 10) + if err != nil { + t.Fatal(err) + } + if len(result.Data.Spans) != 1 || len(result.Data.Logs) != 1 || !result.Data.HasError { + t.Fatalf("indexed Parquet trace detail = %#v", result.Data) + } + if result.Provenance.DataSource != "parquet_index" { + t.Fatalf("trace data source = %q", result.Provenance.DataSource) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatal(err) + } +} + +func TestTraceCombinesIndexedSpansAcrossBatches(t *testing.T) { + svc, mock, repository := newMockService(t) + start := time.Date(2026, 7, 20, 11, 0, 0, 0, time.UTC) + end := start.Add(time.Hour) + if err := repository.Commit(context.Background(), telemetrystore.Batch{ID: "trace-old-root", Spans: []telemetry.Span{{ + Namespace: "prod", TraceID: "split-trace", SpanID: "root", ServiceName: "frontend", + StartUnixNanos: start.Add(10 * time.Minute).UnixNano(), DurationMS: 100, StatusCode: "ERROR", + }}}); err != nil { + t.Fatal(err) + } + if err := repository.Commit(context.Background(), telemetrystore.Batch{ID: "trace-newer-child", Spans: []telemetry.Span{{ + Namespace: "prod", TraceID: "split-trace", SpanID: "child", ParentSpanID: "root", ServiceName: "backend", + StartUnixNanos: start.Add(50 * time.Minute).UnixNano(), DurationMS: 25, StatusCode: "OK", + }}}); err != nil { + t.Fatal(err) + } + mock.ExpectQuery(regexp.QuoteMeta(traceLogsQuery)). + WithArgs("split-trace", start, end, "prod", "prod", 10). + WillReturnRows(sqlmock.NewRows([]string{"time", "severity", "service", "body", "trace_id", "span_id"})) + result, err := svc.Trace(context.Background(), Scope{Namespace: "prod", Start: start, End: end}, "split-trace", "", 10) + if err != nil { + t.Fatal(err) + } + if len(result.Data.Spans) != 2 || !result.Data.HasError || len(result.Data.Services) != 2 || result.Provenance.DataSource != "parquet_index" { + t.Fatalf("multi-batch trace = %#v", result) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatal(err) + } +} + +func TestTraceReadsRecentRootFromParquetIndex(t *testing.T) { + svc, mock, repository := newMockService(t) + start := time.Date(2026, 7, 20, 11, 0, 0, 0, time.UTC) + cutoff := start.Add(30 * time.Minute) + end := start.Add(time.Hour) + if err := repository.Commit(context.Background(), telemetrystore.Batch{ID: "trace-after-rebuild", Spans: []telemetry.Span{{ + Namespace: "prod", TraceID: "new-trace", SpanID: "root", ServiceName: "frontend", + StartUnixNanos: cutoff.Add(time.Minute).UnixNano(), DurationMS: 10, StatusCode: "OK", + }}}); err != nil { + t.Fatal(err) + } + mock.ExpectQuery(regexp.QuoteMeta(traceLogsQuery)). + WithArgs("new-trace", start, end, "prod", "prod", 10). + WillReturnRows(sqlmock.NewRows([]string{"time", "severity", "service", "body", "trace_id", "span_id"})) + result, err := svc.Trace(context.Background(), Scope{Namespace: "prod", Start: start, End: end}, "new-trace", "", 10) + if err != nil { + t.Fatal(err) + } + if len(result.Data.Spans) != 1 || result.Data.Spans[0].SpanID != "root" || result.Provenance.DataSource != "parquet_index" { + t.Fatalf("indexed trace = %#v", result) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatal(err) + } +} + +func TestTraceFiltersIndexedSpansByNamespace(t *testing.T) { + svc, mock, repository := newMockService(t) + start := time.Date(2026, 7, 20, 11, 0, 0, 0, time.UTC) + cutoff := start.Add(30 * time.Minute) + end := start.Add(time.Hour) + if err := repository.Commit(context.Background(), telemetrystore.Batch{ID: "trace-cross-namespace", Spans: []telemetry.Span{ + {Namespace: "prod", TraceID: "shared-trace", SpanID: "child", ParentSpanID: "old-root", StartUnixNanos: cutoff.Add(time.Minute).UnixNano()}, + {Namespace: "staging", TraceID: "shared-trace", SpanID: "root", StartUnixNanos: cutoff.Add(2 * time.Minute).UnixNano()}, + }}); err != nil { + t.Fatal(err) + } + mock.ExpectQuery(regexp.QuoteMeta(traceLogsQuery)). + WithArgs("shared-trace", start, end, "prod", "prod", 10). + WillReturnRows(sqlmock.NewRows([]string{"time", "severity", "service", "body", "trace_id", "span_id"})) + result, err := svc.Trace(context.Background(), Scope{Namespace: "prod", Start: start, End: end}, "shared-trace", "", 10) + if err != nil { + t.Fatal(err) + } + if len(result.Data.Spans) != 1 || result.Data.Spans[0].SpanID != "child" || result.Provenance.DataSource != "parquet_index" { + t.Fatalf("namespace-filtered trace = %#v", result) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatal(err) + } +} + +var _ DB = queryrows.SQLAdapter{} + +func TestLogsBoundsParquetQueryWithLimitAndAggregatedBuckets(t *testing.T) { + svc, mock, repository := newMockService(t) + start := time.Date(2026, 7, 20, 11, 0, 0, 0, time.UTC) + end := start.Add(time.Hour) + if err := repository.Commit(context.Background(), telemetrystore.Batch{ID: "bounded-parquet", Logs: []telemetry.Log{{ + Namespace: "prod", TimeUnixNanos: start.Add(time.Minute).UnixNano(), Severity: "ERROR", + ServiceName: "checkout", Body: "hello", TraceID: "trace-parquet", + }}}); err != nil { + t.Fatal(err) + } + // The sample query must carry the row limit so a wide window cannot stream + // the whole Parquet history through the driver. + mock.ExpectQuery(regexp.QuoteMeta(logEntriesQuery)). + WithArgs(start, end, "prod", "prod", "", "", "", "", "", "", 2). + WillReturnRows(sqlmock.NewRows([]string{"time", "severity", "service", "body", "trace_id", "span_id"}). + AddRow(start.Add(3*time.Minute), "ERROR", "checkout", "newest", "trace-c", ""). + AddRow(start.Add(2*time.Minute), "INFO", "checkout", "older", "trace-b", "")) + // Histogram counts come back aggregated, so a million matching rows cost + // one row per bucket rather than a million transfers. + mock.ExpectQuery(regexp.QuoteMeta(logBucketsQuery)). + WithArgs(start, end, "prod", "prod", "", "", "", "", "", ""). + WillReturnRows(sqlmock.NewRows([]string{"point_time", "severity", "count"}). + AddRow(start, "ERROR", int64(900000)). + AddRow(start.Add(5*time.Minute), "INFO", int64(100000))) + result, err := svc.Logs(context.Background(), Scope{Namespace: "prod", Start: start, End: end}, "", "", "", 2) + if err != nil { + t.Fatal(err) + } + if len(result.Data.Entries) != 2 || result.Data.Entries[0].Body != "newest" { + t.Fatalf("entries = %#v, want the two newest sampled rows", result.Data.Entries) + } + if len(result.Data.Buckets) != 2 { + t.Fatalf("buckets = %#v, want one row per aggregated bucket", result.Data.Buckets) + } + if !strings.Contains(result.Summary, "1000000") { + t.Fatalf("summary = %q, want the aggregated match count", result.Summary) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatal(err) + } +} diff --git a/internal/observability/trace.go b/internal/observability/trace.go index 5a22faee..01e9f16b 100644 --- a/internal/observability/trace.go +++ b/internal/observability/trace.go @@ -6,6 +6,8 @@ import ( "sort" "strings" "time" + + "github.com/labstack/fanout/internal/telemetry" ) const recentTraceQuery = ` @@ -17,19 +19,10 @@ ORDER BY MAX(CASE WHEN upper(status) IN ('ERROR', 'STATUS_CODE_ERROR') THEN 1 EL MAX(end_time) - MIN(start_time) DESC LIMIT 1` -const traceSpansQuery = ` -SELECT span_id, COALESCE(parent_span_id, ''), service, operation, kind, start_time, - duration_ms, COALESCE(status, ''), COALESCE(status_message, '') -FROM spans -WHERE start_time >= ? AND start_time < ? AND (? = '' OR namespace = ?) AND trace_id = ? -ORDER BY start_time ASC, duration_ms DESC -LIMIT ?` - const traceLogsQuery = ` -SELECT time, COALESCE(severity, ''), COALESCE(service, ''), COALESCE(body, ''), - COALESCE(trace_id, ''), COALESCE(span_id, '') +SELECT time, severity, coalesce(service, ''), body, coalesce(trace_id, ''), coalesce(span_id, '') FROM logs -WHERE time >= ? AND time < ? AND (? = '' OR namespace = ?) AND trace_id = ? +WHERE trace_id = ? AND time >= ? AND time < ? AND (? = '' OR namespace = ?) ORDER BY time ASC LIMIT ?` @@ -42,8 +35,7 @@ func (s *Service) Trace(ctx context.Context, scope Scope, traceID, service strin if err != nil { return Result[TraceDetail]{}, err } - traceID = strings.TrimSpace(traceID) - service = strings.TrimSpace(service) + traceID, service = strings.TrimSpace(traceID), strings.TrimSpace(service) if traceID == "" { rows, queryErr := s.db.QueryContext(ctx, recentTraceQuery, scope.Start, scope.End, scope.Namespace, scope.Namespace, service, service) if queryErr != nil { @@ -62,72 +54,75 @@ func (s *Service) Trace(ctx context.Context, scope Scope, traceID, service strin rows.Close() } + dataSource := "parquet_index" data := TraceDetail{TraceID: traceID, Services: []string{}, Spans: []TraceSpan{}, Logs: []LogEntry{}} if traceID != "" { - rows, queryErr := s.db.QueryContext(ctx, traceSpansQuery, scope.Start, scope.End, scope.Namespace, scope.Namespace, traceID, limit) - if queryErr != nil { - return Result[TraceDetail]{}, fmt.Errorf("query trace spans: %w", queryErr) + storedSpans, readErr := s.repository.Trace(ctx, telemetry.TraceQuery{ + TraceID: traceID, Namespace: scope.Namespace, + StartNanos: scope.Start.UnixNano(), EndNanos: scope.End.UnixNano(), Limit: limit, + }) + if readErr != nil { + return Result[TraceDetail]{}, fmt.Errorf("read indexed Parquet trace: %w", readErr) } - serviceSet := map[string]struct{}{} - var first time.Time - var last time.Time - for rows.Next() { - var span TraceSpan - if err := rows.Scan(&span.SpanID, &span.ParentSpanID, &span.Service, &span.Operation, &span.Kind, &span.Start, &span.DurationMS, &span.Status, &span.StatusMessage); err != nil { - rows.Close() - return Result[TraceDetail]{}, fmt.Errorf("scan trace span: %w", err) - } + for _, row := range storedSpans { + data.Spans = append(data.Spans, TraceSpan{SpanID: row.SpanID, ParentSpanID: row.ParentSpanID, Service: row.ServiceName, Operation: row.Name, Kind: row.Kind, Start: time.Unix(0, row.StartUnixNanos).UTC(), DurationMS: row.DurationMS, Status: row.StatusCode, StatusMessage: row.StatusMsg}) + } + + serviceSet := make(map[string]struct{}) + var first, last time.Time + for _, span := range data.Spans { if first.IsZero() || span.Start.Before(first) { first = span.Start } - end := span.Start.Add(time.Duration(span.DurationMS * float64(time.Millisecond))) - if end.After(last) { + if end := span.Start.Add(time.Duration(span.DurationMS * float64(time.Millisecond))); end.After(last) { last = end } if strings.Contains(strings.ToUpper(span.Status), "ERROR") { data.HasError = true } - serviceSet[span.Service] = struct{}{} - data.Spans = append(data.Spans, span) - } - if err := rows.Err(); err != nil { - rows.Close() - return Result[TraceDetail]{}, fmt.Errorf("iterate trace spans: %w", err) + if span.Service != "" { + serviceSet[span.Service] = struct{}{} + } } - rows.Close() if !first.IsZero() { data.DurationMS = last.Sub(first).Seconds() * 1000 } for name := range serviceSet { - if name != "" { - data.Services = append(data.Services, name) - } + data.Services = append(data.Services, name) } sort.Strings(data.Services) - - rows, queryErr = s.db.QueryContext(ctx, traceLogsQuery, scope.Start, scope.End, scope.Namespace, scope.Namespace, traceID, limit) - if queryErr != nil { - return Result[TraceDetail]{}, fmt.Errorf("query trace logs: %w", queryErr) + data.Logs, err = s.traceLogsFromParquet(ctx, scope, traceID, limit) + if err != nil { + return Result[TraceDetail]{}, err } - for rows.Next() { - var entry LogEntry - if err := rows.Scan(&entry.Time, &entry.Severity, &entry.Service, &entry.Body, &entry.TraceID, &entry.SpanID); err != nil { - rows.Close() - return Result[TraceDetail]{}, fmt.Errorf("scan trace log: %w", err) - } - entry.Body = redactLogBody(entry.Body) - data.Logs = append(data.Logs, entry) - } - if err := rows.Err(); err != nil { - rows.Close() - return Result[TraceDetail]{}, fmt.Errorf("iterate trace logs: %w", err) - } - rows.Close() } summary := "No traces found in this telemetry window" if traceID != "" { summary = fmt.Sprintf("Trace %s contains %d spans across %d services", traceID, len(data.Spans), len(data.Services)) } - return Result[TraceDetail]{Schema: TraceSchema, Summary: summary, Data: data, Provenance: s.provenanceFor(scope, "spans + logs")}, nil + return Result[TraceDetail]{Schema: TraceSchema, Summary: summary, Data: data, Provenance: s.provenanceFor(scope, dataSource)}, nil +} + +func (s *Service) traceLogsFromParquet(ctx context.Context, scope Scope, traceID string, limit int) ([]LogEntry, error) { + rows, err := s.db.QueryContext(ctx, traceLogsQuery, traceID, scope.Start, scope.End, scope.Namespace, scope.Namespace, limit) + if err != nil { + return nil, fmt.Errorf("query trace parquet logs: %w", err) + } + logs := make([]LogEntry, 0, limit) + for rows.Next() { + var entry LogEntry + if err := rows.Scan(&entry.Time, &entry.Severity, &entry.Service, &entry.Body, &entry.TraceID, &entry.SpanID); err != nil { + rows.Close() + return nil, fmt.Errorf("scan trace parquet log: %w", err) + } + entry.Body = redactLogBody(entry.Body) + logs = append(logs, entry) + } + if err := rows.Err(); err != nil { + rows.Close() + return nil, fmt.Errorf("iterate trace parquet logs: %w", err) + } + rows.Close() + return logs, nil } diff --git a/internal/query/attr_json_test.go b/internal/query/attr_json_test.go index 00dd939c..cf6f6955 100644 --- a/internal/query/attr_json_test.go +++ b/internal/query/attr_json_test.go @@ -22,7 +22,7 @@ func TestAttrMacroAndQuotedPath(t *testing.T) { // Flat object with dotted keys and a numeric value, as attrsJSON produces. const attrs = `{"http.method":"GET","http.status_code":200,"messaging.system":"kafka"}` if _, err := db.ExecContext(ctx, - `INSERT INTO lake.spans (namespace, service, attributes_json) VALUES ('default','svc',?)`, attrs); err != nil { + `INSERT INTO telemetry.spans (namespace, service, attributes_json) VALUES ('default','svc',?)`, attrs); err != nil { t.Fatalf("insert span: %v", err) } diff --git a/internal/query/duck.go b/internal/query/duck.go index 102ea31f..02c6bdff 100644 --- a/internal/query/duck.go +++ b/internal/query/duck.go @@ -14,31 +14,35 @@ import ( "time" "github.com/duckdb/duckdb-go/v2" - _ "modernc.org/sqlite" // SQLite driver used to put the DuckLake catalog in WAL mode "github.com/labstack/fanout/internal/config" - "github.com/labstack/fanout/internal/lake/writegate" "github.com/labstack/fanout/internal/metrics" + "github.com/labstack/fanout/internal/query/writegate" + "github.com/labstack/fanout/internal/queryrows" + "github.com/labstack/fanout/internal/telemetry" + telemetrystore "github.com/labstack/fanout/internal/telemetry/store" ) type Duck struct { - DB *sql.DB - cfg config.Config - lastMaintenance time.Time - lastMerge time.Time // cadence for the frequent merge-only compaction pass + DB *sql.DB + cfg config.Config + repository *telemetrystore.Repository // rollupLagNanos holds the rollup watermark back from the max ingested // timestamp so late/out-of-order commits aren't skipped. Zero disables the // lag (no trailing window). rollupLagNanos int64 - // writeGate serializes and measures all DuckLake catalog write commits so - // multiple pooled connections never commit to the SQLite catalog concurrently. + // writeGate serializes writes to the rebuildable DuckDB rollup cache. writeGate writegate.WriteGate + // parquetMu pins immutable files for active DuckDB readers. Its reader-first + // gate keeps a queued maintenance publish from stalling unrelated new reads. + parquetMu parquetReadGate // maintHealthMu guards the maintenance health fields below, which the // readiness probe reads while the maintenance pass writes them. - maintHealthMu sync.Mutex - lastMaintenanceOK time.Time - lastMaintenanceAt time.Time - lastMaintenanceErr error + maintHealthMu sync.Mutex + lastMaintenanceOK time.Time + lastMaintenanceAt time.Time + lastMaintenanceErr error + maintenanceFailures int } // MaintenanceHealth reports the maintenance loop's own health: when it last @@ -49,28 +53,54 @@ type Duck struct { // operator instead of only logging. lastAt distinguishes "failing right now" // from a stale error awaiting the hourly retry; a zero lastAt means no pass // has executed since the process started. -func (d *Duck) MaintenanceHealth() (lastOK, lastAt time.Time, lastErr error) { +func (d *Duck) MaintenanceHealth() (lastOK, lastAt time.Time, consecutiveFailures int, lastErr error) { d.maintHealthMu.Lock() defer d.maintHealthMu.Unlock() - return d.lastMaintenanceOK, d.lastMaintenanceAt, d.lastMaintenanceErr + return d.lastMaintenanceOK, d.lastMaintenanceAt, d.maintenanceFailures, d.lastMaintenanceErr } const ( - serviceRollupStateKey = "service_rollup_v2" - serviceRollupRawMaxKey = "service_rollup_v2_rawmax" - edgeRollupStateKey = "edge_rollup_v2" - edgeRollupRawMaxKey = "edge_rollup_v2_rawmax" - EndpointRollupStateKey = "endpoint_rollup_v1" - endpointRollupRawMaxKey = "endpoint_rollup_v1_rawmax" - endpointBackfillStateKey = "endpoint_rollup_v1_backfill_started" - EndpointReadyStateKey = "endpoint_rollup_v1_ready" - EndpointDisabledStateKey = "endpoint_rollup_v1_disabled" - defaultDuckDBPoolSize = 1 + serviceRollupStateKey = "service_rollup_v2" + serviceRollupRawMaxKey = "service_rollup_v2_rawmax" + edgeRollupStateKey = "edge_rollup_v2" + edgeRollupRawMaxKey = "edge_rollup_v2_rawmax" + edgeRollupSubCursorKey = "edge_rollup_v2_substart" + edgeRollupSubWindowKey = "edge_rollup_v2_subwindow_end" + EndpointRollupStateKey = "endpoint_rollup_v1" + endpointRollupRawMaxKey = "endpoint_rollup_v1_rawmax" + endpointBackfillStateKey = "endpoint_rollup_v1_backfill_started" + EndpointReadyStateKey = "endpoint_rollup_v1_ready" + EndpointDisabledStateKey = "endpoint_rollup_v1_disabled" + defaultDuckDBPoolSize = 1 + parquetCompactionCycle = 10 * time.Second + parquetCompactionCycleBudget = 8 * time.Second + parquetMaintenanceBatchLimit = 128 + parquetMaintenancePhaseBudget = 10 * time.Minute ) -// WriteGate returns the shared catalog write gate. The ingest writer must use it -// around appender flushes so writes never overlap rollup/maintenance commits on -// a multi-connection pool. +// parquetDrainBudget bounds how long a publisher waits for readers already +// inside the snapshot to leave. parquetSwapBudget bounds the exclusive window +// that follows — the renames and fsync of at most parquetMaintenanceBatchLimit +// directories — which must be allowed to finish rather than be abandoned with +// every reader already excluded. +const ( + parquetDrainBudget = 2 * defaultWriterGrace + parquetSwapBudget = 30 * time.Second +) + +// parquetStatsWait bounds the gauge refresh, which is the only Parquet reader +// that runs on a loop goroutine rather than behind a request. It is a variable +// so the internal lease can be exercised without a five-second test. +var parquetStatsWait = 5 * time.Second + +// rollupPublicationSafetyLag covers the maximum public SQL hold, publication +// grace, bounded commit retries, and queued Parquet encoding with headroom for +// a busy disk. Rows stamped at request receipt remain inside the recomputed +// tail until their Parquet commit becomes visible. +const rollupPublicationSafetyLag = 5 * time.Minute + +// WriteGate returns the gate that serializes writes to DuckDB's rebuildable +// rollup cache. func (d *Duck) WriteGate() *writegate.WriteGate { return &d.writeGate } // duckDBPoolSize is the effective connection-pool size: the configured value, @@ -172,7 +202,10 @@ func parseDuckBytes(s string) (int64, bool) { } } -func NewDuck(ctx context.Context, cfg config.Config) (*Duck, error) { +func NewDuck(ctx context.Context, cfg config.Config, repository *telemetrystore.Repository) (*Duck, error) { + if repository == nil { + return nil, errors.New("telemetry repository is required") + } if err := os.MkdirAll(cfg.QueryDir(), 0o755); err != nil { return nil, fmt.Errorf("create query dir: %w", err) } @@ -182,40 +215,20 @@ func NewDuck(ctx context.Context, cfg config.Config) (*Duck, error) { dbPath := cfg.QueryDuckDBPath() tempDir := cfg.QueryTempDir() - metadataPath := cfg.TelemetryDuckLakePath() - dataPath := cfg.TelemetryParquetDir() if err := os.MkdirAll(tempDir, 0o755); err != nil { return nil, fmt.Errorf("create temp dir: %w", err) } - if err := os.MkdirAll(dataPath, 0o755); err != nil { - return nil, fmt.Errorf("create telemetry parquet dir: %w", err) - } - - // Put the DuckLake SQLite catalog in WAL mode before DuckDB attaches it, so - // read queries run concurrently with the single writer instead of failing - // with "database is locked" (rollback-journal mode), and a crashed writer - // can't leave the catalog permanently locked. Must run before openDuckDB. - if err := enableCatalogWAL(metadataPath); err != nil { - return nil, fmt.Errorf("enable WAL on DuckLake catalog: %w", err) - } - dsn, err := duckDSN(dbPath, cfg.DuckDBMemory, cfg.DuckDBThreads) if err != nil { return nil, err } - db, err := openDuckDB(ctx, dsn, tempDir, metadataPath, dataPath, duckDBPoolSize(cfg)) + db, err := openDuckDB(ctx, dsn, tempDir, duckDBPoolSize(cfg)) if err != nil { - return nil, fmt.Errorf( - "open duckdb catalog: %w (if the local cache catalog is corrupted, remove %s and %s; DuckLake data remains in %s)", - err, - dbPath, - dbPath+".wal", - dataPath, - ) + return nil, fmt.Errorf("open DuckDB query cache: %w (the cache at %s is rebuildable from Parquet)", err, dbPath) } - d := &Duck{DB: db, cfg: cfg, rollupLagNanos: rollupLagFromConfig(cfg)} + d := &Duck{DB: db, cfg: cfg, repository: repository, rollupLagNanos: int64(rollupPublicationSafetyLag)} if cfg.DuckDBMemory == "" { // Only when the operator hasn't pinned storage.duckdb.memory: keep DuckDB's // cgroup-aware auto limit on big boxes but leave absolute RAM headroom on @@ -224,7 +237,11 @@ func NewDuck(ctx context.Context, cfg config.Config) (*Duck, error) { slog.Warn("apply memory headroom failed; using DuckDB default memory_limit", "err", err) } } - if err := CreateTables(db); err != nil { + if err := CreateCacheTables(db); err != nil { + _ = db.Close() + return nil, err + } + if err := CreateParquetViews(db, repository.Parquet.Dir()); err != nil { _ = db.Close() return nil, err } @@ -280,6 +297,11 @@ func (d *Duck) skipRollupToLatest(ctx context.Context) error { {serviceRollupRawMaxKey, svc}, {edgeRollupStateKey, edge}, {edgeRollupRawMaxKey, edge}, + // Skipping discards any half-aggregated window, so its resume point + // must go too or the next pass would reopen a window behind the + // watermark it just advanced. + {edgeRollupSubCursorKey, 0}, + {edgeRollupSubWindowKey, 0}, // Endpoint queries remain on their raw-span fallback when the operator // explicitly skips historical rollups. Mark this cache disabled instead of // later declaring a new-only, incomplete endpoint cache ready. @@ -293,16 +315,11 @@ func (d *Duck) skipRollupToLatest(ctx context.Context) error { return tx.Commit() } -func openDuckDB(ctx context.Context, dsn, tempDir, metadataPath, dataPath string, maxConns int) (*sql.DB, error) { - // AUTOMATIC_MIGRATION upgrades an older on-disk DuckLake catalog to the format - // the loaded extension requires. Without it, a fanout build that bundles a - // newer DuckLake (e.g. the DuckDB 1.5.3 bump, which needs catalog v1.0) fails - // to attach an existing v0.4 catalog and the server can't boot. Migration is - // in place and forward-only. - attach := fmt.Sprintf("ATTACH IF NOT EXISTS %s AS lake (DATA_PATH %s, AUTOMATIC_MIGRATION true)", - sqlLiteral("ducklake:sqlite:"+metadataPath), - sqlLiteral(dataPath)) - +func openDuckDB(ctx context.Context, dsn, tempDir string, maxConns int) (*sql.DB, error) { + // Every pooled connection must use UTC. DuckDB otherwise inherits the host + // timezone and casts TIMESTAMPTZ rollup buckets into local wall-clock + // TIMESTAMP values, while API windows arrive in UTC. + // // temp_directory is an instance-global setting: re-setting it after the temp // dir has already been used fails with "Cannot switch temporary directory // after the current one has been used". The boot hook runs once per pooled @@ -312,10 +329,8 @@ func openDuckDB(ctx context.Context, dsn, tempDir, metadataPath, dataPath string var tempDirErr error connector, err := duckdb.NewConnector(dsn, func(execer driver.ExecerContext) error { - for _, stmt := range []string{"LOAD ducklake", "LOAD sqlite"} { - if _, err := execer.ExecContext(ctx, stmt, nil); err != nil { - return err - } + if _, err := execer.ExecContext(ctx, "SET TimeZone='UTC'", nil); err != nil { + return fmt.Errorf("set timezone: %w", err) } tempDirOnce.Do(func() { _, tempDirErr = execer.ExecContext(ctx, "SET temp_directory="+sqlLiteral(tempDir), nil) @@ -326,9 +341,6 @@ func openDuckDB(ctx context.Context, dsn, tempDir, metadataPath, dataPath string if tempDirErr != nil { return fmt.Errorf("set temp_directory: %w", tempDirErr) } - if _, err := execer.ExecContext(ctx, attach, nil); err != nil { - return err - } return nil }) if err != nil { @@ -336,12 +348,8 @@ func openDuckDB(ctx context.Context, dsn, tempDir, metadataPath, dataPath string } db := sql.OpenDB(connector) - // DuckLake metadata lives in a SQLite catalog that locks under *concurrent* - // commits from multiple connections. The default pool of 1 serializes - // everything through one handle. Larger pools are allowed (read queries then - // run concurrently), but write commits must still be serialized by the - // caller's write gate (Duck.WriteGate) so two connections never commit at - // once. + // The on-disk DuckDB file contains only rebuildable rollups and views; Parquet + // scans may run concurrently across the machine-sized pool. if maxConns < 1 { maxConns = 1 } @@ -350,49 +358,13 @@ func openDuckDB(ctx context.Context, dsn, tempDir, metadataPath, dataPath string return db, nil } -// enableCatalogWAL switches the DuckLake SQLite catalog at metadataPath to WAL -// journal mode before DuckDB attaches it. The default rollback-journal mode -// makes a committing writer take an exclusive lock that fails concurrent readers -// with "database is locked", and a crashed or stuck writer can leave a hot -// journal that locks the catalog until every connection is dropped (observed in -// prod as a multi-day write+query outage). In WAL mode readers run concurrently -// with the single writer, and the next connection to open the catalog recovers -// the WAL automatically after a crash instead of leaving a lock-holding hot -// journal. journal_mode is persisted in the database header, so this is -// effectively a one-time migration, but it is cheap and idempotent to assert on -// every boot. WAL is the only lever available: the DuckDB sqlite extension -// exposes no busy_timeout knob, so DuckDB's own catalog connections have no -// lock-retry timeout — WAL is what prevents the reader/writer collisions. -func enableCatalogWAL(metadataPath string) (err error) { - db, err := sql.Open("sqlite", metadataPath+"?_pragma=journal_mode(wal)") - if err != nil { - return fmt.Errorf("open catalog: %w", err) - } - defer func() { - // Closing the last connection checkpoints the WAL; surface a failure here - // (e.g. the filesystem can't maintain the -wal/-shm sidecars) instead of - // discovering it later as a mysterious lock. - if cerr := db.Close(); cerr != nil && err == nil { - err = fmt.Errorf("close catalog bootstrap conn: %w", cerr) - } - }() - db.SetMaxOpenConns(1) - - var mode string - if scanErr := db.QueryRow("PRAGMA journal_mode").Scan(&mode); scanErr != nil { - return fmt.Errorf("read journal_mode: %w", scanErr) - } - if !strings.EqualFold(mode, "wal") { - return fmt.Errorf("catalog journal_mode = %q, want wal", mode) - } - return nil -} - func sqlLiteral(v string) string { return "'" + strings.ReplaceAll(v, "'", "''") + "'" } -func (d *Duck) Close() error { return d.DB.Close() } +func (d *Duck) Close() error { + return d.DB.Close() +} // DefaultNamespace returns empty string so queries search all namespaces. func (d *Duck) DefaultNamespace() string { @@ -412,7 +384,13 @@ func (d *Duck) RunRollups(ctx context.Context) { } else if rows > 0 { slog.Info("startup rollup complete", "rows", rows, "duration", time.Since(start)) } - d.updateLakeStats(ctx) + d.updateParquetStats(ctx) + maintenanceDone := make(chan struct{}) + go func() { + defer close(maintenanceDone) + d.runMaintenanceLoop(ctx) + }() + defer func() { <-maintenanceDone }() ticker := time.NewTicker(d.cfg.RollupInterval) defer ticker.Stop() @@ -425,41 +403,35 @@ func (d *Duck) RunRollups(ctx context.Context) { if err != nil { slog.Error("rollup failed", "component", "rollup", "rows", rows, "err", err) } - d.updateLakeStats(ctx) + d.updateParquetStats(ctx) case <-ctx.Done(): return } } } -// updateLakeStats refreshes the per-signal file-count and byte-size gauges -// (fanout_lake_partitions / fanout_lake_size_bytes) from the DuckLake catalog. -// This is the signal that surfaces unbounded file/snapshot growth — the failure -// mode that previously OOM'd the rollup engine — so an operator or soak test can -// watch it climb. It's a cheap read; a failure here must not disturb rollups. -func (d *Duck) updateLakeStats(ctx context.Context) { - // Bound the catalog read so a degraded/bloated DuckLake can't stall the - // rollup loop (this runs inline after rollupOnce in RunRollups). - ctx, cancel := context.WithTimeout(ctx, 15*time.Second) +// updateParquetStats refreshes the per-signal file-count and byte-size gauges. +// +// The wait for the snapshot is bounded. Every other reader carries a request +// deadline, but this one runs on the rollup and maintenance loops under the +// process context, so an unbounded wait here would park those loops for as +// long as a publication stayed stuck. Gauges are refreshed again next tick; +// skipping one refresh costs nothing that waiting would not cost more. +func (d *Duck) updateParquetStats(ctx context.Context) { + statsCtx, cancel := context.WithTimeout(ctx, parquetStatsWait) defer cancel() - rows, err := d.DB.QueryContext(ctx, `SELECT table_name, file_count, file_size_bytes FROM ducklake_table_info('lake')`) - if err != nil { - slog.Warn("lake stats query failed", "err", err) + if err := d.lockParquetRead(statsCtx); err != nil { + slog.Warn("parquet stats skipped", "err", err) return } - defer rows.Close() - for rows.Next() { - var table string - var fileCount, sizeBytes int64 - if err := rows.Scan(&table, &fileCount, &sizeBytes); err != nil { - slog.Warn("lake stats scan failed", "err", err) - return - } - // DuckLake table names (spans/logs/metrics) are the metric signal labels. - metrics.UpdateLakeStats(table, sizeBytes, int(fileCount)) + defer d.parquetMu.RUnlock() + stats, err := d.repository.Parquet.Stats() + if err != nil { + slog.Warn("parquet stats failed", "err", err) + return } - if err := rows.Err(); err != nil { - slog.Warn("lake stats iteration failed", "err", err) + for signal, stat := range stats { + metrics.UpdateParquetStats(signal, stat.Bytes, stat.Files) } } @@ -485,18 +457,126 @@ func (d *Duck) rollupOnce(ctx context.Context) (int, error) { affected += n } - // Compaction must run even when a rollup fails: a failing rollup is exactly - // when compaction matters most, since file/snapshot growth makes every - // retried pass heavier. The frequent merge pass keeps the file count low; the - // hourly maintenance pass handles retention + snapshot expiry + cleanup. - if err := d.runMerge(ctx); err != nil { - slog.Warn("ducklake merge failed", "err", err) + return int(affected), errors.Join(errs...) +} + +func (d *Duck) runMaintenanceLoop(ctx context.Context) { + every := d.cfg.MaintenanceInterval + if every <= 0 { + every = time.Hour + } + run := func() { + if err := d.runRepositoryMaintenance(ctx); err != nil && ctx.Err() == nil { + slog.Warn("telemetry maintenance failed", "err", err) + } + d.updateParquetStats(ctx) + } + run() + maintenanceTicker := time.NewTicker(every) + defer maintenanceTicker.Stop() + var compactionTicker *time.Ticker + var compactionTick <-chan time.Time + if every > parquetCompactionCycle { + compactionTicker = time.NewTicker(parquetCompactionCycle) + compactionTick = compactionTicker.C + defer compactionTicker.Stop() + } + for { + select { + case <-maintenanceTicker.C: + run() + case <-compactionTick: + if err := d.runParquetCompactionCycle(ctx); err != nil && ctx.Err() == nil { + slog.Warn("telemetry compaction cycle failed", "err", err) + } + case <-ctx.Done(): + return + } + } +} + +func (d *Duck) runParquetCompactionCycle(ctx context.Context) error { + if d.repository == nil { + return nil } - if err := d.runMaintenance(ctx); err != nil { - slog.Warn("ducklake maintenance failed", "err", err) + started := time.Now() + if err := d.repository.RecoverParquet(ctx, d); err != nil { + metrics.RecordTelemetryOperation(metrics.TelemetryCompaction, metrics.TelemetryError, time.Since(started).Seconds()) + return err } + compacted, err := d.repository.CompactParquetPass(ctx, d, parquetMaintenanceBatchLimit, parquetCompactionCycleBudget) + result := metrics.TelemetryNoop + if err != nil { + result = metrics.TelemetryError + } else if compacted > 0 { + result = metrics.TelemetrySuccess + } + metrics.RecordTelemetryOperation(metrics.TelemetryCompaction, result, time.Since(started).Seconds()) + return err +} - return int(affected), errors.Join(errs...) +func (d *Duck) runRepositoryMaintenance(ctx context.Context) error { + start := time.Now() + var pruneErr error + if d.repository != nil { + recoveryErr := d.repository.RecoverParquet(ctx, d) + compactStart := time.Now() + var cleanupErr, parquetErr, compactErr error + compacted := 0 + if recoveryErr == nil { + cleanupErr = d.repository.CleanupParquet() + if d.cfg.RetentionDays > 0 { + _, parquetErr = d.repository.PruneParquetPass(ctx, d, time.Now().Add(-time.Duration(d.cfg.RetentionDays)*24*time.Hour).UnixNano(), parquetMaintenanceBatchLimit, parquetMaintenancePhaseBudget) + } + compacted, compactErr = d.repository.CompactParquetPass(ctx, d, parquetMaintenanceBatchLimit, parquetMaintenancePhaseBudget) + } + compactResult := metrics.TelemetryNoop + if recoveryErr != nil || compactErr != nil { + compactResult = metrics.TelemetryError + } else if compacted > 0 { + compactResult = metrics.TelemetrySuccess + } + metrics.RecordTelemetryOperation(metrics.TelemetryCompaction, compactResult, time.Since(compactStart).Seconds()) + pruneErr = errors.Join(pruneErr, recoveryErr, cleanupErr, parquetErr, compactErr) + } + cacheErr := func() error { + unlock, err := d.writeGate.LockContext(ctx, writegate.WriteMaintenance) + if err != nil { + return err + } + defer unlock() + var errs []error + if d.cfg.RetentionDays > 0 { + for _, table := range []string{"service_rollup", "endpoint_rollup", "edge_rollup"} { + if _, err := d.DB.ExecContext(ctx, fmt.Sprintf("DELETE FROM %s WHERE bucket < now() - INTERVAL %d DAY", table, d.cfg.RetentionDays)); err != nil { + errs = append(errs, fmt.Errorf("prune %s: %w", table, err)) + } + } + } + _, checkpointErr := d.DB.ExecContext(ctx, "CHECKPOINT") + return errors.Join(errors.Join(errs...), checkpointErr) + }() + err := errors.Join(pruneErr, cacheErr) + maintenanceResult := metrics.TelemetrySuccess + if err != nil { + maintenanceResult = metrics.TelemetryError + } + metrics.RecordTelemetryOperation(metrics.TelemetryMaintenance, maintenanceResult, time.Since(start).Seconds()) + finished := time.Now() + d.maintHealthMu.Lock() + d.lastMaintenanceAt = finished + if err == nil { + d.lastMaintenanceOK = finished + d.maintenanceFailures = 0 + } else { + d.maintenanceFailures++ + } + d.lastMaintenanceErr = err + d.maintHealthMu.Unlock() + if err == nil { + slog.Info("telemetry maintenance complete", "duration", time.Since(start)) + } + return err } func (d *Duck) refreshServiceRollup(ctx context.Context) (int64, error) { @@ -514,12 +594,15 @@ func (d *Duck) refreshServiceRollup(ctx context.Context) (int64, error) { updateRollupProgress(metrics.RollupService, true, watermark, sourceMax) } }() - - // Serialize against other writers (edge rollup, maintenance, ingest flushes). + // Serialize against other DuckDB writers (edge rollup and maintenance). // The write gate is always acquired before a connection to keep lock ordering // consistent and deadlock-free. unlock := d.writeGate.Lock(writegate.WriteRollupService) defer unlock() + if err := d.lockRollupParquetRead(ctx); err != nil { + return 0, err + } + defer d.parquetMu.RUnlock() tx, err := d.DB.BeginTx(ctx, nil) if err != nil { @@ -635,9 +718,12 @@ func (d *Duck) refreshEndpointRollup(ctx context.Context) (int64, error) { updateRollupProgress(metrics.RollupEndpoint, true, watermark, sourceMax) } }() - unlock := d.writeGate.Lock(writegate.WriteRollupEndpoint) defer unlock() + if err := d.lockRollupParquetRead(ctx); err != nil { + return 0, err + } + defer d.parquetMu.RUnlock() tx, err := d.DB.BeginTx(ctx, nil) if err != nil { @@ -755,9 +841,12 @@ func (d *Duck) refreshEdgeRollup(ctx context.Context) (int64, error) { updateRollupProgress(metrics.RollupEdge, true, watermark, sourceMax) } }() - unlock := d.writeGate.Lock(writegate.WriteRollupEdge) defer unlock() + if err := d.lockRollupParquetRead(ctx); err != nil { + return 0, err + } + defer d.parquetMu.RUnlock() tx, err := d.DB.BeginTx(ctx, nil) if err != nil { @@ -780,7 +869,22 @@ func (d *Duck) refreshEdgeRollup(ctx context.Context) (int64, error) { return 0, err } sourceMax = rawWatermark - if rawWatermark <= lastWatermark { + + // A previous pass that ran out of sub-window budget recorded where to + // resume and left the ingested watermark untouched. Reuse that window + // verbatim so its unprocessed start_time tail is finished before the + // watermark is allowed past it. + subCursor, err := rollupWatermark(ctx, tx, edgeRollupSubCursorKey) + if err != nil { + return 0, err + } + resumeEnd, err := rollupWatermark(ctx, tx, edgeRollupSubWindowKey) + if err != nil { + return 0, err + } + resuming := subCursor > 0 && resumeEnd > lastWatermark + + if rawWatermark <= lastWatermark && !resuming { err := tx.Commit() if err == nil { result = metrics.RollupNoop @@ -795,6 +899,10 @@ func (d *Duck) refreshEdgeRollup(ctx context.Context) (int64, error) { } } windowStart, windowEnd, chunked := rollupWindow(lastWatermark, minIngested, rawWatermark) + if resuming { + windowEnd = resumeEnd + chunked = windowEnd < rawWatermark + } // Same chunking + trailing-window logic as the service rollup (see // refreshServiceRollup): never advance into the lag window below an @@ -832,11 +940,27 @@ WHERE ingested_unix_nano > ? } var totalAffected int64 + completed := true + var nextCursor int64 if minStartT.Valid { subLo := minStartT.Time + if subCursor > 0 { + if resumeAt := time.Unix(0, subCursor).In(subLo.Location()); resumeAt.After(subLo) { + subLo = resumeAt + } + } maxT := maxStartT.Time + processed := 0 for !subLo.After(maxT) { - subHi := subLo.Add(time.Duration(edgeStartChunkNanos)) + if processed == maxEdgeSubWindowsPerPass { + completed = false + nextCursor = subLo.UnixNano() + break + } + subHi, err := edgeSubWindowEnd(ctx, tx, windowStart, windowEnd, subLo, maxT, maxEdgeSpansPerSubWindow) + if err != nil { + return 0, err + } if _, err := tx.ExecContext(ctx, edgeRollupDeleteSQL, windowStart, windowEnd, subLo, subHi); err != nil { return 0, err } @@ -851,14 +975,32 @@ WHERE ingested_unix_nano > ? totalAffected += rows } subLo = subHi + processed++ } } - if err := storeRollupWatermark(ctx, tx, edgeRollupStateKey, newWatermark); err != nil { - return 0, err - } - if err := storeRollupWatermark(ctx, tx, edgeRollupRawMaxKey, rawMaxProcessed); err != nil { - return 0, err + // An unfinished window keeps the ingested watermark where it was: the + // sub-window cursor is the only durable record that part of this window is + // already aggregated, and advancing past it would drop the remainder. + storedWatermark := newWatermark + if completed { + if err := storeRollupWatermark(ctx, tx, edgeRollupStateKey, newWatermark); err != nil { + return 0, err + } + if err := storeRollupWatermark(ctx, tx, edgeRollupRawMaxKey, rawMaxProcessed); err != nil { + return 0, err + } + if err := clearEdgeRollupCursor(ctx, tx); err != nil { + return 0, err + } + } else { + storedWatermark = lastWatermark + if err := storeRollupWatermark(ctx, tx, edgeRollupSubCursorKey, nextCursor); err != nil { + return 0, err + } + if err := storeRollupWatermark(ctx, tx, edgeRollupSubWindowKey, windowEnd); err != nil { + return 0, err + } } if err := tx.Commit(); err != nil { return 0, err @@ -866,16 +1008,28 @@ WHERE ingested_unix_nano > ? result = metrics.RollupSuccess recordedRows = totalAffected - updateRollupProgress(metrics.RollupEdge, true, newWatermark, rawWatermark) + updateRollupProgress(metrics.RollupEdge, true, storedWatermark, rawWatermark) return totalAffected, nil } +// clearEdgeRollupCursor drops the resume point once its window is fully +// aggregated, so the next pass opens a fresh ingested window. +func clearEdgeRollupCursor(ctx context.Context, tx *sql.Tx) error { + if err := storeRollupWatermark(ctx, tx, edgeRollupSubCursorKey, 0); err != nil { + return err + } + return storeRollupWatermark(ctx, tx, edgeRollupSubWindowKey, 0) +} + // rollupChunkNanos caps how much ingest history one rollup pass scans. A pass // that starts far behind — first run on an existing dataset, or recovery after // an outage — catches up one chunk per tick instead of aggregating the whole // backlog in a single statement. Unbounded catch-up is what took prod down on // 2026-06-13 (UTC): the edge rollup's first pass covered 12 days of spans, // spilled 375 GiB to temp, filled the disk, and never committed. +// One-hour chunks preserve fast catch-up while bounding the work admitted to a +// transaction. Publication may wait for the current chunk, but does not cancel +// it and force the same cache work to restart indefinitely. const rollupChunkNanos = int64(time.Hour) // edgeStartChunkNanos bounds how wide a start_time range one edge-rollup @@ -883,6 +1037,59 @@ const rollupChunkNanos = int64(time.Hour) // (catch-up/bulk-load) can't exhaust memory. The pass loops over sub-windows. const edgeStartChunkNanos = int64(30 * time.Minute) +// maxEdgeSpansPerSubWindow makes the start-time window adaptive under dense +// ingest. The one-minute rollup bucket is the indivisible lower bound: a hot +// bucket above this limit still runs as one correct aggregate instead of being +// split into partial results. +const maxEdgeSpansPerSubWindow int64 = 250_000 + +// maxEdgeSubWindowsPerPass bounds how many start_time sub-windows one pass +// processes, and with it how long that pass holds the Parquet read gate. +// +// Chunking ingested time alone does not bound the work: a seed load or a +// backfill compresses a wide start_time range into a narrow ingested window, +// so a single one-hour chunk can expand to hundreds of sub-windows and hold +// the gate for as long as that takes. No timeout can bound that from outside — +// cancelling mid-transaction only discards the work and retries it forever. So +// the pass instead stops at a fixed number of sub-windows, persists where to +// resume, and leaves the ingested watermark where it was. Combined with the +// adaptive row limit above, every pass makes bounded forward progress without +// assuming a uniform event rate. +const maxEdgeSubWindowsPerPass = 8 + +func edgeSubWindowEnd(ctx context.Context, tx *sql.Tx, windowStart, windowEnd int64, subLo, maxT time.Time, rowLimit int64) (time.Time, error) { + maxEnd := maxT.Add(time.Minute) + subHi := subLo.Add(time.Duration(edgeStartChunkNanos)) + if subHi.After(maxEnd) { + subHi = maxEnd + } + for rowLimit > 0 && subHi.Sub(subLo) > time.Minute { + var rows int64 + if err := tx.QueryRowContext(ctx, ` +SELECT COUNT(*) +FROM ( + SELECT 1 + FROM spans + WHERE ingested_unix_nano > ? + AND ingested_unix_nano <= ? + AND start_time >= ? + AND start_time < ? + LIMIT ? +)`, windowStart, windowEnd, subLo, subHi, rowLimit+1).Scan(&rows); err != nil { + return time.Time{}, err + } + if rows <= rowLimit { + break + } + minutes := int64(subHi.Sub(subLo) / time.Minute / 2) + if minutes < 1 { + minutes = 1 + } + subHi = subLo.Add(time.Duration(minutes) * time.Minute) + } + return subHi, nil +} + // rollupWindow bounds one pass's scan to (start, end]. start falls back to // just before the oldest ingested row when there's no stored watermark, so a // first pass doesn't open the window at the epoch. @@ -958,21 +1165,11 @@ SET last_ingested_unix_nano = excluded.last_ingested_unix_nano, // rollupSafetyLagNanos is how far behind the max ingested timestamp the rollup // watermark is held, covering the worst-case delay between a row being stamped at -// ingest and committed to the lake (normal flush latency plus a retry or two). +// ingest and committed to Parquet (the bounded commit retry window plus queueing). func (d *Duck) rollupSafetyLagNanos() int64 { return d.rollupLagNanos } -// rollupLagFromConfig derives the watermark safety lag from the flush interval: -// two flush cycles, with a 30s floor. -func rollupLagFromConfig(cfg config.Config) int64 { - lag := 2 * cfg.FlushInterval - if lag < 30*time.Second { - lag = 30 * time.Second - } - return lag.Nanoseconds() -} - func maxServiceRollupWatermark(ctx context.Context, tx *sql.Tx) (int64, error) { var watermark int64 err := tx.QueryRowContext(ctx, ` @@ -1070,7 +1267,7 @@ span_agg AS ( ON a.namespace = s.namespace AND a.bucket = date_trunc('minute', s.start_time) AND a.service = s.service - -- Bound the scan to the affected bucket range so DuckLake prunes parquet by + -- Bound the scan to the affected bucket range so Telemetry prunes parquet by -- start_time stats instead of scanning all history (the join on a computed -- date_trunc bucket alone does not prune). The range is a provable superset -- of the affected buckets — every row in an affected bucket has start_time in @@ -1370,225 +1567,100 @@ UNION ALL SELECT namespace, bucket, caller, callee, calls, avg_ms, error_rate, edge_type FROM messaging_edges;` -// snapshotGraceMinutes is how long a superseded DuckLake snapshot (and the -// parquet files it references) is kept past compaction before expiry/cleanup -// may reclaim it. It must exceed the longest read query — reads don't hold -// the write gate, so a shorter window lets cleanup delete a file mid-scan — while -// staying well under the maintenance interval so file/snapshot growth stays -// bounded. Queries are sub-second to a few seconds; 10 minutes is ample margin. -const snapshotGraceMinutes = 10 - -// runMerge runs ONLY ducklake_merge_adjacent_files on a short cadence -// (storage.merge_interval, default 1m). Merge consolidates the newest -// small parquet files and deletes nothing, so it's cheap and safe to run often — -// keeping the queryable file count continuously low is what bounds rollup/query -// scan latency. The deletions (expire_snapshots + cleanup_old_files), which -// carry the read race and catalog cost, stay on the hourly runMaintenance -// cadence. Decoupling the two resolves the churn-vs-pileup tension: frequent -// cheap merge keeps scans fast; rare deletes keep the race and overhead away. -func (d *Duck) runMerge(ctx context.Context) error { - start := time.Now() - every := d.cfg.MergeInterval - if every <= 0 { - metrics.RecordDuckLakeOperation(metrics.DuckLakeMerge, metrics.DuckLakeDisabled, 0) - return nil // merge pass disabled - } - if !d.lastMerge.IsZero() && time.Since(d.lastMerge) < every { - metrics.RecordDuckLakeOperation(metrics.DuckLakeMerge, metrics.DuckLakeThrottled, 0) - return nil +// ---- Read query helpers ---- + +// QueryContext executes a read against immutable Parquet files and DuckDB's +// local rollup cache. +func (d *Duck) QueryContext(ctx context.Context, query string, args ...any) (queryrows.Rows, error) { + if err := d.lockParquetRead(ctx); err != nil { + return nil, err } - // merge commits new files — serialize against other writers like maintenance. - err := func() error { - defer d.writeGate.Lock(writegate.WriteMerge)() - _, err := d.DB.ExecContext(ctx, "CALL ducklake_merge_adjacent_files('lake')") - return err - }() - d.lastMerge = time.Now() + rows, err := d.DB.QueryContext(ctx, query, args...) if err != nil { - metrics.RecordDuckLakeOperation(metrics.DuckLakeMerge, metrics.DuckLakeError, time.Since(start).Seconds()) - slog.Error("merge_adjacent_files failed", "err", err) - return fmt.Errorf("merge_adjacent_files: %w", err) + d.parquetMu.RUnlock() + return nil, err } - metrics.RecordDuckLakeOperation(metrics.DuckLakeMerge, metrics.DuckLakeSuccess, time.Since(start).Seconds()) - return nil + return &lockedRows{Rows: rows, unlock: d.parquetMu.RUnlock}, nil } -func (d *Duck) runMaintenance(ctx context.Context) error { - start := time.Now() - every := d.cfg.MaintenanceInterval - if every <= 0 { - every = time.Hour - } - if !d.lastMaintenance.IsZero() && time.Since(d.lastMaintenance) < every { - metrics.RecordDuckLakeOperation(metrics.DuckLakeMaintenance, metrics.DuckLakeThrottled, 0) - return nil - } +type lockedRows struct { + *sql.Rows + unlockOnce sync.Once + unlock func() +} - // Retention deletes and the checkpoint are writes — serialize them too. - unlock := d.writeGate.Lock(writegate.WriteMaintenance) - defer unlock() +func (r *lockedRows) Close() error { + err := r.Rows.Close() + r.release() + return err +} - var errs []error - if d.cfg.RetentionDays > 0 { - stmts := []struct { - name string - sql string - }{ - {name: "lake.spans", sql: fmt.Sprintf("DELETE FROM lake.spans WHERE COALESCE(start_time, ingested_at) < now() - INTERVAL %d DAY", d.cfg.RetentionDays)}, - {name: "lake.logs", sql: fmt.Sprintf("DELETE FROM lake.logs WHERE COALESCE(log_time, observed_time, ingested_at) < now() - INTERVAL %d DAY", d.cfg.RetentionDays)}, - {name: "lake.metrics", sql: fmt.Sprintf("DELETE FROM lake.metrics WHERE COALESCE(metric_time, ingested_at) < now() - INTERVAL %d DAY", d.cfg.RetentionDays)}, - {name: "service_rollup", sql: fmt.Sprintf("DELETE FROM service_rollup WHERE bucket < now() - INTERVAL %d DAY", d.cfg.RetentionDays)}, - {name: "endpoint_rollup", sql: fmt.Sprintf("DELETE FROM endpoint_rollup WHERE bucket < now() - INTERVAL %d DAY", d.cfg.RetentionDays)}, - {name: "edge_rollup", sql: fmt.Sprintf("DELETE FROM edge_rollup WHERE bucket < now() - INTERVAL %d DAY", d.cfg.RetentionDays)}, - } - for _, stmt := range stmts { - res, err := d.DB.ExecContext(ctx, stmt.sql) - if err != nil { - slog.Error("maintenance delete failed", "table", stmt.name, "err", err) - errs = append(errs, fmt.Errorf("%s retention delete: %w", stmt.name, err)) - continue - } - rows, rowsErr := res.RowsAffected() - if rowsErr != nil { - slog.Info("maintenance delete complete", "table", stmt.name) - continue - } - slog.Info("maintenance delete complete", "table", stmt.name, "rows", rows) - } +func (r *lockedRows) Next() bool { + ok := r.Rows.Next() + if !ok { + r.release() } + return ok +} - // DuckLake compaction. Every flush commits a snapshot and writes new parquet - // files; without merge + expiry both grow without bound until per-file - // metadata pins OOM every wide query (prod 2026-06-13: 60k snapshots, 50k - // files averaging 21KB, rollups dead at any memory_limit). Holding the write gate - // here quiesces the dataset against WRITES, the precondition for these calls. - // Order: merge rewrites small files into large ones, expiry releases the - // snapshots that referenced the small ones, cleanup deletes the files no live - // snapshot references. DuckLake never expires the newest snapshot, so a - // low-traffic instance (no new commits within the grace) still keeps a - // readable one. - // - // GRACE WINDOW (now() - snapshotGraceMinutes) on EXPIRY: the write gate does NOT - // serialize reads, so an Overview/diagnose query can be mid-scan against a - // snapshot merge just superseded; expiring + deleting its parquet immediately - // yanks the file out from under the reader ("IO Error: Cannot open file …: No - // such file or directory"). Sparing recently-superseded snapshots keeps their - // files referenced (so cleanup won't delete them) until any in-flight reader - // has finished; they're reclaimed a cycle later. At ingest.flush_interval=15s the grace - // retains ~40 snapshots (4/min × 10min) — bounded, nowhere near the 60k OOM, - // and far below the (default 1h) maintenance cycle. - // - // cleanup stays cleanup_all => true: the bundled DuckLake's - // ducklake_cleanup_old_files does NOT accept an older_than grace (verified by - // benchmark — the call errored every cycle), and it isn't needed: expiry's - // grace already prevents within-grace files from being scheduled for deletion, - // so cleanup_all only unlinks files no live snapshot references. - expireSQL := fmt.Sprintf( - "CALL ducklake_expire_snapshots('lake', older_than => now() - INTERVAL %d MINUTE)", - snapshotGraceMinutes) - for _, stmt := range []struct { - name string - sql string - }{ - {name: "merge_adjacent_files", sql: "CALL ducklake_merge_adjacent_files('lake')"}, - // DuckLake deletes are merge-on-read. Rewriting materializes the retention - // deletes into Parquet so expired rows stop consuming scan and disk budget. - {name: "rewrite_data_files", sql: "CALL ducklake_rewrite_data_files('lake')"}, - {name: "expire_snapshots", sql: expireSQL}, - {name: "cleanup_old_files", sql: "CALL ducklake_cleanup_old_files('lake', cleanup_all => true)"}, - } { - if _, err := d.DB.ExecContext(ctx, stmt.sql); err != nil { - slog.Error("maintenance compaction failed", "step", stmt.name, "err", err) - errs = append(errs, fmt.Errorf("compaction %s: %w", stmt.name, err)) - } - } +func (r *lockedRows) release() { r.unlockOnce.Do(r.unlock) } - // DuckLake's catalog CHECKPOINT (as opposed to the compaction calls above) - // currently trips an internal error on live datasets with nullable string - // fields, which invalidates the whole database connection. Checkpoint only - // the main local cache until the upstream checkpoint path is stable. - if _, err := d.DB.ExecContext(ctx, "CHECKPOINT"); err != nil { - slog.Error("maintenance checkpoint failed", "target", "main", "err", err) - errs = append(errs, fmt.Errorf("checkpoint main: %w", err)) +// QueryRowScan executes a single-row query against immutable Parquet files and +// DuckDB's local rollup cache. +func (d *Duck) QueryRowScan(ctx context.Context, dest []any, query string, args ...any) error { + if err := d.lockParquetRead(ctx); err != nil { + return err } - d.lastMaintenance = time.Now() + defer d.parquetMu.RUnlock() + return d.DB.QueryRowContext(ctx, query, args...).Scan(dest...) +} - err := errors.Join(errs...) - result := metrics.DuckLakeSuccess - if err != nil { - result = metrics.DuckLakeError - } - metrics.RecordDuckLakeOperation(metrics.DuckLakeMaintenance, result, time.Since(start).Seconds()) - d.maintHealthMu.Lock() - d.lastMaintenanceAt = time.Now() - if err == nil { - d.lastMaintenanceOK = d.lastMaintenanceAt +// Trace pins the immutable Parquet snapshot for the full indexed-file read. +func (d *Duck) Trace(ctx context.Context, query telemetry.TraceQuery) ([]telemetry.IndexedSpan, error) { + if err := d.lockParquetRead(ctx); err != nil { + return nil, err } - d.lastMaintenanceErr = err - d.maintHealthMu.Unlock() - return err + defer d.parquetMu.RUnlock() + return d.repository.Parquet.Trace(ctx, query) } -// ---- Read query helpers ---- - -// isTransientLakeIOError reports whether err is a DuckLake compaction race: a -// concurrent maintenance pass (merge + cleanup, which runs without blocking -// reads) unlinked a parquet file this read had planned against. Re-running the -// query re-plans against the current snapshot (the merged file), which succeeds. -func isTransientLakeIOError(err error) bool { - if err == nil { - return false +// PublishParquet limits reader exclusion to the atomic directory swap. +// +// The drain and the swap get separate budgets. Sharing one deadline meant a +// publisher that spent most of it waiting for readers entered the exclusive +// window with almost nothing left, cancelled its own renames, and failed the +// pass — having already paid the full cost of excluding every reader. The swap +// is renames plus fsync against a bounded batch count, so it is budgeted for +// what that work costs rather than for whatever the drain happened to leave. +func (d *Duck) PublishParquet(ctx context.Context, publish func(context.Context) error) error { + drainCtx, cancelDrain := context.WithTimeout(ctx, parquetDrainBudget) + defer cancelDrain() + metrics.ParquetPublishWaiters.Inc() + waitStarted := time.Now() + err := d.parquetMu.LockContext(drainCtx) + metrics.ParquetPublishWaiters.Dec() + metrics.ParquetPublishWait.Observe(time.Since(waitStarted).Seconds()) + if err != nil { + metrics.ParquetPublishTimeouts.Inc() + return fmt.Errorf("wait for Parquet readers: %w", err) } - msg := err.Error() - return strings.Contains(msg, "IO Error") && strings.Contains(msg, "No such file or directory") + defer d.parquetMu.Unlock() + swapCtx, cancelSwap := context.WithTimeout(ctx, parquetSwapBudget) + defer cancelSwap() + return publish(swapCtx) } -// QueryContext runs a read query, retrying briefly on the transient DuckLake -// "file deleted mid-scan" race (reads don't take the write gate, so maintenance can -// unlink a just-merged file underneath them). Cleanup is instantaneous, so a -// short backoff lets the retry re-plan against fresh files. Use this for every -// read that scans the lake instead of DB.QueryContext directly. -func (d *Duck) QueryContext(ctx context.Context, query string, args ...any) (*sql.Rows, error) { - const maxAttempts = 3 - var rows *sql.Rows - var err error - for attempt := 1; attempt <= maxAttempts; attempt++ { - rows, err = d.DB.QueryContext(ctx, query, args...) - if err == nil || attempt == maxAttempts || !isTransientLakeIOError(err) { - return rows, err - } - if rows != nil { - _ = rows.Close() - } - select { - case <-ctx.Done(): - return nil, ctx.Err() - case <-time.After(time.Duration(attempt) * 25 * time.Millisecond): - } +func (d *Duck) lockParquetRead(ctx context.Context) error { + if err := d.parquetMu.RLockContext(ctx); err != nil { + return errors.Join(ErrParquetReadWait, err) } - return rows, err + return nil } -// QueryRowScan is the single-row analogue of QueryContext: it retries the same -// transient DuckLake "file deleted mid-scan" race. For a single-row read the IO -// error surfaces at Scan (not at QueryRowContext), so the retry wraps -// QueryRowContext+Scan together. Use it for lake-scanning single-row reads -// (`*Duck.QueryContext` covers the multi-row ones). -func (d *Duck) QueryRowScan(ctx context.Context, dest []any, query string, args ...any) error { - const maxAttempts = 3 - var err error - for attempt := 1; attempt <= maxAttempts; attempt++ { - err = d.DB.QueryRowContext(ctx, query, args...).Scan(dest...) - if err == nil || attempt == maxAttempts || !isTransientLakeIOError(err) { - return err - } - select { - case <-ctx.Done(): - return ctx.Err() - case <-time.After(time.Duration(attempt) * 25 * time.Millisecond): - } - } - return err +func (d *Duck) lockRollupParquetRead(ctx context.Context) error { + waitCtx, cancel := context.WithTimeout(ctx, rollupReaderLease) + defer cancel() + return d.lockParquetRead(waitCtx) } // ---- Queries for API ---- @@ -1616,7 +1688,7 @@ HAVING SUM(spans) > 0 ORDER BY p95_ms DESC LIMIT 100; `, windowMinutes) - rows, err := d.DB.QueryContext(ctx, q, namespace, namespace) + rows, err := d.QueryContext(ctx, q, namespace, namespace) if err != nil { return nil, err } @@ -1654,7 +1726,7 @@ WHERE time >= now() - INTERVAL %d MINUTE ORDER BY time DESC LIMIT %d; `, windowMinutes, limit) - rows, err := d.DB.QueryContext(ctx, q, namespace, namespace, pattern) + rows, err := d.QueryContext(ctx, q, namespace, namespace, pattern) if err != nil { return nil, err } @@ -1686,7 +1758,7 @@ WHERE bucket >= now() - INTERVAL %d MINUTE GROUP BY bucket ORDER BY bucket ASC; `, windowMinutes) - rows, err := d.DB.QueryContext(ctx, q, namespace, namespace) + rows, err := d.QueryContext(ctx, q, namespace, namespace) if err != nil { return nil, err } @@ -1718,7 +1790,7 @@ GROUP BY service ORDER BY spans_per_minute DESC LIMIT 20; `, windowMinutes, windowMinutes) - rows, err := d.DB.QueryContext(ctx, q, namespace, namespace) + rows, err := d.QueryContext(ctx, q, namespace, namespace) if err != nil { return nil, err } @@ -1751,7 +1823,7 @@ GROUP BY body ORDER BY count DESC LIMIT %d; `, windowMinutes, limit) - rows, err := d.DB.QueryContext(ctx, q, namespace, namespace) + rows, err := d.QueryContext(ctx, q, namespace, namespace) if err != nil { return nil, err } @@ -1793,7 +1865,7 @@ HAVING errors > 0 ORDER BY errors DESC LIMIT %d; `, windowMinutes, limit) - rows, err := d.DB.QueryContext(ctx, q, namespace, namespace) + rows, err := d.QueryContext(ctx, q, namespace, namespace) if err != nil { return nil, err } diff --git a/internal/query/duck_test.go b/internal/query/duck_test.go index f3f37ea8..67a1521b 100644 --- a/internal/query/duck_test.go +++ b/internal/query/duck_test.go @@ -2,15 +2,20 @@ package query import ( "context" + "database/sql" "errors" - "regexp" - "strings" + "fmt" + "os" + "path/filepath" "testing" "time" "github.com/DATA-DOG/go-sqlmock" "github.com/labstack/fanout/internal/config" "github.com/labstack/fanout/internal/metrics" + "github.com/labstack/fanout/internal/query/writegate" + "github.com/labstack/fanout/internal/telemetry" + telemetrystore "github.com/labstack/fanout/internal/telemetry/store" "github.com/prometheus/client_golang/prometheus/testutil" ) @@ -122,246 +127,482 @@ func TestErrorRouteRowStruct(t *testing.T) { } } -func TestRunMaintenanceContinuesAfterDeleteFailure(t *testing.T) { - metrics.DuckLakeOperationTotal.Reset() - db, mock, err := sqlmock.New() +func TestNewDuckUsesSingleConnectionPool(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + cfg := config.Config{ + DataDir: t.TempDir(), + RollupInterval: time.Minute, + DuckDBMemory: "128MB", + } + + repository, err := telemetrystore.Open(cfg.TelemetryDir()) if err != nil { - t.Fatalf("sqlmock.New: %v", err) + t.Fatalf("open telemetry repository: %v", err) } - defer db.Close() + defer repository.Close() + d, err := NewDuck(ctx, cfg, repository) + if err != nil { + t.Fatalf("NewDuck() error = %v", err) + } + defer d.Close() - d := &Duck{ - DB: db, - cfg: config.Config{RetentionDays: 7}, - } - - mock.ExpectExec(regexp.QuoteMeta("DELETE FROM lake.spans WHERE COALESCE(start_time, ingested_at) < now() - INTERVAL 7 DAY")). - WillReturnResult(sqlmock.NewResult(0, 11)) - mock.ExpectExec(regexp.QuoteMeta("DELETE FROM lake.logs WHERE COALESCE(log_time, observed_time, ingested_at) < now() - INTERVAL 7 DAY")). - WillReturnError(errors.New("log delete failed")) - mock.ExpectExec(regexp.QuoteMeta("DELETE FROM lake.metrics WHERE COALESCE(metric_time, ingested_at) < now() - INTERVAL 7 DAY")). - WillReturnResult(sqlmock.NewResult(0, 7)) - mock.ExpectExec(regexp.QuoteMeta("DELETE FROM service_rollup WHERE bucket < now() - INTERVAL 7 DAY")). - WillReturnResult(sqlmock.NewResult(0, 5)) - mock.ExpectExec(regexp.QuoteMeta("DELETE FROM endpoint_rollup WHERE bucket < now() - INTERVAL 7 DAY")). - WillReturnResult(sqlmock.NewResult(0, 4)) - mock.ExpectExec(regexp.QuoteMeta("DELETE FROM edge_rollup WHERE bucket < now() - INTERVAL 7 DAY")). - WillReturnResult(sqlmock.NewResult(0, 3)) - mock.ExpectExec(regexp.QuoteMeta("CALL ducklake_merge_adjacent_files('lake')")). - WillReturnResult(sqlmock.NewResult(0, 0)) - mock.ExpectExec(regexp.QuoteMeta("CALL ducklake_rewrite_data_files('lake')")). - WillReturnResult(sqlmock.NewResult(0, 0)) - mock.ExpectExec(regexp.QuoteMeta("CALL ducklake_expire_snapshots('lake', older_than => now() - INTERVAL 10 MINUTE)")). - WillReturnResult(sqlmock.NewResult(0, 0)) - mock.ExpectExec(regexp.QuoteMeta("CALL ducklake_cleanup_old_files('lake', cleanup_all => true)")). - WillReturnResult(sqlmock.NewResult(0, 0)) - mock.ExpectExec(regexp.QuoteMeta("CHECKPOINT")). - WillReturnResult(sqlmock.NewResult(0, 0)) - - err = d.runMaintenance(context.Background()) - if err == nil { - t.Fatal("runMaintenance() error = nil, want joined error") - } - if d.lastMaintenance.IsZero() { - t.Fatal("runMaintenance() did not update lastMaintenance") - } - if got := testutil.ToFloat64(metrics.DuckLakeOperationTotal.WithLabelValues("maintenance", "error")); got != 1 { - t.Errorf("maintenance error outcomes = %f, want 1", got) + stats := d.DB.Stats() + if stats.MaxOpenConnections != 1 { + t.Fatalf("MaxOpenConnections = %d, want 1 when DuckDBMaxConns is unset", stats.MaxOpenConnections) } - if err := mock.ExpectationsWereMet(); err != nil { - t.Fatalf("unmet sql expectations: %v", err) +} + +func TestNewDuckUsesUTCForEveryConnection(t *testing.T) { + t.Setenv("TZ", "America/Los_Angeles") + ctx := context.Background() + cfg := config.Config{ + DataDir: t.TempDir(), + RollupInterval: time.Minute, + DuckDBMemory: "128MB", + DuckDBMaxConns: 4, + DuckDBThreads: 2, + MaintenanceInterval: time.Hour, + } + repository, err := telemetrystore.Open(cfg.TelemetryDir()) + if err != nil { + t.Fatalf("open telemetry repository: %v", err) + } + defer repository.Close() + d, err := NewDuck(ctx, cfg, repository) + if err != nil { + t.Fatalf("NewDuck() error = %v", err) + } + defer d.Close() + eventTime := time.Date(2026, 8, 27, 16, 0, 30, 0, time.UTC) + if err := repository.Commit(context.Background(), telemetrystore.Batch{ID: "timezone-window", Spans: []telemetry.Span{{ + Namespace: "default", TraceID: "trace-timezone", SpanID: "span-timezone", + ServiceName: "checkout", StartUnixNanos: eventTime.UnixNano(), DurationMS: 5, + StatusCode: "STATUS_CODE_OK", IngestedAt: eventTime.UnixNano(), + }}}); err != nil { + t.Fatalf("commit timezone fixture: %v", err) + } + d.rollupLagNanos = 0 + if _, err := d.rollupOnce(ctx); err != nil { + t.Fatalf("roll up timezone fixture: %v", err) + } + var spans int64 + if err := d.DB.QueryRowContext(ctx, ` +SELECT COALESCE(SUM(spans), 0)::BIGINT +FROM service_rollup +WHERE bucket >= ? AND bucket < ?`, eventTime.Add(-time.Minute), eventTime.Add(time.Minute)).Scan(&spans); err != nil { + t.Fatalf("query UTC rollup window: %v", err) + } + if spans != 1 { + t.Fatalf("UTC rollup window spans = %d, want 1", spans) + } + + connections := make([]*sql.Conn, 0, cfg.DuckDBMaxConns) + defer func() { + for _, connection := range connections { + _ = connection.Close() + } + }() + for range cfg.DuckDBMaxConns { + connection, err := d.DB.Conn(ctx) + if err != nil { + t.Fatalf("open pooled connection: %v", err) + } + connections = append(connections, connection) + } + for i, connection := range connections { + var zone string + if err := connection.QueryRowContext(ctx, "SELECT current_setting('TimeZone')").Scan(&zone); err != nil { + t.Fatalf("connection %d timezone query: %v", i, err) + } + if zone != "UTC" { + t.Fatalf("connection %d timezone = %q, want UTC", i, zone) + } } } -// A merge failure must not stop snapshot expiry, file cleanup, or the -// checkpoint — each compaction step degrades independently. -// Within storage.maintenance_interval of the last pass, runMaintenance -// must short-circuit before issuing ANY SQL — the throttle that keeps the -// retention+compaction cycle off every rollup tick. -// runMerge issues exactly one merge_adjacent_files call when due, and nothing -// on the next call within the storage.merge_interval cadence. -func TestRunMergeExecutesThenThrottles(t *testing.T) { - metrics.DuckLakeOperationTotal.Reset() +func TestQueryContextHoldsParquetLockUntilRowsFinish(t *testing.T) { db, mock, err := sqlmock.New() if err != nil { t.Fatalf("sqlmock.New: %v", err) } defer db.Close() + mock.ExpectQuery("SELECT 1").WillReturnRows(sqlmock.NewRows([]string{"value"}).AddRow(1)) + d := &Duck{DB: db} + rows, err := d.QueryContext(context.Background(), "SELECT 1") + if err != nil { + t.Fatalf("QueryContext: %v", err) + } + lockAcquired := make(chan struct{}) + go func() { + mustLock(&d.parquetMu) + close(lockAcquired) + d.parquetMu.Unlock() + }() + select { + case <-lockAcquired: + t.Fatal("Parquet write lock acquired while query rows were still open") + case <-time.After(25 * time.Millisecond): + } + if !rows.Next() { + t.Fatalf("rows.Next() = false: %v", rows.Err()) + } + var value int + if err := rows.Scan(&value); err != nil { + t.Fatalf("rows.Scan: %v", err) + } + if rows.Next() { + t.Fatal("rows.Next() returned an unexpected second row") + } + select { + case <-lockAcquired: + case <-time.After(time.Second): + t.Fatal("Parquet write lock remained blocked after row iteration completed") + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatal(err) + } +} - d := &Duck{DB: db, cfg: config.Config{MergeInterval: time.Minute}} - mock.ExpectExec(regexp.QuoteMeta("CALL ducklake_merge_adjacent_files('lake')")). - WillReturnResult(sqlmock.NewResult(0, 0)) +func TestQueryRowScanCancelsWhileMaintenanceWaitsForReaders(t *testing.T) { + d := &Duck{} + mustLock(&d.parquetMu) + ctx, cancel := context.WithTimeout(context.Background(), 25*time.Millisecond) + defer cancel() + started := time.Now() + err := d.QueryRowScan(ctx, []any{new(int)}, "SELECT 1") + d.parquetMu.Unlock() + if !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("QueryRowScan error = %v, want deadline exceeded", err) + } + if elapsed := time.Since(started); elapsed > 250*time.Millisecond { + t.Fatalf("QueryRowScan ignored lock deadline for %s", elapsed) + } +} - if err := d.runMerge(context.Background()); err != nil { - t.Fatalf("runMerge() = %v, want nil", err) +func TestPublishParquetHonorsContext(t *testing.T) { + d := &Duck{} + mustRLock(t, &d.parquetMu) + timeoutsBefore := testutil.ToFloat64(metrics.ParquetPublishTimeouts) + ctx, cancel := context.WithTimeout(context.Background(), 25*time.Millisecond) + defer cancel() + called := false + err := d.PublishParquet(ctx, func(context.Context) error { + called = true + return nil + }) + d.parquetMu.RUnlock() + if !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("PublishParquet error = %v, want deadline exceeded", err) } - // Second call is within the cadence → must issue no SQL. - if err := d.runMerge(context.Background()); err != nil { - t.Fatalf("throttled runMerge() = %v, want nil", err) + if called { + t.Fatal("publication ran after its context expired") } - if err := mock.ExpectationsWereMet(); err != nil { - t.Fatalf("unmet sql expectations: %v", err) + if got := waitingParquetWriters(&d.parquetMu); got != 0 { + t.Fatalf("canceled publication remained queued: %d", got) } - if got := testutil.ToFloat64(metrics.DuckLakeOperationTotal.WithLabelValues("merge", "success")); got != 1 { - t.Errorf("merge success outcomes = %f, want 1", got) + if got := testutil.ToFloat64(metrics.ParquetPublishTimeouts); got != timeoutsBefore+1 { + t.Fatalf("publication timeouts = %v, want %v", got, timeoutsBefore+1) } - if got := testutil.ToFloat64(metrics.DuckLakeOperationTotal.WithLabelValues("merge", "throttled")); got != 1 { - t.Errorf("merge throttled outcomes = %f, want 1", got) + if got := testutil.ToFloat64(metrics.ParquetPublishWaiters); got != 0 { + t.Fatalf("publication waiters after timeout = %v, want 0", got) } } -// MergeInterval=0 disables the frequent merge pass entirely. -func TestRunMergeDisabled(t *testing.T) { - metrics.DuckLakeOperationTotal.Reset() +func TestParquetPublicationDoesNotWaitForDuckDBWriteGate(t *testing.T) { + d := &Duck{} + release := d.writeGate.Lock(writegate.WriteRollupService) + defer release() + ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond) + defer cancel() + called := false + if err := d.PublishParquet(ctx, func(context.Context) error { called = true; return nil }); err != nil { + t.Fatalf("publication waited for unrelated DuckDB write gate: %v", err) + } + if !called { + t.Fatal("publication callback did not run") + } +} + +func TestWaitingMaintenanceDoesNotBlockNewReaders(t *testing.T) { db, mock, err := sqlmock.New() if err != nil { - t.Fatalf("sqlmock.New: %v", err) + t.Fatal(err) } defer db.Close() - - d := &Duck{DB: db, cfg: config.Config{MergeInterval: 0}} - if err := d.runMerge(context.Background()); err != nil { - t.Fatalf("disabled runMerge() = %v, want nil", err) + mock.ExpectQuery("SELECT 1").WillReturnRows(sqlmock.NewRows([]string{"value"}).AddRow(1)) + d := &Duck{DB: db} + mustRLock(t, &d.parquetMu) + writerAcquired := make(chan struct{}) + releaseWriter := make(chan struct{}) + writerDone := make(chan struct{}) + go func() { + mustLock(&d.parquetMu) + close(writerAcquired) + <-releaseWriter + d.parquetMu.Unlock() + close(writerDone) + }() + deadline := time.Now().Add(time.Second) + for waitingParquetWriters(&d.parquetMu) == 0 { + if time.Now().After(deadline) { + d.parquetMu.RUnlock() + t.Fatal("maintenance writer did not begin waiting") + } + time.Sleep(time.Millisecond) + } + var value int + if err := d.QueryRowScan(context.Background(), []any{&value}, "SELECT 1"); err != nil { + d.parquetMu.RUnlock() + t.Fatalf("new read blocked behind waiting maintenance: %v", err) + } + if value != 1 { + d.parquetMu.RUnlock() + t.Fatalf("value = %d, want 1", value) + } + d.parquetMu.RUnlock() + select { + case <-writerAcquired: + case <-time.After(time.Second): + t.Fatal("maintenance writer did not run after readers drained") + } + close(releaseWriter) + select { + case <-writerDone: + case <-time.After(time.Second): + t.Fatal("maintenance writer did not release") } if err := mock.ExpectationsWereMet(); err != nil { - t.Fatalf("disabled runMerge should issue no SQL: %v", err) + t.Fatal(err) } - if got := testutil.ToFloat64(metrics.DuckLakeOperationTotal.WithLabelValues("merge", "disabled")); got != 1 { - t.Errorf("merge disabled outcomes = %f, want 1", got) +} + +func TestRollupReadLockHonorsContext(t *testing.T) { + d := &Duck{} + mustLock(&d.parquetMu) + ctx, cancel := context.WithTimeout(context.Background(), 25*time.Millisecond) + defer cancel() + _, err := d.refreshServiceRollup(ctx) + d.parquetMu.Unlock() + if !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("refreshServiceRollup error = %v, want deadline exceeded", err) } } -func TestRunMaintenanceThrottle(t *testing.T) { - metrics.DuckLakeOperationTotal.Reset() +func TestRollupAdmissionLeaseDoesNotCancelAdmittedWork(t *testing.T) { db, mock, err := sqlmock.New() if err != nil { - t.Fatalf("sqlmock.New: %v", err) + t.Fatal(err) } defer db.Close() - - d := &Duck{DB: db, cfg: config.Config{MaintenanceInterval: time.Hour}, lastMaintenance: time.Now()} - if err := d.runMaintenance(context.Background()); err != nil { - t.Fatalf("throttled runMaintenance() = %v, want nil", err) + originalLease := rollupReaderLease + rollupReaderLease = 5 * time.Millisecond + t.Cleanup(func() { rollupReaderLease = originalLease }) + + mock.ExpectBegin() + mock.ExpectQuery("FROM rollup_state").WithArgs(serviceRollupStateKey). + WillReturnRows(sqlmock.NewRows([]string{"last_ingested_unix_nano"}).AddRow(100)) + mock.ExpectQuery("FROM rollup_state").WithArgs(serviceRollupRawMaxKey). + WillReturnRows(sqlmock.NewRows([]string{"last_ingested_unix_nano"}).AddRow(100)) + mock.ExpectQuery("FROM \\(").WillDelayFor(20 * time.Millisecond). + WillReturnRows(sqlmock.NewRows([]string{"watermark"}).AddRow(100)) + mock.ExpectCommit() + + d := &Duck{DB: db} + if _, err := d.refreshServiceRollup(context.Background()); err != nil { + t.Fatalf("admitted rollup was canceled by its admission lease: %v", err) } - // No expectations were registered: if it ran any SQL, this fails. if err := mock.ExpectationsWereMet(); err != nil { - t.Fatalf("throttled runMaintenance should issue no SQL: %v", err) + t.Fatal(err) } - if got := testutil.ToFloat64(metrics.DuckLakeOperationTotal.WithLabelValues("maintenance", "throttled")); got != 1 { - t.Errorf("maintenance throttled outcomes = %f, want 1", got) +} + +func TestIndexedTraceReadHonorsParquetGateContext(t *testing.T) { + repository, err := telemetrystore.Open(t.TempDir()) + if err != nil { + t.Fatal(err) + } + defer repository.Close() + d := &Duck{repository: repository} + mustLock(&d.parquetMu) + ctx, cancel := context.WithTimeout(context.Background(), 25*time.Millisecond) + defer cancel() + _, err = d.Trace(ctx, telemetry.TraceQuery{TraceID: "trace", StartNanos: 1, EndNanos: 2, Limit: 1}) + d.parquetMu.Unlock() + if !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("Trace error = %v, want deadline exceeded", err) + } +} + +func TestParquetStatsWaitForStableNamespace(t *testing.T) { + repository, err := telemetrystore.Open(t.TempDir()) + if err != nil { + t.Fatal(err) + } + defer repository.Close() + d := &Duck{repository: repository} + mustLock(&d.parquetMu) + done := make(chan struct{}) + go func() { + d.updateParquetStats(context.Background()) + close(done) + }() + select { + case <-done: + d.parquetMu.Unlock() + t.Fatal("Parquet stats read while the batch namespace was changing") + case <-time.After(25 * time.Millisecond): + } + d.parquetMu.Unlock() + select { + case <-done: + case <-time.After(time.Second): + t.Fatal("Parquet stats did not resume after publication") } } -func TestRunMaintenanceContinuesAfterCompactionFailure(t *testing.T) { - metrics.DuckLakeOperationTotal.Reset() +func TestRepositoryPublicationDoesNotWaitForDuckDBWrites(t *testing.T) { db, mock, err := sqlmock.New() if err != nil { - t.Fatalf("sqlmock.New: %v", err) + t.Fatal(err) } defer db.Close() + repository, err := telemetrystore.Open(t.TempDir()) + if err != nil { + t.Fatal(err) + } + defer repository.Close() + if err := repository.Commit(context.Background(), telemetrystore.Batch{ID: "expired", Spans: []telemetry.Span{{TraceID: "trace", IngestedAt: 1}}}); err != nil { + t.Fatal(err) + } + mock.ExpectExec("DELETE FROM service_rollup").WillReturnResult(sqlmock.NewResult(0, 0)) + mock.ExpectExec("DELETE FROM endpoint_rollup").WillReturnResult(sqlmock.NewResult(0, 0)) + mock.ExpectExec("DELETE FROM edge_rollup").WillReturnResult(sqlmock.NewResult(0, 0)) + mock.ExpectExec("CHECKPOINT").WillReturnResult(sqlmock.NewResult(0, 0)) + d := &Duck{DB: db, repository: repository, cfg: config.Config{MaintenanceInterval: time.Nanosecond, RetentionDays: 1}} + mustRLock(t, &d.parquetMu) + release := d.writeGate.Lock(writegate.WriteRollupService) + done := make(chan error, 1) + go func() { done <- d.runRepositoryMaintenance(context.Background()) }() + deadline := time.Now().Add(time.Second) + for waitingParquetWriters(&d.parquetMu) == 0 && time.Now().Before(deadline) { + time.Sleep(time.Millisecond) + } + if waiting := waitingParquetWriters(&d.parquetMu); waiting != 1 { + d.parquetMu.RUnlock() + release() + t.Fatalf("publication waiters = %d, want 1 while DuckDB write gate is independently held", waiting) + } + d.parquetMu.RUnlock() + release() + if err := <-done; err != nil { + t.Fatal(err) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatal(err) + } +} - d := &Duck{DB: db, cfg: config.Config{}} - - mock.ExpectExec(regexp.QuoteMeta("CALL ducklake_merge_adjacent_files('lake')")). - WillReturnError(errors.New("merge failed")) - mock.ExpectExec(regexp.QuoteMeta("CALL ducklake_rewrite_data_files('lake')")). - WillReturnResult(sqlmock.NewResult(0, 0)) - mock.ExpectExec(regexp.QuoteMeta("CALL ducklake_expire_snapshots('lake', older_than => now() - INTERVAL 10 MINUTE)")). - WillReturnResult(sqlmock.NewResult(0, 0)) - mock.ExpectExec(regexp.QuoteMeta("CALL ducklake_cleanup_old_files('lake', cleanup_all => true)")). - WillReturnResult(sqlmock.NewResult(0, 0)) - mock.ExpectExec(regexp.QuoteMeta("CHECKPOINT")). - WillReturnResult(sqlmock.NewResult(0, 0)) - - err = d.runMaintenance(context.Background()) - if err == nil { - t.Fatal("runMaintenance() error = nil, want merge error surfaced") +func TestMaintenanceRetriesRetiredDirectoryCleanup(t *testing.T) { + repository, err := telemetrystore.Open(t.TempDir()) + if err != nil { + t.Fatal(err) + } + defer repository.Close() + retired := filepath.Join(repository.Parquet.BatchesDir(), "stale.retired-old") + if err := os.Mkdir(retired, 0o755); err != nil { + t.Fatal(err) + } + db, mock, err := sqlmock.New() + if err != nil { + t.Fatal(err) } - if lastOK, lastAt, lastErr := d.MaintenanceHealth(); lastErr == nil || !lastOK.IsZero() || lastAt.IsZero() { - t.Fatalf("MaintenanceHealth() = (%v, %v, %v), want zero lastOK, non-zero lastAt, non-nil lastErr", lastOK, lastAt, lastErr) + defer db.Close() + mock.ExpectExec("CHECKPOINT").WillReturnResult(sqlmock.NewResult(0, 0)) + d := &Duck{DB: db, repository: repository} + if err := d.runRepositoryMaintenance(context.Background()); err != nil { + t.Fatal(err) } - if got := testutil.ToFloat64(metrics.DuckLakeOperationTotal.WithLabelValues("maintenance", "error")); got != 1 { - t.Errorf("maintenance error outcomes = %f, want 1", got) + if _, err := os.Stat(retired); !os.IsNotExist(err) { + t.Fatalf("retired directory remains after maintenance: %v", err) } if err := mock.ExpectationsWereMet(); err != nil { - t.Fatalf("unmet sql expectations: %v", err) + t.Fatal(err) } } -// A failing rollup must not skip maintenance: file/snapshot growth is what -// makes the failing rollup heavier each retry, so compaction has to run anyway. -func TestRollupOnceRunsMaintenanceDespiteRollupFailure(t *testing.T) { - metrics.RollupComponentTotal.Reset() - metrics.DuckLakeOperationTotal.Reset() +func TestMaintenanceTracksConsecutiveFailures(t *testing.T) { db, mock, err := sqlmock.New() if err != nil { - t.Fatalf("sqlmock.New: %v", err) + t.Fatal(err) } defer db.Close() - - d := &Duck{DB: db, cfg: config.Config{}} - - // All rollup transactions fail at BeginTx. - mock.ExpectBegin().WillReturnError(errors.New("service tx failed")) - mock.ExpectBegin().WillReturnError(errors.New("endpoint tx failed")) - mock.ExpectBegin().WillReturnError(errors.New("edge tx failed")) - // Maintenance still runs: compaction calls + checkpoint (RetentionDays=0 - // skips the TTL deletes). - mock.ExpectExec(regexp.QuoteMeta("CALL ducklake_merge_adjacent_files('lake')")). - WillReturnResult(sqlmock.NewResult(0, 0)) - mock.ExpectExec(regexp.QuoteMeta("CALL ducklake_rewrite_data_files('lake')")). - WillReturnResult(sqlmock.NewResult(0, 0)) - mock.ExpectExec(regexp.QuoteMeta("CALL ducklake_expire_snapshots('lake', older_than => now() - INTERVAL 10 MINUTE)")). - WillReturnResult(sqlmock.NewResult(0, 0)) - mock.ExpectExec(regexp.QuoteMeta("CALL ducklake_cleanup_old_files('lake', cleanup_all => true)")). - WillReturnResult(sqlmock.NewResult(0, 0)) - mock.ExpectExec(regexp.QuoteMeta("CHECKPOINT")). - WillReturnResult(sqlmock.NewResult(0, 0)) - - _, err = d.rollupOnce(context.Background()) - if err == nil { - t.Fatal("rollupOnce() error = nil, want rollup errors surfaced") - } - if lastOK, lastAt, lastErr := d.MaintenanceHealth(); lastErr != nil || lastOK.IsZero() || lastAt.IsZero() { - t.Fatalf("MaintenanceHealth() = (%v, %v, %v), want non-zero lastOK and lastAt, nil lastErr", lastOK, lastAt, lastErr) - } - for _, component := range []string{"service", "endpoint", "edge"} { - if got := testutil.ToFloat64(metrics.RollupComponentTotal.WithLabelValues(component, "error")); got != 1 { - t.Errorf("%s rollup error outcomes = %f, want 1", component, got) + for range 3 { + mock.ExpectExec("CHECKPOINT").WillReturnError(errors.New("injected checkpoint failure")) + } + mock.ExpectExec("CHECKPOINT").WillReturnResult(sqlmock.NewResult(0, 0)) + d := &Duck{DB: db} + for range 3 { + if err := d.runRepositoryMaintenance(context.Background()); err == nil { + t.Fatal("maintenance succeeded despite checkpoint failure") } } - if got := testutil.ToFloat64(metrics.DuckLakeOperationTotal.WithLabelValues("maintenance", "success")); got != 1 { - t.Errorf("maintenance success outcomes = %f, want 1", got) + _, _, failures, _ := d.MaintenanceHealth() + if failures != 3 { + t.Fatalf("consecutive failures = %d, want 3", failures) + } + if err := d.runRepositoryMaintenance(context.Background()); err != nil { + t.Fatal(err) + } + _, _, failures, _ = d.MaintenanceHealth() + if failures != 0 { + t.Fatalf("consecutive failures after recovery = %d, want 0", failures) } if err := mock.ExpectationsWereMet(); err != nil { - t.Fatalf("unmet sql expectations: %v", err) + t.Fatal(err) } } -func TestNewDuckUsesSingleConnectionPool(t *testing.T) { - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() +type failingPublishCompactor struct{ *Duck } - cfg := config.Config{ - DataDir: t.TempDir(), - RollupInterval: time.Minute, - DuckDBMemory: "128MB", - } +func (f failingPublishCompactor) PublishParquet(context.Context, func(context.Context) error) error { + return errors.New("injected publication failure") +} - d, err := NewDuck(ctx, cfg) +func TestMaintenanceRecoversCompactionBeforeRetention(t *testing.T) { + repository, err := telemetrystore.Open(t.TempDir()) if err != nil { - if strings.Contains(err.Error(), "LOAD ducklake") || - strings.Contains(err.Error(), "LOAD sqlite") || - strings.Contains(err.Error(), "ATTACH") { - t.Skipf("DuckLake extensions unavailable: %v", err) + t.Fatal(err) + } + defer repository.Close() + db := openTestDuck(t) + defer db.Close() + for _, table := range []string{"service_rollup", "endpoint_rollup", "edge_rollup"} { + if _, err := db.Exec("CREATE TABLE " + table + " (bucket TIMESTAMP)"); err != nil { + t.Fatal(err) } - t.Fatalf("NewDuck() error = %v", err) } - defer d.Close() - - stats := d.DB.Stats() - if stats.MaxOpenConnections != 1 { - t.Fatalf("MaxOpenConnections = %d, want 1 when DuckDBMaxConns is unset (floored for DuckLake serialization)", stats.MaxOpenConnections) + d := &Duck{DB: db, repository: repository, cfg: config.Config{RetentionDays: 1}} + for i := range 8 { + batch := telemetrystore.Batch{ + ID: fmt.Sprintf("expired-%d", i), + Spans: []telemetry.Span{{TraceID: "trace", SpanID: fmt.Sprintf("span-%d", i), StartUnixNanos: 1, IngestedAt: 1}}, + Logs: []telemetry.Log{{Body: "old", IngestedAt: 1}}, + Metrics: []telemetry.Metric{{Name: "old", IngestedAt: 1}}, + } + if err := repository.Commit(context.Background(), batch); err != nil { + t.Fatal(err) + } + } + if _, err := repository.CompactParquet(context.Background(), failingPublishCompactor{d}, 8); err == nil { + t.Fatal("compaction unexpectedly published") + } + if err := d.runRepositoryMaintenance(context.Background()); err != nil { + t.Fatal(err) + } + if got := repository.RowCount(); got != 0 { + t.Fatalf("retention resurrected pending compaction rows: %d", got) } } @@ -533,3 +774,53 @@ func TestFailedRollupStillPublishesLag(t *testing.T) { }) } } + +func TestMaintenanceRunsOnEveryTick(t *testing.T) { + db, mock, err := sqlmock.New() + if err != nil { + t.Fatal(err) + } + defer db.Close() + mock.ExpectExec("CHECKPOINT").WillReturnResult(sqlmock.NewResult(0, 0)) + mock.ExpectExec("CHECKPOINT").WillReturnResult(sqlmock.NewResult(0, 0)) + d := &Duck{DB: db, cfg: config.Config{MaintenanceInterval: time.Hour}} + if err := d.runRepositoryMaintenance(context.Background()); err != nil { + t.Fatal(err) + } + first := d.lastMaintenanceAt + if err := d.runRepositoryMaintenance(context.Background()); err != nil { + t.Fatal(err) + } + if !d.lastMaintenanceAt.After(first) { + t.Fatal("second maintenance pass was throttled; the ticker is the only intended throttle") + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatal(err) + } +} + +// TestUpdateParquetStatsDoesNotParkOnAStalledPublication pins that the gauge +// refresh gives up rather than waiting out a stuck publication. It is the only +// Parquet reader that runs on the rollup and maintenance loop goroutines under +// the process context, so an unbounded wait here stops those loops entirely — +// the same failure this storage layer has produced repeatedly, one level down. +func TestUpdateParquetStatsDoesNotParkOnAStalledPublication(t *testing.T) { + originalWait := parquetStatsWait + parquetStatsWait = 25 * time.Millisecond + t.Cleanup(func() { parquetStatsWait = originalWait }) + + d := &Duck{} + mustLock(&d.parquetMu) + defer d.parquetMu.Unlock() + + done := make(chan struct{}) + go func() { + defer close(done) + d.updateParquetStats(context.Background()) + }() + select { + case <-done: + case <-time.After(time.Second): + t.Fatal("stats refresh parked on a stalled publication instead of giving up") + } +} diff --git a/internal/query/duck_wal_test.go b/internal/query/duck_wal_test.go deleted file mode 100644 index bfb76801..00000000 --- a/internal/query/duck_wal_test.go +++ /dev/null @@ -1,232 +0,0 @@ -package query - -import ( - "context" - "database/sql" - "path/filepath" - "strings" - "sync" - "testing" - "time" - - "github.com/labstack/fanout/internal/config" - "github.com/labstack/fanout/internal/lake/writegate" -) - -// catalogJournalMode opens the SQLite catalog at path with the modernc driver -// (registered via duck.go's blank import) and returns its journal_mode. -func catalogJournalMode(t *testing.T, path string) string { - t.Helper() - db, err := sql.Open("sqlite", path+"?_pragma=busy_timeout(5000)") - if err != nil { - t.Fatalf("open catalog: %v", err) - } - defer db.Close() - var mode string - if err := db.QueryRow("PRAGMA journal_mode").Scan(&mode); err != nil { - t.Fatalf("read journal_mode: %v", err) - } - return mode -} - -// TestEnableCatalogWAL covers the catalog WAL bootstrap in isolation: it must -// set WAL, be idempotent, and recover (not permanently lock) when a populated -// -wal file from an unclean writer is still present. -func TestEnableCatalogWAL(t *testing.T) { - path := filepath.Join(t.TempDir(), "catalog.sqlite") - - if err := enableCatalogWAL(path); err != nil { - t.Fatalf("enableCatalogWAL: %v", err) - } - if mode := catalogJournalMode(t, path); !strings.EqualFold(mode, "wal") { - t.Fatalf("journal_mode = %q, want wal", mode) - } - - // Write through a connection and abandon it without checkpointing, leaving a - // populated -wal file — the rough shape a crashed/stuck writer leaves behind. - // Re-running the bootstrap must still succeed and report WAL. - db, err := sql.Open("sqlite", path+"?_pragma=busy_timeout(5000)") - if err != nil { - t.Fatalf("open: %v", err) - } - defer db.Close() - if _, err := db.Exec("CREATE TABLE t(id INTEGER)"); err != nil { - t.Fatalf("create: %v", err) - } - if _, err := db.Exec("INSERT INTO t VALUES (1),(2),(3)"); err != nil { - t.Fatalf("insert: %v", err) - } - // Intentionally do NOT close db before re-running the bootstrap. - - if err := enableCatalogWAL(path); err != nil { - t.Fatalf("enableCatalogWAL (idempotent/recovery): %v", err) - } - if mode := catalogJournalMode(t, path); !strings.EqualFold(mode, "wal") { - t.Fatalf("journal_mode after reopen = %q, want wal", mode) - } -} - -// newTestDuck spins up a real DuckDB+DuckLake against a temp dir, skipping when -// the DuckLake/SQLite extensions aren't available in the environment. -func newTestDuck(t *testing.T, maxConns int) *Duck { - t.Helper() - ctx, cancel := context.WithCancel(context.Background()) - t.Cleanup(cancel) - - cfg := config.Config{ - DataDir: t.TempDir(), - RollupInterval: time.Minute, - DuckDBMemory: "128MB", - DuckDBMaxConns: maxConns, - } - d, err := NewDuck(ctx, cfg) - if err != nil { - if strings.Contains(err.Error(), "LOAD ducklake") || - strings.Contains(err.Error(), "LOAD sqlite") || - strings.Contains(err.Error(), "ATTACH") { - t.Skipf("DuckLake extensions unavailable: %v", err) - } - t.Fatalf("NewDuck() error = %v", err) - } - t.Cleanup(func() { d.Close() }) - return d -} - -// TestNewDuckCatalogUsesWAL verifies NewDuck leaves the DuckLake catalog in WAL -// mode end-to-end. -func TestNewDuckCatalogUsesWAL(t *testing.T) { - d := newTestDuck(t, 1) - if mode := catalogJournalMode(t, d.cfg.TelemetryDuckLakePath()); !strings.EqualFold(mode, "wal") { - t.Fatalf("catalog journal_mode = %q, want wal", mode) - } -} - -// TestPoolConcurrentReadsNoLock exercises the WAL fix: with a multi-connection -// pool, concurrent read queries running alongside serialized writes must not -// fail with "database is locked". A start barrier releases all goroutines at -// once so the pool is forced to open multiple physical connections (otherwise a -// low-contention run could funnel through a single reused handle and never -// exercise concurrency). The temp_directory fix is covered separately by -// TestTempDirectorySetOnce. -func TestPoolConcurrentReadsNoLock(t *testing.T) { - d := newTestDuck(t, 4) - ctx := context.Background() - - if _, err := d.DB.ExecContext(ctx, "CREATE TABLE lake.concurrency_probe(id INTEGER, val VARCHAR)"); err != nil { - t.Fatalf("create table: %v", err) - } - - const workers = 8 - const iters = 25 - var wg sync.WaitGroup - errCh := make(chan error, workers*iters) - start := make(chan struct{}) // released once all goroutines are parked - - // Writers: serialized via the shared write gate, exactly as the ingest path - // holds it around appender flushes. - for w := 0; w < workers/2; w++ { - wg.Add(1) - go func(w int) { - defer wg.Done() - <-start - for i := 0; i < iters; i++ { - err := func() error { - defer d.WriteGate().Lock(writegate.WriteIngestSpans)() - _, err := d.DB.ExecContext(ctx, - "INSERT INTO lake.concurrency_probe VALUES (?, ?)", w*iters+i, "x") - return err - }() - if err != nil { - errCh <- err - return - } - } - }(w) - } - // Readers: concurrent, no write gate — must not collide with the writers. - for r := 0; r < workers/2; r++ { - wg.Add(1) - go func() { - defer wg.Done() - <-start - for i := 0; i < iters; i++ { - var n int - if err := d.DB.QueryRowContext(ctx, - "SELECT count(*) FROM lake.concurrency_probe").Scan(&n); err != nil { - errCh <- err - return - } - } - }() - } - close(start) - wg.Wait() - close(errCh) - - for err := range errCh { - msg := strings.ToLower(err.Error()) - if strings.Contains(msg, "database is locked") || - strings.Contains(msg, "switch temporary directory") { - t.Fatalf("regression — pool>1 catalog/temp error: %v", err) - } - t.Fatalf("unexpected error: %v", err) - } -} - -// TestTempDirectorySetOnce deterministically reproduces the "Cannot switch -// temporary directory" regression. temp_directory is an instance-global setting: -// once any connection has spilled to it, re-running SET temp_directory on a later -// connection's boot hook fails. The test forces a spill on a pinned connection, -// then opens a second pooled connection (which runs the boot hook) and asserts it -// doesn't fail. Reverting the sync.Once in openDuckDB makes this test fail. -func TestTempDirectorySetOnce(t *testing.T) { - ctx := context.Background() - // A small memory limit makes a modest sort spill to the temp directory. - // Threads pinned low to keep the per-thread pinned working set inside the - // tiny cap — with one worker per core (the default) the sort OOMs on - // many-core machines before it can spill. - cfg := config.Config{ - DataDir: t.TempDir(), - RollupInterval: time.Minute, - DuckDBMemory: "64MB", - DuckDBThreads: 2, - DuckDBMaxConns: 4, - } - d, err := NewDuck(ctx, cfg) - if err != nil { - if strings.Contains(err.Error(), "LOAD ducklake") || - strings.Contains(err.Error(), "LOAD sqlite") || - strings.Contains(err.Error(), "ATTACH") { - t.Skipf("DuckLake extensions unavailable: %v", err) - } - t.Fatalf("NewDuck() error = %v", err) - } - defer d.Close() - - // Pin connection 1 and force it to spill to the instance temp directory. - conn1, err := d.DB.Conn(ctx) - if err != nil { - t.Fatalf("conn1: %v", err) - } - defer conn1.Close() - var sink int64 - if err := conn1.QueryRowContext(ctx, - "SELECT count(*) FROM (SELECT i FROM range(20000000) t(i) ORDER BY i DESC)").Scan(&sink); err != nil { - t.Fatalf("forced-spill query: %v", err) - } - - // With conn1 still held, this must open a second physical connection and run - // its boot hook. Pre-fix, that boot re-ran SET temp_directory after the temp - // dir was in use and failed with "Cannot switch temporary directory". - conn2, err := d.DB.Conn(ctx) - if err != nil { - if strings.Contains(strings.ToLower(err.Error()), "switch temporary directory") { - t.Fatalf("regression — temp_directory re-set on 2nd connection: %v", err) - } - t.Fatalf("conn2 (boots a fresh pooled connection): %v", err) - } - defer conn2.Close() - if err := conn2.QueryRowContext(ctx, "SELECT 1").Scan(&sink); err != nil { - t.Fatalf("conn2 query: %v", err) - } -} diff --git a/internal/query/edge_backlog_test.go b/internal/query/edge_backlog_test.go index a4834cba..065747d4 100644 --- a/internal/query/edge_backlog_test.go +++ b/internal/query/edge_backlog_test.go @@ -2,12 +2,14 @@ package query import ( "context" - "strings" + "fmt" "testing" "time" "github.com/labstack/fanout/internal/config" "github.com/labstack/fanout/internal/metrics" + "github.com/labstack/fanout/internal/telemetry" + telemetrystore "github.com/labstack/fanout/internal/telemetry/store" "github.com/prometheus/client_golang/prometheus/testutil" ) @@ -26,9 +28,8 @@ func TestEdgeRollupBacklog(t *testing.T) { } d := &Duck{ - DB: db, - cfg: config.Config{RetentionDays: 30, DuckDBMemory: "1GB"}, - lastMaintenance: time.Now(), + DB: db, + cfg: config.Config{RetentionDays: 30, DuckDBMemory: "1GB"}, } ctx := context.Background() @@ -44,7 +45,7 @@ func TestEdgeRollupBacklog(t *testing.T) { _, err := db.ExecContext(ctx, ` WITH input AS (SELECT CAST(? AS TIMESTAMP) AS base_time) -INSERT INTO lake.spans ( +INSERT INTO telemetry.spans ( namespace, trace_id, span_id, parent_span_id, service, operation, kind, start_time, end_time, start_unix_nano, end_unix_nano, duration_ms, @@ -74,7 +75,7 @@ FROM range(?, ?) t(i), input`, _, err = db.ExecContext(ctx, ` WITH input AS (SELECT CAST(? AS TIMESTAMP) AS base_time) -INSERT INTO lake.spans ( +INSERT INTO telemetry.spans ( namespace, trace_id, span_id, parent_span_id, service, operation, kind, start_time, end_time, start_unix_nano, end_unix_nano, duration_ms, @@ -128,7 +129,7 @@ FROM range(?, ?) t(i), input`, } if err := db.QueryRowContext(ctx, ` SELECT count(DISTINCT date_trunc('minute', start_time)) -FROM lake.spans`).Scan(&spanBuckets); err != nil { +FROM telemetry.spans`).Scan(&spanBuckets); err != nil { t.Fatalf("count distinct span minute buckets: %v", err) } if spanBuckets != 180 { @@ -144,22 +145,25 @@ FROM lake.spans`).Scan(&spanBuckets); err != nil { func TestSkipRollupToLatest(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - d, err := NewDuck(ctx, config.Config{DataDir: t.TempDir(), DuckDBMemory: "2GB", RetentionDays: 30}) + cfg := config.Config{DataDir: t.TempDir(), DuckDBMemory: "2GB", RetentionDays: 30} + repository, err := telemetrystore.Open(cfg.TelemetryDir()) + if err != nil { + t.Fatalf("open telemetry repository: %v", err) + } + defer repository.Close() + now := time.Now().UnixNano() + spans := make([]telemetry.Span, 100) + for i := range spans { + spans[i] = telemetry.Span{Namespace: "default", TraceID: "backlog", SpanID: string(rune(i + 1)), ServiceName: "svc", Kind: "SPAN_KIND_CLIENT", StartUnixNanos: now - int64(i)*int64(time.Minute), DurationMS: 10, StatusCode: "STATUS_CODE_OK", IngestedAt: now} + } + if err := repository.Commit(context.Background(), telemetrystore.Batch{ID: "skip-backlog", Spans: spans}); err != nil { + t.Fatalf("commit backlog: %v", err) + } + d, err := NewDuck(ctx, cfg, repository) if err != nil { - if strings.Contains(err.Error(), "ducklake") || strings.Contains(err.Error(), "ATTACH") { - t.Skipf("DuckLake unavailable: %v", err) - } t.Fatalf("NewDuck: %v", err) } defer d.Close() - - if _, err := d.DB.ExecContext(ctx, ` -INSERT INTO lake.spans (namespace, trace_id, span_id, parent_span_id, service, kind, start_time, duration_ms, status, ingested_unix_nano) -SELECT 'default', 'tr-'||i, 'c-'||i, 'p-'||i, 'svc-'||(i%5), 'SPAN_KIND_CLIENT', - now() - ((i % 120) * INTERVAL 1 MINUTE), 10.0, 'STATUS_CODE_OK', epoch_ns(now()) -FROM range(5000) t(i)`); err != nil { - t.Fatalf("insert: %v", err) - } if _, err := d.DB.ExecContext(ctx, ` INSERT INTO endpoint_rollup ( namespace, bucket, service, method, path, calls, error_count, duration_count, duration_buckets @@ -227,12 +231,9 @@ ON CONFLICT (cache_key) DO UPDATE SET last_ingested_unix_nano = 1, updated_at = // Insert one fresh live span (ingested_unix_nano = now) and verify that // rollupOnce picks it up — proves the watermark didn't over-advance and // swallow data that arrived after the skip. - if _, err := d.DB.ExecContext(ctx, ` -INSERT INTO lake.spans (namespace, trace_id, span_id, parent_span_id, service, kind, - start_time, duration_ms, status, ingested_unix_nano) -VALUES ('default', 'tr-live-1', 'sp-live-1', '', 'svc-live', 'SPAN_KIND_SERVER', - now(), 5.0, 'STATUS_CODE_OK', epoch_ns(now()))`); err != nil { - t.Fatalf("insert live span: %v", err) + liveTime := time.Now().Add(time.Millisecond).UnixNano() + if err := repository.Commit(context.Background(), telemetrystore.Batch{ID: "skip-live", Spans: []telemetry.Span{{Namespace: "default", TraceID: "tr-live-1", SpanID: "sp-live-1", ServiceName: "svc-live", Kind: "SPAN_KIND_SERVER", StartUnixNanos: liveTime, DurationMS: 5, StatusCode: "STATUS_CODE_OK", IngestedAt: liveTime}}}); err != nil { + t.Fatalf("commit live span: %v", err) } n2, err := d.rollupOnce(ctx) @@ -243,3 +244,182 @@ VALUES ('default', 'tr-live-1', 'sp-live-1', '', 'svc-live', 'SPAN_KIND_SERVER', t.Error("rollupOnce processed 0 rows for live span after skip-to-latest; watermark may have over-advanced") } } + +// readEdgeRollupState returns the edge rollup's persisted watermark and its +// sub-window resume point. +func readEdgeRollupState(t *testing.T, d *Duck) (watermark, cursor int64) { + t.Helper() + ctx := context.Background() + tx, err := d.DB.BeginTx(ctx, nil) + if err != nil { + t.Fatal(err) + } + defer func() { _ = tx.Rollback() }() + if watermark, err = rollupWatermark(ctx, tx, edgeRollupStateKey); err != nil { + t.Fatal(err) + } + if cursor, err = rollupWatermark(ctx, tx, edgeRollupSubCursorKey); err != nil { + t.Fatal(err) + } + return watermark, cursor +} + +// TestEdgeRollupBoundsSubWindowsPerPass pins the invariant that one rollup pass +// holds the Parquet read gate for an amount of work fixed by configuration, +// never by the shape of the data. +// +// Chunking ingested time does not provide that bound: a seed load or backfill +// compresses a wide start_time range into a narrow ingested window, so a single +// one-hour chunk can expand into an unbounded number of sub-windows and hold +// the gate for as long as that takes — starving every retention and compaction +// publication behind it. No timeout can fix that from the outside; cancelling +// mid-transaction only discards the work and retries it forever. The pass must +// instead stop at a fixed sub-window count, persist where to resume, and leave +// the ingested watermark behind until the window is finished. +func TestEdgeRollupBoundsSubWindowsPerPass(t *testing.T) { + db := openTestDuck(t) + if err := CreateTables(db); err != nil { + t.Fatalf("CreateTables: %v", err) + } + if err := CreateViews(db); err != nil { + t.Fatalf("CreateViews: %v", err) + } + d := &Duck{DB: db, cfg: config.Config{RetentionDays: 30, DuckDBMemory: "1GB"}} + ctx := context.Background() + + // One ingested instant, but start_time spread over 600 minutes: 20 + // sub-windows of edgeStartChunkNanos, well past maxEdgeSubWindowsPerPass. + const ( + rows = 12_000 + spreadMins = 600 + wantWindows = spreadMins / 30 + ) + baseTime := time.Now().UTC().Truncate(time.Minute) + ingestedNano := baseTime.UnixNano() + for _, spec := range []struct{ kind, span, parent, service string }{ + {"SPAN_KIND_SERVER", "parent-%d", "", "svc-a"}, + {"SPAN_KIND_CLIENT", "child-%d", "parent-%d", "svc-b"}, + } { + parent := "NULL" + if spec.parent != "" { + parent = fmt.Sprintf("printf('%s', i)", spec.parent) + } + _, err := db.ExecContext(ctx, fmt.Sprintf(` +WITH input AS (SELECT CAST(? AS TIMESTAMP) AS base_time) +INSERT INTO telemetry.spans ( + namespace, trace_id, span_id, parent_span_id, service, operation, kind, + start_time, end_time, start_unix_nano, end_unix_nano, duration_ms, + status, ingested_at, ingested_unix_nano +) +SELECT 'default', printf('trace-%%d', i %% 2000), printf('%s', i), %s, '%s', 'op', '%s', + base_time - ((i %% %d) * INTERVAL '1' MINUTE), + base_time - ((i %% %d) * INTERVAL '1' MINUTE) + INTERVAL '10' MILLISECOND, + epoch_ns(base_time - ((i %% %d) * INTERVAL '1' MINUTE)), + epoch_ns(base_time - ((i %% %d) * INTERVAL '1' MINUTE)) + 10000000, + 10.0, 'STATUS_CODE_OK', base_time, ? +FROM range(?, ?) t(i), input`, + spec.span, parent, spec.service, spec.kind, spreadMins, spreadMins, spreadMins, spreadMins), + baseTime, ingestedNano, 0, rows) + if err != nil { + t.Fatalf("insert %s: %v", spec.kind, err) + } + } + + // The first pass must stop short and record where to resume, leaving the + // ingested watermark untouched so nothing in the window can be skipped. + if _, err := d.refreshEdgeRollup(ctx); err != nil { + t.Fatalf("first pass: %v", err) + } + watermark, cursor := readEdgeRollupState(t, d) + if cursor == 0 { + t.Fatal("first pass consumed the whole window: the per-pass sub-window bound did not engage") + } + if watermark != 0 { + t.Fatalf("ingested watermark advanced to %d over an unfinished window", watermark) + } + + passes := 1 + for { + if passes > wantWindows+2 { + t.Fatal("edge rollup did not converge: passes are not making forward progress") + } + if _, err := d.refreshEdgeRollup(ctx); err != nil { + t.Fatalf("pass %d: %v", passes+1, err) + } + passes++ + if _, cursor = readEdgeRollupState(t, d); cursor == 0 { + break + } + } + if passes < 2 { + t.Fatalf("converged in %d passes, want the work spread over several", passes) + } + if watermark, _ = readEdgeRollupState(t, d); watermark == 0 { + t.Fatal("ingested watermark never advanced after the window completed") + } + + // Resuming must neither drop nor double-count a bucket: the aggregate has + // to match what a single unbounded pass would have produced. + var edgeBuckets, spanBuckets int64 + if err := db.QueryRowContext(ctx, `SELECT count(DISTINCT bucket) FROM edge_rollup`).Scan(&edgeBuckets); err != nil { + t.Fatal(err) + } + if err := db.QueryRowContext(ctx, ` +SELECT count(DISTINCT date_trunc('minute', start_time)) FROM telemetry.spans`).Scan(&spanBuckets); err != nil { + t.Fatal(err) + } + if spanBuckets != spreadMins { + t.Fatalf("spans cover %d minute buckets, want %d", spanBuckets, spreadMins) + } + if edgeBuckets != spanBuckets { + t.Fatalf("edge_rollup covers %d buckets, spans cover %d — resuming dropped or duplicated work", edgeBuckets, spanBuckets) + } + t.Logf("converged in %d passes over %d sub-windows; buckets edge=%d spans=%d", passes, wantWindows, edgeBuckets, spanBuckets) +} + +func TestEdgeSubWindowShrinksForDenseIngest(t *testing.T) { + db := openTestDuck(t) + if err := CreateTables(db); err != nil { + t.Fatal(err) + } + if err := CreateViews(db); err != nil { + t.Fatal(err) + } + ctx := context.Background() + base := time.Now().UTC().Truncate(time.Minute) + ingested := base.UnixNano() + if _, err := db.ExecContext(ctx, ` +WITH input AS (SELECT CAST(? AS TIMESTAMP) AS base_time) +INSERT INTO telemetry.spans ( + namespace, trace_id, span_id, service, start_time, start_unix_nano, + ingested_at, ingested_unix_nano +) +SELECT + 'default', printf('trace-%d', i), printf('span-%d', i), 'svc', + base_time + ((i % 30) * INTERVAL '1' MINUTE), + epoch_ns(base_time + ((i % 30) * INTERVAL '1' MINUTE)), base_time, ? +FROM range(100) t(i), input`, base, ingested); err != nil { + t.Fatal(err) + } + var affected int64 + if err := db.QueryRowContext(ctx, ` +SELECT COUNT(*) FROM spans +WHERE ingested_unix_nano > ? AND ingested_unix_nano <= ?`, ingested-1, ingested).Scan(&affected); err != nil { + t.Fatal(err) + } + if affected != 100 { + t.Fatalf("affected fixture rows = %d, want 100", affected) + } + tx, err := db.BeginTx(ctx, nil) + if err != nil { + t.Fatal(err) + } + defer func() { _ = tx.Rollback() }() + subHi, err := edgeSubWindowEnd(ctx, tx, ingested-1, ingested, base, base.Add(29*time.Minute), 10) + if err != nil { + t.Fatal(err) + } + if want := base.Add(time.Minute); !subHi.Equal(want) { + t.Fatalf("dense sub-window ended at %s, want %s", subHi, want) + } +} diff --git a/internal/query/ensure_limit_test.go b/internal/query/ensure_limit_test.go index f71c8b5d..9975e292 100644 --- a/internal/query/ensure_limit_test.go +++ b/internal/query/ensure_limit_test.go @@ -3,9 +3,6 @@ package query import ( "context" "testing" - "time" - - "github.com/labstack/fanout/internal/config" ) func TestEnsureLimit_WrapsWithoutClobberingInnerLimits(t *testing.T) { @@ -82,21 +79,3 @@ func TestExecuteSQL_CapsRowsAtMaxRows(t *testing.T) { t.Errorf("RowsReturned = %d, want 5 (capped)", resp.RowsReturned) } } - -func TestRollupLagFromConfig(t *testing.T) { - sec := int64(1_000_000_000) - cases := []struct { - flushSeconds int - wantNanos int64 - }{ - {15, 30 * sec}, // 2×15s = 30s - {20, 40 * sec}, // 2×20s = 40s - {5, 30 * sec}, // 2×5s = 10s, floored to 30s - {0, 30 * sec}, // floored to 30s - } - for _, c := range cases { - if got := rollupLagFromConfig(config.Config{FlushInterval: time.Duration(c.flushSeconds) * time.Second}); got != c.wantNanos { - t.Errorf("rollupLagFromConfig(FlushInterval=%ds) = %d, want %d", c.flushSeconds, got, c.wantNanos) - } - } -} diff --git a/internal/query/hourprune_experiment_test.go b/internal/query/hourprune_experiment_test.go deleted file mode 100644 index be93e7e9..00000000 --- a/internal/query/hourprune_experiment_test.go +++ /dev/null @@ -1,114 +0,0 @@ -package query - -import ( - "context" - "regexp" - "strconv" - "strings" - "testing" - - "github.com/labstack/fanout/internal/config" -) - -// TestHourPartitionPrunesRecentWindow is the within-day-pruning gate experiment -// (design 2026-06-19). It writes a multi-hour span dataset into the real -// hour-partitioned lake, then proves a recent-window query -// (start_time >= now() - 15min) scans only ~the current hour's rows instead of -// the whole dataset — the failure that made the rollup hit 35s as a UTC day -// filled. Run with: go test ./internal/query/ -run HourPartitionPrunes -v -func TestHourPartitionPrunesRecentWindow(t *testing.T) { - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - - d, err := NewDuck(ctx, config.Config{DataDir: t.TempDir(), DuckDBMemory: "512MB"}) - if err != nil { - if strings.Contains(err.Error(), "ducklake") || strings.Contains(err.Error(), "ATTACH") { - t.Skipf("DuckLake unavailable: %v", err) - } - t.Fatalf("NewDuck: %v", err) - } - defer d.Close() - - // Three hours of data, 200k rows each (600k total), each hour in its own - // hour-partition by start_time. ingested_unix_nano is "now" for all (as if - // just ingested) so it can't help pruning — only start_time partitioning can. - const perHour = 200_000 - for _, agoMin := range []int{150, 90, 5} { // 2.5h ago, 1.5h ago, 5min ago - _, err := d.DB.ExecContext(ctx, ` -INSERT INTO lake.spans (namespace, service, trace_id, span_id, start_time, status, status_message, ingested_unix_nano) -SELECT 'default', 'svc-' || (i % 50), md5(i::VARCHAR), md5((i+1)::VARCHAR), - now() - INTERVAL `+strconv.Itoa(agoMin)+` MINUTE + (i % 60) * INTERVAL 1 SECOND, - CASE WHEN i % 20 = 0 THEN 'STATUS_CODE_ERROR' ELSE 'STATUS_CODE_OK' END, - 'err', epoch_ns(now()) -FROM range(`+strconv.Itoa(perHour)+`) t(i)`) - if err != nil { - t.Fatalf("insert hour -%dmin: %v", agoMin, err) - } - } - if _, err := d.DB.ExecContext(ctx, "CALL ducklake_merge_adjacent_files('lake')"); err != nil { - t.Fatalf("merge: %v", err) - } - - var total int64 - if err := d.DB.QueryRowContext(ctx, "SELECT count(*) FROM spans").Scan(&total); err != nil { - t.Fatalf("count total: %v", err) - } - - var fileCount int64 - _ = d.DB.QueryRowContext(ctx, - `SELECT file_count FROM ducklake_table_info('lake') WHERE table_name = 'spans'`).Scan(&fileCount) - - // EXPLAIN ANALYZE the recent-window query (same shape as overviewRecentErrors). - plan := explainAnalyze(t, ctx, d, ` -SELECT service, count(*) FROM spans -WHERE start_time >= now() - INTERVAL 15 MINUTE - AND status = 'STATUS_CODE_ERROR' -GROUP BY service`) - scanned := maxScanCardinality(plan) - - t.Logf("── within-day pruning experiment (hour-partitioned) ──") - t.Logf("total rows : %d (across 3 hour-partitions)", total) - t.Logf("parquet files (merged): %d", fileCount) - t.Logf("recent-15min query scanned: %d rows", scanned) - t.Logf("→ pruned to %.1f%% of the dataset (lower is better; ~1/3 = one hour)", 100*float64(scanned)/float64(total)) - - // The recent-window query must scan well under half the dataset — i.e. it - // pruned to ~the current hour, not all three. (No pruning would scan ~total.) - if scanned >= total/2 { - t.Fatalf("recent-window query scanned %d of %d rows — within-day pruning is NOT working", scanned, total) - } -} - -func explainAnalyze(t *testing.T, ctx context.Context, d *Duck, q string) string { - t.Helper() - rows, err := d.DB.QueryContext(ctx, "EXPLAIN ANALYZE "+q) - if err != nil { - t.Fatalf("explain analyze: %v", err) - } - defer rows.Close() - var sb strings.Builder - for rows.Next() { - var k, v string - if err := rows.Scan(&k, &v); err != nil { - t.Fatalf("scan plan: %v", err) - } - sb.WriteString(v) - sb.WriteString("\n") - } - return sb.String() -} - -// maxScanCardinality pulls the largest actual-rows count from an EXPLAIN ANALYZE -// plan — for a single-table scan+aggregate that is the rows the scan produced. -var cardRe = regexp.MustCompile(`(?i)(\d[\d,]*)\s*Rows`) - -func maxScanCardinality(plan string) int64 { - var max int64 - for _, m := range cardRe.FindAllStringSubmatch(plan, -1) { - n, err := strconv.ParseInt(strings.ReplaceAll(m[1], ",", ""), 10, 64) - if err == nil && n > max { - max = n - } - } - return max -} diff --git a/internal/query/parquet_gate.go b/internal/query/parquet_gate.go new file mode 100644 index 00000000..9945de1a --- /dev/null +++ b/internal/query/parquet_gate.go @@ -0,0 +1,175 @@ +package query + +import ( + "context" + "errors" + "sync" + "time" +) + +// defaultWriterGrace bounds how long readers may keep entering ahead of a +// waiting publisher. Long enough that ordinary dashboard traffic never queues +// behind maintenance, short enough that retention and compaction always run. +const defaultWriterGrace = 30 * time.Second + +// rollupReaderLease bounds only how long rebuildable rollup work waits to enter +// the Parquet snapshot. Once admitted, the bounded rollup chunk runs under its +// caller context so a slow but healthy pass can commit instead of retrying the +// same chunk forever. +var rollupReaderLease = defaultWriterGrace / 2 + +// ErrParquetReadWait distinguishes publication contention from a query error. +var ErrParquetReadWait = errors.New("wait for Parquet publication") + +// parquetReadGate protects Parquet file publication without sync.RWMutex's +// unconditional writer preference. Queuing maintenance must not stall +// unrelated API reads or readiness probes, so readers continue to be admitted +// while a publisher waits — but only until that publisher's grace period +// expires, after which new readers queue so retention and compaction cannot be +// starved by overlapping query traffic. +type parquetReadGate struct { + once sync.Once + mu sync.Mutex + changed chan struct{} + readers int + writer bool + waiting []parquetWaiter + nextWaiter uint64 + writerGrace time.Duration + now func() time.Time +} + +type parquetWaiter struct { + id uint64 + queuedAt time.Time +} + +func (g *parquetReadGate) init() { + g.once.Do(func() { g.changed = make(chan struct{}) }) +} + +func (g *parquetReadGate) notifyLocked() { + close(g.changed) + g.changed = make(chan struct{}) +} + +func (g *parquetReadGate) clock() time.Time { + if g.now != nil { + return g.now() + } + return time.Now() +} + +func (g *parquetReadGate) grace() time.Duration { + if g.writerGrace != 0 { + return g.writerGrace + } + return defaultWriterGrace +} + +// admitsReaderLocked reports whether a new reader may enter. A reader is +// refused once a publisher is active, or once the longest-waiting publisher +// has been queued for longer than the grace period. +func (g *parquetReadGate) admitsReaderLocked() bool { + if g.writer { + return false + } + if len(g.waiting) == 0 { + return true + } + return g.clock().Sub(g.waiting[0].queuedAt) < g.grace() +} + +func (g *parquetReadGate) TryRLock() bool { + g.init() + g.mu.Lock() + defer g.mu.Unlock() + if !g.admitsReaderLocked() { + return false + } + g.readers++ + return true +} + +func (g *parquetReadGate) RLockContext(ctx context.Context) error { + g.init() + g.mu.Lock() + for !g.admitsReaderLocked() { + changed := g.changed + g.mu.Unlock() + select { + case <-ctx.Done(): + return ctx.Err() + case <-changed: + } + g.mu.Lock() + } + g.readers++ + g.mu.Unlock() + return nil +} + +func (g *parquetReadGate) RUnlock() { + g.init() + g.mu.Lock() + g.readers-- + if g.readers < 0 { + g.mu.Unlock() + panic("query: parquetReadGate RUnlock without RLock") + } + if g.readers == 0 { + g.notifyLocked() + } + g.mu.Unlock() +} + +func (g *parquetReadGate) LockContext(ctx context.Context) error { + g.init() + g.mu.Lock() + g.nextWaiter++ + waiter := parquetWaiter{id: g.nextWaiter, queuedAt: g.clock()} + g.waiting = append(g.waiting, waiter) + // Wake any readers parked on an earlier publisher so they re-evaluate this + // publisher's grace, and so the grace clock starts for readers immediately. + g.notifyLocked() + for g.writer || g.readers > 0 { + changed := g.changed + g.mu.Unlock() + select { + case <-ctx.Done(): + g.mu.Lock() + for i, queued := range g.waiting { + if queued.id == waiter.id { + g.waiting = append(g.waiting[:i], g.waiting[i+1:]...) + break + } + } + g.notifyLocked() + g.mu.Unlock() + return ctx.Err() + case <-changed: + } + g.mu.Lock() + } + for i, queued := range g.waiting { + if queued.id == waiter.id { + g.waiting = append(g.waiting[:i], g.waiting[i+1:]...) + break + } + } + g.writer = true + g.mu.Unlock() + return nil +} + +func (g *parquetReadGate) Unlock() { + g.init() + g.mu.Lock() + if !g.writer { + g.mu.Unlock() + panic("query: parquetReadGate Unlock without Lock") + } + g.writer = false + g.notifyLocked() + g.mu.Unlock() +} diff --git a/internal/query/parquet_gate_test.go b/internal/query/parquet_gate_test.go new file mode 100644 index 00000000..db1c260f --- /dev/null +++ b/internal/query/parquet_gate_test.go @@ -0,0 +1,186 @@ +package query + +import ( + "context" + "errors" + "sync" + "testing" + "time" +) + +// fakeClock lets the gate's grace period be crossed deterministically. +type fakeClock struct { + mu sync.Mutex + now time.Time +} + +func waitingParquetWriters(g *parquetReadGate) int { + g.init() + g.mu.Lock() + defer g.mu.Unlock() + return len(g.waiting) +} + +func TestParquetGateRemovesCanceledPublisher(t *testing.T) { + var gate parquetReadGate + mustRLock(t, &gate) + ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond) + defer cancel() + if err := gate.LockContext(ctx); !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("LockContext error = %v, want deadline exceeded", err) + } + if got := waitingParquetWriters(&gate); got != 0 { + t.Fatalf("canceled publisher remained queued: %d", got) + } + gate.RUnlock() + if !gate.TryRLock() { + t.Fatal("canceled publisher continued blocking readers") + } + gate.RUnlock() +} + +func (c *fakeClock) Now() time.Time { + c.mu.Lock() + defer c.mu.Unlock() + return c.now +} + +func (c *fakeClock) Advance(d time.Duration) { + c.mu.Lock() + defer c.mu.Unlock() + c.now = c.now.Add(d) +} + +func waitForQueuedWriter(t *testing.T, gate *parquetReadGate) { + t.Helper() + deadline := time.Now().Add(time.Second) + for waitingParquetWriters(gate) == 0 { + if time.Now().After(deadline) { + t.Fatal("publisher never queued") + } + time.Sleep(time.Millisecond) + } +} + +func waitForQueuedWriters(t *testing.T, gate *parquetReadGate, count int) { + t.Helper() + deadline := time.Now().Add(time.Second) + for waitingParquetWriters(gate) < count { + if time.Now().After(deadline) { + t.Fatalf("publishers queued = %d, want %d", waitingParquetWriters(gate), count) + } + time.Sleep(time.Millisecond) + } +} + +func TestParquetGateAdmitsReadersWhileWriterGraceRuns(t *testing.T) { + clock := &fakeClock{now: time.Date(2026, 8, 26, 12, 0, 0, 0, time.UTC)} + gate := &parquetReadGate{now: clock.Now} + if !gate.TryRLock() { + t.Fatal("first reader was not admitted") + } + go mustLock(gate) + waitForQueuedWriter(t, gate) + clock.Advance(defaultWriterGrace / 2) + if !gate.TryRLock() { + t.Fatal("reader refused while the publisher was still inside its grace period") + } + gate.RUnlock() + gate.RUnlock() +} + +func TestParquetGateQueuesReadersOnceWriterGraceExpires(t *testing.T) { + clock := &fakeClock{now: time.Date(2026, 8, 26, 12, 0, 0, 0, time.UTC)} + gate := &parquetReadGate{now: clock.Now} + if !gate.TryRLock() { + t.Fatal("first reader was not admitted") + } + go mustLock(gate) + waitForQueuedWriter(t, gate) + clock.Advance(defaultWriterGrace + time.Second) + if gate.TryRLock() { + gate.RUnlock() + gate.RUnlock() + t.Fatal("reader admitted past an expired publisher grace; maintenance can starve indefinitely") + } + gate.RUnlock() +} + +func TestParquetGatePublishesAfterOverlappingReadersDrain(t *testing.T) { + clock := &fakeClock{now: time.Date(2026, 8, 26, 12, 0, 0, 0, time.UTC)} + gate := &parquetReadGate{now: clock.Now} + mustRLock(t, gate) + published := make(chan struct{}) + go func() { + mustLock(gate) + close(published) + gate.Unlock() + }() + waitForQueuedWriter(t, gate) + // Overlapping traffic keeps arriving, but only within the grace period. + if !gate.TryRLock() { + t.Fatal("reader refused inside the grace period") + } + clock.Advance(defaultWriterGrace + time.Second) + gate.RUnlock() + gate.RUnlock() + select { + case <-published: + case <-time.After(2 * time.Second): + t.Fatal("publisher never ran after readers drained") + } +} + +func TestParquetGateDistinguishesWritersQueuedAtSameInstant(t *testing.T) { + clock := &fakeClock{now: time.Date(2026, 8, 26, 12, 0, 0, 0, time.UTC)} + gate := &parquetReadGate{now: clock.Now} + mustRLock(t, gate) + acquired := make(chan struct{}, 2) + release := make(chan struct{}, 2) + for range 2 { + go func() { + mustLock(gate) + acquired <- struct{}{} + <-release + gate.Unlock() + }() + } + waitForQueuedWriters(t, gate, 2) + gate.RUnlock() + for range 2 { + select { + case <-acquired: + release <- struct{}{} + case <-time.After(2 * time.Second): + t.Fatal("publisher never acquired gate") + } + } + deadline := time.Now().Add(time.Second) + for waitingParquetWriters(gate) != 0 { + if time.Now().After(deadline) { + t.Fatalf("stale publisher remained queued: %d", waitingParquetWriters(gate)) + } + time.Sleep(time.Millisecond) + } + deadline = time.Now().Add(time.Second) + for !gate.TryRLock() { + if time.Now().After(deadline) { + t.Fatal("reader refused after both publishers completed") + } + time.Sleep(time.Millisecond) + } + gate.RUnlock() +} + +func mustRLock(t *testing.T, gate *parquetReadGate) { + t.Helper() + if err := gate.RLockContext(context.Background()); err != nil { + t.Fatalf("RLockContext: %v", err) + } +} + +func mustLock(gate *parquetReadGate) { + if err := gate.LockContext(context.Background()); err != nil { + panic(err) + } +} diff --git a/internal/query/retry_test.go b/internal/query/retry_test.go deleted file mode 100644 index 0247d0bc..00000000 --- a/internal/query/retry_test.go +++ /dev/null @@ -1,228 +0,0 @@ -package query - -import ( - "context" - "errors" - "regexp" - "testing" - - "github.com/DATA-DOG/go-sqlmock" -) - -// transient is the canonical transient DuckLake IO error string that -// isTransientLakeIOError must classify as retryable. -const transientMsg = "IO Error: Cannot open file: No such file or directory" - -// ---- isTransientLakeIOError ---- - -func TestIsTransientLakeIOError(t *testing.T) { - cases := []struct { - name string - err error - want bool - }{ - { - name: "nil error", - err: nil, - want: false, - }, - { - name: "context.Canceled", - err: context.Canceled, - want: false, - }, - { - name: "generic error", - err: errors.New("syntax error"), - want: false, - }, - { - name: "IO Error only (missing No such file)", - err: errors.New("IO Error: something else"), - want: false, - }, - { - name: "No such file only (missing IO Error)", - err: errors.New("something: No such file or directory"), - want: false, - }, - { - name: "both substrings present", - err: errors.New(transientMsg), - want: true, - }, - } - - for _, tc := range cases { - t.Run(tc.name, func(t *testing.T) { - got := isTransientLakeIOError(tc.err) - if got != tc.want { - t.Errorf("isTransientLakeIOError(%v) = %v, want %v", tc.err, got, tc.want) - } - }) - } -} - -// ---- QueryContext retry tests ---- - -func TestQueryContextRetriesThenSucceeds(t *testing.T) { - db, mock, err := sqlmock.New() - if err != nil { - t.Fatalf("sqlmock.New: %v", err) - } - defer db.Close() - - d := &Duck{DB: db} - ctx := context.Background() - - // First call: transient error → should retry. - mock.ExpectQuery(regexp.QuoteMeta("SELECT 1")). - WillReturnError(errors.New(transientMsg)) - // Second call: success. - mock.ExpectQuery(regexp.QuoteMeta("SELECT 1")). - WillReturnRows(sqlmock.NewRows([]string{"x"}).AddRow(1)) - - rows, err := d.QueryContext(ctx, "SELECT 1") - if err != nil { - t.Fatalf("QueryContext() error = %v, want nil", err) - } - if rows == nil { - t.Fatal("QueryContext() rows = nil, want non-nil") - } - rows.Close() - - if err := mock.ExpectationsWereMet(); err != nil { - t.Fatalf("unmet expectations (retry did not happen): %v", err) - } -} - -func TestQueryContextNonTransientReturnsImmediately(t *testing.T) { - db, mock, err := sqlmock.New() - if err != nil { - t.Fatalf("sqlmock.New: %v", err) - } - defer db.Close() - - d := &Duck{DB: db} - ctx := context.Background() - - // Non-transient error — exactly one call, no retry. - mock.ExpectQuery(regexp.QuoteMeta("SELECT 1")). - WillReturnError(errors.New("syntax error")) - - _, queryErr := d.QueryContext(ctx, "SELECT 1") - if queryErr == nil { - t.Fatal("QueryContext() error = nil, want syntax error") - } - - if err := mock.ExpectationsWereMet(); err != nil { - t.Fatalf("unexpected extra call (retry happened on non-transient error): %v", err) - } -} - -func TestQueryContextExhaustsRetries(t *testing.T) { - db, mock, err := sqlmock.New() - if err != nil { - t.Fatalf("sqlmock.New: %v", err) - } - defer db.Close() - - d := &Duck{DB: db} - ctx := context.Background() - - // All 3 attempts fail with transient error. - for i := 0; i < 3; i++ { - mock.ExpectQuery(regexp.QuoteMeta("SELECT 1")). - WillReturnError(errors.New(transientMsg)) - } - - _, queryErr := d.QueryContext(ctx, "SELECT 1") - if queryErr == nil { - t.Fatal("QueryContext() error = nil after exhausted retries, want transient error") - } - - if err := mock.ExpectationsWereMet(); err != nil { - t.Fatalf("unexpected call count (expected exactly 3 attempts): %v", err) - } -} - -// ---- QueryRowScan retry test ---- - -func TestQueryRowScanRetriesThenSucceeds(t *testing.T) { - db, mock, err := sqlmock.New() - if err != nil { - t.Fatalf("sqlmock.New: %v", err) - } - defer db.Close() - - d := &Duck{DB: db} - ctx := context.Background() - - // First call: transient error (sqlmock surfaces it at Query time, which the - // wrapper classifies and retries — same observable outcome as an error at Scan). - mock.ExpectQuery(regexp.QuoteMeta("SELECT 1")). - WillReturnError(errors.New(transientMsg)) - // Second call: success with one row. - mock.ExpectQuery(regexp.QuoteMeta("SELECT 1")). - WillReturnRows(sqlmock.NewRows([]string{"v"}).AddRow(42)) - - var dst int - if err := d.QueryRowScan(ctx, []any{&dst}, "SELECT 1"); err != nil { - t.Fatalf("QueryRowScan() error = %v, want nil", err) - } - if dst != 42 { - t.Errorf("QueryRowScan() scanned %d, want 42", dst) - } - - if err := mock.ExpectationsWereMet(); err != nil { - t.Fatalf("unmet expectations (retry did not happen): %v", err) - } -} - -// ---- applyMemoryHeadroom tests ---- - -func TestApplyMemoryHeadroomCapsSmallBox(t *testing.T) { - db, mock, err := sqlmock.New() - if err != nil { - t.Fatalf("sqlmock.New: %v", err) - } - defer db.Close() - - ctx := context.Background() - - // 6.4 GiB ≈ 80% of 8 GiB → total≈8 GiB, capped = 8 GiB - 2 GiB = 6 GiB < 6.4 GiB → SET fires. - mock.ExpectQuery(regexp.QuoteMeta("SELECT current_setting('memory_limit')")). - WillReturnRows(sqlmock.NewRows([]string{"v"}).AddRow("6.4 GiB")) - mock.ExpectExec("SET memory_limit="). - WillReturnResult(sqlmock.NewResult(0, 0)) - - if err := applyMemoryHeadroom(ctx, db); err != nil { - t.Fatalf("applyMemoryHeadroom() error = %v, want nil", err) - } - if err := mock.ExpectationsWereMet(); err != nil { - t.Fatalf("SET memory_limit was not issued for small box: %v", err) - } -} - -func TestApplyMemoryHeadroomSkipsBigBox(t *testing.T) { - db, mock, err := sqlmock.New() - if err != nil { - t.Fatalf("sqlmock.New: %v", err) - } - defer db.Close() - - ctx := context.Background() - - // 100 GiB → total≈125 GiB, capped=123 GiB >= 100 GiB limit → no-op (capped >= limit). - mock.ExpectQuery(regexp.QuoteMeta("SELECT current_setting('memory_limit')")). - WillReturnRows(sqlmock.NewRows([]string{"v"}).AddRow("100 GiB")) - // No ExpectExec: if a SET fires, sqlmock reports it as an unexpected call - // and ExpectationsWereMet fails. - - if err := applyMemoryHeadroom(ctx, db); err != nil { - t.Fatalf("applyMemoryHeadroom() error = %v, want nil", err) - } - if err := mock.ExpectationsWereMet(); err != nil { - t.Fatalf("SET memory_limit was unexpectedly issued for big box: %v", err) - } -} diff --git a/internal/query/rollup_test.go b/internal/query/rollup_test.go index 7f4013a6..4c4b3e51 100644 --- a/internal/query/rollup_test.go +++ b/internal/query/rollup_test.go @@ -19,9 +19,8 @@ func TestRollupOnceRebuildsAffectedServiceBuckets(t *testing.T) { } d := &Duck{ - DB: db, - cfg: config.Config{RetentionDays: 30}, - lastMaintenance: time.Now(), + DB: db, + cfg: config.Config{RetentionDays: 30}, } ctx := context.Background() @@ -90,7 +89,7 @@ func TestRollupOnceRebuildsAffectedEndpointBuckets(t *testing.T) { if err := CreateViews(db); err != nil { t.Fatalf("CreateViews failed: %v", err) } - d := &Duck{DB: db, cfg: config.Config{RetentionDays: 30}, lastMaintenance: time.Now()} + d := &Duck{DB: db, cfg: config.Config{RetentionDays: 30}} ctx := context.Background() bucket := time.Now().UTC().Truncate(time.Minute).Add(-2 * time.Minute) @@ -141,9 +140,8 @@ func TestRollupOnceRebuildsAffectedEdgeBuckets(t *testing.T) { } d := &Duck{ - DB: db, - cfg: config.Config{RetentionDays: 30}, - lastMaintenance: time.Now(), + DB: db, + cfg: config.Config{RetentionDays: 30}, } ctx := context.Background() @@ -231,9 +229,8 @@ func TestRollupOnceIgnoresRowsWithoutBucketTimestamp(t *testing.T) { } d := &Duck{ - DB: db, - cfg: config.Config{RetentionDays: 30}, - lastMaintenance: time.Now(), + DB: db, + cfg: config.Config{RetentionDays: 30}, } ctx := context.Background() @@ -250,7 +247,7 @@ func TestRollupOnceIgnoresRowsWithoutBucketTimestamp(t *testing.T) { }) if _, err := db.Exec(` -INSERT INTO lake.logs ( +INSERT INTO telemetry.logs ( namespace, log_time, time_unix_nano, @@ -262,11 +259,11 @@ INSERT INTO lake.logs ( ingested_unix_nano ) VALUES ('ns-a', NULL, 0, 'INFO', 9, 'missing time', 'checkout', now(), 200)`); err != nil { - t.Fatalf("insert lake.logs failed: %v", err) + t.Fatalf("insert telemetry.logs failed: %v", err) } if _, err := db.Exec(` -INSERT INTO lake.metrics ( +INSERT INTO telemetry.metrics ( namespace, metric_time, time_unix_nano, @@ -278,7 +275,7 @@ INSERT INTO lake.metrics ( ingested_unix_nano ) VALUES ('ns-a', NULL, 0, 'cpu.usage', 'gauge', 'checkout', 1.0, now(), 300)`); err != nil { - t.Fatalf("insert lake.metrics failed: %v", err) + t.Fatalf("insert telemetry.metrics failed: %v", err) } if _, err := d.rollupOnce(ctx); err != nil { @@ -306,9 +303,8 @@ func TestRollupOnceMessagingEdgeCountsConsumedMessages(t *testing.T) { } d := &Duck{ - DB: db, - cfg: config.Config{RetentionDays: 30}, - lastMaintenance: time.Now(), + DB: db, + cfg: config.Config{RetentionDays: 30}, } ctx := context.Background() @@ -378,9 +374,8 @@ func TestRollupOnceDropsCallEdgeParentOutsideWindow(t *testing.T) { } d := &Duck{ - DB: db, - cfg: config.Config{RetentionDays: 30}, - lastMaintenance: time.Now(), + DB: db, + cfg: config.Config{RetentionDays: 30}, } ctx := context.Background() @@ -502,9 +497,8 @@ func TestRollupOnceChunksWideBacklog(t *testing.T) { } d := &Duck{ - DB: db, - cfg: config.Config{RetentionDays: 30}, - lastMaintenance: time.Now(), + DB: db, + cfg: config.Config{RetentionDays: 30}, } ctx := context.Background() @@ -634,7 +628,7 @@ func insertRollupTestSpan(t *testing.T, db *sql.DB, span rollupTestSpan) { end := span.start.Add(span.duration) if _, err := db.Exec(` -INSERT INTO lake.spans ( +INSERT INTO telemetry.spans ( namespace, trace_id, span_id, @@ -674,7 +668,7 @@ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, time.Unix(0, span.ingested).UTC(), span.ingested, ); err != nil { - t.Fatalf("insert lake.spans failed: %v", err) + t.Fatalf("insert telemetry.spans failed: %v", err) } } diff --git a/internal/query/rollup_watermark_test.go b/internal/query/rollup_watermark_test.go index 5fa35808..daf92331 100644 --- a/internal/query/rollup_watermark_test.go +++ b/internal/query/rollup_watermark_test.go @@ -8,6 +8,13 @@ import ( "github.com/labstack/fanout/internal/config" ) +func TestRollupPublicationLagExceedsReaderAndPublisherWindow(t *testing.T) { + minimum := defaultWriterGrace + time.Duration(maxQueryTimeoutMs)*time.Millisecond + if rollupPublicationSafetyLag <= minimum { + t.Fatalf("rollup publication lag = %s, must exceed reader+publisher window %s", rollupPublicationSafetyLag, minimum) + } +} + // TestRollupWatermarkPicksUpLateLowIngestedRow reproduces the silent-data-loss // bug: a row that commits with an ingested timestamp below a watermark already // advanced by another signal would be excluded from the rollup forever. The @@ -23,10 +30,9 @@ func TestRollupWatermarkPicksUpLateLowIngestedRow(t *testing.T) { const lag = 2 * time.Second d := &Duck{ - DB: db, - cfg: config.Config{RetentionDays: 30}, - lastMaintenance: time.Now(), - rollupLagNanos: lag.Nanoseconds(), + DB: db, + cfg: config.Config{RetentionDays: 30}, + rollupLagNanos: lag.Nanoseconds(), } ctx := context.Background() @@ -89,10 +95,9 @@ func TestRollupWatermarkLagSurvivesChunkedCatchUp(t *testing.T) { const lag = 2 * time.Second d := &Duck{ - DB: db, - cfg: config.Config{RetentionDays: 30}, - lastMaintenance: time.Now(), - rollupLagNanos: lag.Nanoseconds(), + DB: db, + cfg: config.Config{RetentionDays: 30}, + rollupLagNanos: lag.Nanoseconds(), } ctx := context.Background() @@ -152,10 +157,9 @@ func TestEdgeRollupWatermarkPicksUpLateChild(t *testing.T) { const lag = 2 * time.Second d := &Duck{ - DB: db, - cfg: config.Config{RetentionDays: 30}, - lastMaintenance: time.Now(), - rollupLagNanos: lag.Nanoseconds(), + DB: db, + cfg: config.Config{RetentionDays: 30}, + rollupLagNanos: lag.Nanoseconds(), } ctx := context.Background() bucket := time.Now().UTC().Truncate(time.Minute).Add(-2 * time.Minute) diff --git a/internal/query/schema.go b/internal/query/schema.go index 51051e7f..308a5576 100644 --- a/internal/query/schema.go +++ b/internal/query/schema.go @@ -10,8 +10,9 @@ func GetSchema(dataDir string) string { const schemaTemplate = ` ## Fanout Data Schema -Fanout stores telemetry in DuckLake tables attached under the lake catalog. -The local metadata catalog, query cache, and product state live under {DATA_DIR}. +Fanout stores telemetry in atomic Parquet batches with persistent trace indexes. +DuckDB exposes the Parquet files through the read-only telemetry schema. The +rebuildable query cache and product state live under {DATA_DIR}. Primary query surfaces: - spans view: clean span columns for most queries @@ -22,7 +23,7 @@ Primary query surfaces: - endpoint_rollup table: minute endpoint counts, errors, and mergeable latency histograms ### 1. Spans -Base table: lake.spans +Parquet relation: telemetry.spans Preferred query surface: spans Important columns: @@ -49,7 +50,7 @@ Common queries: - Root spans only: ... WHERE parent_span_id IS NULL OR parent_span_id = '' ### 2. Logs -Base table: lake.logs +Base table: telemetry.logs Preferred query surface: logs Important columns: @@ -70,7 +71,7 @@ Common queries: - Trace-correlated logs: ... WHERE trace_id = '...' ### 3. Metrics -Base table: lake.metrics +Base table: telemetry.metrics Preferred query surface: metrics Important columns: @@ -112,7 +113,7 @@ endpoint_rollup columns: - duration_buckets (STRUCT): cumulative fixed-boundary latency counters ## Query Guidelines -1. Prefer spans, logs, and metrics over raw lake.* tables. +1. Prefer spans, logs, and metrics over raw telemetry.* tables. 2. Always add a recent time filter for large queries. 3. Filter by namespace when relevant. 4. JSON columns are flat objects keyed by the literal attribute name. Attribute keys diff --git a/internal/query/schema_test.go b/internal/query/schema_test.go index 2911cc46..affe3ab4 100644 --- a/internal/query/schema_test.go +++ b/internal/query/schema_test.go @@ -30,7 +30,7 @@ func TestGetSchema(t *testing.T) { "endpoint_rollup", "trace_id", "service", - "lake.spans", + "telemetry.spans", "json_extract_string", } diff --git a/internal/query/sql.go b/internal/query/sql.go index 127a0969..6776ed32 100644 --- a/internal/query/sql.go +++ b/internal/query/sql.go @@ -30,6 +30,11 @@ type SQLResponse struct { // RowMap represents a single row as a map of column name to value type RowMap map[string]interface{} +const ( + defaultQueryTimeoutMs = 15_000 + maxQueryTimeoutMs = 25_000 +) + // ExecuteSQL validates and executes a SQL query func (d *Duck) ExecuteSQL(ctx context.Context, req SQLRequest) (resp SQLResponse) { start := time.Now() @@ -51,15 +56,14 @@ func (d *Duck) ExecuteSQL(ctx context.Context, req SQLRequest) (resp SQLResponse req.MaxRows = 1000 } - // Set default timeout, then clamp to a hard ceiling. A query that outlives - // the DuckLake snapshot grace window can have its parquet files deleted - // mid-scan by maintenance (see snapshotGraceMinutes in duck.go), so a - // caller-supplied TimeoutMs must never exceed it. Half the grace leaves - // ample margin for the longest legitimate scan. - const maxQueryTimeoutMs = snapshotGraceMinutes * 60 * 1000 / 2 // half the snapshot grace + // Set a default timeout and clamp arbitrary SQL so one request cannot occupy + // a DuckDB worker indefinitely. + // Keep untrusted ad-hoc reads shorter than the Parquet publisher grace period. + // Otherwise one query can make maintenance queue, close the reader gate, and + // turn its remaining runtime into an API-wide read outage. timeoutMs := req.TimeoutMs if timeoutMs <= 0 { - timeoutMs = 30000 + timeoutMs = defaultQueryTimeoutMs } if timeoutMs > maxQueryTimeoutMs { timeoutMs = maxQueryTimeoutMs @@ -86,12 +90,8 @@ func (d *Duck) ExecuteSQL(ctx context.Context, req SQLRequest) (resp SQLResponse queryCtx, cancel := context.WithTimeout(ctx, time.Duration(timeoutMs)*time.Millisecond) defer cancel() - // Execute query. The raw `query` tool deliberately uses DB.QueryContext, not - // the d.QueryContext retry wrapper: arbitrary user SQL may stream large/long - // results, and a blind re-plan on a transient lake IO error is less obviously - // safe than for the fixed internal aggregate reads. This path is instead - // bounded by the timeout clamp above (well under the snapshot grace). - rows, err := d.DB.QueryContext(queryCtx, execQuery) + // Execute the query against immutable Parquet files and the local cache. + rows, err := d.QueryContext(queryCtx, execQuery) if err != nil { return SQLResponse{ Error: fmt.Sprintf("Query execution failed: %v", err), diff --git a/internal/query/sql_test.go b/internal/query/sql_test.go index 6216f08c..4bcdd787 100644 --- a/internal/query/sql_test.go +++ b/internal/query/sql_test.go @@ -131,14 +131,14 @@ func TestSQLResponseQueryPlan(t *testing.T) { } func TestDefaultTimeout(t *testing.T) { - // When TimeoutMs is 0, the effective timeout should be 30000 ms. + // When TimeoutMs is 0, the effective timeout should be 15000 ms. // We test the logic directly via the guard in ExecuteSQL. timeoutMs := 0 if timeoutMs <= 0 { - timeoutMs = 30000 + timeoutMs = defaultQueryTimeoutMs } - if timeoutMs != 30000 { - t.Errorf("default timeout = %d, want 30000", timeoutMs) + if timeoutMs != 15000 { + t.Errorf("default timeout = %d, want 15000", timeoutMs) } // Custom timeout is preserved. diff --git a/internal/query/views.go b/internal/query/views.go index 6ea2e9de..eee6f4ba 100644 --- a/internal/query/views.go +++ b/internal/query/views.go @@ -3,10 +3,11 @@ package query import ( "database/sql" "fmt" + "path/filepath" ) const createSpansTable = ` -CREATE TABLE IF NOT EXISTS lake.spans ( +CREATE TABLE IF NOT EXISTS telemetry.spans ( namespace VARCHAR, trace_id VARCHAR, span_id VARCHAR, @@ -45,7 +46,7 @@ CREATE TABLE IF NOT EXISTS lake.spans ( );` const createLogsTable = ` -CREATE TABLE IF NOT EXISTS lake.logs ( +CREATE TABLE IF NOT EXISTS telemetry.logs ( namespace VARCHAR, log_time TIMESTAMP, observed_time TIMESTAMP, @@ -68,7 +69,7 @@ CREATE TABLE IF NOT EXISTS lake.logs ( );` const createMetricsTable = ` -CREATE TABLE IF NOT EXISTS lake.metrics ( +CREATE TABLE IF NOT EXISTS telemetry.metrics ( namespace VARCHAR, metric_time TIMESTAMP, time_unix_nano BIGINT, @@ -195,7 +196,7 @@ SELECT deployment_env, exception_type, exception_message -FROM lake.spans;` +FROM telemetry.spans;` const viewLogs = ` CREATE OR REPLACE VIEW logs AS @@ -219,7 +220,7 @@ SELECT ingested_at, ingested_unix_nano, body_template -FROM lake.logs;` +FROM telemetry.logs;` const viewMetrics = ` CREATE OR REPLACE VIEW metrics AS @@ -244,21 +245,30 @@ SELECT scope_version, ingested_at, ingested_unix_nano -FROM lake.metrics;` +FROM telemetry.metrics;` const macroAttr = ` CREATE OR REPLACE MACRO attr(json_col, key) AS json_extract_string(json_col, '$."' || key || '"');` +// CreateTables creates mutable telemetry tables for query-kernel tests and +// benchmarks. Production startup never calls this function: telemetry is +// exposed exclusively through CreateParquetViews. func CreateTables(db *sql.DB) error { + if _, err := db.Exec(`CREATE SCHEMA IF NOT EXISTS telemetry`); err != nil { + return fmt.Errorf("create telemetry schema: %w", err) + } for _, stmt := range []string{createSpansTable, createLogsTable, createMetricsTable} { if _, err := db.Exec(stmt); err != nil { return fmt.Errorf("create table: %w", err) } } - if err := configureDuckLake(db); err != nil { - return err - } + return CreateCacheTables(db) +} + +// CreateCacheTables creates only DuckDB's rebuildable query accelerators. The +// production telemetry rows themselves live in immutable Parquet batches. +func CreateCacheTables(db *sql.DB) error { if err := ensureCacheTable(db, "service_rollup", createServiceRollupTable, "namespace", "bucket", "service", "spans", "p50_ms", "p95_ms", "error_rate", "log_count", "metric_count"); err != nil { return err @@ -278,45 +288,31 @@ func CreateTables(db *sql.DB) error { return nil } -// CreateViews creates the clean-name views over DuckLake tables plus the attr() macro. -func CreateViews(db *sql.DB) error { - for _, stmt := range []string{macroAttr, viewSpans, viewLogs, viewMetrics} { +// CreateParquetViews exposes the repository's open Parquet files under the +// canonical telemetry schema used by Fanout's SQL kernel. +func CreateParquetViews(db *sql.DB, parquetDir string) error { + if _, err := db.Exec(`CREATE SCHEMA IF NOT EXISTS telemetry`); err != nil { + return err + } + for _, signal := range []string{"spans", "logs", "metrics"} { + pattern := filepath.ToSlash(filepath.Join(parquetDir, "batches", "*.batch", signal+".parquet")) + projection := "*" + if signal == "spans" { + projection = "* EXCLUDE (_trace_hash)" + } + stmt := fmt.Sprintf(`CREATE OR REPLACE VIEW telemetry.%s AS SELECT %s FROM read_parquet(%s, union_by_name=true)`, signal, projection, sqlLiteral(pattern)) if _, err := db.Exec(stmt); err != nil { - return fmt.Errorf("create view/macro: %w", err) + return fmt.Errorf("create parquet view telemetry.%s: %w", signal, err) } } return nil } -func configureDuckLake(db *sql.DB) error { - var loaded int - if err := db.QueryRow(` -SELECT count(*) -FROM duckdb_extensions() -WHERE extension_name = 'ducklake' AND loaded`).Scan(&loaded); err != nil { - return fmt.Errorf("check ducklake extension: %w", err) - } - if loaded == 0 { - return nil - } - - stmts := []string{ - `CALL lake.set_option('parquet_compression', 'zstd')`, - `CALL lake.set_option('target_file_size', '256MB')`, - // Partition by HOUR (not day) of the event time so recent-window scans - // (Overview error queries, rollup aggregations) prune by per-file - // start_time zonemaps to ~the current 1-2 hours' files instead of the whole - // day. day-partitioning let merge produce day-spanning files whose zonemaps - // couldn't prune within a day — the rollup hit 35s and query p95 5s as a UTC - // day filled. hour() is DuckLake's date-inclusive, order-preserving hour - // transform (hours since epoch), so it uniquely identifies a calendar hour. - `ALTER TABLE lake.spans SET PARTITIONED BY (namespace, hour(start_time))`, - `ALTER TABLE lake.logs SET PARTITIONED BY (namespace, hour(log_time))`, - `ALTER TABLE lake.metrics SET PARTITIONED BY (namespace, hour(metric_time))`, - } - for _, stmt := range stmts { +// CreateViews creates stable clean-name views plus the attr() macro. +func CreateViews(db *sql.DB) error { + for _, stmt := range []string{macroAttr, viewSpans, viewLogs, viewMetrics} { if _, err := db.Exec(stmt); err != nil { - return fmt.Errorf("configure ducklake: %w", err) + return fmt.Errorf("create view/macro: %w", err) } } return nil diff --git a/internal/query/views_test.go b/internal/query/views_test.go index dbf64e02..454ad55e 100644 --- a/internal/query/views_test.go +++ b/internal/query/views_test.go @@ -14,9 +14,6 @@ func openTestDuck(t *testing.T) *sql.DB { if err != nil { t.Fatalf("open duckdb: %v", err) } - if _, err := db.Exec(`ATTACH ':memory:' AS lake`); err != nil { - t.Fatalf("attach lake catalog: %v", err) - } t.Cleanup(func() { _ = db.Close() }) return db } diff --git a/internal/lake/writegate/write_gate.go b/internal/query/writegate/write_gate.go similarity index 60% rename from internal/lake/writegate/write_gate.go rename to internal/query/writegate/write_gate.go index a7a5628d..dbeebfa9 100644 --- a/internal/lake/writegate/write_gate.go +++ b/internal/query/writegate/write_gate.go @@ -1,36 +1,40 @@ -// Package writegate serializes and measures DuckLake catalog writes shared by -// the query kernel and telemetry writer. +// Package writegate serializes and measures writes to DuckDB's rebuildable +// rollup cache. package writegate import ( + "context" "sync" "time" "github.com/labstack/fanout/internal/metrics" ) -// WriteOperation is a bounded metric label for a DuckLake catalog write. +// WriteOperation is a bounded metric label for a rollup-cache write. // Keep this list exhaustive: arbitrary strings would create an unbounded // Prometheus label surface. type WriteOperation string const ( - WriteIngestSpans WriteOperation = "ingest_spans" - WriteIngestLogs WriteOperation = "ingest_logs" - WriteIngestMetrics WriteOperation = "ingest_metrics" WriteRollupSkip WriteOperation = "rollup_skip_to_latest" WriteRollupService WriteOperation = "rollup_service" WriteRollupEndpoint WriteOperation = "rollup_endpoint" WriteRollupEdge WriteOperation = "rollup_edge" - WriteMerge WriteOperation = "merge" WriteMaintenance WriteOperation = "maintenance" ) -// WriteGate preserves the existing process-wide sync.Mutex acquisition -// semantics while measuring how long each operation waits for and holds the -// catalog write critical section. The zero value is ready for use. +// WriteGate serializes DuckDB writes while measuring how long each operation +// waits for and holds the critical section. The zero value is ready for use. type WriteGate struct { - mu sync.Mutex + once sync.Once + token chan struct{} +} + +func (g *WriteGate) init() { + g.once.Do(func() { + g.token = make(chan struct{}, 1) + g.token <- struct{}{} + }) } // observe is a package-level seam so a test can prove the observation happens @@ -39,19 +43,33 @@ type WriteGate struct { // cannot tell the two orderings apart. var observe = metrics.RecordWriteGate -// Lock acquires the catalog write gate and returns its release function. +// Lock acquires the cache write gate and returns its release function. // Callers must defer the returned function before acquiring a database // connection, transaction, or appender, and must call it exactly once. func (g *WriteGate) Lock(operation WriteOperation) func() { + unlock, err := g.LockContext(context.Background(), operation) + if err != nil { + panic(err) + } + return unlock +} + +// LockContext acquires the cache write gate, or returns when ctx expires. +func (g *WriteGate) LockContext(ctx context.Context, operation WriteOperation) (func(), error) { + g.init() waitStarted := time.Now() - g.mu.Lock() + select { + case <-g.token: + case <-ctx.Done(): + return nil, ctx.Err() + } acquired := time.Now() return func() { hold := time.Since(acquired) // Unlock before observing: a Prometheus histogram takes its own lock, // and holding the process's hottest critical section across that would // be a throughput regression in the code added to detect one. - g.mu.Unlock() + g.token <- struct{}{} observe(string(operation), acquired.Sub(waitStarted).Seconds(), hold.Seconds()) - } + }, nil } diff --git a/internal/lake/writegate/write_gate_test.go b/internal/query/writegate/write_gate_test.go similarity index 81% rename from internal/lake/writegate/write_gate_test.go rename to internal/query/writegate/write_gate_test.go index 88af1957..e0929fef 100644 --- a/internal/lake/writegate/write_gate_test.go +++ b/internal/query/writegate/write_gate_test.go @@ -1,6 +1,11 @@ package writegate +// These tests cover the query-cache gate; telemetry commits use their own +// repository lock and never pass through DuckDB. + import ( + "context" + "errors" "testing" "time" @@ -8,6 +13,18 @@ import ( dto "github.com/prometheus/client_model/go" ) +func TestWriteGateLockContextHonorsCancellation(t *testing.T) { + var gate WriteGate + unlock := gate.Lock(WriteMaintenance) + ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond) + defer cancel() + if _, err := gate.LockContext(ctx, WriteMaintenance); !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("LockContext error = %v, want deadline exceeded", err) + } + unlock() + gate.Lock(WriteMaintenance)() +} + func TestWriteGateSerializesHoldersInAcquisitionOrder(t *testing.T) { t.Parallel() @@ -20,7 +37,7 @@ func TestWriteGateSerializesHoldersInAcquisitionOrder(t *testing.T) { go func() { defer func() { done <- struct{}{} }() - unlock := gate.Lock(WriteMerge) + unlock := gate.Lock(WriteMaintenance) close(firstEntered) <-releaseFirst unlock() @@ -63,13 +80,13 @@ func TestWriteGateReleasesAfterPanic(t *testing.T) { t.Error("expected panic to propagate") } }() - defer gate.Lock(WriteMerge)() + defer gate.Lock(WriteMaintenance)() panic("boom") }() acquired := make(chan struct{}) go func() { - defer gate.Lock(WriteMerge)() + defer gate.Lock(WriteMaintenance)() close(acquired) }() select { @@ -89,14 +106,16 @@ func TestWriteGateObservesOutsideTheCriticalSection(t *testing.T) { var freeDuringObserve bool restore := swapObserver(t, func(string, float64, float64) { - if gate.mu.TryLock() { + select { + case <-gate.token: freeDuringObserve = true - gate.mu.Unlock() + gate.token <- struct{}{} + default: } }) defer restore() - gate.Lock(WriteMerge)() + gate.Lock(WriteMaintenance)() if !freeDuringObserve { t.Fatal("gate was still held while the observation ran — move the Unlock above observe()") @@ -107,16 +126,16 @@ func TestWriteGateObservesOutsideTheCriticalSection(t *testing.T) { // whether ingest is stalling behind rollups, which is the question this // instrumentation exists to answer. func TestWriteGateRecordsBothWaitAndHoldHistograms(t *testing.T) { - beforeWait := histogramCount(t, "fanout_write_gate_wait_seconds", WriteMerge) - beforeHold := histogramCount(t, "fanout_write_gate_hold_seconds", WriteMerge) + beforeWait := histogramCount(t, "fanout_write_gate_wait_seconds", WriteMaintenance) + beforeHold := histogramCount(t, "fanout_write_gate_hold_seconds", WriteMaintenance) var gate WriteGate - gate.Lock(WriteMerge)() + gate.Lock(WriteMaintenance)() - if after := histogramCount(t, "fanout_write_gate_wait_seconds", WriteMerge); after != beforeWait+1 { + if after := histogramCount(t, "fanout_write_gate_wait_seconds", WriteMaintenance); after != beforeWait+1 { t.Errorf("wait sample count = %d, want %d", after, beforeWait+1) } - if after := histogramCount(t, "fanout_write_gate_hold_seconds", WriteMerge); after != beforeHold+1 { + if after := histogramCount(t, "fanout_write_gate_hold_seconds", WriteMaintenance); after != beforeHold+1 { t.Errorf("hold sample count = %d, want %d", after, beforeHold+1) } } diff --git a/internal/queryrows/rows.go b/internal/queryrows/rows.go new file mode 100644 index 00000000..265489c2 --- /dev/null +++ b/internal/queryrows/rows.go @@ -0,0 +1,32 @@ +package queryrows + +import ( + "context" + "database/sql" +) + +// Rows is the database row-stream surface used by query consumers. Keeping it +// as an interface lets storage engines attach resource-lifetime behavior (for +// example, retaining a Parquet snapshot lock until iteration is complete). +type Rows interface { + Close() error + Columns() ([]string, error) + Err() error + Next() bool + Scan(dest ...any) error +} + +type Queryer interface { + QueryContext(ctx context.Context, query string, args ...any) (Rows, error) +} + +// SQLQueryer is implemented by *sql.DB and sqlmock's database handle. +type SQLQueryer interface { + QueryContext(ctx context.Context, query string, args ...any) (*sql.Rows, error) +} + +type SQLAdapter struct{ DB SQLQueryer } + +func (a SQLAdapter) QueryContext(ctx context.Context, query string, args ...any) (Rows, error) { + return a.DB.QueryContext(ctx, query, args...) +} diff --git a/internal/telemetry/parquet.go b/internal/telemetry/parquet.go new file mode 100644 index 00000000..e945e0c8 --- /dev/null +++ b/internal/telemetry/parquet.go @@ -0,0 +1,1354 @@ +package telemetry + +import ( + "container/heap" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "math" + "os" + "path/filepath" + "sort" + "strings" + "sync" + "time" + + "github.com/parquet-go/parquet-go" + "github.com/parquet-go/parquet-go/compress/zstd" + "github.com/zeebo/xxh3" + "golang.org/x/sync/errgroup" +) + +const ( + BatchSuffix = ".batch" + SchemaBatch = "_schema" + BatchSuffix + batchMetadataVersion = 2 + parquetPageSize = 64 << 10 + parquetRowGroupRows = 50_000 + maxTraceQueryResults = 500 +) + +// commitPublishWait caps how long one ingest commit waits for the publish +// gate. It sits above the publisher's own swap budget so it only fires when a +// publication is genuinely stuck rather than merely slow, and it keeps a stuck +// publication from being indistinguishable from a hung ingest path. +const commitPublishWait = 60 * time.Second + +var syncPublishedDirectory = syncDirectory + +type BatchMetadata struct { + Version uint32 `json:"version"` + ID string `json:"id"` + MinIngestedNanos int64 `json:"min_ingested_nanos"` + MaxIngestedNanos int64 `json:"max_ingested_nanos"` + MinSpanStartNanos int64 `json:"min_span_start_nanos"` + MaxSpanStartNanos int64 `json:"max_span_start_nanos"` + Generation uint32 `json:"generation"` + Spans int `json:"spans"` + Logs int `json:"logs"` + Metrics int `json:"metrics"` +} + +type TraceQuery struct { + TraceID string + Namespace string + StartNanos int64 + EndNanos int64 + Limit int +} + +type storedBatch struct { + metadata BatchMetadata + dir string + traces traceIndex +} + +type ParquetStore struct { + dir string + batchesDir string + stagingDir string + mu sync.RWMutex + publishGate chan struct{} + batches map[string]*storedBatch +} + +type ParquetStats struct { + Files int + Bytes int64 +} + +func OpenParquetStore(dir string) (*ParquetStore, error) { + p := &ParquetStore{ + dir: dir, batchesDir: filepath.Join(dir, "batches"), stagingDir: filepath.Join(dir, "staging"), + publishGate: make(chan struct{}, 1), batches: make(map[string]*storedBatch), + } + p.publishGate <- struct{}{} + for _, path := range []string{p.dir, p.batchesDir} { + if err := os.MkdirAll(path, 0o755); err != nil { + return nil, err + } + } + // Staging is never acknowledged or queried. Removing it is the complete + // recovery protocol for writes interrupted before atomic publication. + if err := os.RemoveAll(p.stagingDir); err != nil { + return nil, fmt.Errorf("clear incomplete Parquet batches: %w", err) + } + if err := os.Mkdir(p.stagingDir, 0o755); err != nil { + return nil, err + } + if err := p.ensureSchemaBatch(); err != nil { + return nil, err + } + if err := p.loadBatches(); err != nil { + return nil, err + } + return p, nil +} + +func (p *ParquetStore) Close() error { return nil } +func (p *ParquetStore) Dir() string { return p.dir } +func (p *ParquetStore) BatchesDir() string { return p.batchesDir } + +func (p *ParquetStore) Pattern(signal string) string { + return filepath.ToSlash(filepath.Join(p.batchesDir, "*"+BatchSuffix, signal+".parquet")) +} + +func (p *ParquetStore) BatchPath(id string) string { + return filepath.Join(p.batchesDir, id+BatchSuffix) +} + +func (p *ParquetStore) StagingPath(id string) string { + return filepath.Join(p.stagingDir, id) +} + +// MergeParquet rewrites immutable files into one native Parquet output. Span +// rows retain the index order required by trace.fidx; the other signals keep +// their input order. Re-encoding also consolidates tiny ingest row groups. +func (p *ParquetStore) MergeParquet(ctx context.Context, signal string, inputs []string, output string) error { + switch signal { + case "spans": + return mergeTypedParquet[spanParquetRow](ctx, inputs, output, true) + case "logs": + return mergeTypedParquet[logParquetRow](ctx, inputs, output, false) + case "metrics": + return mergeTypedParquet[metricParquetRow](ctx, inputs, output, false) + default: + return fmt.Errorf("unsupported Parquet signal %q", signal) + } +} + +func (p *ParquetStore) BatchMetadata() []BatchMetadata { + p.mu.RLock() + defer p.mu.RUnlock() + out := make([]BatchMetadata, 0, len(p.batches)) + for _, batch := range p.batches { + out = append(out, batch.metadata) + } + sort.Slice(out, func(i, j int) bool { + if out[i].Generation != out[j].Generation { + return out[i].Generation < out[j].Generation + } + if out[i].MinIngestedNanos != out[j].MinIngestedNanos { + return out[i].MinIngestedNanos < out[j].MinIngestedNanos + } + return out[i].ID < out[j].ID + }) + return out +} + +func (p *ParquetStore) RowCount() uint64 { + p.mu.RLock() + defer p.mu.RUnlock() + var count uint64 + for _, batch := range p.batches { + count += uint64(batch.metadata.Spans + batch.metadata.Logs + batch.metadata.Metrics) + } + return count +} + +// CommitBatch publishes one atomic batch directory. Preparation is allowed to +// finish; only the final publish-gate wait is capped. A stalled publication +// therefore cannot hang ingest, while a large valid encode does not consume +// the gate budget before it starts waiting. +func (p *ParquetStore) CommitBatch(ctx context.Context, metadata BatchMetadata, spans []Span, logs []Log, metrics []Metric) error { + if err := ValidateBatchID(metadata.ID); err != nil { + return err + } + metadata.Version = batchMetadataVersion + metadata.Spans, metadata.Logs, metadata.Metrics = len(spans), len(logs), len(metrics) + final := p.BatchPath(metadata.ID) + if p.hasBatch(metadata.ID) { + return nil + } + if info, err := os.Stat(final); err == nil && info.IsDir() { + // Adopting a directory left by an interrupted commit means validating + // it — Parquet footers plus a full trace-index walk. That is done + // before taking the gate, so re-registering one batch cannot stall + // every other publisher and ingest worker behind it. + adopted, err := loadRegisteredBatch(final) + if err != nil { + return err + } + if err := p.lockCommitPublish(ctx); err != nil { + return err + } + defer p.unlockPublish() + if err := syncDirectory(p.batchesDir); err != nil { + return err + } + p.installBatch(adopted) + return nil + } else if err != nil && !errors.Is(err, os.ErrNotExist) { + return err + } + + stage := filepath.Join(p.stagingDir, metadata.ID) + if err := os.RemoveAll(stage); err != nil { + return err + } + if err := os.Mkdir(stage, 0o755); err != nil { + return err + } + complete := false + defer func() { + if !complete { + _ = os.RemoveAll(stage) + } + }() + + var spanRows []spanParquetRow + if len(spans) > 0 { + spanRows = make([]spanParquetRow, len(spans)) + for i := range spans { + spanRows[i] = makeSpanParquetRow(spans[i]) + spanRows[i].TraceHash = xxh3.HashString(spanRows[i].TraceID) + if spanRows[i].StartUnixNano > 0 && (metadata.MinSpanStartNanos == 0 || spanRows[i].StartUnixNano < metadata.MinSpanStartNanos) { + metadata.MinSpanStartNanos = spanRows[i].StartUnixNano + } + metadata.MaxSpanStartNanos = max(metadata.MaxSpanStartNanos, spanRows[i].StartUnixNano) + } + sort.Slice(spanRows, func(i, j int) bool { + if spanRows[i].TraceHash != spanRows[j].TraceHash { + return spanRows[i].TraceHash < spanRows[j].TraceHash + } + if spanRows[i].StartUnixNano != spanRows[j].StartUnixNano { + return spanRows[i].StartUnixNano < spanRows[j].StartUnixNano + } + return spanRows[i].SpanID < spanRows[j].SpanID + }) + } + logRows := make([]logParquetRow, len(logs)) + if len(logs) > 0 { + for i := range logs { + logRows[i] = makeLogParquetRow(logs[i]) + } + } + metricRows := make([]metricParquetRow, len(metrics)) + if len(metrics) > 0 { + for i := range metrics { + metricRows[i] = makeMetricParquetRow(metrics[i]) + } + } + var writes errgroup.Group + if len(spanRows) > 0 { + writes.Go(func() error { + if err := writeTypedParquet(filepath.Join(stage, "spans.parquet"), spanRows, parquetPageSize, parquet.SortingWriterConfig(spanSortingColumns())); err != nil { + return fmt.Errorf("write span Parquet: %w", err) + } + return nil + }) + writes.Go(func() error { + if err := writeTraceIndex(filepath.Join(stage, "trace.fidx"), spanRows); err != nil { + return fmt.Errorf("write trace index: %w", err) + } + return nil + }) + } + if len(logRows) > 0 { + writes.Go(func() error { + if err := writeTypedParquet(filepath.Join(stage, "logs.parquet"), logRows, parquetPageSize); err != nil { + return fmt.Errorf("write log Parquet: %w", err) + } + return nil + }) + } + if len(metricRows) > 0 { + writes.Go(func() error { + if err := writeTypedParquet(filepath.Join(stage, "metrics.parquet"), metricRows, parquetPageSize); err != nil { + return fmt.Errorf("write metric Parquet: %w", err) + } + return nil + }) + } + writes.Go(func() error { return writeJSONFile(filepath.Join(stage, "metadata.json"), metadata) }) + if err := writes.Wait(); err != nil { + return err + } + if err := syncDirectory(stage); err != nil { + return err + } + prepared, err := loadStoredBatch(stage) + if err != nil { + return fmt.Errorf("validate staged Parquet batch: %w", err) + } + if prepared.metadata.ID != metadata.ID { + return fmt.Errorf("staged Parquet metadata ID %q does not match batch ID %q", prepared.metadata.ID, metadata.ID) + } + prepared.dir = final + if prepared.metadata.Spans > 0 { + prepared.traces.path = filepath.Join(final, "trace.fidx") + } + + if err := p.lockCommitPublish(ctx); err != nil { + return err + } + defer p.unlockPublish() + if p.hasBatch(metadata.ID) { + complete = true + return os.RemoveAll(stage) + } + if err := os.Rename(stage, final); err != nil { + if info, statErr := os.Stat(final); statErr == nil && info.IsDir() { + complete = true + // The staged copy lost the race and is now unreferenced; drop it + // rather than leaving it to accumulate until the next open. + removeErr := os.RemoveAll(stage) + if err := syncDirectory(p.batchesDir); err != nil { + return errors.Join(err, removeErr) + } + return errors.Join(p.registerBatch(final), removeErr) + } + return fmt.Errorf("publish Parquet batch: %w", err) + } + complete = true + if err := syncDirectory(p.batchesDir); err != nil { + return err + } + p.mu.Lock() + p.batches[metadata.ID] = prepared + p.mu.Unlock() + return nil +} + +func (p *ParquetStore) lockPublish(ctx context.Context) error { + select { + case <-ctx.Done(): + return ctx.Err() + case <-p.publishGate: + return nil + } +} + +func (p *ParquetStore) lockCommitPublish(ctx context.Context) error { + waitCtx, cancel := context.WithTimeout(ctx, commitPublishWait) + defer cancel() + return p.lockPublish(waitCtx) +} + +func (p *ParquetStore) unlockPublish() { p.publishGate <- struct{}{} } + +// RestoreRetiredInputs rolls back a compaction whose durable output vanished. +// The complete namespace change is hidden from readers by publish. +func (p *ParquetStore) RestoreRetiredInputs(inputs []string, replacementID string, publish func(func(context.Context) error) error) error { + if err := ValidateBatchID(replacementID); err != nil { + return err + } + for _, id := range inputs { + if err := ValidateBatchID(id); err != nil { + return err + } + } + type restoredInput struct { + id string + active string + retired string + batch *storedBatch + move bool + } + prepared := make([]restoredInput, 0, len(inputs)) + for _, id := range inputs { + active := p.BatchPath(id) + path := active + move := false + if _, err := os.Stat(active); errors.Is(err, os.ErrNotExist) { + path = filepath.Join(p.batchesDir, id+".retired-"+replacementID) + move = true + } else if err != nil { + return err + } + batch, err := loadStoredBatch(path) + if err != nil { + if errors.Is(err, os.ErrNotExist) { + return fmt.Errorf("input %s is missing", id) + } + return fmt.Errorf("load input %s: %w", id, err) + } + batch.dir = active + if batch.metadata.Spans > 0 { + batch.traces.path = filepath.Join(active, "trace.fidx") + } + prepared = append(prepared, restoredInput{id: id, active: active, retired: path, batch: batch, move: move}) + } + return publish(func(ctx context.Context) error { + if err := p.lockPublish(ctx); err != nil { + return err + } + defer p.unlockPublish() + installActive := func() { + p.mu.Lock() + defer p.mu.Unlock() + for _, input := range prepared { + if _, err := os.Stat(input.active); err == nil { + p.batches[input.id] = input.batch + } else { + delete(p.batches, input.id) + } + } + } + + moved := make([]restoredInput, 0, len(prepared)) + for _, input := range prepared { + if !input.move { + continue + } + if err := os.Rename(input.retired, input.active); err != nil { + var rollbackErr error + for i := len(moved) - 1; i >= 0; i-- { + rollbackErr = errors.Join(rollbackErr, os.Rename(moved[i].active, moved[i].retired)) + } + installActive() + return errors.Join(err, rollbackErr, syncDirectory(p.batchesDir)) + } + moved = append(moved, input) + } + var syncErr error + if len(moved) > 0 { + syncErr = syncDirectory(p.batchesDir) + } + installActive() + return syncErr + }) +} + +// Trace reads only ranges selected by the persistent hash index. Scope filters +// and the limit are applied while decoding so a pathological trace cannot grow +// request memory without bound. +func (p *ParquetStore) Trace(ctx context.Context, query TraceQuery) ([]IndexedSpan, error) { + if query.TraceID == "" { + return nil, nil + } + if query.Limit <= 0 { + return nil, errors.New("trace query limit must be positive") + } + if query.Limit > maxTraceQueryResults { + return nil, fmt.Errorf("trace query limit exceeds %d", maxTraceQueryResults) + } + if query.StartNanos >= query.EndNanos { + return nil, errors.New("trace query time range must be positive") + } + hash := xxh3.HashString(query.TraceID) + p.mu.RLock() + batches := make([]*storedBatch, 0, len(p.batches)) + for _, batch := range p.batches { + batches = append(batches, batch) + } + p.mu.RUnlock() + selected := make(indexedSpanHeap, 0, query.Limit) + for _, batch := range batches { + if err := ctx.Err(); err != nil { + return nil, err + } + if batch.metadata.MaxSpanStartNanos > 0 && + (batch.metadata.MaxSpanStartNanos < query.StartNanos || batch.metadata.MinSpanStartNanos >= query.EndNanos) { + continue + } + match, found, err := batch.traces.Lookup(hash) + if err != nil { + return nil, err + } + if !found { + continue + } + if err := readIndexedTrace(ctx, batch, match, query, &selected); err != nil { + return nil, err + } + } + out := []IndexedSpan(selected) + sort.Slice(out, func(i, j int) bool { + return indexedSpanEarlier(out[i], out[j]) + }) + return out, nil +} + +func readIndexedTrace(ctx context.Context, batch *storedBatch, match traceRange, query TraceQuery, selected *indexedSpanHeap) (err error) { + if match.row > uint64(math.MaxInt64) { + return errors.New("trace index row exceeds Parquet reader limit") + } + file, err := os.Open(filepath.Join(batch.dir, "spans.parquet")) + if err != nil { + return err + } + reader := parquet.NewGenericReader[indexedSpanParquetRow](file) + defer func() { err = errors.Join(err, reader.Close(), file.Close()) }() + if err := reader.SeekToRow(int64(match.row)); err != nil { + return err + } + remaining := match.count + buffer := make([]indexedSpanParquetRow, min(uint64(8192), remaining)) + for remaining > 0 { + if err := ctx.Err(); err != nil { + return err + } + want := min(uint64(len(buffer)), remaining) + n, readErr := reader.Read(buffer[:int(want)]) + if readErr != nil && !errors.Is(readErr, io.EOF) { + return readErr + } + if n == 0 { + return io.ErrUnexpectedEOF + } + for i := range n { + row := buffer[i] + if row.StartUnixNano >= query.EndNanos { + return nil + } + if row.TraceID != query.TraceID || row.StartUnixNano < query.StartNanos || + (query.Namespace != "" && row.Namespace != query.Namespace) { + continue + } + selected.Add(row.span(), query.Limit) + } + remaining -= uint64(n) + } + return nil +} + +type indexedSpanHeap []IndexedSpan + +func (h indexedSpanHeap) Len() int { return len(h) } +func (h indexedSpanHeap) Less(i, j int) bool { + return indexedSpanEarlier(h[j], h[i]) +} +func (h indexedSpanHeap) Swap(i, j int) { h[i], h[j] = h[j], h[i] } +func (h *indexedSpanHeap) Push(value any) { + *h = append(*h, value.(IndexedSpan)) +} +func (h *indexedSpanHeap) Pop() any { + old := *h + value := old[len(old)-1] + *h = old[:len(old)-1] + return value +} + +func (h *indexedSpanHeap) Add(span IndexedSpan, limit int) { + if len(*h) < limit { + heap.Push(h, span) + return + } + if indexedSpanEarlier(span, (*h)[0]) { + (*h)[0] = span + heap.Fix(h, 0) + } +} + +func indexedSpanEarlier(left, right IndexedSpan) bool { + if left.StartUnixNanos != right.StartUnixNanos { + return left.StartUnixNanos < right.StartUnixNanos + } + if left.DurationMS != right.DurationMS { + return left.DurationMS > right.DurationMS + } + return left.SpanID < right.SpanID +} + +// PruneBefore hides complete batches while readers are pinned, then deletes +// the retired directories after publication. +func (p *ParquetStore) PruneBefore(cutoff int64, maxBatches int, publish func(func(context.Context) error) error) (int, error) { + if maxBatches <= 0 { + return 0, nil + } + type candidate struct { + id string + max int64 + } + p.mu.RLock() + candidates := make([]candidate, 0, len(p.batches)) + for id, batch := range p.batches { + if batch.metadata.MaxIngestedNanos > 0 && batch.metadata.MaxIngestedNanos < cutoff { + candidates = append(candidates, candidate{id: id, max: batch.metadata.MaxIngestedNanos}) + } + } + p.mu.RUnlock() + sort.Slice(candidates, func(i, j int) bool { + if candidates[i].max != candidates[j].max { + return candidates[i].max < candidates[j].max + } + return candidates[i].id < candidates[j].id + }) + if len(candidates) > maxBatches { + candidates = candidates[:maxBatches] + } + if len(candidates) == 0 { + return 0, nil + } + var retired []string + var pruneErr error + err := publish(func(ctx context.Context) error { + if err := p.lockPublish(ctx); err != nil { + return err + } + defer p.unlockPublish() + // p.mu is taken only to mutate the in-memory set, never across the + // renames and fsync. Holding it for filesystem work made it a second, + // undeadlined serialization point on the ingest path — CommitBatch's + // first action is a hasBatch read of this same mutex — and one + // invisible to the publish gate's accounting. The renames themselves + // need no additional exclusion: readers are already outside the + // snapshot for the length of this callback, and the publish gate + // serializes this against every other publisher. + p.mu.RLock() + planned := make(map[string]*storedBatch, len(candidates)) + for _, candidate := range candidates { + if batch, exists := p.batches[candidate.id]; exists { + planned[candidate.id] = batch + } + } + p.mu.RUnlock() + + retiredIDs := make([]string, 0, len(planned)) + for _, candidate := range candidates { + batch, exists := planned[candidate.id] + if !exists { + continue + } + path := filepath.Join(p.batchesDir, candidate.id+".retired") + if err := os.Rename(batch.dir, path); err != nil { + pruneErr = errors.Join(pruneErr, err) + continue + } + retiredIDs = append(retiredIDs, candidate.id) + retired = append(retired, path) + } + if len(retired) > 0 { + pruneErr = errors.Join(pruneErr, syncDirectory(p.batchesDir)) + } + p.mu.Lock() + for _, id := range retiredIDs { + delete(p.batches, id) + } + p.mu.Unlock() + return nil + }) + pruneErr = errors.Join(pruneErr, err) + for _, path := range retired { + pruneErr = errors.Join(pruneErr, os.RemoveAll(path)) + } + if len(retired) > 0 { + pruneErr = errors.Join(pruneErr, syncDirectory(p.batchesDir)) + } + return len(retired), pruneErr +} + +func (p *ParquetStore) Stats() (map[string]ParquetStats, error) { + p.mu.RLock() + dirs := make([]string, 0, len(p.batches)) + for _, batch := range p.batches { + dirs = append(dirs, batch.dir) + } + p.mu.RUnlock() + stats := map[string]ParquetStats{"spans": {}, "logs": {}, "metrics": {}} + for _, dir := range dirs { + for signal := range stats { + info, err := os.Stat(filepath.Join(dir, signal+".parquet")) + if errors.Is(err, os.ErrNotExist) { + continue + } + if err != nil { + return nil, err + } + value := stats[signal] + value.Files++ + value.Bytes += info.Size() + stats[signal] = value + } + } + return stats, nil +} + +// PrepareReplacement completes metadata and the trace sidecar for Parquet +// files produced by the compactor. +func (p *ParquetStore) PrepareReplacement(dir string, metadata BatchMetadata) error { + metadata.Version = batchMetadataVersion + metadata.MinSpanStartNanos = 0 + metadata.MaxSpanStartNanos = 0 + if metadata.Spans > 0 { + f, err := os.Open(filepath.Join(dir, "spans.parquet")) + if err != nil { + return err + } + reader := parquet.NewGenericReader[traceParquetRow](f) + index, err := newTraceIndexWriter(filepath.Join(dir, "trace.fidx")) + if err != nil { + _ = reader.Close() + _ = f.Close() + return err + } + rows := 0 + buffer := make([]traceParquetRow, min(8192, metadata.Spans)) + for { + n, readErr := reader.Read(buffer) + for i := range n { + if start := buffer[i].StartUnixNano; start > 0 && (metadata.MinSpanStartNanos == 0 || start < metadata.MinSpanStartNanos) { + metadata.MinSpanStartNanos = start + } + metadata.MaxSpanStartNanos = max(metadata.MaxSpanStartNanos, buffer[i].StartUnixNano) + if err := index.Append(buffer[i].TraceHash); err != nil { + index.Abort() + _ = reader.Close() + _ = f.Close() + return err + } + } + rows += n + if errors.Is(readErr, io.EOF) { + break + } + if readErr != nil { + index.Abort() + _ = reader.Close() + _ = f.Close() + return readErr + } + if n == 0 { + index.Abort() + _ = reader.Close() + _ = f.Close() + return io.ErrNoProgress + } + } + if err := errors.Join(reader.Close(), f.Close()); err != nil { + index.Abort() + return err + } + if rows != metadata.Spans { + index.Abort() + return fmt.Errorf("compacted span count: got %d want %d", rows, metadata.Spans) + } + if err := index.Close(); err != nil { + return err + } + } + if err := writeJSONFile(filepath.Join(dir, "metadata.json"), metadata); err != nil { + return err + } + return syncDirectory(dir) +} + +// PublishReplacement validates a prepared compacted batch, atomically swaps it +// for its inputs while readers are pinned, then deletes retired inputs. +func (p *ParquetStore) PublishReplacement(stage string, metadata BatchMetadata, inputs []string, publish func(func(context.Context) error) error) error { + if err := ValidateBatchID(metadata.ID); err != nil { + return err + } + for _, id := range inputs { + if err := ValidateBatchID(id); err != nil { + return err + } + } + final := p.BatchPath(metadata.ID) + source := stage + if _, err := os.Stat(source); errors.Is(err, os.ErrNotExist) { + source = final + } else if err != nil { + return err + } + replacement, err := loadStoredBatch(source) + if err != nil { + return err + } + replacement.dir = final + if replacement.metadata.Spans > 0 { + replacement.traces.path = filepath.Join(final, "trace.fidx") + } + inputBatches := make(map[string]*storedBatch, len(inputs)) + for _, id := range inputs { + p.mu.RLock() + batch, exists := p.batches[id] + p.mu.RUnlock() + if exists { + inputBatches[id] = batch + continue + } + active := p.BatchPath(id) + path := active + if _, statErr := os.Stat(path); errors.Is(statErr, os.ErrNotExist) { + path = filepath.Join(p.batchesDir, id+".retired-"+metadata.ID) + } else if statErr != nil { + return statErr + } + batch, loadErr := loadStoredBatch(path) + if errors.Is(loadErr, os.ErrNotExist) { + continue + } + if loadErr != nil { + return fmt.Errorf("load compaction input %s: %w", id, loadErr) + } + batch.dir = active + if batch.metadata.Spans > 0 { + batch.traces.path = filepath.Join(active, "trace.fidx") + } + inputBatches[id] = batch + } + retired := make([][2]string, 0, len(inputs)) + err = publish(func(ctx context.Context) error { + if err := p.lockPublish(ctx); err != nil { + return err + } + defer p.unlockPublish() + // p.mu covers only the in-memory set, never the renames and fsync + // below. CommitBatch's first action reads this same mutex, so holding + // it across filesystem work would put ingest behind I/O that the + // publish gate cannot see or bound — the reason PruneBefore keeps its + // own critical section down to the map mutation. + installReplacement := func() { + p.mu.Lock() + defer p.mu.Unlock() + for _, id := range inputs { + delete(p.batches, id) + } + p.batches[metadata.ID] = replacement + } + rollback := func() error { + var rollbackErr error + for i := len(retired) - 1; i >= 0; i-- { + rollbackErr = errors.Join(rollbackErr, os.Rename(retired[i][1], retired[i][0])) + } + restore := make(map[string]*storedBatch, len(inputBatches)) + for id, batch := range inputBatches { + if _, err := os.Stat(batch.dir); err == nil { + restore[id] = batch + } + } + p.mu.Lock() + delete(p.batches, metadata.ID) + for id := range inputBatches { + delete(p.batches, id) + } + for id, batch := range restore { + p.batches[id] = batch + } + p.mu.Unlock() + return errors.Join(rollbackErr, syncDirectory(p.batchesDir)) + } + // Recovery may resume with the output already published. Move it back + // to staging before touching inputs, so rollback can only expose the + // complete old set or the complete replacement, never both. + if source == final { + if err := os.MkdirAll(filepath.Dir(stage), 0o755); err != nil { + return err + } + if err := os.Rename(final, stage); err != nil { + return err + } + source = stage + p.mu.Lock() + delete(p.batches, metadata.ID) + p.mu.Unlock() + } + for _, id := range inputs { + active := p.BatchPath(id) + retiredPath := filepath.Join(p.batchesDir, id+".retired-"+metadata.ID) + if _, statErr := os.Stat(active); statErr == nil { + if err := os.Rename(active, retiredPath); err != nil { + return errors.Join(err, rollback()) + } + retired = append(retired, [2]string{active, retiredPath}) + } else if !errors.Is(statErr, os.ErrNotExist) { + return errors.Join(statErr, rollback()) + } else if _, retiredErr := os.Stat(retiredPath); retiredErr == nil { + retired = append(retired, [2]string{active, retiredPath}) + } else if !errors.Is(retiredErr, os.ErrNotExist) { + return errors.Join(retiredErr, rollback()) + } + } + if source != final { + if err := os.Rename(stage, final); err != nil { + return errors.Join(err, rollback()) + } + source = final + } + if err := syncPublishedDirectory(p.batchesDir); err != nil { + // The namespace already contains only the replacement. Keep the + // in-memory view consistent and let marker recovery retry the fsync. + installReplacement() + return err + } + installReplacement() + return nil + }) + if err != nil { + return err + } + var removeErr error + for _, pair := range retired { + removeErr = errors.Join(removeErr, os.RemoveAll(pair[1])) + } + return errors.Join(removeErr, syncDirectory(p.batchesDir)) +} + +func (p *ParquetStore) ensureSchemaBatch() error { + final := filepath.Join(p.batchesDir, SchemaBatch) + if info, err := os.Stat(final); err == nil && info.IsDir() { + return nil + } else if err != nil && !errors.Is(err, os.ErrNotExist) { + return err + } + stage := filepath.Join(p.stagingDir, "_schema") + if err := os.Mkdir(stage, 0o755); err != nil { + return err + } + if err := writeTypedParquet(filepath.Join(stage, "spans.parquet"), []spanParquetRow{}, parquetPageSize, parquet.SortingWriterConfig(spanSortingColumns())); err != nil { + return err + } + if err := writeTypedParquet(filepath.Join(stage, "logs.parquet"), []logParquetRow{}, parquetPageSize); err != nil { + return err + } + if err := writeTypedParquet(filepath.Join(stage, "metrics.parquet"), []metricParquetRow{}, parquetPageSize); err != nil { + return err + } + if err := syncDirectory(stage); err != nil { + return err + } + if err := os.Rename(stage, final); err != nil { + return err + } + return syncDirectory(p.batchesDir) +} + +func (p *ParquetStore) loadBatches() error { + entries, err := os.ReadDir(p.batchesDir) + if err != nil { + return err + } + for _, entry := range entries { + if !entry.IsDir() || entry.Name() == SchemaBatch || !strings.HasSuffix(entry.Name(), BatchSuffix) { + continue + } + if err := p.registerBatch(filepath.Join(p.batchesDir, entry.Name())); err != nil { + return fmt.Errorf("load Parquet batch %s: %w", entry.Name(), err) + } + } + return nil +} + +func (p *ParquetStore) registerBatch(dir string) error { + batch, err := loadRegisteredBatch(dir) + if err != nil { + return err + } + p.installBatch(batch) + return nil +} + +// loadRegisteredBatch validates a published directory and is safe to call +// outside the publish gate: it only reads files that are already immutable. +func loadRegisteredBatch(dir string) (*storedBatch, error) { + batch, err := loadStoredBatch(dir) + if err != nil { + return nil, err + } + if filepath.Base(dir) != batch.metadata.ID+BatchSuffix { + return nil, fmt.Errorf("parquet batch directory %q does not match metadata ID %q", filepath.Base(dir), batch.metadata.ID) + } + return batch, nil +} + +// ValidatePublishedBatch performs an offline deep validation: startup's +// structural checks plus full Parquet decoding and an exact span/index walk. +// Repair uses it to prove a batch is unreadable before it can be set aside. +func ValidatePublishedBatch(dir string) error { + batch, err := loadRegisteredBatch(dir) + if err != nil { + return err + } + if batch.metadata.Spans > 0 { + if err := verifySpanParquet(filepath.Join(dir, "spans.parquet"), batch.metadata.Spans, batch.traces); err != nil { + return fmt.Errorf("verify spans Parquet: %w", err) + } + } + if batch.metadata.Logs > 0 { + if err := verifyTypedParquet[logParquetRow](filepath.Join(dir, "logs.parquet"), batch.metadata.Logs, nil); err != nil { + return fmt.Errorf("verify logs Parquet: %w", err) + } + } + if batch.metadata.Metrics > 0 { + if err := verifyTypedParquet[metricParquetRow](filepath.Join(dir, "metrics.parquet"), batch.metadata.Metrics, nil); err != nil { + return fmt.Errorf("verify metrics Parquet: %w", err) + } + } + return nil +} + +func verifySpanParquet(path string, expected int, index traceIndex) (err error) { + indexFile, err := os.Open(index.path) + if err != nil { + return err + } + defer func() { err = errors.Join(err, indexFile.Close()) }() + var current traceRange + var have bool + var entry uint64 + flush := func() error { + if !have { + return nil + } + indexed, err := readTraceRangeAt(indexFile, entry) + if err != nil { + return err + } + if indexed != current { + return fmt.Errorf("trace index entry %d does not match span rows", entry) + } + entry++ + have = false + return nil + } + err = verifyTypedParquet[spanParquetRow](path, expected, func(row spanParquetRow, position uint64) error { + if row.TraceHash != xxh3.HashString(row.TraceID) { + return fmt.Errorf("span row %d trace hash does not match trace ID", position) + } + if !have { + current = traceRange{hash: row.TraceHash, row: position, count: 1} + have = true + return nil + } + if current.hash == row.TraceHash { + current.count++ + return nil + } + if current.hash > row.TraceHash { + return fmt.Errorf("span row %d is not sorted by trace hash", position) + } + if err := flush(); err != nil { + return err + } + current = traceRange{hash: row.TraceHash, row: position, count: 1} + have = true + return nil + }) + if err != nil { + return err + } + if err := flush(); err != nil { + return err + } + if entry != index.entries { + return fmt.Errorf("trace index has %d entries; span rows produced %d", index.entries, entry) + } + return nil +} + +func verifyTypedParquet[T any](path string, expected int, visit func(T, uint64) error) (err error) { + file, err := os.Open(path) + if err != nil { + return err + } + defer func() { err = errors.Join(err, file.Close()) }() + reader := parquet.NewGenericReader[T](file) + defer func() { err = errors.Join(err, reader.Close()) }() + buffer := make([]T, 256) + var rows uint64 + for { + n, readErr := reader.Read(buffer) + for i := 0; i < n; i++ { + if visit != nil { + if err := visit(buffer[i], rows); err != nil { + return err + } + } + rows++ + } + clear(buffer[:n]) + if errors.Is(readErr, io.EOF) { + break + } + if readErr != nil { + return readErr + } + if n == 0 { + return io.ErrNoProgress + } + } + if rows != uint64(expected) { + return fmt.Errorf("decoded %d rows; metadata declares %d", rows, expected) + } + return nil +} + +// installBatch publishes a validated batch into the queryable set. +func (p *ParquetStore) installBatch(batch *storedBatch) { + p.mu.Lock() + p.batches[batch.metadata.ID] = batch + p.mu.Unlock() +} + +func loadStoredBatch(dir string) (*storedBatch, error) { + metadata, err := readBatchMetadata(filepath.Join(dir, "metadata.json")) + if err != nil { + return nil, err + } + signals := [...]struct { + name string + count int + }{ + {name: "spans", count: metadata.Spans}, + {name: "logs", count: metadata.Logs}, + {name: "metrics", count: metadata.Metrics}, + } + for _, signal := range signals { + if signal.count == 0 { + continue + } + path := filepath.Join(dir, signal.name+".parquet") + info, err := os.Stat(path) + if err != nil { + return nil, err + } + if !info.Mode().IsRegular() { + return nil, fmt.Errorf("%s Parquet is not a regular file", signal.name) + } + file, err := os.Open(path) + if err != nil { + return nil, err + } + parquetFile, openErr := parquet.OpenFile(file, info.Size()) + closeErr := file.Close() + if err := errors.Join(openErr, closeErr); err != nil { + return nil, fmt.Errorf("open %s Parquet: %w", signal.name, err) + } + if rows := parquetFile.NumRows(); rows != int64(signal.count) { + return nil, fmt.Errorf("%s Parquet has %d rows; metadata declares %d", signal.name, rows, signal.count) + } + } + var traces traceIndex + if metadata.Spans > 0 { + traces, err = loadTraceIndex(filepath.Join(dir, "trace.fidx"), metadata.Spans) + if err != nil { + return nil, err + } + } + return &storedBatch{metadata: metadata, dir: dir, traces: traces}, nil +} + +func (p *ParquetStore) hasBatch(id string) bool { + p.mu.RLock() + _, ok := p.batches[id] + p.mu.RUnlock() + return ok +} + +func spanSortingColumns() parquet.SortingOption { + return parquet.SortingColumns( + parquet.Ascending("_trace_hash"), + parquet.Ascending("start_unix_nano"), + parquet.Ascending("span_id"), + ) +} + +func parquetWriterOptions(pageSize int, extra ...parquet.WriterOption) []parquet.WriterOption { + options := []parquet.WriterOption{ + parquet.Compression(&zstd.Codec{Level: zstd.SpeedFastest, Concurrency: 1}), + parquet.MaxRowsPerRowGroup(parquetRowGroupRows), + parquet.PageBufferSize(pageSize), + } + return append(options, extra...) +} + +type contextParquetRows struct { + context.Context + parquet.Rows +} + +func (r contextParquetRows) ReadRows(rows []parquet.Row) (int, error) { + if err := r.Context.Err(); err != nil { + return 0, err + } + return r.Rows.ReadRows(rows) +} + +func mergeTypedParquet[T any](ctx context.Context, inputs []string, output string, sorted bool) (err error) { + if err := ctx.Err(); err != nil { + return err + } + if len(inputs) == 0 { + return errors.New("merge Parquet requires at least one input") + } + files := make([]*os.File, 0, len(inputs)) + defer func() { + for _, file := range files { + err = errors.Join(err, file.Close()) + } + }() + groups := make([]parquet.RowGroup, 0, len(inputs)) + for _, input := range inputs { + file, openErr := os.Open(input) + if openErr != nil { + return openErr + } + files = append(files, file) + info, statErr := file.Stat() + if statErr != nil { + return statErr + } + parquetFile, parquetErr := parquet.OpenFile(file, info.Size()) + if parquetErr != nil { + return parquetErr + } + groups = append(groups, parquetFile.RowGroups()...) + } + var source parquet.RowGroup + var options []parquet.WriterOption + if sorted { + source, err = parquet.MergeRowGroups(groups, parquet.SortingRowGroupConfig(spanSortingColumns())) + options = append(options, parquet.SortingWriterConfig(spanSortingColumns())) + } else { + source = parquet.MultiRowGroup(groups...) + } + if err != nil { + return err + } + rows := source.Rows() + defer func() { err = errors.Join(err, rows.Close()) }() + out, err := os.OpenFile(output, os.O_CREATE|os.O_EXCL|os.O_RDWR, 0o644) + if err != nil { + return err + } + ok := false + defer func() { + _ = out.Close() + if !ok { + _ = os.Remove(output) + } + }() + writer := parquet.NewGenericWriter[T](out, parquetWriterOptions(parquetPageSize, options...)...) + _, copyErr := parquet.CopyRows(writer, contextParquetRows{Context: ctx, Rows: rows}) + if closeErr := writer.Close(); copyErr != nil || closeErr != nil { + return errors.Join(copyErr, closeErr) + } + if err := out.Close(); err != nil { + return err + } + ok = true + return nil +} + +func writeTypedParquet[T any](path string, rows []T, pageSize int, options ...parquet.WriterOption) error { + f, err := os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_RDWR, 0o644) + if err != nil { + return err + } + ok := false + defer func() { + _ = f.Close() + if !ok { + _ = os.Remove(path) + } + }() + writer := parquet.NewGenericWriter[T](f, parquetWriterOptions(pageSize, options...)...) + if _, err := writer.Write(rows); err != nil { + _ = writer.Close() + return err + } + if err := writer.Close(); err != nil { + return err + } + if err := f.Sync(); err != nil { + return err + } + if err := f.Close(); err != nil { + return err + } + ok = true + return nil +} + +func readBatchMetadata(path string) (BatchMetadata, error) { + data, err := os.ReadFile(path) + if err != nil { + return BatchMetadata{}, err + } + var metadata BatchMetadata + if err := json.Unmarshal(data, &metadata); err != nil { + return BatchMetadata{}, err + } + if metadata.Version != batchMetadataVersion { + return BatchMetadata{}, fmt.Errorf("unsupported batch metadata version %d", metadata.Version) + } + if err := ValidateBatchID(metadata.ID); err != nil { + return BatchMetadata{}, err + } + if metadata.Spans < 0 || metadata.Logs < 0 || metadata.Metrics < 0 { + return BatchMetadata{}, errors.New("parquet batch metadata has a negative row count") + } + if metadata.MinIngestedNanos > 0 && metadata.MaxIngestedNanos > 0 && metadata.MinIngestedNanos > metadata.MaxIngestedNanos { + return BatchMetadata{}, errors.New("parquet batch metadata has an inverted time range") + } + return metadata, nil +} + +func writeJSONFile(path string, value any) error { + data, err := json.Marshal(value) + if err != nil { + return err + } + return writeBytesFile(path, data) +} + +func writeBytesFile(path string, data []byte) error { + f, err := os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o644) + if err != nil { + return err + } + if _, err := f.Write(data); err != nil { + _ = f.Close() + return err + } + if err := f.Sync(); err != nil { + _ = f.Close() + return err + } + return f.Close() +} + +func syncDirectory(dir string) error { + f, err := os.Open(dir) + if err != nil { + return err + } + defer f.Close() + return f.Sync() +} + +// ValidateBatchID is the one grammar for identifiers that become directory +// names under the storage root. It is exported because the store package +// validates marker-supplied IDs against the same rule before joining them to a +// path; a second copy there could drift and leave the two gates disagreeing +// about what is a safe name. +func ValidateBatchID(id string) error { + if id == "" || len(id) > 128 || id[0] == '.' || strings.ContainsAny(id, `/\\`) { + return fmt.Errorf("invalid telemetry batch ID %q", id) + } + for _, r := range id { + if !(r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || r == '-' || r == '_' || r == '.') { + return fmt.Errorf("invalid telemetry batch ID %q", id) + } + } + return nil +} + +type traceParquetRow struct { + TraceHash uint64 `parquet:"_trace_hash"` + StartUnixNano int64 `parquet:"start_unix_nano"` +} diff --git a/internal/telemetry/parquet_rows.go b/internal/telemetry/parquet_rows.go new file mode 100644 index 00000000..f275c15b --- /dev/null +++ b/internal/telemetry/parquet_rows.go @@ -0,0 +1,144 @@ +package telemetry + +type spanParquetRow struct { + Namespace string `parquet:"namespace"` + TraceID string `parquet:"trace_id"` + TraceHash uint64 `parquet:"_trace_hash"` + SpanID string `parquet:"span_id"` + ParentSpanID string `parquet:"parent_span_id"` + Service string `parquet:"service"` + Operation string `parquet:"operation"` + Kind string `parquet:"kind"` + StartTime int64 `parquet:"start_time,timestamp(nanosecond)"` + EndTime int64 `parquet:"end_time,timestamp(nanosecond)"` + StartUnixNano int64 `parquet:"start_unix_nano"` + EndUnixNano int64 `parquet:"end_unix_nano"` + DurationMS float64 `parquet:"duration_ms"` + Status string `parquet:"status"` + StatusMessage string `parquet:"status_message"` + ResourceJSON string `parquet:"resource_json,dict"` + AttributesJSON string `parquet:"attributes_json,dict"` + EventsJSON string `parquet:"events_json,dict"` + LinksJSON string `parquet:"links_json,dict"` + TraceState string `parquet:"trace_state,dict"` + Flags int64 `parquet:"flags"` + ScopeName string `parquet:"scope_name,dict"` + ScopeVersion string `parquet:"scope_version,dict"` + IngestedAt int64 `parquet:"ingested_at,timestamp(nanosecond)"` + IngestedUnixNano int64 `parquet:"ingested_unix_nano"` + HTTPMethod string `parquet:"http_method,dict"` + HTTPStatusCode string `parquet:"http_status_code,dict"` + HTTPRoute string `parquet:"http_route,dict"` + DBSystem string `parquet:"db_system,dict"` + RPCMethod string `parquet:"rpc_method,dict"` + RPCService string `parquet:"rpc_service,dict"` + PeerService string `parquet:"peer_service,dict"` + ServiceVersion string `parquet:"service_version,dict"` + DeploymentEnv string `parquet:"deployment_env,dict"` + ExceptionType string `parquet:"exception_type,dict"` + ExceptionMessage string `parquet:"exception_message,dict"` +} + +func makeSpanParquetRow(r Span) spanParquetRow { + return spanParquetRow{ + Namespace: r.Namespace, TraceID: r.TraceID, SpanID: r.SpanID, ParentSpanID: r.ParentSpanID, + Service: r.ServiceName, Operation: r.Name, Kind: r.Kind, StartTime: r.StartUnixNanos, + EndTime: r.EndUnixNanos, StartUnixNano: r.StartUnixNanos, EndUnixNano: r.EndUnixNanos, + DurationMS: r.DurationMS, Status: r.StatusCode, StatusMessage: r.StatusMsg, + ResourceJSON: string(r.ResourceJSON), AttributesJSON: string(r.AttributesJSON), EventsJSON: string(r.EventsJSON), LinksJSON: string(r.LinksJSON), + TraceState: r.TraceState, Flags: int64(r.Flags), ScopeName: r.ScopeName, ScopeVersion: r.ScopeVersion, + IngestedAt: r.IngestedAt, IngestedUnixNano: r.IngestedAt, HTTPMethod: r.HTTPMethod, + HTTPStatusCode: r.HTTPStatusCode, HTTPRoute: r.HTTPRoute, DBSystem: r.DBSystem, RPCMethod: r.RPCMethod, + RPCService: r.RPCService, PeerService: r.PeerService, ServiceVersion: r.ServiceVersion, + DeploymentEnv: r.DeploymentEnv, ExceptionType: r.ExceptionType, ExceptionMessage: r.ExceptionMessage, + } +} + +type indexedSpanParquetRow struct { + Namespace string `parquet:"namespace"` + TraceID string `parquet:"trace_id"` + SpanID string `parquet:"span_id"` + ParentSpanID string `parquet:"parent_span_id"` + Service string `parquet:"service"` + Operation string `parquet:"operation"` + Kind string `parquet:"kind"` + StartUnixNano int64 `parquet:"start_unix_nano"` + DurationMS float64 `parquet:"duration_ms"` + Status string `parquet:"status"` + StatusMessage string `parquet:"status_message"` +} + +func (r indexedSpanParquetRow) span() IndexedSpan { + return IndexedSpan{ + Namespace: r.Namespace, TraceID: r.TraceID, SpanID: r.SpanID, ParentSpanID: r.ParentSpanID, + ServiceName: r.Service, Name: r.Operation, Kind: r.Kind, StartUnixNanos: r.StartUnixNano, + DurationMS: r.DurationMS, StatusCode: r.Status, StatusMsg: r.StatusMessage, + } +} + +type logParquetRow struct { + Namespace string `parquet:"namespace,dict"` + LogTime int64 `parquet:"log_time,timestamp(nanosecond)"` + ObservedTime int64 `parquet:"observed_time,timestamp(nanosecond)"` + TimeUnixNano int64 `parquet:"time_unix_nano"` + ObservedTimeUnixNano int64 `parquet:"observed_time_unix_nano"` + Severity string `parquet:"severity,dict"` + SeverityNumber int64 `parquet:"severity_number"` + Body string `parquet:"body"` + Service string `parquet:"service,dict"` + TraceID string `parquet:"trace_id"` + SpanID string `parquet:"span_id"` + Flags int64 `parquet:"flags"` + ResourceJSON string `parquet:"resource_json,dict"` + AttributesJSON string `parquet:"attributes_json,dict"` + ScopeName string `parquet:"scope_name,dict"` + ScopeVersion string `parquet:"scope_version,dict"` + IngestedAt int64 `parquet:"ingested_at,timestamp(nanosecond)"` + IngestedUnixNano int64 `parquet:"ingested_unix_nano"` + BodyTemplate string `parquet:"body_template,dict"` +} + +func makeLogParquetRow(r Log) logParquetRow { + return logParquetRow{ + Namespace: r.Namespace, LogTime: FirstPositiveNanos(r.EventUnixNanos, r.TimeUnixNanos, r.ObservedTimeNanos, r.IngestedAt), + ObservedTime: FirstPositiveNanos(r.ObservedTimeNanos, r.EventUnixNanos, r.TimeUnixNanos, r.IngestedAt), TimeUnixNano: r.TimeUnixNanos, + ObservedTimeUnixNano: r.ObservedTimeNanos, Severity: r.Severity, SeverityNumber: int64(r.SeverityNumber), + Body: r.Body, Service: r.ServiceName, TraceID: r.TraceID, SpanID: r.SpanID, Flags: int64(r.Flags), + ResourceJSON: string(r.ResourceJSON), AttributesJSON: string(r.AttributesJSON), ScopeName: r.ScopeName, + ScopeVersion: r.ScopeVersion, IngestedAt: r.IngestedAt, IngestedUnixNano: r.IngestedAt, BodyTemplate: r.BodyTemplate, + } +} + +type metricParquetRow struct { + Namespace string `parquet:"namespace,dict"` + MetricTime int64 `parquet:"metric_time,timestamp(nanosecond)"` + TimeUnixNano int64 `parquet:"time_unix_nano"` + Name string `parquet:"name,dict"` + Description string `parquet:"description,dict"` + Unit string `parquet:"unit,dict"` + MetricType string `parquet:"metric_type,dict"` + Service string `parquet:"service,dict"` + Value float64 `parquet:"value"` + HistBoundsJSON string `parquet:"hist_bounds_json,dict"` + HistCountsJSON string `parquet:"hist_counts_json,dict"` + HistCount int64 `parquet:"hist_count"` + HistSum float64 `parquet:"hist_sum"` + ExemplarsJSON string `parquet:"exemplars_json,dict"` + AttributesJSON string `parquet:"attributes_json,dict"` + ResourceJSON string `parquet:"resource_json,dict"` + ScopeName string `parquet:"scope_name,dict"` + ScopeVersion string `parquet:"scope_version,dict"` + IngestedAt int64 `parquet:"ingested_at,timestamp(nanosecond)"` + IngestedUnixNano int64 `parquet:"ingested_unix_nano"` +} + +func makeMetricParquetRow(r Metric) metricParquetRow { + return metricParquetRow{ + Namespace: r.Namespace, MetricTime: FirstPositiveNanos(r.EventUnixNanos, r.TimeUnixNanos, r.IngestedAt), TimeUnixNano: r.TimeUnixNanos, + Name: r.Name, Description: r.Description, Unit: r.Unit, MetricType: r.Type, Service: r.ServiceName, + Value: r.Value, HistBoundsJSON: string(r.HistBoundsJSON), HistCountsJSON: string(r.HistCountsJSON), + HistCount: r.HistCount, HistSum: r.HistSum, ExemplarsJSON: string(r.ExemplarsJSON), + AttributesJSON: string(r.AttributesJSON), ResourceJSON: string(r.ResourceJSON), ScopeName: r.ScopeName, + ScopeVersion: r.ScopeVersion, IngestedAt: r.IngestedAt, IngestedUnixNano: r.IngestedAt, + } +} diff --git a/internal/telemetry/parquet_test.go b/internal/telemetry/parquet_test.go new file mode 100644 index 00000000..17e60df1 --- /dev/null +++ b/internal/telemetry/parquet_test.go @@ -0,0 +1,557 @@ +package telemetry + +import ( + "context" + "encoding/binary" + "encoding/json" + "errors" + "fmt" + "io" + "math" + "os" + "path/filepath" + "reflect" + "testing" + "time" + + "github.com/parquet-go/parquet-go" +) + +func TestParquetStorePublishesCompleteBatchAndRecovers(t *testing.T) { + dir := t.TempDir() + store, err := OpenParquetStore(dir) + if err != nil { + t.Fatal(err) + } + span := completeTestSpan() + metadata := BatchMetadata{ID: "batch-1", MinIngestedNanos: span.IngestedAt, MaxIngestedNanos: span.IngestedAt} + if err := store.CommitBatch(context.Background(), metadata, []Span{span}, []Log{{Namespace: "tenant", Body: "ready", TimeUnixNanos: 12}}, []Metric{{Namespace: "tenant", Name: "requests", TimeUnixNanos: 13}}); err != nil { + t.Fatal(err) + } + + batchDir := store.BatchPath(metadata.ID) + for _, name := range []string{"spans.parquet", "logs.parquet", "metrics.parquet", "trace.fidx", "metadata.json"} { + if _, err := os.Stat(filepath.Join(batchDir, name)); err != nil { + t.Fatalf("published batch is missing %s: %v", name, err) + } + } + if got := store.RowCount(); got != 3 { + t.Fatalf("row count = %d, want 3", got) + } + if err := store.CommitBatch(context.Background(), metadata, []Span{span}, nil, nil); err != nil { + t.Fatalf("idempotent commit: %v", err) + } + if got := store.RowCount(); got != 3 { + t.Fatalf("idempotent row count = %d, want 3", got) + } + + // An interrupted, unpublished directory is neither cataloged nor retained. + orphan := store.StagingPath("interrupted") + if err := os.Mkdir(orphan, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(orphan, "partial"), []byte("partial"), 0o644); err != nil { + t.Fatal(err) + } + if err := store.Close(); err != nil { + t.Fatal(err) + } + + reopened, err := OpenParquetStore(dir) + if err != nil { + t.Fatal(err) + } + if _, err := os.Stat(orphan); !os.IsNotExist(err) { + t.Fatalf("unpublished staging directory survived restart: %v", err) + } + spans, err := traceAll(reopened, span.TraceID) + if err != nil { + t.Fatal(err) + } + wantIndexed := IndexedSpan{ + Namespace: span.Namespace, TraceID: span.TraceID, SpanID: span.SpanID, ParentSpanID: span.ParentSpanID, + ServiceName: span.ServiceName, Name: span.Name, Kind: span.Kind, StartUnixNanos: span.StartUnixNanos, + DurationMS: span.DurationMS, StatusCode: span.StatusCode, StatusMsg: span.StatusMsg, + } + if !reflect.DeepEqual(spans, []IndexedSpan{wantIndexed}) { + t.Fatalf("trace projection mismatch\n got: %#v\nwant: %#v", spans, []IndexedSpan{wantIndexed}) + } + gotSpan := readOneParquetRow[spanParquetRow](t, filepath.Join(batchDir, "spans.parquet")) + wantSpan := makeSpanParquetRow(span) + wantSpan.TraceHash = gotSpan.TraceHash + if !reflect.DeepEqual(gotSpan, wantSpan) { + t.Fatalf("complete span row mismatch\n got: %#v\nwant: %#v", gotSpan, wantSpan) + } +} + +func TestParquetStoreTraceUsesExactIDAndEventOrder(t *testing.T) { + store, err := OpenParquetStore(t.TempDir()) + if err != nil { + t.Fatal(err) + } + spans := []Span{ + {TraceID: "wanted", SpanID: "later", StartUnixNanos: 30}, + {TraceID: "other", SpanID: "other", StartUnixNanos: 20}, + {TraceID: "wanted", SpanID: "earlier", StartUnixNanos: 10}, + } + if err := store.CommitBatch(context.Background(), BatchMetadata{ID: "batch"}, spans, nil, nil); err != nil { + t.Fatal(err) + } + got, err := traceAll(store, "wanted") + if err != nil { + t.Fatal(err) + } + if len(got) != 2 || got[0].SpanID != "earlier" || got[1].SpanID != "later" { + t.Fatalf("ordered trace = %#v", got) + } + missing, err := traceAll(store, "missing") + if err != nil || len(missing) != 0 { + t.Fatalf("missing trace = %#v, %v", missing, err) + } +} + +func TestValidatePublishedBatchDetectsTraceIndexDataDivergence(t *testing.T) { + store, err := OpenParquetStore(t.TempDir()) + if err != nil { + t.Fatal(err) + } + if err := store.CommitBatch(context.Background(), BatchMetadata{ID: "diverged"}, []Span{{ + TraceID: "trace", SpanID: "span", StartUnixNanos: 1, + }}, nil, nil); err != nil { + t.Fatal(err) + } + batchDir := store.BatchPath("diverged") + indexPath := filepath.Join(batchDir, "trace.fidx") + file, err := os.OpenFile(indexPath, os.O_RDWR, 0) + if err != nil { + t.Fatal(err) + } + var encoded [8]byte + binary.LittleEndian.PutUint64(encoded[:], 1) + if _, err := file.WriteAt(encoded[:], traceIndexHeaderSize); err != nil { + _ = file.Close() + t.Fatal(err) + } + if err := file.Close(); err != nil { + t.Fatal(err) + } + if _, err := loadRegisteredBatch(batchDir); err != nil { + t.Fatalf("startup structural validation unexpectedly caught divergence: %v", err) + } + if err := ValidatePublishedBatch(batchDir); err == nil { + t.Fatal("deep validation accepted a trace index that disagrees with Parquet") + } +} + +func TestParquetStoreTraceFiltersAndBoundsResultsDuringRead(t *testing.T) { + store, err := OpenParquetStore(t.TempDir()) + if err != nil { + t.Fatal(err) + } + spans := make([]Span, 0, 102) + for i := 100; i >= 1; i-- { + spans = append(spans, Span{ + Namespace: "prod", TraceID: "large-trace", SpanID: fmt.Sprintf("span-%03d", i), + StartUnixNanos: int64(i), + }) + } + spans = append(spans, + Span{Namespace: "staging", TraceID: "large-trace", SpanID: "wrong-namespace", StartUnixNanos: 11}, + Span{Namespace: "prod", TraceID: "large-trace", SpanID: "outside-window", StartUnixNanos: 200}, + ) + if err := store.CommitBatch(context.Background(), BatchMetadata{ID: "large"}, spans, nil, nil); err != nil { + t.Fatal(err) + } + + got, err := store.Trace(context.Background(), TraceQuery{ + TraceID: "large-trace", Namespace: "prod", StartNanos: 10, EndNanos: 90, Limit: 3, + }) + if err != nil { + t.Fatal(err) + } + if len(got) != 3 || got[0].SpanID != "span-010" || got[1].SpanID != "span-011" || got[2].SpanID != "span-012" { + t.Fatalf("bounded trace = %#v", got) + } +} + +func TestParquetStorePreservesCompleteLogAndMetricRows(t *testing.T) { + store, err := OpenParquetStore(t.TempDir()) + if err != nil { + t.Fatal(err) + } + logRow := Log{ + Namespace: "tenant", EventUnixNanos: 11, TimeUnixNanos: 12, ObservedTimeNanos: 13, + Severity: "ERROR", SeverityNumber: 17, Body: "declined", ServiceName: "checkout", + TraceID: "trace", SpanID: "span", Flags: 1, ResourceJSON: []byte(`{"host":"one"}`), + AttributesJSON: []byte(`{"attempt":2}`), ScopeName: "scope", ScopeVersion: "1.2.3", + IngestedAt: 14, BodyTemplate: "declined: {reason}", + } + metricRow := Metric{ + Namespace: "tenant", EventUnixNanos: 21, TimeUnixNanos: 22, Name: "request.duration", + Description: "request latency", Unit: "ms", Type: "histogram", ServiceName: "checkout", Value: 23.5, + HistBoundsJSON: []byte(`[1,5,10]`), HistCountsJSON: []byte(`[2,3,4,5]`), HistCount: 14, HistSum: 47, + ExemplarsJSON: []byte(`[{"trace_id":"trace"}]`), AttributesJSON: []byte(`{"route":"/pay"}`), + ResourceJSON: []byte(`{"host":"one"}`), ScopeName: "scope", ScopeVersion: "1.2.3", IngestedAt: 24, + } + if err := store.CommitBatch(context.Background(), BatchMetadata{ID: "complete-signals"}, nil, []Log{logRow}, []Metric{metricRow}); err != nil { + t.Fatal(err) + } + batchDir := store.BatchPath("complete-signals") + gotLog := readOneParquetRow[logParquetRow](t, filepath.Join(batchDir, "logs.parquet")) + if gotLog.LogTime != logRow.EventUnixNanos { + t.Fatalf("log event time = %d, want canonical %d", gotLog.LogTime, logRow.EventUnixNanos) + } + if want := makeLogParquetRow(logRow); !reflect.DeepEqual(gotLog, want) { + t.Fatalf("log row mismatch\n got: %#v\nwant: %#v", gotLog, want) + } + gotMetric := readOneParquetRow[metricParquetRow](t, filepath.Join(batchDir, "metrics.parquet")) + if gotMetric.MetricTime != metricRow.EventUnixNanos { + t.Fatalf("metric event time = %d, want canonical %d", gotMetric.MetricTime, metricRow.EventUnixNanos) + } + if want := makeMetricParquetRow(metricRow); !reflect.DeepEqual(gotMetric, want) { + t.Fatalf("metric row mismatch\n got: %#v\nwant: %#v", gotMetric, want) + } +} + +func TestParquetStoreSkipsTraceIndexesOutsideTimeWindow(t *testing.T) { + store, err := OpenParquetStore(t.TempDir()) + if err != nil { + t.Fatal(err) + } + if err := store.CommitBatch(context.Background(), BatchMetadata{ID: "old-traces"}, []Span{{ + TraceID: "wanted", SpanID: "old", StartUnixNanos: 100, + }}, nil, nil); err != nil { + t.Fatal(err) + } + if err := os.Remove(filepath.Join(store.BatchPath("old-traces"), "trace.fidx")); err != nil { + t.Fatal(err) + } + got, err := store.Trace(context.Background(), TraceQuery{ + TraceID: "wanted", StartNanos: 1_000, EndNanos: 2_000, Limit: 10, + }) + if err != nil { + t.Fatalf("non-overlapping query opened old trace index: %v", err) + } + if len(got) != 0 { + t.Fatalf("non-overlapping trace query returned %#v", got) + } +} + +func TestPublishReplacementValidatesBeforePublication(t *testing.T) { + store, err := OpenParquetStore(t.TempDir()) + if err != nil { + t.Fatal(err) + } + stage := t.TempDir() + called := false + err = store.PublishReplacement(stage, BatchMetadata{ID: "replacement"}, nil, func(publish func(context.Context) error) error { + called = true + return publish(context.Background()) + }) + if err == nil { + t.Fatal("invalid replacement was accepted") + } + if called { + t.Fatal("publication gate entered before replacement validation") + } +} + +func TestPruneReaderWaitDoesNotBlockCommit(t *testing.T) { + store, err := OpenParquetStore(t.TempDir()) + if err != nil { + t.Fatal(err) + } + if err := store.CommitBatch(context.Background(), BatchMetadata{ID: "expired", MaxIngestedNanos: 1}, []Span{{TraceID: "trace"}}, nil, nil); err != nil { + t.Fatal(err) + } + entered := make(chan struct{}) + release := make(chan struct{}) + done := make(chan error, 1) + go func() { + _, err := store.PruneBefore(2, 1, func(prune func(context.Context) error) error { + close(entered) + <-release + return prune(context.Background()) + }) + done <- err + }() + select { + case <-entered: + case <-time.After(time.Second): + t.Fatal("prune did not begin waiting for reader exclusion") + } + commitDone := make(chan error, 1) + go func() { + commitDone <- store.CommitBatch(context.Background(), BatchMetadata{ID: "concurrent", MaxIngestedNanos: 3}, []Span{{TraceID: "trace-2"}}, nil, nil) + }() + select { + case err := <-commitDone: + if err != nil { + t.Fatal(err) + } + case <-time.After(2 * time.Second): + t.Fatal("commit blocked behind prune's reader wait") + } + close(release) + if err := <-done; err != nil { + t.Fatal(err) + } +} + +func TestPrunePublicationContextBoundsStorageLockWait(t *testing.T) { + store, err := OpenParquetStore(t.TempDir()) + if err != nil { + t.Fatal(err) + } + if err := store.CommitBatch(context.Background(), BatchMetadata{ID: "expired", MaxIngestedNanos: 1}, []Span{{TraceID: "trace"}}, nil, nil); err != nil { + t.Fatal(err) + } + if err := store.lockPublish(context.Background()); err != nil { + t.Fatal(err) + } + defer store.unlockPublish() + + ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond) + defer cancel() + removed, err := store.PruneBefore(2, 1, func(prune func(context.Context) error) error { + return prune(ctx) + }) + if !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("PruneBefore error = %v, want deadline exceeded", err) + } + if removed != 0 || len(store.BatchMetadata()) != 1 { + t.Fatalf("timed-out publication removed=%d batches=%d, want the input untouched", removed, len(store.BatchMetadata())) + } +} + +func TestReplacementReaderWaitDoesNotBlockCommit(t *testing.T) { + store, err := OpenParquetStore(t.TempDir()) + if err != nil { + t.Fatal(err) + } + span := []Span{{TraceID: "trace", SpanID: "span", StartUnixNanos: 1}} + if err := store.CommitBatch(context.Background(), BatchMetadata{ID: "input"}, span, nil, nil); err != nil { + t.Fatal(err) + } + output := BatchMetadata{ID: "output"} + if err := store.CommitBatch(context.Background(), output, span, nil, nil); err != nil { + t.Fatal(err) + } + entered := make(chan struct{}) + release := make(chan struct{}) + done := make(chan error, 1) + stage := filepath.Join(t.TempDir(), "missing-stage") + go func() { + done <- store.PublishReplacement(stage, output, []string{"input"}, func(publish func(context.Context) error) error { + close(entered) + <-release + return publish(context.Background()) + }) + }() + select { + case <-entered: + case <-time.After(time.Second): + t.Fatal("replacement did not begin waiting for reader exclusion") + } + commitDone := make(chan error, 1) + go func() { + commitDone <- store.CommitBatch(context.Background(), BatchMetadata{ID: "concurrent"}, []Span{{TraceID: "trace-2"}}, nil, nil) + }() + select { + case err := <-commitDone: + if err != nil { + t.Fatal(err) + } + case <-time.After(2 * time.Second): + t.Fatal("commit blocked behind replacement's reader wait") + } + close(release) + if err := <-done; err != nil { + t.Fatal(err) + } +} + +func TestPublishReplacementKeepsOutputOnlyAfterSyncFailure(t *testing.T) { + store, err := OpenParquetStore(t.TempDir()) + if err != nil { + t.Fatal(err) + } + span := []Span{{TraceID: "trace", SpanID: "span", StartUnixNanos: 1}} + if err := store.CommitBatch(context.Background(), BatchMetadata{ID: "input"}, span, nil, nil); err != nil { + t.Fatal(err) + } + output := BatchMetadata{ID: "output"} + if err := store.CommitBatch(context.Background(), output, span, nil, nil); err != nil { + t.Fatal(err) + } + originalSync := syncPublishedDirectory + syncPublishedDirectory = func(string) error { return errors.New("injected directory sync failure") } + t.Cleanup(func() { syncPublishedDirectory = originalSync }) + + err = store.PublishReplacement(filepath.Join(t.TempDir(), "missing-stage"), output, []string{"input"}, func(publish func(context.Context) error) error { + return publish(context.Background()) + }) + if err == nil { + t.Fatal("replacement succeeded despite injected sync failure") + } + if _, err := os.Stat(store.BatchPath("input")); !os.IsNotExist(err) { + t.Fatalf("input was restored beside live output: %v", err) + } + if _, err := os.Stat(store.BatchPath("output")); err != nil { + t.Fatalf("replacement output missing: %v", err) + } + metadata := store.BatchMetadata() + if len(metadata) != 1 || metadata[0].ID != "output" { + t.Fatalf("live batches after sync failure = %#v, want output only", metadata) + } +} + +func TestPublishReplacementUnpublishesRecoveredOutputBeforeRetiringInputs(t *testing.T) { + store, err := OpenParquetStore(t.TempDir()) + if err != nil { + t.Fatal(err) + } + span := []Span{{TraceID: "trace", SpanID: "span", StartUnixNanos: 1}} + if err := store.CommitBatch(context.Background(), BatchMetadata{ID: "input"}, span, nil, nil); err != nil { + t.Fatal(err) + } + output := BatchMetadata{ID: "output"} + if err := store.CommitBatch(context.Background(), output, span, nil, nil); err != nil { + t.Fatal(err) + } + blockedRetired := filepath.Join(store.BatchesDir(), "input.retired-output") + if err := os.Mkdir(blockedRetired, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(blockedRetired, "block"), []byte("x"), 0o600); err != nil { + t.Fatal(err) + } + stage := filepath.Join(t.TempDir(), "recovery", "output") + err = store.PublishReplacement(stage, output, []string{"input"}, func(publish func(context.Context) error) error { + return publish(context.Background()) + }) + if err == nil { + t.Fatal("replacement succeeded despite blocked input retirement") + } + if _, err := os.Stat(store.BatchPath("output")); !os.IsNotExist(err) { + t.Fatalf("recovered output remained visible beside active input: %v", err) + } + if _, err := os.Stat(store.BatchPath("input")); err != nil { + t.Fatalf("input was not left queryable after rollback: %v", err) + } + if _, err := os.Stat(stage); err != nil { + t.Fatalf("output was not preserved for the next recovery attempt: %v", err) + } + metadata := store.BatchMetadata() + if len(metadata) != 1 || metadata[0].ID != "input" { + t.Fatalf("live batches after rollback = %#v, want input only", metadata) + } +} + +func TestParquetStoreRejectsUnsafeBatchID(t *testing.T) { + store, err := OpenParquetStore(t.TempDir()) + if err != nil { + t.Fatal(err) + } + for _, id := range []string{"", ".hidden", "../escape", "has space"} { + if err := store.CommitBatch(context.Background(), BatchMetadata{ID: id}, []Span{{TraceID: "t"}}, nil, nil); err == nil { + t.Fatalf("CommitBatch accepted unsafe ID %q", id) + } + } +} + +func TestParquetStoreRejectsMetadataRowCountMismatchOnOpen(t *testing.T) { + dir := t.TempDir() + store, err := OpenParquetStore(dir) + if err != nil { + t.Fatal(err) + } + if err := store.CommitBatch(context.Background(), BatchMetadata{ID: "mismatch"}, []Span{{TraceID: "trace"}}, nil, nil); err != nil { + t.Fatal(err) + } + metadataPath := filepath.Join(store.BatchPath("mismatch"), "metadata.json") + metadata, err := readBatchMetadata(metadataPath) + if err != nil { + t.Fatal(err) + } + metadata.Spans++ + data, err := json.Marshal(metadata) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(metadataPath, data, 0o644); err != nil { + t.Fatal(err) + } + if _, err := OpenParquetStore(dir); err == nil { + t.Fatal("OpenParquetStore accepted metadata with the wrong row count") + } +} + +func completeTestSpan() Span { + return Span{ + Namespace: "tenant", TraceID: "0123456789abcdef0123456789abcdef", SpanID: "0123456789abcdef", + ParentSpanID: "fedcba9876543210", ServiceName: "checkout", Name: "POST /orders", Kind: "SERVER", + StartUnixNanos: 10, EndUnixNanos: 20, DurationMS: 0.00001, StatusCode: "ERROR", StatusMsg: "declined", + ResourceJSON: []byte(`{"host":"one"}`), AttributesJSON: []byte(`{"http.request.method":"POST"}`), + EventsJSON: []byte(`[{"name":"exception"}]`), LinksJSON: []byte(`[{"trace_id":"linked"}]`), + TraceState: "vendor=value", Flags: 1, ScopeName: "scope", ScopeVersion: "1.2.3", IngestedAt: 30, + HTTPMethod: "POST", HTTPStatusCode: "500", HTTPRoute: "/orders", DBSystem: "postgresql", + RPCMethod: "Create", RPCService: "orders.v1.Orders", PeerService: "payments", ServiceVersion: "4.5.6", + DeploymentEnv: "production", ExceptionType: "CardDeclined", ExceptionMessage: "declined", + } +} + +func traceAll(store *ParquetStore, traceID string) ([]IndexedSpan, error) { + return store.Trace(context.Background(), TraceQuery{ + TraceID: traceID, StartNanos: math.MinInt64, EndNanos: math.MaxInt64, Limit: maxTraceQueryResults, + }) +} + +func readOneParquetRow[T any](t *testing.T, path string) T { + t.Helper() + file, err := os.Open(path) + if err != nil { + t.Fatal(err) + } + defer file.Close() + reader := parquet.NewGenericReader[T](file) + defer reader.Close() + var rows [1]T + if n, err := reader.Read(rows[:]); err != nil && !errors.Is(err, io.EOF) || n != 1 { + t.Fatalf("read %s: rows=%d err=%v", path, n, err) + } + return rows[0] +} + +// TestCommitBatchDoesNotBlockIndefinitelyOnStalledPublication pins the +// invariant that ingest is never held hostage by a publication. CommitBatch +// used to wait on the publish gate with context.Background(), so a publisher +// stuck inside the gate blocked every OTLP commit until it finished — the +// clients then dropped rows on their own timeouts, turning a latency problem +// into permanent data loss. +func TestCommitBatchDoesNotBlockIndefinitelyOnStalledPublication(t *testing.T) { + store, err := OpenParquetStore(t.TempDir()) + if err != nil { + t.Fatal(err) + } + if err := store.lockPublish(context.Background()); err != nil { + t.Fatal(err) + } + defer store.unlockPublish() + + ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond) + defer cancel() + span := completeTestSpan() + metadata := BatchMetadata{ID: "batch-stalled", MinIngestedNanos: span.IngestedAt, MaxIngestedNanos: span.IngestedAt} + + done := make(chan error, 1) + go func() { done <- store.CommitBatch(ctx, metadata, []Span{span}, nil, nil) }() + select { + case err := <-done: + if !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("commit against a stalled publication = %v, want context.DeadlineExceeded", err) + } + case <-time.After(10 * time.Second): + t.Fatal("commit blocked on a stalled publication instead of honouring its context") + } +} diff --git a/internal/telemetry/rows.go b/internal/telemetry/rows.go new file mode 100644 index 00000000..09b7e4a7 --- /dev/null +++ b/internal/telemetry/rows.go @@ -0,0 +1,114 @@ +// Package telemetry defines the canonical, storage-independent rows emitted by +// the OTLP decoder. These are the only telemetry write contracts in Fanout. +package telemetry + +type Span struct { + Namespace string + TraceID string + SpanID string + ParentSpanID string + ServiceName string + Name string + Kind string + StartUnixNanos int64 + EndUnixNanos int64 + DurationMS float64 + StatusCode string + StatusMsg string + ResourceJSON []byte + AttributesJSON []byte + EventsJSON []byte + LinksJSON []byte + TraceState string + Flags uint32 + ScopeName string + ScopeVersion string + IngestedAt int64 + HTTPMethod string + HTTPStatusCode string + HTTPRoute string + DBSystem string + RPCMethod string + RPCService string + PeerService string + ServiceVersion string + DeploymentEnv string + ExceptionType string + ExceptionMessage string +} + +// IndexedSpan is the narrow projection needed by trace-detail queries. The +// complete authoritative span remains in Parquet and is available through SQL; +// keeping this narrow projection small avoids decoding large JSON columns. +type IndexedSpan struct { + Namespace string + TraceID string + SpanID string + ParentSpanID string + ServiceName string + Name string + Kind string + StartUnixNanos int64 + DurationMS float64 + StatusCode string + StatusMsg string +} + +type Log struct { + Namespace string + EventUnixNanos int64 + TimeUnixNanos int64 + ObservedTimeNanos int64 + Severity string + SeverityNumber int32 + Body string + ServiceName string + TraceID string + SpanID string + Flags uint32 + ResourceJSON []byte + AttributesJSON []byte + ScopeName string + ScopeVersion string + IngestedAt int64 + BodyTemplate string +} + +type Metric struct { + Namespace string + EventUnixNanos int64 + TimeUnixNanos int64 + Name string + Description string + Unit string + Type string + ServiceName string + Value float64 + HistBoundsJSON []byte + HistCountsJSON []byte + HistCount int64 + HistSum float64 + ExemplarsJSON []byte + AttributesJSON []byte + ResourceJSON []byte + ScopeName string + ScopeVersion string + IngestedAt int64 +} + +func NormalizeNamespace(namespace string) string { + if namespace == "" { + return "default" + } + return namespace +} + +// FirstPositiveNanos returns the first usable timestamp in priority order. +func FirstPositiveNanos(values ...int64) int64 { + for _, value := range values { + if value > 0 { + return value + } + } + return 0 +} diff --git a/internal/telemetry/store/compaction.go b/internal/telemetry/store/compaction.go new file mode 100644 index 00000000..f5f0ec6e --- /dev/null +++ b/internal/telemetry/store/compaction.go @@ -0,0 +1,393 @@ +package store + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "math" + "os" + "path/filepath" + "sort" + "time" + + "github.com/labstack/fanout/internal/telemetry" + "golang.org/x/sync/errgroup" +) + +const ( + maxCompactionRows = 25_000_000 + minCompactionInputs = 8 +) + +var parquetSignals = [...]string{"spans", "logs", "metrics"} + +type compactionMarker struct { + Output telemetry.BatchMetadata `json:"output"` + Inputs []string `json:"inputs"` +} + +type compactionKey struct { + day int64 + generation uint32 +} + +// ParquetPublisher lets the query layer exclude readers only for the atomic +// namespace swap; storage owns native merges and crash-safe replacement state. +type ParquetPublisher interface { + PublishParquet(context.Context, func(context.Context) error) error +} + +// CompactParquet combines one same-day, same-generation group. The output is +// prepared outside the query gate and swapped as one batch directory. +func (r *Repository) CompactParquet(ctx context.Context, publisher ParquetPublisher, maxBatches int) (int, error) { + if publisher == nil || maxBatches < minCompactionInputs { + return 0, nil + } + r.compactionMu.Lock() + defer r.compactionMu.Unlock() + markerPath := filepath.Join(r.root, "COMPACTION.json") + if exists, err := pathExists(markerPath); err != nil { + return 0, err + } else if exists { + if err := r.recoverCompaction(ctx, publisher.PublishParquet); err != nil { + return 0, fmt.Errorf("recover pending Parquet compaction: %w", err) + } + } + selected := selectCompactionBatches(r.Parquet.BatchMetadata(), maxBatches) + if len(selected) < 2 { + return 0, nil + } + output := telemetry.BatchMetadata{ + ID: fmt.Sprintf("compact-%d", time.Now().UnixNano()), MinIngestedNanos: math.MaxInt64, + Generation: selected[0].Generation + 1, + } + marker := compactionMarker{Output: output, Inputs: make([]string, 0, len(selected))} + for _, batch := range selected { + marker.Inputs = append(marker.Inputs, batch.ID) + if batch.MinIngestedNanos > 0 { + marker.Output.MinIngestedNanos = min(marker.Output.MinIngestedNanos, batch.MinIngestedNanos) + } + marker.Output.MaxIngestedNanos = max(marker.Output.MaxIngestedNanos, batch.MaxIngestedNanos) + marker.Output.Spans += batch.Spans + marker.Output.Logs += batch.Logs + marker.Output.Metrics += batch.Metrics + } + if marker.Output.MinIngestedNanos == math.MaxInt64 { + marker.Output.MinIngestedNanos = 0 + } + stage := r.compactionStage(marker.Output.ID) + if err := os.RemoveAll(stage); err != nil { + return 0, err + } + if err := os.MkdirAll(stage, 0o755); err != nil { + return 0, err + } + if err := errors.Join(syncDirectory(filepath.Dir(stage)), syncDirectory(r.root)); err != nil { + return 0, err + } + prepared := false + defer func() { + if !prepared { + _ = os.RemoveAll(stage) + } + }() + type mergePlan struct { + signal string + inputs []string + output string + } + plans := make([]mergePlan, 0, len(parquetSignals)) + for _, signal := range parquetSignals { + inputs := make([]string, 0, len(selected)) + for _, batch := range selected { + path := filepath.Join(r.Parquet.BatchPath(batch.ID), signal+".parquet") + if _, err := os.Stat(path); err == nil { + inputs = append(inputs, path) + } else if !errors.Is(err, os.ErrNotExist) { + return 0, err + } + } + if len(inputs) == 0 { + continue + } + plans = append(plans, mergePlan{signal: signal, inputs: inputs, output: filepath.Join(stage, signal+".parquet")}) + } + group, mergeCtx := errgroup.WithContext(ctx) + for _, plan := range plans { + group.Go(func() error { + if err := r.Parquet.MergeParquet(mergeCtx, plan.signal, plan.inputs, plan.output); err != nil { + return fmt.Errorf("compact %s Parquet: %w", plan.signal, err) + } + if err := mergeCtx.Err(); err != nil { + return err + } + if err := syncFile(plan.output); err != nil { + return fmt.Errorf("sync compacted %s Parquet: %w", plan.signal, err) + } + return nil + }) + } + if err := group.Wait(); err != nil { + return 0, err + } + if err := r.Parquet.PrepareReplacement(stage, marker.Output); err != nil { + return 0, err + } + data, err := json.Marshal(marker) + if err != nil { + return 0, err + } + if err := writeDurableFile(markerPath, data); err != nil { + return 0, err + } + if err := syncDirectory(r.root); err != nil { + return 0, err + } + prepared = true + if err := r.completeCompaction(ctx, marker, publisher.PublishParquet); err != nil { + return 0, err + } + return len(selected), nil +} + +func selectCompactionBatches(batches []telemetry.BatchMetadata, maxBatches int) []telemetry.BatchMetadata { + if maxBatches < minCompactionInputs { + return nil + } + groups := make(map[compactionKey][]telemetry.BatchMetadata) + for _, batch := range batches { + if batch.MaxIngestedNanos <= 0 { + continue + } + key := compactionKey{day: batch.MaxIngestedNanos / int64(24*time.Hour), generation: batch.Generation} + groups[key] = append(groups[key], batch) + } + var chosen compactionKey + var selected []telemetry.BatchMetadata + found := false + for key, group := range groups { + candidate := selectBoundedCompactionGroup(group, maxBatches) + if len(candidate) < 2 { + continue + } + if !found || key.day < chosen.day || key.day == chosen.day && key.generation < chosen.generation { + chosen, selected, found = key, candidate, true + } + } + if !found { + return nil + } + return selected +} + +// selectBoundedCompactionGroup keeps the high-reclaim full-group behavior for +// small files, but admits a smaller group when the row ceiling fills first. +// Without the latter, one successful generation can make every later group too +// large for the ceiling and permanently strand those files. +func selectBoundedCompactionGroup(group []telemetry.BatchMetadata, maxBatches int) []telemetry.BatchMetadata { + ordered := append([]telemetry.BatchMetadata(nil), group...) + sort.Slice(ordered, func(i, j int) bool { + left, right := compactionBatchRows(ordered[i]), compactionBatchRows(ordered[j]) + if left != right { + return left < right + } + if ordered[i].MinIngestedNanos != ordered[j].MinIngestedNanos { + return ordered[i].MinIngestedNanos < ordered[j].MinIngestedNanos + } + return ordered[i].ID < ordered[j].ID + }) + candidate := make([]telemetry.BatchMetadata, 0, min(maxBatches, len(ordered))) + var rows int64 + saturated := false + var smallest int64 + for _, batch := range ordered { + batchRows := compactionBatchRows(batch) + if batchRows <= 0 || batchRows > maxCompactionRows { + continue + } + if smallest == 0 { + smallest = batchRows + } + if len(candidate) == maxBatches { + saturated = true + break + } + if rows > maxCompactionRows-batchRows { + saturated = true + break + } + candidate = append(candidate, batch) + rows += batchRows + } + if rows == maxCompactionRows || smallest > 0 && smallest > maxCompactionRows-rows { + saturated = true + } + if len(candidate) == maxBatches || saturated && len(candidate) >= 2 { + return candidate + } + return nil +} + +func compactionBatchRows(batch telemetry.BatchMetadata) int64 { + if batch.Spans < 0 { + return math.MaxInt64 + } + rows := int64(batch.Spans) + for _, count := range [...]int{batch.Logs, batch.Metrics} { + if count < 0 || rows > math.MaxInt64-int64(count) { + return math.MaxInt64 + } + rows += int64(count) + } + return rows +} + +// CompactParquetPass starts complete compactions until the phase budget +// expires. An in-flight merge keeps the caller context so slow, valid work +// commits instead of restarting the same input group on every pass. +func (r *Repository) CompactParquetPass(ctx context.Context, publisher ParquetPublisher, maxBatches int, budget time.Duration) (int, error) { + if maxBatches <= 0 || budget <= 0 { + return 0, nil + } + deadline := time.Now().Add(budget) + total := 0 + for { + count, err := r.CompactParquet(ctx, publisher, maxBatches) + total += count + if err != nil || count == 0 || !time.Now().Before(deadline) { + return total, err + } + } +} + +type parquetPublishFunc func(context.Context, func(context.Context) error) error + +// RecoverParquet resolves a pending compaction marker before any cleanup, +// retention, or new compaction can mutate its rollback set. +func (r *Repository) RecoverParquet(ctx context.Context, publisher ParquetPublisher) error { + r.compactionMu.Lock() + defer r.compactionMu.Unlock() + return r.recoverCompaction(ctx, publisher.PublishParquet) +} + +func (r *Repository) recoverCompaction(ctx context.Context, publish parquetPublishFunc) error { + data, err := os.ReadFile(filepath.Join(r.root, "COMPACTION.json")) + if errors.Is(err, os.ErrNotExist) { + return nil + } + if err != nil { + return err + } + var marker compactionMarker + if err := json.Unmarshal(data, &marker); err != nil { + return err + } + if err := validateCompactionMarker(marker); err != nil { + return err + } + stageExists, err := pathExists(r.compactionStage(marker.Output.ID)) + if err != nil { + return err + } + finalExists, err := pathExists(r.Parquet.BatchPath(marker.Output.ID)) + if err != nil { + return err + } + if !stageExists && !finalExists { + if err := r.Parquet.RestoreRetiredInputs(marker.Inputs, marker.Output.ID, func(swap func(context.Context) error) error { + return publish(ctx, swap) + }); err != nil { + return fmt.Errorf("restore compaction %s inputs: %w", marker.Output.ID, err) + } + if err := os.Remove(filepath.Join(r.root, "COMPACTION.json")); err != nil && !errors.Is(err, os.ErrNotExist) { + return err + } + return syncDirectory(r.root) + } + return r.completeCompaction(ctx, marker, publish) +} + +func (r *Repository) completeCompaction(ctx context.Context, marker compactionMarker, publish parquetPublishFunc) error { + if err := r.Parquet.PublishReplacement(r.compactionStage(marker.Output.ID), marker.Output, marker.Inputs, func(swap func(context.Context) error) error { + return publish(ctx, swap) + }); err != nil { + return err + } + markerPath := filepath.Join(r.root, "COMPACTION.json") + if err := os.Remove(markerPath); err != nil && !errors.Is(err, os.ErrNotExist) { + return err + } + return syncDirectory(r.root) +} + +func (r *Repository) compactionStage(id string) string { + return filepath.Join(r.root, "compaction", id) +} + +func validateCompactionMarker(marker compactionMarker) error { + if err := telemetry.ValidateBatchID(marker.Output.ID); err != nil { + return fmt.Errorf("invalid compaction output: %w", err) + } + if len(marker.Inputs) == 0 { + return errors.New("compaction marker has no inputs") + } + for _, id := range marker.Inputs { + if err := telemetry.ValidateBatchID(id); err != nil { + return fmt.Errorf("invalid compaction input: %w", err) + } + } + return nil +} + +func syncFile(path string) error { + f, err := os.OpenFile(path, os.O_RDWR, 0) + if err != nil { + return err + } + if err := f.Sync(); err != nil { + _ = f.Close() + return err + } + return f.Close() +} + +func syncDirectory(path string) error { + dir, err := os.Open(path) + if err != nil { + return err + } + defer dir.Close() + return dir.Sync() +} + +func pathExists(path string) (bool, error) { + _, err := os.Stat(path) + if err == nil { + return true, nil + } + if errors.Is(err, os.ErrNotExist) { + return false, nil + } + return false, err +} + +func writeDurableFile(path string, data []byte) error { + tmp := path + ".tmp" + f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o600) + if err != nil { + return err + } + if _, err := f.Write(data); err != nil { + _ = f.Close() + return err + } + if err := f.Sync(); err != nil { + _ = f.Close() + return err + } + if err := f.Close(); err != nil { + return err + } + return os.Rename(tmp, path) +} diff --git a/internal/telemetry/store/repair.go b/internal/telemetry/store/repair.go new file mode 100644 index 00000000..0ec6897a --- /dev/null +++ b/internal/telemetry/store/repair.go @@ -0,0 +1,106 @@ +package store + +import ( + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" + "strings" + "time" + + "github.com/labstack/fanout/internal/telemetry" +) + +// BatchIssue is one authoritative batch that startup cannot validate. +type BatchIssue struct { + ID string + Path string + Err error +} + +// VerifyBatches validates every authoritative batch without opening a live +// repository or changing the filesystem. +func VerifyBatches(root string) ([]BatchIssue, error) { + batchesDir := filepath.Join(root, "parquet", "batches") + entries, err := os.ReadDir(batchesDir) + if err != nil { + return nil, fmt.Errorf("read telemetry batches: %w", err) + } + var issues []BatchIssue + for _, entry := range entries { + if !entry.IsDir() || entry.Name() == telemetry.SchemaBatch || !strings.HasSuffix(entry.Name(), telemetry.BatchSuffix) { + continue + } + path := filepath.Join(batchesDir, entry.Name()) + if err := telemetry.ValidatePublishedBatch(path); err != nil { + issues = append(issues, BatchIssue{ + ID: strings.TrimSuffix(entry.Name(), telemetry.BatchSuffix), Path: path, Err: err, + }) + } + } + return issues, nil +} + +// QuarantineBatch atomically sets aside one specifically named unreadable +// batch. Valid data and any batch protected by a live compaction transaction +// are refused. The returned directory remains beside the authoritative set so +// an operator can recover it from a backup or rename it back after repair. +func QuarantineBatch(root, id string) (string, error) { + if err := telemetry.ValidateBatchID(id); err != nil { + return "", err + } + if id+telemetry.BatchSuffix == telemetry.SchemaBatch { + return "", errors.New("the Parquet schema batch cannot be quarantined") + } + if err := ensureBatchOutsideLiveCompaction(root, id); err != nil { + return "", err + } + batchesDir := filepath.Join(root, "parquet", "batches") + source := filepath.Join(batchesDir, id+telemetry.BatchSuffix) + info, err := os.Lstat(source) + if err != nil { + return "", fmt.Errorf("inspect telemetry batch %s: %w", id, err) + } + if !info.IsDir() { + return "", fmt.Errorf("telemetry batch %s is not a directory", id) + } + if err := telemetry.ValidatePublishedBatch(source); err == nil { + return "", fmt.Errorf("telemetry batch %s is valid; refusing to quarantine authoritative data", id) + } + destination := filepath.Join(batchesDir, fmt.Sprintf("%s.quarantined-%d", id, time.Now().UTC().UnixNano())) + if err := os.Rename(source, destination); err != nil { + return "", fmt.Errorf("quarantine telemetry batch %s: %w", id, err) + } + if err := syncDirectory(batchesDir); err != nil { + return destination, fmt.Errorf("quarantined telemetry batch %s at %s but could not sync the directory: %w", id, destination, err) + } + return destination, nil +} + +func ensureBatchOutsideLiveCompaction(root, id string) error { + path := filepath.Join(root, "COMPACTION.json") + data, err := os.ReadFile(path) + if errors.Is(err, os.ErrNotExist) { + return nil + } + if err != nil { + return fmt.Errorf("inspect live compaction marker: %w", err) + } + var marker compactionMarker + if err := json.Unmarshal(data, &marker); err != nil { + return fmt.Errorf("read live compaction marker: %w", err) + } + if err := validateCompactionMarker(marker); err != nil { + return fmt.Errorf("read live compaction marker: %w", err) + } + if marker.Output.ID == id { + return fmt.Errorf("telemetry batch %s belongs to a live compaction; resolve that transaction before repair", id) + } + for _, input := range marker.Inputs { + if input == id { + return fmt.Errorf("telemetry batch %s belongs to a live compaction; resolve that transaction before repair", id) + } + } + return nil +} diff --git a/internal/telemetry/store/repair_test.go b/internal/telemetry/store/repair_test.go new file mode 100644 index 00000000..855f647f --- /dev/null +++ b/internal/telemetry/store/repair_test.go @@ -0,0 +1,142 @@ +package store + +import ( + "context" + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/labstack/fanout/internal/telemetry" +) + +func TestVerifyBatchesReportsOnlyUnreadableBatches(t *testing.T) { + root := t.TempDir() + repository, err := Open(root) + if err != nil { + t.Fatal(err) + } + for _, id := range []string{"good", "broken"} { + if err := repository.Commit(context.Background(), Batch{ + ID: id, Spans: []telemetry.Span{{TraceID: id, SpanID: "span"}}, + }); err != nil { + t.Fatal(err) + } + } + if err := repository.Close(); err != nil { + t.Fatal(err) + } + corruptBatchFile(t, root, "broken", "spans.parquet") + + issues, err := VerifyBatches(root) + if err != nil { + t.Fatal(err) + } + if len(issues) != 1 || issues[0].ID != "broken" || issues[0].Err == nil { + t.Fatalf("issues = %#v", issues) + } +} + +func TestQuarantineBatchRefusesValidData(t *testing.T) { + root := t.TempDir() + repository, err := Open(root) + if err != nil { + t.Fatal(err) + } + if err := repository.Commit(context.Background(), Batch{ + ID: "valid", Spans: []telemetry.Span{{TraceID: "trace", SpanID: "span"}}, + }); err != nil { + t.Fatal(err) + } + if err := repository.Close(); err != nil { + t.Fatal(err) + } + if _, err := QuarantineBatch(root, "valid"); err == nil || !strings.Contains(err.Error(), "valid") { + t.Fatalf("QuarantineBatch valid error = %v", err) + } +} + +func TestQuarantineBatchSetsAsideUnreadableDataAndRestoresStartup(t *testing.T) { + root := t.TempDir() + repository, err := Open(root) + if err != nil { + t.Fatal(err) + } + for _, id := range []string{"good", "broken"} { + if err := repository.Commit(context.Background(), Batch{ + ID: id, Spans: []telemetry.Span{{TraceID: id, SpanID: "span"}}, + }); err != nil { + t.Fatal(err) + } + } + if err := repository.Close(); err != nil { + t.Fatal(err) + } + corruptBatchFile(t, root, "broken", "trace.fidx") + + destination, err := QuarantineBatch(root, "broken") + if err != nil { + t.Fatal(err) + } + if !strings.Contains(filepath.Base(destination), "broken.quarantined-") { + t.Fatalf("destination = %q", destination) + } + if _, err := os.Stat(filepath.Join(root, "parquet", "batches", "broken.batch")); !os.IsNotExist(err) { + t.Fatalf("authoritative batch still exists: %v", err) + } + if info, err := os.Stat(destination); err != nil || !info.IsDir() { + t.Fatalf("quarantine destination = %v, %v", info, err) + } + + reopened, err := Open(root) + if err != nil { + t.Fatalf("startup after quarantine: %v", err) + } + defer reopened.Close() + if got := reopened.RowCount(); got != 1 { + t.Fatalf("row count after quarantine = %d, want 1", got) + } +} + +func TestQuarantineBatchRefusesLiveCompactionMembers(t *testing.T) { + root := t.TempDir() + repository, err := Open(root) + if err != nil { + t.Fatal(err) + } + if err := repository.Commit(context.Background(), Batch{ + ID: "input", Spans: []telemetry.Span{{TraceID: "trace", SpanID: "span"}}, + }); err != nil { + t.Fatal(err) + } + if err := repository.Close(); err != nil { + t.Fatal(err) + } + corruptBatchFile(t, root, "input", "spans.parquet") + marker := compactionMarker{Output: telemetry.BatchMetadata{ID: "replacement"}, Inputs: []string{"input"}} + data, err := json.Marshal(marker) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "COMPACTION.json"), data, 0o644); err != nil { + t.Fatal(err) + } + if _, err := QuarantineBatch(root, "input"); err == nil || !strings.Contains(err.Error(), "live compaction") { + t.Fatalf("QuarantineBatch live input error = %v", err) + } +} + +func TestQuarantineBatchRejectsUnsafeID(t *testing.T) { + if _, err := QuarantineBatch(t.TempDir(), "../outside"); err == nil { + t.Fatal("unsafe batch ID was accepted") + } +} + +func corruptBatchFile(t *testing.T, root, id, name string) { + t.Helper() + path := filepath.Join(root, "parquet", "batches", id+telemetry.BatchSuffix, name) + if err := os.WriteFile(path, []byte("corrupt"), 0o644); err != nil { + t.Fatal(err) + } +} diff --git a/internal/telemetry/store/repository.go b/internal/telemetry/store/repository.go new file mode 100644 index 00000000..c78c1410 --- /dev/null +++ b/internal/telemetry/store/repository.go @@ -0,0 +1,289 @@ +package store + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "log/slog" + "math" + "os" + "path/filepath" + "strings" + "sync" + "time" + + "github.com/labstack/fanout/internal/telemetry" +) + +type Batch struct { + ID string + Spans []telemetry.Span + Logs []telemetry.Log + Metrics []telemetry.Metric +} + +// Repository publishes self-contained Parquet batch directories. The +// directory rename is the transaction and the filesystem is the catalog. +type Repository struct { + root string + Parquet *telemetry.ParquetStore + compactionMu sync.Mutex +} + +func Open(root string) (*Repository, error) { + if err := os.MkdirAll(root, 0o755); err != nil { + return nil, err + } + parquetStore, err := telemetry.OpenParquetStore(filepath.Join(root, "parquet")) + if err != nil { + return nil, err + } + r := &Repository{root: root, Parquet: parquetStore} + if err := r.recoverCompaction(context.Background(), func(ctx context.Context, publish func(context.Context) error) error { return publish(ctx) }); err != nil { + r.logUnresolvedCompaction(err) + _ = r.Close() + return nil, fmt.Errorf("recover Parquet compaction (unresolved marker at %s): %w", + filepath.Join(root, "COMPACTION.json"), err) + } + if err := r.cleanupCompactionArtifacts(); err != nil { + _ = r.Close() + return nil, fmt.Errorf("clean Parquet compaction staging: %w", err) + } + if err := r.cleanupRetired(); err != nil { + _ = r.Close() + return nil, fmt.Errorf("clean retired Parquet batches: %w", err) + } + return r, nil +} + +// compactionRunbookURL is the manual rollback procedure. The startup log links +// to it rather than restating it, because the procedure has preconditions that +// are refined as the failure modes are understood, and a copy here drifts from +// the guide silently — each revision so far has corrected one location and left +// the other saying something the code no longer guarantees. Guidance an +// operator can act on wrongly is worse than guidance in one place they have to +// open. TestUnresolvedCompactionRunbookExists keeps the link honest. +const compactionRunbookURL = "https://fanout.run/guides/troubleshoot#an-unresolved-compaction-blocks-startup" + +// logUnresolvedCompaction spells out the operator's options for a marker that +// blocks startup. +// +// Refusing to boot is only the safe half of the decision. A marker names the +// retired inputs a rollback still needs, and those directories hold the only +// copy of the rows the compaction was merging, so a startup that guessed at +// the rollback set could delete them — which is why recovery fails closed. The +// other half is saying where to look, because the reachable next step for an +// instance that will not start is rm -rf of the data directory, and that +// destroys exactly what failing closed preserved. +// +// The guidance is logged rather than wrapped into the error: the paths and the +// ordering constraint do not fit an error string that composes, and the +// rollback warning is the kind of thing an operator has to be able to read +// once, in full, at the moment the process refuses to come up. +func (r *Repository) logUnresolvedCompaction(err error) { + slog.Error("Parquet compaction is unresolved; Fanout will not start", + "err", err, + "marker", filepath.Join(r.root, "COMPACTION.json"), + "compaction_staging", filepath.Join(r.root, "compaction"), + "retired_inputs", r.Parquet.BatchesDir(), + "preserved_state", "the live marker is retained and startup cleanup will not run after this failure", + "retry", "clear the underlying cause and start again; recovery re-runs on its own", + "runbook", compactionRunbookURL, + "warning", "deleting the marker on its own is not a rollback; cleanup then treats the retired inputs as reclaimable and removes them") +} + +// cleanupCompactionArtifacts drops staging left after compaction recovery has +// completed and consumed its live marker. +func (r *Repository) cleanupCompactionArtifacts() error { + if err := os.RemoveAll(filepath.Join(r.root, "compaction")); err != nil { + return err + } + if err := os.Remove(filepath.Join(r.root, "COMPACTION.json.tmp")); err != nil && !errors.Is(err, os.ErrNotExist) { + return err + } + return syncDirectory(r.root) +} + +func (r *Repository) Close() error { return r.Parquet.Close() } + +// CleanupParquet removes retired inputs only while no retention or compaction +// transaction can still need them for rollback. +func (r *Repository) CleanupParquet() error { + r.compactionMu.Lock() + defer r.compactionMu.Unlock() + return r.cleanupRetired() +} + +func (r *Repository) cleanupRetired() error { + protected, err := r.protectedRetiredSuffix() + if err != nil { + return err + } + entries, err := os.ReadDir(r.Parquet.BatchesDir()) + if err != nil { + return err + } + removed := false + var cleanupErr error + for _, entry := range entries { + name := entry.Name() + if !entry.IsDir() || strings.HasSuffix(name, telemetry.BatchSuffix) || !strings.Contains(name, ".retired") { + continue + } + // The live marker is the only thing that makes a retired directory + // unreclaimable, and it protects exactly one output's inputs. So + // removing COMPACTION.json without first renaming its + // .retired- directories back to .batch does not + // roll the compaction back — it makes the rows here deletable, and + // this loop deletes them on the next pass. + if protected != "" && strings.HasSuffix(name, protected) { + continue + } + if err := os.RemoveAll(filepath.Join(r.Parquet.BatchesDir(), name)); err != nil { + cleanupErr = errors.Join(cleanupErr, err) + } else { + removed = true + } + } + if removed { + cleanupErr = errors.Join(cleanupErr, syncDirectory(r.Parquet.BatchesDir())) + } + return cleanupErr +} + +// protectedRetiredSuffix names the retired-input set a live compaction marker +// may still need. An unreadable marker fails cleanup closed. +func (r *Repository) protectedRetiredSuffix() (string, error) { + data, err := os.ReadFile(filepath.Join(r.root, "COMPACTION.json")) + if errors.Is(err, os.ErrNotExist) { + return "", nil + } + if err != nil { + return "", err + } + var marker compactionMarker + if err := json.Unmarshal(data, &marker); err != nil { + return "", fmt.Errorf("read live compaction marker: %w", err) + } + if err := validateCompactionMarker(marker); err != nil { + return "", err + } + return ".retired-" + marker.Output.ID, nil +} + +func (r *Repository) Commit(ctx context.Context, batch Batch) error { + normalizeBatch(&batch) + if err := validateBatch(batch); err != nil { + return err + } + metadata := telemetry.BatchMetadata{ + ID: batch.ID, MinIngestedNanos: batchMinIngestedNanos(batch), MaxIngestedNanos: batchMaxIngestedNanos(batch), + } + return r.Parquet.CommitBatch(ctx, metadata, batch.Spans, batch.Logs, batch.Metrics) +} + +func (r *Repository) RowCount() uint64 { return r.Parquet.RowCount() } + +func (r *Repository) PruneParquet(ctx context.Context, publisher ParquetPublisher, cutoff int64, maxBatches int) (int, error) { + r.compactionMu.Lock() + defer r.compactionMu.Unlock() + return r.Parquet.PruneBefore(cutoff, maxBatches, func(prune func(context.Context) error) error { + return publisher.PublishParquet(ctx, prune) + }) +} + +// PruneParquetPass starts bounded publications until the phase budget expires. +// The budget is checked between publications so an atomic swap is never +// canceled halfway through. +func (r *Repository) PruneParquetPass(ctx context.Context, publisher ParquetPublisher, cutoff int64, maxBatches int, budget time.Duration) (int, error) { + if maxBatches <= 0 || budget <= 0 { + return 0, nil + } + deadline := time.Now().Add(budget) + total := 0 + for { + count, err := r.PruneParquet(ctx, publisher, cutoff, maxBatches) + total += count + if err != nil || count < maxBatches || !time.Now().Before(deadline) { + return total, err + } + } +} + +func validateBatch(batch Batch) error { + if batch.ID == "" || strings.ContainsAny(batch.ID, `/\\`) { + return errors.New("telemetry batch requires a safe ID") + } + if rows := batchRows(batch); rows == 0 { + return errors.New("telemetry batch is empty") + } + return nil +} + +func normalizeBatch(batch *Batch) { + ingestedAt := time.Now().UnixNano() + for i := range batch.Spans { + batch.Spans[i].Namespace = telemetry.NormalizeNamespace(batch.Spans[i].Namespace) + if batch.Spans[i].IngestedAt == 0 { + batch.Spans[i].IngestedAt = ingestedAt + } + if batch.Spans[i].StartUnixNanos == 0 { + batch.Spans[i].StartUnixNanos = batch.Spans[i].IngestedAt + } + } + for i := range batch.Logs { + batch.Logs[i].Namespace = telemetry.NormalizeNamespace(batch.Logs[i].Namespace) + if batch.Logs[i].IngestedAt == 0 { + batch.Logs[i].IngestedAt = ingestedAt + } + if batch.Logs[i].EventUnixNanos == 0 { + batch.Logs[i].EventUnixNanos = telemetry.FirstPositiveNanos(batch.Logs[i].TimeUnixNanos, batch.Logs[i].ObservedTimeNanos, batch.Logs[i].IngestedAt) + } + } + for i := range batch.Metrics { + batch.Metrics[i].Namespace = telemetry.NormalizeNamespace(batch.Metrics[i].Namespace) + if batch.Metrics[i].IngestedAt == 0 { + batch.Metrics[i].IngestedAt = ingestedAt + } + if batch.Metrics[i].EventUnixNanos == 0 { + batch.Metrics[i].EventUnixNanos = telemetry.FirstPositiveNanos(batch.Metrics[i].TimeUnixNanos, batch.Metrics[i].IngestedAt) + } + } +} + +func batchMaxIngestedNanos(batch Batch) int64 { + var value int64 + for _, row := range batch.Spans { + value = max(value, row.IngestedAt) + } + for _, row := range batch.Logs { + value = max(value, row.IngestedAt) + } + for _, row := range batch.Metrics { + value = max(value, row.IngestedAt) + } + return value +} + +func batchMinIngestedNanos(batch Batch) int64 { + value := int64(math.MaxInt64) + include := func(candidate int64) { + if candidate > 0 { + value = min(value, candidate) + } + } + for _, row := range batch.Spans { + include(row.IngestedAt) + } + for _, row := range batch.Logs { + include(row.IngestedAt) + } + for _, row := range batch.Metrics { + include(row.IngestedAt) + } + if value == math.MaxInt64 { + return 0 + } + return value +} diff --git a/internal/telemetry/store/repository_test.go b/internal/telemetry/store/repository_test.go new file mode 100644 index 00000000..4a5bf334 --- /dev/null +++ b/internal/telemetry/store/repository_test.go @@ -0,0 +1,900 @@ +package store + +import ( + "context" + "database/sql" + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + "time" + + _ "github.com/duckdb/duckdb-go/v2" + "github.com/labstack/fanout/internal/telemetry" +) + +type testParquetCompactor struct { + publishErr error + afterSwap func() error +} + +type testParquetPublisherFunc func(context.Context, func(context.Context) error) error + +func (f testParquetPublisherFunc) PublishParquet(ctx context.Context, publish func(context.Context) error) error { + return f(ctx, publish) +} + +func (c *testParquetCompactor) PublishParquet(ctx context.Context, publish func(context.Context) error) error { + if c.publishErr != nil { + return c.publishErr + } + if err := publish(ctx); err != nil { + return err + } + if c.afterSwap != nil { + return c.afterSwap() + } + return nil +} + +func sqlQuote(value string) string { return "'" + strings.ReplaceAll(value, "'", "''") + "'" } + +func testBatch() Batch { + return Batch{ + ID: "batch-test", + Spans: []telemetry.Span{{ + TraceID: "trace-1", SpanID: "span-1", ServiceName: "api", Name: "GET /", + StartUnixNanos: 100, EndUnixNanos: 200, DurationMS: 0.0001, StatusCode: "OK", IngestedAt: 300, + }}, + Logs: []telemetry.Log{{TimeUnixNanos: 110, Severity: "INFO", Body: "ready", ServiceName: "api", TraceID: "trace-1", IngestedAt: 300}}, + Metrics: []telemetry.Metric{{TimeUnixNanos: 120, Name: "requests", Type: "sum", ServiceName: "api", Value: 1, IngestedAt: 300}}, + } +} + +func TestRepositoryCommitIsIdempotentDurableAndQueryable(t *testing.T) { + dir := t.TempDir() + repository, err := Open(dir) + if err != nil { + t.Fatal(err) + } + batch := testBatch() + if err := repository.Commit(context.Background(), batch); err != nil { + t.Fatal(err) + } + if err := repository.Commit(context.Background(), batch); err != nil { + t.Fatalf("idempotent commit: %v", err) + } + if got := repository.RowCount(); got != 3 { + t.Fatalf("rows = %d, want 3", got) + } + if err := repository.Close(); err != nil { + t.Fatal(err) + } + + reopened, err := Open(dir) + if err != nil { + t.Fatal(err) + } + defer reopened.Close() + if got := reopened.RowCount(); got != 3 { + t.Fatalf("reopened rows = %d, want 3", got) + } + spans, err := traceAll(reopened, "trace-1") + if err != nil || len(spans) != 1 || spans[0].Namespace != "default" { + t.Fatalf("reopened trace = %#v, %v", spans, err) + } + + db := openTestDuckDB(t) + defer db.Close() + for _, signal := range []string{"spans", "logs", "metrics"} { + var count int + if err := db.QueryRowContext(context.Background(), "SELECT count(*) FROM read_parquet(?)", reopened.Parquet.Pattern(signal)).Scan(&count); err != nil { + t.Fatalf("query %s: %v", signal, err) + } + if count != 1 { + t.Fatalf("%s rows = %d, want 1", signal, count) + } + } +} + +func TestRepositoryCommitsOversizedRequestAsOneBatch(t *testing.T) { + repository, err := Open(t.TempDir()) + if err != nil { + t.Fatal(err) + } + defer repository.Close() + spans := make([]telemetry.Span, maxGroupBatchRows+1) + for i := range spans { + spans[i] = telemetry.Span{ + TraceID: "trace", SpanID: fmt.Sprintf("span-%d", i), + StartUnixNanos: int64(i + 1), EndUnixNanos: int64(i + 2), IngestedAt: 1, + } + } + if err := repository.Commit(context.Background(), Batch{ID: "oversized", Spans: spans}); err != nil { + t.Fatal(err) + } + metadata := repository.Parquet.BatchMetadata() + if len(metadata) != 1 || metadata[0].Spans != len(spans) { + t.Fatalf("oversized batch metadata = %#v", metadata) + } +} + +func TestRepositoryHasOneAuthoritativeStorageLayout(t *testing.T) { + dir := t.TempDir() + repository, err := Open(dir) + if err != nil { + t.Fatal(err) + } + defer repository.Close() + if err := repository.Commit(context.Background(), testBatch()); err != nil { + t.Fatal(err) + } + for _, removed := range []string{"wal", "hot", "MANIFEST.json", "MANIFEST.log", "ducklake.sqlite"} { + if _, err := os.Stat(filepath.Join(dir, removed)); !os.IsNotExist(err) { + t.Fatalf("removed storage artifact %s exists: %v", removed, err) + } + } + entries, err := os.ReadDir(repository.Parquet.BatchPath("batch-test")) + if err != nil { + t.Fatal(err) + } + if len(entries) != 5 { + t.Fatalf("batch contains %d files, want Parquet signals, trace index, and metadata", len(entries)) + } +} + +func TestRepositoryCleansUnpublishedCompactionArtifacts(t *testing.T) { + dir := t.TempDir() + staging := filepath.Join(dir, "compaction", "orphan") + if err := os.MkdirAll(staging, 0o755); err != nil { + t.Fatal(err) + } + temporaryMarker := filepath.Join(dir, "COMPACTION.json.tmp") + if err := os.WriteFile(temporaryMarker, []byte("partial"), 0o600); err != nil { + t.Fatal(err) + } + repository, err := Open(dir) + if err != nil { + t.Fatal(err) + } + defer repository.Close() + for _, path := range []string{filepath.Join(dir, "compaction"), temporaryMarker} { + if _, err := os.Stat(path); !os.IsNotExist(err) { + t.Fatalf("unpublished compaction artifact remains at %s: %v", path, err) + } + } +} + +func TestRepositoryPrunesOnlyExpiredBatches(t *testing.T) { + repository, err := Open(t.TempDir()) + if err != nil { + t.Fatal(err) + } + defer repository.Close() + old := testBatch() + old.ID = "old" + old.Spans[0].StartUnixNanos, old.Spans[0].IngestedAt = 100, 100 + old.Logs[0].TimeUnixNanos, old.Logs[0].IngestedAt = 100, 100 + old.Metrics[0].TimeUnixNanos, old.Metrics[0].IngestedAt = 100, 100 + newer := testBatch() + newer.ID = "new" + newer.Spans[0].StartUnixNanos, newer.Spans[0].IngestedAt = 1_000, 1_000 + newer.Logs[0].TimeUnixNanos, newer.Logs[0].IngestedAt = 1_000, 1_000 + newer.Metrics[0].TimeUnixNanos, newer.Metrics[0].IngestedAt = 1_000, 1_000 + if err := repository.Commit(context.Background(), old); err != nil { + t.Fatal(err) + } + if err := repository.Commit(context.Background(), newer); err != nil { + t.Fatal(err) + } + publisher := &testParquetCompactor{afterSwap: func() error { + if repository.compactionMu.TryLock() { + repository.compactionMu.Unlock() + return errors.New("retention published without holding compaction lock") + } + retired, err := filepath.Glob(filepath.Join(repository.Parquet.BatchesDir(), "*.retired")) + if err != nil { + return err + } + if len(retired) != 1 { + return fmt.Errorf("retired directories during publication = %d, want 1", len(retired)) + } + return nil + }} + removed, err := repository.PruneParquet(context.Background(), publisher, 500, 64) + if err != nil || removed != 1 { + t.Fatalf("prune = %d, %v", removed, err) + } + if _, err := os.Stat(repository.Parquet.BatchPath("old")); !os.IsNotExist(err) { + t.Fatalf("expired batch remains: %v", err) + } + if _, err := os.Stat(repository.Parquet.BatchPath("new")); err != nil { + t.Fatalf("current batch missing: %v", err) + } + if got := repository.RowCount(); got != 3 { + t.Fatalf("retained rows = %d, want 3", got) + } +} + +func TestRepositoryDiscardsRecoverableMarkerWithoutOutput(t *testing.T) { + dir := t.TempDir() + repository, err := Open(dir) + if err != nil { + t.Fatal(err) + } + batch := testBatch() + batch.ID = "intact-input" + if err := repository.Commit(context.Background(), batch); err != nil { + t.Fatal(err) + } + marker := compactionMarker{ + Output: telemetry.BatchMetadata{ID: "missing-output", Generation: 1}, + Inputs: []string{batch.ID}, + } + data, err := json.Marshal(marker) + if err != nil { + t.Fatal(err) + } + if err := writeDurableFile(filepath.Join(dir, "COMPACTION.json"), data); err != nil { + t.Fatal(err) + } + retired := filepath.Join(repository.Parquet.BatchesDir(), batch.ID+".retired-"+marker.Output.ID) + if err := os.Rename(repository.Parquet.BatchPath(batch.ID), retired); err != nil { + t.Fatal(err) + } + if err := repository.Close(); err != nil { + t.Fatal(err) + } + + recovered, err := Open(dir) + if err != nil { + t.Fatal(err) + } + defer recovered.Close() + if got := recovered.RowCount(); got != 3 { + t.Fatalf("rows after marker recovery = %d, want 3", got) + } + if _, err := os.Stat(filepath.Join(dir, "COMPACTION.json")); !os.IsNotExist(err) { + t.Fatalf("recoverable marker remains: %v", err) + } +} + +func TestRepositoryRestoresInputsThroughPublicationGate(t *testing.T) { + dir := t.TempDir() + repository, err := Open(dir) + if err != nil { + t.Fatal(err) + } + defer repository.Close() + batch := testBatch() + batch.ID = "retired-input" + if err := repository.Commit(context.Background(), batch); err != nil { + t.Fatal(err) + } + marker := compactionMarker{ + Output: telemetry.BatchMetadata{ID: "missing-output", Generation: 1}, + Inputs: []string{batch.ID}, + } + data, err := json.Marshal(marker) + if err != nil { + t.Fatal(err) + } + if err := writeDurableFile(filepath.Join(dir, "COMPACTION.json"), data); err != nil { + t.Fatal(err) + } + retired := filepath.Join(repository.Parquet.BatchesDir(), batch.ID+".retired-"+marker.Output.ID) + if err := os.Rename(repository.Parquet.BatchPath(batch.ID), retired); err != nil { + t.Fatal(err) + } + + entered := make(chan struct{}) + release := make(chan struct{}) + publisher := testParquetPublisherFunc(func(ctx context.Context, publish func(context.Context) error) error { + close(entered) + <-release + return publish(ctx) + }) + done := make(chan error, 1) + go func() { done <- repository.RecoverParquet(context.Background(), publisher) }() + select { + case <-entered: + case <-time.After(time.Second): + t.Fatal("recovery did not enter the publication gate") + } + if _, err := os.Stat(repository.Parquet.BatchPath(batch.ID)); !os.IsNotExist(err) { + t.Fatalf("input became visible before publication: %v", err) + } + close(release) + if err := <-done; err != nil { + t.Fatal(err) + } + if _, err := os.Stat(repository.Parquet.BatchPath(batch.ID)); err != nil { + t.Fatalf("input was not restored after publication: %v", err) + } +} + +func TestRepositoryPrunePassDrainsWithinBudgetAndOldestFirst(t *testing.T) { + repository, err := Open(t.TempDir()) + if err != nil { + t.Fatal(err) + } + defer repository.Close() + for i := range 5 { + batch := testBatch() + batch.ID = fmt.Sprintf("expired-%d", i) + batch.Spans[0].IngestedAt = int64(i + 1) + batch.Logs[0].IngestedAt = int64(i + 1) + batch.Metrics[0].IngestedAt = int64(i + 1) + if err := repository.Commit(context.Background(), batch); err != nil { + t.Fatal(err) + } + } + publications := 0 + publisher := &testParquetCompactor{afterSwap: func() error { + publications++ + return nil + }} + removed, err := repository.PruneParquetPass(context.Background(), publisher, 10, 2, time.Second) + if err != nil { + t.Fatal(err) + } + if removed != 5 || publications != 3 { + t.Fatalf("removed=%d publications=%d, want 5 across 3 bounded swaps", removed, publications) + } + metadata := repository.Parquet.BatchMetadata() + if len(metadata) != 0 { + t.Fatalf("remaining batches = %#v, want the backlog drained", metadata) + } +} + +func TestRepositoryPrunePassFinishesPublicationAfterBudget(t *testing.T) { + repository, err := Open(t.TempDir()) + if err != nil { + t.Fatal(err) + } + defer repository.Close() + for i := range 4 { + batch := testBatch() + batch.ID = fmt.Sprintf("expired-%d", i) + batch.Spans[0].IngestedAt = int64(i + 1) + batch.Logs[0].IngestedAt = int64(i + 1) + batch.Metrics[0].IngestedAt = int64(i + 1) + if err := repository.Commit(context.Background(), batch); err != nil { + t.Fatal(err) + } + } + publications := 0 + publisher := &testParquetCompactor{afterSwap: func() error { + publications++ + time.Sleep(10 * time.Millisecond) + return nil + }} + removed, err := repository.PruneParquetPass(context.Background(), publisher, 10, 2, time.Millisecond) + if err != nil { + t.Fatal(err) + } + if removed != 2 || publications != 1 { + t.Fatalf("removed=%d publications=%d, want the in-flight 2-batch publication completed once", removed, publications) + } +} + +func TestRepositoryCleanupOwnsRetiredDirectories(t *testing.T) { + repository, err := Open(t.TempDir()) + if err != nil { + t.Fatal(err) + } + defer repository.Close() + retired := filepath.Join(repository.Parquet.BatchesDir(), "old.retired-compacted") + if err := os.Mkdir(retired, 0o755); err != nil { + t.Fatal(err) + } + if err := repository.Commit(context.Background(), Batch{ID: "contains.retired", Spans: []telemetry.Span{{TraceID: "trace"}}}); err != nil { + t.Fatal(err) + } + if err := repository.CleanupParquet(); err != nil { + t.Fatal(err) + } + if _, err := os.Stat(retired); !os.IsNotExist(err) { + t.Fatalf("retired directory remains: %v", err) + } + if _, err := os.Stat(repository.Parquet.BatchPath("contains.retired")); err != nil { + t.Fatalf("published batch with retired in its ID was removed: %v", err) + } +} + +func TestRepositoryRetentionUsesIngestTimeNotEventTime(t *testing.T) { + repository, err := Open(t.TempDir()) + if err != nil { + t.Fatal(err) + } + defer repository.Close() + batch := testBatch() + batch.ID = "future-events" + batch.Spans[0].StartUnixNanos = 1 << 62 + batch.Logs[0].TimeUnixNanos = 1 << 62 + batch.Metrics[0].TimeUnixNanos = 1 << 62 + batch.Spans[0].IngestedAt = 100 + batch.Logs[0].IngestedAt = 100 + batch.Metrics[0].IngestedAt = 100 + if err := repository.Commit(context.Background(), batch); err != nil { + t.Fatal(err) + } + removed, err := repository.PruneParquet(context.Background(), &testParquetCompactor{}, 500, 64) + if err != nil || removed != 1 { + t.Fatalf("prune future-dated events = %d, %v; want one batch expired by ingest time", removed, err) + } +} + +func TestRepositoryCompactsParquetWithoutChangingRows(t *testing.T) { + dir := t.TempDir() + repository, err := Open(dir) + if err != nil { + t.Fatal(err) + } + defer repository.Close() + for i := range minCompactionInputs { + batch := testBatch() + batch.ID = fmt.Sprintf("batch-%d", i) + batch.Spans[0].SpanID = fmt.Sprintf("span-%d", i) + batch.Spans[0].StartUnixNanos = int64(100 + i) + if err := repository.Commit(context.Background(), batch); err != nil { + t.Fatal(err) + } + } + db := openTestDuckDB(t) + defer db.Close() + compactor := &testParquetCompactor{afterSwap: func() error { + retired, err := filepath.Glob(filepath.Join(repository.Parquet.BatchesDir(), "*.retired-*")) + if err != nil { + return err + } + if len(retired) != minCompactionInputs { + return fmt.Errorf("retired compaction inputs during publication = %d, want %d", len(retired), minCompactionInputs) + } + return nil + }} + compactCtx, cancelCompact := context.WithTimeout(context.Background(), 5*time.Second) + defer cancelCompact() + compacted, err := repository.CompactParquet(compactCtx, compactor, minCompactionInputs) + if err != nil { + t.Fatal(err) + } + if compacted != minCompactionInputs { + t.Fatalf("compacted inputs = %d", compacted) + } + metadata := repository.Parquet.BatchMetadata() + if len(metadata) != 1 || metadata[0].Generation != 1 || metadata[0].Spans != minCompactionInputs || metadata[0].Logs != minCompactionInputs || metadata[0].Metrics != minCompactionInputs { + t.Fatalf("compacted metadata = %#v", metadata) + } + if got := repository.RowCount(); got != 3*minCompactionInputs { + t.Fatalf("compacted rows = %d", got) + } + spans, err := traceAll(repository, "trace-1") + if err != nil || len(spans) != minCompactionInputs { + t.Fatalf("compacted trace spans = %d, %v", len(spans), err) + } + for _, signal := range []string{"spans", "logs", "metrics"} { + var count int + if err := db.QueryRowContext(context.Background(), "SELECT count(*) FROM read_parquet(?)", repository.Parquet.Pattern(signal)).Scan(&count); err != nil { + t.Fatal(err) + } + if count != minCompactionInputs { + t.Fatalf("compacted %s rows = %d", signal, count) + } + } + if _, err := os.Stat(filepath.Join(dir, "COMPACTION.json")); !os.IsNotExist(err) { + t.Fatalf("completed compaction marker remains: %v", err) + } +} + +func TestSelectCompactionBatchesRequiresFullSmallFileGroup(t *testing.T) { + const groupSize = 16 + batches := make([]telemetry.BatchMetadata, groupSize) + for i := range batches { + batches[i] = telemetry.BatchMetadata{ID: fmt.Sprintf("batch-%d", i), MaxIngestedNanos: 1, Spans: 1} + } + if selected := selectCompactionBatches(batches[:groupSize-1], groupSize); len(selected) != 0 { + t.Fatalf("selected partial group of %d batches", len(selected)) + } + if selected := selectCompactionBatches(batches, groupSize); len(selected) != groupSize { + t.Fatalf("selected full group = %d, want %d", len(selected), groupSize) + } +} + +func TestSelectCompactionBatchesBuildsRowBoundedGroup(t *testing.T) { + const maxBatches = 16 + batches := make([]telemetry.BatchMetadata, maxBatches) + for i := range batches { + batches[i] = telemetry.BatchMetadata{ + ID: fmt.Sprintf("batch-%d", i), MaxIngestedNanos: 1, Generation: 2, + Spans: 2_000_000, + } + } + selected := selectCompactionBatches(batches, maxBatches) + if len(selected) != 12 { + t.Fatalf("selected %d batches, want 12 below row limit", len(selected)) + } + var rows int + for _, batch := range selected { + rows += batch.Spans + batch.Logs + batch.Metrics + } + if rows > maxCompactionRows { + t.Fatalf("selected %d rows above limit %d", rows, maxCompactionRows) + } +} + +func TestSelectCompactionBatchesCombinesSaturatedLargeFiles(t *testing.T) { + const maxBatches = 16 + batches := make([]telemetry.BatchMetadata, 4) + for i := range batches { + batches[i] = telemetry.BatchMetadata{ + ID: fmt.Sprintf("large-%d", i), MaxIngestedNanos: 1, Generation: 3, + Spans: 6_000_000, + } + } + selected := selectCompactionBatches(batches, maxBatches) + if len(selected) != len(batches) { + t.Fatalf("selected %d saturated large batches, want %d", len(selected), len(batches)) + } +} + +func TestSelectCompactionBatchesSkipsUncompactableOlderGroup(t *testing.T) { + batches := make([]telemetry.BatchMetadata, 2*minCompactionInputs) + for i := range minCompactionInputs { + batches[i] = telemetry.BatchMetadata{ + ID: fmt.Sprintf("oversized-%d", i), MaxIngestedNanos: 1, + Spans: maxCompactionRows + 1, + } + } + for i := minCompactionInputs; i < len(batches); i++ { + batches[i] = telemetry.BatchMetadata{ + ID: fmt.Sprintf("newer-%d", i), MaxIngestedNanos: int64(24*time.Hour) + 1, + Spans: 1, + } + } + selected := selectCompactionBatches(batches, minCompactionInputs) + if len(selected) != minCompactionInputs || !strings.HasPrefix(selected[0].ID, "newer-") { + t.Fatalf("selection did not skip uncompactable older group: %#v", selected) + } +} + +func TestRepositoryRecoversPendingCompactionWithoutRestart(t *testing.T) { + dir := t.TempDir() + repository, err := Open(dir) + if err != nil { + t.Fatal(err) + } + defer repository.Close() + for i := range minCompactionInputs { + batch := testBatch() + batch.ID = fmt.Sprintf("pending-%d", i) + batch.Spans[0].SpanID = fmt.Sprintf("span-%d", i) + if err := repository.Commit(context.Background(), batch); err != nil { + t.Fatal(err) + } + } + compactor := &testParquetCompactor{publishErr: errors.New("publication unavailable")} + if _, err := repository.CompactParquet(context.Background(), compactor, minCompactionInputs); err == nil { + t.Fatal("compaction succeeded despite publication failure") + } + if _, err := os.Stat(filepath.Join(dir, "COMPACTION.json")); err != nil { + t.Fatalf("pending marker: %v", err) + } + compactor.publishErr = nil + if compacted, err := repository.CompactParquet(context.Background(), compactor, minCompactionInputs); err != nil || compacted != 0 { + t.Fatalf("resume compaction = %d, %v", compacted, err) + } + if got := repository.Parquet.BatchMetadata(); len(got) != 1 || got[0].Generation != 1 { + t.Fatalf("recovered metadata = %#v", got) + } + if _, err := os.Stat(filepath.Join(dir, "COMPACTION.json")); !os.IsNotExist(err) { + t.Fatalf("recovered compaction marker remains: %v", err) + } +} + +func TestRepositoryRecoversInterruptedCompactionSwap(t *testing.T) { + dir := t.TempDir() + repository, err := Open(dir) + if err != nil { + t.Fatal(err) + } + for i := range minCompactionInputs { + batch := testBatch() + batch.ID = fmt.Sprintf("recover-%d", i) + batch.Spans[0].SpanID = fmt.Sprintf("span-%d", i) + if err := repository.Commit(context.Background(), batch); err != nil { + t.Fatal(err) + } + } + db := openTestDuckDB(t) + defer db.Close() + output := telemetry.BatchMetadata{ + ID: "compact-recovery", MinIngestedNanos: 100, MaxIngestedNanos: 300, Generation: 1, + Spans: minCompactionInputs, Logs: minCompactionInputs, Metrics: minCompactionInputs, + } + stage := repository.compactionStage(output.ID) + if err := os.MkdirAll(stage, 0o755); err != nil { + t.Fatal(err) + } + for _, signal := range parquetSignals { + query := fmt.Sprintf("SELECT * FROM read_parquet(%s)", sqlQuote(repository.Parquet.Pattern(signal))) + if signal == "spans" { + query += " ORDER BY _trace_hash, start_unix_nano, span_id" + } + stmt := fmt.Sprintf("COPY (%s) TO %s (FORMAT PARQUET, COMPRESSION ZSTD)", query, sqlQuote(filepath.Join(stage, signal+".parquet"))) + if _, err := db.Exec(stmt); err != nil { + t.Fatal(err) + } + } + if err := repository.Parquet.PrepareReplacement(stage, output); err != nil { + t.Fatal(err) + } + inputs := make([]string, minCompactionInputs) + for i := range inputs { + inputs[i] = fmt.Sprintf("recover-%d", i) + } + marker := compactionMarker{Output: output, Inputs: inputs} + data, err := json.Marshal(marker) + if err != nil { + t.Fatal(err) + } + if err := writeDurableFile(filepath.Join(dir, "COMPACTION.json"), data); err != nil { + t.Fatal(err) + } + // Reproduce a crash halfway through retiring inputs. Startup must finish + // the intended publication, not expose a half-old/half-new file set. + for _, id := range inputs[:4] { + if err := os.Rename(repository.Parquet.BatchPath(id), filepath.Join(repository.Parquet.BatchesDir(), id+".retired-"+output.ID)); err != nil { + t.Fatal(err) + } + } + + recovered, err := Open(dir) + if err != nil { + t.Fatal(err) + } + defer recovered.Close() + metadata := recovered.Parquet.BatchMetadata() + if len(metadata) != 1 || metadata[0].ID != output.ID || recovered.RowCount() != 3*minCompactionInputs { + t.Fatalf("recovered metadata = %#v rows=%d", metadata, recovered.RowCount()) + } + spans, err := traceAll(recovered, "trace-1") + if err != nil || len(spans) != minCompactionInputs { + t.Fatalf("recovered trace spans = %d, %v", len(spans), err) + } + if retired, err := filepath.Glob(filepath.Join(recovered.Parquet.BatchesDir(), "*.retired*")); err != nil || len(retired) != 0 { + t.Fatalf("retired inputs after recovery = %v, %v", retired, err) + } + if _, err := os.Stat(filepath.Join(dir, "COMPACTION.json")); !os.IsNotExist(err) { + t.Fatalf("compaction marker remains after recovery: %v", err) + } +} + +func openTestDuckDB(t *testing.T) *sql.DB { + t.Helper() + db, err := sql.Open("duckdb", "") + if err != nil { + t.Fatal(err) + } + return db +} + +func traceAll(repository *Repository, traceID string) ([]telemetry.IndexedSpan, error) { + return repository.Parquet.Trace(context.Background(), telemetry.TraceQuery{ + TraceID: traceID, StartNanos: -1 << 63, EndNanos: 1<<63 - 1, Limit: 500, + }) +} + +// seedFailedCompaction leaves a live COMPACTION.json whose recovery always +// fails, which is the state that used to latch maintenance off permanently. +func seedFailedCompaction(t *testing.T, dir string, repository *Repository) { + t.Helper() + for i := range minCompactionInputs { + batch := testBatch() + batch.ID = fmt.Sprintf("latched-%d", i) + batch.Spans[0].SpanID = fmt.Sprintf("span-%d", i) + if err := repository.Commit(context.Background(), batch); err != nil { + t.Fatal(err) + } + } + compactor := &testParquetCompactor{publishErr: errors.New("publication unavailable")} + if _, err := repository.CompactParquet(context.Background(), compactor, minCompactionInputs); err == nil { + t.Fatal("compaction succeeded despite publication failure") + } + if _, err := os.Stat(filepath.Join(dir, "COMPACTION.json")); err != nil { + t.Fatalf("expected a live marker: %v", err) + } +} + +// TestRepositoryRecoveryFailureStaysLive pins fail-closed recovery. The marker +// and rollback set remain authoritative until recovery succeeds; silently +// bypassing them can make old inputs eligible for deletion. +func TestRepositoryRecoveryFailureStaysLive(t *testing.T) { + dir := t.TempDir() + repository, err := Open(dir) + if err != nil { + t.Fatal(err) + } + defer repository.Close() + seedFailedCompaction(t, dir, repository) + + failing := testParquetPublisherFunc(func(context.Context, func(context.Context) error) error { + return errors.New("publication unavailable") + }) + for attempt := 1; attempt <= 5; attempt++ { + if err := repository.RecoverParquet(context.Background(), failing); err == nil { + t.Fatalf("attempt %d: recovery reported success", attempt) + } + if _, err := os.Stat(filepath.Join(dir, "COMPACTION.json")); err != nil { + t.Fatalf("attempt %d: live marker was removed: %v", attempt, err) + } + } + if _, err := os.Stat(filepath.Join(dir, "COMPACTION.json.failed")); !os.IsNotExist(err) { + t.Fatalf("recovery created a fallback marker: %v", err) + } +} + +// TestRepositoryCleanupPreservesMarkerRollbackSet pins that cleanup never +// deletes the retired inputs a pending marker still needs. Deleting one is +// unrecoverable: the input is gone and its rows were never published under the +// replacement, so the rows exist in no queryable batch at all. +func TestRepositoryCleanupPreservesMarkerRollbackSet(t *testing.T) { + dir := t.TempDir() + repository, err := Open(dir) + if err != nil { + t.Fatal(err) + } + defer repository.Close() + + marker := compactionMarker{ + Output: telemetry.BatchMetadata{ID: "compact-live", Generation: 1}, + Inputs: []string{"input-a"}, + } + data, err := json.Marshal(marker) + if err != nil { + t.Fatal(err) + } + if err := writeDurableFile(filepath.Join(dir, "COMPACTION.json"), data); err != nil { + t.Fatal(err) + } + batches := repository.Parquet.BatchesDir() + needed := filepath.Join(batches, "input-a.retired-compact-live") + stale := filepath.Join(batches, "input-b.retired") + for _, path := range []string{needed, stale} { + if err := os.MkdirAll(path, 0o755); err != nil { + t.Fatal(err) + } + } + + if err := repository.CleanupParquet(); err != nil { + t.Fatal(err) + } + if _, err := os.Stat(needed); err != nil { + t.Fatalf("cleanup deleted the rollback set of a live marker: %v", err) + } + if _, err := os.Stat(stale); !os.IsNotExist(err) { + t.Fatalf("cleanup left an unreferenced retired directory: %v", err) + } +} + +func TestRepositoryOpenFailsClosedOnUnrecoverableMarker(t *testing.T) { + dir := t.TempDir() + repository, err := Open(dir) + if err != nil { + t.Fatal(err) + } + seedFailedCompaction(t, dir, repository) + if err := repository.Close(); err != nil { + t.Fatal(err) + } + // Corrupt the staged output so recovery cannot complete or roll back. + stage := filepath.Join(dir, "compaction") + entries, err := os.ReadDir(stage) + if err != nil { + t.Fatal(err) + } + for _, entry := range entries { + if err := os.WriteFile(filepath.Join(stage, entry.Name(), "metadata.json"), []byte("{"), 0o644); err != nil { + t.Fatal(err) + } + } + + if reopened, err := Open(dir); err == nil { + _ = reopened.Close() + t.Fatal("Open succeeded with an unrecoverable marker") + } + if _, err := os.Stat(filepath.Join(dir, "COMPACTION.json")); err != nil { + t.Fatalf("Open removed the live marker: %v", err) + } + if _, err := os.Stat(stage); err != nil { + t.Fatalf("Open destroyed the staged output: %v", err) + } +} + +func TestRepositoryOpenFailsOnUnreadableMarker(t *testing.T) { + dir := t.TempDir() + repository, err := Open(dir) + if err != nil { + t.Fatal(err) + } + if err := repository.Close(); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, "COMPACTION.json"), []byte("not json"), 0o644); err != nil { + t.Fatal(err) + } + for boot := 1; boot <= 3; boot++ { + if reopened, err := Open(dir); err == nil { + _ = reopened.Close() + t.Fatalf("boot %d accepted an unreadable marker", boot) + } + if _, err := os.Stat(filepath.Join(dir, "COMPACTION.json")); err != nil { + t.Fatalf("boot %d removed the unreadable marker: %v", boot, err) + } + } +} + +func TestRepositoryOpenRejectsUnsafeCompactionMarker(t *testing.T) { + dir := t.TempDir() + repository, err := Open(dir) + if err != nil { + t.Fatal(err) + } + if err := repository.Close(); err != nil { + t.Fatal(err) + } + marker := compactionMarker{ + Output: telemetry.BatchMetadata{ID: "../outside"}, + Inputs: []string{"input"}, + } + data, err := json.Marshal(marker) + if err != nil { + t.Fatal(err) + } + markerPath := filepath.Join(dir, "COMPACTION.json") + if err := os.WriteFile(markerPath, data, 0o600); err != nil { + t.Fatal(err) + } + if reopened, err := Open(dir); err == nil { + _ = reopened.Close() + t.Fatal("Open accepted a marker-controlled path outside storage") + } + if _, err := os.Stat(markerPath); err != nil { + t.Fatalf("Open removed the invalid live marker: %v", err) + } +} + +// TestUnresolvedCompactionRunbookExists keeps the startup log's runbook link +// pointing at a section that exists. The log stopped restating the rollback +// procedure because maintaining it in two places produced a string of +// revisions that corrected one copy and left the other unsafe; a link only +// removes that risk while it resolves, so the anchor is checked here rather +// than trusted. +func TestUnresolvedCompactionRunbookExists(t *testing.T) { + const guide = "../../../site/src/content/docs/guides/troubleshoot.mdx" + data, err := os.ReadFile(guide) + if err != nil { + t.Fatalf("read the runbook the startup log links to: %v", err) + } + _, anchor, found := strings.Cut(compactionRunbookURL, "#") + if !found { + t.Fatalf("runbook URL %q has no anchor", compactionRunbookURL) + } + var headings []string + for _, line := range strings.Split(string(data), "\n") { + title, isHeading := strings.CutPrefix(line, "## ") + if !isHeading { + continue + } + slug := strings.ToLower(strings.TrimSpace(title)) + slug = strings.Join(strings.Fields(slug), "-") + headings = append(headings, slug) + if slug == anchor { + return + } + } + t.Fatalf("runbook anchor %q is not a section in %s; sections are %v", anchor, guide, headings) +} diff --git a/internal/telemetry/store/writer.go b/internal/telemetry/store/writer.go new file mode 100644 index 00000000..6c84252d --- /dev/null +++ b/internal/telemetry/store/writer.go @@ -0,0 +1,328 @@ +package store + +import ( + "context" + "errors" + "fmt" + "log/slog" + "runtime" + "sync" + "time" + + "github.com/google/uuid" + "github.com/labstack/fanout/internal/metrics" +) + +const ( + commitQueueDepth = maxCommitWorkers + commitRetryLimit = 5 + groupAdmissionWindow = 20 * time.Millisecond + maxAdmissionRequests = 512 + maxGroupBatchRows = 50_000 + maxCommitWorkers = 4 + submissionQueueDepth = 256 + writerShutdownGrace = 30 * time.Second +) + +type batchCommitter interface { + Commit(context.Context, Batch) error +} + +type Writer struct { + repository batchCommitter + batchSize int + retryDelay func(int) time.Duration + groupWindow time.Duration + shutdownGrace time.Duration + done chan struct{} + submissions chan submission +} + +type submission struct { + batch Batch + ack chan error +} + +type commitJob struct { + batches []Batch + acks []chan error +} + +func NewWriter(repository *Repository, batchSize int) *Writer { + return &Writer{ + repository: repository, batchSize: batchSize, groupWindow: groupAdmissionWindow, + done: make(chan struct{}), submissions: make(chan submission, submissionQueueDepth), + } +} + +func (w *Writer) Wait() { <-w.done } + +// Submit returns after every row in the request belongs to a durably published +// atomic Parquet batch directory. Delivery is at least once: callers that retry +// an ambiguous request may create another batch because OTLP has no idempotency +// key that survives across requests. +func (w *Writer) Submit(ctx context.Context, batch Batch) error { + if batchRows(batch) == 0 { + return nil + } + request := submission{batch: batch, ack: make(chan error, 1)} + select { + case w.submissions <- request: + metrics.UpdateQueueDepth("batch", len(w.submissions)) + case <-w.done: + return errors.New("telemetry writer is stopped") + case <-ctx.Done(): + return ctx.Err() + } + select { + case err := <-request.ack: + return err + case <-w.done: + return errors.New("telemetry writer stopped before durable acknowledgement") + case <-ctx.Done(): + return ctx.Err() + } +} + +func (w *Writer) Run(ctx context.Context) error { + defer close(w.done) + defer metrics.UpdateQueueDepth("batch", 0) + jobs := make(chan commitJob, commitQueueDepth) + workerCtx, cancelWorkers := context.WithCancel(context.Background()) + defer cancelWorkers() + var workers sync.WaitGroup + for range min(maxCommitWorkers, max(1, runtime.GOMAXPROCS(0))) { + workers.Add(1) + go func() { + defer workers.Done() + w.commitWorker(workerCtx, jobs) + }() + } + + finish := func(graceful bool) error { + close(jobs) + if !graceful { + cancelWorkers() + } + finished := make(chan struct{}) + go func() { workers.Wait(); close(finished) }() + grace := w.shutdownGrace + if grace <= 0 { + grace = writerShutdownGrace + } + timer := time.NewTimer(grace) + defer timer.Stop() + select { + case <-finished: + case <-timer.C: + cancelWorkers() + <-finished + } + return nil + } + + for { + select { + case request := <-w.submissions: + metrics.UpdateQueueDepth("batch", len(w.submissions)) + if err := w.enqueueSubmissions(ctx, request, jobs); err != nil { + return errors.Join(err, finish(false)) + } + case <-ctx.Done(): + return finish(true) + } + } +} + +func (w *Writer) enqueueSubmissions(ctx context.Context, request submission, out chan<- commitJob) error { + requests := make([]submission, 1, maxAdmissionRequests) + requests[0] = request + rows := batchRows(request.batch) + limit := w.batchLimit() + if rows < limit { + window := w.groupWindow + if window <= 0 { + window = groupAdmissionWindow + } + timer := time.NewTimer(window) + defer timer.Stop() + admit: + for len(requests) < maxAdmissionRequests && rows < limit { + select { + case next := <-w.submissions: + requests = append(requests, next) + rows += batchRows(next.batch) + case <-timer.C: + break admit + case <-ctx.Done(): + return ctx.Err() + } + } + } + for len(requests) < maxAdmissionRequests { + select { + case next := <-w.submissions: + requests = append(requests, next) + default: + goto drained + } + } + +drained: + metrics.UpdateQueueDepth("batch", len(w.submissions)) + for len(requests) > 0 { + firstRows := batchRows(requests[0].batch) + // A full batch, a lone request, or a request that cannot share the + // next batch needs no row copy through the group-commit buffer. One + // oversized request remains one atomic directory; the limit is a + // group-commit target, not a durability boundary. + if firstRows >= limit || len(requests) == 1 || firstRows+batchRows(requests[1].batch) > limit { + direct := requests[0] + requests = requests[1:] + direct.batch.ID = uuid.NewString() + if err := enqueueJob(ctx, out, commitJob{batches: []Batch{direct.batch}, acks: []chan error{direct.ack}}); err != nil { + return err + } + continue + } + + batch := Batch{ID: uuid.NewString()} + group := make([]submission, 0, len(requests)) + rows := 0 + for len(requests) > 0 { + next := requests[0] + nextRows := batchRows(next.batch) + if len(group) > 0 && rows+nextRows > limit { + break + } + requests = requests[1:] + group = append(group, next) + rows += nextRows + batch.Spans = append(batch.Spans, next.batch.Spans...) + batch.Logs = append(batch.Logs, next.batch.Logs...) + batch.Metrics = append(batch.Metrics, next.batch.Metrics...) + if rows >= limit { + break + } + } + acks := make([]chan error, len(group)) + for i := range group { + acks[i] = group[i].ack + } + if err := enqueueJob(ctx, out, commitJob{batches: []Batch{batch}, acks: acks}); err != nil { + return err + } + } + return nil +} + +func enqueueJob(ctx context.Context, out chan<- commitJob, job commitJob) error { + select { + case out <- job: + return nil + case <-ctx.Done(): + return ctx.Err() + } +} + +func (w *Writer) commitWorker(ctx context.Context, jobs <-chan commitJob) { + for { + select { + case <-ctx.Done(): + return + case job, ok := <-jobs: + if !ok { + return + } + if err := w.commitJob(ctx, job); err != nil { + for _, ack := range job.acks { + ack <- err + } + slog.Error("telemetry batch commit exhausted retries", "error", err) + continue + } + for _, batch := range job.batches { + metrics.RecordIngest("spans", len(batch.Spans)) + metrics.RecordIngest("logs", len(batch.Logs)) + metrics.RecordIngest("metrics", len(batch.Metrics)) + } + for _, ack := range job.acks { + ack <- nil + } + } + } +} + +func (w *Writer) commitJob(ctx context.Context, job commitJob) error { + for _, batch := range job.batches { + started := time.Now() + var lastErr error + for attempt := 0; attempt < commitRetryLimit; attempt++ { + if err := w.repository.Commit(ctx, batch); err == nil { + lastErr = nil + break + } else { + lastErr = err + } + metrics.FlushErrors.WithLabelValues("batch").Inc() + slog.Warn("telemetry batch commit failed; retrying", "batch_id", batch.ID, "attempt", attempt+1, "error", lastErr) + if attempt+1 == commitRetryLimit { + break + } + delay := defaultCommitRetryDelay(attempt) + if w.retryDelay != nil { + delay = w.retryDelay(attempt) + } + timer := time.NewTimer(delay) + select { + case <-ctx.Done(): + if !timer.Stop() { + <-timer.C + } + return ctx.Err() + case <-timer.C: + } + } + if lastErr != nil { + metrics.FlushErrors.WithLabelValues("failed").Inc() + recordDroppedRows(batch) + return fmt.Errorf("commit telemetry batch %s after %d attempts: %w", batch.ID, commitRetryLimit, lastErr) + } + recordFlushes(batch, time.Since(started).Seconds()) + } + return nil +} + +func batchRows(batch Batch) int { + return len(batch.Spans) + len(batch.Logs) + len(batch.Metrics) +} + +func (w *Writer) batchLimit() int { + limit := min(w.batchSize, maxGroupBatchRows) + if limit <= 0 { + return maxGroupBatchRows + } + return limit +} + +func recordFlushes(batch Batch, durationSec float64) { + if len(batch.Spans) > 0 { + metrics.RecordFlush("spans", durationSec) + } + if len(batch.Logs) > 0 { + metrics.RecordFlush("logs", durationSec) + } + if len(batch.Metrics) > 0 { + metrics.RecordFlush("metrics", durationSec) + } +} + +func recordDroppedRows(batch Batch) { + metrics.RowsDropped.WithLabelValues("spans").Add(float64(len(batch.Spans))) + metrics.RowsDropped.WithLabelValues("logs").Add(float64(len(batch.Logs))) + metrics.RowsDropped.WithLabelValues("metrics").Add(float64(len(batch.Metrics))) +} + +func defaultCommitRetryDelay(attempt int) time.Duration { + return min(250*time.Millisecond*time.Duration(1< uint64((math.MaxInt64-traceIndexHeaderSize)/traceIndexEntrySize) || + info.Size() != traceIndexHeaderSize+int64(count)*traceIndexEntrySize { + return traceIndex{}, errors.New("invalid trace index length") + } + reader := bufio.NewReaderSize(file, 64<<10) + var encoded [traceIndexEntrySize]byte + var previous traceRange + for i := uint64(0); i < count; i++ { + if _, err := io.ReadFull(reader, encoded[:]); err != nil { + return traceIndex{}, err + } + entry := decodeTraceRange(encoded[:]) + if entry.count == 0 || entry.row > uint64(spanCount) || entry.count > uint64(spanCount)-entry.row { + return traceIndex{}, errors.New("trace index entry exceeds span file") + } + if i == 0 && entry.row != 0 { + return traceIndex{}, errors.New("trace index does not start at row zero") + } + if i > 0 && (previous.hash >= entry.hash || previous.row+previous.count != entry.row) { + return traceIndex{}, errors.New("trace index is not strictly sorted and contiguous") + } + previous = entry + } + if count > 0 { + if previous.row+previous.count != uint64(spanCount) { + return traceIndex{}, fmt.Errorf("trace index covers %d of %d span rows", previous.row+previous.count, spanCount) + } + } else if spanCount != 0 { + return traceIndex{}, errors.New("trace index is empty for a non-empty span file") + } + return traceIndex{path: path, entries: count}, nil +} + +func (index traceIndex) Lookup(hash uint64) (match traceRange, found bool, err error) { + if index.entries == 0 { + return traceRange{}, false, nil + } + file, err := os.Open(index.path) + if err != nil { + return traceRange{}, false, err + } + defer func() { err = errors.Join(err, file.Close()) }() + low, high := uint64(0), index.entries + for low < high { + middle := low + (high-low)/2 + entry, readErr := readTraceRangeAt(file, middle) + if readErr != nil { + return traceRange{}, false, readErr + } + if entry.hash < hash { + low = middle + 1 + } else { + high = middle + } + } + if low == index.entries { + return traceRange{}, false, nil + } + entry, err := readTraceRangeAt(file, low) + if err != nil { + return traceRange{}, false, err + } + return entry, entry.hash == hash, nil +} + +func readTraceRangeAt(file *os.File, position uint64) (traceRange, error) { + var encoded [traceIndexEntrySize]byte + offset := int64(traceIndexHeaderSize) + int64(position)*traceIndexEntrySize + if _, err := file.ReadAt(encoded[:], offset); err != nil { + return traceRange{}, err + } + return decodeTraceRange(encoded[:]), nil +} + +func decodeTraceRange(encoded []byte) traceRange { + return traceRange{ + hash: binary.LittleEndian.Uint64(encoded[0:8]), + row: binary.LittleEndian.Uint64(encoded[8:16]), + count: binary.LittleEndian.Uint64(encoded[16:24]), + } +} diff --git a/internal/telemetry/trace_index_test.go b/internal/telemetry/trace_index_test.go new file mode 100644 index 00000000..002691b8 --- /dev/null +++ b/internal/telemetry/trace_index_test.go @@ -0,0 +1,57 @@ +package telemetry + +import ( + "os" + "path/filepath" + "testing" +) + +func TestTraceIndexLookupKeepsEntriesOnDisk(t *testing.T) { + path := filepath.Join(t.TempDir(), "trace.fidx") + rows := make([]spanParquetRow, 10_000) + for i := range rows { + rows[i].TraceHash = uint64(i * 2) + } + if err := writeTraceIndex(path, rows); err != nil { + t.Fatal(err) + } + index, err := loadTraceIndex(path, len(rows)) + if err != nil { + t.Fatal(err) + } + if index.entries != uint64(len(rows)) || index.path != path { + t.Fatalf("index metadata = %#v", index) + } + for _, hash := range []uint64{0, 9_998, 19_998} { + match, found, err := index.Lookup(hash) + if err != nil || !found || match.hash != hash || match.row != hash/2 || match.count != 1 { + t.Fatalf("lookup(%d) = %#v, %v, %v", hash, match, found, err) + } + } + if _, found, err := index.Lookup(9_999); err != nil || found { + t.Fatalf("missing lookup = %v, %v", found, err) + } +} + +func TestTraceIndexRejectsNoncontiguousEntry(t *testing.T) { + path := filepath.Join(t.TempDir(), "trace.fidx") + rows := []spanParquetRow{{TraceHash: 1}, {TraceHash: 2}} + if err := writeTraceIndex(path, rows); err != nil { + t.Fatal(err) + } + file, err := os.OpenFile(path, os.O_RDWR, 0) + if err != nil { + t.Fatal(err) + } + // The second entry must begin at row one; force it to row two. + if _, err := file.WriteAt([]byte{2}, traceIndexHeaderSize+traceIndexEntrySize+8); err != nil { + file.Close() + t.Fatal(err) + } + if err := file.Close(); err != nil { + t.Fatal(err) + } + if _, err := loadTraceIndex(path, len(rows)); err == nil { + t.Fatal("loadTraceIndex accepted a noncontiguous range") + } +} diff --git a/justfile b/justfile index 90d5e12c..c7734b4a 100644 --- a/justfile +++ b/justfile @@ -184,21 +184,31 @@ ui-check: # and `check`: the SVG is committed, so only someone editing a diagram needs d2 # installed at all. # -# The committed SVG was produced by d2 0.7.1. d2's output changes between minor -# versions, so a different version re-renders every file and produces a large -# diff that looks like a change but is not — check `d2 --version` before -# committing one. +# D2 output changes between minor versions, so the recipe pins the generator +# used for committed SVGs instead of depending on a contributor's installed +# binary. -# Render every d2 diagram to SVG. +# Render every d2 diagram to SVG and publish the same assets on the docs site. diagrams: #!/usr/bin/env bash set -euo pipefail - if ! command -v d2 >/dev/null; then - echo "d2 not found — install with: brew install d2" >&2 - exit 1 - fi for src in docs/diagrams/*.d2; do - d2 "$src" "${src%.d2}.svg" + go run oss.terrastruct.com/d2@v0.7.1 "$src" "${src%.d2}.svg" + done + mkdir -p site/public/diagrams + cp docs/diagrams/*.svg site/public/diagrams/ + +# Fail when the site serves a diagram other than the committed documentation +# copy. Regeneration stays a deliberate writing step. +diagrams-check: + #!/usr/bin/env bash + set -euo pipefail + for src in docs/diagrams/*.svg; do + dst="site/public/diagrams/$(basename "$src")" + if ! cmp -s "$src" "$dst"; then + echo "$dst does not match $src; run 'just diagrams'" >&2 + exit 1 + fi done # ── Documentation site ─────────────────────────────────────────────────────── @@ -233,7 +243,7 @@ site: docs-generate # a gate that rewrites the tree it is inspecting cannot tell you whether the # tree was already right. Use `just site` for the writing path during # development. -site-build: docs-generate-check site-deps +site-build: docs-generate-check diagrams-check site-deps cd site && npm run build # Renders the social preview card from docs/media/social-card.typ into diff --git a/site/public/diagrams/architecture.svg b/site/public/diagrams/architecture.svg new file mode 100644 index 00000000..6f054670 --- /dev/null +++ b/site/public/diagrams/architecture.svg @@ -0,0 +1,121 @@ +Clientsfanout — one Go processStorageAnthropic or OpenAIOTLP collector or SDKBrowserExternal MCP hostHTTP :7520routes + auth middlewareIngestOTLP gRPC :4317 + HTTP :4318Embedded React assetsAgent runtimemodel + tool loopMCP server5 typed + 4 dashboard toolsTyped observability contractTelemetry commit workersatomic Parquet batchesQuery kernelDuckDB + rollupsStorage maintenanceretention + native compactionAlert enginerule evaluation + webhooksParquet batches + trace indexesstorage.data_dir/telemetryQuery catalogstorage.data_dir/queryControl SQLitestorage.data_dir/control serves embedded assetsAG-UI streamin-memory transporttyped HTTP APIevaluates rollupsOTLP/gRPC or OTLP/HTTPHTTPS/mcp — OAuthindexed traces + SQL scansatomic replacementusers, sessions, settings, dashboards, threadsrules and fired alertsHTTPS + + + + + + + + + + + + + + + diff --git a/site/public/diagrams/persistence.svg b/site/public/diagrams/persistence.svg new file mode 100644 index 00000000..e1c636dc --- /dev/null +++ b/site/public/diagrams/persistence.svg @@ -0,0 +1,138 @@ +Authenticated OTLP requestsspans, logs, metricsDurable write pathAuthoritative telemetryimmutable *.batch directoriesOrdinary Parquet files are the source of truthReadersBounded maintenanceRebuildable query statestorage.data_dir/queryDuckDB rollups and temporary spill. Not authoritative telemetry.Bounded group admissionconfigured row target + 20 ms windowUp to four commit workersindependent batches encode in parallelStaging batch directorymetadata + signal Parquet + trace.fidx for spansDurable publicationfsync → atomic rename → fsync parentthe filesystem is the catalogIndexed trace readbinary-search .fidx → Parquet rowsDuckDB SQLfilters, ordering, broad scansDuckDB rollupsrebuildable analytical cacheRetention + compaction admissionbounded batch and row groupsNative Parquet mergeprepared outside the reader gateCrash-safe namespace swapmarker + bounded renames + fsync acknowledge only after publicationOrdinary Parquet files are the source of truth + + + + + + + + + + + + +DuckDB rollups and temporary spill. Not authoritative telemetry. + + + + + + + + + + + + + + + + diff --git a/site/src/content/docs/explanation/performance.mdx b/site/src/content/docs/explanation/performance.mdx index e378124e..a78da39f 100644 --- a/site/src/content/docs/explanation/performance.mdx +++ b/site/src/content/docs/explanation/performance.mdx @@ -40,9 +40,12 @@ Before any figure goes back on this page, it has to ship with: `cmd/bench` is the supported load generator. It sends traces, metrics and logs, ramps to find the ingest boundary, and confirms the result at the rate it found. -It can add authenticated dashboard reads on top, which is the part that matters: -ingest and query contend for the same write gate, so an ingest-only number -overstates what an instance does while anyone is looking at it. +It can add authenticated dashboard reads on top, which is the part that matters. +Parquet ingest and DuckDB reads no longer serialize through one write gate, but +they still compete for the same CPU, memory bandwidth, filesystem cache, and +disk. Maintenance adds native Parquet merge work and short reader-exclusive +namespace changes. An ingest-only number therefore still overstates what an +instance does while anyone is looking at it. Build it, create an ingest token through first-admin setup, then: diff --git a/site/src/content/docs/explanation/storage-model.mdx b/site/src/content/docs/explanation/storage-model.mdx index 1d131d37..3d71ac76 100644 --- a/site/src/content/docs/explanation/storage-model.mdx +++ b/site/src/content/docs/explanation/storage-model.mdx @@ -1,71 +1,85 @@ --- title: How storage works -description: What Fanout writes to disk, how it is queried, and why the data directory is one unit. -summary: Parquet under a DuckLake catalog, queried by DuckDB in-process, with SQLite holding application state. +description: What Fanout writes to disk, how it is queried, and why each component exists. +summary: Atomic Parquet ingestion, indexed trace reads, DuckDB analytics, and SQLite control state. read_when: - You are sizing disk, or deciding what to back up. - - You want to know why a rollup lags the raw telemetry. + - You want to understand ingest durability or query performance. status: preview --- -Telemetry lands on disk as Parquet. A DuckLake catalog tracks those files, and -DuckDB queries them in the same process that wrote them. Application state — -users, sessions, dashboards, alert rules, agent history — lives in a separate -SQLite database. +Fanout uses a small embedded storage stack, not a data lake. Telemetry is retained +as ordinary Parquet files and queried by DuckDB in the same process. Application +state—users, sessions, dashboards, alert rules, and agent history—lives in SQLite. -Under the data directory: - -| Path | Holds | +| Component | Purpose | |---|---| -| `telemetry/parquet` | The telemetry itself | -| `telemetry/ducklake.sqlite` | The catalog tracking those files | -| `query/catalog.duckdb` | Query-side state | -| `control/fanout.sqlite` | Users, sessions, dashboards, alerts, agent history | - -These reference each other, which is why [backup](/guides/back-up-and-restore) -takes the whole directory. A copy of `telemetry/parquet` without its catalog is -a directory of files nothing can find. - -## Why two databases - -They have different shapes. Telemetry is append-heavy, queried analytically over -wide time ranges, and never updated — which is what Parquet and DuckDB are good -at. Application state is small, transactional, and updated constantly, which -they are not. Using one engine for both would make one of the two jobs worse. - -## Writes are batched - -Incoming telemetry is buffered and flushed on `FANOUT_FLUSH_INTERVAL` or when -`FANOUT_FLUSH_BATCH_SIZE` rows accumulate. Nothing is queryable until it is -flushed, so the flush interval is the floor on how quickly new telemetry becomes -visible — and the amount of data at risk if the process dies uncleanly. - -## Small files are the thing to manage - -Frequent flushes produce many small Parquet files, and scan cost tracks file -count as much as it tracks bytes. Two passes address it: a frequent merge that -consolidates the newest small files and deletes nothing, and an hourly -maintenance cycle that also applies retention. [Tuning -retention](/guides/tune-retention) covers both. - -## Rollups lag, on purpose - -The overview and the alert engine read pre-aggregated rollups rather than raw -telemetry, recomputed on `FANOUT_ROLLUP_INTERVAL`. A rollup deliberately trails -the newest data by a safety margin, because telemetry arrives out of order — -aggregating right up to the current instant would produce numbers that change -after the fact as late spans land. - -The practical consequence: the newest few seconds are queryable as raw -telemetry before they appear in an overview or fire an alert. That is a -correctness choice, not lag to be tuned away. - -## Everything serialises through one write gate - -Flushes, rollups and maintenance all commit through a shared gate, so DuckDB's -single writer is never contended. Reads run concurrently — the catalog is opened -in WAL mode, which is what makes a connection pool larger than one safe. - -This is the mechanism behind the trade in [why one -binary](/explanation/why-one-binary): maintenance that runs harder takes gate -time that ingest is not getting. +| Atomic Parquet batches | Authoritative retained spans, logs, and metrics | +| Trace sidecar | Persistent row-range index for targeted trace reads from Parquet | +| DuckDB | SQL, log filtering, aggregation, and broad analytical reads | +| SQLite | Transactional application and identity state | + +There is no Iceberg, DuckLake, external catalog, or server database in the +telemetry path. + +
+ +
Fanout persistence flow: authenticated OTLP requests enter bounded group admission and parallel commit workers, publish immutable Parquet batches atomically, then serve indexed trace reads and DuckDB SQL while native maintenance prepares and swaps compacted replacements. + +
Parquet is authoritative; indexes and DuckDB rollups are derived state.
+ + +## Writes are durable immediately + +Concurrent small OTLP requests may share one group-commit batch. Up to four +commit workers encode independent batches in parallel. Each worker writes all +present signals and metadata into a staging directory, adds `trace.fidx` when +spans are present, fsyncs them, and publishes the complete directory with one +atomic rename. Fanout +returns success only after every batch belonging to the request is published. + +A crash before the rename leaves only an unacknowledged staging directory, +which startup removes. A crash after the rename leaves a complete batch that +startup discovers directly from the filesystem. There is no separate WAL, +manifest, catalog, post-acknowledgement flush timer, or acknowledged memory-only +state. + +## One authoritative copy for logs and metrics + +Every signal is written once, to typed Zstandard-compressed Parquet. DuckDB +applies filters, ordering, `LIMIT`, and aggregation inside its vectorized scan. +Trace lookup binary-searches a fixed-width sidecar on disk, seeks directly to the +matching Parquet row ranges, and verifies the complete trace ID after hashing. +The sidecar is an index, not another telemetry representation. + +Dashboard rollups are rebuildable DuckDB caches. SQLite remains independent and +stores only transactional product state. + +## Small files are bounded by compaction + +Atomic ingestion creates immutable batch directories. Maintenance drains every +eligible compaction group, choosing inputs by both count and a 25-million-row +ceiling. It combines same-day, same-generation files, builds a replacement trace +index, and atomically swaps the replacement for its inputs. A durable compaction +marker makes an interrupted swap resumable. Active DuckDB readers pin immutable +files while retention or compaction removes them, so reads never lose an open +input file. + +The expensive merge runs before readers are excluded. Only the bounded directory +swap takes the exclusive reader gate; new ingest batches continue encoding while +that work is prepared. + +Retention is based on ingestion time rather than event time. A service with a +bad clock therefore cannot pin a batch on disk by emitting a far-future event. + +## Rollups lag deliberately + +Overview and alert queries read rebuildable DuckDB rollups. Their watermark +trails committed telemetry by a fixed safety window so out-of-order events from +bounded commit retries are not skipped. Raw committed telemetry is queryable +before it appears in a rollup. diff --git a/site/src/content/docs/explanation/why-one-binary.mdx b/site/src/content/docs/explanation/why-one-binary.mdx index d18d6097..c638d9a5 100644 --- a/site/src/content/docs/explanation/why-one-binary.mdx +++ b/site/src/content/docs/explanation/why-one-binary.mdx @@ -38,10 +38,13 @@ loader binds. a machine. There is no scaling the query layer without also scaling ingest, because they are the same thing. -**They contend.** Maintenance and rollups serialise against ingest through one -write gate. On a busy instance, compacting harder is not free — it is traded -against ingest headroom. [Tuning retention](/guides/tune-retention) is mostly -about managing that trade. +**They contend for one machine.** Parquet ingest, DuckDB reads, rollups, and +native compaction have separate coordination paths, but they consume the same +CPU, memory bandwidth, filesystem cache, and disk. Maintenance excludes readers +only for bounded namespace changes—compaction swaps and retention removals; the +merge itself runs outside that gate. +Compacting harder is still traded against ingest and query headroom. [Tuning +retention](/guides/tune-retention) is mostly about managing that trade. **One process is one failure domain.** Nothing degrades independently. A process that dies takes ingest, query, alerting and the UI with it. diff --git a/site/src/content/docs/guides/back-up-and-restore.mdx b/site/src/content/docs/guides/back-up-and-restore.mdx index 328ff540..d8cae5bd 100644 --- a/site/src/content/docs/guides/back-up-and-restore.mdx +++ b/site/src/content/docs/guides/back-up-and-restore.mdx @@ -11,18 +11,17 @@ status: shipped Fanout is one process and one persistent data directory. Back up the directory as a unit. -That is not a simplification. The directory holds the telemetry catalog and its -Parquet files, query state, and the control SQLite database, and they reference -each other. Copying one subdirectory produces something that looks like a backup -and does not restore. +The directory holds atomic telemetry Parquet batches, rebuildable query state, +and the control SQLite database. Copying one subdirectory produces something +that looks like a backup and does not restore. ## Take a cold backup 1. Record the running version and configuration, keeping secrets out of ordinary logs and tickets. 2. Stop Fanout cleanly and wait for the process to exit. Shutdown closes both - OTLP listeners before draining the lake writer, so a clean exit is what makes - the files on disk consistent. + OTLP listeners before draining the telemetry commit workers, so a clean exit + leaves no accepted request waiting for publication. 3. Copy or snapshot the complete data directory, preserving ownership and permissions. 4. Start Fanout and confirm `/readyz`. @@ -32,8 +31,8 @@ and does not restore. :::caution[Not while it is running] Do not take a recursive copy of a live data directory. A storage-level snapshot is acceptable only if it gives a crash-consistent point-in-time view of the -whole directory at once — a copy that walks the tree while the writer moves -through it captures a catalog and Parquet files from different moments. +whole directory at once — a copy that walks the tree while commits publish can +capture related files from different moments. ::: ## Restore diff --git a/site/src/content/docs/guides/troubleshoot.mdx b/site/src/content/docs/guides/troubleshoot.mdx index 8d1fd0b2..8fd02a01 100644 --- a/site/src/content/docs/guides/troubleshoot.mdx +++ b/site/src/content/docs/guides/troubleshoot.mdx @@ -19,6 +19,95 @@ The one that surprises people is an unrecognised variable. Fanout rejects any release — stops the process instead of being ignored. That is deliberate; see [configuration](/reference/configuration). +## An unresolved compaction blocks startup + +One startup refusal is not about configuration. If a compaction was interrupted +and Fanout can neither finish nor undo it, recovery fails closed and the error +names `telemetry/COMPACTION.json`. + +That is deliberate. The marker lists the batches the compaction retired, and +recovery must not guess whether the original batches or their replacement are +authoritative. A wrong guess could delete the only surviving copy of some rows, +so Fanout stops instead and leaves things where they are: the replacement is +either staged under `telemetry/compaction/` or already published as +`telemetry/parquet/batches/.batch`, and the retired inputs sit under +`telemetry/parquet/batches/`, named `.retired-`. + +A publish that failed partway through may already have removed some of those +retired inputs, so the rollback set is not guaranteed to be complete. That is +what step 2 below checks, and why it has to be checked before anything is +deleted. + +Usually the interruption was transient — a full disk, or the process killed +mid-swap. **Clear the underlying cause and start Fanout again.** Recovery +re-runs on its own and needs no manual step. + +If recovery keeps failing, roll the compaction back by hand. With the process +stopped, use `telemetry/parquet/batches/` as the batch directory: + +1. Read `telemetry/COMPACTION.json`. It is JSON, with an `output` object and an + `inputs` list of batch ids. +2. Back up `telemetry/`, then check every id in `inputs`. Each must exist in + exactly one form: `.batch` (never retired) or + `.retired-` (retired, restorable). **If an id exists in both + forms or neither form, stop here.** Both forms make the authoritative copy + ambiguous; neither means its rows may survive only inside the replacement. + Reconcile the batch from the backup, or complete the compaction instead of + rolling it back. +3. Rename each `.retired-` back to `.batch`. +4. Delete `telemetry/compaction/` if it exists. +5. Delete `telemetry/parquet/batches/.batch` if it exists. Leaving + a published replacement beside the restored inputs duplicates its rows; a + corrupt replacement continues to block startup. +6. Delete `telemetry/COMPACTION.json`. +7. Start Fanout. + +Deleting the whole `telemetry/` directory also clears the refusal, and discards +exactly the telemetry the refusal was protecting. Neither step above needs it. + +:::caution[Verify the set, then restore, then delete] +The order is load-bearing in both directions. Verifying first (step 2) is what +keeps you from deleting a replacement that holds the only surviving copy of an +input's rows. Restoring before deleting the marker (steps 3 and 6) matters +because the marker is what protects the retired directories from cleanup — with +it gone, the next successful startup treats every `*.retired-*` directory as +reclaimable and removes it. Deleting the marker on its own is not a rollback; +it discards the rows the compaction had retired. +::: + +## An unreadable telemetry batch blocks startup + +Fanout also refuses to start when an authoritative `.batch` directory has +invalid metadata, a corrupt Parquet file, a missing signal file, or a damaged +trace index. First stop Fanout and back up `telemetry/`, then verify the full +authoritative set: + +```sh +fanout --config /etc/fanout/fanout.yaml repair verify +``` + +The command is read-only and names every unreadable batch. Restore each named +batch from backup when possible. If one cannot be recovered and discarding +only its telemetry is preferable to leaving the instance offline, set it aside +explicitly: + +```sh +fanout --config /etc/fanout/fanout.yaml repair quarantine --batch +``` + +Quarantine is an atomic rename beside the authoritative batch set, not a +deletion. It refuses a batch that validates successfully and refuses any input +or output protected by a live compaction marker. Preserve the reported +quarantine directory for later recovery, run `repair verify` again, and only +then restart Fanout. + +:::caution[Quarantine makes those rows unavailable] +The command is intentionally batch-specific and never runs during startup. +Every row in the quarantined directory disappears from queries until the batch +is restored, so do not use it as a substitute for fixing a transient disk or +permission error. +::: + ## An exporter is rejected Work down this list in order: diff --git a/site/src/content/docs/guides/tune-retention.mdx b/site/src/content/docs/guides/tune-retention.mdx index 58e276b0..4c4b1494 100644 --- a/site/src/content/docs/guides/tune-retention.mdx +++ b/site/src/content/docs/guides/tune-retention.mdx @@ -1,14 +1,14 @@ --- title: Tune retention description: Control how long telemetry is kept and how aggressively Fanout compacts what it has written. -summary: Retention, the hourly maintenance cycle, and the frequent merge pass that keeps query latency bounded. +summary: Retention, atomic batch sizing, and the maintenance pass that keeps query latency bounded. read_when: - Disk use is growing faster than expected. - Query or rollup latency is climbing on an instance that was fine. status: preview --- -Three settings govern what is on disk and how it is shaped. They do different +These settings govern what is on disk and how it is shaped. They do different jobs and are worth separating before changing any of them. ## How long data is kept @@ -30,41 +30,28 @@ FANOUT_MAINTENANCE_INTERVAL=1h This is the expensive pass: retention deletes plus full compaction. Lower it to reclaim space sooner, at the cost of running the heavy work more often. -## The merge pass +## Group-commit target -```sh -FANOUT_MERGE_INTERVAL=1m -``` - -This is the cheap one, and it is the setting that most often matters for -latency. It consolidates the newest small Parquet files and deletes nothing. -Running it often keeps the queryable file count continuously low, which is what -bounds rollup and query scan time — without the churn, the deletion race or the -catalog cost of the full maintenance pass. `0s` disables it. - -If query latency is climbing on an instance whose data volume has not changed, -this is the first thing to look at: a high file count from many small writes -costs more per scan than the same bytes in fewer files. - -## Flush behaviour - -How often data reaches disk in the first place: +The target number of telemetry rows that concurrent small requests may share in +one atomic Parquet batch: ```sh -FANOUT_FLUSH_INTERVAL=15s -FANOUT_FLUSH_BATCH_SIZE=50000 +FANOUT_INGEST_BATCH_SIZE=50000 ``` -A longer flush interval produces fewer, larger files — less compaction work, at -the cost of newly ingested telemetry taking longer to become queryable, and more -of it being lost if the process dies uncleanly. +A larger target can improve sustained write throughput and create fewer files +under concurrent ingest. Every request is still acknowledged only after its +Parquet batch is durably published. An individual request is never split merely +to satisfy this target: an oversized request remains one atomic batch. The +setting controls admission grouping, not the durability boundary. ## What the knobs interact with -Maintenance and rollups serialise against ingest through the same write gate. -Running maintenance much more often on a busy instance therefore trades disk -against ingest headroom rather than being free. Change one setting at a time and -watch `/-/metrics`. +Maintenance prepares merges while reads continue, then briefly gates readers +while it swaps or removes immutable files. New batches keep encoding and need +only the short publication lock. Ingest, queries, and maintenance still share +CPU, memory bandwidth, and disk, so running maintenance much more often is not +free. Change one setting at a time and watch `/-/metrics`. The full list, with defaults and types, is in the [storage settings](/reference/settings/storage) reference. diff --git a/site/src/content/docs/reference/data-layout.mdx b/site/src/content/docs/reference/data-layout.mdx index 6415ee33..2b547e8a 100644 --- a/site/src/content/docs/reference/data-layout.mdx +++ b/site/src/content/docs/reference/data-layout.mdx @@ -13,17 +13,24 @@ Everything Fanout persists lives under `FANOUT_DATA_DIR` (`./data` by default, | Path | Holds | |---|---| -| `telemetry/parquet/` | The telemetry itself, as Parquet files | -| `telemetry/ducklake.sqlite` | The DuckLake catalog tracking those files | -| `query/catalog.duckdb` | Query-side DuckDB state | +| `telemetry/parquet/batches/*.batch/` | Authoritative atomic batches containing `metadata.json`, present signal Parquet files, and `trace.fidx` when spans are present | +| `telemetry/parquet/batches/_schema.batch/` | Empty schema anchors that keep every DuckDB view queryable | +| `telemetry/parquet/batches/*.retired-*/` | Compaction inputs retained temporarily for crash rollback; a live marker protects its own set | +| `telemetry/parquet/batches/*.quarantined-*/` | Unreadable batches explicitly set aside by `fanout repair quarantine`; preserved but not queryable | +| `telemetry/parquet/staging/` | Unacknowledged writes being prepared for atomic publication | +| `telemetry/compaction/` | Prepared replacement batches during compaction | +| `telemetry/COMPACTION.json` | Durable marker for an interrupted compaction swap. Present only while one is unresolved, and blocks startup until it is — see [troubleshoot](/guides/troubleshoot) | +| `query/catalog.duckdb` | Rebuildable DuckDB rollups and query state | | `query/tmp/` | Spill space for queries that exceed the memory cap | | `control/fanout.sqlite` | Users, sessions, dashboards, alert rules, agent history | ## Why it is one unit -The catalog records which Parquet files exist and what they contain. Separated -from it, `telemetry/parquet/` is a directory of files nothing can locate, and -the catalog alone is an index of files that are not there. +Each published Parquet batch directory is a self-contained crash-recovery unit. +Its metadata records row counts and time bounds; its fixed-width trace sidecar +maps hashes to sorted span row ranges. The sidecar is verified against the +Parquet rows by offline repair, while DuckDB rollups can be rebuilt. Keeping the +complete data directory is the supported and fastest restore path. `control/fanout.sqlite` is independent of both in format but not in meaning: it holds the dashboards and alert rules that refer to the telemetry, and the @@ -33,11 +40,11 @@ So the unit of backup is the whole directory. [Back up and restore](/guides/back-up-and-restore) covers doing that safely, which mostly means doing it while the process is stopped. -## Two databases, on purpose +## Two engines, on purpose Telemetry is append-heavy, queried analytically over wide ranges, and never -updated. Application state is small, transactional and updated constantly. One -engine would serve one of those two badly — [how storage +updated. Application state is small, transactional and updated constantly. +Parquet with DuckDB serves the former; SQLite serves the latter — [how storage works](/explanation/storage-model) covers the reasoning and the write path. ## `query/tmp/` @@ -51,7 +58,7 @@ disk fails the query rather than degrading it. Retention is the main lever — `FANOUT_RETENTION_DAYS`, with expiry applied on the maintenance cycle rather than the moment you change it. File count matters -as much as byte count for query latency, which is what the merge pass exists to +as much as byte count for query latency, which is what the compaction pass exists to bound. [Tune retention](/guides/tune-retention) covers both. :::caution[Do not edit these files] diff --git a/site/src/content/docs/reference/settings/ingest.mdx b/site/src/content/docs/reference/settings/ingest.mdx index 2fda2a27..63d3cd64 100644 --- a/site/src/content/docs/reference/settings/ingest.mdx +++ b/site/src/content/docs/reference/settings/ingest.mdx @@ -19,9 +19,8 @@ as a refusal to start rather than as a default nobody chose. | Setting | Environment variable | Type | Default | |---|---|---|---| | `ingest.advertised_endpoint` | `FANOUT_INGEST_ADVERTISED_ENDPOINT` | string | — | +| `ingest.batch_size` | `FANOUT_INGEST_BATCH_SIZE` | integer | `50000` | | `ingest.default_namespace` | `FANOUT_DEFAULT_NAMESPACE` | string | `default` | -| `ingest.flush_batch_size` | `FANOUT_FLUSH_BATCH_SIZE` | integer | `50000` | -| `ingest.flush_interval` | `FANOUT_FLUSH_INTERVAL` | duration | `15s` | | `ingest.otlp_grpc_addr` | `FANOUT_OTLP_GRPC_ADDR` | string | `127.0.0.1:4317` | | `ingest.otlp_http_addr` | `FANOUT_OTLP_HTTP_ADDR` | string | `127.0.0.1:4318` | diff --git a/site/src/content/docs/reference/settings/storage.mdx b/site/src/content/docs/reference/settings/storage.mdx index df1511ad..87809fdf 100644 --- a/site/src/content/docs/reference/settings/storage.mdx +++ b/site/src/content/docs/reference/settings/storage.mdx @@ -23,7 +23,6 @@ as a refusal to start rather than as a default nobody chose. | `storage.duckdb.memory` | `FANOUT_DUCKDB_MEMORY` | string | — | | `storage.duckdb.threads` | `FANOUT_DUCKDB_THREADS` | integer | — | | `storage.maintenance_interval` | `FANOUT_MAINTENANCE_INTERVAL` | duration | `1h` | -| `storage.merge_interval` | `FANOUT_MERGE_INTERVAL` | duration | `1m` | | `storage.retention_days` | `FANOUT_RETENTION_DAYS` | integer | `30` | | `storage.rollup_interval` | `FANOUT_ROLLUP_INTERVAL` | duration | `1m` | | `storage.rollup_skip_to_latest` | `FANOUT_ROLLUP_SKIP_TO_LATEST` | boolean | `false` | @@ -32,7 +31,7 @@ as a refusal to start rather than as a default nobody chose. ### `storage.duckdb.max_connections` -Caps the DuckDB connection pool. A value of 1 serializes everything through one handle; the machine-sized default lets read queries run concurrently with each other and with ingest flushes. Two things make >1 safe: the DuckLake SQLite catalog is opened in WAL mode (enableCatalogWAL), so readers don't collide with the single writer and a crashed writer can't leave the catalog permanently locked; and write commits are serialized by the shared write gate (Duck.WriteGate, wired into the writer via UseWriteGate in cmd/fanout/main.go, enforced at startup). Without the WAL mode, pool >1 fails with "database is locked". Zero means "size it from the machine" — the same spelling DuckDBThreads uses for deferring to a default. Resolution happens in resolveSizing and is reported in the startup configuration log. +Caps the DuckDB connection pool. Reads scan immutable Parquet concurrently; rollup-cache writes are serialized by the query write gate. Zero means "size it from the machine" — the same spelling DuckDBThreads uses for deferring to a default. Resolution happens in resolveSizing and is reported in the startup configuration log. ### `storage.duckdb.memory` @@ -44,11 +43,7 @@ Caps DuckDB's global query worker pool. Zero leaves DuckDB's own default in plac ### `storage.maintenance_interval` -Throttles the DuckLake maintenance cycle (retention deletes + compaction). Default 1h. Lower it to compact more aggressively, or for soak tests that need to observe file-count staying bounded within minutes rather than hours. - -### `storage.merge_interval` - -The cadence for the cheap, frequent DuckLake file compaction pass (ducklake_merge_adjacent_files only — it consolidates the newest small parquet files and deletes nothing). Run often (default 1m) it keeps the queryable file count continuously low, which is what bounds rollup/query scan latency — WITHOUT the churn, deletion race, or catalog cost of the full hourly maintenance pass (expire + cleanup). 0 disables it. +Controls Parquet retention and compaction, and query-cache checkpointing. ### `storage.rollup_skip_to_latest` diff --git a/site/src/content/docs/start/first-boot.mdx b/site/src/content/docs/start/first-boot.mdx index f55f29c1..00d959a2 100644 --- a/site/src/content/docs/start/first-boot.mdx +++ b/site/src/content/docs/start/first-boot.mdx @@ -19,8 +19,7 @@ Every duration and size has a floor, and the auth settings are checked as a group rather than individually: - Addresses and the data directory must not be empty. -- `FANOUT_FLUSH_INTERVAL` and `FANOUT_ROLLUP_INTERVAL` must be at least `1s`; - `FANOUT_MERGE_INTERVAL` must be `0s` or at least `1s`. +- `FANOUT_ROLLUP_INTERVAL` must be at least `1s`; - `FANOUT_AUTH_MODE` must be `local` or `oidc`. - `FANOUT_SESSION_IDLE_TTL` must be at least `5m`, and the absolute TTL must be positive and no shorter than the idle one. diff --git a/site/src/content/docs/start/what-fanout-is.mdx b/site/src/content/docs/start/what-fanout-is.mdx index bcc27be0..e572f721 100644 --- a/site/src/content/docs/start/what-fanout-is.mdx +++ b/site/src/content/docs/start/what-fanout-is.mdx @@ -13,6 +13,18 @@ OTLP over gRPC and HTTP, writes telemetry to disk as Parquet, answers queries with DuckDB, evaluates alert rules, serves a chat investigator and MCP tools, and hosts the browser client. +
+ + Fanout architecture: collectors, browsers, and MCP hosts connect to one process containing ingest, query, maintenance, alerts, agent, and UI components backed by Parquet, DuckDB query state, and SQLite control state. + +
One process owns the complete path from OTLP ingest to investigation.
+
+ The point of that list is that it is one list. A conventional deployment of the same capability is a collector, a storage backend, a query layer, a dashboard service and an alertmanager, each with its own configuration, failure modes and @@ -24,8 +36,8 @@ restart. | Part | What it does | |---|---| | OTLP ingest | Accepts traces, logs and metrics on gRPC and HTTP | -| Storage | Writes Parquet to a local data directory; no external database | -| Query | DuckDB in-process, over the Parquet it wrote | +| Storage | Atomically publishes indexed Parquet batches to a local data directory | +| Query | Direct indexed trace reads plus in-process DuckDB SQL and rollups | | Alerts | Evaluates persisted rules on a fixed interval against rollups | | Investigator | A chat agent working from the same typed query tools | | MCP server | Exposes those tools to an external agent | diff --git a/site/src/styles/fanout.css b/site/src/styles/fanout.css index 25ccaf78..47890e3c 100644 --- a/site/src/styles/fanout.css +++ b/site/src/styles/fanout.css @@ -186,6 +186,50 @@ h1#_top { outline-offset: 2px; } +/* Architecture diagrams are generated for a light technical canvas. Give them + one explicit blueprint-like media surface in both themes rather than letting + dark mode erase their ink. The border carries hierarchy; no decorative + shadow or gradient competes with the diagram itself. */ +.sl-markdown-content .docs-diagram { + margin: 1.75rem 0 2rem; + padding: clamp(0.75rem, 2vw, 1.25rem); + border: 1px solid var(--sl-color-gray-5); + border-radius: 0.5rem; + background: #fafafa; +} + +.sl-markdown-content .docs-diagram__asset { + display: block; + border-radius: 0.25rem; +} + +.sl-markdown-content .docs-diagram__asset:focus-visible { + outline: 2px solid var(--sl-color-accent); + outline-offset: 4px; +} + +.sl-markdown-content .docs-diagram img { + display: block; + width: 100%; + height: auto; + margin: 0 auto; +} + +.sl-markdown-content .docs-diagram figcaption { + margin-top: 0.75rem; + color: #565b69; + font-family: var(--fo-font-display); + font-size: 0.75rem; + line-height: 1.5; + text-align: center; +} + +@media (max-width: 50rem) { + .sl-markdown-content .docs-diagram { + margin-inline: 0; + } +} + /* Without a minimum, a four-column reference table compresses to fit a phone and every column becomes a ribbon of wrapped text. The minimum makes it overflow instead, which is what the scroll container is there to absorb.