Skip to content

[dmt] Harden chart loading against symlink loops and oversized files#421

Open
ldmonster wants to merge 5 commits into
mainfrom
fix/symlink-occures-scan-loop
Open

[dmt] Harden chart loading against symlink loops and oversized files#421
ldmonster wants to merge 5 commits into
mainfrom
fix/symlink-occures-scan-loop

Conversation

@ldmonster

@ldmonster ldmonster commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Summary

dmt lints modules by handing each chart directory to nelm's chart loader. That loader walks the tree following symbolic links with no cycle detection and reads every file fully into memory. Two classes of pathological input therefore take down the whole process:

  1. Symlink loops — a link that resolves to one of its own ancestors makes the walk recurse forever, reading the same files into memory until the Go runtime aborts with a fatal out-of-memory error. Along the way nelm logs a found symbolic link in path: … line per step, so the process also floods stdout/journald before it dies.
  2. Oversized files — a multi-gigabyte file (generated dashboards, rendered OpenAPI, CRD bundles) read in full just to scan it exhausts memory for no useful result.

This branch makes dmt survive both: it pre-flights every chart directory, lints a cleaned copy when a symlink loop is present (surfacing the loop as a finding instead of crashing), caps the size of any single file it will read, and lets the large-file warning be excluded per module. nelm is bumped to a release whose own loader no longer follows symlink cycles unbounded.

Motivation

A module in the wild shipped a chart with a templates/loop -> .. symlink. Running dmt against it drove nelm's loader into an unbounded walk that consumed all memory and produced an effectively unbounded log stream before the process was killed. Linting should degrade gracefully on hostile input, not crash and flood the logs.

What changed

1. Symlink-loop protection (add symlink resolver)

  • internal/helm/guard.go (new) — a pre-flight pass over the chart dir that mirrors how nelm's loader classifies files (follows symlinks, honours .helmignore) but tracks the resolved real path of every directory on the descent stack, so it can:
    • detect when a symlink resolves to one of its own ancestors (scanChartDirhasLoop), and
    • reject a tree whose cumulative readable size exceeds a generous 2 GiB backstop (a real chart is far smaller; tripping it means the dir is pathological).
    • materializeCleanChartDir copies the chart into a temp dir with symlinks dereferenced and loops skipped, so nelm's loader can walk it safely.
  • internal/helm/render.goRenderChartFromDir now pre-flights the directory: on a symlink loop it renders a cleaned copy (loops skipped, symlinks dereferenced) so linting still proceeds; an oversized tree is a hard error. A new OnSymlinkLoop callback lets the caller surface each skipped loop as a lint finding rather than dropping it silently.
  • internal/module/controller.go — wires OnSymlinkLoop to emit a symlink-loop warn-level finding, so the module is still linted but the loop is reported.
  • internal/module/sympath.go — dmt's own forked Walk gains the same ancestor-based cycle detection (skips an already-visited real path with a warning instead of recursing forever).
  • go.mod / go.sum — bump github.com/werf/nelm v1.24.3 → v1.26.1.

2. Cap the size of any single file dmt reads (add truncate big files)

  • internal/fsutils/fsutils.go — new fsutils.ReadFile, which behaves like os.ReadFile but refuses files larger than MaxLintableFileSize (10 MiB), returning ErrFileTooLarge. Helper IsFileTooLarge(err) lets scanners treat it as "skip this file" rather than a hard failure.
  • Adopted across the file-scanning linters, which now skip (and, where relevant, warn about) oversized files instead of loading them:
    • pkg/linters/no-cyrillic/rules/{files,library}.go
    • pkg/linters/docs/rules/{cyrillic_in_english,no_lang_key}.go
    • pkg/linters/openapi/rules/crds.go
    • pkg/linters/templates/rules/{grafana_dashboards,prometheus_rules,mount_points}.go
    • pkg/linters/container/rules/mount_points.go

3. Exclude the large-file warning per module (add exclude for big files)

  • pkg/config.go / pkg/config/linters_settings.go — new documentation.exclude-rules.file-size config (files / directories). These excludes affect the large-file size warning only; the documentation content checks themselves are unchanged.
  • internal/module/module.gomapDocumentationExclusions wires the new config through (previously documentation had no exclusion mapping).

Behavior changes

  • A chart containing a symlink loop no longer OOMs / floods the log. It is linted from a cleaned copy and the loop is reported as a symlink-loop warning.
  • Files larger than 10 MiB are skipped with a warning rather than read; the warning can be silenced via exclude-rules.file-size.
  • A chart tree whose readable content exceeds 2 GiB is rejected with a clear error instead of being loaded.

Signed-off-by: Pavel Okhlopkov <pavel.okhlopkov@flant.com>
Signed-off-by: Pavel Okhlopkov <pavel.okhlopkov@flant.com>
@ldmonster ldmonster self-assigned this Jul 10, 2026
@ldmonster ldmonster added the bug Something isn't working label Jul 10, 2026
Signed-off-by: Pavel Okhlopkov <pavel.okhlopkov@flant.com>
Signed-off-by: Pavel Okhlopkov <pavel.okhlopkov@flant.com>
@ldmonster ldmonster force-pushed the fix/symlink-occures-scan-loop branch from 85379c7 to ee8fdca Compare July 10, 2026 13:13
@ldmonster ldmonster changed the title [dmt] add symlink cycle terminator, truncate big files [dmt] Harden chart loading against symlink loops and oversized files Jul 10, 2026
…ink-occures-scan-loop

Signed-off-by: Pavel Okhlopkov <pavel.okhlopkov@flant.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant