Skip to content

Reconcile Linux safety fixes and prevent false-success backup, health and log checks #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
blow-tech wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Reconcile Linux safety fixes and prevent false-success backup, health and log checks #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a7d7b35
Fix approved Linux script safety and correctness findings
blow-tech Sep 8, 2026
f8ca573
Prevent checksum evidence overwrite and validate backup publication
blow-tech Sep 9, 2026
6834cd2
Reconcile current main and fail closed on backup, health and log coll…
blow-tech Oct 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .github/workflows/safety.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
name: Isolated script safety
on: [push, pull_request]
permissions:
contents: read
jobs:
fixtures:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
- name: Bash syntax and local fixture regressions
run: python3 -m unittest discover -s tests -v
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@

__pycache__/
*.pyc
209 changes: 45 additions & 164 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,187 +1,68 @@
# 🐧 Linux Admin Scripts
# Linux Admin Scripts

A production-ready collection of Bash scripts for Linux system administration monitoring, alerting, backup, maintenance, and security auditing.
11 October reconciliation: the existing safety fixes are reconciled with current main, preserving all four scripts added since September. `backup_verify.sh`, `healthcare_check.sh`, and `log_summary.sh` now distinguish unavailable evidence from successful checks. The suite contains 28 isolated tests. See [REVIEW-2026-10-11.md](REVIEW-2026-10-11.md) for this run's scope and migration notes.

> All scripts include centralized Slack/Teams/Email alerting, lock files, log rotation, and proper error handling.
9 September continuation: shared backup publication now refuses existing checksum files, symlinks and directories. Both daily and weekly paths publish checksums with no-clobber hard links; a checksum-generation failure cannot report a verified backup. Interrupted publication may still leave an archive without its final checksum, so require a valid checksum pair and investigate retained partial artifacts. This adds two isolated regression tests (12 total); application-consistent restoration remains a lab gate.

---
Bash administration utilities. Validate each script in a disposable lab before production deployment. Syntax checks and fixture tests do not certify an environment or an application backup.

## 📁 Repository Structure
## Entry points

```
linux-admin-scripts/
└── scripts/
├── core/ ← Shared alert engine (load this first)
├── audit/ ← Security & system auditing
├── monitoring/ ← Resource & service monitoring
├── backup/ ← Backup & rotation scripts
└── maintenance/ ← Cleanup & housekeeping
```
Use `bash <script-path>`; several maintained entry points intentionally have no `.sh` extension.

---
| Path | Behavior and scope |
| --- | --- |
| `scripts/monitoring/server-stats.sh` | Local statistics snapshot; unchanged in the September remediation. |
| `scripts/monitoring/cpu_memory_monitor` | CPU/memory observations; local log and stderr alerts. |
| `scripts/monitoring/disk_alert` | One GNU df collection; warning/critical or unavailable collection returns nonzero. |
| `scripts/monitoring/service_uptime <unit> ...` | Explicit service scope, alert-only; never restarts a service. |
| `scripts/backup/backup_script <source> <existing-destination>` | Verified, versioned file archive; retention inventory only. |
| `scripts/backup/backup_rotation.sh` | Versioned daily/weekly archives; requires SRC_DIR, BACKUP_DIR and EXPECTED_BACKUP_SOURCE. |
| `scripts/backup/backup_verify.sh -d <directory>` | Read-only freshness/size check, exact age in seconds; optional `--verify-archive` fails when integrity cannot be verified. |
| `scripts/monitoring/healthcare_check.sh` | CPU, memory, swap and disk checks; failed, empty or malformed collections return UNKNOWN. |
| `scripts/monitoring/log_summary.sh -f <log>` | Read-only log summary; numeric rotations optional; failed reads/decompression/timeouts return UNKNOWN. |
| `scripts/audit/security-audit.sh` | Preserved current-main security audit; not remediated or runtime-validated in this run. |
| `scripts/Nginx_create.sh <dns-name> <template> <owner>` | Configuration preview. `--apply` installs and validates; `--reload` additionally reloads Nginx. |
| `scripts/appache2_confg.sh --plan` | Apache installation plan. `--install` is explicit and may start Apache; never changes UFW. |
| `scripts/audit/LinuxAudit.sh` | Interactive audit. Expensive filesystem scans are opt-in and bounded. |
| `scripts/audit/FastCheck.sh <host> --scan` | Explicitly authorized TLS/connectivity checks; intrusive NSE/DoS menu removed. |
| `scripts/audit/ScanPort.sh <IPv4> --scan` | Explicitly authorized checks of the 24 listed TCP ports. |

## 🔔 Core
The previously documented `system_resource_check` and `maintenance/log_cleanup` scripts are not supplied. Remove references to those nonexistent paths from deployment plans.

| Script | Description | Risk Level |
|---|---|---|
| `scripts/core/alert_engine.sh` | Central alerting module Slack, MS Teams, Email. Sourced by all other scripts. Also provides lock files and log rotation. | Read-only |
## Alerts, state, and privileges

**Configure this first** before running any other script. Set your Slack webhook, Teams webhook, or email inside this file.
`scripts/core/alert_engine.sh` supplies locking, log archival, and timestamped **local stderr alerts only**. It does not send Slack, Teams or email. Integrate stdout/stderr and exit status with your approved monitoring system; external delivery is not configured by this repository. The CPU monitor also writes `/var/log/admin-scripts`; provision that protected log directory before deployment. Log archives are retained; configure a separate reviewed retention policy.

---
Locks use `${XDG_STATE_HOME:-$HOME/.local/state}/admin-scripts`, or `ADMIN_SCRIPTS_STATE_DIR`. The state directory must be owned by the executing account, mode 700, with trusted ancestors. Backup directories and configuration parents must be protected against concurrent untrusted writers. Do not run from or deploy into user-writable directories as root.

## 🔍 Audit
Monitoring needs read access to the selected objects. Backup needs read access to the source and write access to the destination. Nginx `--apply` and Apache `--install` require root and an approved change window. Scan commands contact their target; target-owner approval is required.

| Script | Description | Risk Level |
|---|---|---|
| `scripts/audit/LinuxAudit.sh` | Collects Linux system audit information — users, services, packages, network details, and configuration data. | Read-only / information gathering |
## Backup configuration and migration

---
1. Provision an existing, protected destination and verify the intended mounted device. Read-only check: `findmnt -n -o SOURCE -T <BackupDirectory>`.
2. Pin the reviewed value in `EXPECTED_BACKUP_SOURCE`; do not dynamically accept whichever device is mounted at runtime.
3. Set the source/destination explicitly. Archives include a source-path job identifier and unique timestamp/PID; old daily files are never accepted merely because they exist.
4. Creation writes to `.partial.*`, checks gzip/tar integrity, then publishes without overwriting an existing archive and writes a SHA256 sidecar. Failures retain clearly marked partial artifacts and return nonzero.
5. Retention prints job-specific, nonrecursive candidates; it never deletes previous backups. Review retention and capacity separately. Existing legacy archives are retained.
6. Test restoration into an isolated directory and compare representative files, metadata, ACLs and application consistency. These are file archives, not consistent live-database backups; this implementation does not promise ACL/xattr preservation. Use an application-aware backup when required.

## 📊 Monitoring
Weekly filenames include the ISO week-year. Publication requires same-filesystem hard-link support. Interrupted runs after publication but before the sidecar completes must be investigated; do not infer completion from filename alone. No automatic cleanup of failed artifacts is performed.

| Script | Description | Risk Level |
|---|---|---|
| `scripts/monitoring/server-stats.sh` | Displays server statistics CPU, memory, disk usage, uptime, and running processes. | Read-only |
| `scripts/monitoring/cpu_memory_monitor.sh` | Monitors CPU and memory usage. Sends WARNING/CRITICAL alerts when thresholds are exceeded. Uses `vmstat` and `/proc/meminfo` for accuracy. | Read-only |
| `scripts/monitoring/disk_alert.sh` | Monitors all disk mountpoints. Two alert levels: WARNING at 75%, CRITICAL at 90%. Skips tmpfs/devtmpfs. | Read-only |
| `scripts/monitoring/service_uptime_checker.sh` | Checks critical services (nginx, docker, mysql). Auto-restarts if down, retries N times, escalates to CRITICAL alert with journal logs if restart fails. | Read/Write (restarts services) |
| `scripts/monitoring/system_resource_check.sh` | Full hourly system snapshot — CPU, RAM, swap, disk, top processes, network connections, last logins. Saves report to log file. | Read-only |
## Configuration changes

---
Nginx defaults to stdout preview. Templates are operator-supplied and must be reviewed, including certificate paths and any includes. Existing web roots/sites are refused. Applying checks the baseline first, creates only the new targets, tests the complete configuration, and rolls back only those new targets on failure. Ownership changes are limited to the newly created directory. If `--reload` fails, the previous file configuration is restored and a rollback reload is attempted; a failure is reported for operator intervention. A successful `--apply` without `--reload` stages the site for the next reload.

## 💾 Backup
Apache installation and firewall activation are separate changes. Before any manual firewall activation, record existing policy, the actual management port/source, console access, rollback commands, and an independent management-session validation. This script does not activate the firewall.

| Script | Description | Risk Level |
|---|---|---|
| `scripts/backup/backup_rotation.sh` | Daily + weekly backup with rotation. Pre-flight checks for source directory and free disk space. Alerts on failure. | Read/Write |
| `scripts/backup/backup_script.sh` | Compressed backup with archive integrity verification (`tar -tzf`). Cleans backups older than N days. | Read/Write |
For filesystem audit scans, set `ENABLE_FILESYSTEM_SCAN=1` and an explicit absolute `AUDIT_SCAN_ROOT` only after approving metadata I/O. Results can be incomplete on permission errors or timeout and must be reviewed accordingly.

---
## Validation

## 🧹 Maintenance
The three October checkers use exit 3 for UNKNOWN/incomplete evidence and exit 2 for invalid usage; `--help` exits 0. Backup and health checkers return 1 for failed checks or exceeded thresholds. Integrations must alert on every nonzero exit, not just 1. Backup age is compared in seconds: a file one second older than the configured threshold fails; a future timestamp is UNKNOWN. Unsupported archive extensions cannot satisfy `--verify-archive`.

| Script | Description | Risk Level |
|---|---|---|
| `scripts/maintenance/log_cleanup.sh` | Deletes log files older than N days. Optional `--archive` flag compresses logs before deletion. Protects `audit.log` and `secure.log`. | Read/Write (deletes files) |
Log rotation discovery now includes only `.N` and `.N.gz`. Date-named rotations must be supplied explicitly with `-f`. Log scanning has a 60-second timeout per file (plus five seconds to terminate), configurable with `--timeout`; this does not bound total disk space or total runtime across many rotations. Compressed logs are decompressed once with status checked. Health collectors use 30-second timeouts, and memory/swap share one `free` call. These scripts require Bash and GNU/Linux tools (`timeout`, NUL-aware `sort`, GNU `find`/`stat`, plus `gzip`/`tar` or `unzip` for the relevant formats).

---
`python3 -m unittest discover -s tests -v` performs Bash syntax validation, local backup creation/restore fixtures, and mocked systemctl/df/configuration-preview checks. It makes no production calls. GitHub Actions runs the same tests. Nginx reload/rollback, full disks, application restore, production mounts and notification delivery still require environment-specific lab validation.

## ⚙️ Setup

### 1. Clone the repository

```bash
git clone https://github.com/blow-tech/linux-admin-scripts.git
cd linux-admin-scripts
```

### 2. Deploy to server

```bash
sudo mkdir -p /opt/admin-scripts
sudo cp -r scripts/* /opt/admin-scripts/
sudo chmod +x /opt/admin-scripts/**/*.sh
sudo mkdir -p /var/log/admin-scripts
```

### 3. Configure alerts

Edit `/opt/admin-scripts/core/alert_engine.sh`:

```bash
SLACK_ENABLED=true
SLACK_WEBHOOK_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"

TEAMS_ENABLED=true
TEAMS_WEBHOOK_URL="https://outlook.office.com/webhook/YOUR-WEBHOOK-URL"

SMTP_CONFIGURED=true
ALERT_EMAIL="admin@yourcompany.com"
```

### 4. Make scripts executable

```bash
chmod +x scripts/monitoring/cpu_memory_monitor.sh
chmod +x scripts/monitoring/disk_alert.sh
chmod +x scripts/monitoring/service_uptime_checker.sh
chmod +x scripts/monitoring/system_resource_check.sh
chmod +x scripts/backup/backup_rotation.sh
chmod +x scripts/backup/backup_script.sh
chmod +x scripts/maintenance/log_cleanup.sh
chmod +x scripts/core/alert_engine.sh
```

### 5. Run a script

```bash
./scripts/monitoring/cpu_memory_monitor.sh
./scripts/monitoring/disk_alert.sh
./scripts/audit/LinuxAudit.sh
./scripts/backup/backup_rotation.sh
./scripts/maintenance/log_cleanup.sh /var/log 14 --archive
```

---

## ⏰ Recommended Crontab

```bash
sudo crontab -e
```

```cron
# CPU & Memory — every 5 minutes
*/5 * * * * /opt/admin-scripts/monitoring/cpu_memory_monitor.sh

# Disk Alert — every 30 minutes
*/30 * * * * /opt/admin-scripts/monitoring/disk_alert.sh

# Service Health — every 5 minutes
*/5 * * * * /opt/admin-scripts/monitoring/service_uptime_checker.sh

# System Resource Snapshot — every hour
0 * * * * /opt/admin-scripts/monitoring/system_resource_check.sh

# Daily Backup — 2AM every day
0 2 * * * /opt/admin-scripts/backup/backup_rotation.sh

# Log Cleanup — 3AM every Sunday
0 3 * * 0 /opt/admin-scripts/maintenance/log_cleanup.sh /var/log 14 --archive
```

---

## 📋 Log Files

All logs are written to `/var/log/admin-scripts/`:

| File | Contents |
|---|---|
| `alerts.log` | All alerts from all scripts |
| `cpu_memory_monitor.log` | CPU/memory check history |
| `disk_alert.log` | Disk usage check history |
| `service_uptime_checker.log` | Service status & restart history |
| `system_resource_check.log` | Hourly resource snapshots |
| `backup_rotation.log` | Daily/weekly backup history |
| `backup_script.log` | Backup job history |
| `log_cleanup.log` | Cleanup activity |

---

## 🛡️ Script Safety Features

Every script in this repo includes:

- `set -euo pipefail` — exits immediately on any error
- Lock files — prevents duplicate cron executions
- Log rotation — log files are capped automatically
- Pre-flight validation — checks directories and disk space before acting
- Alert integration — every failure sends a real-time notification

---

## 👤 Author
Linux System Administrator
GitHub: [@blow-tech](https://github.com/blow-tech)
The September 2026 remediation corresponds to review findings H2-H5, H9, M1-M2 and the identified Linux performance items. See the pull request for exact validation evidence. Keep existing deployments pinned until their schedule and parameters have been migrated and approved.
42 changes: 42 additions & 0 deletions REVIEW-2026-10-11.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Linux script remediation — 11 October 2026

This change continues the existing Linux draft PR under the user's renewed approval to fix and improve GitHub scripts. It does not merge, deploy, change schedules, or execute scripts against production.

## Reconciliation

Reviewed main: `81ae9b030416ec45e3cb21605dc1e63a079c0b6a`.
Previous PR head: `f8ca5736d1ceb6fa75d492df67960b3eb51d5a4d`.

All 17 main files and all files in the previous PR snapshot were checked against GitHub blob hashes before editing. Main's ten shared operational scripts still match the original September base; the prior ten-script safety changes can be retained without discarding later source edits. Main added four scripts, all retained. Three receive the fixes below; `security-audit.sh` remains unchanged. README conflicts are resolved with accurate paths, local-only alert behavior, and current entry points and migration notes.

## New fixes in this run

| Script | Fix and evidence |
| --- | --- |
| `scripts/backup/backup_verify.sh` | Compare age in seconds, reject future timestamps and invalid arguments, handle newline filenames with NUL records, retain enumeration failures, and reject unsupported requested integrity checks. Fixtures cover the exact age boundary, stale/future files, denied collection, unusual names, valid/corrupt archives and unsupported formats. |
| `scripts/monitoring/healthcare_check.sh` | Failed/empty/malformed CPU, memory or disk data becomes UNKNOWN; validate thresholds, collect memory/swap once, preserve mountpoint spaces, use bounded external collectors. Fixtures cover healthy/high-usage paths and failed/malformed collectors. |
| `scripts/monitoring/log_summary.sh` | Distinguish grep no-match from errors, validate regex/numeric arguments, check decompression and scan exit codes, timeout individual scans, restrict rotations, and avoid head-induced SIGPIPE on large summaries. Fixtures cover corrupt gzip, invalid regex, read errors/timeouts, numeric rotations and large output. |

## Previous fixes retained and reconciled

Ten scripts: `scripts/backup/backup_script`, `scripts/backup/backup_rotation.sh`, `scripts/monitoring/service_uptime`, `scripts/monitoring/disk_alert`, `scripts/monitoring/cpu_memory_monitor`, `scripts/Nginx_create.sh`, `scripts/appache2_confg.sh`, `scripts/audit/FastCheck.sh`, `scripts/audit/ScanPort.sh`, and `scripts/audit/LinuxAudit.sh`.

Two shared modules: `scripts/core/alert_engine.sh` and `scripts/core/backup_engine.sh`.

They retain verified versioned backups and preservation of prior backup/checksum evidence; preview-only retention; explicit service scope without automatic restarts; local alert/locking dependencies; Nginx preview and scoped apply/rollback; separate Apache installation and firewall administration; syntax/input fixes and bounded audit scanning. These are existing PR fixes, not thirteen newly discovered fixes in this run.

## Validation

- 28 isolated Python unittest cases passed locally: 12 retained safety regressions plus 16 collection regressions. The syntax case checks all 17 Bash entry points/modules.
- Disposable archive round-trip and no-overwrite fixtures passed. No production source, mount, tenant, server, notification destination or network scanning target was used.
- Failed and malformed command responses are injected with temporary PATH mocks. Timeout handling uses injected timeout exit status; this is not a real hung-process integration test.
- Existing GitHub Actions workflow runs the same suite. The PR and Actions status are the source of truth for remote validation.
- No speedup benchmark was performed. Removing the duplicate `free` call is verified by an invocation-count fixture.

## Migration and remaining limits

Checkers now return 3 for UNKNOWN, 2 for usage and 0 for successful help. Alert on every nonzero result. Date-named log rotations require explicit `-f`; automatic inclusion is limited to `.N`/`.N.gz`. Backup `--verify-archive` cannot pass unsupported formats. The README describes changed backup, service, configuration and audit interfaces retained from September.

Application-consistent backup restoration, real Nginx apply/reload/rollback, actual mount/capacity behavior and platform integration still require disposable lab validation before deployment. Backup verification reads metadata and archives in separate operations; run against completed, stable backups. File archive/checksum publication is not a single transaction; incomplete artifacts remain available for investigation. Log scan timeout limits are per input, not a storage or total-job budget.

This run addresses the Linux administration PR only. The separate PowerShell, audit-toolkit and infrastructure findings in the 5 October review remain outside this change; no claim is made that every repository is fixed or production-certified.
Loading
Loading