Politico → Kindle

Automated newsletter ingestion, transformation, and delivery pipeline

End-to-end system that ingests subscription emails, transforms them into Kindle-optimized documents, and delivers them automatically via email relay — built with production-grade email infrastructure.

---

✨ Overview

This project automatically converts POLITICO newsletters (received via email) into Kindle-friendly EPUB documents and delivers them to a Kindle device on a schedule.

It is designed as a robust backend pipeline, not a script:

• real mail server (Postfix, Dovecot)
• Maildir ingestion
• deduplication & archiving
• HTML content cleaning & reflow
• EPUB generation optimized for Kindle
• authenticated SMTP relay delivery
• scheduled execution with cron
• idempotent, observable, restart-safe

This system runs continuously on a Linux server with zero manual intervention.

---

🏗 Architecture

Gmail subscription emails
        |
        v
Postfix (inbound SMTP)
        |
        v
Maildir queue: ~/Maildir/new
        |
        v
politico-to-kindle pipeline
  - parse .eml
  - extract preferred body (HTML/text)
  - clean/normalize content
  - render Kindle-optimized layout
  - generate EPUB
  - dedupe + archive
        |
        v
Postfix (submission, 587)
        |
        v
SMTP2GO relay (587 + STARTTLS)
        |
        v
Amazon Send-to-Kindle

---

🚀 Features

Email ingestion

• Receives forwarded newsletters via standard SMTP
• Uses Maildir semantics for safe, atomic processing
• Handles HTML and plaintext emails

Content transformation

• Cleans newsletter HTML (removes tracking, ads, layout noise)
• Reflows content for e-ink page width
• Preserves headings, links, sections
• Produces Kindle-friendly EPUB (reflowable, adjustable font size)

Reliability

• Deduplication via message fingerprinting
• Idempotent: safe to re-run at any time
• Archiving of original .eml and generated artifacts
• Queue-based processing (Maildir/new → processed)

Delivery

• Authenticated SMTP submission
• External relay (SMTP2GO) for deliverability
• TLS enforced
• Works without outbound port 25

Automation

• Cron-scheduled runs (morning + evening)
• Locking to prevent concurrent runs
• Fully unattended operation

---

🛠 Tech Stack

• Python 3.12
• Postfix (SMTP, submission)
• Dovecot (Maildir delivery)
• SMTP2GO (relay)
• Kindle Send-to-Kindle
• BeautifulSoup (HTML processing)
• Linux cron
• Maildir format
• TLS / SMTP AUTH
• SPF / DKIM aware

---

📁 Repository Structure

politico-to-kindle/
├── src/politico_to_kindle/
│ ├── ingest/ # Maildir reading, dedupe
│ ├── transform/ # HTML extraction & cleaning
│ ├── render/ # Kindle HTML / EPUB generation
│ ├── send/ # SMTP submission
│ └── main.py # Orchestration
├── scripts/
│ └── run.sh # Production entrypoint
├── config/
│ └── production.env # Runtime configuration
├── runtime/
│ ├── archive/ # Archived .eml / epub
│ └── state/ # Deduplication state
└── README.md

---

⚙️ Configuration

Environment variables (example):

MAILDIR_NEW=/home/politico/Maildir/new
ARCHIVE_DIR=/opt/politico-to-kindle/runtime/archive
STATE_DIR=/opt/politico-to-kindle/runtime/state

SMTP_HOST=MAIL.DOMAIN.COM
SMTP_PORT=587
SMTP_USER=politico
SMTP_PASS=********

KINDLE_EMAIL=your_kindle@kindle.com
MAIL_DOMAIN=its-kaster.de