FYSummarizer

FYSummarizer is an AI-powered application that helps users stay informed without endless scrolling. It retrieves recent discussions from X and uses a multi-stage LLM summarization pipeline to generate concise overviews of topics or accounts you care about.

Why I Built It

I wanted a way to keep up with conversations on social media without spending hours doomscrolling. FYSummarizer retrieves relevant posts from X and uses a multi-stage LLM pipeline to generate concise summaries.

Features

• Search for up to five keywords or X accounts.
• Retrieve recent posts relevant to your search.
• Generate concise summaries using a hierarchical LLM pipeline.
• Reduce information overload while staying up to date.

Getting Started

1. Clone the repository.
2. Install the Python dependencies.
3. Configure your OpenAI API key.
4. Configure your ScrapeBadger credentials.
5. Launch the local PHP server.

Architecture

                User
                  │
                  ▼
      Browser (HTML/CSS/JavaScript)
                  │
            POST request (Fetch API)
                  │
                  ▼
             handler.php
       Input validation & routing
                  │
       Launch Python subprocess
                  │
                  ▼
             backend.py
        ├─ Retrieve posts (ScrapeBadger)
        ├─ Clean & batch content
        ├─ Generate batch summaries
        └─ Merge into final summary
                  │
                  ▼
          JSON response (bullets)
                  │
                  ▼
         Browser renders results

Design

Rather than implementing the entire application in one language, FYSummarizer uses PHP as a lightweight web layer while delegating data retrieval and LLM processing to a Python backend. Communication between the two components occurs through a JSON interface, allowing the AI pipeline to remain independent of the web frontend.

Tech Stack

Frontend

• HTML5
• CSS3
• Vanilla JavaScript (DOM manipulation, Fetch API)

Backend

• PHP (request routing and session management)
• Python 3.10 (data retrieval and summarization pipeline)

AI

• OpenAI API (configurable model)

Data Collection

• ScrapeBadger SDK for retrieving public posts from X

Development

• Git
• GitHub

Challenges

API Rate Limits

The free scraping tier limited requests to five per minute. To accommodate this restriction, searches were queued with delays between batches, influencing how multiple keywords and accounts could be processed in a single request.

Ambiguous Search Results

Keyword searches often returned unrelated people or topics with the same name. Prompt engineering and filtering strategies were refined to encourage the LLM to focus on the intended context instead of summarizing irrelevant posts.

LLM Context Window

Large collections of posts exceeded model context limits. The solution was a hierarchical summarization pipeline: summarize batches independently, then merge those summaries into a final result.

Reliable Structured Output

LLM responses occasionally returned incomplete or malformed JSON. The backend includes validation and error handling to detect parsing failures before returning results to the frontend.

Cross-Language Integration

The web application uses PHP while the AI pipeline is implemented in Python. User input is safely passed between the two using sanitized command-line arguments, and responses are exchanged through a JSON interface.

What I Learned

Building FYSummarizer taught me how to:

• Integrate multiple external APIs into a single application.
• Design multi-stage LLM workflows for large datasets.
• Bridge PHP and Python components through a clean JSON interface.
• Handle API rate limits and unreliable external services.
• Engineer prompts that improve summarization quality for noisy social media data.

Limitations

• Requires OpenAI API credentials.
• Requires ScrapeBadger credentials.
• Performance is constrained by external API rate limits.

Future Work

• Support additional social media platforms.
• Save previous summaries.
• Compare conversations over time.
• Allow users to customize summary length and style.