Distributed Text Analysis on AWS

A distributed system that runs heavy NLP analysis (POS tagging, dependency parsing, constituency parsing with the Stanford Parser) over large text files, using a manager–worker architecture on AWS EC2, S3 and SQS.

Several local clients can submit jobs at the same time. A manager instance splits each job into tasks, scales EC2 workers up to match the load, and returns a single HTML summary to each client.

Authors: Stav Balaish (@stavb1510) · Kfir Bar Kochva (@kfirbr2) Built for the Distributed Systems Programming course at Ben-Gurion University (2025).

---

Architecture

Local App ──NEW_JOB──▶ [Manager queue] ──▶ Manager (EC2)
    ▲                                         │  splits input into tasks
    │                                         ▼
    │                                 [Worker task queue] ──▶ Workers (EC2 × N)
    │                                         ▲                    │ Stanford NLP
    │                                         └──── results ◀──────┘ → S3
    └──JOB_DONE── [per-client reply queue] ◀── Manager builds HTML summary

Local application (client)

• Starts a manager instance if none is running (bootstrapped with an EC2 User Data script).
• Uploads the input file to S3 under a unique key.
• Creates its own temporary SQS reply queue and sends NEW_JOB with that queue's URL.

Manager

• Multi-threaded loop (ExecutorService) that handles new jobs and worker reports concurrently, so a large job never blocks the others.
• Pushes one WORKER_TASK per input line to a shared queue and launches workers according to the n tasks-per-worker ratio (capped at 18 workers + 1 manager).
• Tracks every job's state in a ConcurrentHashMap; when a job completes, it builds the HTML summary with presigned S3 URLs, so the bucket never has to be public.

Workers

• On boot: install Java, pull the code from S3 and start polling.
• Run the Stanford Parser with a 6 GB heap, upload the result to S3 and report back over SQS.

Design decisions

• Reply-queue pattern for multiple clients. Each client listens on its own queue, so one client can never consume another client's JOB_DONE message.
• Fault tolerance via SQS visibility timeout. A 60-minute timeout means a task held by a worker that crashed or hung becomes visible again and is picked up by another worker.
• No credentials on EC2. Manager and workers authenticate through an IAM instance profile; no keys are passed to the instances or stored in scripts.
• Thin JAR deployment. Dependencies (~150 MB) are uploaded to S3 once; only the small application JAR is uploaded on each run, and the instances build the classpath at startup.
• Instance type m4.large — the 8 GB of RAM are needed for constituency parsing of whole books.

Running

Requirements: JDK 8+, Maven, AWS credentials in ~/.aws/credentials.

mvn clean package
java -jar target/distributedtextanalysis-1.0-SNAPSHOT.jar <inputFile> <outputFile.html> <n> [terminate]

Argument	Meaning
inputFile	Local file with one <analysis type>\t<text URL> per line (see input-sample.txt)
outputFile.html	Name of the HTML summary to write
n	Tasks per worker — the manager launches tasks / n workers
terminate	Optional: shut down all workers and the manager when done

The summary is an HTML page that links each input line to its analysis result through a presigned S3 URL.

Results

• Sample input: ~2 minutes. Full input (Project Gutenberg books): 20–30 minutes with n = 1.
• Two clients running in parallel each received the correct summary.
• terminate cleanly shut down all workers and the manager.

Tech

Java 8 · Maven · AWS SDK (EC2, S3, SQS, IAM) · Stanford CoreNLP / Parser