Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
135 changes: 65 additions & 70 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,35 @@
# JoblessYu

JoblessYu is a Discord bot built in Go that scrapes IT job listings from Vietnamese and global job boards (ITViec, Indeed, LinkedIn), classifies them with Groq AI, and serves them via an interactive, 100% ephemeral (`"Only you can see this"`) slash command with rich filtering and pagination.
JoblessYu is a high-performance Discord bot built in Go that scrapes IT job listings from Vietnamese and global job boards (ITViec, Indeed, LinkedIn), classifies them with Groq AI (`llama-3.1-8b-instant`), and serves them via an interactive, 100% ephemeral (`"Only you can see this"`) slash command with multi-keyword search, modal page jumps, and dual real-time Discord Hub status cards.

## Architecture

```
DAILY 5:00 AM ICT (automated cron)
DAILY 5:00 AM ICT (automated cron) / Manual Trigger (`make scrape`)
├── 1. SCRAPE
│ ├── Python jobspy → Indeed (40 jobs) + LinkedIn (40 jobs)
│ ├── Go Colly → ITViec (40 jobs, 24h freshness filter)
├── 1. SCRAPE (Target: 30 ITViec + 30 Indeed + 30 LinkedIn)
│ ├── Python jobspy → Indeed (30 jobs) + LinkedIn (30 jobs)
│ ├── Go Colly → ITViec (30 jobs, 24h freshness filter)
│ └── Cross-Site Deduplication (7-day window → merges alternate URLs)
├── 2. AI ENRICHMENT (Groq — llama-3.1-8b-instant)
│ ├── Classifies: level, type, expertise, tags, salary, remote, summary
│ ├── 1-job request loop with 18s throttle (~5,050 TPM safely under 6,000 TPM limit)
│ ├── Regex fallback only when Groq is unreachable (network errors)
│ ├── Jobs with empty descriptions (<50 chars) are deleted
│ └── Bilingual: handles Vietnamese + English + mixed JDs
├── 3. SERVING (/jobs command — 100% Ephemeral & Private)
├── 3. REAL-TIME DISCORD HUB (Dual Static Pinned Cards)
│ ├── Card 1: System Status (🟢 ONLINE / 🔴 OFFLINE, version, active job pool)
│ ├── Card 2: Scrape Summary (Timestamps, fresh roles, levels & location breakdown)
│ └── Hybrid DB Sync: Postgres LISTEN jobs_changed + 30s monitor + 500ms debounce
├── 4. SERVING (/jobs command — 100% Ephemeral & Private)
│ ├── Filter menu & job result cards are tagged "Only you can see this"
│ ├── Interactive filter panel: Level, Location, Position, Job Type
│ ├── Self-healing pagination cache with automatic DB recovery
│ ├── Modal Page Jump ("Go to Page") with self-healing DB fallback recovery
│ └── Multi-platform apply links (Indeed, LinkedIn, ITViec)
└── 4. WEEKLY CLEANUP (Monday 4:55 AM)
└── 5. WEEKLY CLEANUP (Monday 4:55 AM)
└── Auto-delete jobs older than 30 days
```

Expand Down Expand Up @@ -61,108 +66,98 @@ The AI classifies each job into one of 24 canonical IT expertise categories base

## Prerequisites

- **Go 1.26** or later
- **Python 3.13** (for jobspy scraper)
- **PostgreSQL Database** ([Neon](https://neon.tech/) — free tier)
- **Go 1.24** or later
- **Python 3.11+** (for jobspy scraper)
- **PostgreSQL Database** ([Neon](https://neon.tech/) — free tier serverless Postgres)
- **Groq API Key** ([GroqCloud](https://console.groq.com/) — free tier)
- **Discord Bot Token** ([Developer Portal](https://discord.com/developers/applications))

## Setup
## Setup & Deployment

### 1. Clone and install dependencies
### 1. Local Environment Setup

```bash
git clone https://github.com/Itea-Lab/JoblessYu.git
cd JoblessYu
go mod download
```

### 2. Python scraper setup

```bash
# On Linux/macOS:
# Set up Python virtual environment:
python3 -m venv scraper-python/.venv
source scraper-python/.venv/bin/activate
pip install -r scraper-python/requirements.txt

# On Windows (PowerShell):
py -3.13 -m venv scraper-python\.venv
.\scraper-python\.venv\Scripts\Activate.ps1
pip install -r scraper-python/requirements.txt
```

### 3. Environment variables

Copy `.env.example` to `.env` and fill in your credentials:
### 2. Environment Variables (`.env`)

```env
DISCORD_BOT_TOKEN=your_discord_bot_token
DISCORD_GUILD_ID=your_guild_id
DISCORD_CHANNEL_ID=your_hub_channel_id
DATABASE_URL=your_neon_postgres_connection_string
GROQ_API_KEY=your_groq_api_key
AI_MODEL=llama-3.1-8b-instant
JOB_RETENTION_DAYS=30
```

### 4. Database migrations
### 3. Database Migrations

```bash
make migrate
```

### 5. Run the bot
### 4. Running Docker Container (Cloud-Ready)

```bash
make bot
```
# Build multi-stage hybrid container:
docker build -t joblessyu .

## How it works

### Daily pipeline (5:00 AM ICT)
# Run container with HTTP /healthz probe on port 8080:
docker run -d --env-file .env -p 8080:8080 joblessyu
```

1. **Scrape (JobSpy + Colly)**
- Python JobSpy scrapes Indeed + LinkedIn (40 jobs each, posts from last 24 hours) using ITViec's canonical IT search query and non-IT title filters
- Go Colly scrapes ITViec (40 jobs, filtered by "Posted X ago" ≤ 24h)
- Cross-site deduplication computes `dedup_hash` (Company + Title + JobType) over a 7-day window, merging duplicate multi-platform URLs into `alternate_urls` JSONB
## How It Works

2. **AI Enrichment (Groq)**
- Batch enrichment processes all un-enriched jobs (2-hour window)
- Groq (`llama-3.1-8b-instant`) classifies: level (`Intern`, `Fresher`, `Junior`, `Senior`), type (`Full-time`, `Part-time`, `Contract`), expertise (24 categories), tags, salary, remote, summary
- JDs truncated to 1,500 chars to stay within Groq's free-tier TPM limit (8,000 TPM)
- 18s throttle between calls (~3.3 calls/min, ~5,050 TPM — 63% of limit)
- Regex fallback only when Groq is unreachable (network errors only)
- Jobs with empty descriptions (<50 chars) are deleted
- Bilingual: handles Vietnamese titles ("Chuyên viên" → Junior), experience phrases, skill inference
### 1. Dual-Card Discord Hub Architecture
- **Card 1 (`🟢 ONLINE` / `🔴 OFFLINE`)**: Displays real-time bot lifecycle status, current version (`v1.2.0`), retention policy, and active job pool size. Automatically switches to `🔴 OFFLINE` when gracefully stopped.
- **Card 2 (`🌅 SCRAPE SUMMARY`)**: Displays user-centric job insights:
- **Timestamps**: Last scrape time & next scheduled 05:00 AM ICT scrape.
- **Fresh Roles Today**: Count of new job listings ingested today.
- **Experience Level Breakdown**: `🎓 Intern / Fresher`, `🌱 Junior`, `🚀 Senior`, `⚡ Lead / Manager`.
- **Top Locations**: `🏙️ Ho Chi Minh`, `🏛️ Ha Noi`, `🌊 Da Nang`, `💻 Remote`.

3. **Serving (/jobs command — 100% Ephemeral & Private)**
- Pure DB read — instant response, no AI calls during user interaction
- Responses are tagged `"Only you can see this"` to preserve user privacy and keep channels clean
- Filter dropdowns: Position (24 categories), Experience Level (`Intern`, `Fresher`, `Junior`, `Senior`, `All`), Location (`Ho Chi Minh`, `Ha Noi`, `All`), Job Type (`Full-time`, `Part-time`, `Contract`, `All`)
- Interactive pagination with Prev/Next buttons and self-healing DB fallback recovery
### 2. Real-Time Hybrid Database Synchronization
- Uses Postgres trigger `notify_jobs_changed()` (`LISTEN jobs_changed`) combined with a 30s fail-safe ticker and 500ms debouncer.
- Automatically updates Discord status cards whenever job records are added, updated, or deleted.

4. **Weekly cleanup (Monday 4:55 AM)**
- Jobs older than 30 days auto-deleted to keep DB lean (Neon free tier)
### 3. Interactive Ephemeral Job Search (`/jobs`)
- Pure DB query — 100% ephemeral (`"Only you can see this"`).
- Multi-select checkbox filtering by Level, Location, Position, and Job Type (compact fixed-height layout).
- Ephemeral pagination with `◀️ Prev`, `Page X/Y`, `Next ▶️`, and `🔢 Go to Page` modal jump.

## Make commands
## Make Commands

```bash
make bot # Start Discord bot + cron scheduler
make scrape # Full pipeline: jobspy (Indeed+LinkedIn) + Colly (ITViec) + AI enrichment
make enrich # AI enrichment only (processes un-enriched DB jobs)
make test # Run Go tests with race detector + coverage
make lint # Run Go vet + Python ruff
make migrate # Apply all SQL migrations to Neon DB
make clean # Remove build artifacts (bot binary, jobs.json)
make bot # Start Discord bot + cron scheduler + real-time DB listener
make dev # Alias for `make bot`
make scrape # Full pipeline: jobspy (Indeed+LinkedIn) + Colly (ITViec) + AI enrichment
make enrich # AI enrichment only (processes un-enriched DB jobs)
make test # Run Go tests with race detector + coverage
make test-notify # Test Discord notifications & UI cards (Online/Offline status, Scrape Summary)
make eval-ai # Run AI Evaluation & Hallucination Benchmark Suite
make lint # Run Go vet + Python ruff static analysis
make lint-go # Run Go vet static analysis
make lint-python # Run ruff check on Python scraper scripts
make migrate # Apply SQL migrations to Neon DB
make clean # Remove build artifacts
```

## Tech stack
## Tech Stack

| Component | Technology | Free tier |
| Component | Technology | Free Tier Capabilities |
|---|---|---|
| Bot | Go + discordgo | — |
| Scraper (Indeed + LinkedIn) | Python + jobspy | — |
| Scraper (ITViec) | Go + Colly | — |
| AI enrichment | Groq (llama-3.1-8b-instant) | 30 RPM, 8K TPM, 1K RPD |
| Database | Neon Postgres (serverless) | 0.5 GB storage |
| Scheduler | robfig/cron (Go) | — |
| CI | GitHub Actions (Go test + Python lint) | 2,000 min/month |
| Bot Gateway | Go + discordgo | Dual static pinned cards + ephemeral components |
| Scrapers | Python JobSpy (Indeed, LinkedIn) + Go Colly (ITViec) | 30/30/30 target scrape distribution |
| AI Enrichment | Groq (`llama-3.1-8b-instant`) | 1-job request loop @ 18s delay (~5,050 TPM) |
| Database | Neon PostgreSQL (Serverless) | GIN Trigram indexes (`pg_trgm`) + LISTEN/NOTIFY |
| Container | Docker Multi-stage (Go 1.24 static + Python 3.11) | HTTP `/healthz` probe on port 8080 |
| CI | GitHub Actions | Automated Go test + static analysis |
17 changes: 17 additions & 0 deletions docs/AI/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,23 @@
> Emergency backup context. Grand scheme from the foundation.
> Roll up progress-log entries here at slice boundaries so context survives session resets.

## [Slice O] — 2026-08-11

### Summary
Multi-Checkbox `/jobs` Search UI/UX, Fixed-Height Select Menu Layout & `skills.md` Alignment. Upgraded `/jobs` filter dropdowns (`select_position`, `select_level`, `select_location`, `select_type`) to use multi-select checkboxes (`MinValues: 0`, `MaxValues: 3..5`). Shortened chip labels (`Web Dev`, `Mobile Dev`, `IT Management`) and moved detailed role descriptions into `Description` fields inside dropdown items to keep select menu height fixed (~40px single-line) without flexing or vertical expanding. Enforced mutual exclusivity for "All" in `normalizeSelectionValues`. Expanded Seniority Levels (`Intern`, `Fresher`, `Junior`, `Middle`, `Senior`, `Lead / Manager`) to match `skills.md` and Locations (`Ho Chi Minh`, `Ha Noi`, `Da Nang`, `Remote / WFH`) to match Card 2 metrics. Built dynamic slice-based SQL query generator (`FetchRawJobs`) in `store.go`. Fully documented all 12 Makefile targets in `README.md`.

### Files touched
- modified: `internal/bot/embeds.go` (configured multi-select select menus with short chip labels, rich descriptions, and optimal `MaxValues`)
- modified: `internal/bot/handlers.go` (updated `handleSweeperComponent` and `normalizeSelectionValues` to handle multi-select choices and mutual exclusivity)
- modified: `internal/job/entry.go` (expanded `JobQuery` struct with slice fields `Levels`, `JobTypes`, `Locations`, `Expertises`)
- modified: `internal/job/store.go` (updated `FetchRawJobs` query builder for multi-value SQL filtering)
- modified: `internal/bot/embeds_test.go` (added unit tests for multi-select component generation and slice label formatters)
- modified: `README.md` (documented all 12 Makefile targets)
- modified: `docs/AI/progress-log.md` (recorded Slice O progress)
- modified: `docs/AI/changelog.md` (this entry)

---

## [Slice N] — 2026-08-11

### Summary
Expand Down
27 changes: 27 additions & 0 deletions docs/AI/progress-log.md
Original file line number Diff line number Diff line change
Expand Up @@ -513,11 +513,38 @@ Major architecture overhaul: replaced Firecrawl with hybrid jobspy+Colly scraper
- Expanded `GetPipelineSummaryStats` ILIKE queries in `store.go` to use multi-keyword recognition matching Vietnamese diacritics (`Hà Nội`, `HN`, `Hồ Chí Minh`, `HCM`, `SG`, `Đà Nẵng`, `Remote`, `Junior / Mid`, `Senior`, `Lead`), ensuring 100% database categorization accuracy.

### Verification
- `make eval-ai`: PASS (100% Level, 100% Expertise, 100% Location, 0% Hallucination Rate — 6/6 test cases passed).
- `go test -v -race -cover ./...`: PASS (100% test pass rate across all packages).
- `go vet ./...`: PASS (0 warnings / 0 errors).
- `make scrape`: PASS (Updated Card 1 & Card 2 live in Discord).
- `Migration 007`: Applied & active on Neon PostgreSQL DB.

---

## 2026-08-11 — Multi-Checkbox `/jobs` Search UI/UX, Fixed-Height Select Layout & `skills.md` Alignment (Slice O)

### Completed
- **Multi-Select Checkbox Dropdowns (`internal/bot/embeds.go`, `internal/bot/handlers.go`)**:
- Configured `select_position`, `select_level`, `select_location`, and `select_type` select menus with `MinValues: &zero` and `MaxValues: 3..5` to render native checkboxes inside Discord dropdowns.
- Enabled checking multiple options per dropdown in a single interaction without individual selection delays or UI lockups.
- **Fixed-Height Compact UI Box Layout (`internal/bot/embeds.go`)**:
- Shortened option labels shown inside selected chips (e.g. `Web Dev`, `Mobile Dev`, `DevOps & SRE`, `IT Management`), keeping checked tags on a single clean line (~40px fixed height) without vertical flexing.
- Moved rich role details and tech keywords to `Description` fields inside dropdown items.
- Enforced mutual exclusivity for "All" options in `normalizeSelectionValues` to prevent "All" from displaying alongside specific checked tags.
- **`skills.md` & Card 2 Metrics Alignment (`internal/bot/embeds.go`)**:
- Expanded Seniority Levels to include `Intern`, `Fresher`, `Junior`, `Middle`, `Senior`, and `Lead / Manager` (matching `skills.md`).
- Expanded Locations to include `Ho Chi Minh`, `Ha Noi`, `Da Nang`, and `Remote / WFH` (matching Card 2 metrics).
- **Dynamic Multi-Value SQL Query Engine (`internal/job/entry.go`, `internal/job/store.go`, `internal/bot/handlers.go`)**:
- Added slice fields (`Levels`, `JobTypes`, `Locations`, `Expertises`) to `JobQuery`.
- Updated `FetchRawJobs` query builder in `store.go` to construct parameterized `IN (...)` and combined `OR` queries across PostgreSQL.
- **Documentation & Makefile Alignment (`README.md`, `Makefile`)**:
- Updated `README.md` to document all 12 Makefile targets (`make bot`, `make dev`, `make scrape`, `make enrich`, `make test`, `make test-notify`, `make eval-ai`, `make lint`, `make lint-go`, `make lint-python`, `make migrate`, `make clean`).

### Verification
- `go test -v -race -cover ./...`: PASS (100% test pass rate across all packages).
- `go vet ./...`: PASS (0 warnings / 0 errors).
- `make eval-ai`: PASS (100% Level, 100% Expertise, 100% Location, 0% Hallucination Rate — 6/6 test cases passed).




8 changes: 4 additions & 4 deletions internal/bot/bot.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,10 @@ type cachedJobs struct {
}

type criteriaState struct {
PositionValue string
LevelValue string
LocationValue string
JobTypeValue string
Positions []string
Levels []string
Locations []string
JobTypes []string
}

type cachedCriteria struct {
Expand Down
Loading
Loading