diff --git a/auth0-react-router/SKILL.md b/auth0-react-router/SKILL.md new file mode 100644 index 0000000..328fb10 --- /dev/null +++ b/auth0-react-router/SKILL.md @@ -0,0 +1,212 @@ +--- +name: auth0-react-router +description: Use when adding Auth0 login and logout to React Router v7 web applications - integrates @auth0/auth0-react-router SDK for React Router v7 regular web apps with server-side session management, JWE-encrypted cookies, middleware-based route protection, and optional SPA mode +--- + +# Auth0 React Router Integration + +Adds Auth0 authentication to [React Router v7](https://reactrouter.com/) framework-mode applications using `@auth0/auth0-react-router`. The SDK operates in **RWA mode** by default (server-side OIDC with JWE-encrypted session cookies) and optionally in **SPA mode** (client-side PKCE via `@auth0/auth0-spa-js`) when `VITE_AUTH0_*` env vars are present. + +> **Agent instruction:** Before providing SDK setup instructions, fetch the latest release version: +> ```bash +> gh api repos/auth0/auth0-react-router/releases/latest --jq '.tag_name' 2>/dev/null \ +> || gh api repos/auth0/auth0-react-router/releases --jq '.[0].tag_name' +> ``` +> The first command returns the latest stable release. If it returns an error (pre-release only), fall back to the second command which returns the most recent release including pre-releases. Use the returned version in all dependency lines. + +## Prerequisites + +- Node.js 18+ +- React Router v7 (framework mode — `react-router.config.ts` present) +- React ≥ 18.0.0 +- npm 9+ or equivalent +- Node.js 20+ for the bootstrap setup script + +## When NOT to Use + +| Use Case | Recommended Skill | +|----------|------------------| +| React Router used as a SPA (no SSR/server) | Use Auth0 React SDK (`@auth0/auth0-react`) | +| Next.js application | Use `auth0-nextjs` skill | +| Express/Fastify backend API (JWT bearer only) | Use `auth0-node-api` skill | +| Plain React SPA (Vite/CRA, not React Router) | Use `auth0-react` skill | +| Adding Auth0 to a React Router SPA without `react-router.config.ts` | Use SPA SDK instead | + +## Quick Start Workflow + +> **Agent instruction:** +> +> 1. **Check for credentials in the prompt first.** If the user's prompt contains an Auth0 domain, client ID, and client secret, use them directly — skip the bootstrap script and write `.env` manually. Do NOT ask for credentials that are already provided. +> +> 2. **Otherwise offer setup choice:** +> Use `AskUserQuestion`: "How would you like to configure Auth0?" +> - Option A: "Automatic setup (recommended)" — runs `scripts/bootstrap.mjs` +> - Option B: "Manual setup" — provide credentials manually +> +> 3. **Install the SDK:** +> ```bash +> npm install @auth0/auth0-react-router +> ``` +> +> 4. **Create Auth0Server instance** (`app/auth0.server.ts`): +> ```ts +> import { Auth0Server, registerAuth0Instance } from '@auth0/auth0-react-router/server' +> export const auth0 = new Auth0Server() +> registerAuth0Instance(auth0) +> ``` +> +> 5. **Register auth routes** — add to `app/routes.ts`: +> ```ts +> route('auth/*', 'routes/auth.$.tsx'), +> ``` +> Create `app/routes/auth.$.tsx`: +> ```ts +> import { handleAuth } from '@auth0/auth0-react-router/server' +> import { auth0 } from '../auth0.server' +> export const loader = ({ request }) => handleAuth(auth0, request) +> export const action = ({ request }) => handleAuth(auth0, request) +> ``` +> +> 6. **Wrap app in Auth0Provider** in `app/root.tsx`: +> ```tsx +> import { Auth0Provider } from '@auth0/auth0-react-router' +> import { rootAuthLoader } from '@auth0/auth0-react-router/server' +> import { auth0 } from './auth0.server' +> export const loader = ({ request }) => rootAuthLoader(auth0, request) +> export default function Root() { +> return ( +> +> +> +> ) +> } +> ``` +> If using a custom route config, ensure the root route has `id: 'root'`. +> +> 7. **Verify build:** +> ```bash +> npm run build +> ``` +> If it fails, check that `app/root.tsx` has `id: 'root'`, that `Auth0Provider` is not imported from `/server`, and that `.env` is present with all required vars. +> +> 8. **Failcheck:** If verification fails after 5–6 iterations, use `AskUserQuestion` to ask the user whether to continue troubleshooting or document the blocker. + +## Detailed Documentation + +- **[Setup Guide](./references/setup.md)** — Auth0 Dashboard configuration, bootstrap script, `.env` setup, secret management, and verification steps +- **[Integration Patterns](./references/integration.md)** — Protected routes, session/token utilities, middleware, role-based auth, API bearer tokens, SPA mode, error handling, and testing +- **[API Reference & Testing](./references/api.md)** — All env vars, session helpers, middleware API, error types, testing checklist, and security considerations + +## Common Mistakes + +| Mistake | Fix | +|---------|-----| +| App type in Auth0 Dashboard set to **SPA** instead of **Regular Web Application** | Create a new application of type "Regular Web Application" | +| Missing `AUTH0_SESSION_SECRET` (must be ≥ 32 chars) | Generate with `openssl rand -base64 32` and add to `.env` | +| `AUTH0_CLIENT_SECRET` committed to source control | Move to `.env`, add `.env` to `.gitignore` | +| `Auth0Provider` imported from `/server` instead of `/client` | Use `import { Auth0Provider } from '@auth0/auth0-react-router'` | +| `rootAuthLoader` called without passing `auth0` instance | Signature is `rootAuthLoader(auth0, request)` | +| Root route missing `id: 'root'` in custom route configs | Add `id: 'root'` to the root route definition | +| Callback URL mismatch (e.g., port 3000 vs 5173) | Check `AUTH0_APP_BASE_URL` and Auth0 Dashboard Allowed Callback URLs | +| Setting both `AUTH0_*` and `VITE_AUTH0_*` env vars | Use exactly one mode — RWA uses `AUTH0_*`, SPA uses `VITE_AUTH0_*` | +| `getAccessToken` throwing `TokenError` on expiry | Add `offline_access` to scope + enable Refresh Token grant in Auth0 Dashboard | +| Domain includes `https://` prefix | Use hostname only: `example.us.auth0.com` (no scheme) | + +## Dual-Mode Architecture + +The SDK auto-detects the authentication mode at render time from environment variables: + +| Mode | Activation | Token Location | Suitable For | +|------|-----------|---------------|-------------| +| **RWA** (default) | `AUTH0_*` env vars | Server-side JWE cookie | SSR apps, sensitive data | +| **SPA** | `VITE_AUTH0_*` env vars | Browser (memory or localStorage) | Client-heavy apps | + +**Do not set both sets of env vars** — the modes are mutually exclusive. + +In RWA mode, `rootAuthLoader` returns a `BrowserSession` (user profile only; no tokens ever reach the browser). In SPA mode, `@auth0/auth0-spa-js` handles token storage client-side. + +## Entry Points + +The package has 6 tree-shakeable entry points to enforce bundle boundaries: + +| Entry | Safe For | Contents | +|-------|----------|----------| +| `@auth0/auth0-react-router` | Browser | Alias for `/client` | +| `/client` | Browser | `Auth0Provider`, hooks, components | +| `/server` | Server only | Handlers, session utils, middleware | +| `/errors` | Both | Typed error classes | +| `/types` | Both | TypeScript interfaces | +| `/testing` | Tests | Mock factories, `WithAuth` | + +**Critical:** Never import from `/server` inside client-side code. React Router's bundler tree-shakes it away, but incorrect imports can leak server code into the browser bundle. + +## Related Skills + +- **[auth0-quickstart](/auth0-quickstart)** — Initial Auth0 account and tenant setup +- **[auth0-nextjs](/auth0-nextjs)** — Next.js App Router integration (similar SSR pattern) +- **[auth0-aspnetcore-authentication](/auth0-aspnetcore-authentication)** — WEB_REGULAR reference for .NET + +## Quick Reference + +### Server Utilities (`@auth0/auth0-react-router/server`) + +| Function | Signature | Returns | +|----------|-----------|---------| +| `handleAuth` | `(auth0, request)` | Response (login/callback/logout dispatch) | +| `handleLogin` | `(auth0, request, opts?)` | Response | +| `handleCallback` | `(auth0, request, opts?)` | Response | +| `handleLogout` | `(auth0, request, opts?)` | Response | +| `rootAuthLoader` | `(auth0, request)` | `{ session: BrowserSession \| null }` | +| `getSession` | `(request)` | `Auth0Session \| null` | +| `requireSession` | `(request)` | `Auth0Session` (throws 302 if not auth'd) | +| `getUser` | `(request)` | `Auth0User \| null` | +| `requireUser` | `(request)` | `Auth0User` (throws 302 if not auth'd) | +| `getAccessToken` | `(request)` | `string` (auto-refreshes; throws `TokenError` on failure) | +| `updateSession` | `(request, session)` | `Response` with updated cookie | +| `deleteSession` | `(request, opts?)` | `Response` clearing session | +| `createApiClient` | `(request, opts)` | `fetch`-based client with `Authorization` header | +| `requireClaims` | `(request, opts?)` | `JWTPayload` (bearer token; 401/403 on failure) | +| `getClaims` | `(request)` | `JWTPayload \| null` | + +### Middleware (`@auth0/auth0-react-router/server`) + +| Export | Purpose | +|--------|---------| +| `auth0Middleware` | Global middleware — populates `auth0SessionContext`, `auth0UserContext` | +| `defineRouteAuth(opts)` | Per-route middleware factory — `{ middleware }`, throws 403 on role mismatch | +| `bearerTokenMiddleware` | API routes — validates Bearer token, populates `auth0ClaimsContext` | +| `auth0SessionContext` | Context key for `Auth0Session` | +| `auth0UserContext` | Context key for `Auth0User` | +| `auth0ClaimsContext` | Context key for `JWTPayload` | + +### Client Components/Hooks (`@auth0/auth0-react-router` or `/client`) + +| Export | Purpose | +|--------|---------| +| `Auth0Provider` | Root provider — wraps entire app | +| `useAuth0()` | Full auth context (includes `getAccessToken` in SPA mode) | +| `useUser()` | Current `Auth0User \| null` | +| `useSession()` | Full `Auth0Session \| null` | +| `SignedIn` / `SignedOut` | Conditional rendering | +| `AuthLoading` | Renders during SPA init | +| `RequireAuth` | Client-side redirect guard | +| `RequireRole` | Role-based conditional render | +| `LoginButton` | Renders `` | +| `LogoutButton` | Renders `` | +| `Auth0ErrorBoundary` | Catches `Auth0Error` subclasses | +| `withAuthenticationRequired(Component, opts)` | HOC redirect guard | + +### Error Classes (`@auth0/auth0-react-router/errors`) + +All extend `Auth0Error` with `.code` (string) and `.statusCode` (number). + +`AuthenticationError` · `SessionExpiredError` · `MissingSessionError` · `TokenError` · `BearerTokenError` · `CallbackError` · `InsufficientScopeError` · `ConfigurationError` + +## References + +- [GitHub Repository](https://github.com/auth0/auth0-react-router) +- [npm Package](https://www.npmjs.com/package/@auth0/auth0-react-router) +- [Auth0 Dashboard](https://manage.auth0.com/) +- [Auth0 Documentation](https://auth0.com/docs) +- [React Router v7 Docs](https://reactrouter.com/start/framework/installation) +- [Report Issues](https://github.com/auth0/auth0-react-router/issues) diff --git a/auth0-react-router/evals/PROMPT.md b/auth0-react-router/evals/PROMPT.md new file mode 100644 index 0000000..db9e714 --- /dev/null +++ b/auth0-react-router/evals/PROMPT.md @@ -0,0 +1,18 @@ +--- +skills: auth0-react-router +--- + +## Agent System + +You are a software developer adding Auth0 authentication to a React Router v7 application. +You have access to tools for reading/writing files, running commands, and fetching URLs. +Use these tools to complete the integration task below. + +## Task + +Add Auth0 authentication to a React Router v7 application using the @auth0/auth0-react-router SDK. + +**Auth0 Credentials:** +- Domain: `dev-example.auth0.com` +- Client ID: `abc123def456ghi789jkl012` +- Client Secret: `sample_secret_def456uvw789` diff --git a/auth0-react-router/evals/benchmark-config.json b/auth0-react-router/evals/benchmark-config.json new file mode 100644 index 0000000..61bae42 --- /dev/null +++ b/auth0-react-router/evals/benchmark-config.json @@ -0,0 +1,49 @@ +{ + "metadata": { + "skill_name": "auth0-react-router", + "sdk_type": "WEB_REGULAR", + "framework": "React Router v7", + "language": "TypeScript", + "package": "@auth0/auth0-react-router", + "generated_by": "quickstart-skill-generator" + }, + "configurations": { + "baseline": { + "name": "Baseline (no tools)", + "description": "Single LLM call, no tools, no skill — pure training data knowledge", + "skill_context": "none", + "tools": "none" + }, + "without_skill": { + "name": "Agent Only", + "description": "Agent has tools but no skill context", + "skill_context": "none", + "tools": "full" + }, + "with_skill": { + "name": "Agent + Skill", + "description": "Agent has tools and generated SKILL.md in context", + "skill_context": "full", + "tools": "full" + } + }, + "scaffold": { + "sdk_type": "WEB_REGULAR", + "framework": "React Router v7", + "language": "TypeScript" + }, + "scoring": { + "dimensions": { + "correctness": 0.20, + "hallucination": 0.15, + "security": 0.15, + "skill_differentiation": 0.30, + "semantic": 0.20 + }, + "thresholds": { + "skill_valuable": { "delta_gte": 0.30 }, + "skill_acceptable": { "with_skill_pass_rate_gte": 0.85 }, + "needs_improvement": { "delta_lt": 0.15 } + } + } +} diff --git a/auth0-react-router/evals/evals.json b/auth0-react-router/evals/evals.json new file mode 100644 index 0000000..5982535 --- /dev/null +++ b/auth0-react-router/evals/evals.json @@ -0,0 +1,33 @@ +{ + "skill_name": "auth0-react-router", + "evals": [ + { + "id": 1, + "prompt": "Add Auth0 authentication to a React Router v7 application using the @auth0/auth0-react-router SDK.\n\n**Auth0 Credentials:**\n- Domain: `dev-example.auth0.com`\n- Client ID: `abc123def456ghi789jkl012`\n- Client Secret: `sample_secret_def456uvw789`", + "expected_output": "Working React Router v7 web app with Auth0 middleware, session management, login/logout, and protected routes", + "expectations": [ + "Auth0 SDK installed via any valid package manager", + "Has correct import from @auth0/auth0-react-router package", + "Auth0Server initialized with correct constructor pattern", + "Auth0Provider wrapper present in root component (required for client-side session hydration)", + "Auth middleware route handlers registered (handleAuth or individual handlers)", + "Login route or handler present", + "Logout route or handler present", + "Session management configured (rootAuthLoader or session helpers used)", + "Protected route or authentication guard present", + "Auth0 domain dev-example.auth0.com written to .env config file", + "Client ID abc123def456ghi789jkl012 written to .env config file", + "Does not use SPA-only SDK (@auth0/auth0-react or auth0-js) in a server-rendered React Router app", + "Client secret not hardcoded in source (must be in .env only)", + "Does not store tokens in localStorage", + "Correctly integrates Auth0 into React Router v7 with middleware, session, and protected routes", + "Uses current SDK version (1.0.0-beta.2) in dependency declaration", + "Calls registerAuth0Instance(auth0) to enable standalone session helpers", + "AUTH0_SESSION_SECRET env var present (required for JWE cookie encryption, minimum 32 chars)", + "Uses React Router splat route pattern for auth callback (auth/* or auth.$.tsx)", + "Does not use deprecated Auth0 packages (passport-auth0, express-jwt, nextjs-auth0, Lock)", + "Skill-specific advanced patterns present: version declaration, registerAuth0Instance, session secret, or splat route" + ] + } + ] +} diff --git a/auth0-react-router/evals/graders.json b/auth0-react-router/evals/graders.json new file mode 100644 index 0000000..351548d --- /dev/null +++ b/auth0-react-router/evals/graders.json @@ -0,0 +1,113 @@ +[ + { + "type": "matches", + "pattern": "npm install @auth0/auth0-react-router|yarn add @auth0/auth0-react-router|pnpm add @auth0/auth0-react-router", + "description": "Auth0 SDK installed via any valid package manager" + }, + { + "type": "matches", + "pattern": "from ['\"]@auth0/auth0-react-router(/server|/errors|/types|/testing)?['\"]", + "description": "Has correct import from @auth0/auth0-react-router package" + }, + { + "type": "contains_any", + "values": ["new Auth0Server(", "Auth0Server({"], + "description": "Auth0Server initialized with correct constructor pattern" + }, + { + "type": "contains", + "value": "Auth0Provider", + "description": "Auth0Provider wrapper present in root component (required for client-side session hydration)" + }, + { + "type": "matches", + "pattern": "handleAuth|handleLogin.*handleCallback.*handleLogout", + "description": "Auth middleware route handlers registered (handleAuth or individual handlers)" + }, + { + "type": "matches", + "pattern": "handleLogin|/auth/login|route.*auth", + "description": "Login route or handler present" + }, + { + "type": "matches", + "pattern": "handleLogout|/auth/logout", + "description": "Logout route or handler present" + }, + { + "type": "matches", + "pattern": "rootAuthLoader|AUTH0_SESSION_SECRET|getSession|requireSession", + "description": "Session management configured (rootAuthLoader or session helpers used)" + }, + { + "type": "matches", + "pattern": "requireSession|requireUser|RequireAuth|withAuthenticationRequired|defineRouteAuth", + "description": "Protected route or authentication guard present" + }, + { + "type": "file_contains", + "file_pattern": ".env", + "value": "dev-example.auth0.com", + "description": "Auth0 domain written to .env config file" + }, + { + "type": "file_contains", + "file_pattern": ".env", + "value": "abc123def456ghi789jkl012", + "description": "Client ID written to .env config file" + }, + { + "type": "not_contains_any", + "values": ["@auth0/auth0-react\"", "@auth0/auth0-react'", "from 'auth0-js'", "from \"auth0-js\""], + "description": "Does not use SPA-only SDK (@auth0/auth0-react or auth0-js) in a server-rendered React Router app" + }, + { + "type": "not_contains", + "value": "AUTH0_CLIENT_SECRET=\"sample_secret", + "description": "Client secret not hardcoded in source (must be in .env only)" + }, + { + "type": "not_contains", + "value": "localStorage.setItem", + "description": "Does not store tokens in localStorage" + }, + { + "type": "judge", + "question": "Does the solution correctly integrate Auth0 into a React Router v7 web application with server-side authentication handlers, session management, and protected routes?", + "examples": "PASS: Uses Auth0Server with handleAuth for login/callback/logout routes, uses rootAuthLoader in root.tsx with Auth0Provider, protects routes with requireSession or requireUser.\nFAIL: Uses SPA-style token flow with @auth0/auth0-react instead of server-side sessions.\nFAIL: Stores client secret in source code instead of .env.\nFAIL: Missing Auth0Provider or rootAuthLoader in root.tsx.", + "framework": "react-router", + "description": "Correctly integrates Auth0 into React Router v7 with middleware, session, and protected routes" + }, + { + "type": "matches", + "pattern": "auth0-react-router.*1\\.0\\.0-beta\\.2|1\\.0\\.0-beta\\.2.*auth0-react-router|\"@auth0/auth0-react-router\": \"\\^?1\\.0\\.0-beta\\.2\"", + "description": "Uses current SDK version (1.0.0-beta.2) in dependency declaration" + }, + { + "type": "matches", + "pattern": "registerAuth0Instance", + "description": "Calls registerAuth0Instance(auth0) to enable standalone session helpers" + }, + { + "type": "matches", + "pattern": "AUTH0_SESSION_SECRET", + "description": "AUTH0_SESSION_SECRET env var present (required for JWE cookie encryption, minimum 32 chars)" + }, + { + "type": "matches", + "pattern": "/auth/callback|auth/\\*|auth\\.\\$\\.tsx", + "description": "Uses React Router splat route pattern for auth callback (auth/* or auth.$.tsx)" + }, + { + "type": "not_contains_any", + "values": ["passport-auth0", "express-jwt", "@auth0/nextjs-auth0", "auth0-lock"], + "description": "Does not use deprecated Auth0 packages (passport-auth0, express-jwt, nextjs-auth0, Lock)" + }, + { + "type": "judge", + "question": "Check ONLY for these skill-specific advanced patterns (NOT basic SDK usage like handleAuth, Auth0Provider, getSession, SignedIn, SignedOut). Look for at least 2 of: (1) SDK version 1.0.0-beta.2 in a package.json dependency declaration, (2) registerAuth0Instance(auth0) called after Auth0Server instantiation, (3) AUTH0_SESSION_SECRET in .env or referenced in code, (4) /auth/callback or auth/* splat route pattern. These are advanced patterns taught by the skill that are NOT part of standard Auth0 quickstart knowledge. Answer YES only if at least 2 of these advanced patterns are present, NO otherwise.", + "examples": "PASS: Uses version 1.0.0-beta.2 AND calls registerAuth0Instance AND has AUTH0_SESSION_SECRET in .env.\nFAIL: Only uses basic handleAuth with no registerAuth0Instance or SESSION_SECRET.", + "framework": "react-router", + "description": "Skill-specific advanced patterns present: version declaration, registerAuth0Instance, session secret, or splat route" + } +] diff --git a/auth0-react-router/evals/graders.ts b/auth0-react-router/evals/graders.ts new file mode 100644 index 0000000..9ff0186 --- /dev/null +++ b/auth0-react-router/evals/graders.ts @@ -0,0 +1,98 @@ +/** + * Compatibility shim for auth0-evals. + * + * Reads graders.json (rich format with custom types, tier, description) + * and exports defineGraders() returning auth0-evals GraderDef[] format. + * + * Custom type mappings: + * file_contains → contains (loses file-specificity) + * contains_any → contains (first value only) + * not_contains_any → not_contains (first value only) + * all → flattened sub-graders + * judge.examples → stripped + */ +import { readFileSync } from "fs"; +import { join, dirname } from "path"; +import { fileURLToPath } from "url"; + +const __dirname = dirname(fileURLToPath(import.meta.url)); + +/** auth0-evals GraderDef — the format runGraders() expects */ +export interface GraderDef { + kind: string; + name: string; + needle?: string; + pattern?: string; + question?: string; + framework?: string; +} + +/** Rich grader from graders.json (superset of auth0-evals types) */ +interface RichGrader { + type: string; + value?: string; + values?: string[]; + pattern?: string; + description?: string; + question?: string; + examples?: string; + framework?: string; + file_pattern?: string; + tier?: number; + graders?: RichGrader[]; +} + +function mapGrader(g: RichGrader): GraderDef | GraderDef[] { + const name = g.description ?? ""; + + switch (g.type) { + case "contains": + return { kind: "contains", needle: g.value, name }; + + case "file_contains": + return { kind: "contains", needle: g.value, name }; + + case "contains_any": + return { kind: "contains", needle: g.values?.[0], name }; + + case "not_contains": + return { kind: "not_contains", needle: g.value, name }; + + case "not_contains_any": + return { kind: "not_contains", needle: g.values?.[0], name }; + + case "matches": + return { kind: "matches", pattern: g.pattern, name }; + + case "all": + return (g.graders ?? []).flatMap((sub) => { + const mapped = mapGrader(sub); + return Array.isArray(mapped) ? mapped : [mapped]; + }); + + case "judge": + return { + kind: "judge", + question: g.question, + framework: g.framework, + name: g.question?.slice(0, 80) ?? name, + }; + + default: + return { kind: g.type, name }; + } +} + +/** + * Reads graders.json and returns auth0-evals compatible GraderDef[]. + * Custom types are mapped to standard primitives (contains, not_contains, matches, judge). + */ +export function defineGraders(): GraderDef[] { + const raw: RichGrader[] = JSON.parse( + readFileSync(join(__dirname, "graders.json"), "utf-8") + ); + return raw.flatMap((g) => { + const mapped = mapGrader(g); + return Array.isArray(mapped) ? mapped : [mapped]; + }); +} diff --git a/auth0-react-router/evals/package.json b/auth0-react-router/evals/package.json new file mode 100644 index 0000000..2dd4054 --- /dev/null +++ b/auth0-react-router/evals/package.json @@ -0,0 +1,16 @@ +{ + "name": "auth0-react-router-evals", + "version": "1.0.0", + "description": "Eval runner for auth0-react-router skill", + "type": "module", + "scripts": { + "eval": "node run-evals.mjs", + "eval:3x": "node run-evals.mjs --runs 3", + "eval:sequential": "node run-evals.mjs --sequential", + "eval:grade-only": "node run-evals.mjs --grade-only" + }, + "dependencies": { + "execa": "^9.0.0", + "ora": "^8.0.0" + } +} diff --git a/auth0-react-router/evals/run-evals.mjs b/auth0-react-router/evals/run-evals.mjs new file mode 100644 index 0000000..085d095 --- /dev/null +++ b/auth0-react-router/evals/run-evals.mjs @@ -0,0 +1,1216 @@ +#!/usr/bin/env node + +import fs from "node:fs" +import path from "node:path" +import os from "node:os" +import readline from "node:readline/promises" +import { $ } from "execa" +import ora from "ora" + +// --------------------------------------------------------------------------- +// Paths +// --------------------------------------------------------------------------- + +const EVAL_DIR = path.dirname(new URL(import.meta.url).pathname) +const SKILL_DIR = path.resolve(EVAL_DIR, "..") + +// --------------------------------------------------------------------------- +// Model selection — null means use CLI default +// --------------------------------------------------------------------------- + +const modelFlag = process.argv.indexOf("--model") +const MODEL = modelFlag !== -1 ? process.argv[modelFlag + 1] : null + +// --------------------------------------------------------------------------- +// User input helpers +// --------------------------------------------------------------------------- + +async function confirm(message) { + const rl = readline.createInterface({ input: process.stdin, output: process.stdout }) + const answer = await rl.question(`${message} (y/N): `) + rl.close() + return answer.toLowerCase() === "y" || answer.toLowerCase() === "yes" +} + +async function prompt(message) { + const rl = readline.createInterface({ input: process.stdin, output: process.stdout }) + const answer = await rl.question(`${message} `) + rl.close() + return answer.trim() +} + +// --------------------------------------------------------------------------- +// File scanning +// --------------------------------------------------------------------------- + +const SOURCE_EXTENSIONS = new Set([ + ".js", ".ts", ".jsx", ".tsx", ".mjs", ".cjs", + ".swift", ".kt", ".java", ".cs", ".go", ".py", ".rb", ".php", ".dart", + ".vue", ".svelte", ".astro", + ".gradle", ".kts", + ".xml", ".plist", ".json", ".env", ".yaml", ".yml", ".toml", ".properties", + ".html", ".css", ".scss", + ".csproj", ".sln", + ".lock", + ".pbxproj", ".resolved", ".podspec", +]) + +function collectSourceFiles(dir, files = []) { + if (!fs.existsSync(dir)) return files + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const full = path.join(dir, entry.name) + if (entry.isDirectory()) { + if (entry.name === "node_modules" || entry.name === ".git" || entry.name === "build" || entry.name === "dist" || entry.name === ".gradle") continue + collectSourceFiles(full, files) + } else if (SOURCE_EXTENSIONS.has(path.extname(entry.name))) { + files.push(full) + } + } + return files +} + +function readAllSources(dir) { + const files = collectSourceFiles(dir) + const contents = [] + for (const f of files) { + try { + contents.push({ path: f, content: fs.readFileSync(f, "utf-8") }) + } catch { + // skip unreadable files + } + } + return contents +} + +// --------------------------------------------------------------------------- +// Grader execution +// --------------------------------------------------------------------------- + +function gradeFileContains(grader, workspaceDir) { + // Glob for files matching the pattern in the workspace + const pattern = grader.file_pattern + const matchingFiles = [] + + function walkAndMatch(dir, globPattern) { + // Convert simple glob to check function + // Supports: **/.env*, **/strings.xml, **/appsettings*.json, **/*.plist, **/environment*.ts, **/application*.properties + const filename = globPattern.replace(/^\*\*\//, "") + const isWildcard = filename.includes("*") + + function matchesPattern(name) { + if (!isWildcard) return name === filename + // Escape regex-special chars, then convert * to [^/]* + const escaped = filename.replace(/[.+?^${}()|[\]\\]/g, "\\$&").replace(/\*/g, "[^/]*") + const regex = new RegExp("^" + escaped + "$") + return regex.test(name) + } + + if (!fs.existsSync(dir)) return + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const full = path.join(dir, entry.name) + if (entry.isDirectory()) { + if (["node_modules", ".git", "build", "dist", ".gradle"].includes(entry.name)) continue + walkAndMatch(full, globPattern) + } else if (matchesPattern(entry.name)) { + try { + matchingFiles.push({ path: full, content: fs.readFileSync(full, "utf-8") }) + } catch { /* skip unreadable */ } + } + } + } + + walkAndMatch(workspaceDir, pattern) + + if (matchingFiles.length === 0) { + return { pass: false, detail: `No files matching "${pattern}" found in workspace` } + } + + for (const f of matchingFiles) { + if (f.content.includes(grader.value)) { + return { pass: true, detail: `Found "${grader.value}" in ${path.relative(workspaceDir, f.path)}` } + } + } + + const fileNames = matchingFiles.map((f) => path.relative(workspaceDir, f.path)).join(", ") + return { pass: false, detail: `"${grader.value}" not found in matching files: ${fileNames}` } +} + +function gradeContains(grader, sources) { + for (const src of sources) { + if (src.content.toLowerCase().includes(grader.value.toLowerCase())) { + return { pass: true, detail: `Found "${grader.value}" in ${path.basename(src.path)}` } + } + } + return { pass: false, detail: `"${grader.value}" not found in any source file` } +} + +function gradeContainsAny(grader, sources) { + // Pass if ANY of the values is found in any source file + const allContent = sources.map((s) => s.content.toLowerCase()).join("\n") + for (const value of grader.values) { + if (allContent.includes(value.toLowerCase())) { + return { pass: true, detail: `Found "${value}" in workspace` } + } + } + return { pass: false, detail: `None of [${grader.values.join(", ")}] found in any source file` } +} + +function gradeNotContains(grader, sources) { + for (const src of sources) { + if (src.content.toLowerCase().includes(grader.value.toLowerCase())) { + return { pass: false, detail: `Found "${grader.value}" in ${path.basename(src.path)} (should not be present)` } + } + } + return { pass: true, detail: `"${grader.value}" correctly absent` } +} + +function gradeNotContainsAny(grader, sources) { + // Pass only if NONE of the values is found in any source file + for (const src of sources) { + const lower = src.content.toLowerCase() + for (const value of grader.values) { + if (lower.includes(value.toLowerCase())) { + return { pass: false, detail: `Found "${value}" in ${path.basename(src.path)} (should not be present)` } + } + } + } + return { pass: true, detail: `None of [${grader.values.join(", ")}] found (correct)` } +} + +function gradeMatches(grader, sources) { + let regex + try { + regex = new RegExp(grader.pattern) + } catch { + return { pass: false, detail: `Invalid regex pattern: ${grader.pattern}` } + } + for (const src of sources) { + if (regex.test(src.content)) { + return { pass: true, detail: `Pattern matched in ${path.basename(src.path)}` } + } + } + return { pass: false, detail: `Pattern /${grader.pattern}/ not matched in any source file` } +} + +function gradeAll(grader, sources, workspaceDir) { + // Pass only if ALL sub-graders pass + const subResults = [] + for (const sub of grader.graders) { + const result = gradeSync(sub, sources, workspaceDir) + subResults.push(result) + if (!result.pass) { + return { pass: false, detail: `Sub-grader failed: ${sub.description || sub.type} — ${result.detail}` } + } + } + return { pass: true, detail: `All ${grader.graders.length} sub-graders passed` } +} + +function gradeSync(grader, sources, workspaceDir) { + // Synchronous grader dispatch (for use in all/any composites — excludes judge) + switch (grader.type) { + case "contains": return gradeContains(grader, sources) + case "contains_any": return gradeContainsAny(grader, sources) + case "file_contains": return gradeFileContains(grader, workspaceDir) + case "not_contains": return gradeNotContains(grader, sources) + case "not_contains_any": return gradeNotContainsAny(grader, sources) + case "matches": return gradeMatches(grader, sources) + default: return { pass: false, detail: `Unsupported sub-grader type: ${grader.type}` } + } +} + +async function gradeJudge(grader, sources, workspaceDir) { + // Collect a summary of the workspace files for the judge + const fileSummary = sources + .slice(0, 20) // limit to 20 files to avoid context overflow + .map((s) => `--- ${path.relative(workspaceDir, s.path)} ---\n${s.content.slice(0, 3000)}`) + .join("\n\n") + + // Build judge prompt — include examples if provided (few-shot) + let questionBlock = grader.question + if (grader.examples) { + questionBlock += `\n\n## Examples\n${grader.examples}` + } + + const judgePrompt = `You are evaluating code quality. Review the following source files and answer this question: + +${questionBlock} + +Answer with exactly "YES" or "NO" on the first line, followed by a brief explanation. + +${fileSummary}` + + const judgeArgs = ["-p", judgePrompt, "--permission-mode", "dontAsk", "--no-session-persistence"] + if (MODEL) judgeArgs.push("--model", MODEL) + + try { + const { stdout } = await $({ + timeout: 60000, + })`claude ${judgeArgs}` + const firstLine = stdout.trim().split("\n")[0].toUpperCase() + const pass = firstLine.startsWith("YES") + return { pass, detail: stdout.trim().slice(0, 300) } + } catch (e) { + return { pass: false, detail: `Judge failed: ${e.message}` } + } +} + +async function runGraders(graders, workspaceDir) { + const sources = readAllSources(workspaceDir) + const results = [] + + // First pass: run all graders + for (const grader of graders) { + let result + switch (grader.type) { + case "contains": + result = gradeContains(grader, sources) + break + case "contains_any": + result = gradeContainsAny(grader, sources) + break + case "file_contains": + result = gradeFileContains(grader, workspaceDir) + break + case "not_contains": + result = gradeNotContains(grader, sources) + break + case "not_contains_any": + result = gradeNotContainsAny(grader, sources) + break + case "matches": + result = gradeMatches(grader, sources) + break + case "all": + result = gradeAll(grader, sources, workspaceDir) + break + case "judge": + result = await gradeJudge(grader, sources, workspaceDir) + break + default: + result = { pass: false, detail: `Unknown grader type: ${grader.type}` } + } + + results.push({ + type: grader.type, + description: grader.description, + ...result, + }) + } + + // Second pass: invalidate not_contains passes when no positive graders passed. + // If the agent wrote zero integration code, not_contains graders trivially pass + // (nothing bad can be present when nothing is present). This inflates scores for + // empty/untouched workspaces. Demote these to FAIL with an explanatory detail. + const positiveTypes = new Set(["contains", "contains_any", "file_contains", "matches"]) + const anyPositivePassed = results.some((r) => positiveTypes.has(r.type) && r.pass) + + if (!anyPositivePassed) { + for (const r of results) { + if ((r.type === "not_contains" || r.type === "not_contains_any") && r.pass) { + r.pass = false + r.detail = `Invalidated: no positive graders passed (agent likely wrote no integration code). Original: ${r.detail}` + } + } + } + + return results +} + +// --------------------------------------------------------------------------- +// Token parsing +// --------------------------------------------------------------------------- + +function parseTokenUsage(stdout) { + // Try to parse token usage from claude CLI output + // claude CLI outputs a summary line like: "Total tokens: 12345" or JSON with token info + let tokens = 0 + try { + // Try JSON output format first + const jsonMatch = stdout.match(/"total_tokens"\s*:\s*(\d+)/) + if (jsonMatch) { + tokens = parseInt(jsonMatch[1], 10) + } else { + // Try summary line format + const lineMatch = stdout.match(/[Tt]otal\s+tokens?\s*[=:]\s*([\d,]+)/) + if (lineMatch) { + tokens = parseInt(lineMatch[1].replace(/,/g, ""), 10) + } else { + // Try cost line (tokens in parentheses) + const costMatch = stdout.match(/(\d[\d,]*)\s*tokens?\s*(?:used|total|consumed)/i) + if (costMatch) { + tokens = parseInt(costMatch[1].replace(/,/g, ""), 10) + } + } + } + } catch { + // If parsing fails, return 0 + } + return tokens +} + +// --------------------------------------------------------------------------- +// Baseline mode — extract code blocks from LLM response +// --------------------------------------------------------------------------- + +function extractCodeBlocks(text) { + // Extract fenced code blocks with optional filename hints + const blocks = [] + const regex = /```(\w+)?(?:\s+(?:\/\/|#|