diff --git a/README.md b/README.md index 9cf67cb..8e30e29 100644 --- a/README.md +++ b/README.md @@ -64,10 +64,39 @@ The prompt supports: - User and AI turns in the transcript. - Session history with delete controls. - Uploaded file references for future capability-specific parsing. -- Action preview cards, such as Fleet-Ops create-order previews. +- Action preview cards, such as Fleet-Ops create-order previews (several per answer). +- Confirmation cards for console actions the AI offers, such as opening a page or a create dialog. +- Thumbs up/down feedback on answers. Closing the prompt only hides it. A chat session continues until the user starts a new chat, ends the current chat, or deletes the session from history. +## How Answers Are Grounded + +With OpenAI or Claude enabled, each prompt runs as a tool-calling conversation. The model looks information up before answering instead of relying on memory: + +- **Documentation**: `search_docs` and `read_doc` search the official Fleetbase documentation (fleetbase.io/docs). Answers link the pages they used. +- **Company data**: `count_records`, `group_count`, and `list_records` query the allowlisted resources extensions register, scoped to the organization and the user's permissions. Invalid filters return errors instead of being ignored. +- **Console actions**: `find_console_commands` and `propose_console_command` offer to open a page or a dialog (for example IAM › Users › Create user). The user sees a confirmation card and nothing runs until they confirm; the server re-checks permissions at that moment. +- **Extension tools**: modules add their own, such as Fleet-Ops record search and order drafts. + +Every tool call is recorded as a task step. Turning off **Look up answers with tools** in the provider settings falls back to the older single-request mode. + +### Who sees what + +Only users whose `type` is `admin` are system administrators. Everyone else, including organization "Administrator" roles, is answered as an organization user: documentation pages and sections about system setup, service credentials, environment variables, and self-hosting are filtered out on the server, admin console actions are never offered, and the system prompt tells the model to refer those needs to the system administrator. + +### Documentation index + +Documentation is indexed from the fleetbase.io sitemap into `ai_knowledge_documents` and `ai_knowledge_chunks` (MySQL full-text search), refreshed weekly, and tagged by audience (`server/config/ai.php` → `knowledge.docs`). A snapshot ships in `server/resources/ai-knowledge` and loads automatically when nothing is indexed, so instances without internet access still have documentation. + +```bash +php artisan ai:sync-docs # crawl fleetbase.io/docs +php artisan ai:sync-docs --snapshot # load the packaged snapshot +php artisan ai:sync-docs --write-snapshot=server/resources/ai-knowledge/docs-snapshot.json.gz +``` + +System administrators can check the index and trigger a sync in **Admin → AI Config → Knowledge Base**. + ## Capability Framework Fleetbase AI does not give providers arbitrary database access. Modules expose AI functionality explicitly by registering capabilities. @@ -95,6 +124,40 @@ Create-order actions use a dedicated compact preview component designed for the Orders are not created until the user confirms the preview. +## Console Commands for Extensions + +Extensions register console actions from their service providers. Definitions are plain arrays, so an extension does not need to depend on this package's classes: + +```php +$this->callAfterResolving(\Fleetbase\Ai\Support\Commands\AiCommandRegistry::class, function ($commands) { + $commands->registerMany([[ + 'id' => 'my-extension.widgets.create', + 'label' => 'Create widget', + 'breadcrumb' => 'My Extension › Widgets', + 'description' => 'Open the new widget form.', + 'steps' => [ + ['type' => 'navigate', 'route' => 'console.my-extension.widgets.index'], + ['type' => 'service', 'engine' => '@vendor/my-extension-engine', 'service' => 'widget-actions', 'method' => 'modal.create'], + ], + 'permissions' => ['my-extension create widget'], + ]]); +}); +``` + +`service` steps call a method on the engine's resource-action service, so the same dialog opens from the AI prompt as from the extension's own pages. Verify routes and service methods resolve with: + +```bash +node scripts/verify-ai-commands.mjs --console ../../console --packages .. +``` + +## Logs, Feedback, and Evaluation + +- Users rate answers with thumbs up or down. Ratings are filterable in **Admin → AI Config → Task & Chat Logs**, alongside degraded answers (a capability failed) and cut-off answers. +- The log view shows each conversation in full: every prompt and answer, and, per answer, every step including each tool call's arguments and results and the exact system prompt and messages sent to the model. It requires the `ai view audit logs` permission. +- Export logs from the admin view, or with `php artisan ai:export-logs --from=2026-09-01 --format=jsonl`. Exports use the same filters as the log view, contain user content, require `ai view audit logs`, and are recorded in the access log. +- `php artisan ai:eval` runs the golden cases in `server/resources/ai-eval/cases.json` (drawn from real conversations) against the configured provider and reports a pass rate. It calls the provider, so it asks for confirmation; use `--model` to compare models. +- `php artisan ai:replay ` re-runs a recorded turn through the current runtime and prints both answers. + ## Development Checks Frontend checks: @@ -108,8 +171,8 @@ Frontend checks: Backend checks: ```bash -php -l server/src/routes.php -php -l server/src/Http/Controllers/Internal/AiSessionController.php +composer test:unit +composer test:lint ``` The local template-lint configuration may print `Invalid rule configuration found: no-down-event-binding` while still exiting successfully. diff --git a/addon/components/admin/ai-admin/filter-bar.hbs b/addon/components/admin/ai-admin/filter-bar.hbs new file mode 100644 index 0000000..45a73fa --- /dev/null +++ b/addon/components/admin/ai-admin/filter-bar.hbs @@ -0,0 +1,46 @@ +
+
+ {{yield to="before"}} + {{#each @primary as |field|}} + + {{/each}} + {{#if this.secondary.length}} +
+ {{#if this.showMore}} +
+ {{#each this.secondary as |field|}} + + {{/each}} +
+ {{/if}} +
diff --git a/addon/components/admin/ai-admin/filter-bar.js b/addon/components/admin/ai-admin/filter-bar.js new file mode 100644 index 0000000..d614140 --- /dev/null +++ b/addon/components/admin/ai-admin/filter-bar.js @@ -0,0 +1,33 @@ +import Component from '@glimmer/component'; +import { tracked } from '@glimmer/tracking'; +import { action } from '@ember/object'; + +/** + * The compact toolbar at the top of the AI admin views. `@primary` fields sit in the toolbar itself, + * `@secondary` fields in a second row opened with "More filters", whose count shows how many of them + * are set. Changes are reported through `@onChange(field)`; the view applies them. + */ +export default class AdminAiAdminFilterBarComponent extends Component { + @tracked showMore = false; + + get secondary() { + return this.args.secondary ?? []; + } + + get secondaryActiveCount() { + return this.args.filters.activeCount(this.secondary.map((field) => (field === 'review' ? ['degraded', 'truncated'] : field === 'session_status' ? 'status' : field)).flat()); + } + + get canClear() { + return this.args.filters.hasAny; + } + + @action toggleMore() { + this.showMore = !this.showMore; + } + + @action clear() { + this.args.filters.clear(); + this.args.onChange?.('clear'); + } +} diff --git a/addon/components/admin/ai-admin/filter-field.hbs b/addon/components/admin/ai-admin/filter-field.hbs new file mode 100644 index 0000000..08a6e1c --- /dev/null +++ b/addon/components/admin/ai-admin/filter-field.hbs @@ -0,0 +1,81 @@ +{{#if (eq @field "search")}} + +{{else if (eq @field "session_status")}} + +{{else if (eq @field "feedback")}} + +{{else if (eq @field "model")}} + - - - - - - -
-
- - - - -