Skip to content

INDEXING.md

Latest commit

 

History

History
145 lines (110 loc) · 8.21 KB

INDEXING.md

File metadata and controls

145 lines (110 loc) · 8.21 KB
Raw

Indexing API

Glean Code exposes 32 of the 37 documented Indexing API endpoints — read/debug, single-record writes, bulk uploads, and long-running process-all triggers. All Indexing-API commands require a separate indexing token (Client API tokens cannot reach /api/index/v1):

/config set indexing_token <token-or-secure-ref>

The token can be a literal value or the secure reference token.secure.indexing (resolved from $GLEAN_INDEXING_TOKEN at request time — see Secure tokens). Get a real token from your Glean admin UI (workspace settings → API tokens → Indexing).

Read & debug

These are non-destructive lookups — start here when answering questions like "is this doc indexed?", "why can't user X see doc Y?", or "what's the upload count for datasource Z?"

Command Purpose
/datasources.status <name> Full status for one datasource: visibility, counts, last 5 processing events
/datasources.config <name> Live config: object types, ACL settings, trusted domains, icon URL
/datasources.list --with-status All datasources with uploaded/indexed counts and coverage
/documents.status Upload + indexing status for one document
/documents.count Document count for a custom datasource
/users.count User count for a custom datasource
/documents.access Whether a specific user has access to a specific document
/debug.document Per-doc debug payload (status + uploaded permissions)
/debug.documents Bulk debug for many documents (--from-file)
/debug.user Per-user debug payload (status + uploaded groups) 
/datasources.config gdrive
/documents.access --datasource gdrive --object-type Article --id doc-1 --user alice@example.com
/debug.user gdrive alice@example.com

Single-record write

Each write command takes a JSON request body via --from-file. Deletes use convenience flags. All accept --version <n> for optimistic concurrency.

Command Purpose
/index.document Index one document — supports --path <file> mode, see below
/index.delete-document Delete one document by id
/index.permissions Update document ACL
/index.user Index one user
/index.delete-user Delete one user
/index.group Index one group
/index.delete-group Delete one group
/index.membership Index one group membership
/index.delete-membership Delete one group membership 
/index.document --from-file ./doc.json
/index.document --path ./README.md --datasource custom1 --object-type Article --public
/index.delete-document --datasource gdrive --object-type Article --id doc-1
/index.permissions --from-file ./perms.json

Indexing from local files (--path)

/index.document and /index.bulk-documents both accept --path <file-or-dir> as an alternative to --from-file. The CLI walks the path, builds DocumentDefinitions for you, and POSTs them. Pair with --dry-run to see exactly what would be sent.

Flag Purpose
--path A file (single mode) or directory (bulk mode)
--datasource Required. Datasource name applied to every walked file
--object-type Required. e.g. Article, Wiki
--public Make all docs world-readable. Mutually exclusive with --acl-from-file
--acl-from-file JSON file with a DocumentPermissionsDefinition applied to every doc
--include Comma-separated globs. Default: *.txt,*.md,*.markdown,*.html,*.htm,*.json
--exclude Comma-separated globs. Default skips .git, node_modules, __pycache__, .DS_Store
--max-bytes Skip files larger than this. Default 5 MB
--id-prefix Prepended to the path-derived id slug (e.g. --id-prefix proj → proj-team-onboarding)
--view-url-prefix Base URL prepended to relative paths. Defaults to file:// per file
--dry-run Print the assembled request body and exit without calling the API

Supported file types: .txt, .md, .markdown, .html, .htm, .json. Binary formats (PDF, .docx, etc.) are out of scope for v1.

Behaviour:

  • Path-derived ids: team/onboarding.md → team-onboarding. Stable across re-runs, debuggable, datasource-safe
  • HTML files are sent as htmlContent; everything else as textContent
  • Mock mode works for --path exactly like every other indexing command (token still required)
  • /index.document --path <dir> errors and points you at /index.bulk-documents — single mode is single-file only
  • The bulk command warns when more than 500 files are matched. v1 sends them in one POST; auto-paging across isFirstPage/isLastPage is planned for v2
# Single Markdown file, public ACL, dry-run first
/index.document --path ./README.md \
                --datasource custom1 --object-type Article \
                --public --dry-run

# Walk a folder, only .md and .txt, with a fixed ACL from disk
/index.bulk-documents --path ./content/ \
                      --datasource custom1 --object-type Article \
                      --acl-from-file ./perms.json \
                      --include "*.md,*.txt" --exclude "**/draft/**"

Bulk + paged uploads

Bulk endpoints use the standard upload-paging contract (uploadId, isFirstPage, isLastPage, optional forceRestartUpload). Wrap your full request body in a JSON file and pass it via --from-file. The documents-side commands also accept --path (see above).

Command Endpoint family
/index.documents Paged document index
/index.bulk-documents Bulk document index
/index.bulk-users Bulk user index
/index.bulk-groups Bulk group index
/index.bulk-memberships Bulk group memberships
/people.bulk-employees Bulk employee records (org chart)
/people.bulk-teams Bulk team records (org chart)
/people.index-employee-list Versioned employee list
/shortcuts.bulk-index Bulk shortcuts via Indexing API ⚠ distinct from Client API /shortcuts.*
/shortcuts.upload Upload shortcuts via Indexing API

Process-all (long-running)

Trigger a tenant-wide reprocess after a bulk upload completes. These commands accept an optional --datasource filter where applicable.

Command Purpose
/index.process-all-documents Reprocess all uploaded documents
/index.process-all-memberships Reprocess all uploaded memberships
/people.process-all-employees-teams Reprocess all uploaded employees + teams

Token rotation

/indexing.rotate-token
/config set indexing_token <new-raw-secret>

/indexing.rotate-token prints the new raw secret — store it immediately, the old one is invalidated.

Mock mode for indexing

All 32 indexing commands work in mock mode as long as an indexing token is set in config — it can be any non-empty string (e.g. mock_idx_token). The CLI returns realistic shapes (datasource configs, doc/user counts, debug payloads, accept-style write responses) so you can rehearse a workflow before pointing at a live tenant.

REST paths

See REST_PATHS.md for the full list of Indexing API paths this client targets.

← Back to README · Command Reference