diff --git a/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/CHANGELOG.md b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/CHANGELOG.md new file mode 100644 index 000000000..156c27c36 --- /dev/null +++ b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/CHANGELOG.md @@ -0,0 +1,35 @@ +# Changelog + +All notable changes to this project will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## Unreleased + +### Added + +- Initial release of `loongsuite-instrumentation-llama-index`: automatic + instrumentation for LlamaIndex (`llama-index-core`) via its native + instrumentation dispatcher, projecting LlamaIndex spans/events onto + OpenTelemetry spans that follow the ARMS gen-ai semantic conventions + (LLM / EMBEDDING / RETRIEVER / RERANKER / TASK / CHAIN / AGENT / TOOL), with + parent/child trace relationships preserved from `parent_span_id`. + ([#18](https://github.com/alibaba/loongsuite-python/issues/18)) +- Span-kind and content-capture semantics are sourced from the shared GenAI + util (`opentelemetry-util-genai`): span kinds come from `GenAiSpanKindValues` + and content capture is governed by the standard + `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` switch, so this package + behaves consistently with the rest of loongsuite. + +### Fixed + +- A class name containing `Agent` no longer forces internal methods to `AGENT` + spans: `call_tool` is now a `TOOL` span, agent-loop steps (`setup_agent`, + `take_step`, `finalize`, `handle_tool_call_results`, ...) are `CHAIN` spans, + and only a genuine agent invocation (`run`/`chat` on an agent) is `AGENT`. +- `astructured_predict` / `stream_structured_predict` / + `astream_structured_predict` are now classified as `LLM` calls. +- `uninstrument()` no longer strands spans that were open when it ran: the + handler stops creating new spans and drains (ends) any still-open spans + before it is detached from the dispatcher. diff --git a/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/LICENSE b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/LICENSE new file mode 100644 index 000000000..261eeb9e9 --- /dev/null +++ b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/README.md b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/README.md new file mode 100644 index 000000000..c7a3bae8c --- /dev/null +++ b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/README.md @@ -0,0 +1,48 @@ +# LoongSuite LlamaIndex Instrumentation + +OpenTelemetry instrumentation for [LlamaIndex](https://github.com/run-llama/llama_index) +(`llama-index-core`). + +Instead of monkey-patching call sites, this package attaches to LlamaIndex's +**native instrumentation dispatcher** (`llama_index.core.instrumentation`) and +re-projects the span/event stream LlamaIndex already emits onto OpenTelemetry +spans that follow the ARMS gen-ai semantic conventions. LlamaIndex threads a +`parent_span_id` through every instrumented call, so the resulting OTel trace +preserves the logical parent/child structure (e.g. `query` → `retrieve` / +`synthesize`, `chat` → `complete`). + +## Installation + +```bash +pip install loongsuite-instrumentation-llama-index +``` + +## Usage + +```python +from opentelemetry.instrumentation.llama_index import LlamaIndexInstrumentor + +LlamaIndexInstrumentor().instrument() +``` + +## Span kinds + +| LlamaIndex span | `gen_ai.span.kind` | +| --------------------------------------- | ------------------ | +| `*.chat` / `*.complete` / `*.predict` | `LLM` | +| `*.get_*_embedding*` | `EMBEDDING` | +| retriever `*.retrieve` | `RETRIEVER` | +| reranker / node-postprocessor | `RERANKER` | +| response synthesizer `*.synthesize` | `TASK` | +| query engine `*.query` | `CHAIN` | +| chat engine / agent `*.chat` / `*.run` | `AGENT` | + +## Content capture + +Message text is captured on span attributes by default. To suppress +`gen_ai.input.messages` / `gen_ai.output.messages` while keeping the +structural spans and token metrics: + +```bash +export OTEL_INSTRUMENTATION_LLAMA_INDEX_CAPTURE_CONTENT=false +``` diff --git a/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/pyproject.toml b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/pyproject.toml new file mode 100644 index 000000000..f6f758b83 --- /dev/null +++ b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/pyproject.toml @@ -0,0 +1,53 @@ +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[project] +name = "loongsuite-instrumentation-llama-index" +dynamic = ["version"] +description = "LoongSuite LlamaIndex Instrumentation" +license = "Apache-2.0" +requires-python = ">=3.10" +authors = [ + { name = "OpenTelemetry Authors", email = "cncf-opentelemetry-contributors@lists.cncf.io" }, +] +classifiers = [ + "Development Status :: 4 - Beta", + "Intended Audience :: Developers", + "License :: OSI Approved :: Apache Software License", + "Programming Language :: Python", + "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", +] +dependencies = [ + "opentelemetry-api >= 1.37.0", + "opentelemetry-instrumentation >= 0.58b0", + "opentelemetry-semantic-conventions >= 0.58b0", +] + +[project.optional-dependencies] +instruments = [ + "llama-index-core >= 0.11.0", +] + +[project.entry-points.opentelemetry_instrumentor] +llama_index = "opentelemetry.instrumentation.llama_index:LlamaIndexInstrumentor" + +[project.urls] +Homepage = "https://github.com/alibaba/loongsuite-python/tree/main/instrumentation-loongsuite/loongsuite-instrumentation-llama-index" +Repository = "https://github.com/alibaba/loongsuite-python" + +[tool.hatch.version] +path = "src/opentelemetry/instrumentation/llama_index/version.py" + +[tool.hatch.build.targets.sdist] +include = [ + "/src", + "/tests", +] + +[tool.hatch.build.targets.wheel] +packages = ["src/opentelemetry"] diff --git a/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/src/opentelemetry/instrumentation/llama_index/__init__.py b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/src/opentelemetry/instrumentation/llama_index/__init__.py new file mode 100644 index 000000000..e4d1f07fb --- /dev/null +++ b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/src/opentelemetry/instrumentation/llama_index/__init__.py @@ -0,0 +1,695 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +OpenTelemetry LlamaIndex Instrumentation + +Provides automatic instrumentation for LlamaIndex (``llama-index-core``) by +attaching to its **native instrumentation dispatcher** +(``llama_index.core.instrumentation``) rather than monkey-patching call +sites. LlamaIndex already emits a rich span/event stream through a root +``Dispatcher``; this package registers a ``BaseSpanHandler`` and a +``BaseEventHandler`` on that dispatcher and re-projects the stream onto +OpenTelemetry spans that follow the ARMS gen-ai semantic conventions +(see ``arms_docs/trace/gen-ai.md``). + +Why the dispatcher seam (and not wrapt) +--------------------------------------- +LlamaIndex assigns every instrumented call a span ``id_`` and, crucially, a +``parent_span_id`` that reflects the *logical* call tree — e.g. an +``llm.chat`` span parents the ``llm.complete`` span it triggers, a +``query`` span parents ``retrieve`` / ``synthesize`` children, and so on. +Reconstructing that tree by hand via ``wrapt`` would be brittle and would +miss the contextvar-propagated relationships LlamaIndex maintains across +threads and async tasks. Consuming ``parent_span_id`` directly yields a +faithful OTel trace whose parent/child structure matches LlamaIndex's own +view. + +Span kind mapping (ARMS gen-ai semconv) +--------------------------------------- +LlamaIndex span ids are of the form ``.-``. The +``.`` prefix is classified into an ARMS ``gen_ai.span.kind``: + + * ``*.chat`` / ``*.complete`` / ``*.predict`` / ``*.stream*`` → LLM + * ``*.get_text_embedding*`` / ``*.get_query_embedding*`` → EMBEDDING + * ``*.retrieve`` / retriever spans → RETRIEVER + * reranker / postprocessor spans → RERANKER + * ``*.synthesize`` / response-synthesizer spans → TASK + * ``*.query`` / query-engine spans → CHAIN + * ``*.chat`` on chat engines / ``*.run`` on agents → AGENT + * everything else → CHAIN + +Events (``LLMChatEndEvent``, ``EmbeddingEndEvent`` ...) carry request model, +messages and — when the provider returns it — token usage. These are folded +onto the currently-open OTel span as gen-ai attributes. + +Content capture +--------------- +Message text is written onto span attributes only when the shared GenAI +util's content-capture switch enables it -- set +``OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT`` to ``SPAN_ONLY`` or +``SPAN_AND_EVENT`` to record ``gen_ai.input.messages`` / +``gen_ai.output.messages``; ``NO_CONTENT`` (the default) keeps the +structural spans and token metrics without message text. This is the same +control every other loongsuite instrumentation uses. +""" + +import json +import logging +import threading +from typing import Any, Collection, Dict, Optional + +from opentelemetry import context as context_api +from opentelemetry import trace as trace_api +from opentelemetry.instrumentation.instrumentor import BaseInstrumentor +from opentelemetry.instrumentation.llama_index.package import _instruments +from opentelemetry.trace import SpanKind, Status, StatusCode +from opentelemetry.util.genai.extended_semconv.gen_ai_extended_attributes import ( + GEN_AI_SPAN_KIND, + GenAiSpanKindValues, +) +from opentelemetry.util.genai.types import ContentCapturingMode +from opentelemetry.util.genai.utils import get_content_capturing_mode + +logger = logging.getLogger(__name__) + +# ── Framework identifier ───────────────────────────────────────────────────── +_FRAMEWORK = "llama_index" + +# ── GenAI semantic-convention attribute keys (ARMS gen-ai semconv) ─────────── +# Strings inlined to avoid a hard dependency on private aliyun packages that +# aren't published to PyPI. Values track the ARMS gen-ai semconv, matching the +# sibling loongsuite instrumentation packages (terminus2, litellm, ...). +_GEN_AI_SPAN_KIND = GEN_AI_SPAN_KIND # shared GenAI-util semconv key +_GEN_AI_OPERATION_NAME = "gen_ai.operation.name" +_GEN_AI_FRAMEWORK = "gen_ai.framework" +_GEN_AI_REQUEST_MODEL = "gen_ai.request.model" +_GEN_AI_RESPONSE_MODEL = "gen_ai.response.model" +_GEN_AI_PROVIDER_NAME = "gen_ai.provider.name" +_GEN_AI_USAGE_INPUT_TOKENS = "gen_ai.usage.input_tokens" +_GEN_AI_USAGE_OUTPUT_TOKENS = "gen_ai.usage.output_tokens" +_GEN_AI_USAGE_TOTAL_TOKENS = "gen_ai.usage.total_tokens" +_GEN_AI_INPUT_MESSAGES = "gen_ai.input.messages" +_GEN_AI_OUTPUT_MESSAGES = "gen_ai.output.messages" + +# ── Span kind values ───────────────────────────────────────────────────────── +# Span-kind literals below are sourced from the shared GenAI util +# (GenAiSpanKindValues) so this package and the rest of loongsuite speak one +# span-kind vocabulary. TASK/CHAIN have no shared-util member yet, so they +# stay local literals until one exists. +_SPAN_KIND_LLM = GenAiSpanKindValues.LLM.value +_SPAN_KIND_EMBEDDING = GenAiSpanKindValues.EMBEDDING.value +_SPAN_KIND_RETRIEVER = GenAiSpanKindValues.RETRIEVER.value +_SPAN_KIND_RERANKER = GenAiSpanKindValues.RERANKER.value +_SPAN_KIND_TOOL = GenAiSpanKindValues.TOOL.value +_SPAN_KIND_AGENT = GenAiSpanKindValues.AGENT.value +_SPAN_KIND_TASK = "TASK" +_SPAN_KIND_CHAIN = "CHAIN" + +# ── Operation-name values ──────────────────────────────────────────────────── +_OP_CHAT = "chat" +_OP_EMBEDDING = "embedding" +_OP_EXECUTE_TOOL = "execute_tool" +_OP_STEP = "step" +_OP_RETRIEVE = "retrieve" +_OP_RERANK = "rerank" +_OP_TASK = "task" +_OP_CHAIN = "chain" +_OP_INVOKE_AGENT = "invoke_agent" + +# ── Content capture toggle ─────────────────────────────────────────────────── +# Content-message capture is delegated to the shared GenAI util so it is +# governed by the same OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT +# switch (and NO_CONTENT/SPAN_ONLY/EVENT_ONLY/SPAN_AND_EVENT modes) as every +# other loongsuite instrumentation, rather than a package-specific env var. +_CONTENT_ON_SPAN_MODES = frozenset( + {ContentCapturingMode.SPAN_ONLY, ContentCapturingMode.SPAN_AND_EVENT} +) + + +def _capture_content() -> bool: + """True when message content should be written onto spans. + + Reads the shared util's content-capturing mode; unset defaults to no + capture there, so callers that want content set the standard env var. + """ + try: + return get_content_capturing_mode() in _CONTENT_ON_SPAN_MODES + except Exception: # pragma: no cover - defensive: never break tracing + return False + + +def _span_id_prefix(id_: str) -> str: + """Return the ``.`` portion of a LlamaIndex span id. + + LlamaIndex span ids look like ``MockLLM.chat-8c7b...-uuid``. The class and + method carry the semantic meaning; the uuid suffix is per-invocation. + """ + if not id_: + return "" + # The uuid is appended with a '-' separator; the class/method prefix never + # contains a '-' in LlamaIndex's naming scheme. + return id_.split("-", 1)[0] + + +def _classify(prefix: str) -> tuple: + """Map a ``.`` prefix to (span_kind, operation_name). + + Classification is **method-first**: the invoked method carries the + operation's true meaning, and the class name is only a fallback. This + ordering matters because several class names embed a misleading keyword — + e.g. ``RetrieverQueryEngine.query`` is a *query engine* (CHAIN), not a + retriever, despite ``Retriever`` appearing in the class name. + """ + lower = prefix.lower() + method = lower.rsplit(".", 1)[-1] if "." in lower else lower + + # ---- method-first mapping (authoritative) ---- + # Tool execution: an agent's ``call_tool`` runs the selected tool, so it is a + # TOOL span, not an AGENT span -- even though it lives on an ``*Agent`` class. + # Checked before everything else so the class name can never override it. + if method in ("call_tool", "acall_tool"): + return _SPAN_KIND_TOOL, _OP_EXECUTE_TOOL + # Internal agent-loop machinery (setup/init/step/finalize/tool-result + # handling). These are steps *inside* an agent invocation, not the agent + # invocation itself, so they must not inherit AGENT from the class name. + if method in ( + "setup_agent", + "init_run", + "take_step", + "atake_step", + "run_step", + "arun_step", + "_run_step", + "finalize", + "afinalize", + "handle_tool_call_results", + "aggregate_tool_results", + ): + return _SPAN_KIND_CHAIN, _OP_STEP + # Embedding + if "embedding" in method or "embed" in method: + return _SPAN_KIND_EMBEDDING, _OP_EMBEDDING + # LLM calls + if method in ( + "chat", + "achat", + "complete", + "acomplete", + ) or method.startswith( + ("stream_chat", "astream_chat", "stream_complete", "astream_complete") + ): + # chat/complete invoked on a chat engine or agent is the agent turn. + if "agent" in lower or "chatengine" in lower or "chat_engine" in lower: + return _SPAN_KIND_AGENT, _OP_INVOKE_AGENT + return _SPAN_KIND_LLM, _OP_CHAT + # Structured prediction is an LLM call; include the async and streaming + # dispatcher-instrumented variants so they get LLM (not the CHAIN fallback). + if method in ( + "predict", + "apredict", + "structured_predict", + "astructured_predict", + "stream_structured_predict", + "astream_structured_predict", + ): + return _SPAN_KIND_LLM, _OP_CHAT + # Query engine (must precede the retriever class-substring fallback so + # RetrieverQueryEngine.query is a CHAIN, not a RETRIEVER). + if method in ("query", "aquery"): + return _SPAN_KIND_CHAIN, _OP_CHAIN + # Retrieval + if "retrieve" in method: + return _SPAN_KIND_RETRIEVER, _OP_RETRIEVE + # Rerank / postprocess + if "rerank" in method or "postprocess" in method: + return _SPAN_KIND_RERANKER, _OP_RERANK + # Synthesis + if "synthesize" in method: + return _SPAN_KIND_TASK, _OP_TASK + # Agent / workflow run: the true agent invocation entrypoint. + if method in ("run", "arun"): + return _SPAN_KIND_AGENT, _OP_INVOKE_AGENT + + # ---- class-name fallback (method was not decisive) ---- + # NB: no blanket ``"agent" in lower -> AGENT`` here. Only a genuine agent + # invocation (handled above via ``run``/``chat`` on an agent) is AGENT; an + # unrecognized method on an ``*Agent`` class is an internal step, so it falls + # through to CHAIN rather than masquerading as another whole agent turn. + if "queryengine" in lower or "query_engine" in lower: + return _SPAN_KIND_CHAIN, _OP_CHAIN + if "retriever" in lower: + return _SPAN_KIND_RETRIEVER, _OP_RETRIEVE + if "rerank" in lower or "postprocessor" in lower: + return _SPAN_KIND_RERANKER, _OP_RERANK + if "synthesizer" in lower: + return _SPAN_KIND_TASK, _OP_TASK + return _SPAN_KIND_CHAIN, _OP_CHAIN + + +def _messages_to_json(messages: Any) -> Optional[str]: + """Serialize LlamaIndex ChatMessage list into the GenAI message schema.""" + if not messages: + return None + out = [] + try: + for m in messages: + role = getattr(m, "role", None) + role = getattr(role, "value", role) + content = getattr(m, "content", None) + if content is None: + content = str(m) + out.append( + { + "role": str(role) if role is not None else "user", + "parts": [{"type": "text", "content": str(content)}], + } + ) + return json.dumps(out, ensure_ascii=False, separators=(",", ":")) + except Exception: + try: + return json.dumps(str(messages), ensure_ascii=False) + except Exception: + return None + + +def _extract_usage(raw: Any) -> Dict[str, int]: + """Best-effort extraction of token usage from a provider raw response. + + Supports both dict-shaped (``{"usage": {...}}``) and object-shaped + (``resp.usage.prompt_tokens``) payloads across providers. + """ + usage: Dict[str, int] = {} + if raw is None: + return usage + u = None + if isinstance(raw, dict): + u = raw.get("usage") + else: + u = getattr(raw, "usage", None) + if u is None: + return usage + + def _get(obj, *names): + for n in names: + if isinstance(obj, dict): + if n in obj and obj[n] is not None: + return obj[n] + else: + v = getattr(obj, n, None) + if v is not None: + return v + return None + + prompt = _get(u, "prompt_tokens", "input_tokens") + completion = _get(u, "completion_tokens", "output_tokens") + total = _get(u, "total_tokens") + if isinstance(prompt, int): + usage["input"] = prompt + if isinstance(completion, int): + usage["output"] = completion + if isinstance(total, int): + usage["total"] = total + elif "input" in usage and "output" in usage: + usage["total"] = usage["input"] + usage["output"] + return usage + + +# ═══════════════════════════════════════════════════════════════════════════ +# Dispatcher span handler → OTel spans +# ═══════════════════════════════════════════════════════════════════════════ + + +def _build_span_handler(tracer): + """Construct the OTel span handler bound to ``tracer``. + + The handler class is built lazily inside this function so importing this + module does not require ``llama-index-core`` to be installed (mirrors the + other loongsuite packages, whose instrumentation dependency is optional + at import time). + """ + from llama_index.core.instrumentation.span_handlers.base import ( + BaseSpanHandler, + ) + + class _OtelSpanHandler(BaseSpanHandler): + """Project LlamaIndex spans onto OTel spans, preserving the tree. + + ``BaseSpanHandler`` is a pydantic model, so instance mutable state + (the tracer and the id→span/token maps) is stored via + ``object.__setattr__`` to bypass pydantic field validation. + """ + + model_config = {"arbitrary_types_allowed": True} + + def __init__(self, otel_tracer, **kwargs: Any): + super().__init__(**kwargs) + object.__setattr__(self, "_otel_tracer", otel_tracer) + object.__setattr__(self, "_otel_spans", {}) + object.__setattr__(self, "_otel_tokens", {}) + object.__setattr__(self, "_otel_lock", threading.Lock()) + object.__setattr__(self, "_otel_stopped", False) + + # -- helpers ------------------------------------------------------- + def _spans(self) -> Dict[str, Any]: + return object.__getattribute__(self, "_otel_spans") + + def _tokens(self) -> Dict[str, Any]: + return object.__getattribute__(self, "_otel_tokens") + + def _lock(self): + return object.__getattribute__(self, "_otel_lock") + + def _tracer(self): + return object.__getattribute__(self, "_otel_tracer") + + def _stopped(self) -> bool: + return object.__getattribute__(self, "_otel_stopped") + + def stop_and_drain(self) -> None: + """Stop creating new spans and finish any that are still open. + + LlamaIndex only dispatches exit/drop to handlers still attached to + the dispatcher, so a handler removed mid-flight would strand every + span open when uninstrument ran (their SDK spans and context tokens + would never close). Flip the stopped flag first -- so no new spans + are created while callers wind down -- then end whatever remains. + """ + object.__setattr__(self, "_otel_stopped", True) + spans = self._spans() + with self._lock(): + leftover_ids = list(spans.keys()) + for id_ in leftover_ids: + self._finish(id_) + + def class_name(self) -> str: # pydantic-friendly identity + return "OtelSpanHandler" + + # -- lifecycle ----------------------------------------------------- + def new_span( + self, + id_: str, + bound_args, + instance: Optional[Any] = None, + parent_span_id: Optional[str] = None, + tags: Optional[Dict[str, Any]] = None, + **kwargs: Any, + ): + if self._stopped(): + # Uninstrument in progress: create no new spans, but leave + # already-open ones for exit/drop to finish. + return None + prefix = _span_id_prefix(id_) + span_kind, op_name = _classify(prefix) + + spans = self._spans() + with self._lock(): + parent_span = ( + spans.get(parent_span_id) if parent_span_id else None + ) + parent_ctx = ( + trace_api.set_span_in_context(parent_span) + if parent_span is not None + else None + ) + + span = self._tracer().start_span( + prefix or "llama_index.span", + context=parent_ctx, + kind=SpanKind.INTERNAL, + ) + span.set_attribute(_GEN_AI_SPAN_KIND, span_kind) + span.set_attribute(_GEN_AI_OPERATION_NAME, op_name) + span.set_attribute(_GEN_AI_FRAMEWORK, _FRAMEWORK) + span.set_attribute("llama_index.span.id", id_) + + # Attach so that sibling LlamaIndex spans created via contextvars + # (and any nested OTel instrumentation) parent correctly even when + # LlamaIndex does not thread the parent id through. + ctx = trace_api.set_span_in_context(span) + token = context_api.attach(ctx) + + with self._lock(): + spans[id_] = span + self._tokens()[id_] = token + return None + + def _finish(self, id_: str, err: Optional[BaseException] = None): + spans = self._spans() + tokens = self._tokens() + with self._lock(): + span = spans.pop(id_, None) + token = tokens.pop(id_, None) + if token is not None: + try: + context_api.detach(token) + except Exception: # pragma: no cover - defensive + pass + if span is not None: + if err is not None: + span.record_exception(err) + span.set_status(Status(StatusCode.ERROR)) + else: + span.set_status(Status(StatusCode.OK)) + span.end() + + def prepare_to_exit_span( + self, + id_: str, + bound_args, + instance: Optional[Any] = None, + result: Optional[Any] = None, + **kwargs: Any, + ): + self._finish(id_) + return None + + def prepare_to_drop_span( + self, + id_: str, + bound_args, + instance: Optional[Any] = None, + err: Optional[BaseException] = None, + **kwargs: Any, + ): + self._finish(id_, err=err) + return None + + return _OtelSpanHandler(tracer) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Dispatcher event handler → enrich OTel spans +# ═══════════════════════════════════════════════════════════════════════════ + + +def _build_event_handler(span_handler): + from llama_index.core.instrumentation.event_handlers.base import ( + BaseEventHandler, + ) + + class _OtelEventHandler(BaseEventHandler): + """Fold LlamaIndex events onto the matching open OTel span.""" + + model_config = {"arbitrary_types_allowed": True} + + def __init__(self, handler, **kwargs: Any): + super().__init__(**kwargs) + object.__setattr__(self, "_handler", handler) + + @classmethod + def class_name(cls) -> str: + return "OtelEventHandler" + + def _span_for(self, event): + handler = object.__getattribute__(self, "_handler") + spans = handler._spans() + span_id = getattr(event, "span_id", None) + if not span_id: + return None + with handler._lock(): + return spans.get(span_id) + + def handle(self, event, **kwargs: Any) -> Any: + span = self._span_for(event) + if span is None or not span.is_recording(): + return None + + name = event.class_name() + capture = _capture_content() + + # ---- request model (start events) ---- + model_dict = getattr(event, "model_dict", None) + if isinstance(model_dict, dict): + model = model_dict.get("model") or model_dict.get("model_name") + if model: + span.set_attribute(_GEN_AI_REQUEST_MODEL, str(model)) + + # ---- input messages ---- + if capture: + messages = getattr(event, "messages", None) + if messages and name.endswith("StartEvent"): + js = _messages_to_json(messages) + if js: + span.set_attribute(_GEN_AI_INPUT_MESSAGES, js) + prompt = getattr(event, "prompt", None) + if prompt and name.endswith("StartEvent"): + span.set_attribute( + _GEN_AI_INPUT_MESSAGES, + _messages_to_json( + [ + type( + "M", + (), + {"role": "user", "content": prompt}, + )() + ] + ) + or "", + ) + + # ---- end events: response + usage ---- + if name.endswith("EndEvent"): + response = getattr(event, "response", None) + if response is not None: + raw = getattr(response, "raw", None) + usage = _extract_usage(raw) or _extract_usage(response) + if "input" in usage: + span.set_attribute( + _GEN_AI_USAGE_INPUT_TOKENS, usage["input"] + ) + if "output" in usage: + span.set_attribute( + _GEN_AI_USAGE_OUTPUT_TOKENS, usage["output"] + ) + if "total" in usage: + span.set_attribute( + _GEN_AI_USAGE_TOTAL_TOKENS, usage["total"] + ) + if capture: + msg = getattr(response, "message", None) + text = None + if msg is not None: + text = getattr(msg, "content", None) + if text is None: + text = str(response) + span.set_attribute( + _GEN_AI_OUTPUT_MESSAGES, + _messages_to_json( + [ + type( + "M", + (), + {"role": "assistant", "content": text}, + )() + ] + ) + or "", + ) + # completion end carries a plain response string + if capture: + messages_out = getattr(event, "messages", None) + if messages_out and _GEN_AI_OUTPUT_MESSAGES not in ( + span.attributes or {} + ): + js = _messages_to_json(messages_out) + if js: + span.set_attribute(_GEN_AI_OUTPUT_MESSAGES, js) + + # embedding end: record chunk count + chunks = getattr(event, "chunks", None) + if chunks is not None: + try: + span.set_attribute( + "gen_ai.embedding.chunk_count", len(chunks) + ) + except Exception: + pass + return None + + return _OtelEventHandler(span_handler) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Instrumentor +# ═══════════════════════════════════════════════════════════════════════════ + + +class LlamaIndexInstrumentor(BaseInstrumentor): + """Instrumentor for LlamaIndex (``llama-index-core``). + + Registers a span handler and an event handler on LlamaIndex's root + dispatcher. ``uninstrument`` removes them so no spans are produced + afterwards. + """ + + def __init__(self): + super().__init__() + self._span_handler = None + self._event_handler = None + + def instrumentation_dependencies(self) -> Collection[str]: + return _instruments + + def _instrument(self, **kwargs: Any) -> None: + import llama_index.core.instrumentation as instrumentation + + tracer_provider = kwargs.get("tracer_provider") + tracer = trace_api.get_tracer( + __name__, "", tracer_provider=tracer_provider + ) + + dispatcher = instrumentation.get_dispatcher() + span_handler = _build_span_handler(tracer) + event_handler = _build_event_handler(span_handler) + + dispatcher.add_span_handler(span_handler) + dispatcher.add_event_handler(event_handler) + + self._span_handler = span_handler + self._event_handler = event_handler + + def _uninstrument(self, **kwargs: Any) -> None: + try: + import llama_index.core.instrumentation as instrumentation + + dispatcher = instrumentation.get_dispatcher() + if self._span_handler is not None: + # Close any spans still open BEFORE detaching, so removing the + # handler cannot strand them (the dispatcher only routes + # exit/drop to still-attached handlers). + try: + self._span_handler.stop_and_drain() + except Exception as e: # pragma: no cover - defensive + logger.debug("Could not drain open spans: %s", e) + try: + dispatcher.span_handlers = [ + h + for h in dispatcher.span_handlers + if h is not self._span_handler + ] + except Exception as e: # pragma: no cover - defensive + logger.debug("Could not detach span handler: %s", e) + if self._event_handler is not None: + try: + dispatcher.event_handlers = [ + h + for h in dispatcher.event_handlers + if h is not self._event_handler + ] + except Exception as e: # pragma: no cover - defensive + logger.debug("Could not detach event handler: %s", e) + finally: + self._span_handler = None + self._event_handler = None diff --git a/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/src/opentelemetry/instrumentation/llama_index/package.py b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/src/opentelemetry/instrumentation/llama_index/package.py new file mode 100644 index 000000000..d1ba33092 --- /dev/null +++ b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/src/opentelemetry/instrumentation/llama_index/package.py @@ -0,0 +1,15 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_instruments = ("llama-index-core >= 0.11.0",) diff --git a/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/src/opentelemetry/instrumentation/llama_index/version.py b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/src/opentelemetry/instrumentation/llama_index/version.py new file mode 100644 index 000000000..3a48c29e5 --- /dev/null +++ b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/src/opentelemetry/instrumentation/llama_index/version.py @@ -0,0 +1,15 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +__version__ = "0.10.0.dev" diff --git a/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/test-requirements.txt b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/test-requirements.txt new file mode 100644 index 000000000..147d9342e --- /dev/null +++ b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/test-requirements.txt @@ -0,0 +1,8 @@ +pytest +pytest-asyncio +pytest-forked==1.6.0 +setuptools +llama-index-core + +-e opentelemetry-instrumentation +-e instrumentation-loongsuite/loongsuite-instrumentation-llama-index diff --git a/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/tests/__init__.py b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/tests/__init__.py new file mode 100644 index 000000000..b0a6f4284 --- /dev/null +++ b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/tests/__init__.py @@ -0,0 +1,13 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/tests/conftest.py b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/tests/conftest.py new file mode 100644 index 000000000..658549ef1 --- /dev/null +++ b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/tests/conftest.py @@ -0,0 +1,79 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Test configuration for LlamaIndex instrumentation tests. + +These tests exercise the instrumentation against the *real* +``llama-index-core`` dispatcher and a real in-process LLM/embedding stub +(``MockLLM`` / ``MockEmbedding``), asserting on the OTel spans exported to an +``InMemorySpanExporter``. No network access or provider credentials are +required. +""" + +from __future__ import annotations + +import os +import sys +from pathlib import Path + +import pytest + +# --------------------------------------------------------------------------- +# Ensure workspace src is importable when running from the package dir. +# --------------------------------------------------------------------------- +_SRC = Path(__file__).resolve().parents[1] / "src" +if _SRC.is_dir() and str(_SRC) not in sys.path: + sys.path.insert(0, str(_SRC)) + + +def pytest_configure(config: pytest.Config): + os.environ["OTEL_SEMCONV_STABILITY_OPT_IN"] = "gen_ai_latest_experimental" + + +from opentelemetry.instrumentation.llama_index import LlamaIndexInstrumentor +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import SimpleSpanProcessor +from opentelemetry.sdk.trace.export.in_memory_span_exporter import ( + InMemorySpanExporter, +) + + +@pytest.fixture(scope="function", name="span_exporter") +def fixture_span_exporter(): + exporter = InMemorySpanExporter() + yield exporter + exporter.clear() + + +@pytest.fixture(scope="function", name="tracer_provider") +def fixture_tracer_provider(span_exporter): + provider = TracerProvider() + provider.add_span_processor(SimpleSpanProcessor(span_exporter)) + return provider + + +@pytest.fixture(scope="function") +def instrument(tracer_provider): + """Instrument LlamaIndex, yield the instrumentor, then uninstrument. + + Uninstrument is important for RED tests: it must leave the dispatcher + with no OTel handler so that a subsequent call produces zero spans. + """ + instrumentor = LlamaIndexInstrumentor() + instrumentor.instrument( + tracer_provider=tracer_provider, + skip_dep_check=True, + ) + yield instrumentor + instrumentor.uninstrument() diff --git a/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/tests/test_instrumentor.py b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/tests/test_instrumentor.py new file mode 100644 index 000000000..d2afcca6c --- /dev/null +++ b/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/tests/test_instrumentor.py @@ -0,0 +1,290 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for LlamaIndexInstrumentor. + +Every span-producing assertion has a paired RED check: without the +instrumentor active (before ``instrument`` / after ``uninstrument``), the +same LlamaIndex call produces **zero** OTel spans. This guards against the +test passing for reasons unrelated to the instrumentation. +""" + +from __future__ import annotations + +import pytest + +from opentelemetry.instrumentation.llama_index import ( + _GEN_AI_FRAMEWORK, + _GEN_AI_OPERATION_NAME, + _GEN_AI_SPAN_KIND, + _SPAN_KIND_AGENT, + _SPAN_KIND_CHAIN, + _SPAN_KIND_EMBEDDING, + _SPAN_KIND_LLM, + _SPAN_KIND_TOOL, + LlamaIndexInstrumentor, + _classify, + _span_id_prefix, +) + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _mock_llm(): + from llama_index.core.llms import MockLLM + + return MockLLM(max_tokens=8) + + +def _chat_once(llm): + from llama_index.core.llms import ChatMessage + + return llm.chat([ChatMessage(role="user", content="hi there")]) + + +# --------------------------------------------------------------------------- +# Pure classification unit tests (no dispatcher needed) +# --------------------------------------------------------------------------- + + +def test_span_id_prefix_strips_uuid(): + assert ( + _span_id_prefix("MockLLM.chat-8c7b6315-7a0e-401b-9185-59474d2632c0") + == "MockLLM.chat" + ) + assert _span_id_prefix("") == "" + + +@pytest.mark.parametrize( + "prefix,expected_kind", + [ + ("MockLLM.chat", _SPAN_KIND_LLM), + ("OpenAI.complete", _SPAN_KIND_LLM), + ("MockEmbedding.get_text_embedding", _SPAN_KIND_EMBEDDING), + ("VectorIndexRetriever.retrieve", "RETRIEVER"), + ("LLMRerank.postprocess_nodes", "RERANKER"), + ("CompactAndRefine.synthesize", "TASK"), + ("RetrieverQueryEngine.query", "CHAIN"), + ("ReActAgent.run", "AGENT"), + # #273: agent-internal machinery must NOT inherit AGENT from the + # class name -- only a genuine agent invocation (run/chat) is AGENT. + ("FunctionAgent.call_tool", _SPAN_KIND_TOOL), + ("ReActAgent.call_tool", _SPAN_KIND_TOOL), + ("FunctionAgent.take_step", _SPAN_KIND_CHAIN), + ("FunctionAgent.setup_agent", _SPAN_KIND_CHAIN), + ("FunctionAgent.finalize", _SPAN_KIND_CHAIN), + ("ReActAgent.handle_tool_call_results", _SPAN_KIND_CHAIN), + # Copilot: async/streaming structured prediction are LLM calls. + ("OpenAI.astructured_predict", _SPAN_KIND_LLM), + ("OpenAI.stream_structured_predict", _SPAN_KIND_LLM), + ("OpenAI.astream_structured_predict", _SPAN_KIND_LLM), + ], +) +def test_classify(prefix, expected_kind): + kind, _op = _classify(prefix) + assert kind == expected_kind + + +def test_agent_internal_methods_are_not_agent_spans(): + # Direct guard for the #273 review: a class named *Agent* must not turn + # setup/parse/call_tool into AGENT spans. call_tool is TOOL; the rest are + # internal steps (CHAIN), and only run/chat is the AGENT invocation. + assert _classify("FunctionAgent.run")[0] == _SPAN_KIND_AGENT + assert _classify("FunctionAgent.call_tool")[0] == _SPAN_KIND_TOOL + for internal in ("take_step", "setup_agent", "init_run", "finalize"): + kind, _op = _classify(f"FunctionAgent.{internal}") + assert kind != _SPAN_KIND_AGENT, internal + assert kind == _SPAN_KIND_CHAIN, internal + + +# --------------------------------------------------------------------------- +# RED: no spans when not instrumented +# --------------------------------------------------------------------------- + + +def test_red_no_spans_without_instrumentation(span_exporter, tracer_provider): + """Baseline: driving an LLM chat with NO instrumentor active must not + produce any OTel spans on our exporter.""" + llm = _mock_llm() + _chat_once(llm) + assert span_exporter.get_finished_spans() == () + + +# --------------------------------------------------------------------------- +# GREEN: spans appear and nest correctly when instrumented +# --------------------------------------------------------------------------- + + +def test_green_chat_produces_llm_span(instrument, span_exporter): + llm = _mock_llm() + _chat_once(llm) + + spans = span_exporter.get_finished_spans() + assert len(spans) >= 1 + + chat_spans = [ + s + for s in spans + if s.attributes.get(_GEN_AI_SPAN_KIND) == _SPAN_KIND_LLM + ] + assert chat_spans, f"no LLM span among {[s.name for s in spans]}" + for s in chat_spans: + assert s.attributes.get(_GEN_AI_FRAMEWORK) == "llama_index" + assert s.attributes.get(_GEN_AI_OPERATION_NAME) == "chat" + + +def test_green_chat_complete_share_trace_and_nest(instrument, span_exporter): + """MockLLM.chat internally calls MockLLM.complete. The two spans must + share one trace_id and the complete span must be a child of the chat + span — proving parent_span_id is faithfully mapped.""" + llm = _mock_llm() + _chat_once(llm) + + spans = span_exporter.get_finished_spans() + assert len(spans) >= 2, [s.name for s in spans] + + trace_ids = {s.context.trace_id for s in spans} + assert len(trace_ids) == 1, f"spans split across traces: {trace_ids}" + + chat = next(s for s in spans if s.name.endswith(".chat")) + complete = next(s for s in spans if s.name.endswith(".complete")) + + # Assert the EXACT parent id, not just 'some exported span': a broken + # parent_span_id mapping must not be able to satisfy this test. + assert complete.parent is not None + assert complete.parent.span_id == chat.context.span_id + assert complete.context.trace_id == chat.context.trace_id + + +def test_green_embedding_span(instrument, span_exporter): + from llama_index.core.embeddings import MockEmbedding + + emb = MockEmbedding(embed_dim=4) + emb.get_text_embedding("hello world") + + spans = span_exporter.get_finished_spans() + emb_spans = [ + s + for s in spans + if s.attributes.get(_GEN_AI_SPAN_KIND) == _SPAN_KIND_EMBEDDING + ] + assert emb_spans, f"no EMBEDDING span among {[s.name for s in spans]}" + + +# --------------------------------------------------------------------------- +# RED after uninstrument: teardown must stop span production +# --------------------------------------------------------------------------- + + +def test_red_uninstrument_stops_spans(span_exporter, tracer_provider): + instrumentor = LlamaIndexInstrumentor() + instrumentor.instrument( + tracer_provider=tracer_provider, skip_dep_check=True + ) + llm = _mock_llm() + _chat_once(llm) + assert len(span_exporter.get_finished_spans()) >= 1 + + instrumentor.uninstrument() + span_exporter.clear() + + _chat_once(_mock_llm()) + assert span_exporter.get_finished_spans() == (), ( + "spans still produced after uninstrument" + ) + + +# --------------------------------------------------------------------------- +# Lifecycle: double instrument / uninstrument is safe +# --------------------------------------------------------------------------- + + +def test_instrument_is_idempotent_on_uninstrument( + span_exporter, tracer_provider +): + instrumentor = LlamaIndexInstrumentor() + instrumentor.instrument( + tracer_provider=tracer_provider, skip_dep_check=True + ) + instrumentor.uninstrument() + # second uninstrument must not raise + instrumentor.uninstrument() + + +# --------------------------------------------------------------------------- +# Content capture is governed by the shared GenAI util's switch (#273) +# --------------------------------------------------------------------------- + + +def _chat_span(span_exporter): + return next( + s + for s in span_exporter.get_finished_spans() + if s.attributes.get(_GEN_AI_SPAN_KIND) == _SPAN_KIND_LLM + ) + + +def test_content_captured_when_shared_util_enables_span_content( + instrument, span_exporter, monkeypatch +): + # SPAN_ONLY via the standard shared-util env => input messages on the span. + monkeypatch.setenv( + "OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT", "SPAN_ONLY" + ) + _chat_once(_mock_llm()) + span = _chat_span(span_exporter) + assert "gen_ai.input.messages" in span.attributes + + +def test_content_suppressed_when_shared_util_disables_content( + instrument, span_exporter, monkeypatch +): + # NO_CONTENT (the shared-util default) => structural span but no messages. + monkeypatch.setenv( + "OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT", "NO_CONTENT" + ) + _chat_once(_mock_llm()) + span = _chat_span(span_exporter) + assert "gen_ai.input.messages" not in span.attributes + assert "gen_ai.output.messages" not in span.attributes + # the structural span itself is still emitted + assert span.attributes.get(_GEN_AI_FRAMEWORK) == "llama_index" + + +# --------------------------------------------------------------------------- +# Uninstrument must not strand spans that were open when it ran (Copilot #2) +# --------------------------------------------------------------------------- + + +def test_uninstrument_drains_open_spans(span_exporter, tracer_provider): + """A span left open at uninstrument time must still be ended (exported), + not stranded because the handler was detached before it closed.""" + from llama_index.core.instrumentation import get_dispatcher + + instrumentor = LlamaIndexInstrumentor() + instrumentor.instrument(tracer_provider=tracer_provider, skip_dep_check=True) + + dispatcher = get_dispatcher() + # Manually open a span through the dispatcher and DO NOT close it. + dispatcher.span_enter(id_="ManualThing.run-abc", bound_args=None, instance=None) + assert span_exporter.get_finished_spans() == (), "span ended too early" + + # Uninstrument while that span is still open: it must be drained (ended). + instrumentor.uninstrument() + ended = span_exporter.get_finished_spans() + assert any(s.name == "ManualThing.run" for s in ended), ( + f"open span was stranded, not drained: {[s.name for s in ended]}" + ) diff --git a/loongsuite-distro/src/loongsuite/distro/bootstrap_registry/loongsuite_instrumentation_llama_index.py b/loongsuite-distro/src/loongsuite/distro/bootstrap_registry/loongsuite_instrumentation_llama_index.py new file mode 100644 index 000000000..6c43f0b3d --- /dev/null +++ b/loongsuite-distro/src/loongsuite/distro/bootstrap_registry/loongsuite_instrumentation_llama_index.py @@ -0,0 +1,30 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# DO NOT EDIT. THIS FILE WAS AUTOGENERATED FROM INSTRUMENTATION PACKAGES. +# RUN `python scripts/loongsuite/generate_loongsuite_bootstrap.py` TO REGENERATE. +# +# Generated with options: +# --upstream-version: (from source) +# --loongsuite-version: (from source) + +REGISTRY = { + "source": "loongsuite", + "package": "loongsuite-instrumentation-llama-index", + "instrumentation": "loongsuite-instrumentation-llama-index==0.10.0.dev", + "libraries": [ + "llama-index-core >= 0.11.0", + ], + "default": False, +} diff --git a/pyproject.toml b/pyproject.toml index ecd8c7fa6..6c91d6843 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -185,6 +185,8 @@ ignore = [ "instrumentation-loongsuite/loongsuite-instrumentation-claw-eval/**/*.py" = ["PLC0415"] "instrumentation-loongsuite/loongsuite-instrumentation-claw-eval/src/opentelemetry/instrumentation/claw_eval/internal/wrappers.py" = ["E402"] "instrumentation-loongsuite/loongsuite-instrumentation-claw-eval/tests/**/*.py" = ["E402", "F811"] +"instrumentation-loongsuite/loongsuite-instrumentation-llama-index/**/*.py" = ["PLC0415"] +"instrumentation-loongsuite/loongsuite-instrumentation-llama-index/tests/**/*.py" = ["E402", "F811"] "instrumentation-loongsuite/loongsuite-instrumentation-minisweagent/**/*.py" = ["PLC0415"] "instrumentation-loongsuite/loongsuite-instrumentation-minisweagent/tests/**/*.py" = ["E402", "F811"] "instrumentation-loongsuite/loongsuite-instrumentation-openhands/**/*.py" = ["PLC0415"] diff --git a/tox-loongsuite.ini b/tox-loongsuite.ini index 3906e904d..e7479350e 100644 --- a/tox-loongsuite.ini +++ b/tox-loongsuite.ini @@ -103,6 +103,10 @@ envlist = py3{10,11,12,13}-test-loongsuite-instrumentation-crewai lint-loongsuite-instrumentation-crewai + ; loongsuite-instrumentation-llama-index + py3{10,11,12,13}-test-loongsuite-instrumentation-llama-index + lint-loongsuite-instrumentation-llama-index + ; loongsuite-instrumentation-qwenpaw py3{11,12,13}-test-loongsuite-instrumentation-qwenpaw-latest py3{10,11,12,13}-test-loongsuite-instrumentation-qwenpaw-{v1,legacy} @@ -250,6 +254,9 @@ deps = crewai: -r {toxinidir}/instrumentation-loongsuite/loongsuite-instrumentation-crewai/tests/test-requirements.txt + llama-index: {[testenv]test_deps} + llama-index: -r {toxinidir}/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/test-requirements.txt + qwenpaw-latest: -r {toxinidir}/instrumentation-loongsuite/loongsuite-instrumentation-qwenpaw/tests/requirements.latest.txt qwenpaw-v1: -r {toxinidir}/instrumentation-loongsuite/loongsuite-instrumentation-qwenpaw/tests/requirements.v1.txt qwenpaw-legacy: -r {toxinidir}/instrumentation-loongsuite/loongsuite-instrumentation-qwenpaw/tests/requirements.oldest.txt @@ -365,6 +372,8 @@ commands = test-loongsuite-instrumentation-crewai: pytest {toxinidir}/instrumentation-loongsuite/loongsuite-instrumentation-crewai/tests {posargs} lint-loongsuite-instrumentation-crewai: python -m ruff check {toxinidir}/instrumentation-loongsuite/loongsuite-instrumentation-crewai + test-loongsuite-instrumentation-llama-index: pytest {toxinidir}/instrumentation-loongsuite/loongsuite-instrumentation-llama-index/tests {posargs} + lint-loongsuite-instrumentation-llama-index: python -m ruff check {toxinidir}/instrumentation-loongsuite/loongsuite-instrumentation-llama-index test-util-genai: pytest {toxinidir}/util/opentelemetry-util-genai/tests {posargs} lint-util-genai: sh -c "cd util && pylint --rcfile ../.pylintrc opentelemetry-util-genai"