Skip to content

Commit ce25bc8

Browse files
hollenberryCopilot
andauthored
Copilot usage metrics: rename Passive users to Chat-first users, add chat and vscode_agent feature engagement (#64386)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9526898b-57dc-484a-a65f-3ac2048de06e
1 parent 6de6538 commit ce25bc8

5 files changed

Lines changed: 45 additions & 33 deletions

File tree

‎content/copilot/concepts/billing-and-usage/copilot-usage-metrics/copilot-metrics.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -168,7 +168,7 @@ Users are grouped into the following phases:
168168

169169
| Phase | What it represents |
170170
|:--|:--|
171-
| Passive users | The user has not met the engagement threshold for a phase during the period. A passive user may still be using {% data variables.product.prodname_copilot_short %} regularly, for example by asking questions in {% data variables.copilot.copilot_chat_short %} or agent mode without applying any code edits. In the API, this phase is labeled `No Cohort`. |
171+
| Chat-first users | The user is active but hasn't used a qualifying feature on at least two distinct days in the trailing 28-day window. For example, a chat-first user may be asking questions in {% data variables.copilot.copilot_chat_short %} or agent mode without applying any code edits. In the API, this phase is labeled `No Cohort`. |
172172
| Phase 1: Code first | The user engaged with code completions and/or agent edits, where {% data variables.product.prodname_copilot_short %} writes changes directly into files in the IDE. Using {% data variables.copilot.copilot_chat_short %} or agent mode without producing code completion or agent edit activity does not qualify on its own. |
173173
| Phase 2: Agent first | The user engaged with a single {% data variables.product.github %}-based agent surface, such as {% data variables.copilot.copilot_cloud_agent %}, {% data variables.copilot.copilot_code-review_short %}, or {% data variables.copilot.copilot_cli_short %}. Both active engagement with {% data variables.copilot.copilot_code-review_short %} and having it automatically assigned to review a pull request count as the same surface. |
174174
| Phase 3: Multi-agent | The user engaged with two or more {% data variables.product.github %}-based agent surfaces, or with the {% data variables.copilot.github_copilot_app %}. |
@@ -179,17 +179,17 @@ Phase assignment is recalculated daily from the trailing 28-day window, so a use
179179

180180
The impact dashboard counts every user who was active during the trailing 28-day window and groups each user by their current phase. The phase classification rules are the same, but the dashboard population is broader than the `totals_by_ai_adoption_phase.total_engaged_users` field in API and NDJSON reports, which counts users who were active on a specific day.
181181

182-
Users who haven't met any phase's threshold are grouped into **Passive users**, which is not a measure of inactivity, but a signal that a user's engagement hasn't yet reached the level needed to reliably classify their adoption depth. For example, a user with fewer than two active days in the trailing 28-day window, or one who only lightly uses a surface like {% data variables.copilot.copilot_chat_short %} on {% data variables.product.prodname_dotcom_the_website %}, shows as a passive user.
182+
Users who haven't met any phase's threshold are grouped into **Chat-first users**, which is not a measure of inactivity, but a signal that a user's engagement hasn't yet reached the level needed to reliably classify their adoption depth. For example, a user with fewer than two active days in the trailing 28-day window, or one who only lightly uses a surface like {% data variables.copilot.copilot_chat_short %} on {% data variables.product.prodname_dotcom_the_website %}, shows as a chat-first user.
183183

184184
Because phase assignment is based on **which surfaces a user engages with**, not simply how many actions they take, a user with high completion volume but no agent usage stays in Phase 1, while a user with lighter but broader usage across agent surfaces progresses to Phase 2 or Phase 3.
185185

186186
For the underlying schema fields, see [AI adoption phase fields](/copilot/reference/copilot-usage-metrics/copilot-usage-metrics#ai-adoption-phase-fields).
187187

188-
The impact dashboard's **adoption multiplier** divides the average amount of pull requests merged for engaged users, which are ones in Phase 1, 2, or 3, by the amount of pull requests merged for Passive users.
188+
The impact dashboard's **adoption multiplier** divides the average amount of pull requests merged for engaged users, which are ones in Phase 1, 2, or 3, by the amount of pull requests merged for chat-first users.
189189

190190
This shows the relative impact of deeper adoption, independent of how many users fall into each phase. This is a measurement to understand the potential lift that becoming a more engaged user can provide.
191191

192-
For example, if users in the engaged cohorts averaged 20 pull requests/user/month and the passive cohort averaged 10, then the equation is 20 / 10 = 2x multiplier.
192+
For example, if engaged users averaged 20 merged pull requests per user per month and chat-first users averaged 10, the multiplier is 20 / 10 = 2x.
193193

194194
### Understanding industry comparisons
195195

‎content/copilot/reference/copilot-usage-metrics/copilot-usage-metrics.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -53,8 +53,8 @@ These metrics appear in the impact dashboard, which groups users into adoption c
5353
|:--|:--|
5454
| Engagement trends | How the organization's adoption cohort mix and pull request throughput (total pull requests merged per month) have changed over the last six months. |
5555
| Feature engagement | Number of active users who engaged with each included {% data variables.product.prodname_copilot_short %} feature on at least two distinct days during the inclusive 28-day window. |
56-
| Adoption cohort distribution | Share of licensed users grouped into each adoption phase (Passive users, Phase 1, Phase 2, Phase 3) for the period. Passive users correspond to the `No Cohort` value in the API. |
57-
| Adoption multiplier | Compares engaged users (Phase 1, 2, or 3) against passive users on code shipped (pull requests merged per user per month) and time to merge pull requests, to show the relative impact of deeper {% data variables.product.prodname_copilot_short %} adoption. |
56+
| Adoption cohort distribution | Share of licensed users grouped into each adoption phase (Chat-first users, Phase 1, Phase 2, Phase 3) for the period. Chat-first users correspond to the `No Cohort` value in the API. |
57+
| Adoption multiplier | Compares engaged users (Phase 1, 2, or 3) against chat-first users on code shipped (pull requests merged per user per month) and time to merge pull requests, to show the relative impact of deeper {% data variables.product.prodname_copilot_short %} adoption. |
5858
| Recommendations | Suggested actions, such as configuring {% data variables.copilot.copilot_cloud_agent %} or enabling {% data variables.copilot.copilot_code-review_short %}, based on the organization's current cohort distribution. |
5959

6060
## Code generation dashboard metrics
@@ -236,17 +236,17 @@ The 28-day aggregate reports (`enterprise-28-day` and `organization-28-day`) are
236236

237237
#### {% data variables.product.prodname_copilot_short %} feature engagement fields
238238

239-
The optional `copilot_feature_engagement` object summarizes the report's rolling active-user population and engagement with seven features. For the `day_totals` entry whose `day` matches `report_end_day`, `copilot_feature_engagement.active_user_count` equals the sum of the mutually exclusive `totals_by_ai_adoption_phase[].users_in_phase_28d` values.
239+
The optional `copilot_feature_engagement` object summarizes the report's rolling active-user population and engagement with nine features. For the `day_totals` entry whose `day` matches `report_end_day`, `copilot_feature_engagement.active_user_count` equals the sum of the mutually exclusive `totals_by_ai_adoption_phase[].users_in_phase_28d` values.
240240

241-
Counts for individual features overlap because a user can engage with more than one feature. Do not add the `engaged_user_count` values together. The initial feature set reports active and passive {% data variables.copilot.copilot_code-review_short %} as separate features. It does not include {% data variables.copilot.copilot_chat_short %} or {% data variables.product.prodname_vscode_shortname %} Agent.
241+
Counts for individual features overlap because a user can engage with more than one feature. Do not add the `engaged_user_count` values together. The feature set reports active and passive {% data variables.copilot.copilot_code-review_short %} as separate features.
242242

243-
{% data variables.copilot.copilot_chat_short %} is planned for a later release after 28 consecutive days of data have been collected and report consumers support the additional entry.
243+
The array also includes {% data variables.copilot.copilot_chat_short %} (`chat`) and the {% data variables.product.prodname_vscode_shortname %} Agents window (`vscode_agent`). These two features are reported for engagement only and aren't used to classify adoption phases. `chat` covers {% data variables.copilot.copilot_chat_short %} use captured by usage metrics, including agent mode. Users counted under `agent_edit` are also counted under `chat`. Reports for windows before these features were added can contain only the original seven entries.
244244

245245
| Field | Type | Nullable | Description |
246246
|:--|:--|:--|:--|
247247
| `copilot_feature_engagement.active_user_count` | `integer` | No | Nonnegative number of distinct active users in the inclusive 28-day report window. A valid zero-user report sets this field to `0`. |
248-
| `copilot_feature_engagement.totals_by_feature` | `array` | No | Feature engagement counts. The initial array contains exactly one entry for each of these values: `code_completion`, `agent_edit`, `code_review_passive`, `code_review_active`, `cloud_agent`, `copilot_cli`, and `github_app`. A valid zero-user report includes all seven entries with `engaged_user_count` set to `0`. |
249-
| `copilot_feature_engagement.totals_by_feature[].feature` | `string` | No | Feature identifier. In this object, `github_app` represents the {% data variables.copilot.github_copilot_app_short %}. This differs from the `copilot_app` value used in activity breakdown arrays such as `totals_by_feature`. |
248+
| `copilot_feature_engagement.totals_by_feature` | `array` | No | Feature engagement counts, sorted alphabetically by `feature`. The array contains exactly one entry for each of these values: `agent_edit`, `chat`, `cloud_agent`, `code_completion`, `code_review_active`, `code_review_passive`, `copilot_cli`, `github_app`, and `vscode_agent`. A valid zero-user report includes all nine entries with `engaged_user_count` set to `0`. |
249+
| `copilot_feature_engagement.totals_by_feature[].feature` | `string` | No | Feature identifier. In this object, `github_app` represents the {% data variables.copilot.github_copilot_app_short %}. Activity breakdown arrays such as `totals_by_feature` use `copilot_app` instead. The `vscode_agent` value represents the dedicated {% data variables.product.prodname_vscode_shortname %} Agents window, not agent mode in the editor. |
250250
| `copilot_feature_engagement.totals_by_feature[].engaged_user_count` | `integer` | No | Nonnegative number of distinct active users who engaged with the feature on at least two distinct days during the inclusive 28-day report window. |
251251

252252
### User-teams fields
@@ -611,7 +611,7 @@ A user is classified into a phase based on the {% data variables.product.prodnam
611611

612612
| Phase | Engagement criteria |
613613
|:--|:--|
614-
| No Cohort (displayed as "Passive users" in the impact dashboard) | The user isn't engaged with any feature that qualifies for a phase. This doesn't mean the user is inactive. A user in this group can still be using {% data variables.product.prodname_copilot_short %} regularly and consuming premium requests, but hasn't used a qualifying feature on at least two distinct days within the trailing 28-day window. The most common case is conversational use of {% data variables.copilot.copilot_chat_short %} or agent mode, recorded as `chat_panel_agent_mode`, that never produces `code_completion` or `agent_edit` activity. Copying code out of a chat response also doesn't produce either signal. |
614+
| No Cohort (displayed as "Chat-first users" in the impact dashboard) | The user isn't engaged with any feature that qualifies for a phase. This doesn't mean the user is inactive. A user in this group can still be using {% data variables.product.prodname_copilot_short %} regularly and consuming premium requests, but hasn't used a qualifying feature on at least two distinct days within the trailing 28-day window. The most common case is conversational use of {% data variables.copilot.copilot_chat_short %} or agent mode, recorded as `chat_panel_agent_mode`, that never produces `code_completion` or `agent_edit` activity. Copying code out of a chat response also doesn't produce either signal. |
615615
| Phase 1: Code first | Engaged with inline code completions (the `code_completion` value of the `feature` dimension) or with agent edits written directly to files (the `agent_edit` value of the `feature` dimension). |
616616
| Phase 2: Agent first | Engaged with exactly one {% data variables.product.github %}-based agent surface: {% data variables.copilot.copilot_cli_short %} (the `copilot_cli` value of the `feature` dimension), {% data variables.copilot.copilot_cloud_agent %} (`used_copilot_cloud_agent`), or {% data variables.copilot.copilot_code-review_short %} (`used_copilot_code_review_active` or `used_copilot_code_review_passive`). Active and passive {% data variables.copilot.copilot_code-review_short %} count together as a single surface. |
617617
| Phase 3: Multi-agent | Engaged with the {% data variables.copilot.github_copilot_app_short %} (the `copilot_app` value of the `feature` dimension), or engaged with two or more of the agent surfaces listed for Phase 2. |

‎content/copilot/reference/copilot-usage-metrics/example-schema.md‎

Lines changed: 24 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -309,7 +309,7 @@ In this example, `used_copilot_code_review_active` and `used_copilot_code_review
309309
"loc_suggested_to_delete_sum" : 6,
310310
"missing_client_telemetry_user_count" : 0,
311311
"monthly_active_agent_users" : 0,
312-
"monthly_active_chat_users" : 0,
312+
"monthly_active_chat_users" : 2,
313313
"monthly_active_copilot_cloud_agent_users" : 1,
314314
"monthly_active_copilot_code_review_users" : 1,
315315
"monthly_active_local_byok_users" : 2,
@@ -567,26 +567,32 @@ In this example, `used_copilot_code_review_active` and `used_copilot_code_review
567567
"copilot_feature_engagement" : {
568568
"active_user_count" : 2,
569569
"totals_by_feature" : [ {
570-
"feature" : "code_completion",
571-
"engaged_user_count" : 2
572-
}, {
573570
"feature" : "agent_edit",
574571
"engaged_user_count" : 1
575572
}, {
576-
"feature" : "code_review_passive",
573+
"feature" : "chat",
574+
"engaged_user_count" : 2
575+
}, {
576+
"feature" : "cloud_agent",
577577
"engaged_user_count" : 1
578+
}, {
579+
"feature" : "code_completion",
580+
"engaged_user_count" : 2
578581
}, {
579582
"feature" : "code_review_active",
580583
"engaged_user_count" : 1
581584
}, {
582-
"feature" : "cloud_agent",
585+
"feature" : "code_review_passive",
583586
"engaged_user_count" : 1
584587
}, {
585588
"feature" : "copilot_cli",
586589
"engaged_user_count" : 1
587590
}, {
588591
"feature" : "github_app",
589592
"engaged_user_count" : 1
593+
}, {
594+
"feature" : "vscode_agent",
595+
"engaged_user_count" : 1
590596
} ]
591597
},
592598
"report_end_day" : "2025-10-01",
@@ -614,7 +620,7 @@ In this example, `used_copilot_code_review_active` and `used_copilot_code_review
614620
"loc_suggested_to_delete_sum" : 3,
615621
"missing_client_telemetry_user_count" : 1,
616622
"monthly_active_agent_users" : 0,
617-
"monthly_active_chat_users" : 0,
623+
"monthly_active_chat_users" : 2,
618624
"monthly_active_copilot_cloud_agent_users" : 0,
619625
"monthly_active_copilot_code_review_users" : 0,
620626
"monthly_active_local_byok_users" : 0,
@@ -776,26 +782,32 @@ In this example, `used_copilot_code_review_active` and `used_copilot_code_review
776782
"copilot_feature_engagement" : {
777783
"active_user_count" : 2,
778784
"totals_by_feature" : [ {
779-
"feature" : "code_completion",
780-
"engaged_user_count" : 2
781-
}, {
782785
"feature" : "agent_edit",
783786
"engaged_user_count" : 1
784787
}, {
785-
"feature" : "code_review_passive",
788+
"feature" : "chat",
789+
"engaged_user_count" : 2
790+
}, {
791+
"feature" : "cloud_agent",
786792
"engaged_user_count" : 1
793+
}, {
794+
"feature" : "code_completion",
795+
"engaged_user_count" : 2
787796
}, {
788797
"feature" : "code_review_active",
789798
"engaged_user_count" : 1
790799
}, {
791-
"feature" : "cloud_agent",
800+
"feature" : "code_review_passive",
792801
"engaged_user_count" : 1
793802
}, {
794803
"feature" : "copilot_cli",
795804
"engaged_user_count" : 1
796805
}, {
797806
"feature" : "github_app",
798807
"engaged_user_count" : 1
808+
}, {
809+
"feature" : "vscode_agent",
810+
"engaged_user_count" : 1
799811
} ]
800812
},
801813
"report_end_day" : "2025-10-01",

0 commit comments

Comments
 (0)