.NET: Update compaction docs to show different ways to use a Compaction Strategy with their implications

[marcominerva]

Hi!

I'm wondering what the difference is between defining a compaction strategy like this:

var compactionStrategy = new ContextWindowCompactionStrategy(
            maxContextWindowTokens: 4000,
            maxOutputTokens: 800);

var agent = chatClient
    .AsAIAgent(new ChatClientAgentOptions
    {
        ChatOptions = new()
        {
            Instructions = "You are a helpful assistant.",
        },
        AIContextProviders = [new CompactionProvider(compactionStrategy)]
    });

and instead of using it as a ChatReducer:

var agent = chatClient
    .AsAIAgent(new ChatClientAgentOptions
    {
        ChatOptions = new()
        {
            Instructions = "You are a helpful assistant.",
        },
        ChatHistoryProvider = new InMemoryChatHistoryProvider(new()
        {
            ChatReducer = compactionStrategy.AsChatReducer()
        })
    });

I'm asking because I read the documentation at https://learn.microsoft.com/en-us/agent-framework/agents/conversations/compaction?#registering-through-chatclientagentoptions, but it doesn't mention using the compaction strategy as a ChatReducer.

More generally, what is the recommended way to register a compaction strategy? For example, in Agent Harness I noticed that the strategy is registered both as a ChatReducer and via UseAIContextProviders(...) on the ChatClientBuilder:

agent-framework/dotnet/src/Microsoft.Agents.AI.Harness/HarnessAgent.cs

Line 195 in 7b6d257

ChatReducer = compactionStrategy.AsChatReducer(),

agent-framework/dotnet/src/Microsoft.Agents.AI.Harness/HarnessAgent.cs

Line 241 in 7b6d257

chatClientBuilder = chatClientBuilder.UseAIContextProviders(compactionProvider);

Should one of these approaches be preferred, or do they serve different purposes?

[westey-m]

@marcominerva, as you point out, there are three different places you can register the compaction, and our docs indeed need to be improved to explain this better.

Let's take each one in turn:

1. As part of ChatClient pipeline

This option is mostly useful for cases where the user may have very long running tool calling loops, with non-service-managed chat history. The compaction would need to be below the FunctionInvokingChatClient in the stack. In this case, let's say the tool calling loop goes on for 100 iterations, and by the time we get to 50, the number of messages being sent to the service in each request is getting close to filling the entire context, so we want to compact the input before calling the service.

For cases where chat history is service-managed (e.g. Reponses with response ids or conversation ids) this case is largely irrelevant, since only new input is passed to the service for each call, and the service itself adds the rest of the chat history and it is responsible for compaction.

The same rules applies if you have enabled per-service-call chat history persistence, unless you add the compaction below the PerServiceCallChatHistoryPersistingChatClient in the stack. Note that doing compaction at this level can be inefficient, since you are running it for each service call.

2. As an AIContextProvider on the Agent

This option allows compaction to the start of any agent run, once per run. This is equivalent to when the provider is added above the FunctionInvokingChatClient in the ChatClient stack.

Similar to no 1 above, this is really not useful for service-managed chat history.

3. As a reducer on the InMemoryChatHistoryProvider

This option allows compaction to run on the chat history stored in the session, so chat history is permanently updated and changes are persisted. This can be more efficient than options 1 & 2, since the results of a single compaction run may impact many subsequent agent runs.

This can be combined with per-service-call chat history persistence as well, to achieve persistent compaction just before service invocation.

Of course, not all ChatHistoryProviders may support compaction. For summarization, you typically need to load the oldest messages, summarize them and then replace them with the summary in storage, meaning that it ends up being quite closely tied to the storage mechanism and requires the ChatHistoryProvider to explicitly support this load/compact/replace functionality. When it does not, number 1 & 2 are useful ways of adding compaction on a per-run / per-service-call basis, on the output from the ChatHistoryProvider + next input.

Let me know if this makes sense.

[marcominerva]

Thanks for the reply @westey-m ! The overall picture is much clearer now.

I think updating the documentation with a few examples showing when to adopt each strategy would be a great improvement and would make it much easier for others to choose the right approach.