Skip to content
This repository was archived by the owner on Jan 29, 2024. It is now read-only.
This repository was archived by the owner on Jan 29, 2024. It is now read-only.

Low throughput #11

@EdvardD

Description

@EdvardD

I'm trying to use the library to process messages in the subscription with big number of messages. I have this code:

#include <chrono>
#include <memory>
#include <list>
#include <spdlog/spdlog.h>
#include <spdlog/sinks/stdout_sinks.h>
#include "pubsuber/Pubsuber.h"
#include "base/logging.h"

using namespace std::chrono;

int main() {
        pubsuber::ClientOptions opts = pubsuber::ClientOptions::CreateDefault("my-team");

        auto sink = std::make_shared<spdlog::sinks::stdout_sink_mt>();
        sink->set_pattern("%D-%T.%f [%n] [%t] [%L]: %v");
        opts.SetLogSink(sink);
        opts.SetLogLevel(spdlog::level::info);

        pubsuber::ClientPtr client;

        pubsuber::RetryCountPolicy countPolicy{4};
        pubsuber::MaxRetryTimePolicy timePolicy{30s};
        pubsuber::ExponentialBackoffPolicy backoffPolicy{250ms, 3s, 2.0};

        client = pubsuber::Client::Create(std::move(opts), std::move(timePolicy), std::move(countPolicy), std::move(backoffPolicy));
        client->SetLogLevel(spdlog::level::info);

        auto subscription = client->GetSubscription("MY-CHANNEL");


        std::list<pubsuber::MessagePtr> messages;
        std::mutex mmm;
        size_t count = 0;
        subscription->Receive([&messages, &count, &mmm](pubsuber::MessagePtr&& m) {
                std::unique_lock l(mmm);
                if (++count % 1000 == 0) {
                        LOG(Info, "count: " << count << " message_size: " << m->Payload().size());
                        messages.clear();
                }
                messages.push_back(std::move(m));
                messages.back()->Ack();
        });

        while (true) {}

  return 0;
}

When I start the application there is a long queue of unacked messages (~100K) in the subscription.
I see that throughput doesn't allow to read them fast: I see throughput 1K messages in a minute.
Is this an expected speed to process messages? Is it possible to setup library to get a higher throughput?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions