Rate limiting on bulk DELETE-s

[BaurzhanSakhariev]

Hi there, I was reading docs in https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html

And got a question about this part

at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned Amazon S3 prefix.

Let's say I have a structure like

a/
└── b/
    ├── 1/
    │   ├── file1_1
    │   ├── file1_2
    ...
    │   ├── file1_1000
    ├── 2/
    │   ├── file2_1
    │   ├── file2_2
    ...
    │   ├── file2_1000

And deleting using bulk deletes (DeleteObjectsRequest).

1. Do I get it right, that requests containing only files from one sub-directory, say [file1_1, ...file1_1000] will be rate limited for the common prefix a/b/1 and entire request counts as 1?

2. Let's say a single request contains a mix - 500 files from a/b/1 and 500 files from a/b/2. How rate limit is applied here?
   a) it's counted as 1, content doesn't matter -> probably not the case as it would contradict with "per partitioned Amazon S3 prefix"
   b) it's counted as 2, because we have 2 common prefixes in the request.
   c) it's counted as 1000 because each item is handled separately?

   From https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html

    > The request can contain a list of up to 1,000 keys that you want to delete. 
        ...For each key, Amazon S3 performs a delete operation and returns the result...
   

   d) Same like c but deferred/heuristic, ie there are some stats that are being gathered
   https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html
   >And Amazon S3 automatically scales to high request rates
   ie it's not exactly 1000 but smth close to it. And I fire X bulk requests with same prefix in a second throughout
   long time window, I can expect that rate will be close to X*1000, and the longer the time window, the closer it's to X*1000

Bonus question in case it's 2b - I think implication here is that optimal way to keep request rate low is to batch files with common prefix together and avoid mixing files with different prefixes in a single request, is this correct?

---

For GET -> List API i think it's trivial as it's set by a user explcitily

ListObjectsRequest listObjectsRequest = new ListObjectsRequest();
listObjectsRequest.setBucketName("bucket");
listObjectsRequest.setPrefix("prefix");
list = clientReference.client().listObjects(listObjectsRequest);

meaning that I can fire 5500 such requests with different prefixes in a second, is it correct?

Thanks!

[BaurzhanSakhariev]

https://forums.veeam.com/object-storage-as-backup-target-f52/high-s3-api-request-rate-during-delete-operations-exponential-backoff-algorithm-t70741.html

s3 allow deleting multiple objects with a single HTTP request using the "DeleteObjects" API. The single HTTP request is tracked in CloudWatch, however internally, S3 processes a DELETE operation for each object in the original request. This explains why you can see about 350 in the metrics, while I see more than 3500 "internal" operations.

https://www.perfectscale.io/blog/aws-s3-throttling

Use prefixes for your objects, this will be a guardrail for AWS to partition it. Even if you have prefixes it doesn’t mean that AWS will give you 3.500 RPS for every prefix, extended load for them causes partitioning and thus increased throughput.

so for 1: it's 1000 ops, 1 request (+1 in CloudWatch) and 1000 files hitting same prefix a/b/1
for 2: neither a, nor b, nor c, nor d, rather "corrected c + d" -> it's 500 in a/b/1 and 500 in a/b/2 but after some time/sustained load