RFC: Add limited read

[Xuanwo]

Which issue does this PR close?

Refs #7938.

Rationale for this change

A bounded read range is exact, so callers that only need up to a small number of bytes must stat the object first or handle a short object as an error. OpenDAL needs an at-most read for probes such as file header and format detection without an extra metadata request.

What changes are included in this PR?

This PR proposes ReadOptions::limit and Operator::read_with(...).limit(n). The proposal keeps exact bounded range behavior, uses the existing raw open and read split, and carries only a private exact boolean in core instead of adding a new raw operation or planning abstraction.

Are there any user-facing changes?

Yes. The proposed API returns between zero and the configured limit on clean EOF. Existing reads that do not set limit keep their current behavior. This PR contains the RFC only; it does not implement the API.

AI Usage Statement

This proposal was drafted with assistance from OpenAI Codex powered by GPT-5.

[dentiny]

I'm not sure this aligns with the current behavior? I checked with quick unit test seems we return RangeNotSatisfied for out-of-bound access.

Example-1 for memory backend

// under core/core/src/services/memory/backend.rs
#[cfg(test)]
mod tests {
    use super::MEMORY_SCHEME;
    use crate::Buffer;
    use crate::BytesRange;
    use crate::ErrorKind;
    use crate::Operator;
    use crate::Result;

    #[tokio::test]
    async fn test_bounded_read_beyond_eof_returns_range_not_satisfied() -> Result<()> {
        let op = Operator::via_iter(MEMORY_SCHEME, [])?;
        op.write("test", Buffer::from("abc")).await?;

        // RFC-7945 proposes normalizing range(4096..).limit(1024) into this
        // bounded physical range. Memory rejects it before core can apply
        // at-most completion semantics.
        let physical_range = BytesRange::new(4096, Some(1024));
        let err = op
            .reader("test")
            .await?
            .read(physical_range)
            .await
            .unwrap_err();

        assert_eq!(err.kind(), ErrorKind::RangeNotSatisfied);

        Ok(())
    }
}

Example-2 for local filesystem backend

// under core/services/fs/src/backend.rs
#[cfg(test)]
mod tests {
    use opendal_core::Buffer;
    use opendal_core::BytesRange;
    use opendal_core::ErrorKind;
    use opendal_core::Operator;

    use super::*;

    #[tokio::test]
    async fn test_bounded_read_beyond_eof_returns_range_not_satisfied() -> Result<()> {
        let root = tempfile::TempDir::new().expect("tempdir must be created");
        let op = Operator::new(FsBuilder::default().root(&root.path().to_string_lossy()))?;
        op.write("test", Buffer::from("abc")).await?;

        let physical_range = BytesRange::new(4096, Some(1024));
        let err = op
            .reader("test")
            .await?
            .read(physical_range)
            .await
            .unwrap_err();

        assert_eq!(err.kind(), ErrorKind::RangeNotSatisfied);

        Ok(())
    }
}

[Xuanwo]

Sorry I didn't get this. We are going to add a new option called limit, and this examples shows how limit works. It's true and of course that current tests will fail.

[dentiny]

The RFC here mentioned

If the offset is at or beyond the end of the object, the result is empty.

It reads to me that, with limit option specified, users will get an empty buffer instead of an error.

• Our current behavior seems to reject and explicitly return RangeNotSatisfied (see my rust code above) -- it's a behavior change
• HTTP spec indicates 416 error code is intended for out-of-range request: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/416#malformed_range_request
• S3 documents that Range follows the HTTP Range header semantics: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html?utm_source=chatgpt.com#API_GetObject_RequestSyntax

So I'm wondering if it's better to keep the current behavior?

[Xuanwo]

Oh, yes. Makes sense 👍

[akx]

Yeah, starting a read past end of file sounds like it should be RangeNotSatisfied, even if short reads would be allowed by the caller. 👍

[akx]

I'm not sure this rejection is needed. If suffix ranges are currently supported anyway, a suffix range (END-1024..END) + limit 512, say, is equivalent to (END-1024..END-512) and resolvable just the same?

[akx]

I'm not sure about this. Could a service do a better job if it knew about the request?