📦🔧：import large GitHub files as raw content

[DerekNonGeneric]

GitHub omits inline content and reports encoding: "none" for repository files
larger than 1 MB, which made the importer pass an invalid encoding to Buffer.

This uses the same authenticated client to fetch raw bytes at the original ref,
then decodes them exactly as the inline path does. Mocked HTTP tests cover auth,
ref stability, JSON text, ordinary base64 files, and request/body failures.

Fixes #98.

Summary by CodeRabbit

• Bug Fixes
  • Large repository files can now be retrieved when GitHub omits their contents from file metadata. Requests preserve authentication and the specified branch or reference.
  • Download errors are reported, and file contents are returned as UTF-8 text.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Essentials

Run ID: a184db69-8bdc-4b51-849a-0ee5591ab51e

📥 Commits

Reviewing files that changed from the base of the PR and between 8177c45 and 35e848d.

📒 Files selected for processing (3)

• .changeset/large-files-use-raw-contents.md
• packages/gh-file-importer/src/index.ts
• packages/gh-file-importer/test/large-files.ts

Included review availability: This review used your included allowance. 2 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 4 reviews per hour.

---

📝 Walkthrough

Walkthrough

fetchFileText now requests raw file content when GitHub returns encoding none. The raw request preserves authentication and the requested repository location, including an optional ref. The importer decodes the response as UTF-8 and propagates request and body-read errors.

Changes

Large-file retrieval

Layer / File(s)	Summary
Raw-content fallback and validation packages/gh-file-importer/src/index.ts, packages/gh-file-importer/test/large-files.ts, .changeset/large-files-use-raw-contents.md	fetchFileText includes the requested path and optional ref in its metadata request. When the response encoding is none, it requests raw content with the authenticated Octokit client and decodes the response as UTF-8. Tests cover request parameters, base64 decoding, JSON-like text, HTTP errors, and body-read errors. The changeset records a patch release for @openinf/gh-file-importer.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant fetchFileText
  participant Octokit
  participant GitHub as GitHub Contents API
  fetchFileText->>Octokit: Request file metadata with repository, path, and optional ref
  Octokit->>GitHub: Fetch file metadata
  GitHub-->>Octokit: Return encoding none and omitted content
  fetchFileText->>Octokit: Request raw content with the same location
  Octokit->>GitHub: Fetch raw content
  GitHub-->>Octokit: Return response stream
  fetchFileText->>fetchFileText: Decode response as UTF-8

Loading

Merge Risk: ⚪ Minimal · up to 35e84

The large-file fallback has no identified merge-blocking issue; it is ready for normal merge checks.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 2 files. (1 skipped: 1 …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: importing large GitHub files through raw content. The emojis add noise, but they do not prevent the title from being specific and understandable.
Linked Issues check	✅ Passed	The implementation satisfies the coding requirements in [#98]. fetchFileText preserves owner, repo, path, and the requested ref for both requests. It uses the same Octokit client with the ra…
Out of Scope Changes check	✅ Passed	The changes remain within [#98]. The source change implements the raw Contents API fallback. The tests verify the required behavior. The changeset documents the same behavior. No unrelated product beh…

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 2 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

_{Comment @coderabbitai help to get the list of available commands.}