Skip to content

Make Hash platform independent. Always 8byte little endian - #173

Open
bjornkellner wants to merge 2 commits into
masterfrom
platform_independent
Open

bjornkellner wants to merge 2 commits into
masterfrom
platform_independent

Conversation

@bjornkellner

@bjornkellner bjornkellner commented Nov 6, 2026 •

Copy link
Copy Markdown

Noticed that the hash we use is platform dependent.
Is that by design? Intuitively I would have thought that the hash was platform independent.

@bjornkellner
bjornkellner requested a balanced review from Copilot November 6, 2026 09:02
@bjornkellner bjornkellner changed the title Hash is is not platform dependant. Always 8byte little endian Make Hash platform independent. Always 8byte little endian Nov 6, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The README and matching spec descriptions need to reflect fixed 64-bit little-endian hashes, since the current native-format guidance can produce incompatible primary keys on big-endian or 32-bit platforms.

1 open issue
What changed in this PR

This PR makes URL database hashes platform-independent by explicitly decoding SHA-256 digest bytes as 64-bit little-endian integers.

Changes:

  • Use unsigned 64-bit little-endian decoding for documentdb_hash.
  • Use signed 64-bit little-endian decoding for autopingdb_hash.
File Description
lib/​twingly/​url/​hasher.rb Replaces native integer decoding with explicit 64-bit little-endian formats.

Review effort: Balanced

Comment thread lib/twingly/url/hasher.rb

def documentdb_hash(url)
SHA256_DIGEST.digest(url).unpack("L!")[0]
SHA256_DIGEST.digest(url).unpack("Q<")[0]

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will fix unless this PR is rejected.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the README and matching spec descriptions to specify fixed 64-bit little-endian values, preserving signedness. Commit: 67897b1.

@roback

roback commented Nov 10, 2026

Copy link
Copy Markdown
Member

Is that by design? Intuitively I would have thought that the hash was platform independent.

No, I can't imagine we've made it platform dependent on purpose. I haven't really thought about what L! actually means (docs).

The hasher was originally just trying to replicate our legacy C# implementation, but if it was platform dependent it probably wasn't by design either 😅

@roback roback left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice find! I think this is a good change, even if it most likely won't matter in reality.

Could be nice to verify that the output is the same for a bunch of URLs, even though it should be according to the documentation.

Comment thread lib/twingly/url/hasher.rb
Comment on lines 26 to 28
def autopingdb_hash(url)
SHA256_DIGEST.digest(url).unpack("q")[0]
SHA256_DIGEST.digest(url).unpack("q<")[0]
end

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't use this method anywhere, and the same with the taskdb_hash method above. We should probably also rename documentdb_hash to something better, as none of the old databases they're referencing exists anymore.

Nothing we need to do here though, just thought I would mention it.

Co-authored-by: bjornkellner <7229061+bjornkellner@users.noreply.github.com>
@bjornkellner

Copy link
Copy Markdown
Author

Nice find! I think this is a good change, even if it most likely won't matter in reality.

Could be nice to verify that the output is the same for a bunch of URLs, even though it should be according to the documentation.

Thanks, yes it will probably won't every matter. I tried it on a few urls, not a lot thought. There is also one in the test. I don't think this change need to verified with a lot of urls.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants