Repository navigation
Make Hash platform independent. Always 8byte little endian - #173
bjornkellner wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The README and matching spec descriptions need to reflect fixed 64-bit little-endian hashes, since the current native-format guidance can produce incompatible primary keys on big-endian or 32-bit platforms.
1 open issue
What changed in this PR
This PR makes URL database hashes platform-independent by explicitly decoding SHA-256 digest bytes as 64-bit little-endian integers.
Changes:
- Use unsigned 64-bit little-endian decoding for
documentdb_hash. - Use signed 64-bit little-endian decoding for
autopingdb_hash.
| File | Description |
|---|---|
lib/twingly/url/hasher.rb |
Replaces native integer decoding with explicit 64-bit little-endian formats. |
Review effort: Balanced
|
|
||
| def documentdb_hash(url) | ||
| SHA256_DIGEST.digest(url).unpack("L!")[0] | ||
| SHA256_DIGEST.digest(url).unpack("Q<")[0] |
There was a problem hiding this comment.
Will fix unless this PR is rejected.
There was a problem hiding this comment.
Updated the README and matching spec descriptions to specify fixed 64-bit little-endian values, preserving signedness. Commit: 67897b1.
No, I can't imagine we've made it platform dependent on purpose. I haven't really thought about what The hasher was originally just trying to replicate our legacy C# implementation, but if it was platform dependent it probably wasn't by design either 😅 |
roback
left a comment
There was a problem hiding this comment.
Nice find! I think this is a good change, even if it most likely won't matter in reality.
Could be nice to verify that the output is the same for a bunch of URLs, even though it should be according to the documentation.
| def autopingdb_hash(url) | ||
| SHA256_DIGEST.digest(url).unpack("q")[0] | ||
| SHA256_DIGEST.digest(url).unpack("q<")[0] | ||
| end |
There was a problem hiding this comment.
We don't use this method anywhere, and the same with the taskdb_hash method above. We should probably also rename documentdb_hash to something better, as none of the old databases they're referencing exists anymore.
Nothing we need to do here though, just thought I would mention it.
Co-authored-by: bjornkellner <7229061+bjornkellner@users.noreply.github.com>
Thanks, yes it will probably won't every matter. I tried it on a few urls, not a lot thought. There is also one in the test. I don't think this change need to verified with a lot of urls. |

Noticed that the hash we use is platform dependent.
Is that by design? Intuitively I would have thought that the hash was platform independent.