Hashing files versus hashing text
A hash function does not operate on text. It operates on bytes. Everything confusing about hashing comes back to that single fact: the same characters on screen can be different byte sequences, and different byte sequences produce completely different hashes. If you hash "the same thing" twice and get two different digests, the bytes were not actually the same.
Text becomes bytes before it is hashed
When you hash a string, something has to turn those characters into bytes first, and that something is a text encoding. The word café is five characters but a different number of bytes depending on the encoding chosen.
| Text | Encoding | Bytes | Byte count |
|---|---|---|---|
café | UTF-8 | 63 61 66 C3 A9 | 5 |
café | UTF-16LE | 63 00 61 00 66 00 E9 00 | 8 |
café | Latin-1 | 63 61 66 E9 | 4 |
Three different byte sequences, three different hashes, for text that looks identical. ASCII-only text sidesteps this because its bytes are the same across these encodings, which is why people often never notice the issue until a non-ASCII character appears. If a hash you compute does not match one someone else computed, the first thing to check is whether you both used UTF-8.
A related trap is an invisible byte-order mark. Some editors prepend a BOM (EF BB BF in UTF-8) when saving, and that BOM is part of the bytes, so it changes the hash even though nothing visible changed.
Line endings quietly change the hash
The other silent source of mismatches is line endings. Windows terminates lines with carriage return plus line feed (\r\n, bytes 0D 0A), while Unix and macOS use a lone line feed (\n, byte 0A). A single newline is one extra byte on Windows.
"line1\nline2" -> 11 bytes
"line1\r\nline2" -> 12 bytes (different hash)
Version control tools make this worse by rewriting line endings on checkout. Git with core.autocrlf=true stores \n but hands you \r\n on a Windows checkout, so a file hashed after cloning on Windows will not match the same file hashed on Linux. When you need a stable hash of text across platforms, normalize line endings first, usually to \n, and fix the encoding to UTF-8 without a BOM. Then the bytes are deterministic.
Verifying a download
Verifying a file is the opposite problem and much cleaner, because a binary file has no encoding ambiguity. Its bytes are its bytes. A publisher posts a checksum, you hash the file you received, and the two digests either match or they do not.
The workflow:
- Download the file and the published hash, ideally the hash over a separate channel so an attacker cannot swap both.
- Compute the hash of the file locally. On the command line that is
sha256sum file.isoon Linux,shasum -a 256 file.isoon macOS, orGet-FileHash file.isoin PowerShell. - Compare the full digest, not just the first few characters. A truncated comparison can collide by chance and defeats the point.
If the hashes differ, the file is not the one the publisher hashed. That could be an incomplete download, a corrupted transfer, or a tampered file. You cannot tell which from the hash alone, only that the bytes are not identical.
One practical note: pick the algorithm the publisher used. If they posted a SHA-256, compute SHA-256, not MD5 or SHA-1. Mixing algorithms guarantees a mismatch that means nothing.
To check your understanding of how encoding and whitespace shift a digest, hash the same word with and without a trailing newline, or with an accented character, and watch the output change completely. The Hash Generator computes several algorithms at once so you can compare them on the same input.