diff --git a/content/posts/five-dollars-instead-of-an-open-port.md b/content/posts/five-dollars-instead-of-an-open-port.md index bcf2b59..e1807fa 100644 --- a/content/posts/five-dollars-instead-of-an-open-port.md +++ b/content/posts/five-dollars-instead-of-an-open-port.md @@ -2,16 +2,16 @@ title: "Five dollars a month instead of an open port" date: 2021-03-16T21:50:00+07:00 draft: false -description: "Every home service you add tempts you to forward one more port on a modem you do not own. Renting a small server as a VPN hub does not remove the internet-facing listener; it replaces several you cannot patch with one you can." -summary: "The honest reason the homelab exists. Port forwarding on the ISP modem accumulates: one listener per service, on a box you do not administer. A 5 USD virtual server does not make the listener disappear - it moves it somewhere you control." +description: "Every home service you add tempts you to forward one more port. Renting a small server as a VPN hub does not remove the internet-facing listener; it replaces a growing set of them, some on appliance firmware nobody patches, with one you administer." +summary: "The honest reason the homelab exists. Port forwarding accumulates: one internet-reachable service per forward, several of them on hardware nobody will patch. A 5 USD virtual server does not make the listener disappear - it moves it somewhere you control." tags: ["homelab", "networking", "vpn", "security"] --- > *Archive note. Written in March 2021 in Hanoi, for a blog I have since retired, and the earliest post in that blog's homelab sequence - everything else grew out of it. Prices and products are of their time, and the arrangement itself did not last long: I moved to the Netherlands within about a year, and a VPN hub hosted by a Vietnamese carrier stops being the obvious choice once you live on the other side of the planet from it. What follows chronologically is [mining ETH in a homelab](/posts/mining-eth-in-a-homelab/).* -**TL;DR.** You want to reach a machine at home from outside. You can forward a port on the ISP modem, or you can rent a VPS - a virtual private server, a small Linux machine rented by the month - and run a VPN on it that everything dials out to. Both designs put a process on the internet where strangers can reach it. The difference is how many, and whose. Forwarding grows one internet-facing listener per service, on firmware you did not choose and cannot patch. The VPN hub has exactly one listener, on a machine you administer, and that count does not move when the service list grows. The reason this matters is not the first service. It is the fourth. +**TL;DR.** You want to reach a machine at home from outside. The textbook answer is to forward a port on the ISP modem. On my line that was not available at all - it sat behind carrier-grade NAT, with no public address to forward from - but the argument against it stands even where it is. What does work is renting a VPS - a virtual private server, a small Linux machine rented by the month - and running a VPN on it that everything dials out to. Both designs put a process on the internet where strangers can reach it. The difference is how many, and whose. Forwarding grows one internet-reachable service per forward, some of them on appliance firmware nobody will ever patch. The VPN hub has two, both on a machine you administer: WireGuard, and the SSH you administer it with. That count does not move when the service list grows. The reason this matters is not the first service. It is the fourth. -Concretely: the VPN was WireGuard, deployed with [Algo](https://github.com/trailofbits/algo) - Ansible scripts from Trail of Bits that stand up a personal WireGuard or IPsec server with hardened defaults - on a virtual machine rented from Viettel IDC - a Vietnamese cloud provider whose small instances sit in much the same bracket as a DigitalOcean droplet, and the hosting arm of the same carrier that supplied the line at home. Both of those choices do more work in the argument than they look, so they get their own sections. +Concretely: the VPN was WireGuard, deployed with [Algo](https://github.com/trailofbits/algo) - Ansible scripts from Trail of Bits that stand up a personal WireGuard or IPsec server with hardened defaults - on a virtual machine rented from Viettel IDC - a Vietnamese cloud provider, and the hosting arm of the same carrier that supplied the line at home. Both of those choices do more work in the argument than they look, so they get their own sections. ## The network everyone starts with @@ -19,15 +19,15 @@ Home networks are not standardised. Two flats with identical floor plans end up That is enough for what most people need. Web, streaming, games with low latency. Nothing about it is wrong. -Then one day you want to reach the PC at home while you are not at home, and to do it safely. +Then one day you want to reach the PC at home while you are not at home, and to do it safely. And you discover that the textbook answer is not on the menu. ## The ready-made answers The hosted remote-desktop tools do this job and do it well: -- **TeamViewer** is free only while your use counts as personal, and the moment it decides otherwise you are paying. The figure I had in mind at the time was 24.90 USD. I did not write down whether that was per month or per year, and I am not going to reconstruct it now - which means I cannot honestly claim the VPS was cheaper, only that it was not obviously more expensive and did considerably more. +- **TeamViewer** is free only while your use counts as personal, and the moment it decides otherwise you are paying. The figure I had in mind at the time was 24.90 USD. I did not write down whether that was per month or per year, and without that the comparison cannot be made at all: against a monthly licence the VPS at 60 USD a year is about a fifth of the price, and against an annual one it is more than double. I am not going to pick the branch that flatters the decision. - **AnyDesk** is the same shape with fewer features. -- **Chrome Remote Desktop** is genuinely free, and it is what I would still point a non-technical person at. It also renegotiates to a direct route instead of hauling traffic through Google's servers. Its catch is small: it runs in Chrome, so some key combinations never reach the far machine - press Ctrl-W expecting to close a window over there and you close a tab over here. +- **Chrome Remote Desktop** is genuinely free, and it is what I would still point a non-technical person at. It also tries to negotiate a direct peer-to-peer route rather than hauling traffic through Google's relays, though behind carrier-grade NAT - which my line was - it falls back to relaying, and the signalling goes through Google either way. Its catch is small: it runs in Chrome, so some key combinations never reach the far machine - press Ctrl-W expecting to close a window over there and you close a tab over here. So I should be straight about the decision, because the cost framing flatters it. Chrome Remote Desktop was free and solved the stated problem. Paying 5 USD a month to escape a hotkey collision would be absurd. What I actually wanted was a general-purpose machine with a public address, which happens to solve remote access as a side effect and then keeps being useful. The remote desktop was the excuse, not the reason. @@ -35,25 +35,35 @@ Buying domestically mattered more than the price did, though the price was compe Point everything at a VPN running on that VPS and the devices stop being on separate networks. They address each other as though they shared a LAN, which means the built-in remote-desktop protocols - RDP on Windows, VNC elsewhere - work with no product tier involved. Laptop, phone, tablet, all reaching the machines at home. -Worth naming what that does not buy: the VPS is a third party in the path, and unlike Chrome Remote Desktop's direct route, it is in the path for every packet. I traded a company I do not control for a rented box I do, which is a real improvement in authority and a real regression in hop count. +Worth naming what that does not buy: the VPS is in the path for every packet, always, where Chrome Remote Desktop is in the path only when it cannot get out of it. I traded a company I do not control for a rented box I do, which is an improvement in authority and a regression in hop count. ## What is actually running on it WireGuard, put there by Algo. -Algo is worth naming rather than leaving as "a VPN", because half of what makes this design defensible is which server you expose. It is Ansible, run from your own laptop against a fresh cloud instance, and its opinions are the point. It declines to install OpenVPN or Tor. It drops legacy cipher suites and protocols - no L2TP, no IKEv1, no RSA. It does not put its security on TLS. Users are declared in a config file before deployment and each one gets a generated WireGuard config and a QR code, so adding a phone is scanning a square rather than editing anything on the server. It builds on an Ubuntu LTS with unattended security upgrades switched on. +Algo is worth naming rather than leaving as "a VPN", because half of what makes this design defensible is which server you expose. It is Ansible, run from your own laptop against a fresh cloud instance, and its opinions are the point. It declines to install OpenVPN or Tor. It drops legacy cipher suites and protocols - no L2TP, no IKEv1, no RSA. It does not put its security on TLS. Users are declared in a config file before deployment and each one gets a generated WireGuard config and a QR code, so adding a phone is scanning a square rather than editing anything on the server. It builds on an Ubuntu LTS - long-term support, the flavour that receives five years of security patches - with unattended security upgrades switched on. -The WireGuard part matters more than the automation. WireGuard does not reply to packets that fail authentication. A port scan of the VPS finds nothing on that port, because the server says nothing to anyone who cannot already prove they belong. That is a materially different exposure profile from a forwarded port on a modem, where whatever is behind it answers and identifies itself to anybody who knocks. The listener exists, and it is silent. +The WireGuard part matters more than the automation. WireGuard does not reply to packets that fail authentication, so it offers no banner, no version, and no handshake to anyone who cannot already prove they belong. + +That is worth stating carefully rather than as a magic trick. A scanner still learns something: a UDP port that returns silence while the rest of the host returns port-unreachable is a fingerprint, and anyone looking for WireGuard will recognise it. What it does not get is an identified service to look up in a vulnerability database. Compare that with a forwarded port, where the thing behind it answers, names itself, and frequently gives its version. The listener exists and is findable; it just refuses to introduce itself. It is also small. WireGuard's kernel implementation is a few thousand lines against the hundreds of thousands in the IPsec and OpenVPN stacks, and that ratio is the whole reason it is possible to feel relaxed about running one on a public address. ## The port forward, and why I did not -The obvious objection: run the VPN server on the ISP modem, forward one port, save the 5 USD. If the contract has no static IP, dynamic DNS will give you a free name that tracks the address as it moves. People do this and it works. +The obvious objection: run the VPN server at home behind the ISP modem, forward one port to it, save the 5 USD. If the address changes, dynamic DNS gives you a free name that tracks it. People do this and it works. + +Be precise about what a forward actually does, because I was sloppy about this for years. It is a translation rule on the modem, not a service running there. The modem listens for nothing; it rewrites the destination and hands the packet to a machine on the inside. The exposed process is on that machine. + +Which is the part that matters. Some of those machines you patch: the VMs, the PC. Some you cannot. A camera appliance runs whatever firmware its vendor last felt like shipping, and no update you can apply will change that. The forward makes it reachable from the entire internet, and the modem in the middle is a device you did not choose and do not administer, so you cannot even put a useful rule in front of it. + +So the exposure is not "a listener on the ISP's box". It is one internet-reachable service per forward, spread across machines of mixed provenance, some of which will never receive another update. That is the argument, and it is the reason I would still do this somewhere that hands out a routable address without being asked. -The catch is that it works only where the ISP lets you configure the modem at all. Plenty of contracts ship locked firmware, and where that is true this alternative does not lose on security, it simply is not available. Where it is available, here is what it costs. +There was a second reason, and in my flat it was decisive: there was no address to forward from. The line sat behind carrier-grade NAT, CGNAT for short. The ISP hands out a private address on the WAN side of your modem and shares one real public address among a great many subscribers. Nothing outside can address your line, because your line has no address of its own. -Forwarding a port puts a process in listening state, reachable from the entire internet, on a device you did not choose, do not administer, and cannot patch. Your ISP picked it and ships its firmware on its own schedule. When a vulnerability lands in that firmware, your options are to wait. +That is worth separating from the usual advice, because dynamic DNS is normally offered as the answer to "my IP changes" and people reach for it here too. It does not help. Dynamic DNS keeps a name pointed at an address that is yours but moves. Under CGNAT the address is not yours at any moment, and pointing a name at the carrier's shared address sends strangers to the carrier, not to you. The modem will still show you a port-forwarding page. It will still accept the rule. The rule will do nothing. + +Outbound connections work fine under CGNAT - that is the whole point of it - so a tunnel dialled outward to something with a real address was the only shape available anyway. The security argument is why I would choose it; the NAT is why I could not have chosen otherwise. Since I already owned a domain, I bought the VPS and pointed a DNS record at it through Cloudflare, so I had a name rather than an address to remember. That is all Cloudflare is doing here - a VPN is not HTTP and does not travel through a caching proxy. @@ -73,33 +83,37 @@ The list does not terminate. Each addition, alone, is one small hole and a reaso Two honest caveats before the conclusion. -**The count grows only if you let it.** Three of those four services speak HTTP. One reverse proxy on one VM fronts all of them behind a single port, and now the growth is in proxy configuration rather than in forwarded ports. That is a real answer to the problem and it is what many people do. It also leaves you one internet-facing listener on a VM you patch - which, as the next section admits, is exactly where the VPN hub lands. The two designs are closer relatives than this post originally implied. +**The count grows only if you let it.** Three of those four services speak HTTP. One reverse proxy - a single web server that accepts every request and forwards it to whichever internal service the hostname belongs to - on one VM fronts all of them behind a single port, and now the growth is in proxy configuration rather than in forwarded ports. That is a real answer to the problem and it is what many people do. It also leaves you one internet-facing listener on a VM you patch - which, as the next section admits, is exactly where the VPN hub lands. The two designs are closer relatives than this post originally implied. + +**Not everything can join a VPN.** The camera is the first example in the escalation above, and a camera appliance almost certainly cannot run a VPN client, nor can most home-automation hubs. Those devices never join the tunnel. They stay on the LAN and speak to nothing outside it; what joins the tunnel are the clients, and the tunnel gives those clients a route onto the LAN where the appliances already live. -**Not everything can join a VPN.** The camera is the first example in my own escalation, and a camera appliance almost certainly cannot run a VPN client, nor can most home-automation hubs. Those devices never join the tunnel. They stay on the LAN and speak to nothing outside it; what joins the tunnel are the clients, and the tunnel gives those clients a route onto the LAN where the appliances already live. The appliance does not need to participate for this to work, which is fortunate, because it cannot. +That route does not appear by itself, and this is the step Algo does not do for you. Out of the box it builds a road-warrior server: clients dial in and reach the internet, not each other's home networks. To reach the camera you need one always-on machine at home to be a peer as well, carrying the LAN prefix in its allowed addresses and routing for the subnet, with the matching prefix configured on the client side. A machine that is already running anyway is the natural choice, which is the recurring theme of this entire homelab. The appliance still does not participate, which is fortunate, because it cannot. ## What the hub actually buys -The VPN hub does not remove the internet-facing listener. It has one: the VPN server itself, on a public address, which is the whole point of it being reachable. +The VPN hub does not remove the internet-facing listener. It has at least one: the VPN server itself, on a public address, which is the whole point of it being reachable. + +I spent this entire post arguing against internet-facing services, so let me state the real claim rather than the flattering one. The gain is not zero exposure, and it is not even down to one: Algo leaves SSH running, because that is how you administer the box. Call it two, of which one is silent and the other is the conventional hardened thing everybody already runs. -I spent this entire post arguing against internet-facing listeners, so let me state the real claim rather than the flattering one. The gain is not zero exposure. It is that the count stops tracking the service list, and that the one remaining listener sits on a machine I chose, administer and can patch the same day a vulnerability is published. Four listeners on ISP firmware become one listener on my own box, and the fifth service adds none. +The gain is that the count stops tracking the service list. Four reachable services across machines of mixed provenance become two on a single box I chose, administer, and can patch the day a fix ships. The fifth service adds none, and neither does the tenth. Where the internet-facing listener lives, in two designs -Both designs have something listening on the internet; the diagram is about how many and on whose hardware. Left: the internet reaches the ISP modem, and each service the owner adds - camera, blog, home automation, media - needs its own forwarded inbound port there, so the number of internet-reachable listeners grows with the service list, on firmware the owner did not choose and cannot patch. Right: the internet reaches one listener, a WireGuard server on a rented virtual machine the owner administers, which does not answer packets that fail authentication and so does not announce itself to a port scan. The ISP modem forwards nothing; the home network and any device away from home both dial outward to that hub and meet on it. Adding a fifth service leaves the listener count at one. Red marks an internet-facing listener in both panels, so the single red arrow on the right is the honest admission that the exposure moved rather than vanished. +Both designs expose something to the internet; the diagram is about how many things, and whose hardware runs them. Red marks a path reachable from the internet, not a box. Left: the internet reaches the ISP modem, which runs no service of its own and only rewrites destinations, handing each forwarded port to a machine behind it. Every service the owner adds - camera, blog, home automation, media - needs its own forward, so the number of internet-reachable services grows with the list. Two of those machines are the owner's VMs and get patched; the camera runs vendor firmware that will not be updated again. Right: the internet reaches one machine, the rented virtual server, running WireGuard and the SSH used to administer it. WireGuard does not answer packets that fail authentication, so it offers a scanner no banner and no version to look up. The ISP modem forwards nothing; the home network and any device away from home both dial outward to that hub and meet on it. Adding a fifth service leaves that count unchanged. The single red arrow on the right is the honest admission that the exposure moved rather than vanished. -Four listeners, none of them yours -the count grows with the service list +One reachable service per forward +the modem translates; the service runs behind it internet ISP modem -4 ports forwarded +4 forwards, no listener @@ -112,18 +126,18 @@ I spent this entire post arguing against internet-facing listeners, so let me st home auto media -firmware you did not choose and cannot patch -when a bug lands in it, your option is to wait +the camera runs vendor firmware you cannot patch +the VMs are yours; the appliance never gets another update no single one of these feels like the wrong call -One listener, and it is yours +Two listeners, and they are yours the count does not move when the list grows internet -VPS: WireGuard -1 listener, silent to scans +VPS: WireGuard + SSH +WireGuard names nothing laptop, phone away from home @@ -139,15 +153,15 @@ I spent this entire post arguing against internet-facing listeners, so let me st green dials outward; nothing at home accepts a connection appliances never join the tunnel - the clients do, and reach them -The hub does not remove the exposure. It relocates it. -Four listeners on firmware you cannot patch become one on a machine you can, and the fifth service adds none. +The hub does not remove the exposure. It relocates and caps it. +A growing set of reachable services, some unpatchable, becomes a fixed two on a box you administer. ## What it costs, stated plainly Three bills come with this that the 5 USD does not cover. -**You now run a server.** The modem was condemned for being unpatchable; the replacement is patchable and therefore must be patched. Algo takes the worst of that away by building on an Ubuntu LTS with unattended security upgrades on by default, which is more maintenance than most home routers ever receive. It does not take all of it away: unattended upgrades cover distribution packages, not the day the LTS reaches end of life and the whole instance needs rebuilding. A VPN server abandoned across that boundary is worse than the port forward it replaced. +**You now run a server.** The modem was condemned for being unpatchable; the replacement is patchable and therefore must be patched. Algo takes the worst of that away by building on an Ubuntu LTS with unattended upgrades on by default, which is more maintenance than most home routers ever receive. It does not take all of it away: unattended upgrades cover distribution packages, not the day that support window closes and the whole instance needs rebuilding. A VPN server abandoned across that boundary is worse than the port forward it replaced. **It is a single point of failure.** A forwarded port fails one service at a time. A hub fails everything at once, including the route you would use to get back in and fix it. That is a genuinely worse failure mode and the only mitigation is a second way in, which costs either money or the exact exposure you were avoiding. diff --git a/content/posts/merklision-merkle-second-preimage.md b/content/posts/merklision-merkle-second-preimage.md index ef58b14..27a4a0d 100644 --- a/content/posts/merklision-merkle-second-preimage.md +++ b/content/posts/merklision-merkle-second-preimage.md @@ -1,19 +1,21 @@ --- -title: "Merklision: composing two Merkle bugs into 2047 collisions" +title: "Merklision: composing two Merkle bugs into 2048 collisions" date: 2020-10-24T17:00:00+07:00 draft: false -description: "A SVATTT 2020 qualifier challenge asked for 2020 distinct inputs that all hash to one Merkle root. Neither of the two implementation gaps gets close on its own. Alternating between them does, because each one's output is the other one's input." -summary: "The challenge wanted 2020 distinct strings colliding on one Merkle root. A second-preimage attack gives about ten. An odd-length duplication bug gives about ten more. Alternating the two gives 2047, which I measured rather than derived." +description: "A SVATTT 2020 qualifier challenge asked for 2020 distinct inputs that all hash to one Merkle root. A second-preimage rewrite gives ten, an odd-length padding bug gives a thousand, and alternating between them gives one answer at every length up to 2048, because each rule's output is the other rule's input." +summary: "The challenge wanted 2020 distinct strings colliding on one Merkle root. A second-preimage attack gives about ten. An odd-length duplication bug gives about a thousand. Alternating the two gives one answer per length up to 2048. My 2020 solver reported 2047, and the one it missed took five years to notice." tags: ["ctf", "cryptography", "merkle-tree", "writeup"] --- -> *Archive note. Written in October 2020 for a blog I have since retired, and reproduced here as it stood. The challenge server is long gone and nothing here has been re-verified against a current implementation.* +> *Archive note. Written in October 2020 for a blog I have since retired. The challenge server is long gone, but the challenge source and my solver survive in a gist, and this version has been corrected against both, with the solver re-run in 2025. Several counts in the original were wrong, and each correction is marked where it occurs.* -**TL;DR.** A CTF challenge hands you a random string of `n` characters and asks for 2020 other distinct strings whose Merkle hash matches. A textbook second-preimage attack on a Merkle tree yields about `log2(n)` collisions, roughly ten. A second bug - odd-length input silently duplicating its last element - yields about ten more. Neither is enough, and twenty is not 2020. What closes the gap is that the two rules compose: each one's output is a valid input to the other, so you alternate them up the tree instead of picking one. Choosing `n = 1025`, which makes every level odd, produced 2047 collisions in eleven seconds. I measured that number rather than deriving it, and the distinction matters - see below. +**TL;DR.** A CTF challenge hands you a random string of `n` characters and asks for 2020 distinct strings whose Merkle hash matches. A textbook second-preimage attack on a Merkle tree yields about `log2(n)` collisions, roughly ten. A second bug - odd-length input silently duplicating its last element - yields about a thousand, half of what is needed. What closes the gap is that the two rules compose: each one's output is a valid input to the other, so you alternate them up the tree instead of picking one. Choosing `n = 1025`, which makes every row odd down to the final pair, gives one answer at every length from 1 to 2048. My solver printed 2047 on the night, and where the missing one went is the most instructive part of this writeup. ## The challenge -This was the crypto challenge in the SVATTT 2020 online qualifier - Sinh vien voi An Toan Thong Tin, the ASEAN Student Contest in Information Security - authored by `ndh`. I had played SVATTT from 2015 to 2017 on the crypto side and wrote a chain of elliptic-curve challenges for the 2018 edition, so the first thing I do with one of these is look at who set it. That tells you the ratio. Some authors write challenges that are 50% thinking and 50% code. `ndh` writes challenges that are 80% thinking and 20% code, and this one held to that. +This was the crypto challenge in the SVATTT 2020 online qualifier - Sinh vien voi An Toan Thong Tin, the ASEAN Student Contest in Information Security - authored by `ndh`. I had played SVATTT from 2015 to 2017 on the crypto side and wrote a chain of elliptic-curve challenges for the 2018 edition. I was not competing in 2020. The university team that was competing asked me, after the round closed, how the crypto challenge should have gone, and this writeup is the answer. + +The first thing I do with one of these is look at who set it. That tells you the ratio. Some authors write challenges that are 50% thinking and 50% code. `ndh` writes challenges that are 80% thinking and 20% code, and this one held to that. The server does three things: @@ -25,15 +27,15 @@ Two details from the source matter more than they look. The cap `n <= 2020` is what makes this a challenge rather than a formality: without it you would name an enormous `n` and let a single rule run away. You pick `n` within the cap, and that choice is the whole exploit. -The submission format is a comma-separated list of base64 blobs, decoded straight back into a tuple of byte strings. Elements are arbitrary bytes. There is no alphabet restriction on what you send back, which turns out to matter a great deal. Nor is there any check that a submission differs from the original tuple - only that the 2020 are pairwise distinct - so the original is itself a legal answer. +The submission format is a comma-separated list of base64 blobs, decoded straight back into a tuple of byte strings. Elements are arbitrary bytes. There is no alphabet restriction on what you send back, which turns out to matter a great deal. Nor is there any check that a submission differs from the original tuple - only that the 2020 are pairwise distinct - so the original is itself a legal answer. And nothing caps the length of a submitted list; the cap is on `n` alone. ## Merkle hashes, briefly -A Merkle tree hashes a block of data by hashing each leaf, then hashing each adjacent pair of hashes, and repeating until one root remains. Here the hash is SHA-256 truncated to its first 6 bytes, a 48-bit digest, which the challenge does to keep the base64 traffic small. It is the structure under Bitcoin and Ethereum block bodies, and its point is that you can verify one leaf against the root without holding the rest of the data. +A Merkle tree hashes a block of data by hashing each leaf, then hashing each adjacent pair of hashes, and repeating until one root remains. Here the hash is SHA-256 truncated to its first 6 bytes, a 48-bit digest, which the source says is to keep the base64 traffic small. It is the structure under Bitcoin and Ethereum block bodies, and its point is that you can verify one leaf against the root without holding the rest of the data. -Three pieces of notation. `h(x)` is the 6-byte digest of one element. `||` is byte concatenation. `M(...)` is the Merkle root of a whole input list, which is what the challenge compares. +Three pieces of notation. `h(x)` is the 6-byte digest of any byte string - a leaf, or a pair of digests glued together. `||` is byte concatenation. `M(...)` is the Merkle root of a whole input list, which is what the challenge compares. -The build, stated exactly, because the detail matters below: hash every element to get a row of digests, then repeatedly replace each adjacent pair `a, b` with `h(a || b)` until one value is left. +The build, stated exactly, because the detail matters below: hash every element to get a row of digests, then repeatedly replace each adjacent pair `a, b` with `h(a || b)` until one value is left. A one-element input never enters the loop, so `M([x])` is just `h(x)`. The property that breaks it is that every row is just a list of byte strings, and the function that consumes a row does not record which row it was. As far as `M` is concerned, rows are interchangeable. That is the root cause of everything below. @@ -43,15 +45,15 @@ A second-preimage attack means: given one input, find a different input with the Take leaves `L1, L2, L3, L4`. The first pass produces digests `h(L1) h(L2) h(L3) h(L4)`. The row above is `h(h(L1)||h(L2))` and `h(h(L3)||h(L4))`, and the root is the hash of those two concatenated. -So what two-element input reproduces that root? Not the internal nodes themselves. The implementation hashes every element before pairing anything, so you have to submit the values that *hash to* the internal nodes. The node is `h(h(L1)||h(L2))`, so the element you send is `h(L1)||h(L2)`: the two child digests, concatenated, twelve bytes. +So what two-element input reproduces that root? Call this a rewrite: submitting a higher row in place of the leaves. Not the internal nodes themselves. The implementation hashes every element before pairing anything, so you have to submit the values that *hash to* the internal nodes. The node is `h(h(L1)||h(L2))`, so the element you send is `h(L1)||h(L2)`: the two child digests, concatenated, twelve bytes. ``` -M(L1, L2, L3, L4) == M(h(L1)||h(L2), h(L3)||h(L4)) +M(L1, L2, L3, L4) = M(h(L1)||h(L2), h(L3)||h(L4)) ``` -The original writeup glossed this, and the source settles it. The solver builds its rewrite rows as `[hashes[i] + hashes[i+1] for i in range(0, len(hashes), 2)]`, and `+` on `bytes` in Python is concatenation - not addition, and not a further hash. Submitting a twelve-byte element is legal because the wire format base64-decodes to arbitrary bytes and the challenge never inspects them. +The original writeup glossed this, and the source settles it. The solver builds its rewrite rows as `[hashes[i] + hashes[i+1] for i in range(0, len(hashes), 2)]`, run after the odd-row duplicate has already been appended so the pairing is even, and `+` on `bytes` in Python is concatenation - not addition, and not a further hash. Submitting a twelve-byte element is legal because the wire format base64-decodes to arbitrary bytes and the challenge never inspects them. -Every level gives you one such rewrite, so a tree over `n` leaves gives about `log2(n)` of them. With `n` capped at 2020 that is about ten. Two hundred times short. +Every row gives you one such rewrite, so a tree over `n` leaves gives about `log2(n)` of them. With `n` capped at 2020 that is about ten. Two hundred times short. ## Rule two: the odd-length duplication @@ -61,19 +63,20 @@ The implementation has a second gap. When a row has an odd number of elements, t M(1, 2, 3, 4, 5) = M(1, 2, 3, 4, 5, 5) ``` -That is worth one collision per odd row, and there are only about ten rows. Another ten. Still short, and adding the two rules gives twenty, not 2020. +The original writeup counted this as one collision per odd row, about ten in total, and I repeated that for years. It is wrong, and badly. Nothing stops you appending a second copy, and a third: the padding cascades, and every length from 1026 up to 2048 collides with the original. Rule two on its own is worth 1023 collisions, not ten. + +So the framing the original built - two rules worth ten each, twenty against a requirement of 2020, hopeless without some trick - was never the real situation. Rule two alone gets you halfway. What the composition buys is the rest, and a much tidier account of where the total comes from. Two Merkle collision rules and their composition -Left: the four generated leaves are hashed to digests, each adjacent pair of digests is concatenated and hashed into an internal node, and the two nodes combine into the root. Right: a two-element submission where each element is the concatenation of two child digests. Because the implementation hashes every element before pairing, hashing those concatenations reproduces the internal nodes exactly, and therefore the same root. Note the elements submitted are the concatenated digests, twelve bytes each, not the six-byte internal nodes. Colour marks role: blue is what the server generated, red is what the attacker submits. Below: the odd-length rule repeats a trailing element without changing the root, and alternating the two rules is what produces the collision count the post derives. +Left: the four generated leaves are hashed to digests, each adjacent pair of digests is concatenated and hashed into an internal node, and the two nodes combine into the root. Right: a two-element submission where each element is the concatenation of two child digests. Because the implementation hashes every element before pairing, hashing those concatenations reproduces the internal nodes exactly, and therefore the same root. Note the elements submitted are the concatenated digests, twelve bytes each, not the six-byte internal nodes. Colour marks role: blue is what the server generated and hashes up from, red is what the attacker submits instead, and green is the root, which is the one value identical on both sides and the whole point of the attack. Below: the odd-length rule repeats a trailing element without changing the root - for instance the five-element list and the same list with its last element doubled - and alternating the two rules across rows is what the post derives the collision count from. - What the server generated -four leaves, hashed pairwise upward +blue is the server side, at both rows root @@ -107,7 +110,7 @@ That is worth one collision per odd row, and there are only about ten rows. Anot 12 bytes each: the child digests, concatenated -hashing them reproduces the blue row, so the root matches +hashing them reproduces the blue digest row above not the 6-byte nodes themselves - elements get hashed first input length 2 @@ -118,76 +121,85 @@ That is worth one collision per odd row, and there are only about ten rows. Anot ## Composing the rules -Treating the two rules as separate buckets gives twenty collisions. They are not separate. Each one changes the shape of the input, and the changed shape is eligible for the other rule. +Treating the two rules as separate buckets gives ten from one, a thousand from the other, and no way to add them. They are not separate. Each one changes the shape of the input, and the changed shape is eligible for the other rule. Start from `1, 2, 3, 4, 5`: 1. Odd length, so pad: collides with `1, 2, 3, 4, 5, 5`. -2. That is six elements, so collapse a level: collides with the three concatenated digest pairs of that row. +2. That is six elements, so rewrite one row up: collides with the three concatenated digest pairs of that row. 3. That row is three long, so pad it: repeat its last element to get four. -4. Expand back down a level and you are holding `1, 2, 3, 4, 5, 5, 5, 5`, a plain list of the original single-letter elements again. +4. Undo the rewrite - expand back down one row - and you are holding `1, 2, 3, 4, 5, 5, 5, 5`, a plain list of the original single-letter elements again. -One five-element input produced four collisions from the bottom two levels alone, and the last of them is longer than anything the first rule reaches by itself. There is a ceiling: padding cannot grow an input past the first power of two at or above `n`, which for `n = 5` is 8. +The original writeup stopped there and counted four. It missed one: `1, 2, 3, 4, 5, 5, 5` collides too, at length 7. The bottom two rows of a five-element input yield lengths 5 through 8 and 3 through 4, which is six answers, one of them the original. The rows above add one each, at lengths 2 and 1, for eight answers in total; the next section says why the count comes out to exactly one per length. There is a ceiling: padding cannot grow an input past the first power of two at or above `n`, which for `n = 5` is 8. -An earlier draft of this post claimed step 4 mattered because it lands back in the original alphabet, and that the server might be fussy about what a submission contains. Reading the source, it is not: elements arrive base64-decoded as arbitrary bytes and nothing inspects them. Rows of concatenated digests are just as acceptable as rows of letters. Step 4 is a consequence of the composition, not a precaution. +An earlier draft of this republished version claimed step 4 mattered because it lands back in the original alphabet, and that the server might be fussy about what a submission contains. Reading the source, it is not: elements arrive base64-decoded as arbitrary bytes and nothing inspects them. Rows of concatenated digests are just as acceptable as rows of letters. Step 4 is a consequence of the composition, not a precaution. -So the choice of `n` is the actual exploit. You want every row on the way up to have an odd count, so the padding rule fires at each one. Working the recurrence downward, `n = (((2*2-1)*2-1)*2-1)...` while staying under the cap gives `n = 1025`, whose padding ceiling is 2048. +So the choice of `n` is the actual exploit. You want every row on the way up, down to the last pair, to have an odd count, so the padding rule fires at each one. Working the recurrence downward, `n = (((2*2-1)*2-1)*2-1)...` while staying under the cap gives `n = 1025`, whose padding ceiling is 2048. -## Where 2047 comes from +## Where the count comes from -The original writeup asked how many collisions that yields and answered, in as many words, that I did not know. Having the source back, the answer is exact and a little deflating. +The original writeup asked how many collisions that yields and answered, in as many words, that I did not know. Having the source back, the answer is exact, and it is not the number the original printed. -Every answer the search produces is a list, and the lists come out at one distinct length each, from 2 elements up to 2048. That is it. One answer per length: +Take the rows of the tree for `n = 1025`: 1025, 513, 257, 129, 65, 33, 17, 9, 5, 3, 2, and then the root. -``` -2048 - 2 + 1 = 2047 -``` +Rule one lets you submit any one of those rows, expressed as concatenated digest pairs. Rule two then lets you pad whichever row you submitted, repeating its trailing element, up to that row's own ceiling. So each row contributes a contiguous band of lengths, and the bands turn out to tile the whole range without overlapping: -The ceiling of 2048 is the padding limit, the floor of 2 is the shortest row a Merkle tree can have above the root, and every length in between is reachable. The requirement was 2020, so the margin was 27 lengths - closer than it felt at the time. +| row | lengths it yields | answers | +|---|---|---| +| the leaves, 1025 | 1025 to 2048 | 1024 | +| 513 | 513 to 1024 | 512 | +| 257 | 257 to 512 | 256 | +| 129 | 129 to 256 | 128 | +| 65 | 65 to 128 | 64 | +| 33 | 33 to 64 | 32 | +| 17 | 17 to 32 | 16 | +| 9 | 9 to 16 | 8 | +| 5 | 5 to 8 | 4 | +| 3 | 3 to 4 | 2 | +| 2 | 2 | 1 | +| the root | 1 | 1 | -One wrinkle worth stating because it affects the number you can actually submit: the solver seeds its result set with the original tuple before searching, so 2047 counts the input itself. There are 2046 genuine collisions. The challenge never checks a submission against the original, only that the 2020 are pairwise distinct, so submitting it is legal and the point is moot - but 2047 and 2046 both being defensible answers is exactly the sort of thing a writeup should not leave implicit. - -I re-ran the solver against the original challenge source while preparing this post. It reproduces 2047, every length from 2 to 2048 present exactly once, in about five seconds on current hardware. +``` +1024 + 512 + 256 + 128 + 64 + 32 + 16 + 8 + 4 + 2 + 1 + 1 = 2048 +``` - -Why the answer count is 2047 -The search emits one valid answer at each possible list length. The shortest is a two-element list, the longest is a 2048-element list, which is the padding ceiling for a 1025-element input, and every length in between is reachable exactly once. The count is therefore 2048 minus 2 plus 1, which is 2047, against a requirement of 2020 - a margin of 27. The original input, 1025 elements long, sits inside that range and is counted by the solver, so 2046 of the 2047 are collisions distinct from it. - - - -One answer per list length -every length from 2 to 2048 is reachable, and each is reached exactly once - -length 2 -shortest row - -length 3 - -length 4 - -... - -length 1025 -the original - -length 2048 -padding ceiling - - - - - - -The count -2048 - 2 + 1 = 2047 -required: 2020 -margin: 27 lengths - -2047 or 2046? -2047 counts the original input, which the solver -seeds into its own result set before searching. -2046 are collisions distinct from it. Either clears -2020, and the challenge accepts the original too. +Twelve bands, each half the one above it until the last two, covering every length from 1 to 2048 exactly once. That is both why the answers come out one per length and where the total comes from. The search reaches the shorter bands without padding them directly: pad the leaves to some length, rewrite one row up, and you are holding the 513-row padded to half that length. Other collisions exist in principle - a 48-bit digest guarantees them - but these are the ones the two rules produce, and because the bands tile, the count of those is exact. + +The last band deserves a paragraph, because the original solver never found it and an earlier draft of this republished version argued it could not exist. A one-element submission `[x]` has `M([x]) = h(x)`, which looks like it demands a preimage of the 48-bit root. It does not. The root is `h(a || b)` for the two nodes beneath it, so `x = a || b` is a twelve-byte element that hashes to the root directly. It is rule one applied to the top row, nothing more. + +So why did the 2020 run print 2047? The solver as I wrote it that night harvested rewrite rows inside the padding branch: it recorded the concatenated pairs of a row only when that row had just been padded. Every row of a 1025-element tree is odd down to the final pair, so every row was harvested - except the final pair itself, which has two elements, needs no padding, and was skipped. The one answer it never produced is the one-element row, the rewrite of the root. I tidied the gist later the same day, which moved the harvest outside the branch, and never ran the tidied version: the console block below is from the version that ran first, and the number in it is the one I remembered for five years. The tidied solver finds 2048. + +One wrinkle worth stating because it affects the number you can actually submit: the solver seeds its result set with the original tuple before searching, so its count includes the input itself. The challenge never checks a submission against the original, only that the 2020 are pairwise distinct, so submitting it is legal. The requirement was 2020, so the margin was 27 lengths as the solver counted them that night - closer than it felt at the time. + + +Twelve bands of lengths, tiling the range +Each row of the tree can be submitted in place of the leaves, and then padded up to that row's own ceiling, so each row yields a contiguous band of list lengths. The leaf row of 1025 yields lengths 1025 to 2048, which is 1024 answers; the row of 513 yields 513 to 1024, which is 512; and so on halving down to the final pair, which yields the single length 2, and the rewrite of the root, the concatenation of its two children, which yields the single length 1. The twelve bands abut without overlapping and cover every length from 1 to 2048 exactly once, so the total is 1024 plus 512 plus 256 and so on down to 1 plus 1, which is 2048, two to the eleventh. The 2020 solver reported 2047 because it never harvested the final pair, which is even and so never padded. + +Each row yields a band of lengths, and the bands tile +width is proportional to the count; every band is half the one above it, until the last two + +leaf row 1025: lengths 1025-2048, 1024 answers + +row 513: 513-1024, 512 + +257: 256 + +129 + +... + +The sum +1024 + 512 + 256 + 128 + 64 + + 32 + 16 + 8 + 4 + 2 + 1 + 1 += 2048 = 2^11 +twelve terms: eleven rows, plus the rewrite of the root + +Against the requirement +required: 2020 pairwise-distinct submissions +available: 2048, one at every length from 1 to 2048 +the 2020 solver found 2047: it skipped the final pair, +which is even, so never padded, so never harvested +counts include the original tuple, which the challenge accepts ## Solving it @@ -195,11 +207,15 @@ I re-ran the solver against the original challenge source while preparing this p There is a clever way to enumerate the collisions and a stupid way. I took the stupid way, because the challenge scores a flag and not an algorithm: 1. Take the original tuple and append copies of its last element, one more each time, walking every length from 1025 up to 2048. -2. Keep a padded candidate only if its Merkle hash still equals the target. Most lengths do not survive this. +2. Keep a padded candidate only if its Merkle hash still equals the target. 3. For each survivor, walk the tree and collect the rewrite rows that rule one licenses - the concatenated digest pairs at each row where padding occurred. Add the survivor itself alongside them. 4. Check every one of those against the target again, drop anything already seen, and keep the rest. -Step 2 is the filter, and it is why this is the stupid way. Padding to an arbitrary length does not collide in general; the loop emits candidates cheaply and lets the hash comparison discard most of them, rather than being constructed to only produce valid ones. The recorded rewrites in step 3 come from the rows the implementation had to pad, which is exactly where the two rules meet. +Step 2 is a guard rather than a filter, and this is where I had the algorithm wrong for years. I assumed most padded lengths would fail the check and that the loop was cheap generation plus aggressive discarding. Running it says otherwise: all 1024 lengths from 1025 to 2048 pass. They have to, or the total could not reach the ceiling. The comparison never rejects anything at this step; it only proves the rule. + +Step 3 is where the missing answer went. "Where padding occurred" is the bug, stated as a design decision: the final pair is never padded, so its rewrite row was never collected. + +It is still the stupid way, just not for the reason I thought. The search re-derives and re-hashes candidates it could have enumerated directly, and leans on a set to absorb the duplicates - of which there are many, since the same answer is reachable by more than one route. ```console $ python ./merklision.py @@ -208,12 +224,16 @@ There are 2047 collisions Takes 11 sec to produce ``` -2047 against a requirement of 2020, in eleven seconds on 2020 hardware. Both the challenge and the solver are in [the merklision gist](https://gist.github.com/minhtt159/af8e19e2ac7088be48889ccd5c6e0e0b); every claim in this post about how the challenge behaves was checked against that source rather than against my memory of it, and several of them needed correcting. +2047 against a requirement of 2020, in eleven seconds on 2020 hardware. Both the challenge and the solver are in [the merklision gist](https://gist.github.com/minhtt159/af8e19e2ac7088be48889ccd5c6e0e0b); every claim in this post about how the challenge behaves was checked against that source rather than against my memory of it, and several of them needed correcting. The gist as it stands today, re-run while preparing this post, produces 2048 - every length from 1 to 2048 present exactly once - in about six seconds on a laptop five years newer than the one that took eleven. + +And 2048 is `2^11`, which is not a coincidence at all. The bands from the leaves down to the final pair are a geometric series, 1024 halving to 1, eleven terms, one per row of a 1025-element tree; they sum to 2047, which is the `2^11 - 1` I remembered. The root's rewrite adds the twelfth band, one more answer, and closes the sum to the ceiling. The harvest bug dropped exactly that term. My old intuition, that the power of two came from a binary choice at each level, had the right shape and the wrong mechanism: the doubling is in how many lengths a row can reach, not in a decision taken at it. + +Running the same search at smaller sizes shows it holding: `n = 5` gives 8 answers against a ceiling of 8, `n = 9` gives 16, `n = 17` gives 32, each covering every length from 1 to its own ceiling exactly once. The first-night solver gives 7, 15 and 31 for the same inputs, one short every time, which is how the bug showed itself. -Note what 2047 is not. It is tempting to read it as `2^11 - 1`, one binary choice per level of a 1025-element tree with the empty case removed, and I believed something like that for years. The lengths say otherwise: the answers are one per list length from 2 to 2048, so the count is a range size and the resemblance to a power of two is just the padding ceiling being one. +This is also what makes `n = 1025` the right pick rather than a lucky one. Every row down to the last pair lands on an odd count, so the padding rule fires at each one and every band is available. Choose 1026 instead and the leaf row is even and does not pad, so the biggest band - the leaf row's 1024 answers - is gone; the rows above it are still odd and still pad, so what is left is the 513-row's band and everything under it. ## What it was worth -Five teams solved it inside the four-hour window. The original version of this writeup drew a conclusion from that about four-person teams each landing one challenge perfectly; I have left the observation in and the inference out, because the solve count alone does not support it and I never knew how many teams competed. +About five teams solved it inside the four-hour window, as it was reported to me; I was not there. The original version of this writeup drew a conclusion from that about four-person teams each landing one challenge perfectly. I have left the observation in and the inference out, because the solve count alone does not support it and I never knew how many teams competed. -The lesson generalises past the challenge. Neither bug is exotic: one is the documented second-preimage weakness of unvalidated Merkle constructions, the other is a one-line padding convenience. Individually each is worth ten collisions and would probably survive a review as a curiosity. What made them worth 2047 is that one rule's output is the other rule's input, and nothing in the implementation stops you from alternating between them all the way up the tree. +The lesson generalises past the challenge. Neither bug is exotic: one is the documented second-preimage weakness of unvalidated Merkle constructions, the other is a one-line padding convenience. One is worth ten collisions on its own and the other about a thousand, and either would probably survive a review as a curiosity. What made them worth every length up to 2048 is that one rule's output is the other rule's input, and nothing in the implementation stops you from alternating between them all the way up the tree. diff --git a/content/posts/mining-eth-in-a-homelab.md b/content/posts/mining-eth-in-a-homelab.md index 15ff936..c132d17 100644 --- a/content/posts/mining-eth-in-a-homelab.md +++ b/content/posts/mining-eth-in-a-homelab.md @@ -7,22 +7,22 @@ summary: "The goal was to make idle hardware pay the electricity bill. The route tags: ["homelab", "proxmox", "kubernetes", "gpu", "virtualization"] --- -> *Archive note. This is two posts from a blog I have since retired, stitched together because they were two halves of one arc. The passthrough half was written in April 2021; the Kubernetes half in September 2021. The container insight that joins them landed in between, in July, and has [its own post](/posts/the-economics-that-make-cryptojacking-work/). Ethereum left proof of work in 2022, so the mining is history. The mechanics are the part that outlived it.* +> *Archive note. This is two posts from a blog I have since retired, stitched together because they were two halves of one arc. The passthrough half was written in April 2021; the Kubernetes half in September 2021, so this post carries material from five months after its own date. The container insight that joins them landed in between, in July, and has [its own post](/posts/the-economics-that-make-cryptojacking-work/). Ethereum left proof of work in 2022, so the mining is history. The mechanics are the part that outlived it.* -**TL;DR.** During the Covid-era GPU shortage I had spare PCIe slots in machines that were already powered on for other reasons, and a few cards accumulated over the years. Mining ETH on them was a way to make that hardware cover the electricity, water and internet bills. Getting a GPU into a virtual machine meant IOMMU passthrough on Proxmox, because the cards that virtualise themselves properly cost enterprise money. The Windows guest turned out to be the unstable part. Containers removed the passthrough step entirely, and once the miner was a container, Kubernetes could place it on whichever machines had a card to give it. I never wrote down what it earned, so this post cannot tell you whether it paid - only what it cost and how it was built. The scheduling pattern is what I actually kept. +**TL;DR.** During the Covid-era GPU shortage I had spare PCIe slots in machines that were already powered on for other reasons, and a few cards accumulated over the years. Mining ETH on them was a way to make that hardware cover the electricity, water and internet bills. Getting a GPU into a virtual machine meant IOMMU passthrough on Proxmox, because the cards that virtualise themselves properly cost enterprise money. The Windows guest turned out to be the unstable part. Moving the miner into a container on bare-metal nodes removed the passthrough step, and once the miner was a container, Kubernetes could place it on whichever machines had a card to give it. I never wrote down what it earned, so this post cannot tell you whether it paid - only what it cost and how it was built. The scheduling pattern is what I actually kept. ## Why bother Four things lined up in early 2021: 1. Lockdowns gave people time to trade, and crypto and equities both moved, mostly upward. -2. The same lockdowns broke the semiconductor supply chain. GPUs were the scarce part, and GPUs are what mines ETH, so mining got harder and the price went up together. +2. The same lockdowns broke the semiconductor supply chain. GPUs were the scarce part, and GPUs are what mine ETH, so mining got harder and the price went up together. 3. I was already running a homelab, which means several PCs, workstations and servers at home. Almost none of them needed graphics, so I had a lot of empty PCIe slots. 4. I already owned a handful of GPUs, accumulated across years of student machine-building. Running them at full duty felt like getting value out of money already spent. So: the goal was to cover the monthly bills, and a bit beyond. The condition attached is worth stating, because it is the whole economic argument. I would not have done this without the background to run it, and I would not have built a mining rig from nothing - the premise was spare capacity in machines already earning their keep. -That premise leaked. Over the following months I bought a GPU for this and a case to put it in, which are both on the "from scratch" side of the line I had just drawn. I am leaving the original framing in rather than quietly tightening it, because the drift from "use what I have" to "buy one more thing" is the most honest part of the story and it is what a payback calculation would have caught. I never ran one. +That premise leaked. Over the following months I bought two GPUs for this and a case to put them in, which are both on the "from scratch" side of the line I had just drawn. I am leaving the original framing in rather than quietly tightening it, because the drift from "use what I have" to "buy one more thing" is the most honest part of the story and it is what a payback calculation would have caught. I never ran one. ## Not building a rig @@ -34,23 +34,25 @@ I went the other way. The workstations already existed, so adding GPUs to them r Mining software runs on Windows or on Linux, so the card needs to reach an operating system. Since I virtualise rather than installing on bare metal, a guest by default has no idea what hardware exists underneath it. Handing it real hardware needs IOMMU, the input-output memory management unit, which is what lets a device be mapped safely into a guest's address space. -In server hardware the clean answer is SR-IOV, single-root IO virtualization, where the card presents itself as several virtual functions. Cards that do that are priced accordingly. Compare an RTX A6000 against an RTX 3090 sometime: the same chip, the same CUDA core count, wildly different price. Enterprise cards were out of reach, so consumer cards it was, and the one I bought for this was an RTX 3080 at 24.5 million VND. +In server hardware the clean answer is hardware-assisted partitioning, where one card presents itself to several guests at once. Cards that do that are priced accordingly. Compare an RTX A6000 against an RTX 3090 sometime: the same GA102 die, similar core counts, wildly different price. And the price of the card is not the whole barrier - NVIDIA gates the virtualisation behind a separate software licence, so buying the expensive card is necessary rather than sufficient. Enterprise cards were out of reach, so consumer cards it was. The two I bought for this were an RTX 3060 and an RTX 3080, the latter at 24.5 million VND. Consumer parts work for passthrough as long as the CPU and motherboard support IOMMU. I tend to buy high-end desktop CPUs, so this was never the blocker. For the hypervisor there were three serious options: QEMU-KVM, Xen, and VMware ESXi. I was poor and partial to open source, which picks QEMU-KVM. Several distributions expose it, and the friendliest of those is Proxmox. -Then Proxmox got a Windows guest with the GPU passed through. Windows over HiveOS was a personal call - I needed Windows for other things anyway, and one card, an RTX 3060, needed a driver unlock that only existed on Windows in order to reach 48 MH/s, megahashes per second, the rate at which it can attempt the mining puzzle. Without that unlock the card ran at a fraction of its capability, because the vendor shipped it deliberately limited to make it unattractive to miners. Down the rabbit hole: once it had been running a while, the Windows guest itself turned out to be the least reliable component in the stack, and rebooting it was a recurring chore rather than a one-off. +Then Proxmox got a Windows guest with the GPU passed through. Windows over HiveOS was not really a preference. One card, an RTX 3060, needed a driver unlock that only existed on Windows in order to reach 48 MH/s - megahashes per second, the rate at which it can attempt the mining puzzle - and that decided it. Without that unlock the card ran at a fraction of its capability, because the vendor shipped it deliberately limited to make it unattractive to miners. Down the rabbit hole: once it had been running a while, the Windows guest itself turned out to be the least reliable component in the stack, and rebooting it was a recurring chore rather than a one-off. -With the guest up, the rest is mundane. Install the miner, join a pool. My hashrate was small, so I picked ethermine for its 0.1 ETH payout floor, which suits a small operation - a higher floor means waiting months to see anything. +With the guest up, the rest is mundane. Install the miner and join a pool - a pool being the arrangement where many small miners submit work together and split the proceeds, because a lone consumer card has no realistic chance of finding a block on its own. Pools differ in the balance they owe you before they will pay it out, and I picked ethermine, whose 0.1 ETH floor suited a small hashrate. That threshold is the thing that decides whether payouts are a regular event or a theoretical one. I did not keep records of which it turned out to be. One quiet benefit of virtualising: when a bare-metal rig hangs, someone has to walk over and power-cycle it. Under Proxmox the guest reboots from a web interface, the same as VMware or VirtualBox. -Heat was the real problem. GPUs crammed into a workstation chassis cook. I bought a 4U case - four rack units tall, so roughly 18 cm of internal height against the 8 or so a desktop tower gives a card, which is enough room for the airflow the cards actually want - and dedicated it to this. Thermally that configuration ran about three months without drama. That is a claim about the hardware, not about the Windows guest running on it; those two things were stable and unstable respectively, at the same time. +Heat was the real problem. GPUs crammed into a workstation chassis cook. I bought a 4U case - four rack units, about 17.8 cm of external height, tall enough to take a full-height card standing up with airflow around it rather than pressed against its neighbour - and dedicated it to this. Thermally that configuration ran about three months without drama. That is a claim about the hardware, not about the Windows guest running on it; those two things were stable and unstable respectively, at the same time. ## The part that outlived the mining -By July, a more experienced friend had pointed out the obvious thing I walked past: run the miner in a Docker container and the passthrough problem disappears. He was right. Containers skip the passthrough setup, and they avoid the CPU cost of emulating a machine around the card - a virtual machine spends real cycles on virtualised interrupts and device access that a container simply does not incur. For a miner that overhead is small, because mining hammers the GPU and barely touches the CPU. The simplification is the part worth having. +By July, a more experienced friend had pointed out the obvious thing I walked past: run the miner in a Docker container and the passthrough problem goes away. That is true with one condition worth stating, because the post above spends a long time establishing the opposite habit: it only goes away if the machine holding the card runs Linux on the metal. Put Kubernetes nodes inside Proxmox guests and the card still has to be passed into the guest with IOMMU, and you have moved the problem down a layer rather than removed it. The nodes I built for this ran on the metal, so the hypervisor left the picture entirely. + +With that settled, containers skip the passthrough setup, and they avoid the CPU cost of emulating a machine around the card - a virtual machine spends real cycles on virtualised interrupts and device access that a container simply does not incur. For a miner that overhead is small, because mining hammers the GPU and barely touches the CPU. The simplification is the part worth having. Once the miner is a container, it is just another workload, and the whole apparatus for running workloads applies to it. That is where this stopped being about mining. @@ -117,20 +119,19 @@ Once the miner is a container, it is just another workload, and the whole appara ## Building the container path -The miner itself is someone else's tool. Phoenix Miner and T-Rex are both easy to drive and take a one percent developer fee, switching over to mine for their author for that share of the time. T-Rex has good Linux support, so that is the one I wrapped. +The miner itself is someone else's tool. Phoenix Miner and T-Rex are both easy to drive and take a developer fee - 0.65 and one percent respectively - by switching over to mine for their author for that share of the time. T-Rex has good Linux support, so that is the one I wrapped. Using T-Rex is a matter of reading its command-line options and filling in the fields. Nothing about it needs modification, which is the point - the less of someone else's tool you touch, the less you maintain. -**Build the image.** Write a Dockerfile on the development machine, so that what runs in production is what you tested. Docker Compose can expose a GPU to a container for local testing. Worth knowing: in production it is not Docker that enables the GPU for the container. That is the device plugin's job, and it is a different mechanism with different failure modes. +**Build the image.** Write a Dockerfile on the development machine, so that what runs in production is what you tested. Docker Compose can expose a GPU to a container for local testing. Worth knowing: in production it is not Docker driving this. The device plugin discovers the cards, advertises them to the scheduler as an allocatable resource, and tells the kubelet which card a given container got. The injection of the device nodes and driver libraries is then done by NVIDIA's container runtime - the same machinery underneath `docker --gpus`. Same mechanism, different component driving it, which matters when you are working out which layer to debug. -**Build it in CI.** Push the repository to GitHub and set up a workflow that publishes the image. GitHub hosts a container registry with a free tier around 500 MB, which is far more than a mining container needs. After that the pipeline builds the package on every push and the cluster pulls it. GitLab, Docker Hub or a self-hosted registry would work identically - GitHub just kept the number of accounts down. +**Build it in CI.** Push the repository to GitHub and set up a workflow that publishes the image. The original post put GitHub's free registry allowance at around 500 MB and called that ample; that figure is the private-package storage quota, and public packages are not metered the same way. Either way the interesting question is your base image - anything built on a CUDA runtime base is well past 500 MB before your own code lands. After that the pipeline builds the package on every push and the cluster pulls it. GitLab, Docker Hub or a self-hosted registry would work identically - GitHub just kept the number of accounts down. -**Deploy it.** With the image in a registry, four things matter on the Kubernetes side: +**Deploy it.** With the image in a registry, three things matter on the Kubernetes side: - Install a device plugin on the GPU nodes. The [NVIDIA k8s-device-plugin](https://github.com/NVIDIA/k8s-device-plugin) advertises the cards as a schedulable resource. - Set `replicas` to the number of agents you want. Where the node pool is backed by a cloud scale set - a managed group that creates identical machines on demand, which every major provider offers under its own name - that number can grow as far as the budget does. - Set a `nodeSelector` so the miner only lands on nodes that have a card. Without it the scheduler will happily place the pod somewhere with nothing to mine on. -- Set `hostAliases` for the pool's domains. This pins the names to addresses in the pod's own `/etc/hosts` so the lookups never reach the cluster resolver. The original post recommended this to avoid awkward questions from whoever runs the DNS server, which is worth naming for what it is: that is evasion, and the [companion post](/posts/the-economics-that-make-cryptojacking-work/) argues the defender's side of exactly this. Left in because it is what the post said. A working example is at [minhtt159/ghcr-tictactoe](https://github.com/minhtt159/ghcr-tictactoe). The name is left over from what the repository started as, a throwaway for testing GitHub container registry publishing; what it holds is the Dockerfile and workflow described above. Fork it, change the wallet address, and it runs. diff --git a/content/posts/the-economics-that-make-cryptojacking-work.md b/content/posts/the-economics-that-make-cryptojacking-work.md index a7281ed..91e2759 100644 --- a/content/posts/the-economics-that-make-cryptojacking-work.md +++ b/content/posts/the-economics-that-make-cryptojacking-work.md @@ -7,15 +7,17 @@ summary: "A miner that schedules itself onto any GPU node is a workload like any tags: ["security", "threat-model", "kubernetes", "gpu", "cloud"] --- -> *Archive note. Written in July 2021 for a blog I have since retired, when Ethereum still ran on proof of work. EIP-1559, the fee-market change then weeks away, redirected most of each transaction fee to be destroyed rather than paid to whoever mined the block, cutting miner income at a stroke. Ethereum left proof of work entirely the following year. The coin is gone and the numbers with it; the economic argument is the part that still holds, and it applies to whatever GPUs turn out to be scarce for next.* +> *Archive note. Written in July 2021 for a blog I have since retired, when Ethereum still ran on proof of work, meaning its network was secured by machines racing to solve a puzzle and paid for doing it - which is what made a graphics card into an income source. EIP-1559, the fee-market change then days away, redirected most of each transaction fee to be destroyed rather than paid to whoever mined the block, cutting miner income at a stroke. Ethereum left proof of work entirely the following year. The coin is gone and the numbers with it; the economic argument is the part that still holds, and it applies to whatever GPUs turn out to be scarce for next.* -**TL;DR.** Once a miner is a container scheduled by a replica count, running it on rented cloud GPUs is a straightforward calculation: mined value minus rental cost, minus the fees the miner and the pool take off the top. At mid-2021 rates that subtraction came out negative often enough that renting to mine was mostly not worth doing. The interesting case is what happens when the cost term is zero, because someone else is paying for the compute. Then there is no threshold to clear and no break-even to wait for, and that is the entire economic engine behind cryptojacking. If your organisation runs GPUs, that is what makes the credentials in front of them worth stealing. +**TL;DR.** Cryptojacking is mining cryptocurrency on computers somebody else is paying for. This post is about why the arithmetic makes that so attractive, from someone who spent a while doing the paid-for version. + +Once a miner is a container scheduled by a replica count, running it on rented cloud GPUs is a straightforward calculation: mined value minus rental cost, minus the fees the miner and the pool take off the top. At mid-2021 rates that subtraction came out negative often enough that renting to mine was mostly not worth doing. The interesting case is what happens when the cost term is zero, because someone else is paying for the compute. Then there is no threshold to clear and no break-even to wait for, and that is the entire economic engine behind cryptojacking. If your organisation runs GPUs, that is what makes the credentials in front of them worth stealing. A warning about what this post is not: I never wrote down the instance prices, the hashrates, or what any of it earned. The argument here is about the *shape* of the calculation, and you should not take "usually negative" from me as a measured result. It was my impression at the time. ## Where this picks up -Three months earlier I had been [mining ETH in a homelab](/posts/mining-eth-in-a-homelab/) by passing a GPU into a Windows virtual machine, which is a fiddly way to spend a weekend. Since then someone more experienced had pointed out that the passthrough was unnecessary: a container skips it, and once the miner is a container, Kubernetes will place it against an advertised GPU resource like any other workload. Working through what that implied is what this post is. The build instructions came later, in September, and are folded into that earlier post. +Three months earlier I had been [mining ETH in a homelab](/posts/mining-eth-in-a-homelab/) by passing a GPU into a Windows virtual machine, which is a fiddly way to spend a weekend. Since then a more experienced friend had pointed out that the passthrough was unnecessary: a container skips it, and once the miner is a container, Kubernetes will place it against an advertised GPU resource like any other workload. Working through what that implied is what this post is. The build instructions came later still, in September, and are folded into that earlier post - which therefore carries material from after this one despite its earlier date. Three months of running the original setup produced some coin - I did not record how much - some experience, and one observation I had not expected to make, which is that I had been looking at an attack surface without recognising it. @@ -23,13 +25,13 @@ Three months of running the original setup produced some coin - I did not record Cloud providers are generous with GPU instances, and renting one to mine on is entirely possible. Renting cloud capacity to farm a coin was certainly in the air: acquaintances were renting plain compute instances - a VPS, a virtual private server, with no GPU attached - to plot Chia, which is a storage-and-CPU workload rather than a GPU one. That is the same instinct applied to different hardware, not evidence about GPUs, and I should not have offered it as though it were. -The obstacle is not renting one instance; it is managing many. A single VM tops out at around four GPUs. Suppose the price moves and you want ten GPU instances. You can deploy them quickly with a tool like `ansible`, but when one of them breaks you have to work out *which* one and go fix it by hand. Ten machines with individual identities is ten things to babysit. +The obstacle is not renting one instance; it is managing many. The original post asserted that a single VM tops out at around four GPUs, which was already wrong when written - the large instance types of the day carried eight accelerators, and some carried sixteen. The argument survives the correction, because the ceiling is a ceiling wherever it sits: past it you are managing a fleet rather than a machine. Suppose the price moves and you want ten GPU instances. You can deploy them quickly with a tool like `ansible`, but when one of them breaks you have to work out *which* one and go fix it by hand. Ten machines with individual identities is ten things to babysit. Kubernetes answers that directly. On a cloud provider it grows the instance count when you need capacity and shrinks it with one action, and a pod that dies is rescheduled rather than investigated. The unit of work becomes fungible. It does start to sound like passive income. None of this is new. Back in 2017, when ETH last ran up hard, people were already building on this stack to rent GPUs and mine. In 2018 the price crashed and the technique stopped being discussed as a business. That is worth stating precisely, because the rest of this post argues the economics are permanent: what the crash ended was the version where you pay for the compute. The version where you do not has no price floor to fall through, and it never went away. -## When information technology does return on investment +## When IT does the ROI maths The idea is a few lines of configuration. The hard question is not how, it is when: when is it worth renting at all? @@ -45,7 +47,7 @@ If that comes out positive you rent. If it comes out negative, renting is not th Three ways to fund the same miner, and what each one has to clear -The same container image feeds three funding models, so the three columns differ only in who pays for the node underneath. Owning the hardware carries capital, power and cooling, and must clear a break-even period measured in months; this post does not work out whether it does. Renting cloud GPUs carries an hourly cost that the mined value has to exceed, which at the rates of the time it usually did not. Running on compute someone else pays for carries no cost term at all, so the subtraction has nothing to clear and the answer is always yes. That missing cost line, not any difference in the software, is what makes the third column an attack rather than a business. +The same container image feeds three funding models, so the three columns differ only in who pays for the node underneath. Owning the hardware carries capital, power and cooling, and must clear a break-even period measured in months; this post does not work out whether it does. A dead card reschedules its pod like any other, but you are the one buying the replacement. Renting cloud GPUs carries an hourly cost that the mined value has to exceed, which the author believed it usually did not, though he recorded no figures to check that against. Running on compute someone else pays for carries no cost term at all, so the subtraction has nothing to clear and the answer is always yes. That missing cost line, not any difference in the software, is what makes the third column an attack rather than a business. @@ -60,7 +62,7 @@ If that comes out positive you rent. If it comes out negative, renting is not th Own the hardware capital: cards, case, power supply running: electricity, cooling -recovery: reschedule; you buy +recovery: reschedule, then buy must clear: break-even, in months answer: not worked out here @@ -71,7 +73,7 @@ If that comes out positive you rent. If it comes out negative, renting is not th recovery: reschedule the pod must clear: mined minus rental -answer: usually no, in 2021 +answer: my impression, no Pay for neither capital: none @@ -96,7 +98,9 @@ That is not a hypothetical. It is the standing economic reason that idle compute Two things fit that description uncomfortably well in any organisation that runs GPUs. -The first is machine-learning serving. Mining is not the only reason to buy a GPU fleet; the far more common one is running models, and a business that serves model predictions puts an inference endpoint - an internal service that takes a request and answers it using a model on a GPU - in front of that hardware. The credentials for such a service are handled by more systems than anyone has fully enumerated, and by design they sit in front of exactly the hardware a miner wants. +The first is machine-learning serving. Mining is not the only reason to buy a GPU fleet; the far more common one is running models, and a business that serves model predictions puts an inference endpoint - an internal service that takes a request and answers it using a model on a GPU - in front of that hardware. + +A credential for that endpoint is not by itself enough, and it is worth being exact about why, because the sloppy version of this claim is everywhere. Being able to send inference requests is not being able to run a miner; sitting next to a GPU is not executing on it. What matters is the surrounding platform rather than the endpoint: the control-plane credential that can deploy a new model server, the model-loading path that will fetch and unpickle an artifact you supply, the notebook service that exists precisely to run arbitrary code on the expensive hardware. Those meet the bar. The inference key on its own does not, and an attacker who has one is looking for the next hop. The second is anything that shapes how a node comes up. A cloud scale set - the managed group that creates identical machines on demand, which every major provider offers under its own name - takes an initialisation script that runs on every instance it ever creates, with that instance's authority. If that script can be edited, editing it once is enough, and nobody reads it again after the first time it works. @@ -104,13 +108,14 @@ The second is anything that shapes how a node comes up. A cloud scale set - the The defensive reading is not "watch for miners". Signature-matching a known miner binary catches the lazy version and nothing else. The useful reading is that GPU capacity is a directly monetisable asset on your balance sheet, and should be protected the way other directly monetisable assets are: -- **Treat inference and GPU-scheduling credentials as money.** Scope them, rotate them, and alert on use from somewhere they have never been used from. They are not read-only API keys; they are a way to spend your compute budget. +- **Treat GPU-scheduling credentials as money, and inference credentials as the hop before them.** Scope them, rotate them, and alert on use from somewhere they have never been used from. They are not read-only API keys; they are a way to spend your compute budget. - **Version and review node initialisation the way you review production code.** A mutable startup script attached to a scale set is an unreviewed root shell that runs on every machine the group creates. -- **Alert on utilisation, not on binaries.** Sustained high GPU utilisation with no corresponding job, or instance counts that grow without a workload behind them, describe the condition regardless of which miner is running. The billing anomaly and the capacity anomaly are the same event seen from two directions. +- **Alert on utilisation, not on binaries or names.** DNS-based detection is barely better than signatures: a pod spec can pin a pool's name to an address in the pod's own hosts file, so the lookup never reaches your resolver, and anyone who can submit a pod spec can do it. What does not dodge is the resource itself - sustained GPU utilisation with no job behind it, or instance counts growing without a workload to explain them. The billing anomaly and the capacity anomaly are the same event seen from two directions. +- **Assume the miner will hide in the noise.** The obvious counter to utilisation alerting is to throttle, sitting at a level that looks like ordinary load, and an attacker paying nothing has no reason to be greedy. That is an argument for baselining what each workload normally draws rather than alerting on a single global threshold. - **Ask what a compromised credential is worth in cash.** Most credential risk gets modelled as data loss, because that is what a breach usually costs. A credential in front of a GPU fleet has a second price: what an attacker can bill you for by simply using it. GPUs made that second price obvious earlier than the rest of the estate did, because the hardware was scarce and the resale value of its output was quoted publicly, by the hour. ## Closing -EIP-1559 is coming, and at the rates involved renting GPUs legitimately to mine ETH cannot be made profitable anyway. That kills this particular instance of the arithmetic, not the arithmetic. Some future coin, or some future thing GPUs are scarce for, will rebuild the same subtraction, and the same term will be the one worth removing. +EIP-1559 is coming, and my impression at the time - an impression, per the caveat at the top, not a figure I ever wrote down - was that renting GPUs at market rates to mine ETH was not clearing its own cost. That kills this particular instance of the arithmetic, not the arithmetic. Some future coin, or some future thing GPUs are scarce for, will rebuild the same subtraction, and the same term will be the one worth removing. -The original version of this post signed off by joking that publishing it would cut the number of instances I could quietly rent. I have kept the line in view because it dates the piece honestly, but it does not survive the argument above it: if renting is not profitable, publishing costs me nothing, and the only version the disclosure could affect is the version I was arguing against. The interesting work was never the mining. +The original version of this post signed off by joking that publishing it would cut the number of instances I could quietly rent. Two things about that line. The renting was on my own accounts, with my own card, the same way the hardware at home was mine - "quietly" meant without telling anyone, not without paying. And the joke does not survive the argument above it anyway: if renting is barely profitable, publishing costs me nothing, and the only version the disclosure could affect is the version I was arguing against. The interesting work was never the mining.