Return-Path: Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 052CFC0032 for ; Thu, 19 Oct 2023 08:12:26 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id DF515424C9 for ; Thu, 19 Oct 2023 08:12:25 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org DF515424C9 Authentication-Results: smtp2.osuosl.org; dkim=pass (2048-bit key) header.d=acinq.fr header.i=@acinq.fr header.a=rsa-sha256 header.s=google header.b=GwgMSFIT X-Virus-Scanned: amavisd-new at osuosl.org X-Spam-Flag: NO X-Spam-Score: -2.088 X-Spam-Level: X-Spam-Status: No, score=-2.088 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, T_SPF_PERMERROR=0.01] autolearn=ham autolearn_force=no Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2BfDDeH1Q2BD for ; Thu, 19 Oct 2023 08:12:22 +0000 (UTC) Received: from mail-lf1-x133.google.com (mail-lf1-x133.google.com [IPv6:2a00:1450:4864:20::133]) by smtp2.osuosl.org (Postfix) with ESMTPS id BC40A400C7 for ; Thu, 19 Oct 2023 08:12:21 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org BC40A400C7 Received: by mail-lf1-x133.google.com with SMTP id 2adb3069b0e04-507a936f4a9so6001402e87.1 for ; Thu, 19 Oct 2023 01:12:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=acinq.fr; s=google; t=1697703139; x=1698307939; darn=lists.linuxfoundation.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=JC7/akCsGbOecDTAxWfbwLKp/Hdejpm70pMXWVoCyWE=; b=GwgMSFIT1wZyoIqhzlCYEIMl70XoClC2DDb0BiaNzYvYn1dimFXQGU+upPD64uOg+j jdJHaUiz+IIrzw6rUNDmnW7J93yQSrrnIgPouylkGIED7p5oADo+cNR/O9/TLT4T1rUG sdpaVnrJYMhdmG3wj+emaawoMdiMnB+Q43O72Wa1dRoGPvdXQGKwAju1zXfpABm6206E GwiumzN9ftK6bXkSW2Ll6pOhZAoxzR3HJ6Yqjlb3B92c4w0tuQwhMBA0TCDmnHO8JXS4 7GEUS0txBfZ5qiErH6srxweW2g7wc06LGpuokEb5ZhTKBJaQM/o1WulETJt6OhYUSV9L Kglg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697703139; x=1698307939; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=JC7/akCsGbOecDTAxWfbwLKp/Hdejpm70pMXWVoCyWE=; b=o4kJz6j6VvYQoJSsrAy9hGFriAnw0PxqJXh9Jjk2tPPBxmOqDUBS69U/2O+j6Pd02p cLSp9CysLsmJSgQa2zgCzRgjCpUmENAFXbm1cCPZMv4OaXTMI9qTzoFEqcsThaCChVz7 fxDHbcWPpABP+BZlLORuqOe7YCFBpIg/byua/CwndB705XRUu2r1BlkaRFOZxFKOAdLD QMSG73knkPI4D7ooTwWCBtOGKPcUzqHOMBHiYDW7Bx8mzhsZjVqfxiU1q8RcTXl+Can3 HEFpjGUdXSDcuewLpToxBP4iuDGZBXD992mQETYZsK8KbB3WpBnKo2gNbG7MIcz1GZ6Z NcoQ== X-Gm-Message-State: AOJu0YyqT+ie6HQHZUlvuZNEJOh7kTxEcxaRSO/jHNcOj0ksPrW70cHU m/vai3i1fSZeqhUviiMyz6ObSbPGqdlQ6KErRn8+gg== X-Google-Smtp-Source: AGHT+IEJV/b4OQhjXfHxm2BE3Iq1LGXwSpOa8l+qfieyW+Bqo8b8TDoeC3OYcWzkT5Vvza0Id/nzg//9lBIfbLXpKYM= X-Received: by 2002:ac2:47eb:0:b0:507:a66b:c9a1 with SMTP id b11-20020ac247eb000000b00507a66bc9a1mr847497lfp.17.1697703139479; Thu, 19 Oct 2023 01:12:19 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Bastien TEINTURIER Date: Thu, 19 Oct 2023 10:12:08 +0200 Message-ID: To: Antoine Riard , Bitcoin Protocol Discussion Content-Type: multipart/alternative; boundary="00000000000090385906080d509c" X-Mailman-Approved-At: Thu, 19 Oct 2023 08:47:03 +0000 Cc: "lightning-dev\\\\@lists.linuxfoundation.org" Subject: Re: [bitcoin-dev] [Lightning-dev] Full Disclosure: CVE-2023-40231 / CVE-2023-40232 / CVE-2023-40233 / CVE-2023-40234 "All your mempool are belong to us" X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Oct 2023 08:12:26 -0000 --00000000000090385906080d509c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks Antoine for your work on this issue. I confirm that eclair v0.9.0 contains the migitations described. Eclair has been watching the mempool for preimages since its very early versions (years ago), relying on Bitcoin Core's ZMQ notifications for incoming transactions. I believe this guarantees that we see the HTLC success transaction, even if it is immediately replaced (as long as we don't overflow the ZMQ limits). I agree with Matt though that more fundamental work most likely needs to happen at the bitcoin layer to allow L2 protocols to be more robust against that class of attacks. Thanks, Bastien Le mer. 18 oct. 2023 =C3=A0 11:07, Antoine Riard via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> a =C3=A9crit : > The disclosure mails noted a 3rd mitigation beyond mempool scanning and > transaction re-signing / re-broadcasting, namely bumping CLTV delta. > > Generally bumping CLTV delta is a basic line of mitigations for a lot of > lightning attacks, as it gives opportunity to node operators to intervene > and re-broadcast their time-sensitive transactions on other interfaces (e= .g > a secondary full-node if the first one is eclipsed). > > About the second mitigation transaction re-signing, if done correctly at > least sounds to put an economic cost (denominated in fees / feerates) on > the attack. This is unclear to me if the game-theory of this cost holds. > > One thing which sounds to me making the attack harder is stratum v2 > deployment, as you're increasing the number of miners which might do thei= r > own block templates, and therefore the number of miners' mempools where a= n > attacker has to successfully continuously replace in cycles channels > counterparties transactions. > > A replacement buffer or history of transactions at the mempool level migh= t > be a mitigation to this attack. I believe this is yet to be seen if it ca= n > be made robust enough. > > I don't know if folks like tadge or rusty who have been involved in the > early design of lightning have more ideas of mitigations. Fees was noted = as > a hard issue in the original paper. > > Le mer. 18 oct. 2023 =C3=A0 01:17, Matt Corallo a > =C3=A9crit : > >> There appears to be some confusion about this issue and the mitigations. >> To be clear, the deployed >> mitigations are not expected to fix this issue, its arguable if they >> provide anything more than a PR >> statement. >> >> There are two discussed mitigations here - mempool scanning and >> transaction re-signing/re-broadcasting. >> >> Mempool scanning relies on regularly checking the mempool of a local nod= e >> to see if we can catch the >> replacement cycle mid-cycle. It only works if wee see the first >> transaction before the second >> transaction replaces it. >> >> Today, a large majority of lightning nodes run on machines with a Bitcoi= n >> node on the same IP >> address, making it very clear what the "local node" of the lightning nod= e >> is. An attacker can >> trivially use this information to connect to said local node and do the >> replacement quickly, >> preventing the victim from seeing the replacement. >> >> More generally, however, similar discoverability is true for mining >> pools. An attacker performing >> this attack is likely to do the replacement attack on a miner's node >> directly, potentially reducing >> the reach of the intermediate transaction to only miners, such that the >> victim can never discover it >> at all. >> >> The second mitigation is similarly pathetic. Re-signing and >> re-broadcasting the victim's transaction >> in an attempt to get it to miners even if its been removed may work, if >> the attacker is super lazy >> and didn't finish writing their attack system. If the attacker is >> connected to a large majority of >> hashrate (which has historically been fairly doable), they can simply do >> their replacement in a >> cycle aggressively and arbitrarily reduce the probability that the >> victim's transaction gets confirmed. >> >> Now, the above is all true in a spherical cow kinda world, and the P2P >> network has plenty of slow >> nodes and strange behavior. Its possible that these mitigations might, b= y >> some stroke of luck, >> happen to catch such an attack and prevent it, because something took >> longer than the attacker >> intended or whatever. But, that's a far cry from any kind of material >> "fix" for the issue. >> >> Ultimately the only fix for this issue will be when miners keep a histor= y >> of transactions they've >> seen and try them again after they may be able to enter the mempool >> because of an attack like this. >> >> Matt >> >> On 10/16/23 12:57 PM, Antoine Riard wrote: >> > (cross-posting mempool issues identified are exposing lightning chan t= o >> loss of funds risks, other >> > multi-party bitcoin apps might be affected) >> > >> > Hi, >> > >> > End of last year (December 2022), amid technical discussions on eltoo >> payment channels and >> > incentives compatibility of the mempool anti-DoS rules, a new >> transaction-relay jamming attack >> > affecting lightning channels was discovered. >> > >> > After careful analysis, it turns out this attack is practical and >> immediately exposed lightning >> > routing hops carrying HTLC traffic to loss of funds security risks, >> both legacy and anchor output >> > channels. A potential exploitation plausibly happening even without >> network mempools congestion. >> > >> > Mitigations have been designed, implemented and deployed by all major >> lightning implementations >> > during the last months. >> > >> > Please find attached the release numbers, where the mitigations should >> be present: >> > - LDK: v0.0.118 - CVE-2023 -40231 >> > - Eclair: v0.9.0 - CVE-2023-40232 >> > - LND: v.0.17.0-beta - CVE-2023-40233 >> > - Core-Lightning: v.23.08.01 - CVE-2023-40234 >> > >> > While neither replacement cycling attacks have been observed or >> reported in the wild since the last >> > ~10 months or experimented in real-world conditions on bitcoin mainet, >> functional test is available >> > exercising the affected lightning channel against bitcoin core mempool >> (26.0 release cycle). >> > >> > It is understood that a simple replacement cycling attack does not >> demand privileged capabilities >> > from an attacker (e.g no low-hashrate power) and only access to basic >> bitcoin and lightning >> > software. Yet I still think executing such an attack successfully >> requests a fair amount of bitcoin >> > technical know-how and decent preparation. >> > >> > From my understanding of those issues, it is yet to be determined if >> the mitigations deployed are >> > robust enough in face of advanced replacement cycling attackers, >> especially ones able to combine >> > different classes of transaction-relay jamming such as pinnings or >> vetted with more privileged >> > capabilities. >> > >> > Please find a list of potential affected bitcoin applications in this >> full disclosure report using >> > bitcoin script timelocks or multi-party transactions, albeit no >> immediate security risk exposure as >> > severe as the ones affecting lightning has been identified. Only >> cursory review of non-lightning >> > applications has been conducted so far. >> > >> > There is a paper published summarizing replacement cycling attacks on >> the lightning network: >> > >> https://github.com/ariard/mempool-research/blob/2023-10-replacement-pape= r/replacement-cycling.pdf >> > < >> https://github.com/ariard/mempool-research/blob/2023-10-replacement-pape= r/replacement-cycling.pdf >> > >> > >> > ## Problem >> > >> > A lightning node allows HTLCs forwarding (in bolt3's parlance accepted >> HTLC on incoming link and >> > offered HTLC on outgoing link) should settle the outgoing state with >> either a success or timeout >> > before the incoming state timelock becomes final and an asymmetric >> defavorable settlement might >> > happen (cf "Flood & Loot: A Systematic Attack on The Lightning Network= " >> section 2.3 for a classical >> > exposition of this lightning security property). >> > >> > Failure to satisfy this settlement requirement exposes a forwarding ho= p >> to a loss of fund risk where >> > the offered HTLC is spent by the outgoing link counterparty's >> HTLC-preimage and the accepted HTLC is >> > spent by the incoming link counterparty's HTLC-timeout. >> > >> > The specification mandates the incoming HTLC expiration timelock to be >> spaced out by an interval of >> > `cltv_expiry_delta` from the outgoing HTLC expiration timelock, this >> exact interval value being an >> > implementation and node policy setting. As a minimal value, the >> specification recommends 34 blocks >> > of interval. If the timelock expiration I of the inbound HTLC is equal >> to 100 from chain tip, the >> > timelock expiration O of the outbound HTLC must be equal to 66 blocks >> from chain tip, giving a >> > reasonable buffer of reaction to the lightning forwarding node. >> > >> > In the lack of cooperative off-chain settlement of the HTLC on the >> outgoing link negotiated with the >> > counterparty (either `update_fulfill_htlc` or `update_fail_htlc`) when >> O is reached, the lightning >> > node should broadcast its commitment transaction. Once the commitment >> is confirmed (if anchor and >> > the 1 CSV encumbrance is present), the lightning node broadcasts and >> confirms its HTLC-timeout >> > before I height is reached. >> > >> > Here enter a replacement cycling attack. A malicious channel >> counterparty can broadcast its >> > HTLC-preimage transaction with a higher absolute fee and higher feerat= e >> than the honest HTLC-timeout >> > of the victim lightning node and triggers a replacement. Both for >> legacy and anchor output channels, >> > a HTLC-preimage on a counterparty commitment transaction is malleable, >> i.e additional inputs or >> > outputs can be added. The HTLC-preimage spends an unconfirmed and >> unrelated to the channel parent >> > transaction M and conflicts its child. >> > >> > As the HTLC-preimage spends an unconfirmed input that was already >> included in the unconfirmed and >> > unrelated child transaction (rule 2), pays an absolute higher fee of a= t >> least the sum paid by the >> > HTLC-timeout and child transaction (rule 3) and the HTLC-preimage >> feerate is greater than all >> > directly conflicting transactions (rule 6), the replacement is >> accepted. The honest HTLC-timeout is >> > evicted out of the mempool. >> > >> > In an ulterior move, the malicious counterparty can replace the parent >> transaction itself with >> > another candidate N satisfying the replacement rules, triggering the >> eviction of the malicious >> > HTLC-preimage from the mempool as it was a child of the parent T. >> > >> > There is no spending candidate of the offered HTLC output for the >> current block laying in network >> > mempools. >> > >> > This replacement cycling tricks can be repeated for each rebroadcast >> attempt of the HTLC-timeout by >> > the honest lightning node until expiration of the inbound HTLC timeloc= k >> I. Once this height is >> > reached a HTLC-timeout is broadcast by the counterparty's on the >> incoming link in collusion with the >> > one on the outgoing link broadcasting its own HTLC-preimage. >> > >> > The honest Lightning node has been "double-spent" in its HTLC >> forwarding. >> > >> > As a notable factor impacting the success of the attack, a lightning >> node's honest HTLC-timeout >> > might be included in the block template of the miner winning the block >> race and therefore realizes a >> > spent of the offered output. In practice, a replacement cycling attack >> might over-connect to miners' >> > mempools and public reachable nodes to succeed in a fast eviction of >> the HTLC-timeout by its >> > HTLC-preimage. As this latter transaction can come with a better >> ancestor-score, it should be picked >> > up on the flight by economically competitive miners. >> > >> > A functional test exercising a simple replacement cycling of a HTLC >> transaction on bitcoin core >> > mempool is available: >> > https://github.com/ariard/bitcoin/commits/2023-test-mempool >> > >> > >> > ## Deployed LN mitigations >> > >> > Aggressive rebroadcasting: As the replacement cycling attacker benefit= s >> from the HTLC-timeout being >> > usually broadcast by lightning nodes only once every block, or less th= e >> replacement cycling >> > malicious transactions paid only equal the sum of the absolute fees >> paid by the HTLC, adjusted with >> > the replacement penalty. Rebroadcasting randomly and multiple times >> before the next block increases >> > the absolute fee cost for the attacker. >> > >> > Implemented and deployed by Eclair, Core-Lightning, LND and LDK . >> > >> > Local-mempool preimage monitoring: As the replacement cycling attacker >> in a simple setup broadcast >> > the HTLC-preimage to all the network mempools, the honest lightning >> node is able to catch on the >> > flight the unconfirmed HTLC-preimage, before its subsequent mempool >> replacement. The preimage can be >> > extracted from the second-stage HTLC-preimage and used to fetch the >> off-chain inbound HTLC with a >> > cooperative message or go on-chain with it to claim the accepted HTLC >> output. >> > >> > Implemented and deployed by Eclair and LND. >> > >> > CLTV Expiry Delta: With every jammed block comes an absolute fee cost >> paid by the attacker, a risk >> > of the HTLC-preimage being detected or discovered by the honest >> lightning node, or the HTLC-timeout >> > to slip in a winning block template. Bumping the default CLTV delta >> hardens the odds of success of a >> > simple replacement cycling attack. >> > >> > Default setting: Eclair 144, Core-Lightning 34, LND 80 and LDK 72. >> > >> > ## Affected Bitcoin Protocols and Applications >> > >> > From my understanding the following list of Bitcoin protocols and >> applications could be affected by >> > new denial-of-service vectors under some level of network mempools >> congestion. Neither tests or >> > advanced review of specifications (when available) has been conducted >> for each of them: >> > - on-chain DLCs >> > - coinjoins >> > - payjoins >> > - wallets with time-sensitive paths >> > - peerswap and submarine swaps >> > - batch payouts >> > - transaction "accelerators" >> > >> > Inviting their developers, maintainers and operators to investigate ho= w >> replacement cycling attacks >> > might disrupt their in-mempool chain of transactions, or fee-bumping >> flows at the shortest delay. >> > Simple flows and non-multi-party transactions should not be affected t= o >> the best of my understanding. >> > >> > ## Open Problems: Package Malleability >> > >> > Pinning attacks have been known for years as a practical vector to >> compromise lightning channels >> > funds safety, under different scenarios (cf. current bip331's >> motivation section). Mitigations at >> > the mempool level have been designed, discussed and are under >> implementation by the community >> > (ancestor package relay + nverrsion=3D3 policy). Ideally, they should >> constraint a pinning attacker to >> > always attach a high feerate package (commitment + CPFP) to replace th= e >> honest package, or allow a >> > honest lightning node to overbid a malicious pinning package and get >> its time-sensitive transaction >> > optimistically included in the chain. >> > >> > Replacement cycling attack seem to offer a new way to neutralize the >> design goals of package relay >> > and its companion nversion=3D3 policy, where an attacker package RBF a >> honest package out of the >> > mempool to subsequently double-spend its own high-fee child with a >> transaction unrelated to the >> > channel. As the remaining commitment transaction is pre-signed with a >> minimal relay fee, it can be >> > evicted out of the mempool. >> > >> > A functional test exercising a simple replacement cycling of a >> lightning channel commitment >> > transaction on top of the nversion=3D3 code branch is available: >> > https://github.com/ariard/bitcoin/commits/2023-10-test-mempool-2 >> > >> > >> > ## Discovery >> > >> > In 2018, the issue of static fees for pre-signed lightning transaction= s >> is made more widely known, >> > the carve-out exemption in mempool rules to mitigate in-mempool packag= e >> limits pinning and the >> > anchor output pattern are proposed. >> > >> > In 2019, bitcoin core 0.19 is released with carve-out support. >> Continued discussion of the anchor >> > output pattern as a dynamic fee-bumping method. >> > >> > In 2020, draft of anchor output submitted to the bolts. Initial findin= g >> of economic pinning against >> > lightning commitment and second-stage HTLC transactions. Subsequent >> discussions of a >> > preimage-overlay network or package-relay as mitigations. Public call >> made to inquiry more on >> > potential other transaction-relay jamming attacks affecting lightning. >> > >> > In 2021, initial work in bitcoin core 22.0 of package acceptance. >> Continued discussion of the >> > pinning attacks and shortcomings of current mempool rules during >> community-wide online workshops. >> > Later the year, in light of all issues for bitcoin second-layers, a >> proposal is made about killing >> > the mempool. >> > >> > In 2022, bip proposed for package relay and new proposed v3 policy >> design proposed for a review and >> > implementation. Mempoolfullrbf is supported in bitcoin core 24.0 and >> conceptual questions about >> > alignment of mempool rules w.r.t miners incentives are investigated. >> > >> > Along this year 2022, eltoo lightning channels design are discussed, >> implemented and reviewed. In >> > this context and after discussions on mempool anti-DoS rules, I >> discovered this new replacement >> > cycling attack was affecting deployed lightning channels and >> immediately reported the finding to >> > some bitcoin core developers and lightning maintainers. >> > >> > ## Timeline >> > >> > - 2022-12-16: Report of the finding to Suhas Daftuar, Anthony Towns, >> Greg Sanders and Gloria Zhao >> > - 2022-12-16: Report to LN maintainers: Rusty Russell, Bastien >> Teinturier, Matt Corallo and Olaoluwa >> > Osuntunkun >> > - 2022-12-23: Sharing to Eugene Siegel (LND) >> > - 2022-12-24: Sharing to James O'Beirne and Antoine Poinsot >> (non-lightning potential affected projects) >> > - 2022-01-14: Sharing to Gleb Naumenko (miners incentives and >> cross-layers issuers) and initial >> > proposal of an early public disclosure >> > - 2022-01-19: Collection of analysis if other second-layers and >> multi-party applications affected. >> > LN mitigations development starts. >> > - 2023-05-04: Sharing to Wilmer Paulino (LDK) >> > - 2023-06-20: LN mitigations implemented and progressively released. >> Week of the 16 october proposed >> > for full disclosure. >> > - 2023-08-10: CVEs assigned by MITRE >> > - 2023-10-05: Pre-disclosure of LN CVEs numbers and replacement cyclin= g >> attack existence to >> > security@bitcoincore.org . >> > - 2023-10-16: Full disclosure of CVE-2023-40231 / CVE-2023-40232 / >> CVE-2023-40233 / CVE-2023-40234 >> > and replacement cycling attacks >> > >> > ## Conclusion >> > >> > Despite the line of mitigations adopted and deployed by current major >> lightning implementations, I >> > believe replacement cycling attacks are still practical for advanced >> attackers. Beyond this new >> > attack might come as a way to partially or completely defeat some of >> the pinning mitigations which >> > have been working for years as a community. >> > >> > As of today, it is uncertain to me if lightning is not affected by a >> more severe long-term package >> > malleability critical security issue under current consensus rules, an= d >> if any other time-sensitive >> > multi-party protocol, designed or deployed isn't de facto affected too >> (loss of funds or denial of >> > service). >> > >> > Assuming analysis on package malleability is correct, it is unclear to >> me if it can be corrected by >> > changes in replacement / eviction rules or mempool chain of >> transactions processing strategy. >> > Inviting my technical peers and the bitcoin community to look more on >> this issue, including to >> > dissent. I'll be the first one pleased if I'm fundamentally wrong on >> those issues, or if any element >> > has not been weighted with the adequate technical accuracy it deserves= . >> > >> > Do not trust, verify. All mistakes and opinions are my own. >> > >> > Antoine >> > >> > "meet with Triumph and Disaster. And treat those two impostors just th= e >> same" - K. >> > >> > _______________________________________________ >> > Lightning-dev mailing list >> > Lightning-dev@lists.linuxfoundation.org >> > https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev >> > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > --00000000000090385906080d509c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thanks Antoine for your work on this issue.

I confi= rm that eclair v0.9.0 contains the migitations described.

Eclair has= been watching the mempool for preimages since its very early
versions (= years ago), relying on Bitcoin Core's ZMQ notifications for
incoming= transactions. I believe this guarantees that we see the HTLC
success tr= ansaction, even if it is immediately replaced (as long as we
don't o= verflow the ZMQ limits).

I agree with Matt though that more fundamen= tal work most likely needs to
happen at the bitcoin layer to allow L2 pr= otocols to be more robust
against that class of attacks.

Thanks,<= br>Bastien

Le=C2=A0mer. 18 oct. 2023 =C3=A0=C2=A011:07, Antoine Riard via bi= tcoin-dev <bitc= oin-dev@lists.linuxfoundation.org> a =C3=A9crit=C2=A0:
The disclosu= re mails noted a 3rd mitigation beyond=C2=A0mempool scanning and transactio= n re-signing / re-broadcasting, namely bumping CLTV delta.

Generally bumping CLTV delta is a basic line of mitigations for a lot of= lightning attacks, as it gives opportunity to node operators to intervene = and re-broadcast their time-sensitive transactions on other interfaces (e.g= a secondary full-node if the first one is eclipsed).

<= div>About the second mitigation transaction re-signing, if done correctly a= t least sounds to put an economic cost (denominated in fees / feerates) on = the attack. This is unclear to me if the game-theory of this cost holds.

One thing which sounds to me making the attack harde= r is stratum v2 deployment, as you're increasing the number of miners w= hich might do their own block templates, and therefore the number of miners= ' mempools where=C2=A0an attacker has to successfully continuously repl= ace in cycles channels counterparties transactions.

A replacement buffer or history of transactions at the mempool level migh= t be a mitigation to this attack. I believe this is yet to be seen if it ca= n be made robust enough.

I don't know if folks= like tadge or rusty who have been involved in the early design of lightnin= g have more ideas of mitigations. Fees was noted as a hard issue in the ori= ginal paper.

Le=C2=A0mer. 18 oct. 2023 =C3=A0=C2=A001:17, Matt Corallo= <lf-lists= @mattcorallo.com> a =C3=A9crit=C2=A0:
There appears to be some confusion about this = issue and the mitigations. To be clear, the deployed
mitigations are not expected to fix this issue, its arguable if they provid= e anything more than a PR
statement.

There are two discussed mitigations here - mempool scanning and transaction= re-signing/re-broadcasting.

Mempool scanning relies on regularly checking the mempool of a local node t= o see if we can catch the
replacement cycle mid-cycle. It only works if wee see the first transaction= before the second
transaction replaces it.

Today, a large majority of lightning nodes run on machines with a Bitcoin n= ode on the same IP
address, making it very clear what the "local node" of the lightn= ing node is. An attacker can
trivially use this information to connect to said local node and do the rep= lacement quickly,
preventing the victim from seeing the replacement.

More generally, however, similar discoverability is true for mining pools. = An attacker performing
this attack is likely to do the replacement attack on a miner's node di= rectly, potentially reducing
the reach of the intermediate transaction to only miners, such that the vic= tim can never discover it
at all.

The second mitigation is similarly pathetic. Re-signing and re-broadcasting= the victim's transaction
in an attempt to get it to miners even if its been removed may work, if the= attacker is super lazy
and didn't finish writing their attack system. If the attacker is conne= cted to a large majority of
hashrate (which has historically been fairly doable), they can simply do th= eir replacement in a
cycle aggressively and arbitrarily reduce the probability that the victim&#= 39;s transaction gets confirmed.

Now, the above is all true in a spherical cow kinda world, and the P2P netw= ork has plenty of slow
nodes and strange behavior. Its possible that these mitigations might, by s= ome stroke of luck,
happen to catch such an attack and prevent it, because something took longe= r than the attacker
intended or whatever. But, that's a far cry from any kind of material &= quot;fix" for the issue.

Ultimately the only fix for this issue will be when miners keep a history o= f transactions they've
seen and try them again after they may be able to enter the mempool because= of an attack like this.

Matt

On 10/16/23 12:57 PM, Antoine Riard wrote:
> (cross-posting mempool issues identified are exposing lightning chan t= o loss of funds risks, other
> multi-party bitcoin apps might be affected)
>
> Hi,
>
> End of last year (December 2022), amid technical discussions on eltoo = payment channels and
> incentives compatibility of the mempool anti-DoS rules, a new transact= ion-relay jamming attack
> affecting lightning channels was discovered.
>
> After careful analysis, it turns out this attack is practical and imme= diately=C2=A0exposed lightning
> routing hops carrying HTLC traffic to loss of funds security risks, bo= th legacy and anchor=C2=A0output
> channels. A potential exploitation plausibly happening even without ne= twork mempools congestion.
>
> Mitigations have been designed, implemented and deployed by all major = lightning implementations
> during the last months.
>
> Please find attached the release numbers, where the mitigations should= be present:
> - LDK: v0.0.118 - CVE-2023 -40231
> - Eclair: v0.9.0 - CVE-2023-40232
> - LND: v.0.17.0-beta - CVE-2023-40233
> - Core-Lightning: v.23.08.01 - CVE-2023-40234
>
> While neither replacement cycling attacks have been observed or report= ed in the wild since the last
> ~10 months or experimented in real-world conditions on bitcoin mainet,= functional test is available
> exercising the affected lightning channel against bitcoin core mempool= (26.0 release cycle).
>
> It is understood that a simple replacement cycling attack does not dem= and privileged capabilities
> from an attacker (e.g no low-hashrate power) and only access to basic = bitcoin and lightning
> software. Yet I still think executing such an attack successfully requ= ests a fair amount of bitcoin
> technical know-how and decent preparation.
>
>=C2=A0 From my understanding of those issues, it is yet to be determine= d if the mitigations deployed are
> robust enough in face of advanced replacement cycling attackers, espec= ially ones able to combine
> different classes of transaction-relay jamming such as pinnings or vet= ted with more privileged
> capabilities.
>
> Please find a list of potential affected bitcoin applications in this = full disclosure report using
> bitcoin script timelocks or multi-party transactions, albeit no immedi= ate security risk exposure as
> severe as the ones affecting lightning has been identified. Only curso= ry review of non-lightning
> applications has been conducted so far.
>
> There is a paper published summarizing replacement cycling attacks on = the lightning network:
> https://github.com/ariard/mempool-research/blob/2023-10-replacement-paper= /replacement-cycling.pdf
> <https://github.com/ariard/mempool-research/blob/2023-10-replacement-p= aper/replacement-cycling.pdf>
>
>=C2=A0 =C2=A0## Problem
>
> A lightning node allows HTLCs forwarding (in bolt3's parlance acce= pted HTLC on incoming link and
> offered HTLC on outgoing link) should settle the outgoing state with e= ither a success or timeout
> before the incoming state timelock becomes final and an asymmetric def= avorable settlement might
> happen (cf "Flood & Loot: A Systematic Attack on The Lightnin= g Network" section 2.3 for a classical
> exposition of this lightning security property).
>
> Failure to satisfy this settlement requirement exposes a forwarding ho= p to a loss of fund risk where
> the offered HTLC is spent by the outgoing link counterparty's HTLC= -preimage and the accepted HTLC is
> spent by the incoming link counterparty's HTLC-timeout.
>
> The specification mandates the incoming HTLC expiration timelock to be= spaced out by an interval of
> `cltv_expiry_delta` from the outgoing HTLC expiration timelock, this e= xact interval value being an
> implementation and node policy setting. As a minimal value, the specif= ication recommends 34 blocks
> of interval. If the timelock expiration I of the inbound HTLC is equal= to 100 from chain tip, the
> timelock expiration O of the outbound HTLC must be equal to 66 blocks = from chain tip, giving a
> reasonable buffer of reaction to the lightning forwarding node.
>
> In the lack of cooperative off-chain settlement of the HTLC on the out= going link negotiated with the
> counterparty (either `update_fulfill_htlc` or `update_fail_htlc`) when= O is reached, the lightning
> node should broadcast its commitment transaction. Once the commitment = is confirmed (if anchor and
> the 1 CSV encumbrance is present), the lightning node broadcasts and c= onfirms its HTLC-timeout
> before I height is reached.
>
> Here enter a replacement cycling attack. A malicious channel counterpa= rty can broadcast its
> HTLC-preimage transaction with a higher absolute fee and higher feerat= e than the honest HTLC-timeout
> of the victim lightning node and triggers a replacement. Both for lega= cy and anchor output channels,
> a HTLC-preimage on a counterparty commitment transaction is malleable,= i.e additional inputs or
> outputs can be added. The HTLC-preimage spends an unconfirmed and unre= lated to the channel parent
> transaction M and conflicts its child.
>
> As the HTLC-preimage spends an unconfirmed input that was already incl= uded in the unconfirmed and
> unrelated child transaction (rule 2), pays an absolute higher fee of a= t least the sum paid by the
> HTLC-timeout and child transaction (rule 3) and the HTLC-preimage feer= ate is greater than all
> directly conflicting transactions (rule 6), the replacement is accepte= d. The honest HTLC-timeout is
> evicted out of the mempool.
>
> In an ulterior move, the malicious counterparty can replace the parent= transaction itself with
> another candidate N satisfying the replacement rules, triggering the e= viction of the malicious
> HTLC-preimage from the mempool as it was a child of the parent T.
>
> There is no spending candidate of the offered HTLC output for the curr= ent block laying in network
> mempools.
>
> This replacement cycling tricks can be repeated for each rebroadcast a= ttempt of the HTLC-timeout by
> the honest lightning node until expiration of the inbound HTLC timeloc= k I. Once this height is
> reached a HTLC-timeout is broadcast by the counterparty's on the i= ncoming link in collusion with the
> one on the outgoing link broadcasting its own HTLC-preimage.
>
> The honest Lightning node has been "double-spent" in its HTL= C forwarding.
>
> As a notable factor impacting the success of the attack, a lightning n= ode's honest HTLC-timeout
> might be included in the block template of the miner winning the block= race and therefore realizes a
> spent of the offered output. In practice, a replacement cycling attack= might over-connect to miners'
> mempools and public reachable nodes to succeed in a fast eviction of t= he HTLC-timeout by its
> HTLC-preimage. As this latter transaction can come with a better ances= tor-score, it should be picked
> up on the flight by economically competitive miners.
>
> A functional test exercising a simple replacement cycling of a HTLC tr= ansaction on bitcoin core
> mempool is available:
> https://github.com/ariard/bitcoin/co= mmits/2023-test-mempool
> <https://github.com/ariard/bitcoi= n/commits/2023-test-mempool>
>
> ## Deployed LN mitigations
>
> Aggressive rebroadcasting: As the replacement cycling attacker benefit= s from the HTLC-timeout being
> usually broadcast by lightning nodes only once every block, or less th= e replacement cycling
> malicious transactions paid only equal the sum of the absolute fees pa= id by the HTLC, adjusted with
> the replacement penalty. Rebroadcasting randomly and multiple times be= fore the next block increases
> the absolute fee cost for the attacker.
>
> Implemented and deployed by Eclair, Core-Lightning, LND and LDK .
>
> Local-mempool preimage monitoring: As the replacement cycling attacker= in a simple setup broadcast
> the HTLC-preimage to all the network mempools, the honest lightning no= de is able to catch on the
> flight the unconfirmed HTLC-preimage, before its subsequent mempool re= placement. The preimage can be
> extracted from the second-stage HTLC-preimage and used to fetch the of= f-chain inbound HTLC with a
> cooperative message or go on-chain with it to claim the accepted HTLC = output.
>
> Implemented and deployed by Eclair and LND.
>
> CLTV Expiry Delta: With every jammed block comes an absolute fee cost = paid by the attacker, a risk
> of the HTLC-preimage being detected or discovered by the honest lightn= ing node, or the HTLC-timeout
> to slip in a winning block template. Bumping the default CLTV delta ha= rdens the odds of success of a
> simple replacement cycling attack.
>
> Default setting: Eclair 144, Core-Lightning 34, LND 80 and LDK 72.
>
> ## Affected Bitcoin Protocols and Applications
>
>=C2=A0 From my understanding the following list of Bitcoin protocols an= d applications could be affected by
> new denial-of-service vectors under some level of network mempools con= gestion. Neither tests or
> advanced review of specifications (when available) has been conducted = for each of them:
> - on-chain DLCs
> - coinjoins
> - payjoins
> - wallets with time-sensitive paths
> - peerswap and submarine swaps
> - batch payouts
> - transaction "accelerators"
>
> Inviting their developers, maintainers and operators to investigate ho= w replacement cycling attacks
> might disrupt their in-mempool chain of transactions, or fee-bumping f= lows at the shortest delay.
> Simple flows and non-multi-party transactions should not be affected t= o the best of my understanding.
>
> ## Open Problems: Package Malleability
>
> Pinning attacks have been known for years as a practical vector to com= promise lightning channels
> funds safety, under different scenarios (cf. current bip331's moti= vation section). Mitigations at
> the mempool level have been designed, discussed and are under implemen= tation by the community
> (ancestor package relay=C2=A0+ nverrsion=3D3 policy). Ideally, they sh= ould constraint a pinning attacker to
> always attach a high feerate package (commitment=C2=A0+ CPFP) to repla= ce the honest package, or allow a
> honest lightning node to overbid a malicious pinning package and get i= ts time-sensitive transaction
> optimistically included in the chain.
>
> Replacement cycling attack seem to offer a new way to neutralize the d= esign goals of package relay
> and its companion nversion=3D3 policy, where an attacker package RBF a= honest package out of the
> mempool to subsequently double-spend its own high-fee child with a tra= nsaction unrelated to the
> channel. As the remaining commitment transaction is pre-signed with a = minimal relay fee, it can be
> evicted out of the mempool.
>
> A functional test exercising a simple replacement cycling of a lightni= ng channel commitment
> transaction on top of the nversion=3D3 code branch is available:
> https://github.com/ariard/bitco= in/commits/2023-10-test-mempool-2
> <https://github.com/ariard/b= itcoin/commits/2023-10-test-mempool-2>
>
> ## Discovery
>
> In 2018, the issue of static fees for pre-signed lightning transaction= s is made more widely known,
> the carve-out exemption in mempool rules to mitigate in-mempool packag= e limits pinning and the
> anchor output pattern are proposed.
>
> In 2019, bitcoin core 0.19 is released with carve-out support. Continu= ed discussion of the anchor
> output pattern as a dynamic fee-bumping method.
>
> In 2020, draft of anchor output submitted to the bolts. Initial findin= g of economic pinning against
> lightning commitment and second-stage HTLC transactions. Subsequent di= scussions of a
> preimage-overlay network or package-relay as mitigations. Public call = made to inquiry more on
> potential other transaction-relay jamming attacks affecting lightning.=
>
> In 2021, initial work in bitcoin core 22.0 of package acceptance. Cont= inued discussion of the
> pinning attacks and shortcomings of current mempool rules during commu= nity-wide online workshops.
> Later the year, in light of all issues for bitcoin second-layers, a pr= oposal is made about killing
> the mempool.
>
> In 2022, bip proposed for package relay and new proposed v3 policy des= ign proposed for a review and
> implementation. Mempoolfullrbf is supported in bitcoin core 24.0 and c= onceptual questions about
> alignment of mempool rules w.r.t miners incentives are investigated. >
> Along this year 2022, eltoo lightning channels design are discussed, i= mplemented and reviewed. In
> this context and after discussions on mempool anti-DoS rules, I discov= ered this new replacement
> cycling attack was affecting deployed lightning channels and immediate= ly reported the finding to
> some bitcoin core developers and lightning maintainers.
>
> ## Timeline
>
> - 2022-12-16: Report of the finding to Suhas Daftuar, Anthony Towns, G= reg Sanders and Gloria Zhao
> - 2022-12-16: Report to LN maintainers: Rusty Russell, Bastien Teintur= ier, Matt Corallo and Olaoluwa
> Osuntunkun
> - 2022-12-23: Sharing to Eugene Siegel (LND)
> - 2022-12-24: Sharing to James O'Beirne and Antoine Poinsot (non-l= ightning potential affected projects)
> - 2022-01-14: Sharing to Gleb Naumenko (miners incentives and cross-la= yers issuers) and initial
> proposal of an early public disclosure
> - 2022-01-19: Collection of analysis if other second-layers and multi-= party applications affected.
> LN mitigations development starts.
> - 2023-05-04: Sharing to Wilmer Paulino (LDK)
> - 2023-06-20: LN mitigations implemented and progressively released. W= eek of the 16 october proposed
> for full disclosure.
> - 2023-08-10: CVEs assigned by MITRE
> - 2023-10-05: Pre-disclosure of LN CVEs numbers and replacement cyclin= g attack existence to
> security= @bitcoincore.org <mailto:security@bitcoincore.org>.
> - 2023-10-16: Full disclosure of CVE-2023-40231 / CVE-2023-40232 / CVE= -2023-40233 / CVE-2023-40234
> and replacement cycling attacks
>
> ## Conclusion
>
> Despite the line of mitigations adopted and deployed by current major = lightning implementations, I
> believe replacement cycling attacks are still practical for advanced a= ttackers. Beyond this new
> attack might come as a way to partially or completely defeat some of t= he pinning mitigations which
> have been working for years as a community.
>
> As of today, it is uncertain to me if lightning is not affected by a m= ore severe long-term package
> malleability critical security issue under current consensus rules, an= d if any other time-sensitive
> multi-party protocol, designed or deployed isn't de facto affected= too (loss of funds or denial of
> service).
>
> Assuming analysis on package malleability is correct, it is unclear to= me if it can be corrected by
> changes in replacement / eviction rules or mempool chain of transactio= ns processing strategy.
> Inviting my technical peers and the bitcoin community to look more on = this issue, including to
> dissent. I'll be the first one pleased if I'm fundamentally wr= ong on those issues, or if any element
> has not been weighted with the adequate technical accuracy it deserves= .
>
> Do not trust, verify. All mistakes and opinions are my own.
>
> Antoine
>
> "meet with Triumph and Disaster. And treat those two impostors ju= st the same" - K.
>
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.o= rg/mailman/listinfo/lightning-dev
_______________________________________________
bitcoin-dev mailing list
= bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mail= man/listinfo/bitcoin-dev
--00000000000090385906080d509c--