path: root/9b/be1fcc2b9e26c00c4a03f4de29839c9c7d168d

Return-Path: <antoine.riard@gmail.com>
Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133])
 by lists.linuxfoundation.org (Postfix) with ESMTP id EF903C0032
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Fri,  3 Nov 2023 19:57:49 +0000 (UTC)
Received: from localhost (localhost [127.0.0.1])
 by smtp2.osuosl.org (Postfix) with ESMTP id D6DDE40412
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Fri,  3 Nov 2023 19:57:49 +0000 (UTC)
DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org D6DDE40412
Authentication-Results: smtp2.osuosl.org;
 dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com
 header.a=rsa-sha256 header.s=20230601 header.b=eJylbEGN
X-Virus-Scanned: amavisd-new at osuosl.org
X-Spam-Flag: NO
X-Spam-Score: -2.098
X-Spam-Level: 
X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5
 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
 DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from smtp2.osuosl.org ([127.0.0.1])
 by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id jfP3fP7jWd3w
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Fri,  3 Nov 2023 19:57:48 +0000 (UTC)
Received: from mail-il1-x135.google.com (mail-il1-x135.google.com
 [IPv6:2607:f8b0:4864:20::135])
 by smtp2.osuosl.org (Postfix) with ESMTPS id C6CF74032C
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Fri,  3 Nov 2023 19:57:47 +0000 (UTC)
DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org C6CF74032C
Received: by mail-il1-x135.google.com with SMTP id
 e9e14a558f8ab-35931064a3bso9302115ab.3
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Fri, 03 Nov 2023 12:57:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20230601; t=1699041467; x=1699646267;
 darn=lists.linuxfoundation.org; 
 h=to:subject:message-id:date:from:in-reply-to:references:mime-version
 :from:to:cc:subject:date:message-id:reply-to;
 bh=dZUss3soWYIFQ7Fxxi/e0T3UDsl24P8AAjNaG5847Ek=;
 b=eJylbEGN2nON9IBBGkHkWoPmhsD17LOrQqhG48iclFwsXncDCbeesSXpRb/v1QuBBn
 C/7aU8jeEq1faMDgFyub9WiXIlZ6OFFi+1QAK6vg3DRLUU9vPJopZvici1oEAGQzDS0Y
 qUEljfZsV18XukgXESNCb/+Os3HqO0X4sUDZGIzFXnqPnCM59gMsjPnydS2cmAKVlZhX
 jHiPiRsXMWT0gOuZoKKMol63/gBZ9HY2hbqFk/Yr5WhiMr5hdgDy65+302ZlXqF8Jyoi
 ppxj4ZHno9Il3lpaAyVZcDVa+T2Hkua0bAkO+Nz43a73+J83SipcCyb3GGwsqaCZAaRv
 ytbA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20230601; t=1699041467; x=1699646267;
 h=to:subject:message-id:date:from:in-reply-to:references:mime-version
 :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;
 bh=dZUss3soWYIFQ7Fxxi/e0T3UDsl24P8AAjNaG5847Ek=;
 b=u9CaCuYvYYtf4y5ASPHkktMRfyaooDRMaMK8Voq3m/cSq/7RiZeoEQjD0kikf5z+9B
 pWOAJ7LSYKEg//PG+5zSVzXdkYAIjxGcG3D5c39uFgSMClqlv2pYpeOWk4tRZtg/ss17
 AKinXPkeMSGwEcs8QyFoCGYVfW4k3CmMhpWK9O2iPvj3F4G7xWAi9XQnQEBPT5kC9DUK
 hbiT5ITo2URPGFJdnCO4VoL4BL216Zj4VYk2VNd8IS9ZwIn5BKX5stv2HmdRtsdkUEkg
 j/0n5wVkyq7oRbPiJSRjno4obfld4BrjNT+Sgiu+rDipv9HyJZFJhWxLi/lBoF5MqRX6
 /giw==
X-Gm-Message-State: AOJu0YxHU2UD6PjU8eoQ1q8Sw1wLYP1wMR3sXV6Igb61oPqXeGkgwhzg
 p0486E14G8Y/fpTRJtXScHHDzed7Tgdoc2YSxVX3dnU++pzrmKym
X-Google-Smtp-Source: AGHT+IG35fEvjQxYwiZ7BRxPHykEW7zivLfG23gtcHkkBbg51sigmfVBFDuV+Kk5d2DYGwJXMBcNy1NGQFBS81f0oPs=
X-Received: by 2002:a05:6e02:221b:b0:359:62e:e25b with SMTP id
 j27-20020a056e02221b00b00359062ee25bmr29861243ilf.8.1699041466553; Fri, 03
 Nov 2023 12:57:46 -0700 (PDT)
MIME-Version: 1.0
References: <mailman.10748.1698911611.1202.lightning-dev@lists.linuxfoundation.org>
 <CAHTn92z9RhrHd=quYwfbj9y9gvA4aGX=JGNv9UggR4cWSZE9Xw@mail.gmail.com>
In-Reply-To: <CAHTn92z9RhrHd=quYwfbj9y9gvA4aGX=JGNv9UggR4cWSZE9Xw@mail.gmail.com>
From: Antoine Riard <antoine.riard@gmail.com>
Date: Fri, 3 Nov 2023 19:57:35 +0000
Message-ID: <CALZpt+HQ0KMufd7Hmkpq+opOT7H9rZJccqzy=v6s3bao5zQAzQ@mail.gmail.com>
To: John Carvalho <john@synonym.to>, 
 Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
Content-Type: multipart/alternative; boundary="000000000000129a31060944eb15"
X-Mailman-Approved-At: Fri, 03 Nov 2023 20:59:42 +0000
Subject: Re: [bitcoin-dev] The Pinning & Replacement Problem Set
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>, 
 <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>, 
 <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Fri, 03 Nov 2023 19:57:50 -0000

--000000000000129a31060944eb15
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi John,

I think lightning and other second time-sensitive layers being hit by
safety issues whenever the blocks are full is common knowledge as the issue
is being described in the paper under the "forced expiration spam" issues
arising spontaneously within an environment with high block space demand. I
believe you have known variants of those issues with the "flood & loot"
style scenarios.

What is coming as a new problem with the novel replacement issues lay in
the fact that your counterparty channel might always defer the confirmation
of your honest on-chain spend, in a way which is compatible with miners
incentives.

While this has been commonized by the wide deployment of bip-125-style RBF
over network mempools, this ability has been always available to any party
reaching out directly to miners out-of-band with consensus-valid
transactions. In a world where miners are pseudonymous (as we're mostly
seeing today, not considering pools) miners motivated by maximizing their
satoshi-denominated income is a reasonable assumption in my opinion.

As an ecosystem beyond core to fix sustainably pinning and replacement
problems, and I agree with you here, we are stuck with a "between Scylla
and Charybdis" serious safety issue, I'm seeing the following options.
Matching yours, rephrasing in my own words to ensure correct understanding.

1. We can revert to a static world with no replacement-by-fee mechanism as
a widely deployed network policy.

I think in a world where miners are in competition you can always reach out
to them with out-of-band higher fees packages than available in their local
mempool.

2. We can design, implement and deploy policies to better capture
transaction-issuer intent on the way future spend candidates are allowed
(or not) to replace the current in-mempool transaction.

Sadly, I think with lightning and other second time-sensitive layers, there
is no "single" transaction-issuer intent, though you might have as many
intents that you have counterparties within your off-chain states, all with
competing interests.

3. We can remove all policy and let the network of nodes and the economic
ecosystem evolve on its own.

I think a lot of mempool policy in place is actually anti-DoS measures
which are protecting at least the lowest-performance full-nodes, and those
measures are the source of issues for pinning. "Pure" RBF sounds to me only
to make adversarial replacement issues worse (at least nversion=3D3-style
policy introduces some assumptions on max package size if widely deployed).

4. We can do nothing and let a fragmented mempool environment, where every
large lightning and bitcoin business have out-of-band relationships with
miners and pools to support their packages, with some service-level safety
agreements.

I think this option was weighted as a way to solve pinning issues at the
commitment and second-state level years ago by the lightning community,
though it was brushed aside as bringing rampant centralization of the
lightning network, where small lightning nodes might not have privileged
access to miners [0].

5. We can design and implement some magical consensus-change or alter the
processing requirements (bandwidth, CPU perf, I/O disk) of full-nodes on
the peer-to-peer network to align incentives between miners and lightning
and time-sensitive second-layers.

I think this option represents more or less what are the trade-offs of what
is discussed by the "reverse locktime" new bitcoin script opcode idea or
replacement cache at the mempool-level in core [1].

Best,
Antoine

[0] https://github.com/lightning/bolts/issues/783
[1] https://github.com/bitcoin/bitcoin/issues/28699

Le jeu. 2 nov. 2023 =C3=A0 11:38, John Carvalho via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> a =C3=A9crit :

> Good morning,
>
> All layers and technologies "on" Bitcoin will fail in situations where
> miners misbehave or the blocks & mempool remain consistently, overly full=
.
> Consider this as a "law" of Bitcoin/blockchains.
>
> In hindsight (for you, not me) it was very unwise to start messing with
> mempool policies, like with RBF, mempoolfullrbf. First-seen policy brough=
t
> a fragile harmony and utility to Bitcoin, which we were lucky to have for
> as long as we could.
>
> The engineers intentionally broke this. Mempoofullrbf washes away the
> ability for users to express their intent on whether to pin or replace a
> transaction, and now Lightning has BOTH pinning and replacement problems.
> You could argue this was inevitable. The point here is that layers have
> mempool and miner problems.
>
> Core has a few choices here, as I see it:
>
> 1. They can try to revert mempoolfullrbf and re-establish first-seen
> policy. Hard to say whether this would work, or whether it would be
> enough...
>
> 2. They can create additional policies that are enforced by default that
> allow people to flag how they want their txn handled, as in, a "pin this"
> vs "replace this" aspect to every txn. This is probably the best option, =
as
> it allows miners to know what people want and give it to them. This is
> utility, thus incentive-compatible.
>
> 3. Remove all policy and let the market evolve to deal with the chaos.
> Which is similar to the next option: do nothing.
>
> 4. Do nothing and allow a fractured mempool environment to evolve where
> large businesses form private contracts with miners and pools to support
> their own unsupported policies, so they can provide better experiences to
> customers and users.
>
> 5. Invent some miracle magical crypto cure that I am not capable of
> imagining at this time.
>
> I disclaim some ignorance to details of how other mempool research might
> affect these options and problems, but I am skeptical the dynamics
> discussed here can be removed entirely.
>
> --John Carvalho
> CEO, Synonym.to <http://synonym.to/>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

--000000000000129a31060944eb15
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi John,<div><br></div><div>I think lightning and other se=
cond time-sensitive layers being hit by safety issues whenever the blocks a=
re full is common knowledge as the issue is being described in the paper un=
der the &quot;forced expiration spam&quot; issues arising spontaneously wit=
hin an environment with high block space demand. I believe you have known v=
ariants of those issues with the &quot;flood &amp; loot&quot; style scenari=
os.</div><div><br></div><div>What is coming as a new problem with the novel=
 replacement issues lay in the fact that your counterparty channel might al=
ways defer the confirmation of your honest on-chain spend, in a way which i=
s compatible with=C2=A0miners incentives.</div><div><br></div><div>While th=
is has been commonized by the wide deployment of bip-125-style RBF over net=
work mempools, this ability=C2=A0has been always available to any party rea=
ching out directly=C2=A0to miners out-of-band with consensus-valid transact=
ions. In a world where miners=C2=A0are pseudonymous (as we&#39;re mostly se=
eing today, not considering pools) miners motivated by maximizing their sat=
oshi-denominated income is a reasonable assumption in my opinion.</div><div=
><br></div><div>As an ecosystem beyond core to fix sustainably pinning and =
replacement problems, and I agree with you here, we are stuck with a &quot;=
between Scylla and Charybdis&quot; serious safety issue, I&#39;m seeing the=
 following options. Matching yours, rephrasing in my own words to ensure co=
rrect understanding.</div><div><br></div><div>1. We can revert to a static =
world with no replacement-by-fee mechanism as a widely deployed network pol=
icy.</div><div><br></div><div>I think in a world where miners are in compet=
ition you can always reach out to them with out-of-band higher fees package=
s than available in their local mempool.</div><div><br></div><div>2. We can=
 design, implement and deploy policies to better capture transaction-issuer=
 intent on the way future spend candidates are allowed (or not) to replace =
the current in-mempool transaction.</div><div><br></div><div>Sadly, I think=
 with lightning and other second time-sensitive layers, there is no &quot;s=
ingle&quot; transaction-issuer intent, though you might have as many intent=
s that you have counterparties within your off-chain states, all with compe=
ting=C2=A0interests.</div><div><br></div><div>3. We can remove all policy a=
nd let the network of nodes and the economic ecosystem evolve on its own.</=
div><div><br></div><div>I think a lot of mempool policy in place is actuall=
y anti-DoS measures which are protecting at least the lowest-performance fu=
ll-nodes, and those measures are the source of issues for pinning. &quot;Pu=
re&quot; RBF sounds to me only to make adversarial replacement issues worse=
 (at least nversion=3D3-style policy introduces some assumptions on max pac=
kage size if widely deployed).</div><div><br></div><div>4. We can do nothin=
g and let a fragmented mempool environment, where every large lightning and=
 bitcoin business have out-of-band relationships with miners and pools to s=
upport their packages, with some service-level safety agreements.</div><div=
><br></div><div>I think this option was weighted as a way to solve pinning =
issues at the commitment and second-state level years ago by the lightning =
community, though it was brushed aside as bringing rampant centralization o=
f the lightning network, where small lightning nodes might not have privile=
ged access to miners [0].</div><div><br></div><div>5. We can design and imp=
lement some magical consensus-change or alter the processing requirements (=
bandwidth, CPU perf, I/O disk) of full-nodes on the peer-to-peer network to=
 align incentives between miners and lightning and time-sensitive second-la=
yers.</div><div><br></div><div>I think this option represents more or less =
what are the trade-offs of what is discussed by the &quot;reverse locktime&=
quot; new bitcoin script opcode idea or replacement cache at the mempool-le=
vel in core [1].</div><div><br></div><div>Best,</div><div>Antoine</div><div=
><br></div><div>[0]=C2=A0<a href=3D"https://github.com/lightning/bolts/issu=
es/783">https://github.com/lightning/bolts/issues/783</a></div><div>[1]=C2=
=A0<a href=3D"https://github.com/bitcoin/bitcoin/issues/28699">https://gith=
ub.com/bitcoin/bitcoin/issues/28699</a></div></div><br><div class=3D"gmail_=
quote"><div dir=3D"ltr" class=3D"gmail_attr">Le=C2=A0jeu. 2 nov. 2023 =C3=
=A0=C2=A011:38, John Carvalho via bitcoin-dev &lt;<a href=3D"mailto:bitcoin=
-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a>&g=
t; a =C3=A9crit=C2=A0:<br></div><blockquote class=3D"gmail_quote" style=3D"=
margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;bord=
er-left-color:rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div dir=
=3D"ltr"><div dir=3D"ltr" class=3D"gmail_signature"><div dir=3D"ltr"><div d=
ir=3D"ltr"><div dir=3D"ltr">Good morning,=C2=A0</div><div dir=3D"ltr"><br><=
/div><div dir=3D"ltr">All layers and technologies &quot;on&quot; Bitcoin wi=
ll fail in situations where miners misbehave or the blocks &amp; mempool re=
main consistently, overly full. Consider this as a &quot;law&quot; of Bitco=
in/blockchains.=C2=A0</div><div dir=3D"ltr"><br></div><div dir=3D"ltr">In h=
indsight (for you, not me) it was very unwise to start messing with mempool=
 policies, like with RBF, mempoolfullrbf. First-seen policy brought a fragi=
le harmony and utility to Bitcoin, which we were lucky to have for as long =
as we could.=C2=A0</div><div dir=3D"ltr"><br></div><div dir=3D"ltr">The eng=
ineers intentionally broke this. Mempoofullrbf washes away the ability for =
users to express their intent on whether to pin or replace a transaction, a=
nd now Lightning has BOTH pinning and replacement problems. You could argue=
 this was inevitable. The point here is that layers have mempool and miner =
problems.</div><div dir=3D"ltr"><br></div><div dir=3D"ltr">Core has a few c=
hoices here, as I see it:=C2=A0</div><div dir=3D"ltr"><br></div><div dir=3D=
"ltr">1. They can try to revert mempoolfullrbf and re-establish first-seen =
policy. Hard to say whether this would work, or whether it would be enough.=
..=C2=A0</div><div dir=3D"ltr"><br></div><div dir=3D"ltr">2. They can creat=
e additional policies that are enforced by default that allow people to fla=
g how they want their txn handled, as in, a &quot;pin this&quot; vs &quot;r=
eplace this&quot; aspect to every txn. This is probably the best option, as=
 it allows miners to know what people want and give it to them. This is uti=
lity, thus incentive-compatible.=C2=A0</div><div dir=3D"ltr"><br></div><div=
 dir=3D"ltr">3. Remove all policy and let the market evolve to deal with th=
e chaos. Which is similar to the next option: do nothing.=C2=A0</div><div d=
ir=3D"ltr"><br></div><div dir=3D"ltr">4. Do nothing and allow a fractured m=
empool environment to evolve where large businesses form private contracts =
with miners and pools to support their own unsupported policies, so they ca=
n provide better experiences to customers and users.=C2=A0</div><div dir=3D=
"ltr"><br></div><div dir=3D"ltr">5. Invent some miracle magical crypto cure=
 that I am not capable of imagining at this time.</div><div dir=3D"ltr"><br=
></div><div>I disclaim some ignorance to details of how other mempool resea=
rch might affect these options and problems, but I am skeptical the dynamic=
s discussed here can be removed entirely.</div><div dir=3D"ltr"><br>--John =
Carvalho</div><div dir=3D"ltr" style=3D"color:rgb(34,34,34)">CEO,=C2=A0<a h=
ref=3D"http://synonym.to/" style=3D"color:rgb(17,85,204)" target=3D"_blank"=
>Synonym.to</a></div></div></div></div></div></div>
_______________________________________________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">=
bitcoin-dev@lists.linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" =
rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail=
man/listinfo/bitcoin-dev</a><br>
</blockquote></div>

--000000000000129a31060944eb15--