MIME-Version: 1.0
References: <mailman.10748.1698911611.1202.lightning-dev@lists.linuxfoundation.org>
 <CAHTn92z9RhrHd=quYwfbj9y9gvA4aGX=JGNv9UggR4cWSZE9Xw@mail.gmail.com>
In-Reply-To: <CAHTn92z9RhrHd=quYwfbj9y9gvA4aGX=JGNv9UggR4cWSZE9Xw@mail.gmail.com>
From: Antoine Riard <antoine.riard@gmail.com>
Date: Fri, 3 Nov 2023 19:57:35 +0000
Message-ID: <CALZpt+HQ0KMufd7Hmkpq+opOT7H9rZJccqzy=v6s3bao5zQAzQ@mail.gmail.com>
To: John Carvalho <john@synonym.to>, 
 Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
Content-Type: multipart/alternative; boundary="000000000000129a31060944eb15"
Subject: Re: [bitcoin-dev] The Pinning & Replacement Problem Set
Precedence: list

--000000000000129a31060944eb15
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi John,

I think lightning and other second time-sensitive layers being hit by
safety issues whenever the blocks are full is common knowledge as the issue
is being described in the paper under the "forced expiration spam" issues
arising spontaneously within an environment with high block space demand. I
believe you have known variants of those issues with the "flood & loot"
style scenarios.

What is coming as a new problem with the novel replacement issues lay in
the fact that your counterparty channel might always defer the confirmation
of your honest on-chain spend, in a way which is compatible with miners
incentives.

While this has been commonized by the wide deployment of bip-125-style RBF
over network mempools, this ability has been always available to any party
reaching out directly to miners out-of-band with consensus-valid
transactions. In a world where miners are pseudonymous (as we're mostly
seeing today, not considering pools) miners motivated by maximizing their
satoshi-denominated income is a reasonable assumption in my opinion.

As an ecosystem beyond core to fix sustainably pinning and replacement
problems, and I agree with you here, we are stuck with a "between Scylla
and Charybdis" serious safety issue, I'm seeing the following options.
Matching yours, rephrasing in my own words to ensure correct understanding.

1. We can revert to a static world with no replacement-by-fee mechanism as
a widely deployed network policy.

I think in a world where miners are in competition you can always reach out
to them with out-of-band higher fees packages than available in their local
mempool.

2. We can design, implement and deploy policies to better capture
transaction-issuer intent on the way future spend candidates are allowed
(or not) to replace the current in-mempool transaction.

Sadly, I think with lightning and other second time-sensitive layers, there
is no "single" transaction-issuer intent, though you might have as many
intents that you have counterparties within your off-chain states, all with
competing interests.

3. We can remove all policy and let the network of nodes and the economic
ecosystem evolve on its own.

I think a lot of mempool policy in place is actually anti-DoS measures
which are protecting at least the lowest-performance full-nodes, and those
measures are the source of issues for pinning. "Pure" RBF sounds to me only
to make adversarial replacement issues worse (at least nversion=3D3-style
policy introduces some assumptions on max package size if widely deployed).

4. We can do nothing and let a fragmented mempool environment, where every
large lightning and bitcoin business have out-of-band relationships with
miners and pools to support their packages, with some service-level safety
agreements.

I think this option was weighted as a way to solve pinning issues at the
commitment and second-state level years ago by the lightning community,
though it was brushed aside as bringing rampant centralization of the
lightning network, where small lightning nodes might not have privileged
access to miners [0].

5. We can design and implement some magical consensus-change or alter the
processing requirements (bandwidth, CPU perf, I/O disk) of full-nodes on
the peer-to-peer network to align incentives between miners and lightning
and time-sensitive second-layers.

I think this option represents more or less what are the trade-offs of what
is discussed by the "reverse locktime" new bitcoin script opcode idea or
replacement cache at the mempool-level in core [1].

Best,
Antoine

[0] https://github.com/lightning/bolts/issues/783
[1] https://github.com/bitcoin/bitcoin/issues/28699

Le jeu. 2 nov. 2023 =C3=A0 11:38, John Carvalho via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> a =C3=A9crit :

> Good morning,
>
> All layers and technologies "on" Bitcoin will fail in situations where
> miners misbehave or the blocks & mempool remain consistently, overly full=
.
> Consider this as a "law" of Bitcoin/blockchains.
>
> In hindsight (for you, not me) it was very unwise to start messing with
> mempool policies, like with RBF, mempoolfullrbf. First-seen policy brough=
t
> a fragile harmony and utility to Bitcoin, which we were lucky to have for
> as long as we could.
>
> The engineers intentionally broke this. Mempoofullrbf washes away the
> ability for users to express their intent on whether to pin or replace a
> transaction, and now Lightning has BOTH pinning and replacement problems.
> You could argue this was inevitable. The point here is that layers have
> mempool and miner problems.
>
> Core has a few choices here, as I see it:
>
> 1. They can try to revert mempoolfullrbf and re-establish first-seen
> policy. Hard to say whether this would work, or whether it would be
> enough...
>
> 2. They can create additional policies that are enforced by default that
> allow people to flag how they want their txn handled, as in, a "pin this"
> vs "replace this" aspect to every txn. This is probably the best option, =
as
> it allows miners to know what people want and give it to them. This is
> utility, thus incentive-compatible.
>
> 3. Remove all policy and let the market evolve to deal with the chaos.
> Which is similar to the next option: do nothing.
>
> 4. Do nothing and allow a fractured mempool environment to evolve where
> large businesses form private contracts with miners and pools to support
> their own unsupported policies, so they can provide better experiences to
> customers and users.
>
> 5. Invent some miracle magical crypto cure that I am not capable of
> imagining at this time.
>
> I disclaim some ignorance to details of how other mempool research might
> affect these options and problems, but I am skeptical the dynamics
> discussed here can be removed entirely.
>
> --John Carvalho
> CEO, Synonym.to <http://synonym.to/>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

--000000000000129a31060944eb15
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi John,<div><br></div><div>I think lightning and other se=
cond time-sensitive layers being hit by safety issues whenever the blocks a=
re full is common knowledge as the issue is being described in the paper un=
der the &quot;forced expiration spam&quot; issues arising spontaneously wit=
hin an environment with high block space demand. I believe you have known v=
ariants of those issues with the &quot;flood &amp; loot&quot; style scenari=
os.</div><div><br></div><div>What is coming as a new problem with the novel=
 replacement issues lay in the fact that your counterparty channel might al=
ways defer the confirmation of your honest on-chain spend, in a way which i=
s compatible with=C2=A0miners incentives.</div><div><br></div><div>While th=
is has been commonized by the wide deployment of bip-125-style RBF over net=
work mempools, this ability=C2=A0has been always available to any party rea=
ching out directly=C2=A0to miners out-of-band with consensus-valid transact=
ions. In a world where miners=C2=A0are pseudonymous (as we&#39;re mostly se=
eing today, not considering pools) miners motivated by maximizing their sat=
oshi-denominated income is a reasonable assumption in my opinion.</div><div=
><br></div><div>As an ecosystem beyond core to fix sustainably pinning and =
replacement problems, and I agree with you here, we are stuck with a &quot;=
between Scylla and Charybdis&quot; serious safety issue, I&#39;m seeing the=
 following options. Matching yours, rephrasing in my own words to ensure co=
rrect understanding.</div><div><br></div><div>1. We can revert to a static =
world with no replacement-by-fee mechanism as a widely deployed network pol=
icy.</div><div><br></div><div>I think in a world where miners are in compet=
ition you can always reach out to them with out-of-band higher fees package=
s than available in their local mempool.</div><div><br></div><div>2. We can=
 design, implement and deploy policies to better capture transaction-issuer=
 intent on the way future spend candidates are allowed (or not) to replace =
the current in-mempool transaction.</div><div><br></div><div>Sadly, I think=
 with lightning and other second time-sensitive layers, there is no &quot;s=
ingle&quot; transaction-issuer intent, though you might have as many intent=
s that you have counterparties within your off-chain states, all with compe=
ting=C2=A0interests.</div><div><br></div><div>3. We can remove all policy a=
nd let the network of nodes and the economic ecosystem evolve on its own.</=
div><div><br></div><div>I think a lot of mempool policy in place is actuall=
y anti-DoS measures which are protecting at least the lowest-performance fu=
ll-nodes, and those measures are the source of issues for pinning. &quot;Pu=
re&quot; RBF sounds to me only to make adversarial replacement issues worse=
 (at least nversion=3D3-style policy introduces some assumptions on max pac=
kage size if widely deployed).</div><div><br></div><div>4. We can do nothin=
g and let a fragmented mempool environment, where every large lightning and=
 bitcoin business have out-of-band relationships with miners and pools to s=
upport their packages, with some service-level safety agreements.</div><div=
><br></div><div>I think this option was weighted as a way to solve pinning =
issues at the commitment and second-state level years ago by the lightning =
community, though it was brushed aside as bringing rampant centralization o=
f the lightning network, where small lightning nodes might not have privile=
ged access to miners [0].</div><div><br></div><div>5. We can design and imp=
lement some magical consensus-change or alter the processing requirements (=
bandwidth, CPU perf, I/O disk) of full-nodes on the peer-to-peer network to=
 align incentives between miners and lightning and time-sensitive second-la=
yers.</div><div><br></div><div>I think this option represents more or less =
what are the trade-offs of what is discussed by the &quot;reverse locktime&=
quot; new bitcoin script opcode idea or replacement cache at the mempool-le=
vel in core [1].</div><div><br></div><div>Best,</div><div>Antoine</div><div=
><br></div><div>[0]=C2=A0<a href=3D"https://github.com/lightning/bolts/issu=
es/783">https://github.com/lightning/bolts/issues/783</a></div><div>[1]=C2=
=A0<a href=3D"https://github.com/bitcoin/bitcoin/issues/28699">https://gith=
ub.com/bitcoin/bitcoin/issues/28699</a></div></div><br><div class=3D"gmail_=
quote"><div dir=3D"ltr" class=3D"gmail_attr">Le=C2=A0jeu. 2 nov. 2023 =C3=
=A0=C2=A011:38, John Carvalho via bitcoin-dev &lt;<a href=3D"mailto:bitcoin=
-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a>&g=
t; a =C3=A9crit=C2=A0:<br></div><blockquote class=3D"gmail_quote" style=3D"=
margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;bord=
er-left-color:rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div dir=
=3D"ltr"><div dir=3D"ltr" class=3D"gmail_signature"><div dir=3D"ltr"><div d=
ir=3D"ltr"><div dir=3D"ltr">Good morning,=C2=A0</div><div dir=3D"ltr"><br><=
/div><div dir=3D"ltr">All layers and technologies &quot;on&quot; Bitcoin wi=
ll fail in situations where miners misbehave or the blocks &amp; mempool re=
main consistently, overly full. Consider this as a &quot;law&quot; of Bitco=
in/blockchains.=C2=A0</div><div dir=3D"ltr"><br></div><div dir=3D"ltr">In h=
indsight (for you, not me) it was very unwise to start messing with mempool=
 policies, like with RBF, mempoolfullrbf. First-seen policy brought a fragi=
le harmony and utility to Bitcoin, which we were lucky to have for as long =
as we could.=C2=A0</div><div dir=3D"ltr"><br></div><div dir=3D"ltr">The eng=
ineers intentionally broke this. Mempoofullrbf washes away the ability for =
users to express their intent on whether to pin or replace a transaction, a=
nd now Lightning has BOTH pinning and replacement problems. You could argue=
 this was inevitable. The point here is that layers have mempool and miner =
problems.</div><div dir=3D"ltr"><br></div><div dir=3D"ltr">Core has a few c=
hoices here, as I see it:=C2=A0</div><div dir=3D"ltr"><br></div><div dir=3D=
"ltr">1. They can try to revert mempoolfullrbf and re-establish first-seen =
policy. Hard to say whether this would work, or whether it would be enough.=
..=C2=A0</div><div dir=3D"ltr"><br></div><div dir=3D"ltr">2. They can creat=
e additional policies that are enforced by default that allow people to fla=
g how they want their txn handled, as in, a &quot;pin this&quot; vs &quot;r=
eplace this&quot; aspect to every txn. This is probably the best option, as=
 it allows miners to know what people want and give it to them. This is uti=
lity, thus incentive-compatible.=C2=A0</div><div dir=3D"ltr"><br></div><div=
 dir=3D"ltr">3. Remove all policy and let the market evolve to deal with th=
e chaos. Which is similar to the next option: do nothing.=C2=A0</div><div d=
ir=3D"ltr"><br></div><div dir=3D"ltr">4. Do nothing and allow a fractured m=
empool environment to evolve where large businesses form private contracts =
with miners and pools to support their own unsupported policies, so they ca=
n provide better experiences to customers and users.=C2=A0</div><div dir=3D=
"ltr"><br></div><div dir=3D"ltr">5. Invent some miracle magical crypto cure=
 that I am not capable of imagining at this time.</div><div dir=3D"ltr"><br=
></div><div>I disclaim some ignorance to details of how other mempool resea=
rch might affect these options and problems, but I am skeptical the dynamic=
s discussed here can be removed entirely.</div><div dir=3D"ltr"><br>--John =
Carvalho</div><div dir=3D"ltr" style=3D"color:rgb(34,34,34)">CEO,=C2=A0<a h=
ref=3D"http://synonym.to/" style=3D"color:rgb(17,85,204)" target=3D"_blank"=
>Synonym.to</a></div></div></div></div></div></div>
_______________________________________________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">=
bitcoin-dev@lists.linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" =
rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail=
man/listinfo/bitcoin-dev</a><br>
</blockquote></div>

--000000000000129a31060944eb15--