path: root/ac/2475bc74a7db991519c1aba43cb1c6e62ceeda

Return-Path: <jlrubin@mit.edu>
Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136])
 by lists.linuxfoundation.org (Postfix) with ESMTP id 87695C0175;
 Thu, 23 Apr 2020 01:17:44 +0000 (UTC)
Received: from localhost (localhost [127.0.0.1])
 by silver.osuosl.org (Postfix) with ESMTP id 76BF2204B8;
 Thu, 23 Apr 2020 01:17:44 +0000 (UTC)
X-Virus-Scanned: amavisd-new at osuosl.org
Received: from silver.osuosl.org ([127.0.0.1])
 by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id yyrM1F4oqJth; Thu, 23 Apr 2020 01:17:43 +0000 (UTC)
X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6
X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6
Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11])
 by silver.osuosl.org (Postfix) with ESMTPS id 0BCDE2046A;
 Thu, 23 Apr 2020 01:17:42 +0000 (UTC)
Received: from mail-io1-f43.google.com (mail-io1-f43.google.com
 [209.85.166.43]) (authenticated bits=0)
 (User authenticated as jlrubin@ATHENA.MIT.EDU)
 by outgoing.mit.edu (8.14.7/8.12.4) with ESMTP id 03N1Hf0J031074
 (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128 verify=NOT);
 Wed, 22 Apr 2020 21:17:41 -0400
Received: by mail-io1-f43.google.com with SMTP id i3so4602168ioo.13;
 Wed, 22 Apr 2020 18:17:41 -0700 (PDT)
X-Gm-Message-State: AGi0PuZOpqA3uvhRNVtpIix9GkLTOYe82gzDxrwYHkpUBuHwP993TbTF
 Syriv3Nz2cRR7K6PzI0c3W8RQc+Swvb4K9UtJ78=
X-Google-Smtp-Source: APiQypKWQvykRnaVFo/1Rz7X4k2NduTEC+xmz6pS9XGG0kSSDwWX6zZVWe46MLUlurkiGaC9+SRvkm3Rgo6ICa3h0fs=
X-Received: by 2002:a02:94a3:: with SMTP id x32mr1156071jah.20.1587604660750; 
 Wed, 22 Apr 2020 18:17:40 -0700 (PDT)
MIME-Version: 1.0
References: <CAO3Pvs8Dx8ATXfQRA3p7DzzHcYZz1nF4g=+ZHvSukL0jMbxAOw@mail.gmail.com>
 <9F7F7A00-FFA5-48E8-9BA3-8D71A55B2659@mattcorallo.com>
 <CAO3Pvs_hNhEjX_tAatnij-4vd1cgPk8DPxOdrgDVuia7h=z5UQ@mail.gmail.com>
In-Reply-To: <CAO3Pvs_hNhEjX_tAatnij-4vd1cgPk8DPxOdrgDVuia7h=z5UQ@mail.gmail.com>
From: Jeremy <jlrubin@mit.edu>
Date: Wed, 22 Apr 2020 18:18:05 -0700
X-Gmail-Original-Message-ID: <CAD5xwhhKirBiF8ADE-H366opYA1idteC-qWYvQ8EUeAG4HMiZA@mail.gmail.com>
Message-ID: <CAD5xwhhKirBiF8ADE-H366opYA1idteC-qWYvQ8EUeAG4HMiZA@mail.gmail.com>
To: Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
Content-Type: multipart/alternative; boundary="000000000000d90d5605a3eb03c4"
Cc: lightning-dev <lightning-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] RBF Pinning with Counterparties and Competing
	Interest
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>, 
 <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>, 
 <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Thu, 23 Apr 2020 01:17:44 -0000

--000000000000d90d5605a3eb03c4
Content-Type: text/plain; charset="UTF-8"

Hi everyone,

Sorry to just be getting to a response here. Hadn't noticed it till now.

*(Plug: If anyone or their organizations would like to assist in funding
the work described below for a group of developers, I've been working to
put resources together for funding the above for a few months now, and I
think it would be high leverage towards seeing this through. There are a
lot of unsexy tasks to do  that aren't coming up with a solution
(e.g.,writing a myriad of Mempool stress test scenarios) that can be a well
defined full-time job for someone to do.)*

I've been working on exactly this problem in the mempool for months now.
I'm deeply familiar with the issues here and the types of pinning possible.
I think everyone can recognize that with my work on OP_CTV I want nothing
more than the mempool to be able to accept whatever long chains we can
throw at it, but I'm pretty well steeped at this point in the obstacles to
doing that.

I don't think that we should be entertaining further carve outs at the
moment, unless it is really trivial. Every new carve out rule added to the
way that the mempool operates is removing complexity invariants we aim to
preserve in the mempool in order to keep nodes operational. Many of these
invariants are well documented, some are not. I'm happy to go off list for
a more thorough discussion with anyone qualified to have it; this isn't the
best venue for that discussion.

From my point of view the path forward here is to dedicate more development
resources towards finishing the mempool project I began. You can see the
outstanding work here: https://github.com/bitcoin/bitcoin/projects/14,
contributing review towards moving those PRs forward will greatly improve
our ability to consider a stopgap carve out measure.

The current focus of this work is primarily on:

1) Testing Construction to better test & catch regressions or
vulnerabilities introduced or extant in mempool
2) Refactoring algorithms in mempool to reduce constant factors &
asymptotics
3) Package Relay


None of these fix the exact problem at hand though, but here's part of how
they can help us:

If we finish up the algorithmic refactors I've been working on it seems
plausible to do a one-off increase of descendants limits to say, 100
descendants with no restriction. However, we could use the opportunity to
use the 75 descendant increase exclusively for a new carve out, and apply
some new stricter rules in that extra space. There are a few anti-pinning
countermeasures that you can apply in that space that you would not
generally want in the mempool. An example of one is that any new
transaction must pay more feerate and absolute fee than every child in that
space. Or that only the highest fee paying branch of the excess
transactions are mineable, no others. Another would be disabling RBF past
that watermark. In all likelihood, different subsystems interacting with
the mempool will require a different set of restrictions each with the
current architecture, I don't think there's a magic bullet.

Package relay is a promising approach for a future pinning solution as
there are opportunities to attach to packages compact proofs of improved
fee efficiency for pinned transactions. But the ground work for package
relay needs to come first. This is theoretically possible with our current
architecture of the mempool and can probably address much of the pinning
concerns by replacing pinning with more rational eviction policies.

Longer term I've been working on plans and designs to completely re-do the
mempool's architecture to make it behave for arbitrary cases. It's possible
to one day lift all preemptively enforced (e.g., before acceptance)
descendants limits, which can solve this problem for good. There is more
than one potentially good solution here, and a conjunction of them can be
used as they affect independent sub systems. But this work will probably
take years to complete to the point where restrictions can realistically be
lifted.

If developers would like to coordinate resources around completing this
work and making more regular progress on it I'm happy to help point people
to specific tasks that need to be done in order to accelerate this and help
serialize the work so that we can not get into rebase hell.

Originally I had the plug at the top as a closing note, but I figured
people might miss it.

Best,

Jeremy


--
@JeremyRubin <https://twitter.com/JeremyRubin>

--000000000000d90d5605a3eb03c4
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><div class=3D"gmail_default" style=3D"fon=
t-family:arial,helvetica,sans-serif;font-size:small;color:#000000">Hi every=
one,</div><div class=3D"gmail_default" style=3D"font-family:arial,helvetica=
,sans-serif;font-size:small;color:#000000"><br></div><div class=3D"gmail_de=
fault" style=3D"font-family:arial,helvetica,sans-serif;font-size:small;colo=
r:#000000">Sorry to just be getting to a response here. Hadn&#39;t noticed =
it till now.</div><div class=3D"gmail_default" style=3D"font-family:arial,h=
elvetica,sans-serif;font-size:small;color:#000000"><br></div><div class=3D"=
gmail_default" style=3D"font-family:arial,helvetica,sans-serif;font-size:sm=
all;color:#000000"><i>(Plug: If anyone or their organizations would like to=
 assist in funding the work described below for a group of developers, I&#3=
9;ve been working to put resources together for funding the above for a few=
 months now, and I think it would be high leverage towards seeing this thro=
ugh. There are a lot of unsexy tasks to do=C2=A0 that aren&#39;t coming up =
with a solution (e.g.,writing a myriad of Mempool stress test scenarios) th=
at can be a well defined full-time job for someone to do.)</i></div><div cl=
ass=3D"gmail_default" style=3D"font-family:arial,helvetica,sans-serif;font-=
size:small;color:#000000"><br></div><div class=3D"gmail_default" style=3D"f=
ont-family:arial,helvetica,sans-serif;font-size:small;color:#000000">I&#39;=
ve been working on exactly this problem in the mempool for months now. I&#3=
9;m deeply familiar with the issues here and the types of pinning possible.=
 I think everyone can recognize that with my work on OP_CTV I want nothing =
more than the mempool to be able to accept whatever long chains we can thro=
w at it, but I&#39;m pretty well steeped at this point in the obstacles to =
doing that.</div><div class=3D"gmail_default" style=3D"font-family:arial,he=
lvetica,sans-serif;font-size:small;color:#000000"><br></div><div class=3D"g=
mail_default" style=3D"font-family:arial,helvetica,sans-serif;font-size:sma=
ll;color:#000000">I don&#39;t think that we should be entertaining further =
carve outs at the moment,=C2=A0unless it is really trivial. Every new carve=
 out rule added to the way that the mempool operates is removing complexity=
 invariants we aim to preserve in the mempool in order to keep nodes operat=
ional. Many of these invariants are well documented, some are not. I&#39;m =
happy to go off list for a more thorough discussion with anyone qualified t=
o have it; this isn&#39;t the best venue for that discussion.</div><div cla=
ss=3D"gmail_default" style=3D"font-family:arial,helvetica,sans-serif;font-s=
ize:small;color:#000000"><br></div><div class=3D"gmail_default" style=3D"fo=
nt-family:arial,helvetica,sans-serif;font-size:small;color:#000000">From my=
 point of view the path forward here is to dedicate more development resour=
ces towards finishing the mempool project I began. You can see the outstand=
ing work here:=C2=A0<a href=3D"https://github.com/bitcoin/bitcoin/projects/=
14" style=3D"font-family:Arial,Helvetica,sans-serif">https://github.com/bit=
coin/bitcoin/projects/14</a>, contributing review towards moving those PRs =
forward will greatly improve our ability to consider a stopgap carve out me=
asure.</div><div class=3D"gmail_default" style=3D"font-family:arial,helveti=
ca,sans-serif;font-size:small;color:#000000"><br></div><div class=3D"gmail_=
default" style=3D"font-family:arial,helvetica,sans-serif;font-size:small;co=
lor:#000000">The current focus of this work is primarily on:</div><div clas=
s=3D"gmail_default" style=3D"font-family:arial,helvetica,sans-serif;font-si=
ze:small;color:#000000"><br></div><div class=3D"gmail_default" style=3D"fon=
t-family:arial,helvetica,sans-serif;font-size:small;color:#000000">1) Testi=
ng Construction to better test &amp; catch regressions or vulnerabilities i=
ntroduced or extant in mempool</div><div class=3D"gmail_default" style=3D"f=
ont-family:arial,helvetica,sans-serif;font-size:small;color:#000000">2) Ref=
actoring algorithms in mempool to reduce constant factors &amp; asymptotics=
</div><div class=3D"gmail_default" style=3D"font-family:arial,helvetica,san=
s-serif;font-size:small;color:#000000">3) Package Relay</div><div class=3D"=
gmail_default" style=3D"font-family:arial,helvetica,sans-serif;font-size:sm=
all;color:#000000"><br></div><div class=3D"gmail_default" style=3D"font-fam=
ily:arial,helvetica,sans-serif;font-size:small;color:#000000"></div><div cl=
ass=3D"gmail_default" style=3D"font-family:arial,helvetica,sans-serif;font-=
size:small;color:#000000"><br></div><div class=3D"gmail_default" style=3D"f=
ont-family:arial,helvetica,sans-serif;font-size:small;color:#000000">None o=
f these fix the exact problem at hand though, but here&#39;s part of how th=
ey can help us:</div><div class=3D"gmail_default" style=3D"font-family:aria=
l,helvetica,sans-serif;font-size:small;color:#000000"><br></div><div class=
=3D"gmail_default" style=3D"font-family:arial,helvetica,sans-serif;font-siz=
e:small;color:#000000">If we finish up the algorithmic refactors I&#39;ve b=
een working on it seems plausible to do a one-off increase of descendants l=
imits to say, 100 descendants with no restriction. However, we could use th=
e opportunity to use the 75 descendant increase exclusively for a new carve=
 out, and apply some new stricter rules in that extra space. There are a fe=
w anti-pinning countermeasures that you can apply in that space that you wo=
uld not generally want in the mempool. An example of one is that any new tr=
ansaction must pay more feerate and absolute fee than every child in that s=
pace. Or that only the highest fee paying branch of the excess transactions=
 are mineable, no others. Another would be disabling RBF past that watermar=
k. In all likelihood, different subsystems interacting with the mempool wil=
l require a different set of restrictions each with the current architectur=
e, I don&#39;t think there&#39;s a magic bullet.</div><div class=3D"gmail_d=
efault" style=3D"font-family:arial,helvetica,sans-serif;font-size:small;col=
or:#000000"><br></div><div class=3D"gmail_default" style=3D"font-family:ari=
al,helvetica,sans-serif;font-size:small;color:#000000">Package relay is a p=
romising approach for a future pinning solution as there are opportunities =
to attach to packages compact proofs of improved fee efficiency for pinned =
transactions. But the ground work for package relay needs to come first. Th=
is is theoretically possible with our current architecture of the mempool a=
nd can probably address much of the pinning concerns by replacing pinning w=
ith more rational eviction policies.<br></div><div class=3D"gmail_default" =
style=3D"font-family:arial,helvetica,sans-serif;font-size:small;color:#0000=
00"><br></div><div class=3D"gmail_default" style=3D"font-family:arial,helve=
tica,sans-serif;font-size:small;color:#000000">Longer term I&#39;ve been wo=
rking on plans and designs to completely re-do the mempool&#39;s architectu=
re to make it behave for arbitrary cases. It&#39;s possible to one day lift=
 all preemptively enforced (e.g., before acceptance) descendants limits, wh=
ich can solve this problem for good. There is more than one potentially goo=
d solution here, and a conjunction of them can be used as they affect indep=
endent sub systems. But this work will probably take years to complete to t=
he point where restrictions can realistically be lifted.</div><div class=3D=
"gmail_default" style=3D"font-family:arial,helvetica,sans-serif;font-size:s=
mall;color:#000000"><br></div><div class=3D"gmail_default" style=3D"font-fa=
mily:arial,helvetica,sans-serif;font-size:small;color:#000000">If developer=
s would like to coordinate resources around completing this work and making=
 more regular progress on it I&#39;m happy to help point people to specific=
 tasks that need to be done in order to accelerate this and help serialize =
the work so that we can not get into rebase hell.</div><div class=3D"gmail_=
default" style=3D"font-family:arial,helvetica,sans-serif;font-size:small;co=
lor:#000000"><br></div><div class=3D"gmail_default" style=3D"font-family:ar=
ial,helvetica,sans-serif;font-size:small;color:#000000">Originally I had th=
e plug at the top as a closing note, but I figured people might miss it.</d=
iv><div class=3D"gmail_default" style=3D"font-family:arial,helvetica,sans-s=
erif;font-size:small;color:#000000"><br></div><div class=3D"gmail_default" =
style=3D"font-family:arial,helvetica,sans-serif;font-size:small;color:#0000=
00">Best,</div><div class=3D"gmail_default" style=3D"font-family:arial,helv=
etica,sans-serif;font-size:small;color:#000000"><br></div><div class=3D"gma=
il_default" style=3D"font-family:arial,helvetica,sans-serif;font-size:small=
;color:#000000">Jeremy</div><div class=3D"gmail_default" style=3D"font-fami=
ly:arial,helvetica,sans-serif;font-size:small;color:#000000"><br></div><div=
 class=3D"gmail_default" style=3D"font-family:arial,helvetica,sans-serif;fo=
nt-size:small;color:#000000"><br></div><div><div dir=3D"ltr" class=3D"gmail=
_signature" data-smartmail=3D"gmail_signature"><div dir=3D"ltr">--<br><a hr=
ef=3D"https://twitter.com/JeremyRubin" target=3D"_blank">@JeremyRubin</a></=
div></div></div></div></div>

--000000000000d90d5605a3eb03c4--