Return-Path: Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 87695C0175; Thu, 23 Apr 2020 01:17:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 76BF2204B8; Thu, 23 Apr 2020 01:17:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yyrM1F4oqJth; Thu, 23 Apr 2020 01:17:43 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) by silver.osuosl.org (Postfix) with ESMTPS id 0BCDE2046A; Thu, 23 Apr 2020 01:17:42 +0000 (UTC) Received: from mail-io1-f43.google.com (mail-io1-f43.google.com [209.85.166.43]) (authenticated bits=0) (User authenticated as jlrubin@ATHENA.MIT.EDU) by outgoing.mit.edu (8.14.7/8.12.4) with ESMTP id 03N1Hf0J031074 (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128 verify=NOT); Wed, 22 Apr 2020 21:17:41 -0400 Received: by mail-io1-f43.google.com with SMTP id i3so4602168ioo.13; Wed, 22 Apr 2020 18:17:41 -0700 (PDT) X-Gm-Message-State: AGi0PuZOpqA3uvhRNVtpIix9GkLTOYe82gzDxrwYHkpUBuHwP993TbTF Syriv3Nz2cRR7K6PzI0c3W8RQc+Swvb4K9UtJ78= X-Google-Smtp-Source: APiQypKWQvykRnaVFo/1Rz7X4k2NduTEC+xmz6pS9XGG0kSSDwWX6zZVWe46MLUlurkiGaC9+SRvkm3Rgo6ICa3h0fs= X-Received: by 2002:a02:94a3:: with SMTP id x32mr1156071jah.20.1587604660750; Wed, 22 Apr 2020 18:17:40 -0700 (PDT) MIME-Version: 1.0 References: <9F7F7A00-FFA5-48E8-9BA3-8D71A55B2659@mattcorallo.com> In-Reply-To: From: Jeremy Date: Wed, 22 Apr 2020 18:18:05 -0700 X-Gmail-Original-Message-ID: Message-ID: To: Bitcoin Protocol Discussion Content-Type: multipart/alternative; boundary="000000000000d90d5605a3eb03c4" Cc: lightning-dev Subject: Re: [bitcoin-dev] RBF Pinning with Counterparties and Competing Interest X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Apr 2020 01:17:44 -0000 --000000000000d90d5605a3eb03c4 Content-Type: text/plain; charset="UTF-8" Hi everyone, Sorry to just be getting to a response here. Hadn't noticed it till now. *(Plug: If anyone or their organizations would like to assist in funding the work described below for a group of developers, I've been working to put resources together for funding the above for a few months now, and I think it would be high leverage towards seeing this through. There are a lot of unsexy tasks to do that aren't coming up with a solution (e.g.,writing a myriad of Mempool stress test scenarios) that can be a well defined full-time job for someone to do.)* I've been working on exactly this problem in the mempool for months now. I'm deeply familiar with the issues here and the types of pinning possible. I think everyone can recognize that with my work on OP_CTV I want nothing more than the mempool to be able to accept whatever long chains we can throw at it, but I'm pretty well steeped at this point in the obstacles to doing that. I don't think that we should be entertaining further carve outs at the moment, unless it is really trivial. Every new carve out rule added to the way that the mempool operates is removing complexity invariants we aim to preserve in the mempool in order to keep nodes operational. Many of these invariants are well documented, some are not. I'm happy to go off list for a more thorough discussion with anyone qualified to have it; this isn't the best venue for that discussion. From my point of view the path forward here is to dedicate more development resources towards finishing the mempool project I began. You can see the outstanding work here: https://github.com/bitcoin/bitcoin/projects/14, contributing review towards moving those PRs forward will greatly improve our ability to consider a stopgap carve out measure. The current focus of this work is primarily on: 1) Testing Construction to better test & catch regressions or vulnerabilities introduced or extant in mempool 2) Refactoring algorithms in mempool to reduce constant factors & asymptotics 3) Package Relay None of these fix the exact problem at hand though, but here's part of how they can help us: If we finish up the algorithmic refactors I've been working on it seems plausible to do a one-off increase of descendants limits to say, 100 descendants with no restriction. However, we could use the opportunity to use the 75 descendant increase exclusively for a new carve out, and apply some new stricter rules in that extra space. There are a few anti-pinning countermeasures that you can apply in that space that you would not generally want in the mempool. An example of one is that any new transaction must pay more feerate and absolute fee than every child in that space. Or that only the highest fee paying branch of the excess transactions are mineable, no others. Another would be disabling RBF past that watermark. In all likelihood, different subsystems interacting with the mempool will require a different set of restrictions each with the current architecture, I don't think there's a magic bullet. Package relay is a promising approach for a future pinning solution as there are opportunities to attach to packages compact proofs of improved fee efficiency for pinned transactions. But the ground work for package relay needs to come first. This is theoretically possible with our current architecture of the mempool and can probably address much of the pinning concerns by replacing pinning with more rational eviction policies. Longer term I've been working on plans and designs to completely re-do the mempool's architecture to make it behave for arbitrary cases. It's possible to one day lift all preemptively enforced (e.g., before acceptance) descendants limits, which can solve this problem for good. There is more than one potentially good solution here, and a conjunction of them can be used as they affect independent sub systems. But this work will probably take years to complete to the point where restrictions can realistically be lifted. If developers would like to coordinate resources around completing this work and making more regular progress on it I'm happy to help point people to specific tasks that need to be done in order to accelerate this and help serialize the work so that we can not get into rebase hell. Originally I had the plug at the top as a closing note, but I figured people might miss it. Best, Jeremy -- @JeremyRubin --000000000000d90d5605a3eb03c4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi every= one,

Sorry to just be getting to a response here. Hadn't noticed = it till now.

(Plug: If anyone or their organizations would like to= assist in funding the work described below for a group of developers, I= 9;ve been working to put resources together for funding the above for a few= months now, and I think it would be high leverage towards seeing this thro= ugh. There are a lot of unsexy tasks to do=C2=A0 that aren't coming up = with a solution (e.g.,writing a myriad of Mempool stress test scenarios) th= at can be a well defined full-time job for someone to do.)

I'= ve been working on exactly this problem in the mempool for months now. I= 9;m deeply familiar with the issues here and the types of pinning possible.= I think everyone can recognize that with my work on OP_CTV I want nothing = more than the mempool to be able to accept whatever long chains we can thro= w at it, but I'm pretty well steeped at this point in the obstacles to = doing that.

I don't think that we should be entertaining further = carve outs at the moment,=C2=A0unless it is really trivial. Every new carve= out rule added to the way that the mempool operates is removing complexity= invariants we aim to preserve in the mempool in order to keep nodes operat= ional. Many of these invariants are well documented, some are not. I'm = happy to go off list for a more thorough discussion with anyone qualified t= o have it; this isn't the best venue for that discussion.

From my= point of view the path forward here is to dedicate more development resour= ces towards finishing the mempool project I began. You can see the outstand= ing work here:=C2=A0https://github.com/bit= coin/bitcoin/projects/14, contributing review towards moving those PRs = forward will greatly improve our ability to consider a stopgap carve out me= asure.

The current focus of this work is primarily on:

1) Testi= ng Construction to better test & catch regressions or vulnerabilities i= ntroduced or extant in mempool
2) Ref= actoring algorithms in mempool to reduce constant factors & asymptotics=
3) Package Relay


None o= f these fix the exact problem at hand though, but here's part of how th= ey can help us:

If we finish up the algorithmic refactors I've b= een working on it seems plausible to do a one-off increase of descendants l= imits to say, 100 descendants with no restriction. However, we could use th= e opportunity to use the 75 descendant increase exclusively for a new carve= out, and apply some new stricter rules in that extra space. There are a fe= w anti-pinning countermeasures that you can apply in that space that you wo= uld not generally want in the mempool. An example of one is that any new tr= ansaction must pay more feerate and absolute fee than every child in that s= pace. Or that only the highest fee paying branch of the excess transactions= are mineable, no others. Another would be disabling RBF past that watermar= k. In all likelihood, different subsystems interacting with the mempool wil= l require a different set of restrictions each with the current architectur= e, I don't think there's a magic bullet.

Package relay is a p= romising approach for a future pinning solution as there are opportunities = to attach to packages compact proofs of improved fee efficiency for pinned = transactions. But the ground work for package relay needs to come first. Th= is is theoretically possible with our current architecture of the mempool a= nd can probably address much of the pinning concerns by replacing pinning w= ith more rational eviction policies.

Longer term I've been wo= rking on plans and designs to completely re-do the mempool's architectu= re to make it behave for arbitrary cases. It's possible to one day lift= all preemptively enforced (e.g., before acceptance) descendants limits, wh= ich can solve this problem for good. There is more than one potentially goo= d solution here, and a conjunction of them can be used as they affect indep= endent sub systems. But this work will probably take years to complete to t= he point where restrictions can realistically be lifted.

If developer= s would like to coordinate resources around completing this work and making= more regular progress on it I'm happy to help point people to specific= tasks that need to be done in order to accelerate this and help serialize = the work so that we can not get into rebase hell.

Originally I had th= e plug at the top as a closing note, but I figured people might miss it.

Best,

Jeremy


--000000000000d90d5605a3eb03c4--