MIME-Version: 1.0
References: <CAM1a7P04apCwwmqNp4VLRam5_uk59tVRWv74UVD_g0-DUGNghw@mail.gmail.com>
In-Reply-To: <CAM1a7P04apCwwmqNp4VLRam5_uk59tVRWv74UVD_g0-DUGNghw@mail.gmail.com>
From: Gloria Zhao <gloriajzhao@gmail.com>
Date: Tue, 26 Oct 2021 19:16:51 +0100
Message-ID: <CAFXO6=Jk0MAqQ6u5JCrpC3eMv=bF3DT6wH6Y60zb_b-beU4mcg@mail.gmail.com>
To: lisa neigut <niftynei@gmail.com>, 
 Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
Content-Type: multipart/alternative; boundary="000000000000f5256705cf457bd0"
Subject: Re: [bitcoin-dev] death to the mempool, long live the mempool
Precedence: list

--000000000000f5256705cf457bd0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Lisa,

Some background for people who are not familiar with mempool:

The mempool is a cache of unconfirmed transactions, designed in a way
to help miners efficiently pick the highest feerate packages to
include in new blocks. It stores more than a block's worth of
transactions because transaction volume fluctuates and any rational
miner would try to maximize the fees in their blocks; in a reorg, we
also don't want to completely forget what transactions were in the
now-stale tip.

In Bitcoin Core, full nodes keep a mempool by default. The additional
requirements for keeping a mempool are minimal (300MB storage, can be
configured to be lower) because anyone, anywhere in the world, should
be able to run a node and broadcast a Bitcoin payment without special
connectivity to some specific set of people or expensive/inaccessible
hardware. Perhaps connecting directly to miners can be a solution for
some people, but I don't think it's healthy for the network.

Some benefits of keeping a mempool as a non-mining node include:
- Fee estimation based on your node's knowledge of unconfirmed
transactions + historical data.
- Dramatically increased block validation (and thus propagation)
speed, since you cache signature and script results of transactions
before they are confirmed.
- Reduced block relay bandwidth usage (Bitcoin Core nodes use BIP152
compact block relay), as you don't need to re-request the block
transactions you already have in your mempool.
- Wallet ability to send/receive transactions that spend unconfirmed output=
s.

> I had the realization that the mempool is obsolete and should be eliminat=
ed.

I assume you mean that the mempool should still exist but be turned
off for non-mining nodes. A block template producer needs to keep
unconfirmed transactions somewhere.
On Bitcoin Core today, you can use the -blocksonly config option to
ignore incoming transactions (effectively switching off your mempool),
but there are strong reasons not to do this:
- It is trivial for your peers to detect that all transactions
broadcasted by your node =3D from your wallet. Linking your node to your
transactions is a very bad privacy leak.
- You must query someone else for what feerate to put on your transaction.
- You can't use BIP152 compact block relay, so your network bandwidth
usage spikes at every block. You also can't cache validation results,
so your block validation speed slows down.

> Removing the mempool would greatly reduce the bandwidth requirement for r=
unning a node...

If you're having problems with your individual node's bandwidth usage,
you can also decrease the number of connections you make or turn off
incoming connections. There are efforts to reduce transaction relay
bandwidth usage network-wide [1].

> Removing the mempool would... keep intentionality of transactions private=
 until confirmed/irrevocable...

I'm confused - what is the purpose of keeping a transaction private
until it confirms? Don't miners still see your transaction? A
confirmed transaction is not irrevocable; reorgs happen.

> Removing the mempool would... naturally resolve all current issues inhere=
nt in package relay and rbf rules.

Removing the mempool does not help with this. How does a miner decide
whether a conflicting transaction is an economically-advantageous
replacement or a DoS attack? How do you submit your CPFP if the parent
is below the mempool minimum feerate? Do they already have a different
relay/mempool implementation handling all of these problems but don't
aren't sharing it with the rest of the community?

> Initial feerate estimation would need to be based on published blocks, no=
t pending transactions (as this information would no longer be available), =
or from direct interactions with block producers.

There are many reasons why using only published blocks for fee
estimates is a flawed design, including:

- The miner of a block can artificially inflate the feerate of the
transactions in their mempool simply by including a few of their own
transactions that pay extremely high feerates. This costs them
nothing, as they collect the fees.
- A miner constructs a block based on the transactions in their
mempool. Your transaction's feerate may have been enough to be
included 2 blocks ago or a week ago, but it will be compared to the
other unconfirmed transactions available to the miner now. They can
tell you what's in their mempool or what the next-block feerate is,
but you would be a fool to believe them.

See also [2],[3].

> Provided the number of block template producing actors remains beneath, s=
ay 1000, it=E2=80=99d be quite feasible to publish a list of tor endpoints =
that nodes can independently  + directly submit their transactions to. In f=
act, merely allowing users to select their own list of endpoints to use alt=
ernatively to the mempool would be a low effort starting point for the even=
tual replacement.

As a thought experiment, let's imagine we have some public registry of
mining nodes' tor endpoints and we use it for this secondary
direct-to-miner transaction relay network. If the registry is
maintained (by who?) and accurate (based on whose word?), it is a
point of failure for transaction censorship and deanonymization, as
well as an additional barrier to becoming a miner, encouraging
centralization.
The other possibility is that the registry is not accurate. In fact,
unless the registry requires miners to identify themselves (which
others on this thread have already pointed out is ill-advised), this
should be treated similarly to regular addr gossip. We would never
automatically trust that the entity behind the endpoint provides the
service it advertises, is an honest node that won't simply blackhole
our transaction, or even belongs to a Bitcoin node at all.

Best,
Gloria

[1]: https://arxiv.org/pdf/1905.10518.pdf
[2]: https://bitcointechtalk.com/an-introduction-to-bitcoin-core-fee-estima=
tion-27920880ad0
[3]: https://gist.github.com/morcos/d3637f015bc4e607e1fd10d8351e9f41


On Tue, Oct 26, 2021 at 8:38 AM lisa neigut via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Hi all,
>
> In a recent conversation with @glozow, I had the realization that the
> mempool is obsolete and should be eliminated.
>
> Instead, users should submit their transactions directly to mining pools,
> preferably over an anonymous communication network such as tor. This can
> easily be achieved by mining pools running a tor onion node for this
> express purpose (or via a lightning network extension etc)
>
> Mempools make sense in a world where mining is done by a large number of
> participating nodes, eg where the block template is constructed by a
> majority of the participants on the network. In this case, it is necessar=
y
> to socialize pending transaction data to all participants, as you don=E2=
=80=99t
> know which participant will be constructing the winning block template.
>
> In reality however, mempool relay is unnecessary where the majority of
> hashpower and thus block template creation is concentrated in a
> semi-restricted set.
>
> Removing the mempool would greatly reduce the bandwidth requirement for
> running a node, keep intentionality of transactions private until
> confirmed/irrevocable, and naturally resolve all current issues inherent =
in
> package relay and rbf rules. It also resolves the recent minimum relay
> questions, as relay is no longer a concern for unmined transactions.
>
> Provided the number of block template producing actors remains beneath,
> say 1000, it=E2=80=99d be quite feasible to publish a list of tor endpoin=
ts that
> nodes can independently  + directly submit their transactions to. In fact=
,
> merely allowing users to select their own list of endpoints to use
> alternatively to the mempool would be a low effort starting point for the
> eventual replacement.
>
> On the other hand, removing the mempool would greatly complicate solo
> mining and would also make BetterHash proposals, which move the block
> template construction away from a centralized mining pool back to the
> individual miner, much more difficult. It also makes explicit the target
> for DoS attacks.
>
> A direct communication channel between block template construction venues
> and transaction proposers also provides a venue for direct feedback wrt
> acceptable feerates at the time, which both makes transaction confirmatio=
n
> timelines less variable as well as provides block producers a mechanism f=
or
> (independently) enforcing their own minimum security budget. In other
> words, expressing a minimum acceptable feerate for continued operation.
>
> Initial feerate estimation would need to be based on published blocks, no=
t
> pending transactions (as this information would no longer be available), =
or
> from direct interactions with block producers.
>
>
> ~niftynei
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

--000000000000f5256705cf457bd0
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><pre>Hi Lisa,<br><br>Some background for people who are no=
t familiar with mempool:

The mempool is a cache of unconfirmed transactions, designed in a way to he=
lp miners efficiently pick the highest feerate packages to include in new b=
locks. It stores more than a block&#39;s worth of transactions because tran=
saction volume fluctuates and any rational miner would try to maximize the =
fees in their blocks; in a reorg, we also don&#39;t want to completely forg=
et what transactions were in the now-stale tip.

In Bitcoin Core, full nodes keep a mempool by default. The additional requi=
rements for keeping a mempool are minimal (300MB storage, can be configured=
 to be lower) because anyone, anywhere in the world, should be able to run =
a node and broadcast a Bitcoin payment without special connectivity to some=
 specific set of people or expensive/inaccessible hardware. Perhaps connect=
ing directly to miners can be a solution for some people, but I don&#39;t t=
hink it&#39;s healthy for the network.

Some benefits of keeping a mempool as a non-mining node include:
- Fee estimation based on your node&#39;s knowledge of unconfirmed transact=
ions + historical data.
- Dramatically increased block validation (and thus propagation) speed, sin=
ce you cache signature and script results of transactions before they are c=
onfirmed.
- Reduced block relay bandwidth usage (Bitcoin Core nodes use BIP152 compac=
t block relay), as you don&#39;t need to re-request the block transactions =
you already have in your mempool.
- Wallet ability to send/receive transactions that spend unconfirmed output=
s.

&gt; I had the realization that the mempool is obsolete and should be elimi=
nated.

I assume you mean that the mempool should still exist but be turned off for=
 non-mining nodes. A block template producer needs to keep unconfirmed tran=
sactions somewhere.
On Bitcoin Core today, you can use the -blocksonly config option to ignore =
incoming transactions (effectively switching off your mempool), but there a=
re strong reasons not to do this:
- It is trivial for your peers to detect that all transactions broadcasted =
by your node =3D from your wallet. Linking your node to your transactions i=
s a very bad privacy leak.
- You must query someone else for what feerate to put on your transaction.
- You can&#39;t use BIP152 compact block relay, so your network bandwidth u=
sage spikes at every block. You also can&#39;t cache validation results, so=
 your block validation speed slows down.

&gt; Removing the mempool would greatly reduce the bandwidth requirement fo=
r running a node...

If you&#39;re having problems with your individual node&#39;s bandwidth usa=
ge, you can also decrease the number of connections you make or turn off in=
coming connections. There are efforts to reduce transaction relay bandwidth=
 usage network-wide [1].

&gt; Removing the mempool would... keep intentionality of transactions priv=
ate until confirmed/irrevocable...

I&#39;m confused - what is the purpose of keeping a transaction private unt=
il it confirms? Don&#39;t miners still see your transaction? A confirmed tr=
ansaction is not irrevocable; reorgs happen.

&gt; Removing the mempool would... naturally resolve all current issues inh=
erent in package relay and rbf rules.

Removing the mempool does not help with this. How does a miner decide wheth=
er a conflicting transaction is an economically-advantageous replacement or=
 a DoS attack? How do you submit your CPFP if the parent is below the mempo=
ol minimum feerate? Do they already have a different relay/mempool implemen=
tation handling all of these problems but don&#39;t aren&#39;t sharing it w=
ith the rest of the community?

&gt; Initial feerate estimation would need to be based on published blocks,=
 not pending transactions (as this information would no longer be available=
), or from direct interactions with block producers.

There are many reasons why using only published blocks for fee estimates is=
 a flawed design, including:

- The miner of a block can artificially inflate the feerate of the transact=
ions in their mempool simply by including a few of their own transactions t=
hat pay extremely high feerates. This costs them nothing, as they collect t=
he fees.
- A miner constructs a block based on the transactions in their mempool. Yo=
ur transaction&#39;s feerate may have been enough to be included 2 blocks a=
go or a week ago, but it will be compared to the other unconfirmed transact=
ions available to the miner now. They can tell you what&#39;s in their memp=
ool or what the next-block feerate is, but you would be a fool to believe t=
hem.

See also [2],[3].

&gt; Provided the number of block template producing actors remains beneath=
, say 1000, it=E2=80=99d be quite feasible to publish a list of tor endpoin=
ts that nodes can independently  + directly submit their transactions to. I=
n fact, merely allowing users to select their own list of endpoints to use =
alternatively to the mempool would be a low effort starting point for the e=
ventual replacement.

As a thought experiment, let&#39;s imagine we have some public registry of =
mining nodes&#39; tor endpoints and we use it for this secondary direct-to-=
miner transaction relay network. If the registry is maintained (by who?) an=
d accurate (based on whose word?), it is a point of failure for transaction=
 censorship and deanonymization, as well as an additional barrier to becomi=
ng a miner, encouraging centralization.
The other possibility is that the registry is not accurate. In fact, unless=
 the registry requires miners to identify themselves (which others on this =
thread have already pointed out is ill-advised), this should be treated sim=
ilarly to regular addr gossip. We would never automatically trust that the =
entity behind the endpoint provides the service it advertises, is an honest=
 node that won&#39;t simply blackhole our transaction, or even belongs to a=
 Bitcoin node at all.

Best,
Gloria

[1]: <a href=3D"https://arxiv.org/pdf/1905.10518.pdf">https://arxiv.org/pdf=
/1905.10518.pdf</a>
[2]: <a href=3D"https://bitcointechtalk.com/an-introduction-to-bitcoin-core=
-fee-estimation-27920880ad0">https://bitcointechtalk.com/an-introduction-to=
-bitcoin-core-fee-estimation-27920880ad0</a>
[3]: <a href=3D"https://gist.github.com/morcos/d3637f015bc4e607e1fd10d8351e=
9f41">https://gist.github.com/morcos/d3637f015bc4e607e1fd10d8351e9f41</a></=
pre></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_at=
tr">On Tue, Oct 26, 2021 at 8:38 AM lisa neigut via bitcoin-dev &lt;<a href=
=3D"mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfo=
undation.org</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" styl=
e=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);paddin=
g-left:1ex"><div><div dir=3D"auto">Hi all,</div><div dir=3D"auto"><br></div=
><div dir=3D"auto">In a recent conversation with @glozow, I had the realiza=
tion that the mempool is obsolete and should be eliminated.</div><div dir=
=3D"auto"><br></div><div dir=3D"auto">Instead, users should submit their tr=
ansactions directly to mining pools, preferably over an anonymous communica=
tion network such as tor. This can easily be achieved by mining pools runni=
ng a tor onion node for this express purpose (or via a lightning network ex=
tension etc)</div><div dir=3D"auto"><br></div><div dir=3D"auto">Mempools ma=
ke sense in a world where mining is done by a large number of participating=
 nodes, eg where the block template is constructed by a majority of the par=
ticipants on the network. In this case, it is necessary to socialize pendin=
g transaction data to all participants, as you don=E2=80=99t know which par=
ticipant will be constructing the winning block template.</div><div dir=3D"=
auto"><br></div><div dir=3D"auto">In reality however, mempool relay is unne=
cessary where the majority of hashpower and thus block template creation is=
 concentrated in a semi-restricted set.=C2=A0</div><div dir=3D"auto"><br></=
div><div dir=3D"auto">Removing the mempool would greatly reduce the bandwid=
th requirement for running a node, keep intentionality of transactions priv=
ate until confirmed/irrevocable, and naturally resolve all current issues i=
nherent in package relay and rbf rules. It also resolves the recent minimum=
 relay questions, as relay is no longer a concern for unmined transactions.=
</div><div dir=3D"auto"><br></div><div dir=3D"auto">Provided the number of =
block template producing actors remains beneath, say 1000, it=E2=80=99d be =
quite feasible to publish a list of tor endpoints that nodes can independen=
tly =C2=A0+ directly submit their transactions to. In fact, merely allowing=
 users to select their own list of endpoints to use alternatively to the me=
mpool would be a low effort starting point for the eventual replacement.</d=
iv><div dir=3D"auto"><br></div><div dir=3D"auto">On the other hand, removin=
g the mempool would greatly complicate solo mining and would also make Bett=
erHash proposals, which move the block template construction away from a ce=
ntralized mining pool back to the individual miner, much more difficult. It=
 also makes explicit the target for DoS attacks.</div><div dir=3D"auto"><br=
></div><div dir=3D"auto">A direct communication channel between block templ=
ate construction venues and transaction proposers also provides a venue for=
 direct feedback wrt acceptable feerates at the time, which both makes tran=
saction confirmation timelines less variable as well as provides block prod=
ucers a mechanism for (independently) enforcing their own minimum security =
budget. In other words, expressing a minimum acceptable feerate for continu=
ed operation.</div><div dir=3D"auto"><br></div><div dir=3D"auto">Initial fe=
erate estimation would need to be based on published blocks, not pending tr=
ansactions (as this information would no longer be available), or from dire=
ct interactions with block producers.</div><div dir=3D"auto"><br></div><div=
 dir=3D"auto"><br></div><div dir=3D"auto">~niftynei</div>
</div>
_______________________________________________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">=
bitcoin-dev@lists.linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" =
rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail=
man/listinfo/bitcoin-dev</a><br>
</blockquote></div>

--000000000000f5256705cf457bd0--