Date: Thu, 27 Oct 2022 09:52:10 +1000
From: Anthony Towns <aj@erisian.com.au>
To: Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
Message-ID: <Y1nIKjQC3DkiSGyw@erisian.com.au>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Subject: [bitcoin-dev] On mempool policy consistency
Precedence: list

Hi *,

TLDR: Yes, this post is too long, and there's no TLDR. If it's any
consolation, it took longer to write than it does to read?

On Tue, Jun 15, 2021 at 12:55:14PM -0400, Antoine Riard via bitcoin-dev wrote:
> Subject: Re: [bitcoin-dev] Proposal: Full-RBF in Bitcoin Core 24.0
> I'm writing to propose deprecation of opt-in RBF in favor of full-RBF

> If there is ecosystem agreement on switching to full-RBF, but 0.24 sounds
> too early, let's defer it to 0.25 or 0.26. I don't think Core has a
> consistent deprecation process w.r.t to policy rules heavily relied-on by
> Bitcoin users, if we do so let sets a precedent satisfying as many folks as
> we can.

One precedent that seems to be being set here, which to me seems fairly
novel for bitcoin core, is that we're about to start supporting and
encouraging nodes to have meaningfully different mempool policies. From
what I've seen, the baseline expectation has always been that while
certainly mempools can and will differ, policies will be largely the same:

  Firstly, there is no "the mempool". There is no global mempool. Rather
  each node maintains its own mempool and accepts and rejects transaction
  to that mempool using their own internal policies. Most nodes have
  the same policies, but due to different start times, relay delays,
  and other factors, not every node has the same mempool, although they
  may be very similar.

  - https://bitcoin.stackexchange.com/questions/98585/how-to-find-if-two-transactions-in-mempool-are-conflicting

Up until now, the differences between node policies supported by different
nodes running core have been quite small, with essentially the following
options available:

 -minrelaytxfee, -maxmempool - changes the lowest fee rate you'll accept

 -mempoolexpiry - how long to keep txs in the mempool

 -datacarrier - reject txs creating OP_RETURN outputs

 -datacarriersize - maximum size of OP_RETURN data

 -permitbaremultisig - prevent relay of bare multisig

 -bytespersigop - changes how SIGOP accounting works for relay and
 mining prioritisation

as well as these, marked as "debug only" options (only shown with
-help-debug):

 -incrementalrelayfee - make it easier/harder to spam txs by only
 slightly bumping the fee; marked as a "debug only" option

 -dustrelayfee - make it easier/harder to create uneconomic utxos;
 marked as a "debug only" option

 -limit{descendant,ancestor}{count,size} - changes how large the
 transaction chains can be; marked as a "debug only" option

and in theory, but not available on mainnet:

 -acceptnonstdtxn - relay/mine non standard transactions

There's also the "prioritisetransaction" rpc, which can cause you to keep
a low feerate transaction in your mempool longer than you might otherwise.

I think that -minrelaytxfee, -maxmempool and -mempoolexpiry are the only
ones of those options commonly set, and those only rarely result in any
differences in the txs at the top of the mempool.

There are also quite a few parameters that aren't even runtime
configurable:

 - MAX_STANDARD_TX_WEIGHT
 - MIN_STANDARD_TX_NONWITNESS_SIZE (see also #26265)
 - MAX_P2SH_SIGOPS (see also #26348)
 - MAX_STANDARD_TX_SIGOPS_COST
 - MAX_STANDARD_P2WSH_STACK_ITEMS
 - MAX_STANDARD_P2WSH_STACK_ITEM_SIZE
 - MAX_STANDARD_TAPSCRIPT_STACK_ITEM_SIZE
 - MAX_STANDARD_P2WSH_SCRIPT_SIZE
 - MAX_STANDARD_SCRIPTSIG_SIZE
 - EXTRA_DESCENDANT_TX_SIZE_LIMIT
 - MAX_REPLACEMENT_CANDIDATES

And other plausible options aren't configurable even at compile time
-- eg, core doesn't implement BIP 125's inherited signalling rule so
there's no way to enable it; core doesn't allow opting out of BIP 125
rule 3 ratchet on absolute fee; core doesn't allow CPFP carveout with
more than 1 ancestor; core doesn't allow opting out of LOW_S checks
(even via -acceptnonstdtxn); etc.

We also naturally have different mempool policies between different
releases: eg, expansions of policy, such as allowing OP_RETURN or
expanding it from 40 to 80 bytes or new soft forks where old nodes won't
relay transactions that use the new; and also occassional restrictions
in policy, such as the LOW_S requirement.


While supporting and encouraging different mempool polices might be new
for core, it's not new for knots: knots changes some of these defaults
(-permitbaremultisig defaults to false, -datacarriersize is reduced to
42), allows the use of -acceptnonstdtxn on mainnet, and introduces new
options including -spkreuse and -mempoolreplacement (giving the latter
full rbf behaviour by default). Knots also includes a `-corepolicy`
option to make it easy to get a configuration matching core's defaults.


I think gmaxwell's take from Feb 2015 (in the context of how restrictive
policy on OP_RETURN data should be) was a reasonable description for
core's approach up until now:

  There is also a matter of driving competent design rather than lazy
  first thing that works. E.g. In stealth addresses the early proposals
  use highly inefficient single ECDH point per output instead of simply
  pooling them. Network behavior is one of the few bits of friction
  driving good technical design rather than "move fast, break things, and
  force everyone else onto my way of doing thing rather than discussing
  the design in public". No one wants to be an outright gatekeeper,
  but the network is a shared resource and it's perfectly reasonable
  node behavior to be stingy about the perpetual storage impact of the
  transactions they're willing to process, especially when it comes to
  neutral technical criteria like the amount of network irrelevant data
  stuffed in transactions.

  There is also a very clear pattern we've seen in the past where
  people take anything the system lets them do as strong evidence that
  they have a irrevocable right to use the system in that way, and that
  their only responsibility-- and if their usage harms the system it's
  the responsibility of the system to not permit it. [...
  ...] For mitigating these risks it's optimal if transactions
  seem as uniform and indistinguishable as reasonably possible.

  - https://github.com/bitcoin/bitcoin/pull/5286#issuecomment-72564175

Perhaps see also sdaftuar in Nov 2015,

  To me the most important question is, is priority something that miners
  want to use?

  If a non-negligible amount of hashpower intends to use it in their
  transaction selection, then I think it makes sense for nodes to use it
  too, because it's generally helpful to have your mempool predict the
  UTXO as much as possible, and for nodes to be able to have reasonable
  fee and priority estimates (which won't happen unless they track the
  priority transactions somehow -- I'm presuming that miners run with
  much bigger mempools than regular nodes).

  If the answer is no, then that's fine and I don't see a reason to push
  in this direction. I sort of assumed there was enough hashpower mining
  with priority, since last time I checked estimatepriority was still
  giving meaningful results for low-ish blockheights, but I haven't done
  any kind of real analysis.

  - https://github.com/bitcoin/bitcoin/pull/6992#issuecomment-155969455

or in June 2019,

  What this PR is proposing is to get rid of a command-line option that is
  (a) a footgun for users and (b) does not reflect what I believe to be
  the understanding most users have, which is that [X txs] are expected
  to propagate well on the network.

  ..

  I don't think this rises to the level that Luke is concerned about,
  namely a prelude to forcing a common relay policy on all nodes. In
  particular I do agree it makes sense that we offer some ways of
  customizing policy parameters (eg the mempool size, min relay fee,
  etc). Instead, I think the justification for this change is that we
  should not support behaviors we think are harmful to the ecosystem
  overall and have no legitimate use-case, and we should eliminate ways
  that users might inadvertently shoot themselves in the foot.

  - https://github.com/bitcoin/bitcoin/pull/16171#issuecomment-500393271

(or see discussion in https://github.com/bitcoin/bitcoin/pull/7219)

I don't mean to imply the above are saying "there's one way to do
things and it's this way", or that the old way of doing things should
necessarily be the way we keep doing things. Just that previously core
has tended towards designing a single policy that works as well as it
can for everyone and the ecosystem as a whole. (I'm also not saying that
fullrbf can't work well for everyone or the ecosystem as a whole)


By contrast, I think the most common response to pushback against the
full rbf option has been along the lines of "it's just an option, we
don't want to force people", eg:

  Blaming the default false -mempoolfullrbf option for a full RBF network
  would be holding Bitcoin Core developers responsible for the decisions
  of individual node operators and miners. I don't think having the
  option (again, default false) can directly cause a full RBF network,
  and likewise, I don't think removing this option removes the "risk"
  of a full RBF network.
   - glozow
     https://github.com/bitcoin/bitcoin/pull/26287#issuecomment-1274949400

  NACK. This is a default false option.
   - achow101
     https://github.com/bitcoin/bitcoin/pull/26287#issuecomment-1274953204

  Erecting artificial barriers to prevent or make it difficult for users
  to do what they want to do, is not appropriate behaviour.
   - luke-jr
     https://github.com/bitcoin/bitcoin/pull/26287#issuecomment-1290721905

  I'm in general against removing options.
   - instagibbs
     https://github.com/bitcoin/bitcoin/pull/26287#issuecomment-1292030700

I think this differs from what core has done in the past, in that
previously we've tried to ensure a new policy is good for everyone (or as
nearly as it can be), and then enabled it as soon as it's implemented.
Any options that have been added have either been to control resource
usage in ways that don't significantly effect tx propagation, to
allow people to revert to the old behaviour when the new behaviour is
controversial (eg the -mempoolreplacement=0 option from 0.12 to 0.18),
and to make it easier to test/debug the implementation.

Giving people a new relay behaviour they can opt-in to when we aren't
confident enough to turn on by default doesn't match the approach I've
seen core take in the past.


If this is going to be an ongoing shift in how core sees relay/mempool
policy, I think that's significant and worth paying attention to.

I don't think it's necessary to have that shift to roll out full rbf.
The other approach would be either:

 * set -mempoolfullrbf=true as the default for 24.0, and just have the
   command line param there in case people want to do a
   "UserRejectedMempoolPolicy" campaign to get everyone to opt-out

 * revert it for now because we don't think mainnet is ready for fullrbf
   yet, and introduce it as default true for 25.0 or 26.0 or 27.0 or
   to activate at some scheduled date in that timeframe (potentially
   backporting it to previous releases to help with adoption too,
   whatever). same effect as the previous option, just with a bit more
   advanced notice and time to prepare

I don't think anyone's proposed the first (which I interpret as "most of
us don't think mainnet is ready for fullrbf today"), but the comments
above are all pushback by people arguing against (the first step of)
the second approach, and they seem to be winning the day.

It's also possible that this is something of a one time thing: full rbf
has been controversial for ages, but widely liked by devs, and other
attempts (eg making it available in knots) haven't actually achieved
much of a result in practice. So maybe this is just a special case and
not a precedent, and when people propose other default false options,
there will be substantially more resistance to them being merged,
despite all the talk about users having options that's going on right now.


Assuming it is the change of direction it appears to be -- and all of
the above is really just justification for that assumption -- then like
I said, I think it's worth seriously considering what it means for people
to choose their own relay/mempool policies and for you to expect to have
different mempool policies to many or most of your potential peers.


One thing maybe worth noting is that is that you can still only choose
your policy from options that people write code for -- if it wasn't
something you could get by running knots or compiling a rejected PR
yourself, it won't magically become more possible now.  Presumably it
would mean that once a PR is written, it might get better review (rather
than being dismissed as not suitable for everyone), and there would be
less maintenance burden than if it had to be manually rebased every
release, though (or at least the maintenance burden would be shared
across everyone working on the codebase).


The second thing is that whatever your relay policy is, you still
need a path all the way to miners through nodes that will accept your
transaction at every step. If you're making your mempool more restrictive
(eg -permitbaremultisig=0, -datacarrier=0), that's easy for you (though
you're making life more difficult for people who do create those sorts
of txs); but if you want a more permissive policy (package relay,
version-3-rbf, full-rbf), you might need to do some work.

The cutoff for that is probably something like "do 30% of listening
nodes have a compatible policy"? If they do, then you'll have about a
95% chance of having at least one of your outbound peers accept your tx,
just by random chance. If erlay allows increasing your outbound count to
12 connections instead of 8; that might reduce down to needing just 20%
of listening nodes (~93%).

But for cases where less than 30% (20%) of network supports your preferred
policy, you probably need to do something cleverer.

One approach is to set a service bit and preferentially peer with other
nodes that advertise that service bit; knots does the first half of this
for fullrbf, and both halves have been proposed for core in #25600.
Preferential peering was previously done for the segwit deployment,
though in that case it was necessary not just for tx propogation but
also for ensuring block propogation, making it effectively a consensus
critical issue.

Another approach is having a separate relay network -- eg, lightning nodes
already have a gossip network, and might want to help their own ecosystem
by ensuring unilateral channel closes and justice transactions are quickly
relayed. Using their own gossip network to relay the transaction around,
and each lightning node adding it to their local bitcoind's mempool and
allowing it to propogate (or not) from there as normal, would also be a
way of allowing transactions to propogate well. It does mean that miners
would either need to also participate in lightning gossip directly, or
that miners would need to connect to *many* peers to be confident of
seeing those transactions (eg, if only 2% of the network would see a
tx, you'd need to make 228 connections to have a 99% chance of seeing
the tx). You can't currently do something like this, because all the
relay policies are also applied when adding txs to the mempool via RPC,
and there's no convenient way to remove txs from the mempool.

A case where something like that might occur is in preventing L2
transactions from pinning attacks -- so you might have a high-fee,
low-feerate transaction that's been widely propogated, sitting in the
bottom of people's mempools, and you want to replace it with a smaller,
higher-feerate transaction, but don't want to pay a higher absolute fee,
and are thus blocked by BIP 125 rule 3. Perhaps 98% of the network is
unwilling to deviate from BIP 125 rule 3 for you; because that would
make it easy for random griefers to spam their mempool with large txs
then delete them while only paying a small fee; but your L2 peers may be
able to decode your replacement transaction and be sure that you aren't
going to spam them, and thus will happily relay it.

From a technical point-of-view, that's largely fine; the downside is it
increases the centralisation pressure on mining: whether that's by having
to connect to substantially more nodes, or having to parse through more
spam, you can't just run your mining operation off a standard install
of bitcoin core anymore, but need to actively opt-in to find all the
weird unusual ways people are sending transactions around in order to
actually collect as much in fees as your competitors are.

That's probably moderately bad for privacy as well -- if lightning or
coinjoins need special relay rules that most nodes haven't opted into,
it's potentially easy to use that to find the bitcoin nodes on the
network that are participating in those protocols, and from there to
either identify the operator, or run a DoS attack to make it hard for you
to keep doing what you want. Obviously if you're setting a service bit to
get better routing, you've given up that privacy already. Likewise if the
government or random vandals are opposed to bitcoin mining, and miners
have to have special configuration on their nodes that distinguish them
from regular users, then perhaps that makes it easier to find or shut
down their operations.

There are a few efficiencies to be gained from similar mempool policies as
well: more reliable compact block reconstruction (if you're not missing
any transactions, you avoid a round-trip) and presumably more efficient
set reconstruction with erlay. You'll also waste less bandwidth sending
transactions that the other node is only going to reject. Both those
depend on how many transactions are going to rely on unusual mempool
policies in the first place though.

ariard wrote:

  I know I've advocated in the past to turn RBF support by default in
  the past. Though after gathering a lot of feedbacks, this approach
  of offering the policy flexiblity to the interested users only and
  favoring a full-rbf gradual deployment sounds better to me.

  - https://github.com/bitcoin/bitcoin/pull/25353#issuecomment-1157137026

I guess all the above leads me to think that gradual deployments of
mempool policies are likely the worse approach: even when they're not
hurting anyone, it makes them hard to use during the gradual phase,
and getting around that comes with worrying compromises on privacy and
centralisation; and when they are problematic for some, the indeterminate
nature of a gradual deployment means it's hard to plan for when that
risk is going to eventuate.


Theoretically, one way to recover the good parts of core deciding on
what's good for the network might be for people outside of core to
recommend a mempool configuration; then core can just have an option
to make that easy, similar to "-std=c++17" for a C++ compiler, and much
the same as knots' "-corepolicy" option.

Presuming anyone actually wants to take on that job, and listen to the
concerns of zeroconf businesses, lightning and coinjoin devs, miners, etc;
and can come up with something that keeps most of them happy, and that
70% or 90% of the network ends up just following those recommendations
because it's easy, it works, and it's recommended by all the apps they
want to use, then that could work great:

 * miners don't need to do anything special, so there's no new
   mining centralisation pressure
 * miners and users don't reveal what they're doing with bitcoin by the way
   they configure their nodes, so there's no privacy problems
 * devs can be fairly confident in how they have to design their apps
   in order to get their transactions to most hashpower
 * devs don't have to add new p2p layers to make it happen
 * at least there's someone to talk to when you're trying to figure out
   how to make some new project possible when it's inhibited by current
   relay policies and you don't have to try to convince everyone to
   upgrade on your own
 * core devs just provide options, and don't have to worry about being
   seen as gatekeepers

The "downside" in that scenario is that users/dev aren't making much
actual use of all the choices core is offering by making different
options available; but the upside is that that choice is at least readily
available should whoever is coming up with these policy become out of
step with what people actually want.

One thing that might make an approach like that difficult is that core
has historically been happy to remove options that don't seem useful
anymore: eg the ability to turn of BIP 125 support (#16171), and priority
transactions (#9602). Perhaps that's fine if you're trying to actively
craft a single mempool/relay policy that's good enough for almost everyone
(after all, it makes the code simpler and more efficient, and reduces
the number of footguns); all you're doing is leaving a minority of people
who want weird things to run a fork, and that's going to happen anyway.

But if people are following policy developed outside of core, core
might well disagree with them and decide "no that's a stupid policy,
no one should do that" and remove some feature that others thing should
continue to be normal. Beyond the examples above, there's already talk of
removing the ability to disable fullrbf support in #26305, for instance.
If that happens, then the people maintaining the policy will instead
end up maintaining an entire fork of bitcoin core, and all we've done
is transition to people running software from a different repo, and a
different set of maintainers.

If we're really going to a world where core's eager to add new options,
and reluctant to remove them, at least if anyone at all finds them
interesting, that's presumably a non-issue, though.

Cheers,
aj