summaryrefslogtreecommitdiff
path: root/ea/72643bf55f3b81cd2486df43786ad6bcbdf6fb
blob: 6147386bf8899e2d51833a5fcc14ace7a7e7ac20 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
Return-Path: <gloriajzhao@gmail.com>
Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136])
 by lists.linuxfoundation.org (Postfix) with ESMTP id 1F12CC0012
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Mon, 14 Mar 2022 10:29:32 +0000 (UTC)
Received: from localhost (localhost [127.0.0.1])
 by smtp3.osuosl.org (Postfix) with ESMTP id F1D7F60D9C
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Mon, 14 Mar 2022 10:29:31 +0000 (UTC)
X-Virus-Scanned: amavisd-new at osuosl.org
X-Spam-Flag: NO
X-Spam-Score: -2.098
X-Spam-Level: 
X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5
 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
 DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: smtp3.osuosl.org (amavisd-new);
 dkim=pass (2048-bit key) header.d=gmail.com
Received: from smtp3.osuosl.org ([127.0.0.1])
 by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id 5o2yHOLpQL96
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Mon, 14 Mar 2022 10:29:29 +0000 (UTC)
X-Greylist: whitelisted by SQLgrey-1.8.0
Received: from mail-yw1-x112c.google.com (mail-yw1-x112c.google.com
 [IPv6:2607:f8b0:4864:20::112c])
 by smtp3.osuosl.org (Postfix) with ESMTPS id B0A6360D4E
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Mon, 14 Mar 2022 10:29:28 +0000 (UTC)
Received: by mail-yw1-x112c.google.com with SMTP id
 00721157ae682-2dbfe58670cso157037607b3.3
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Mon, 14 Mar 2022 03:29:28 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to
 :cc; bh=/fxR1SeMCpd4grS8pvMxA8IFfAsqkSWu9bMoW8fAZI4=;
 b=RlF6DjFFONZOq2VjyTkjQHUjlIpT92z7Fv8fHoYNxlt4hhfT4X9DIxichUwvv6Oo5U
 r60onvK16/qBZ9lyfMemvG8IctHAPC7XlrCK4hopebYMRe0OWk8VwEaFcHQ+dr44UeJY
 WVjZKXm1KhGcoRlpj4BsJ5orYMVFMFAWnV+P+7nkGeWi4ANANyf8qMbtKeCYEyadlOQu
 aYEmXLWvybUEvN8IpOUWuOwVaQGViEnsC96woip9aVdYGAYdiyjWR3x8oj1qRvQSTls5
 u1lai5CtQNXSmrLtfEhNpm2Y67wlfGR2jtKNJHtySxnB1DIuR1zdezDvkQKoC5T8OQqM
 RTPg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to:cc;
 bh=/fxR1SeMCpd4grS8pvMxA8IFfAsqkSWu9bMoW8fAZI4=;
 b=cQcfKmbBeXGx+f2XMSFcfJ4idjDIEFtzwgR4XPhVbsKuY0q34YqVtK/xTIc8YuxcpK
 JijKRgOeBYGlcu4W46j8SjGPpjEwBJqrXO8MvSlFrqbWjacVpvDJpgXH8jNIjHM4qe6j
 MbipGfkQIIUSbPF8Ijp1ouYv8s/lT8NapRPAaBOHpREJJh7P2WkYQZNx+BfHtEUWyYl/
 frMRsrbMjGHZpUneoHVkK6Jvbqlf/K5/X8qEshdtAjjYw+jw1gtJxKsy1FnW7LYybVB+
 TCh7A4pPYD15I0wlco08Cb5R2EuK0Xe5o/JVc0Vzm/qg1naI3DiS7PBVwkYNf8BUFyTD
 OZtA==
X-Gm-Message-State: AOAM5301oqmyQLnQcU/tev0Cl7wIoBPwKSSuuegkItNO0m02gJL3Db80
 QGLGTf1qYbEjaAJ2Oy0C12V95XxVN9DGmSYV9hEaVf3mnIE=
X-Google-Smtp-Source: ABdhPJzAn2PoHYKre3/grdS8WCA2b+YwDRBWs7+p6jFmUZqDXdr/lymRapFSSfkYNDQd2He2JSma5s9DI/gxxr1GYJg=
X-Received: by 2002:a0d:fe85:0:b0:2e5:7500:761f with SMTP id
 o127-20020a0dfe85000000b002e57500761fmr253142ywf.1.1647253767440; Mon, 14 Mar
 2022 03:29:27 -0700 (PDT)
MIME-Version: 1.0
References: <CAFXO6=LGbaur6XQrE+6a6mAAHXduOCXoWPTgPosxAG59ZkK6Gg@mail.gmail.com>
 <CALZpt+EjqKbhnN_5jy3kvYpMvjN8=iwRzMLSM7yS8_j-WzLrBQ@mail.gmail.com>
 <CACdvm3P1co1HDFKNxpHRe_JX_UPNw_P5qgL5cHCM=Qs+kR=B_A@mail.gmail.com>
 <GlEfqW7mh2W3uHkxDxwb5RSj-O_zbTUi4wa67oRz3erHRM1ykxT0BrcJrqulCOqrRLVJ4Bp8KVSOj0yJGB7rwcFGlZDyMrTsndPFO89hAQc=@protonmail.com>
 <CACdvm3P_-1DPxcWkd1J-PckPF1oRTtVB5zz5e3+VQ0Mko1T=hQ@mail.gmail.com>
 <CAFXO6=+WFUueqDh21NTZzA5EcSQjX2owFn0+dr0ua_BRLfV4QQ@mail.gmail.com>
 <20220208045850.GA6538@erisian.com.au>
 <CAFXO6=KMveswFvYdFCjsvt7a-Af+act4K3p8UrJXGyBO8E1o+w@mail.gmail.com>
 <CAGpPWDY5W8G8je7yQRPF12PtVGeaZ9Pi98LacjrAs+RGEWqv_w@mail.gmail.com>
 <CAGpPWDY8rrUp0CmKm6Bf2dGnj2S8JJjRHD-dhn8LDxidbwV2Ug@mail.gmail.com>
In-Reply-To: <CAGpPWDY8rrUp0CmKm6Bf2dGnj2S8JJjRHD-dhn8LDxidbwV2Ug@mail.gmail.com>
From: Gloria Zhao <gloriajzhao@gmail.com>
Date: Mon, 14 Mar 2022 10:29:16 +0000
Message-ID: <CAFXO6=+i4ad9b-pC-ZtchmPqeQjsmj+CUyWrO3dx2p3ub9DxsQ@mail.gmail.com>
To: Billy Tetrud <billy.tetrud@gmail.com>
Content-Type: multipart/alternative; boundary="000000000000a9cb0405da2b27bd"
X-Mailman-Approved-At: Mon, 14 Mar 2022 10:33:50 +0000
Cc: Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>,
 Anthony Towns <aj@erisian.com.au>
Subject: Re: [bitcoin-dev] Improving RBF Policy
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>, 
 <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>, 
 <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Mon, 14 Mar 2022 10:29:32 -0000

--000000000000a9cb0405da2b27bd
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Billy,

> We should expect miners will be using a more complex, more optimal way of
determining what blocks they're working on [...] we should instead run with
the assumption that miners keep all potentially relevant transactions in
their mempools, including potentially many conflicting transctions, in
order to create the most profitable blocks. And therefore we shouldn't put
the constraint on normal non-mining full nodes to do that same more-complex
mempool behavior or add any complexity for the purpose of denying
transaction replacements.

> I think a lot of the complexity in these ideas is because of the attempt
to match relay rules with miner
inclusion rules.

I think the assumption that miners are using a completely different
implementation of mempool and block template building is false. IIUC, most
miners use Bitcoin Core and perhaps configure their node differently (e.g.
larger mempool and different minfeerates), but also use `getblocktemplate`
which means the same ancestor package-based mining algorithm.

Of course, I'm not a miner, so if anybody is a miner or has seen miners'
setups, please correct me if I'm wrong.

In either case, we would want our mining algorithm to result in block
templates that are as close as possible to perfectly incentive compatibile.

Fundamentally, I believe default mempool policy (which perhaps naturally
creates a network-wide transaction relay policy) should be as close to the
mining code as possible. Imagine node A only keeps 1 block's worth of
transactions, and node B keeps a (default) 300MB mempool. The contents of
node A's mempool should be as close as possible to a block template
generated from node B's mempool. Otherwise, node A's mempool is not very
useful - their fee estimation is flawed and compact block relay won't do
them much good if they need to re-request a lot of block transactions.
Next, imagine that node B is a miner. It would be very suboptimal if the
mining code was ancestor package-based (i.e. supports CPFP), but the
mempool policy only cared about individual feerate, and evicted low-fee
parents despite their high-fee children. It's easy to see why this would be
suboptimal.
Attempting to match mempool policy with the mining algorithm is also
arguably the point of package relay. Our mining code uses ancestor packages
which is good, but we only submit transactions one at a time to the
mempool, so a transaction's high-fee children can't be considered until
they are all already in the mempool. Package relay allows us to start
thinking about ancestor packages immediately when evaluating transactions
for submission to the mempool.

The attempt to match policy with miner inclusion rules is deliberate and
necessary.

> I want to echo James O'Beirne's opinion on this that this may be the
wrong path to go down (a path of more complexity without much gain). He
said: "Special consideration for "what should be in the next block" and/or
the caching of block templates seems like an imposing dependency, dragging
in a bunch of state and infrastructure to a question that should be solely
limited to mempool feerate aggregates and the feerate of the particular txn
package a wallet is concerned with."

It seems that I under-explained the purpose of building/caching block
templates in my original post, since both you and James have the same
misunderstanding. Since RBF's introduction, we have improved to an ancestor
package-based mining algorithm. This supports CPFP (incentive compatible)
and it is now common to see more complex "families" of transactions as
users fee-bump transactions (market is working, yay). On the other hand, we
no longer have an accurate way of determining a transaction's "mining
score," i.e., the feerate of this transaction's ancestor package when it is
included in a block template using our current mining algorithm.

This limitation is a big blocker in proposing new fee/feerate RBF rules.
For example, if we say "the transaction needs a better feerate," this is
obviously flawed, since the original transactions may have very
high-feerate children, and the replacement transaction may have low feerate
parents. So what we really want is "the transaction needs to be more
incentive compatible to mine based on our mining algorithm," but we have no
way of getting that information right now.

In my original post, I [described 4 heuristics to get transaction's "mining
score"][1] using the current data cached in the mempool (e.g. max ancestor
feerate of descendant set), as well as why they don't work. As such, the
best way to calculate a transaction's mining score AFAICT is to grab all of
the related transactions and build a mini "block template" with them. The
[implementation][2] I sent last week also cuts out some of the fluff, so
the pseudocode looks like this:

// Get ALL connected entries (ancestors, descendants, siblings, cousins,
coparents, etc.)
vector<TxMempoolEntry> cluster =3D mempool.GetAllTransactionsRelatedTo(txid=
);
sort(cluster, ancestorfeerate);

// For deducting ancestors when they are included separately
vector<ModifiedTxMempoolEntry> modified_entries;

while (!cluster.empty() and !modified_entries.empty()) {
    iter =3D BetterAncestorFeerateOf(cluster, modified_entries);
    best_ancestor_package =3D GetAncestorSet(iter);
    mining_score =3D Feerate(best_ancestor_package);
    for (entry : best_ancestor_package) {
       mining_scores[entry] =3D mining_score;
       for (descendant : GetAllDescendants(entry) {
           modified_entries[descendant].DeductAncestorFromAccounting(entry)=
;
       }
    }
}

Perhaps somebody will come up with a better way to do this, but this is my
solution, and I hope I've now sufficiently described why I've made an
effort to think about this. It's not because I want to make things more
fancy or complicated, but because we currently have no way of accurately
determining a transaction's mining score. The reason I proposed a
persistent block template is so we can efficiently query the mining score
of all transactions in the mempool.

> However, I do think that these DOS concerns are quite overblown. I wrote
up a comment on your rbf-improvements.md
<https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_=
comment_id=3D4093100#gistcomment-4093100>
detailing
my thought process on that. The summary is that as long as the fee-rate
relay rule is maintained, any "spam" is actually paid for, either by the
"first" transaction in the spam chain, or by the "spam" itself.

The DoS concern is not overblown. I recommend you re-read the [current RBF
policy][3]; Rule 3 and 4 *are* the feerate relay rules. Removing Rule 3
means allowing recycled fees, so new transactions are not necessarily
"paying" anything.

[1]:
https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_c=
omment_id=3D4093100#mining-score-of-a-mempool-transaction
[2]: https://github.com/glozow/bitcoin/tree/2022-02-mining-score
[3]:
https://github.com/bitcoin/bitcoin/blob/master/doc/policy/mempool-replaceme=
nts.md

Best,
Gloria

On Sat, Mar 12, 2022 at 8:18 AM Billy Tetrud <billy.tetrud@gmail.com> wrote=
:

> In reading through more of the discussion, it seems the idea I presented
> above might basically be a reformulation of t-bast's rate-limiting idea
> presented in this comment
> <https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalin=
k_comment_id=3D4081349#gistcomment-4081349>.
> Perhaps he could comment on whether that characterization is accurate.
>
> On Fri, Mar 11, 2022 at 10:22 AM Billy Tetrud <billy.tetrud@gmail.com>
> wrote:
>
>> Hi Gloria,
>>
>> >  1. Transaction relay rate limiting
>>
>> I have a similar concern as yours, that this could prevent higher
>> fee-rate transactions from being broadcast.
>>
>> > 2. Staggered broadcast of replacement transactions: within some time
>> interval, maybe accept multiple replacements for the same prevout, but o=
nly
>> relay the original transaction.
>>
>> By this do you mean basically having a batching window where, on
>> receiving a replacement transaction, a node will wait for a period of ti=
me,
>> potentially receiving many replacements for the same transaction (or man=
y
>> separate conflicting transactions), and only broadcasting the "best" one=
(s)
>> at the end of that time window?
>>
>> Its an interesting idea, but it would produce a problem. Every hop that
>> replacement transaction takes would be delayed by this staggered window.=
 If
>> the window were 3 minutes and transactions generally take 20 hops to get=
 to
>> the majority of miners, a "best-case average" delay might be 3.75 minute=
s
>> (noting that among your 8 nodes, its quite likely one of them would have=
 a
>> window ending much sooner than 3 minutes). Some (maybe 3% of) nodes woul=
d
>> experience delays of more than 20 minutes. That kind of delay isn't grea=
t.
>>
>> However it made me think of another idea: a transaction replacement
>> broadcast cooldown. What if nodes kept track of the time they broadcaste=
d
>> the last replacement for a package and had a relay cooldown after the la=
st
>> replacement was broadcasted? A node receiving a replacement would relay =
the
>> replacement immediately if the package its replacing was broadcasted mor=
e
>> than X seconds ago, and otherwise it would wait until the time when that
>> package was broadcasted at least X seconds ago to broadcast it. Any
>> replacements it receives during that waiting period would replace as
>> normal, meaning the unrebroadcasted replacement would never be
>> broadcasted, and only the highest value replacement would be broadcasted=
 at
>> the end of the cooldown.
>>
>> This wouldn't prevent a higher-fee-rate transaction from being
>> broadcasted (like rate limiting could), but would still be effective at
>> limiting unnecessary data transmission. Another benefit is that in the
>> non-adversarial case, replacement transactions wouldn't be subject to an=
y
>> delay at all (while in the staggered broadcast idea, most replacements
>> would experience some delay). And in the adversarial case, where a
>> malicious actor broadcasts a low-as-possible-value replacement just befo=
re
>> yours, the worst case delay is just whatever the cooldown period is. I
>> would imagine that maybe 1 minute would be a reasonable worst-case delay=
.
>> This would limit spam for a transaction that makes it into a block to ~1=
0x
>> (9 to 1). I don't see much of a downside to doing this beyond just the
>> slight additional complexity of relay rules (and considering it could sa=
ve
>> substantial additional code complexity, even that is a benefit).
>>
>> All a node would need to do is keep a timestamp on each transaction they
>> receive for when it was broadcasted and check it when a replacement come=
s
>> in. If now-broadcastDate < cooldown, set a timer for cooldown -
>> (now-broadcastDate) to broadcast it. If another replacement comes in, cl=
ear
>> that timer and repeat using the original broadcast date (since the
>> unbroadcast transaction doesn't have a broadcast date yet).
>>
>> I think it might also be useful to note that eliminating "extra data"
>> caused by careless or malicious actors (spam or whatever you want to cal=
l
>> it) should not be the goal. It is impossible to prevent all spam. What w=
e
>> should be aiming for is more specific: we should attempt to design a sys=
tem
>> where spam is manageable. Eg if our goal is to ensure that a bitcoin nod=
e
>> uses no more than 10% of the bandwidth of a "normal" user, if current
>> non-spam traffic only requires 1% of a "normal" users's bandwidth, then =
the
>> network can bear a 9 to 1 ratio of spam. When a node spins up, there is =
a
>> lot more data to download and process. So we know that all full nodes ca=
n
>> handle at least as much traffic as they handle during IBD. What's the
>> difference between those amounts? I'm not sure, but I would guess that I=
BD
>> is at least a couple times more demanding than a fully synced node. So I
>> might suggest that as long as spam can be kept below a ratio of maybe 2 =
to
>> 1, we should consider the design acceptable (and therefore more complexi=
ty
>> unnecessary).
>>
>> The 1 minute broadcast cooldown I mentioned before wouldn't be quite
>> sufficient to achieve that ratio. But a 3.33 minute cooldown would be.
>> Whether this is "too much" is something that would have to be discussed,=
 I
>> suspect a worst-case adversarial 3.33 minute delay would not be "too muc=
h".
>> Doing this could basically eliminate any risk of actual service denial v=
ia
>> replacement transactions.
>>
>> However, I do think that these DOS concerns are quite overblown. I wrote
>> up a comment on your rbf-improvements.md
>> <https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permali=
nk_comment_id=3D4093100#gistcomment-4093100> detailing
>> my thought process on that. The summary is that as long as the fee-rate
>> relay rule is maintained, any "spam" is actually paid for, either by the
>> "first" transaction in the spam chain, or by the "spam" itself. Even
>> without something like a minimum RBF relay delay limiting how much spam
>> could be created, the economics of the fee-rate rule already sufficientl=
y
>> mitigate the issue of spam.
>> On Wed, Mar 9, 2022 at 9:37 AM Gloria Zhao via bitcoin-dev <
>> bitcoin-dev@lists.linuxfoundation.org> wrote:
>>
>>> Hi RBF friends,
>>>
>>> Posting a summary of RBF discussions at coredev (mostly on transaction
>>> relay rate-limiting), user-elected descendant limit as a short term
>>> solution to unblock package RBF, and mining score, all open for feedbac=
k:
>>>
>>> One big concept discussed was baking DoS protection into the p2p level
>>> rather than policy level. TLDR: The fees are not paid to the node opera=
tor,
>>> but to the miner. While we can use fees to reason about the cost of an
>>> attack, if we're ultimately interested in preventing resource exhaustio=
n,
>>> maybe we want to "stop the bleeding" when it happens and bound the amou=
nt
>>> of resources used in general. There were two main ideas:
>>>
>>> 1. Transaction relay rate limiting (i.e. the one you proposed above or
>>> some variation) with a feerate-based priority queue
>>> 2. Staggered broadcast of replacement transactions: within some time
>>> interval, maybe accept multiple replacements for the same prevout, but =
only
>>> relay the original transaction.
>>>
>>> Looking to solicit feedback on these ideas and the concept in general.
>>> Is it a good idea (separate from RBF) to add rate-limiting in transacti=
on
>>> relay? And is it the right direction to think about RBF DoS protection =
this
>>> way?
>>>
>>> A lingering concern that I have about this idea is it would then be
>>> possible to impact the propagation of another person=E2=80=99s transact=
ion, i.e.,
>>> an attacker can censor somebody=E2=80=99s transaction from ever being a=
nnounced by
>>> a node if they send enough transactions to fill up the rate limit.
>>> Obviously this would be expensive since they're spending a lot on fees,=
 but
>>> I imagine it could be profitable in some situations to spend a few thou=
sand
>>> dollars to prevent anyone from hearing about a transaction for a few ho=
urs.
>>> This might be a non-issue in practice if the rate limit is generous and
>>> traffic isn=E2=80=99t horrendous, but is this a problem?
>>>
>>> And if we don't require an increase in (i.e. addition of "new") absolut=
e
>>> fees, users are essentially allowed to =E2=80=9Crecycle=E2=80=9D fees. =
In the scenario
>>> where we prioritize relay based on feerate, users could potentially be
>>> placed higher in the queue, ahead of other users=E2=80=99 transactions,=
 multiple
>>> times, without ever adding more fees to the transaction. Again, maybe t=
his
>>> isn=E2=80=99t a huge deal in practice if we set the parameters right, b=
ut it seems=E2=80=A6
>>> not great, in principle.
>>>
>>> ---------
>>>
>>> It's probably also a good idea to point out that there's been some
>>> discussion happening on the gist containing my original post on this th=
read
>>> (https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff).
>>>
>>> Suhas and Matt [proposed][0] adding a policy rule allowing users to
>>> specify descendant limits on their transactions. For example, some nth =
bit
>>> of nSequence with nVersion 3 means "this transaction won't have more th=
an X
>>> vbytes of descendants" where X =3D max(1000, vsizeof(tx)) or something.=
 It
>>> solves the pinning problem with package RBF where the attacker's packag=
e
>>> contains a very large and high-fee descendant.
>>>
>>> We could add this policy and deploy it with package RBF/package relay s=
o
>>> that LN can use it by setting the user-elected descendant limit flag on
>>> commitment transactions. (Otherwise package RBF is blocked until we fin=
d a
>>> more comprehensive solution to the pinning attack).
>>>
>>> It's simple to [implement][1] as a mempool policy, but adds some
>>> complexity for wallets that use it, since it limits their use of UTXOs =
from
>>> transactions with this bit set.
>>>
>>> ---------
>>>
>>> Also, coming back to the idea of "we can't just use {individual,
>>> ancestor} feerate," I'm interested in soliciting feedback on adding a
>>> =E2=80=9Cmining score=E2=80=9D calculator. I've implemented one [here][=
2] which takes the
>>> transaction in question, grabs all of the connected mempool transaction=
s
>>> (including siblings, coparents, etc., as they wouldn=E2=80=99t be in th=
e ancestor
>>> nor descendant sets), and builds a =E2=80=9Cblock template=E2=80=9D usi=
ng our current
>>> mining algorithm. The mining score of a transaction is the ancestor fee=
rate
>>> at which it is included.
>>>
>>> This would be helpful for something like ancestor-aware funding and
>>> fee-bumping in the wallet: [3], [4]. I think if we did the rate-limited
>>> priority queue for transaction relay, we'd want to use something like t=
his
>>> as the priority value. And for RBF, we probably want to require that a
>>> replacement have a higher mining score than the original transactions. =
This
>>> could be computationally expensive to do all the time; it could be good=
 to
>>> cache it but that could make mempool bookkeeping more complicated. Also=
, if
>>> we end up trying to switch to a candidate set-based algorithm for minin=
g,
>>> we'd of course need a new calculator.
>>>
>>> [0]:
>>> https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permali=
nk_comment_id=3D4058140#gistcomment-4058140
>>> [1]: https://github.com/glozow/bitcoin/tree/2022-02-user-desclimit
>>> [2] https://github.com/glozow/bitcoin/tree/2022-02-mining-score
>>> [3]: https://github.com/bitcoin/bitcoin/issues/9645
>>> [4]: https://github.com/bitcoin/bitcoin/issues/15553
>>>
>>> Best,
>>> Gloria
>>>
>>> On Tue, Feb 8, 2022 at 4:58 AM Anthony Towns <aj@erisian.com.au> wrote:
>>>
>>>> On Mon, Feb 07, 2022 at 11:16:26AM +0000, Gloria Zhao wrote:
>>>> > @aj:
>>>> > > I wonder sometimes if it could be sufficient to just have a relay
>>>> rate
>>>> > > limit and prioritise by ancestor feerate though. Maybe something
>>>> like:
>>>> > > - instead of adding txs to each peers setInventoryTxToSend
>>>> immediately,
>>>> > >   set a mempool flag "relayed=3Dfalse"
>>>> > > - on a time delay, add the top N (by fee rate) "relayed=3Dfalse" t=
xs
>>>> to
>>>> > >   each peer's setInventoryTxToSend and mark them as "relayed=3Dtru=
e";
>>>> > >   calculate how much kB those txs were, and do this again after
>>>> > >   SIZE/RATELIMIT seconds
>>>>
>>>> > > - don't include "relayed=3Dfalse" txs when building blocks?
>>>>
>>>> The "?" was me not being sure that point is a good suggestion...
>>>>
>>>> Miners might reasonably decide to have no rate limit, and always relay=
,
>>>> and never exclude txs -- but the question then becomes is whether they
>>>> hear about the tx at all, so rate limiting behaviour could still be a
>>>> potential problem for whoever made the tx.
>>>>
>>>> > Wow cool! I think outbound tx relay size-based rate-limiting and
>>>> > prioritizing tx relay by feerate are great ideas for preventing
>>>> spammers
>>>> > from wasting bandwidth network-wide. I agree, this would slow the lo=
w
>>>> > feerate spam down, preventing a huge network-wide bandwidth spike.
>>>> And it
>>>> > would allow high feerate transactions to propagate as they should,
>>>> > regardless of how busy traffic is. Combined with inbound tx request
>>>> > rate-limiting, might this be sufficient to prevent DoS regardless of
>>>> the
>>>> > fee-based replacement policies?
>>>>
>>>> I think you only want to do outbound rate limits, ie, how often you se=
nd
>>>> INV, GETDATA and TX messages? Once you receive any of those, I think
>>>> you have to immediately process / ignore it, you can't really sensibly
>>>> defer it (beyond the existing queues we have that just build up while
>>>> we're busy processing other things first)?
>>>>
>>>> > One point that I'm not 100% clear on: is it ok to prioritize the
>>>> > transactions by ancestor feerate in this scheme? As I described in t=
he
>>>> > original post, this can be quite different from the actual feerate w=
e
>>>> would
>>>> > consider a transaction in a block for. The transaction could have a
>>>> high
>>>> > feerate sibling bumping its ancestor.
>>>> > For example, A (1sat/vB) has 2 children: B (49sat/vB) and C
>>>> (5sat/vB). If
>>>> > we just received C, it would be incorrect to give it a priority equa=
l
>>>> to
>>>> > its ancestor feerate (3sat/vB) because if we constructed a block
>>>> template
>>>> > now, B would bump A, and C's new ancestor feerate is 5sat/vB.
>>>> > Then, if we imagine that top N is >5sat/vB, we're not relaying C. If
>>>> we
>>>> > also exclude C when building blocks, we're missing out on good fees.
>>>>
>>>> I think you're right that this would be ugly. It's something of a
>>>> special case:
>>>>
>>>>  a) you really care about C getting into the next block; but
>>>>  b) you're trusting B not being replaced by a higher fee tx that
>>>>     doesn't have A as a parent; and
>>>>  c) there's a lot of txs bidding the floor of the next block up to a
>>>>     level in-between the ancestor fee rate of 3sat/vB and the tx fee
>>>>     rate of 5sat/vB
>>>>
>>>> Without (a), maybe you don't care about it getting to a miner quickly.
>>>> If your trust in (b) was misplaced, then your tx's effective fee rate
>>>> will drop and (because of (c)), you'll lose anyway. And if the spam en=
ds
>>>> up outside of (c)'s range, either the rate limiting won't take effect
>>>> (spam's too cheap) and you'll be fine, or you'll miss out on the block
>>>> anyway (spam's paying more than your tx rate) and you never had any ho=
pe
>>>> of making it in.
>>>>
>>>> Note that we already rate limit via INVENTORY_BROADCAST_MAX /
>>>> *_INVENTORY_BROADCAST_INTERVAL; which gets to something like 10,500 tx=
s
>>>> per 10 minutes for outbound connections. This would be a weight based
>>>> rate limit instead-of/in-addition-to that, I guess.
>>>>
>>>> As far as a non-ugly approach goes, I think you'd have to be smarter
>>>> about
>>>> tracking the "effective fee rate" than the ancestor fee rate manages;
>>>> maybe that's something that could fall out of Murch and Clara's
>>>> candidate
>>>> set blockbuilding ideas [0] ?
>>>>
>>>> Perhaps that same work would also make it possible to come up with
>>>> a better answer to "do I care that this replacement would invalidate
>>>> these descendents?"
>>>>
>>>> [0] https://github.com/Xekyo/blockbuilding
>>>>
>>>> > > - keep high-feerate evicted txs around for a while in case they ge=
t
>>>> > >   mined by someone else to improve compact block relay, a la the
>>>> > >   orphan pool?
>>>> > Replaced transactions are already added to vExtraTxnForCompact :D
>>>>
>>>> I guess I was thinking that it's just a 100 tx LRU cache, which might
>>>> not be good enough?
>>>>
>>>> Maybe it would be more on point to have a rate limit apply only to
>>>> replacement transactions?
>>>>
>>>> > For wallets, AJ's "All you need is for there to be *a* path that
>>>> follows
>>>> > the new relay rules and gets from your node/wallet to perhaps 10% of
>>>> > hashpower" makes sense to me (which would be the former).
>>>>
>>>> Perhaps a corollarly of that is that it's *better* to have the mempool
>>>> acceptance rule only consider economic incentives, and have the spam
>>>> prevention only be about "shall I tell my peers about this?"
>>>>
>>>> If you don't have that split; then the anti-spam rules can prevent you
>>>> from getting the tx in the mempool at all; whereas if you do have the
>>>> split, then even if the bitcoind anti-spam rules are blocking you at
>>>> every turn, you can still send your tx to miners by some other route,
>>>> and then they can add it to their mempool directly without any hassle.
>>>>
>>>> Cheers,
>>>> aj
>>>>
>>>> _______________________________________________
>>> bitcoin-dev mailing list
>>> bitcoin-dev@lists.linuxfoundation.org
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>
>>

--000000000000a9cb0405da2b27bd
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi Billy,</div><div><br></div>&gt; We should expect m=
iners will be using a more complex, more optimal way of determining what bl=
ocks they&#39;re working on [...] we should instead run with the assumption=
 that miners keep all potentially relevant transactions in their mempools, =
including potentially many conflicting transctions, in order to create the =
most profitable blocks. And therefore we shouldn&#39;t put the constraint o=
n normal non-mining full nodes to do that same more-complex mempool behavio=
r or add any complexity for the purpose of denying transaction replacements=
.<br><br>&gt; I think a lot of the complexity in these ideas is because of =
the attempt to match relay rules with miner <br>inclusion rules.<br><br>I t=
hink the assumption that miners are using a completely different implementa=
tion of mempool and block template building is false. IIUC, most miners use=
 Bitcoin Core and perhaps configure their node differently (e.g. larger mem=
pool and different minfeerates), but also use `getblocktemplate` which mean=
s the same ancestor package-based mining algorithm.<br><br>Of course, I&#39=
;m not a miner, so if anybody is a miner or has seen miners&#39; setups, pl=
ease correct me if I&#39;m wrong.<br><br>In either case, we would want our =
mining algorithm to result in block templates that are as close as possible=
 to perfectly incentive compatibile.<br><br>Fundamentally, I believe defaul=
t mempool policy (which perhaps naturally creates a network-wide transactio=
n relay policy) should be as close to the mining code as possible. Imagine =
node A only keeps 1 block&#39;s worth of transactions, and node B keeps a (=
default) 300MB mempool. The contents of node A&#39;s mempool should be as c=
lose as possible to a block template generated from node B&#39;s mempool. O=
therwise, node A&#39;s mempool is not very useful - their fee estimation is=
 flawed and compact block relay won&#39;t do them much good if they need to=
 re-request a lot of block transactions.<br>Next, imagine that node B is a =
miner. It would be very suboptimal if the mining code was ancestor package-=
based (i.e. supports CPFP), but the mempool policy only cared about individ=
ual feerate, and evicted low-fee parents despite their high-fee children. I=
t&#39;s easy to see why this would be suboptimal.<br>Attempting to match me=
mpool policy with the mining algorithm is also arguably the point of packag=
e relay. Our mining code uses ancestor packages which is good, but we only =
submit transactions one at a time to the mempool, so a transaction&#39;s hi=
gh-fee children can&#39;t be considered until they are all already in the m=
empool. Package relay allows us to start thinking about ancestor packages i=
mmediately when evaluating transactions for submission to the mempool.<br><=
br>The attempt to match policy with miner inclusion rules is deliberate and=
 necessary.<br><br>&gt; I want to echo James O&#39;Beirne&#39;s opinion on =
this that this may be the wrong path to go down (a path of more complexity =
without much gain). He said: &quot;Special consideration for &quot;what sho=
uld be in the next block&quot; and/or the caching of block templates seems =
like an imposing dependency, dragging in a bunch of state and infrastructur=
e to a question that should be solely limited to mempool feerate aggregates=
 and the feerate of the particular txn package a wallet is concerned with.&=
quot;<br><br>It seems that I under-explained the purpose of building/cachin=
g block templates in my original post, since both you and James have the sa=
me misunderstanding. Since RBF&#39;s introduction, we have improved to an a=
ncestor package-based mining algorithm. This supports CPFP (incentive compa=
tible) and it is now common to see more complex &quot;families&quot; of tra=
nsactions as users fee-bump transactions (market is working, yay). On the o=
ther hand, we no longer have an accurate way of determining a transaction&#=
39;s &quot;mining score,&quot; i.e., the feerate of this transaction&#39;s =
ancestor package when it is included in a block template using our current =
mining algorithm.<br><br>This limitation is a big blocker in proposing new =
fee/feerate RBF rules. For example, if we say &quot;the transaction needs a=
 better feerate,&quot; this is obviously flawed, since the original transac=
tions may have very high-feerate children, and the replacement transaction =
may have low feerate parents. So what we really want is &quot;the transacti=
on needs to be more incentive compatible to mine based on our mining algori=
thm,&quot; but we have no way of getting that information right now.<br><br=
>In my original post, I [described 4 heuristics to get transaction&#39;s &q=
uot;mining score&quot;][1] using the current data cached in the mempool (e.=
g. max ancestor feerate of descendant set), as well as why they don&#39;t w=
ork. As such, the best way to calculate a transaction&#39;s mining score AF=
AICT is to grab all of the related transactions and build a mini &quot;bloc=
k template&quot; with them. The [implementation][2] I sent last week also c=
uts out some of the fluff, so the pseudocode looks like this:<br><br>// Get=
 ALL connected entries (ancestors, descendants, siblings, cousins, coparent=
s, etc.)<br>vector&lt;TxMempoolEntry&gt; cluster =3D mempool.GetAllTransact=
ionsRelatedTo(txid);<br>sort(cluster, ancestorfeerate);<br><br>// For deduc=
ting ancestors when they are included separately<br>vector&lt;ModifiedTxMem=
poolEntry&gt; modified_entries;<br><br>while (!cluster.empty() and !modifie=
d_entries.empty()) {<br>=C2=A0 =C2=A0 iter =3D BetterAncestorFeerateOf(clus=
ter, modified_entries);<br>=C2=A0 =C2=A0 best_ancestor_package =3D GetAnces=
torSet(iter);<br>=C2=A0 =C2=A0 mining_score =3D Feerate(best_ancestor_packa=
ge);<br>=C2=A0 =C2=A0 for (entry : best_ancestor_package) {<br>=C2=A0=C2=A0=
 =C2=A0 =C2=A0 mining_scores[entry] =3D mining_score;<br>=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 for (descendant : GetAllDescendants(entry) {<br>=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 modified_entries[des=
cendant].DeductAncestorFromAccounting(entry);<br>=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 }<br>=C2=A0=C2=A0=C2=A0 }<br>}<br><br>Perhaps somebody will co=
me up with a better way to do this, but this is my solution, and I hope I&#=
39;ve now sufficiently described why I&#39;ve made an effort to think about=
 this. It&#39;s not because I want to make things more fancy or complicated=
, but because we currently have no way of accurately determining a transact=
ion&#39;s mining score. The reason I proposed a persistent block template i=
s so we can efficiently query the mining score of all transactions in the m=
empool.<br><div><br></div><div>&gt; However, I do think that these DOS conc=
erns are quite overblown. I wrote up <a href=3D"https://gist.github.com/glo=
zow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=3D4093100#gistcom=
ment-4093100" target=3D"_blank">a comment on your rbf-improvements.md</a>=
=C2=A0detailing
 my thought process on that. The summary is that as long as the fee-rate
 relay rule is maintained, any &quot;spam&quot; is actually paid for, eithe=
r by=20
the &quot;first&quot; transaction in the spam chain, or by the &quot;spam&q=
uot; itself.</div><div><br></div><div>The DoS concern is not overblown. I r=
ecommend you re-read the [current RBF policy][3]; Rule 3 and 4 *are* the fe=
erate relay rules. Removing Rule 3 means allowing recycled fees, so new tra=
nsactions are not necessarily &quot;paying&quot; anything.<br></div><br>[1]=
: <a href=3D"https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9f=
f?permalink_comment_id=3D4093100#mining-score-of-a-mempool-transaction">htt=
ps://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comm=
ent_id=3D4093100#mining-score-of-a-mempool-transaction</a><br><div>[2]: <a =
href=3D"https://github.com/glozow/bitcoin/tree/2022-02-mining-score">https:=
//github.com/glozow/bitcoin/tree/2022-02-mining-score</a></div><div>[3]: <a=
 href=3D"https://github.com/bitcoin/bitcoin/blob/master/doc/policy/mempool-=
replacements.md">https://github.com/bitcoin/bitcoin/blob/master/doc/policy/=
mempool-replacements.md</a></div><div><br></div><div>Best,</div><div>Gloria=
<br></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gm=
ail_attr">On Sat, Mar 12, 2022 at 8:18 AM Billy Tetrud &lt;<a href=3D"mailt=
o:billy.tetrud@gmail.com">billy.tetrud@gmail.com</a>&gt; wrote:<br></div><b=
lockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-le=
ft:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr">In reading=
 through more of the discussion, it seems the idea I presented above might =
basically be a reformulation of t-bast&#39;s rate-limiting idea presented i=
n <a href=3D"https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d3783b9f=
f?permalink_comment_id=3D4081349#gistcomment-4081349" target=3D"_blank">thi=
s comment</a>. Perhaps he could comment on whether that characterization is=
 accurate.=C2=A0</div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=
=3D"gmail_attr">On Fri, Mar 11, 2022 at 10:22 AM Billy Tetrud &lt;<a href=
=3D"mailto:billy.tetrud@gmail.com" target=3D"_blank">billy.tetrud@gmail.com=
</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:=
0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">=
<div dir=3D"ltr"><div>Hi Gloria,</div><div dir=3D"ltr"><br></div><div dir=
=3D"ltr">&gt;=C2=A0

1. Transaction relay rate limiting</div><div dir=3D"ltr"><br></div><div>I h=
ave a similar concern as yours, that this could prevent higher fee-rate tra=
nsactions from being broadcast.=C2=A0</div><div dir=3D"ltr"><br></div><div =
dir=3D"ltr"><div>&gt; 2.=C2=A0<span>Staggered</span>=C2=A0broadcast of repl=
acement transactions: within some time interval, maybe accept multiple repl=
acements for the same prevout, but only relay the original transaction.<br>=
</div></div><br><div>By this do you mean basically having a batching window=
 where, on receiving a replacement transaction, a node will wait for a peri=
od of time, potentially receiving many replacements for the same transactio=
n (or many separate conflicting transactions), and only broadcasting the &q=
uot;best&quot; one(s) at the end of that time window?=C2=A0</div><div><br><=
/div><div>Its an interesting idea, but it would produce a problem. Every ho=
p that replacement transaction takes would be delayed by this staggered win=
dow. If the window were 3 minutes and transactions generally take 20 hops t=
o get to the majority of miners, a &quot;best-case average&quot; delay migh=
t be 3.75 minutes (noting that among your 8 nodes, its=C2=A0quite likely on=
e of them would have a window ending much sooner than 3 minutes). Some (may=
be 3% of) nodes would experience delays of more than 20 minutes. That kind =
of delay isn&#39;t great.=C2=A0</div><div><br></div><div>However it made me=
 think of another idea: a transaction replacement broadcast cooldown. What =
if nodes kept track of the time they broadcasted the last replacement for a=
 package and had a relay cooldown after the last replacement was broadcaste=
d? A node receiving a replacement would relay the replacement immediately i=
f the package its replacing was broadcasted more than X seconds ago, and ot=
herwise it would wait until the time when that package was broadcasted at l=
east X seconds ago to broadcast it. Any replacements it receives during tha=
t waiting period would replace as normal, meaning the unrebroadcasted repla=
cement would never be broadcasted,=C2=A0and only the highest value replacem=
ent would be broadcasted at the end of the cooldown.</div><div><br></div><d=
iv>This wouldn&#39;t prevent a higher-fee-rate transaction from being broad=
casted (like rate limiting could), but would still be effective=C2=A0at lim=
iting unnecessary data transmission. Another benefit is that in the non-adv=
ersarial case, replacement transactions wouldn&#39;t be subject to any dela=
y at all (while in the staggered broadcast idea, most replacements would ex=
perience some delay). And in the adversarial case, where a malicious actor =
broadcasts a low-as-possible-value replacement just before yours, the=C2=A0=
worst case delay is just whatever the cooldown period is. I would imagine t=
hat maybe 1 minute would be a reasonable worst-case delay. This would limit=
 spam for a transaction that makes it into a block to ~10x (9 to 1). I don&=
#39;t see much of a downside to doing this beyond just the slight additiona=
l complexity of relay rules (and considering it could save substantial addi=
tional code complexity, even that is a benefit).=C2=A0</div><div><br></div>=
<div>All a node would need to do is keep a timestamp on each transaction th=
ey receive for when it was broadcasted and check it when a replacement come=
s in. If now-broadcastDate &lt; cooldown, set a timer for cooldown - (now-b=
roadcastDate) to broadcast it. If another replacement comes in, clear that =
timer and repeat using the original broadcast date (since the unbroadcast t=
ransaction doesn&#39;t have a broadcast date yet).=C2=A0</div><div><br></di=
v><div>I think it might also be useful to note that eliminating &quot;extra=
 data&quot; caused by careless or malicious actors (spam or whatever you wa=
nt to call it) should not be the goal. It is impossible to prevent all spam=
. What we should be aiming for is more specific: we should attempt to desig=
n a system where spam is manageable. Eg if our goal is to ensure that a bit=
coin node uses no more than 10% of the bandwidth of a &quot;normal&quot; us=
er, if current non-spam traffic only requires 1% of a &quot;normal&quot; us=
ers&#39;s=C2=A0bandwidth, then the network can bear a 9 to 1 ratio of spam.=
 When a node spins up, there is a lot more data to download and process. So=
 we know that all full nodes can handle at least as much traffic as they ha=
ndle during IBD. What&#39;s the difference between those amounts? I&#39;m n=
ot sure, but I would guess that IBD is at least a couple times more demandi=
ng than a fully synced node. So I might suggest that as long as spam can be=
 kept below a ratio of maybe 2 to 1, we should consider the design acceptab=
le (and therefore more complexity unnecessary).=C2=A0</div><div><br></div><=
div>The 1 minute broadcast cooldown I mentioned before wouldn&#39;t be quit=
e sufficient to achieve that ratio. But a 3.33 minute cooldown would be. Wh=
ether this is &quot;too much&quot; is something that would have to be discu=
ssed,=C2=A0I suspect a worst-case adversarial 3.33 minute delay would not b=
e &quot;too much&quot;. Doing this could basically eliminate any risk of ac=
tual service denial via replacement transactions.=C2=A0</div><div><br></div=
><div>However, I do think that these DOS concerns are quite overblown. I wr=
ote up <a href=3D"https://gist.github.com/glozow/25d9662c52453bd08b4b4b1d37=
83b9ff?permalink_comment_id=3D4093100#gistcomment-4093100" target=3D"_blank=
">a comment on your rbf-improvements.md</a>=C2=A0detailing my thought proce=
ss on that. The summary is that as long as the fee-rate relay rule is maint=
ained, any &quot;spam&quot; is actually paid for, either by the &quot;first=
&quot; transaction in the spam chain, or by the &quot;spam&quot; itself. Ev=
en without something like a minimum RBF relay delay limiting how much spam =
could be created, the economics of the fee-rate rule already sufficiently m=
itigate the issue of spam.=C2=A0<br></div></div><div class=3D"gmail_quote">=
<div dir=3D"ltr" class=3D"gmail_attr">On Wed, Mar 9, 2022 at 9:37 AM Gloria=
 Zhao via bitcoin-dev &lt;<a href=3D"mailto:bitcoin-dev@lists.linuxfoundati=
on.org" target=3D"_blank">bitcoin-dev@lists.linuxfoundation.org</a>&gt; wro=
te:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px =
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"=
ltr"><div>Hi RBF friends,<br></div><div><br></div><div>Posting a summary of=
 RBF discussions at coredev (mostly on transaction relay rate-limiting), us=
er-elected descendant limit as a short term solution to unblock package RBF=
, and mining score, all open for feedback:<br><br>One big concept discussed=
 was baking DoS protection into the p2p level rather than policy level. TLD=
R: The fees are not paid to the node operator, but to the miner. While we c=
an use fees to reason about the cost of an attack, if we&#39;re ultimately =
interested in preventing resource exhaustion, maybe we want to &quot;stop t=
he bleeding&quot; when it happens and bound the amount of resources used in=
 general. There were two main ideas:<br><br>1. Transaction relay rate limit=
ing (i.e. the one you proposed above or some variation) with a feerate-base=
d priority queue<br>2. Staggered broadcast of replacement transactions: wit=
hin some time interval, maybe accept multiple replacements for the same pre=
vout, but only relay the original transaction.<br><br>Looking to solicit fe=
edback on these ideas and the concept in general. Is it a good idea (separa=
te from RBF) to add rate-limiting in transaction relay? And is it the right=
 direction to think about RBF DoS protection this way?<br><br>A lingering c=
oncern that I have about this idea is it would then be possible to impact t=
he propagation of another person=E2=80=99s transaction, i.e., an attacker c=
an censor somebody=E2=80=99s transaction from ever being announced by a nod=
e if they send enough transactions to fill up the rate limit. Obviously thi=
s would be expensive since they&#39;re spending a lot on fees, but I imagin=
e it could be profitable in some situations to spend a few thousand dollars=
 to prevent anyone from hearing about a transaction for a few hours. This m=
ight be a non-issue in practice if the rate limit is generous and traffic i=
sn=E2=80=99t horrendous, but is this a problem?<br><br>And if we don&#39;t =
require an increase in (i.e. addition of &quot;new&quot;) absolute fees, us=
ers are essentially allowed to =E2=80=9Crecycle=E2=80=9D fees. In the scena=
rio where we prioritize relay based on feerate, users could potentially be =
placed higher in the queue, ahead of other users=E2=80=99 transactions, mul=
tiple times, without ever adding more fees to the transaction. Again, maybe=
 this isn=E2=80=99t a huge deal in practice if we set the parameters right,=
 but it seems=E2=80=A6 not great, in principle.<br></div><div><br></div><di=
v>---------<br></div><div><br></div><div>It&#39;s probably also a good idea=
 to point out that there&#39;s been some discussion happening on the gist c=
ontaining my original post on this thread (<a href=3D"https://gist.github.c=
om/glozow/25d9662c52453bd08b4b4b1d3783b9ff" target=3D"_blank">https://gist.=
github.com/glozow/25d9662c52453bd08b4b4b1d3783b9ff</a>).</div><div><br></di=
v><div>Suhas and Matt [proposed][0] adding a policy rule allowing users to =
specify descendant limits on their transactions. For example, some nth bit =
of nSequence with nVersion 3 means &quot;this transaction won&#39;t have mo=
re than X vbytes of descendants&quot; where X =3D max(1000, vsizeof(tx)) or=
 something. It solves the pinning problem with package RBF where the attack=
er&#39;s package contains a very large and high-fee descendant.</div><div><=
br></div><div>We could add this policy and deploy it with package RBF/packa=
ge relay so that LN can use it by setting the user-elected descendant limit=
 flag on commitment transactions. (Otherwise package RBF is blocked until w=
e find a more comprehensive solution to the pinning attack).</div><div><br>=
</div><div>It&#39;s simple to [implement][1] as a mempool policy, but adds =
some complexity for wallets that use it, since it limits their use of UTXOs=
 from transactions with this bit set.<br></div><div><br></div><div>--------=
-</div><div><br></div><div>Also, coming back to the idea of &quot;we can&#3=
9;t just use {individual, ancestor} feerate,&quot; I&#39;m interested in so=
liciting feedback on adding a =E2=80=9Cmining score=E2=80=9D calculator. I&=
#39;ve implemented one [here][2] which takes the transaction in question, g=
rabs all of the connected mempool transactions (including siblings, coparen=
ts, etc., as they wouldn=E2=80=99t be in the ancestor nor descendant sets),=
 and builds a =E2=80=9Cblock template=E2=80=9D using our current mining alg=
orithm. The mining score of a transaction is the ancestor feerate at which =
it is included.<br></div><div><br></div><div>This would be helpful for some=
thing like ancestor-aware funding and fee-bumping in the wallet: [3], [4]. =
I think if we did the rate-limited priority queue for transaction relay, we=
&#39;d want to use something like this as the priority value. And for RBF, =
we probably want to require that a replacement have a higher mining score t=
han the original transactions. This could be computationally expensive to d=
o all the time; it could be good to cache it but that could make mempool bo=
okkeeping more complicated. Also, if we end up trying to switch to a candid=
ate set-based algorithm for mining, we&#39;d of course need a new calculato=
r.<br></div><div><br></div><div>[0]: <a href=3D"https://gist.github.com/glo=
zow/25d9662c52453bd08b4b4b1d3783b9ff?permalink_comment_id=3D4058140#gistcom=
ment-4058140" target=3D"_blank">https://gist.github.com/glozow/25d9662c5245=
3bd08b4b4b1d3783b9ff?permalink_comment_id=3D4058140#gistcomment-4058140</a>=
<br>[1]: <a href=3D"https://github.com/glozow/bitcoin/tree/2022-02-user-des=
climit" target=3D"_blank">https://github.com/glozow/bitcoin/tree/2022-02-us=
er-desclimit</a></div><div>[2] <a href=3D"https://github.com/glozow/bitcoin=
/tree/2022-02-mining-score" target=3D"_blank">https://github.com/glozow/bit=
coin/tree/2022-02-mining-score</a><br>[3]: <a href=3D"https://github.com/bi=
tcoin/bitcoin/issues/9645" target=3D"_blank">https://github.com/bitcoin/bit=
coin/issues/9645</a><br>[4]: <a href=3D"https://github.com/bitcoin/bitcoin/=
issues/15553" target=3D"_blank">https://github.com/bitcoin/bitcoin/issues/1=
5553</a><br></div><br><div>Best,</div><div>Gloria<br></div></div><br><div c=
lass=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Tue, Feb 8, 2=
022 at 4:58 AM Anthony Towns &lt;<a href=3D"mailto:aj@erisian.com.au" targe=
t=3D"_blank">aj@erisian.com.au</a>&gt; wrote:<br></div><blockquote class=3D=
"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(2=
04,204,204);padding-left:1ex">On Mon, Feb 07, 2022 at 11:16:26AM +0000, Glo=
ria Zhao wrote:<br>
&gt; @aj:<br>
&gt; &gt; I wonder sometimes if it could be sufficient to just have a relay=
 rate<br>
&gt; &gt; limit and prioritise by ancestor feerate though. Maybe something =
like:<br>
&gt; &gt; - instead of adding txs to each peers setInventoryTxToSend immedi=
ately,<br>
&gt; &gt;=C2=A0 =C2=A0set a mempool flag &quot;relayed=3Dfalse&quot;<br>
&gt; &gt; - on a time delay, add the top N (by fee rate) &quot;relayed=3Dfa=
lse&quot; txs to<br>
&gt; &gt;=C2=A0 =C2=A0each peer&#39;s setInventoryTxToSend and mark them as=
 &quot;relayed=3Dtrue&quot;;<br>
&gt; &gt;=C2=A0 =C2=A0calculate how much kB those txs were, and do this aga=
in after<br>
&gt; &gt;=C2=A0 =C2=A0SIZE/RATELIMIT seconds<br>
<br>
&gt; &gt; - don&#39;t include &quot;relayed=3Dfalse&quot; txs when building=
 blocks?<br>
<br>
The &quot;?&quot; was me not being sure that point is a good suggestion...<=
br>
<br>
Miners might reasonably decide to have no rate limit, and always relay,<br>
and never exclude txs -- but the question then becomes is whether they<br>
hear about the tx at all, so rate limiting behaviour could still be a<br>
potential problem for whoever made the tx.<br>
<br>
&gt; Wow cool! I think outbound tx relay size-based rate-limiting and<br>
&gt; prioritizing tx relay by feerate are great ideas for preventing spamme=
rs<br>
&gt; from wasting bandwidth network-wide. I agree, this would slow the low<=
br>
&gt; feerate spam down, preventing a huge network-wide bandwidth spike. And=
 it<br>
&gt; would allow high feerate transactions to propagate as they should,<br>
&gt; regardless of how busy traffic is. Combined with inbound tx request<br=
>
&gt; rate-limiting, might this be sufficient to prevent DoS regardless of t=
he<br>
&gt; fee-based replacement policies?<br>
<br>
I think you only want to do outbound rate limits, ie, how often you send<br=
>
INV, GETDATA and TX messages? Once you receive any of those, I think<br>
you have to immediately process / ignore it, you can&#39;t really sensibly<=
br>
defer it (beyond the existing queues we have that just build up while<br>
we&#39;re busy processing other things first)?<br>
<br>
&gt; One point that I&#39;m not 100% clear on: is it ok to prioritize the<b=
r>
&gt; transactions by ancestor feerate in this scheme? As I described in the=
<br>
&gt; original post, this can be quite different from the actual feerate we =
would<br>
&gt; consider a transaction in a block for. The transaction could have a hi=
gh<br>
&gt; feerate sibling bumping its ancestor.<br>
&gt; For example, A (1sat/vB) has 2 children: B (49sat/vB) and C (5sat/vB).=
 If<br>
&gt; we just received C, it would be incorrect to give it a priority equal =
to<br>
&gt; its ancestor feerate (3sat/vB) because if we constructed a block templ=
ate<br>
&gt; now, B would bump A, and C&#39;s new ancestor feerate is 5sat/vB.<br>
&gt; Then, if we imagine that top N is &gt;5sat/vB, we&#39;re not relaying =
C. If we<br>
&gt; also exclude C when building blocks, we&#39;re missing out on good fee=
s.<br>
<br>
I think you&#39;re right that this would be ugly. It&#39;s something of a<b=
r>
special case:<br>
<br>
=C2=A0a) you really care about C getting into the next block; but<br>
=C2=A0b) you&#39;re trusting B not being replaced by a higher fee tx that<b=
r>
=C2=A0 =C2=A0 doesn&#39;t have A as a parent; and<br>
=C2=A0c) there&#39;s a lot of txs bidding the floor of the next block up to=
 a<br>
=C2=A0 =C2=A0 level in-between the ancestor fee rate of 3sat/vB and the tx =
fee<br>
=C2=A0 =C2=A0 rate of 5sat/vB<br>
<br>
Without (a), maybe you don&#39;t care about it getting to a miner quickly.<=
br>
If your trust in (b) was misplaced, then your tx&#39;s effective fee rate<b=
r>
will drop and (because of (c)), you&#39;ll lose anyway. And if the spam end=
s<br>
up outside of (c)&#39;s range, either the rate limiting won&#39;t take effe=
ct<br>
(spam&#39;s too cheap) and you&#39;ll be fine, or you&#39;ll miss out on th=
e block<br>
anyway (spam&#39;s paying more than your tx rate) and you never had any hop=
e<br>
of making it in.<br>
<br>
Note that we already rate limit via INVENTORY_BROADCAST_MAX /<br>
*_INVENTORY_BROADCAST_INTERVAL; which gets to something like 10,500 txs<br>
per 10 minutes for outbound connections. This would be a weight based<br>
rate limit instead-of/in-addition-to that, I guess.<br>
<br>
As far as a non-ugly approach goes, I think you&#39;d have to be smarter ab=
out<br>
tracking the &quot;effective fee rate&quot; than the ancestor fee rate mana=
ges;<br>
maybe that&#39;s something that could fall out of Murch and Clara&#39;s can=
didate<br>
set blockbuilding ideas [0] ?<br>
<br>
Perhaps that same work would also make it possible to come up with<br>
a better answer to &quot;do I care that this replacement would invalidate<b=
r>
these descendents?&quot;<br>
<br>
[0] <a href=3D"https://github.com/Xekyo/blockbuilding" rel=3D"noreferrer" t=
arget=3D"_blank">https://github.com/Xekyo/blockbuilding</a><br>
<br>
&gt; &gt; - keep high-feerate evicted txs around for a while in case they g=
et<br>
&gt; &gt;=C2=A0 =C2=A0mined by someone else to improve compact block relay,=
 a la the<br>
&gt; &gt;=C2=A0 =C2=A0orphan pool?<br>
&gt; Replaced transactions are already added to vExtraTxnForCompact :D<br>
<br>
I guess I was thinking that it&#39;s just a 100 tx LRU cache, which might<b=
r>
not be good enough?<br>
<br>
Maybe it would be more on point to have a rate limit apply only to<br>
replacement transactions?<br>
<br>
&gt; For wallets, AJ&#39;s &quot;All you need is for there to be *a* path t=
hat follows<br>
&gt; the new relay rules and gets from your node/wallet to perhaps 10% of<b=
r>
&gt; hashpower&quot; makes sense to me (which would be the former).<br>
<br>
Perhaps a corollarly of that is that it&#39;s *better* to have the mempool<=
br>
acceptance rule only consider economic incentives, and have the spam<br>
prevention only be about &quot;shall I tell my peers about this?&quot;<br>
<br>
If you don&#39;t have that split; then the anti-spam rules can prevent you<=
br>
from getting the tx in the mempool at all; whereas if you do have the<br>
split, then even if the bitcoind anti-spam rules are blocking you at<br>
every turn, you can still send your tx to miners by some other route,<br>
and then they can add it to their mempool directly without any hassle.<br>
<br>
Cheers,<br>
aj<br>
<br>
</blockquote></div>
_______________________________________________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">=
bitcoin-dev@lists.linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" =
rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail=
man/listinfo/bitcoin-dev</a><br>
</blockquote></div>
</blockquote></div>
</blockquote></div>

--000000000000a9cb0405da2b27bd--