Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 2F1AE7F6 for ; Sat, 27 Jul 2019 16:10:08 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mail.bluematt.me (mail.bluematt.me [69.59.18.99]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 35668FE for ; Sat, 27 Jul 2019 16:10:05 +0000 (UTC) Received: from [192.168.1.67] (unknown [69.59.18.155]) by mail.bluematt.me (Postfix) with ESMTPSA id 8CA3082CA3; Sat, 27 Jul 2019 16:10:03 +0000 (UTC) Content-Type: multipart/alternative; boundary=Apple-Mail-FE382715-0483-4F15-993C-AB8315ADCD40 Mime-Version: 1.0 (1.0) From: Matt Corallo X-Mailer: iPhone Mail (16G77) In-Reply-To: <0DD0A8C8-374E-4E4F-AB08-C51612A28E16@jonasschnelli.ch> Date: Sat, 27 Jul 2019 12:10:03 -0400 Content-Transfer-Encoding: 7bit Message-Id: <4661BB32-3725-4E10-9698-25CDBA2F7ACB@mattcorallo.com> References: <59fad2b6-9b15-ffec-116e-91d27ce29f80@mattcorallo.com> <0DD0A8C8-374E-4E4F-AB08-C51612A28E16@jonasschnelli.ch> To: Jonas Schnelli , Bitcoin Protocol Discussion X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,HTML_MESSAGE, MIME_QP_LONG_LINE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org X-Mailman-Approved-At: Sat, 27 Jul 2019 20:17:43 +0000 Cc: Andreas Schildbach Subject: Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jul 2019 16:10:08 -0000 --Apple-Mail-FE382715-0483-4F15-993C-AB8315ADCD40 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable This conversation went off the rails somewhat. I don't think there's any imm= ediate risk of NODE_BLOOM peers being unavailable. This is a defaults change= , not a removal of the code to serve BIP 37 peers (nor would I suggest remov= ing said code while people still want to use them - the maintenance burden i= sn't much). Looking at historical upgrade cycles, ignoring any other factors= , there will be a large number of nodes serving NODE_BLOOM for many years. Even more importantly, if you need them, run a node or two. As long as no on= e is exploiting the issues with them such a node isn't *too* expensive. Or d= on't, I guarantee you chainanalysis or some competitor of theirs will very v= ery happily serve bloom-filtered clients as long as such clients want to dea= nonymize themselves. We already see a plurality of nodes on the network are c= learly not run-of-the-mill Core nodes, many of which are likely deanonimizat= ion efforts. In some cases BIP 137 is a replacement, in some cases, indeed, it is not. I a= gree at a protocol level we shouldn't be passing judgement about how users w= ish to interact with the Bitcoin system (aside from not putting our own, per= sonal, effort into building such things) but that isn't what's happening her= e. This is an important DoS fix for the average node, and I don't really und= erstand the argument that this is going to break existing BIP 37 wallets, bu= t if it makes you feel any better I can run some beefy BIP 37 nodes. Matt > On Jul 26, 2019, at 06:04, Jonas Schnelli via bitcoin-dev wrote: >=20 >=20 >> 1) It causes way too much traffic for mobile users, and likely even too >> much traffic for fixed lines in not so developed parts of the world. >=20 > Yes. It causes more traffic than BIP37. > Basic block filters for current last ~7 days (1008 blocks) are about 19MB (= just the filters). > On top, you will probably fetch a handful of irrelevant blocks due to the = FPs and due to true relevant txns. > A over-the-thumb estimation: ~25MB per week of catch-up. > If you where offline for a month: ~108MB >=20 > Thats certainly more then BIP37 BF (measured 1.6MB total traffic with andr= oid schildbach wallet restore blockchain for 8 week [7 weeks headers, 1week m= erkleblocks]). >=20 > But lets look at it like this: for an additional, say 25MB per week (maybe= a bit more), you get the ability to filter blocks without depending on serv= ing peers who may compromise your financial privacy. > Also, if you keep the filters, further rescans do consume the same or less= bandwidth than BF BIP37. > In other words: you have the chance to potentially increase privacy by con= suming bandwidth in the range of a single audio podcast per week. >=20 > I would say the job of protocol developers is protect users privacy where i= t=E2=80=99s possible (as a default). > It=E2=80=99s probably a debatable point wether 25MB per week of traffic is= worth a potential increase in privacy, though I absolutely think 25MB/week i= s an acceptable tradeoff. > Saving traffic is possible by using BIP37 or stratum/electrum=E2=80=A6 but= developers should make sure users are __warned about the consequences__! >=20 > Additionally, it looks like, peer operators are not endless being willing t= o serve =E2=80=93 for free =E2=80=93 a CPU/disk intense service with no bene= fits for the network. I would question wether a decentralised form of BIP37 i= s sustainable in the long run (if SPV wallet provider bootstrap a net range o= f NODE_BLOOM peers to make it more reliable on the network would be snake-oi= l). >=20 >=20 >>=20 >> 2) It filters blocks only. It doesn't address unconfirmed transactions. >=20 > Well, unconfirmed transaction are uncertain for various reasons. >=20 > BIP158 won't allow you to filter the mempool. > But as soon as you are connected to the network, you may fetch tx with inv= /getdata and pick out the relevant ones (causes also traffic). > Unclear and probably impossible with the current BIP158 specs to fetch tra= nsactions that are not in active relay and are not in a block (mempool txns,= at least this is true with the current observed relay tactics). >=20 >=20 >> 3) Afaik, it enforces/encourages address re-use. This stems from the >> fact that the server decides on the filter and in particular on the >> false positive rate. On wallets with many addresses, a hardcoded filter >> will be too blurry and thus each block will be matched. So wallets that >> follow the "one address per incoming payment" pattern (e.g. HD wallets) >> at some point will be forced to wrap their key chains back to the >> beginning. If I'm wrong on this one please let me know. >=20 > I=E2=80=99m probably the wrong guy to ask (haven=E2=80=99t made the number= s) but last time I rescanned a Core wallet (in my dev branch) with block fil= ters (and a Core wallet has >2000 addresses by default) it fetched a low and= acceptable amount of false positive blocks. > (Maybe someone who made the numbers step in here.) >=20 > Though, large wallets =E2=80=93 AFAIK =E2=80=93 also operate badly with BI= P37. >=20 >>=20 >> 4) The filters are not yet committed to the blockchain. Until that >> happens we'd have to trust a server to provide correct filters. >=20 > I wouldn=E2=80=99t say so. It=E2=80=99s on a similar level than BIP37. > BIP37 is not =E2=80=93 and can not =E2=80=93 be committed to the blockchai= n. > You fully trust the peer that it won=E2=80=99t=E2=80=A6 > a) create fake unconfirmed transactions (would be the same if a BIP158 wal= let would show you unconfirmed transaction) > b) lies by omission (you will miss relevant transactions, eventually swipe= your wallet and loose coins) >=20 > IMO, the point b) is true for BIP37 and BIP158 (as long as not commited). > In both cases, you can reduce the trust by comparing between peers / filte= r-providers. >=20 > b) is conceptually solvable with a soft-fork (commitment) in BIP158 (not w= ith BIP37). >=20 > Additionally, block-filters will, very likely, be useful for other feature= s (load/rescan an [old] wallet on a prune peer that has the filters construc= ted). >=20 >=20 >=20 > There is probably no clear answer like =E2=80=9EX is better than Y=E2=80=9C= . >=20 > Personally I would like to see developers being more honest/transparent to= users about the implications of the used filtering,... and giving them choi= ces. > Imagine a user can choose between =E2=80=9EElectrum / BIP37 / BIP158=E2=80= =9C depending on his needs for privacy and availability of bandwidth. Eventu= ally also taking the future usage of this wallet (will he load old private k= eys, will he receive money daily, etc.) into account. >=20 > Plus, full node hardware appliances that run at home (or in a trusted envi= ronment) are solving many of these issues plus adding a bunch of great featu= res =E2=80=93 if done right. >=20 > /jonas > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev --Apple-Mail-FE382715-0483-4F15-993C-AB8315ADCD40 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
Thi= s conversation went off the rails somewhat. I don't think there's any immedi= ate risk of NODE_BLOOM peers being unavailable. This is a defaults change, n= ot a removal of the code to serve BIP 37 peers (nor would I suggest removing= said code while people still want to use them - the maintenance burden isn'= t much). Looking at historical upgrade cycles, ignoring any other factors, t= here will be a large number of nodes serving NODE_BLOOM for many years.

Even more importantly, if you n= eed them, run a node or two. As long as no one is exploiting the issues with= them such a node isn't *too* expensive. Or don't, I guarantee you chainanal= ysis or some competitor of theirs will very very happily serve bloom-filtere= d clients as long as such clients want to deanonymize themselves. We already= see a plurality of nodes on the network are clearly not run-of-the-mill Cor= e nodes, many of which are likely deanonimization efforts.

In some cases BIP 137 is a replacement, in s= ome cases, indeed, it is not. I agree at a protocol level we shouldn't be pa= ssing judgement about how users wish to interact with the Bitcoin system (as= ide from not putting our own, personal, effort into building such things) bu= t that isn't what's happening here. This is an important DoS fix for the ave= rage node, and I don't really understand the argument that this is going to b= reak existing BIP 37 wallets, but if it makes you feel any better I can run s= ome beefy BIP 37 nodes.

Mat= t

On Jul 26, 2019, at 06:04, Jonas Schnelli via bi= tcoin-dev <bitco= in-dev@lists.linuxfoundation.org> wrote:


1) It causes way too much traffic for mobile users, an= d likely even too
much traffic for f= ixed lines in not so developed parts of the world.

Yes. It ca= uses more traffic than BIP37.
Basic block filters for current last= ~7 days (1008 blocks) are about 19MB (just the filters).
On top, y= ou will probably fetch a handful of irrelevant blocks due to the FPs and due= to true relevant txns.
A over-the-thumb estimation: ~25MB per wee= k of catch-up.
If you where offline for a month: ~108MB
=
Thats certainly more then BIP37 BF (measured 1.6MB= total traffic with android schildbach wallet restore blockchain for 8 week [= 7 weeks headers, 1week merkleblocks]).

B= ut lets look at it like this: for an additional, say 25MB per week (maybe a b= it more), you get the ability to filter blocks without depending on serving p= eers who may compromise your financial privacy.
Also, if you keep t= he filters, further rescans do consume the same or less bandwidth than BF BI= P37.
In other words: you have the chance to potentially increase p= rivacy by consuming bandwidth in the range of a single audio podcast per wee= k.

I would say the job of protocol devel= opers is protect users privacy where it=E2=80=99s possible (as a default).
It=E2=80=99s probably a debatable point wether 25MB per week of tra= ffic is worth a potential increase in privacy, though I absolutely think 25M= B/week is an acceptable tradeoff.
Saving traffic is possible by us= ing BIP37 or stratum/electrum=E2=80=A6 but developers should make sure users= are __warned about the consequences__!

= Additionally, it looks like, peer operators are not endless being willing to= serve =E2=80=93 for free =E2=80=93 a CPU/disk intense service with no benef= its for the network. I would question wether a decentralised form of BIP37 i= s sustainable in the long run (if SPV wallet provider bootstrap a net range o= f NODE_BLOOM peers to make it more reliable on the network would be snake-oi= l).



2) It filters b= locks only. It doesn't address unconfirmed transactions.

Well, unco= nfirmed transaction are uncertain for various reasons.

BIP158 won't allow you to filter the mempool.
But as= soon as you are connected to the network, you may fetch tx with inv/getdata= and pick out the relevant ones (causes also traffic).
Unclear and= probably impossible with the current BIP158 specs to fetch transactions tha= t are not in active relay and are not in a block (mempool txns, at least thi= s is true with the current observed relay tactics).


3) Afaik, it enforces/encourages address re-use. This stems from the
fact that the server decides on the fi= lter and in particular on the
false p= ositive rate. On wallets with many addresses, a hardcoded filter
will be too blurry and thus each block will be= matched. So wallets that
follow t= he "one address per incoming payment" pattern (e.g. HD wallets)
at some point will be forced to wrap their key= chains back to the
beginning. If I= 'm wrong on this one please let me know.

I=E2=80=99m probably= the wrong guy to ask (haven=E2=80=99t made the numbers) but last time I res= canned a Core wallet (in my dev branch) with block filters (and a Core walle= t has >2000 addresses by default) it fetched a low and acceptable amount o= f false positive blocks.
(Maybe someone who made the numbers step i= n here.)

Though, large wallets =E2=80=93= AFAIK =E2=80=93 also operate badly with BIP37.


4) The filters are not yet committed to the blockchain. Until that=
happens we'd have to trust a server to pr= ovide correct filters.

I wouldn=E2=80=99t say so. It=E2=80=99s on a= similar level than BIP37.
BIP37 is not =E2=80=93 and can not =E2=80= =93 be committed to the blockchain.
You fully trust the peer that i= t won=E2=80=99t=E2=80=A6
a) create fake unconfirmed transactions (= would be the same if a BIP158 wallet would show you unconfirmed transaction)=
b) lies by omission (you will miss relevant transactions, eventua= lly swipe your wallet and loose coins)

I= MO, the point b) is true for BIP37 and BIP158 (as long as not commited).
In both cases, you can reduce the trust by comparing between peers / f= ilter-providers.

b) is conceptually solv= able with a soft-fork (commitment) in BIP158 (not with BIP37).
Additionally, block-filters will, very likely, be use= ful for other features (load/rescan an [old] wallet on a prune peer that has= the filters constructed).



There is probably no clear answer lik= e =E2=80=9EX is better than Y=E2=80=9C.

= Personally I would like to see developers being more honest/transparent to u= sers about the implications of the used filtering,... and giving them choice= s.
Imagine a user can choose between =E2=80=9EElectrum / BIP37 / B= IP158=E2=80=9C depending on his needs for privacy and availability of bandwi= dth. Eventually also taking the future usage of this wallet (will he load ol= d private keys, will he receive money daily, etc.) into account.
<= br class=3D"">
Plus, full node hardware appliances that run at hom= e (or in a trusted environment) are solving many of these issues plus adding= a bunch of great features =E2=80=93 if done right.

/jonas
_______________________________________________
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.orghttps://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
= --Apple-Mail-FE382715-0483-4F15-993C-AB8315ADCD40--