Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 33615AAE for ; Fri, 26 Jul 2019 10:04:40 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from bitcoin.jonasschnelli.ch (bitcoinsrv.jonasschnelli.ch [138.201.55.219]) by smtp1.linuxfoundation.org (Postfix) with ESMTP id B85EF709 for ; Fri, 26 Jul 2019 10:04:38 +0000 (UTC) Received: from [192.168.0.4] (cable-static-140-182.teleport.ch [87.102.140.182]) by bitcoin.jonasschnelli.ch (Postfix) with ESMTPSA id 5B26115E055F; Fri, 26 Jul 2019 12:04:37 +0200 (CEST) From: Jonas Schnelli Content-Type: multipart/signed; boundary="Apple-Mail=_EBD2572C-F977-492A-B1E2-46C7CC7DF621"; protocol="application/pgp-signature"; micalg=pgp-sha256 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Date: Fri, 26 Jul 2019 12:04:32 +0200 References: <59fad2b6-9b15-ffec-116e-91d27ce29f80@mattcorallo.com> To: Andreas Schildbach , Bitcoin Protocol Discussion In-Reply-To: Message-Id: <0DD0A8C8-374E-4E4F-AB08-C51612A28E16@jonasschnelli.ch> X-Mailer: Apple Mail (2.3445.104.11) X-Virus-Scanned: clamav-milter 0.100.3 at bitcoinsrv.jonasschnelli.ch X-Virus-Status: Clean X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,HTML_MESSAGE, MIME_QP_LONG_LINE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org X-Mailman-Approved-At: Sat, 27 Jul 2019 14:38:20 +0000 Subject: Re: [bitcoin-dev] Bitcoin Core to disable Bloom-based Filtering by default X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2019 10:04:40 -0000 --Apple-Mail=_EBD2572C-F977-492A-B1E2-46C7CC7DF621 Content-Type: multipart/alternative; boundary="Apple-Mail=_82AAF962-04AB-4EEF-B6AE-787E4F6F8CB4" --Apple-Mail=_82AAF962-04AB-4EEF-B6AE-787E4F6F8CB4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > 1) It causes way too much traffic for mobile users, and likely even = too > much traffic for fixed lines in not so developed parts of the world. Yes. It causes more traffic than BIP37. Basic block filters for current last ~7 days (1008 blocks) are about = 19MB (just the filters). On top, you will probably fetch a handful of irrelevant blocks due to = the FPs and due to true relevant txns. A over-the-thumb estimation: ~25MB per week of catch-up. If you where offline for a month: ~108MB Thats certainly more then BIP37 BF (measured 1.6MB total traffic with = android schildbach wallet restore blockchain for 8 week [7 weeks = headers, 1week merkleblocks]). But lets look at it like this: for an additional, say 25MB per week = (maybe a bit more), you get the ability to filter blocks without = depending on serving peers who may compromise your financial privacy. Also, if you keep the filters, further rescans do consume the same or = less bandwidth than BF BIP37. In other words: you have the chance to potentially increase privacy by = consuming bandwidth in the range of a single audio podcast per week. I would say the job of protocol developers is protect users privacy = where it=E2=80=99s possible (as a default). It=E2=80=99s probably a debatable point wether 25MB per week of traffic = is worth a potential increase in privacy, though I absolutely think = 25MB/week is an acceptable tradeoff. Saving traffic is possible by using BIP37 or stratum/electrum=E2=80=A6 = but developers should make sure users are __warned about the = consequences__! Additionally, it looks like, peer operators are not endless being = willing to serve =E2=80=93 for free =E2=80=93 a CPU/disk intense service = with no benefits for the network. I would question wether a = decentralised form of BIP37 is sustainable in the long run (if SPV = wallet provider bootstrap a net range of NODE_BLOOM peers to make it = more reliable on the network would be snake-oil). >=20 > 2) It filters blocks only. It doesn't address unconfirmed = transactions. Well, unconfirmed transaction are uncertain for various reasons. BIP158 won't allow you to filter the mempool. But as soon as you are connected to the network, you may fetch tx with = inv/getdata and pick out the relevant ones (causes also traffic). Unclear and probably impossible with the current BIP158 specs to fetch = transactions that are not in active relay and are not in a block = (mempool txns, at least this is true with the current observed relay = tactics). > 3) Afaik, it enforces/encourages address re-use. This stems from the > fact that the server decides on the filter and in particular on the > false positive rate. On wallets with many addresses, a hardcoded = filter > will be too blurry and thus each block will be matched. So wallets = that > follow the "one address per incoming payment" pattern (e.g. HD = wallets) > at some point will be forced to wrap their key chains back to the > beginning. If I'm wrong on this one please let me know. I=E2=80=99m probably the wrong guy to ask (haven=E2=80=99t made the = numbers) but last time I rescanned a Core wallet (in my dev branch) with = block filters (and a Core wallet has >2000 addresses by default) it = fetched a low and acceptable amount of false positive blocks. (Maybe someone who made the numbers step in here.) Though, large wallets =E2=80=93 AFAIK =E2=80=93 also operate badly with = BIP37. >=20 > 4) The filters are not yet committed to the blockchain. Until that > happens we'd have to trust a server to provide correct filters. I wouldn=E2=80=99t say so. It=E2=80=99s on a similar level than BIP37. BIP37 is not =E2=80=93 and can not =E2=80=93 be committed to the = blockchain. You fully trust the peer that it won=E2=80=99t=E2=80=A6 a) create fake unconfirmed transactions (would be the same if a BIP158 = wallet would show you unconfirmed transaction) b) lies by omission (you will miss relevant transactions, eventually = swipe your wallet and loose coins) IMO, the point b) is true for BIP37 and BIP158 (as long as not = commited). In both cases, you can reduce the trust by comparing between peers / = filter-providers. b) is conceptually solvable with a soft-fork (commitment) in BIP158 (not = with BIP37). Additionally, block-filters will, very likely, be useful for other = features (load/rescan an [old] wallet on a prune peer that has the = filters constructed). There is probably no clear answer like =E2=80=9EX is better than Y=E2=80=9C= . Personally I would like to see developers being more honest/transparent = to users about the implications of the used filtering,... and giving = them choices. Imagine a user can choose between =E2=80=9EElectrum / BIP37 / BIP158=E2=80= =9C depending on his needs for privacy and availability of bandwidth. = Eventually also taking the future usage of this wallet (will he load old = private keys, will he receive money daily, etc.) into account. Plus, full node hardware appliances that run at home (or in a trusted = environment) are solving many of these issues plus adding a bunch of = great features =E2=80=93 if done right. /jonas --Apple-Mail=_82AAF962-04AB-4EEF-B6AE-787E4F6F8CB4 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
1) It causes way too much = traffic for mobile users, and likely even too
much traffic for fixed lines in = not so developed parts of the world.

Yes. It causes more traffic than = BIP37.
Basic block filters for current last ~7 days (1008 = blocks) are about 19MB (just the filters).
On top, you will = probably fetch a handful of irrelevant blocks due to the FPs and due to = true relevant txns.
A over-the-thumb estimation: ~25MB per = week of catch-up.
If you where offline for a month: = ~108MB

Thats certainly more then = BIP37 BF (measured 1.6MB total traffic with android schildbach wallet = restore blockchain for 8 week [7 weeks headers, 1week = merkleblocks]).

But lets look at it = like this: for an additional, say 25MB per week (maybe a bit more), you = get the ability to filter blocks without depending on serving peers who = may compromise your financial privacy.
Also, if you keep the = filters, further rescans do consume the same or less bandwidth than BF = BIP37.
In other words: you have the chance to potentially = increase privacy by consuming bandwidth in the range of a single audio = podcast per week.

I would say the = job of protocol developers is protect users privacy where it=E2=80=99s = possible (as a default).
It=E2=80=99s probably a debatable = point wether 25MB per week of traffic is worth a potential increase in = privacy, though I absolutely think 25MB/week is an acceptable = tradeoff.
Saving traffic is possible by using BIP37 or = stratum/electrum=E2=80=A6 but developers should make sure users are = __warned about the consequences__!

Additionally, it looks like, peer operators are = not endless being willing to serve =E2=80=93 for free =E2=80=93 a = CPU/disk intense service with no benefits for the network. I would = question wether a decentralised form of BIP37 is sustainable in the long = run (if SPV wallet provider bootstrap a net range of NODE_BLOOM peers to = make it more reliable on the network would be snake-oil).



2) It filters = blocks only. It doesn't address unconfirmed transactions.

Well, = unconfirmed transaction are uncertain for various reasons.

BIP158 won't allow you to filter the = mempool.
But as soon as you are connected to the network, you = may fetch tx with inv/getdata and pick out the relevant ones (causes = also traffic).
Unclear and probably impossible with the = current BIP158 specs to fetch transactions that are not in active relay = and are not in a block (mempool txns, at least this is true with the = current observed relay tactics).


3) Afaik, it enforces/encourages = address re-use. This stems from the
fact that the server decides on the filter and in particular = on the
false = positive rate. On wallets with many addresses, a hardcoded = filter
will be too = blurry and thus each block will be matched. So wallets that
follow the "one address per = incoming payment" pattern (e.g. HD wallets)
at some point will be forced to = wrap their key chains back to the
beginning. If I'm wrong on this one please let me = know.

I=E2=80= =99m probably the wrong guy to ask (haven=E2=80=99t made the numbers) = but last time I rescanned a Core wallet (in my dev branch) with block = filters (and a Core wallet has >2000 addresses by default) it fetched = a low and acceptable amount of false positive blocks.
(Maybe = someone who made the numbers step in here.)

Though, large wallets =E2=80=93 AFAIK =E2=80=93 = also operate badly with BIP37.


4) The filters are not yet committed to the blockchain. Until = that
happens we'd = have to trust a server to provide correct filters.

I wouldn=E2=80=99t= say so. It=E2=80=99s on a similar level than BIP37.
BIP37 is = not =E2=80=93 and can not =E2=80=93 be committed to the = blockchain.
You fully trust the peer that it = won=E2=80=99t=E2=80=A6
a) create fake unconfirmed transactions = (would be the same if a BIP158 wallet would show you unconfirmed = transaction)
b) lies by omission (you will miss relevant = transactions, eventually swipe your wallet and loose = coins)

IMO, the point b) is true for = BIP37 and BIP158 (as long as not commited).
In both cases, you = can reduce the trust by comparing between peers / = filter-providers.

b) is conceptually = solvable with a soft-fork (commitment) in BIP158 (not with = BIP37).

Additionally, block-filters = will, very likely, be useful for other features (load/rescan an [old] = wallet on a prune peer that has the filters constructed).



There is probably no clear answer like =E2=80=9EX = is better than Y=E2=80=9C.

Personally = I would like to see developers being more honest/transparent to users = about the implications of the used filtering,... and giving them = choices.
Imagine a user can choose between =E2=80=9EElectrum / = BIP37 / BIP158=E2=80=9C depending on his needs for privacy and = availability of bandwidth. Eventually also taking the future usage of = this wallet (will he load old private keys, will he receive money daily, = etc.) into account.

Plus, full node = hardware appliances that run at home (or in a trusted environment) are = solving many of these issues plus adding a bunch of great features =E2=80=93= if done right.

/jonas
= --Apple-Mail=_82AAF962-04AB-4EEF-B6AE-787E4F6F8CB4-- --Apple-Mail=_EBD2572C-F977-492A-B1E2-46C7CC7DF621 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEyhopCNzi8TB0xizeHrd2uwPHki0FAl060DAACgkQHrd2uwPH ki2M0xAA0SC3LaxoPZTG2SVOY3HHD8m7WYevKZilpVWNm5uARU1Y2EawmWe9ZvG0 HHz62klMs72O+gMbz9lBX7e23DIbqlYSccrmeiySM0NFr8jTLzKV4ezOIK7eUJkn KfqKP6MPdOYbikCZQG/xecxwYPZKOV8Wy8o1d3258qg8eZDOcxJiPa6AhhfehZ9S J0vWfCHf7jv8KjXARdP6M52O76CtITGwQKs7bVAjDnxktNymPEBUmvr7oRcSf6Zl suXw73X/q0C5NvdFes+uu3Q3Oscgu4n0kbLlyVyCFo9UATvCWgoXV/1MlIL9Sbn0 apYxiORwOEBvMAnqnNVrFf+DZHK2smec7sJobDvEUOzl8vE1jAHSiG/uxDy5T8aR mYKeN8OYuqtLAKj1gVYk3D9Kt0RkhO8szHeyk6LCyZKc1TO1CembbTolxcoNq4cF VMNO4DX88ltsB4zJg71+nnFng0HbNJKiEhgWUVyIEks+uWncjz8wzd75s9sPV8Fr feYWFJtJGMjBeTHpcTQthCNuXRGusI3EEkxtvIrKErjOtRWfDAE9BWBDTMACKbp9 Ipxj09t47g1fKj6tIuet7aee7GDmTVfVIHLIbZWUHPd+sWe9e8qnQpIHlehop7hk 0xMvvp9Pcwg/SYGL1VfmWxQMeBTdVqZ3N/YTIA9fWJfX3vyGT8M= =U9aA -----END PGP SIGNATURE----- --Apple-Mail=_EBD2572C-F977-492A-B1E2-46C7CC7DF621--