Received: from sog-mx-2.v43.ch3.sourceforge.com ([172.29.43.192] helo=mx.sourceforge.net) by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1YPA6m-0006oI-K6 for bitcoin-development@lists.sourceforge.net; Sat, 21 Feb 2015 13:28:36 +0000 Received-SPF: pass (sog-mx-2.v43.ch3.sourceforge.com: domain of gmail.com designates 74.125.82.182 as permitted sender) client-ip=74.125.82.182; envelope-from=mh.in.england@gmail.com; helo=mail-we0-f182.google.com; Received: from mail-we0-f182.google.com ([74.125.82.182]) by sog-mx-2.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1YPA6j-000643-SB for bitcoin-development@lists.sourceforge.net; Sat, 21 Feb 2015 13:28:36 +0000 Received: by wevm14 with SMTP id m14so10200772wev.8 for ; Sat, 21 Feb 2015 05:28:27 -0800 (PST) MIME-Version: 1.0 X-Received: by 10.194.93.134 with SMTP id cu6mr4514559wjb.79.1424525307877; Sat, 21 Feb 2015 05:28:27 -0800 (PST) Sender: mh.in.england@gmail.com Received: by 10.194.188.11 with HTTP; Sat, 21 Feb 2015 05:28:27 -0800 (PST) In-Reply-To: References: Date: Sat, 21 Feb 2015 14:28:27 +0100 X-Google-Sender-Auth: oC0HUha2RqjwrGnlYTO3lyDlz2s Message-ID: From: Mike Hearn To: Adam Back Content-Type: multipart/alternative; boundary=047d7bb7092cf31f66050f9924e4 X-Spam-Score: -0.5 (/) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (mh.in.england[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record 1.0 HTML_MESSAGE BODY: HTML included in message 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1YPA6j-000643-SB Cc: Bitcoin Dev Subject: Re: [Bitcoin-development] bloom filtering, privacy X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Feb 2015 13:28:36 -0000 --047d7bb7092cf31f66050f9924e4 Content-Type: text/plain; charset=UTF-8 Let's put the UTXO commitments/anti-fraud proofs to one side for a moment. I would like to see them happen one day, but they aren't critical to these protocols and are just proving to be a distraction. > Then they make fresh random connections to different nodes and request > download of the respective individual transactions from the full node. > ... About privacy the node can make different random connections to > different nodes to fetch addresses ..... The full node cant > correlate the addresses as belonging to the same person by correlating > the download requests for them, because they are made via different > nodes. Apologies for the wall of text, but I don't think this will work nor solve any real problem. And I must justify such a strong statement clearly. *First: technical issues* When you download the per-block Bloom filter and test, what you get back is a set of script elements (addresses, keys, OP_RETURN tags etc). But then in the next step you are saying that you connect to random peers and request individual transactions. We don't know that at this point. All we know are a set of addresses that possibly matched. So I think what you mean is "wallets connect to random peers and request transactions in block N that match a given set of addresses". This is what Bloom filtering already does, of course. Doing the test against the per-block filter first doesn't seem to buy us much because with thousands of transactions per block, even a very tiny FP rate will still trigger a match on every single one. The second problem I see is that we can't do this in parallel because of the following edge case: wallet contains key K and someone sends it money using an OP_CHECKSIG output. The input which spends this output does not contain any predictable data, thus we do not know what to look for in the following blocks to detect a spend of it until we have seen the first transaction and know its hash. In practice this means we must either scan through the chain in sequence and update our matching criteria if we see such an output (this is what the Bloom filtering protocol already does server-side), or we must constrain the user such that output scripts always force repetition of predictable data - this is what mostly happens today due to pay-to-address outputs, but not always, and correctness is more important than completeness. If we can't do it in parallel then we must suffer a node round-trip for every single block we traverse, because we can't request long runs of blocks with a single command. That latency will kill performance dead. It's a non starter. But let's imagine we don't care about OP_CHECKSIG outputs and are willing to ignore them. There are cases where they are the best and most efficient technical solution, but let's put that to one side. The primary difference after making the above changes are that no one node gets a filter containing *all* our keys and addresses. I don't think a per block pre-test filter would gain us much efficiency so from a privacy perspective this is what it boils down to - sharding of the scan. But we can already do this with the current Bloom filtering protocol. BitcoinJ doesn't do so because having multiple parallel scans uses up network IOPs which are a resource of unknown quantity, and because stepping through the chain in parallel with multiple peers complicates the chain sync implementation quite a bit. *Second: this doesn't solve any real problem* Who cares about collecting Bloom filters off the wire? Commercial fraudsters? Doubtful. There are much easier ways to steal money. Spies? Yes! Without a doubt NSA/GCHQ are building or have built databases of IP addresses to Bitcoin addresses and are correlating it via XKEYSCORE with other identifiable information. However, just requesting data from different nodes doesn't help with that, because they are doing DPI and can still see all the connections, so can still combine all the filters or received transactions. Ah, you say, but we're requesting everything via Tor. Yes, about that. We've implemented that already. Some wallets even use it by default, like Alon & Chris' Bitcoin Authenticator wallet. It's just one line of code to activate. Unfortunately there are severe practical problems to using Tor: 1. If you don't have a warm consensus then booting it up is very slow. We're already slower than our competitors like blockchain.info and VISA/MasterCard, we can't make this any worse. This one is possibly not that big a deal and can be solved with more technical tricks. 2. Bitcoin Core's DoS strategy means anyone can block all of Tor quite trivially. So we'd need some complicated fallback mechanism to disable Tor remotely, in case someone did this. 3. Bitcoin wire traffic isn't encrypted or authenticated so it makes it much easier for trolls to tamper with lots of wire traffic at once, whereas without Tor it's much harder. Let's ignore the fact that the Tor project insists on poking the law enforcement bear with rusty nails, and has been receiving tipoffs about plans to seize directory authorities. How much Bitcoin wallets should rely on Tor sticking around is a debate for some other time. There's a much simpler way to fix all of this - add opportunistic encryption to the wire protocol. --047d7bb7092cf31f66050f9924e4 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Let's put the UTXO commitments/anti-fraud proofs to one side for a mom= ent. I would like to see them happen one day, but they aren't critical = to these protocols and are just proving to be a distraction.

=
=C2=A0
Then they make fresh = random connections to different nodes and request
download of the respective individual transactions from the full node.
<= /blockquote>

...

About privacy the node can make different random connections = to
different nodes to fetch addresses .....=C2=A0The full node cant
correlate the addresses as belonging to the same person by correlating
the download requests for them, because they are made via different
nodes.

Apologies for the wall of text, but = I don't think this will work nor solve any real problem. And I must jus= tify such a strong statement clearly.

First: te= chnical issues

When you download the per-block= Bloom filter and test, what you get back is a set of script elements (addr= esses, keys, OP_RETURN tags etc). But then in the next step you are saying = that you connect to random peers and request individual transactions. We do= n't know that at this point. All we know are a set of addresses that po= ssibly matched. So I think what you mean is "wallets connect to random= peers and request transactions in block N that match a given set of addres= ses".=C2=A0

This is what Bloom filtering alre= ady does, of course. Doing the test against the per-block filter first does= n't seem to buy us much because with thousands of transactions per bloc= k, even a very tiny FP rate will still trigger a match on every single one.=

The second problem I see is that we can't do = this in parallel because of the following edge case: wallet contains key K = and someone sends it money using an OP_CHECKSIG output. The input which spe= nds this output does not contain any predictable data, thus we do not know = what to look for in the following blocks to detect a spend of it until we h= ave seen the first transaction and know its hash.=C2=A0

In practice this means we must either scan through the chain in seque= nce and update our matching criteria if we see such an output (this is what= the Bloom filtering protocol already does server-side), or we must constra= in the user such that output scripts always force repetition of predictable= data - this is what mostly happens today due to pay-to-address outputs, bu= t not always, and correctness is more important than completeness.

If we can't do it in parallel then we must suffer a no= de round-trip for every single block we traverse, because we can't requ= est long runs of blocks with a single command. That latency will kill perfo= rmance dead. It's a non starter.

But let's= imagine we don't care about OP_CHECKSIG outputs and are willing to ign= ore them. There are cases where they are the best and most efficient techni= cal solution, but let's put that to one side.

= The primary difference after making the above changes are that no one node = gets a filter containing all=C2=A0our keys and addresses. I don'= t think a per block pre-test filter would gain us much efficiency so from a= privacy perspective this is what it boils down to - sharding of the scan.<= /div>

But we can already do this with the current Bloom = filtering protocol. BitcoinJ doesn't do so because having multiple para= llel scans uses up network IOPs which are a resource of unknown quantity, a= nd because stepping through the chain in parallel with multiple peers compl= icates the chain sync implementation quite a bit.

= Second: this doesn't solve any real problem

=
Who cares about collecting Bloom filters off the wire?

<= /div>
Commercial fraudsters? Doubtful. There are much easier ways to st= eal money.

Spies? Yes! Without a doubt NSA/GCHQ ar= e building or have built databases of IP addresses to Bitcoin addresses and= are correlating it via XKEYSCORE with other identifiable information.

However, just requesting data from different nodes doe= sn't help with that, because they are doing DPI and can still see all t= he connections, so can still combine all the filters or received transactio= ns.

Ah, you say, but we're requesting everythi= ng via Tor.=C2=A0

Yes, about that. We've imple= mented that already. Some wallets even use it by default, like Alon & C= hris' Bitcoin Authenticator wallet. It's just one line of code to a= ctivate.

Unfortunately there are severe practical = problems to using Tor:
  1. If you don't have a warm conse= nsus then booting it up is very slow. We're already slower than our com= petitors like blockchain.info and VI= SA/MasterCard, we can't make this any worse.

    This one is possibl= y not that big a deal and can be solved with more technical tricks.

    =
  2. Bitcoin Core's DoS strategy means anyone can block all of Tor = quite trivially. So we'd need some complicated fallback mechanism to di= sable Tor remotely, in case someone did this.

  3. Bitcoin wire = traffic isn't encrypted or authenticated so it makes it much easier for= trolls to tamper with lots of wire traffic at once, whereas without Tor it= 's much harder.
Let's ignore the fact that the To= r project insists on poking the law enforcement bear with rusty nails, and = has been receiving tipoffs about plans to seize directory authorities. How = much Bitcoin wallets should rely on Tor sticking around is a debate for som= e other time.

There's a much simpler way= to fix all of this - add opportunistic encryption to the wire protocol.
--047d7bb7092cf31f66050f9924e4--