path: root/46/b886c979c7a86915102645598f1d777774b007

Received: from sog-mx-3.v43.ch3.sourceforge.com ([172.29.43.193]
	helo=mx.sourceforge.net)
	by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
	(envelope-from <mh.in.england@gmail.com>) id 1Uz6sY-0006wl-Ej
	for bitcoin-development@lists.sourceforge.net;
	Tue, 16 Jul 2013 15:09:26 +0000
Received-SPF: pass (sog-mx-3.v43.ch3.sourceforge.com: domain of gmail.com
	designates 209.85.219.49 as permitted sender)
	client-ip=209.85.219.49; envelope-from=mh.in.england@gmail.com;
	helo=mail-oa0-f49.google.com; 
Received: from mail-oa0-f49.google.com ([209.85.219.49])
	by sog-mx-3.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
	(Exim 4.76) id 1Uz6sW-0000XE-OW
	for bitcoin-development@lists.sourceforge.net;
	Tue, 16 Jul 2013 15:09:26 +0000
Received: by mail-oa0-f49.google.com with SMTP id n9so955468oag.22
	for <bitcoin-development@lists.sourceforge.net>;
	Tue, 16 Jul 2013 08:09:19 -0700 (PDT)
MIME-Version: 1.0
X-Received: by 10.60.43.226 with SMTP id z2mr2601628oel.76.1373987359327; Tue,
	16 Jul 2013 08:09:19 -0700 (PDT)
Sender: mh.in.england@gmail.com
Received: by 10.76.23.36 with HTTP; Tue, 16 Jul 2013 08:09:19 -0700 (PDT)
In-Reply-To: <8EE501AA-1601-4C28-A32E-80F17D219D3A@grabhive.com>
References: <B87F1213-5BD8-43F5-9744-F69947561ED5@grabhive.com>
	<CANEZrP2xh=m8yWLt-o2UrrUVfU+cUYBuMVxkVFF5mVMtWdRwOQ@mail.gmail.com>
	<C1727B41-AF61-44FA-BE17-5FE4425FDEA8@grabhive.com>
	<CANEZrP0_H9+prDSF92q8a4QzP=fzDM6cTDv0+KcfV9NF9thkmw@mail.gmail.com>
	<3E7894A0-06F3-453D-87F8-975A244EBACF@include7.ch>
	<CANEZrP2jmWkDbpJEm0vd2CKF-prFNbz_ZeNJfDWtSCKb8k5ZXA@mail.gmail.com>
	<2BDA0943-22BB-4405-9AF0-86FB41FD04A6@include7.ch>
	<CANEZrP0McSrVzwv=-qimPyX41EEDmyQdYW5QjPr_i+KWyJZSZw@mail.gmail.com>
	<2F20A509-13A9-4C84-86D7-A15C21BACD53@include7.ch>
	<CANEZrP2yQvmvwP_ZULdS2i+X6L9MeZ+DfidiuZPD2EHwLsN2MA@mail.gmail.com>
	<2A1C412D-414E-4C41-8E20-F0D21F801328@grabhive.com>
	<CANEZrP12V_5Ak0f91RsMziuqXysde102rGeSko=qPBjefy3AeA@mail.gmail.com>
	<8EE501AA-1601-4C28-A32E-80F17D219D3A@grabhive.com>
Date: Tue, 16 Jul 2013 17:09:19 +0200
X-Google-Sender-Auth: -1Mu57FH6OBQ-WH7-CSqrrq5x_w
Message-ID: <CANEZrP0ZG+91Bm9odkNWYja52_MLTgDz6DSGkPzNJLGteDpF-w@mail.gmail.com>
From: Mike Hearn <mike@plan99.net>
To: Wendell <w@grabhive.com>
Content-Type: multipart/alternative; boundary=001a11333dce7a8ff804e1a25c11
X-Spam-Score: -0.5 (/)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
	See http://spamassassin.org/tag/ for more details.
	-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
	sender-domain
	0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
	(mh.in.england[at]gmail.com)
	-0.0 SPF_PASS               SPF: sender matches SPF record
	1.0 HTML_MESSAGE           BODY: HTML included in message
	0.1 DKIM_SIGNED            Message has a DKIM or DK signature,
	not necessarily valid
	-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
X-Headers-End: 1Uz6sW-0000XE-OW
Cc: Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Subject: Re: [Bitcoin-development] SPV bitcoind? (was: Introducing
	BitcoinKit.framework)
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
	<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
	<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Tue, 16 Jul 2013 15:09:26 -0000

--001a11333dce7a8ff804e1a25c11
Content-Type: text/plain; charset=UTF-8

You'd want to create and get merged patches in the following order:

1) Be able to store just block headers in the blkXXXX.dat files instead of
full block contents. At this point you are still *downloading* full blocks,
but they are not being stored. The contents are still sent to the wallet
for extracting relevant transactions though (see SyncWithWallets).  You
also need to disable listening and addr announcements to the P2P network at
this point. You need to be able to re-org and do all the usual things
without storing block contents. You also need to short-circuit the leveldbs
so they aren't created or used. All that needs to be unit tested. You need
to also rewrite the mempool logic so it throws out irrelevant transactions.
The RPC interface needs to adjust itself so you can't try to start mining,
query the utxo set, etc.

At this point you have an SPV node, albeit one that still downloads the
entire block chain. However total disk storage used will be much lower.
Getting this written and reviewed is a big chunk of work but is the hardest
part. Once it's done you can breath easy.

2) Next step, use getheaders to catch up with the chain until the
min(wallet birthdays) is reached. You can see in Satoshi's patch where he
adds support for receiving "headers" messages. Because key times are
recorded as dates and you don't know the dates of blocks in advance, you
need to download headers until you see one that goes past the key birthday
minus some slack period, then throw out the headers you downloaded and
switch to downloading full blocks again from that point onwards.

3) Next step, implement client side support for Bloom filtering. Switch
from downloading full blocks to filteredblocks, verify the Merkle branches
then apply them to the wallet. Watch out for accidental re-orderings of
transactions here from block order (e.g. if you accidentally insert them
into a std::map or other unordered collection it can lead to bugs). Come up
with some way to decide on a FP rate. Probably you want a fairly high FP
rate for desktop wallets.

4) Next step (optional), implement monitoring of broadcast propagation for
transactions that are received. SPV clients cannot verify unconfirmed
transactions so you can either just give up entirely and accept any old
garbage, or assume a non-MITMd internet connection and use network
propagation as a rough proxy for "likely to be valid and mined upon".

4) Optimize!

How much you need to optimize really depends on a lot of things. I found
that to be competitive with Electrum/blockchain.info I had to do a ton of
optimizations including very aggressive checkpointing so new users don't
have to download more than a month or twos worth of headers, as downloading
all the headers was becoming a bottleneck. You'd need to download about
16mb+ of data at the moment to grab all the headers and on a weakass mobile
phone with a weak Dalvik VM and 3G internet this was way too much. I also
had to spend some time profiling to ensure we weren't accidentally
thrashing the UI due to too-fast updates, we weren't bottlenecking on
updating last seen block data in the wallet, we weren't accidentally
de/reserializing messages redundantly etc.

After about 3-4 evenings of non-stop profiling and optimising I ended up
with a relatively flat profile whilst doing initial catchup and chain sync.
On a desktop I bet you can get away with much less optimisation because
your CPUs, network and disk tend to be much stronger.



On Tue, Jul 16, 2013 at 4:16 PM, Wendell <w@grabhive.com> wrote:

> Hello everyone,
>
> In the previous thread, I expressed interest in seeing an SPV bitcoind,
> further stating that I would fund such work. Mike Hearn followed up with
> some of Satoshi's old code for this, which is now quite broken. The offer
> and interest on my side still stand, as more diversity in SPV options seems
> like the right way to go.
>
> Time-permitting, I would really appreciate feedback from knowledgable
> parties about the possible approaches to an SPV bitcoind. We at Hive
> ideally want to see something that could one be merge into master, rather
> than a fork.
>
> -wendell
>
> grabhive.com | twitter.com/grabhive
>
>
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

--001a11333dce7a8ff804e1a25c11
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">You&#39;d want to create and get merged patches in the fol=
lowing order:<div><br></div><div>1) Be able to store just block headers in =
the blkXXXX.dat files instead of full block contents. At this point you are=
 still *downloading* full blocks, but they are not being stored. The conten=
ts are still sent to the wallet for extracting relevant transactions though=
 (see SyncWithWallets). =C2=A0You also need to disable listening and addr a=
nnouncements to the P2P network at this point. You need to be able to re-or=
g and do all the usual things without storing block contents. You also need=
 to short-circuit the leveldbs so they aren&#39;t created or used. All that=
 needs to be unit tested. You need to also rewrite the mempool logic so it =
throws out irrelevant transactions. The RPC interface needs to adjust itsel=
f so you can&#39;t try to start mining, query the utxo set, etc.</div>
<div><br></div><div>At this point you have an SPV node, albeit one that sti=
ll downloads the entire block chain. However total disk storage used will b=
e much lower. Getting this written and reviewed is a big chunk of work but =
is the hardest part. Once it&#39;s done you can breath easy.</div>
<div><br></div><div>2) Next step, use getheaders to catch up with the chain=
 until the min(wallet birthdays) is reached. You can see in Satoshi&#39;s p=
atch where he adds support for receiving &quot;headers&quot; messages. Beca=
use key times are recorded as dates and you don&#39;t know the dates of blo=
cks in advance, you need to download headers until you see one that goes pa=
st the key birthday minus some slack period, then throw out the headers you=
 downloaded and switch to downloading full blocks again from that point onw=
ards.</div>
<div><br></div><div>3) Next step, implement client side support for Bloom f=
iltering. Switch from downloading full blocks to filteredblocks, verify the=
 Merkle branches then apply them to the wallet. Watch out for accidental re=
-orderings of transactions here from block order (e.g. if you accidentally =
insert them into a std::map or other unordered collection it can lead to bu=
gs). Come up with some way to decide on a FP rate. Probably you want a fair=
ly high FP rate for desktop wallets.</div>
<div><br></div><div>4) Next step (optional), implement monitoring of broadc=
ast propagation for transactions that are received. SPV clients cannot veri=
fy unconfirmed transactions so you can either just give up entirely and acc=
ept any old garbage, or assume a non-MITMd internet connection and use netw=
ork propagation as a rough proxy for &quot;likely to be valid and mined upo=
n&quot;.</div>
<div><br></div><div>4) Optimize!</div><div><br></div><div>How much you need=
 to optimize really depends on a lot of things. I found that to be competit=
ive with Electrum/<a href=3D"http://blockchain.info">blockchain.info</a> I =
had to do a ton of optimizations including very aggressive checkpointing so=
 new users don&#39;t have to download more than a month or twos worth of he=
aders, as downloading all the headers was becoming a bottleneck. You&#39;d =
need to download about 16mb+ of data at the moment to grab all the headers =
and on a weakass mobile phone with a weak Dalvik VM and 3G internet this wa=
s way too much. I also had to spend some time profiling to ensure we weren&=
#39;t accidentally thrashing the UI due to too-fast updates, we weren&#39;t=
 bottlenecking on updating last seen block data in the wallet, we weren&#39=
;t accidentally de/reserializing messages redundantly etc.</div>
<div><br></div><div>After about 3-4 evenings of non-stop profiling and opti=
mising I ended up with a relatively flat profile whilst doing initial catch=
up and chain sync. On a desktop I bet you can get away with much less optim=
isation because your CPUs, network and disk tend to be much stronger.</div>
<div><br></div></div><div class=3D"gmail_extra"><br><br><div class=3D"gmail=
_quote">On Tue, Jul 16, 2013 at 4:16 PM, Wendell <span dir=3D"ltr">&lt;<a h=
ref=3D"mailto:w@grabhive.com" target=3D"_blank">w@grabhive.com</a>&gt;</spa=
n> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hello everyone,<br>
<br>
In the previous thread, I expressed interest in seeing an SPV bitcoind, fur=
ther stating that I would fund such work. Mike Hearn followed up with some =
of Satoshi&#39;s old code for this, which is now quite broken. The offer an=
d interest on my side still stand, as more diversity in SPV options seems l=
ike the right way to go.<br>

<br>
Time-permitting, I would really appreciate feedback from knowledgable parti=
es about the possible approaches to an SPV bitcoind. We at Hive ideally wan=
t to see something that could one be merge into master, rather than a fork.=
<br>

<br>
-wendell<br>
<br>
<a href=3D"http://grabhive.com" target=3D"_blank">grabhive.com</a> | <a hre=
f=3D"http://twitter.com/grabhive" target=3D"_blank">twitter.com/grabhive</a=
><br>
<br>
<br>
---------------------------------------------------------------------------=
---<br>
See everything from the browser to the database with AppDynamics<br>
Get end-to-end visibility with application monitoring from AppDynamics<br>
Isolate bottlenecks and diagnose root cause in seconds.<br>
Start your free trial of AppDynamics Pro today!<br>
<a href=3D"http://pubads.g.doubleclick.net/gampad/clk?id=3D48808831&amp;iu=
=3D/4140/ostg.clktrk" target=3D"_blank">http://pubads.g.doubleclick.net/gam=
pad/clk?id=3D48808831&amp;iu=3D/4140/ostg.clktrk</a><br>
_______________________________________________<br>
Bitcoin-development mailing list<br>
<a href=3D"mailto:Bitcoin-development@lists.sourceforge.net">Bitcoin-develo=
pment@lists.sourceforge.net</a><br>
<a href=3D"https://lists.sourceforge.net/lists/listinfo/bitcoin-development=
" target=3D"_blank">https://lists.sourceforge.net/lists/listinfo/bitcoin-de=
velopment</a><br>
</blockquote></div><br></div>

--001a11333dce7a8ff804e1a25c11--