Received-SPF: pass (sog-mx-4.v43.ch3.sourceforge.com: domain of gmail.com
	designates 209.85.219.44 as permitted sender)
	client-ip=209.85.219.44; envelope-from=mh.in.england@gmail.com;
	helo=mail-oa0-f44.google.com; 
MIME-Version: 1.0
Sender: mh.in.england@gmail.com
In-Reply-To: <2F20A509-13A9-4C84-86D7-A15C21BACD53@include7.ch>
References: <B87F1213-5BD8-43F5-9744-F69947561ED5@grabhive.com>
	<CANEZrP2xh=m8yWLt-o2UrrUVfU+cUYBuMVxkVFF5mVMtWdRwOQ@mail.gmail.com>
	<C1727B41-AF61-44FA-BE17-5FE4425FDEA8@grabhive.com>
	<CANEZrP0_H9+prDSF92q8a4QzP=fzDM6cTDv0+KcfV9NF9thkmw@mail.gmail.com>
	<3E7894A0-06F3-453D-87F8-975A244EBACF@include7.ch>
	<CANEZrP2jmWkDbpJEm0vd2CKF-prFNbz_ZeNJfDWtSCKb8k5ZXA@mail.gmail.com>
	<2BDA0943-22BB-4405-9AF0-86FB41FD04A6@include7.ch>
	<CANEZrP0McSrVzwv=-qimPyX41EEDmyQdYW5QjPr_i+KWyJZSZw@mail.gmail.com>
	<2F20A509-13A9-4C84-86D7-A15C21BACD53@include7.ch>
Date: Tue, 16 Jul 2013 11:51:57 +0200
Message-ID: <CANEZrP2yQvmvwP_ZULdS2i+X6L9MeZ+DfidiuZPD2EHwLsN2MA@mail.gmail.com>
From: Mike Hearn <mike@plan99.net>
To: Jonas Schnelli <jonas.schnelli@include7.ch>, 
	Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Content-Type: multipart/alternative; boundary=047d7b67812685670d04e19ded42
Subject: Re: [Bitcoin-development] Introducing BitcoinKit.framework
Precedence: list

--047d7b67812685670d04e19ded42
Content-Type: text/plain; charset=UTF-8

Let's re-add the list as this is a topic of general interest.

On Tue, Jul 16, 2013 at 11:36 AM, Jonas Schnelli <jonas.schnelli@include7.ch
> wrote:

> What do you think about extending the BitcoinKit.framework (based on
> bitcoind) so that the kit could handle the very basic SPV concept
> (getHeaders aka fast catchup, wallet-birthday, bloom filters)?
> Maybe it would be possible to only port some of the bitcoinj work to c++
> (and use it for extending BitcoinKit or bitcoind)?
> Maybe then it could also be a starting point for someone who likes to
> create a SPV mode for bitcoind?
>

Making bitcoind/Bitcoin-Qt support SPV mode was the original plan some
years ago, Satoshi even sent me some code he wrote that did the first
parts, but it was incomplete.

At the time, I decided to do a separate implementation for a few different
reasons. One is that my understanding of his code wasn't so good back then
and I lacked confidence to change it. Especially as there were no unit
tests back then (and still aren't any for most of it), making invasive
changes to the core validation code was and is highly risky. A separate
code base seemed to reduce the risk a lot.

Another reason is that Satoshi encouraged me to write a simple
re-implementation that people could learn from. And I wanted a documented,
object oriented API that people could use to build a variety of apps.

Yet another reason was bitcoind is security critical code that scrapes
complex data structures from untrusted sources on the internet, and it's
written in an unmanaged language. Ordinarily this would be a recipe for
disaster as a single overflow or memory management error could lead to
hacking and theft on a massive scale. It's like taking a chainsaw and using
it to carve an ice sculpture. Satoshi, incredibly, pulled it off, mostly by
using advanced C++ features that made his code hard to read for many people
and by being very, very careful. I was not convinced I could do such a good
job and was worried about accidentally introducing vulnerabilities.

A final reason is that it was clear that the bitcoind codebase would need
serious changes for mobiles, beyond that required for ordinary SPV support.
For example, Satoshi's code assumes it has access to block headers via a
std::map and that assumption is made in a lot of places. On Android phones,
you can't fit all the block headers in RAM. bitcoinj uses a circular ring
buffer of the last N thousand headers for this reason. It's quite different
to how bitcoind works.

All that said, it was a ton of work and it's still unclear that it was the
right call.

Anyway, your situation is a little different. Firstly you don't care about
mobiles, your app is intended for desktops. So the changes required are
less invasive. Also, there are more unit tests and more people with a good
understanding of the code these days, so perhaps the risk of introducing
bugs is lower. And these days we have some nice APIs for building apps so
that need is already met.

If you wanted to implement SPV mode in bitcoind, Gavin or I could send you
Satoshi's old patch although of course it is no longer usable. It would
indicate the basic cut lines though.

--047d7b67812685670d04e19ded42
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Let&#39;s re-add the list as this is a topic of general in=
terest.<div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, Ju=
l 16, 2013 at 11:36 AM, Jonas Schnelli <span dir=3D"ltr">&lt;<a href=3D"mai=
lto:jonas.schnelli@include7.ch" target=3D"_blank">jonas.schnelli@include7.c=
h</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div style=3D"word-wrap:break-word"><div><di=
v class=3D"im"></div></div><div>What do you think about extending the Bitco=
inKit.framework (based on bitcoind) so that the kit could handle the very b=
asic SPV concept (getHeaders aka fast catchup, wallet-birthday, bloom filte=
rs)?<br>
</div><div>Maybe it would be possible to only port some of the bitcoinj wor=
k to c++ (and use it for extending BitcoinKit or bitcoind)?</div><div>Maybe=
 then it could also be a starting point for someone who likes to create a S=
PV mode for bitcoind?</div>
</div></blockquote><div><br></div><div>Making bitcoind/Bitcoin-Qt support S=
PV mode was the original plan some years ago, Satoshi even sent me some cod=
e he wrote that did the first parts, but it was incomplete.</div><div><br>
</div><div>At the time, I decided to do a separate implementation for a few=
 different reasons. One is that my understanding of his code wasn&#39;t so =
good back then and I lacked confidence to change it. Especially as there we=
re no unit tests back then (and still aren&#39;t any for most of it), makin=
g invasive changes to the core validation code was and is highly risky. A s=
eparate code base seemed to reduce the risk a lot.</div>
<div><br></div><div>Another reason is that Satoshi encouraged me to write a=
 simple re-implementation that people could learn from. And I wanted a docu=
mented, object oriented API that people could use to build a variety of app=
s.</div>
<div><br></div><div>Yet another reason was bitcoind is security critical co=
de that scrapes complex data structures from untrusted sources on the inter=
net, and it&#39;s written in an unmanaged language. Ordinarily this would b=
e a recipe for disaster as a single overflow or memory management error cou=
ld lead to hacking and theft on a massive scale. It&#39;s like taking a cha=
insaw and using it to carve an ice sculpture. Satoshi, incredibly, pulled i=
t off, mostly by using advanced C++ features that made his code hard to rea=
d for many people and by being very, very careful. I was not convinced I co=
uld do such a good job and was worried about accidentally introducing vulne=
rabilities.</div>
<div><br></div><div>A final reason is that it was clear that the bitcoind c=
odebase would need serious changes for mobiles, beyond that required for or=
dinary SPV support. For example, Satoshi&#39;s code assumes it has access t=
o block headers via a std::map and that assumption is made in a lot of plac=
es. On Android phones, you can&#39;t fit all the block headers in RAM. bitc=
oinj uses a circular ring buffer of the last N thousand headers for this re=
ason. It&#39;s quite different to how bitcoind works.</div>
<div><br></div><div>All that said, it was a ton of work and it&#39;s still =
unclear that it was the right call.</div><div><br></div><div>Anyway, your s=
ituation is a little different. Firstly you don&#39;t care about mobiles, y=
our app is intended for desktops. So the changes required are less invasive=
. Also, there are more unit tests and more people with a good understanding=
 of the code these days, so perhaps the risk of introducing bugs is lower. =
And these days we have some nice APIs for building apps so that need is alr=
eady met.</div>
<div><br></div><div>If you wanted to implement SPV mode in bitcoind, Gavin =
or I could send you Satoshi&#39;s old patch although of course it is no lon=
ger usable. It would indicate the basic cut lines though.=C2=A0</div></div>=
</div>
</div>

--047d7b67812685670d04e19ded42--