path: root/3f/4a2b57a2b99d3b8fa7524674434d7971a71f63

Return-Path: <david.vorick@gmail.com>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
	[172.17.192.35])
	by mail.linuxfoundation.org (Postfix) with ESMTPS id 59C4C89C
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Wed, 19 Apr 2017 17:30:35 +0000 (UTC)
X-Greylist: whitelisted by SQLgrey-1.7.6
Received: from mail-wm0-f47.google.com (mail-wm0-f47.google.com [74.125.82.47])
	by smtp1.linuxfoundation.org (Postfix) with ESMTPS id DC2AD202
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Wed, 19 Apr 2017 17:30:32 +0000 (UTC)
Received: by mail-wm0-f47.google.com with SMTP id r190so28533342wme.1
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Wed, 19 Apr 2017 10:30:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
	h=mime-version:in-reply-to:references:from:date:message-id:subject:cc; 
	bh=yoibd5brTMadOrKSEblPzUM7M9Fz1TNgxzjQRTZkrS0=;
	b=gbgJS8liWilIu/gp2eGYohjh5KBtE3ZczVblT+zaG6I2DvdXcwlJQ+mfN6eVrehdOk
	XqLRUsOmP0ZNZO81rQ9QQD6WAbOVI1DiaFMhmm1//aSBDvBtohKFVQdHpSGinqt1Z5Rl
	JKHV1f2FCLOReRBmHCWTd1j8a/xVLrcxwvd4EgfS+iBKbEX1IZkO+dCSwKceRtcR0puF
	Ggei3EoZwPHZdaADnSThJV26J05NcE2sjeOJefQ/f8gwR8Bdz7wJGWbvCnaDAC0NTL9A
	bMahd7xyXpb9gbAOWYo2UV1ujlpldBz+qTQX6RMoCFHQKALqN7YXfvP5sjAzGp5Ps3Tv
	siCQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20161025;
	h=x-gm-message-state:mime-version:in-reply-to:references:from:date
	:message-id:subject:cc;
	bh=yoibd5brTMadOrKSEblPzUM7M9Fz1TNgxzjQRTZkrS0=;
	b=bsAgXKch9TB1wOlgaXNbVHRqY0rP7NvvYqW1dTf7xLo+8EYOhL97F1CzAeK6vSk9nA
	JXq1W8/W4XfeOdLRp2sSVEiib2kzNu/L2cDrZW6ETIIrtpEgUwpN1EMZGG0LBHaAfrJ0
	XxikyysHQNBs+DJFpijsP4cm5UvEh8di/dH2mmd/ADzvhUrs6+yP6ZQVhfLm0YTuupdV
	iUQyycNMNLDIV20p0d1Nr8Ujv4p9dXc/3o3Pg0J/eZX/xZ4sqi2s3AQ8eoy3d2s1B7X/
	4bsJiVNaUzBRvgB+JAfF5nT3Ms/JlubZESg5hZh9m4joyL4xCQYoCdgylkKoxlEBv1TO
	yRiA==
X-Gm-Message-State: AN3rC/4R/GPP7GQTnXP5qG9fd1+nwLyiEvksTPkmU+bHEU9BacWCdTVm
	ca/B2nMd0it3kbwAl7mMHb2MVrxWww==
X-Received: by 10.28.97.2 with SMTP id v2mt4014330wmb.88.1492623031148; Wed,
	19 Apr 2017 10:30:31 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.28.45.19 with HTTP; Wed, 19 Apr 2017 10:30:30 -0700 (PDT)
In-Reply-To: <19dbfef2-3791-8fe7-1c00-c4052c3d6c45@gmail.com>
References: <CAFVRnypbQQ-vsSLqv48cYaqTCty4R1DmFRqfAvxe4mAqyQNXxQ@mail.gmail.com>
	<19dbfef2-3791-8fe7-1c00-c4052c3d6c45@gmail.com>
From: David Vorick <david.vorick@gmail.com>
Date: Wed, 19 Apr 2017 13:30:30 -0400
Message-ID: <CAFVRnyrQ3CMPW0=dtR-xnW1bF8cD9o5yvD67w25=w9wxVyJT9w@mail.gmail.com>
Content-Type: multipart/alternative; boundary=001a114a4c868e727e054d88616c
X-Spam-Status: No, score=-1.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,HTML_MESSAGE,MISSING_HEADERS,
	RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
	smtp1.linux-foundation.org
Cc: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] Small Nodes: A Better Alternative to Pruned Nodes
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Wed, 19 Apr 2017 17:30:35 -0000

--001a114a4c868e727e054d88616c
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On Tue, Apr 18, 2017 at 3:43 AM, Jonas Schnelli <dev@jonasschnelli.ch>
wrote:

>
> Hi Dave
>
> *1. I agree that we need to have a way for pruned nodes to partially serv=
e
> historical blocks.*
> My personal measurements told me that around ~80% of historical block
> serving are between tip and -1=E2=80=99000 blocks.
> Currently, Core nodes have only two modes of operations, =E2=80=9Eserver =
all
> historical blocks=E2=80=9C or =E2=80=9Enone=E2=80=9C.
> This makes little sense especially if you prune to a target size of, lets
> say, 80GB (~80% of the chain).
> Ideally, there would be a mode where your full node can signal a third
> mode =E2=80=9EI keep the last 1000 blocks=E2=80=9C (or make this more dyn=
amic).
>

That probably makes sense with small nodes too. The past 1000 blocks are
such a small footprint compared to the rest of the chain.


>
> *2. Bootstrapping new peers*
> I=E2=80=99m not sure if full nodes must be the single point of historical=
 data
> storage. Full nodes provide a valuable service (verification, relay,
> filtering, etc.). I=E2=80=99m not sure if serving historical blocks is on=
e of them.
> Historical blocks could be made available on CDN=E2=80=99s or other file =
storage
> networks. You are going to verify them anyways,... the serving part is pu=
re
> data storage.
> I=E2=80=99m also pretty sure that some users have stopping running full n=
odes
> because their upstream bandwidth consumption (because of serving historic=
al
> blocks) was getting intolerable.
> Especially =E2=80=9Econsumer=E2=80=9C peers must have been hit by this (l=
ittle experience
> in how to reduce traffic, upstream in general is bad for
> consumers-connections, little resources in general).
>

Perhaps it is not, but I would think that it would be pretty
straightforward to configure a global bandwidth limit within Bitcoin. I
know many torrent clients, and clients for protocols like Tor and i2p
include the ability to set both speed limits and monthly bandwidth limits.
Shipping core with sane default limits is probably sufficient to solve
bandwidth issues for most users. I don't know if default limits may result
in today's archive nodes pulling less weight though - changing the defaults
to have limits may slow the network down as a whole.

In my experience (living in a city where most people have uncapped
connections), disk usage is usually the bigger issue, but certainly
bandwidth is a known problem (especially for rural users) as well.


>
> Having a second option built into full nodes (or as an external bootstrap
> service/app) how to download historical blocks during bootstrapping could
> probably be a relieve for "small nodes=E2=80=9C.
> It could be a little daemon that downloads historical blocks from CDN=E2=
=80=99s,
> etc. and feeds them into your full node over p2p/8333 and kickstarts your
> bootstrapping without bothering valuable peers.
> Or, the alternative download, could be built into the full nodes main
> logic.
> And, if it wasn=E2=80=99t obvious, this must not bypass the verification!
>

I worry about any type of CDN being a central point of failure. CDNs cost
money, which means someone is footing the bill. Torrenting typically relies
on a DHT, which is much easier to attack than Bitcoin's peer network. It's
possible that a decentralized CDN could be used, but I don't think any yet
exist (though I am building one for unrelated reasons) which are both
sufficiently secure and incentive-compatible to be considered as an
alternative to using full nodes to bootstrap.

I just don't want to end up in a situation where 90% of users are getting
their blockchain from the same 3 websites or centralized services. And I
also don't want to rely on any p2p solution which would not stand up to a
serious adversary. Right now, I think the bitcoin p2p network is by
significant margin the best we've got. The landscape for decentralized data
distribution is evolving rapidly though, perhaps in a few years there will
be a superior alternative.


> *To your proposal:*
> - Isn=E2=80=99t there a tiny finger-printing element if peers have to pic=
k an
> segmentation index?
> - SPV bloom filter clients can=E2=80=99t use fragmented blocks to filter =
txns?
> Right? How could they avoid connecting to those peers?
>
> </jonas>
>

Yes, there is finger-print that happens if you have nodes pick an index.
And the fingerprint gets a lot worse if you have a node pick multiple
indexes. Though, isn't it already required that nodes have some sort of IP
address or hidden service domain? I want to say that the fingerprint
created by picking an index is not a big deal, because it can be separated
from activity like transaction relaying and mining. Though, I am not
certain and perhaps it is a problem.

To be honest, I hadn't really considered SPV nodes at the time of writing.
Small nodes would still be seeing all of the new blocks, and per your
suggestion may also be storing the 1000 or so most recent blocks, but SPV
nodes would still need some way to find all of their historical
transactions. The problem is not fetching blocks, it's figuring out which
blocks are worth fetching. It may be sufficient to have nodes store a bloom
filter for each block indicating which addresses had activity. The bloom
filter would probably only need to be about 1% of the size of the full
block. That's not too much overhead (now you are storing 21% of the block
instead of just 20%), and would retain SPV compatibility.

On Mon, Apr 17, 2017 at 12:17 PM, praxeology_guy <
praxeology_guy@protonmail.com> wrote:

>
> FYI With unspent coin snapshots, needing archive data becomes less
> important.  People can synchronize from a later snapshot instead of the
> genesis block.  See https://petertodd.org/2016/delayed-txo-commitments
> for my current favorite idea.
>
>
This is something that I think could help a lot too. If the build processes
included verifying the chain and then creating a utxo snapshot at say,
block 400,000, then nodes would no longer need to download, store, upload,
or share blocks prior to that height. It means that a reorg deeper than
that point would hardfork the network. But a reorg 60k blocks deep is going
to cause problems worse than a network split. Then, the only people who
would ever need to care about the early blocks are developers, and it's
more reasonable to expect developers to go through a longer process and
have more resources than everyday users.

On Mon, Apr 17, 2017 at 6:14 AM, Aymeric Vitte via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> While I fully agree with the intent (increasing full nodes so a big miner
> waking up in a bad mood can't threaten the world any longer every day as =
it
> is now) I am not sure to get the interest of this proposal, because:
>
> - it's probably not a good idea to encourage the home users to run full
> nodes, there are many people running servers far from their capacity that
> could easily run efficient full nodes
>
Running a full node is the only way to avoid needing to trust others. It's
also how you make your opinion worthwhile for events like hard forks and
UASFs. If decentralization is the primary motivation, it absolutely makes
sense to encourage people to run their own full nodes. Without a full node,
you are at the mercy of the governance decisions by those who do have full
nodes. But if you have a full node, you can chose to opt-out of any upgrade
(example: ethereum classic nodes).


> - if someone can't allocate 100 GB today to run a full node, then we can'=
t
> expect him to allocate more in the future
>
That's why I'm proposing something to decrease the storage requirements.


> - this proposal is a kind of reinventing torrents, while limiting the
> number of connections to something not efficient at all, I don't see why
> something that is proven to be super efficient (torrents) would be needed
> to be reinvented, I am not saying that it should be used as the bittorren=
t
> network is doing but the concepts can be reused
>
It's different from torrents in that it uses specialized erasure coding to
make sure that every block is always available, even if an adversary is
running around targeting all the nodes with a particular piece.


> - I don't get at all the concept of "archival" nodes since it's another
> useless step toward centralization
>
"archival" nodes are simply nodes with the full blockchain. Nobody can
bootstrap on the network without them. Today, every bitcoin-core node is an
archival node by default.

> I think the only way to increase full nodes it to design an incentive for
> people to run them
>
The primary incentive is the sovereignty that it gives you. Running a
Bitcoin full node gives you security and protection against political
garbage that you can't get any other way. The network does currently depend
on altruism to allow people to download the blockchain, but as long as we
can keep the resource requirements of this altruism low, I think we can
expect it to continue. This proposal attempts to keep those requirements
low.



On Tue, Apr 18, 2017 at 6:50 AM, Tom Zander <tomz@freedommail.ch> wrote:

> On Monday, 17 April 2017 08:54:49 CEST David Vorick via bitcoin-dev wrote=
:
> > The best alternative today to storing the full blockchain is to run a
> > pruned node
>
> The idea looks a little overly complex to me.
>
> I suggested something similar which is a much simpler version;
> https://zander.github.io/scaling/Pruning/
>
> > # Random pruning mode
> >
> > There is a large gap between the two current modes of everything
> > (currently 75GB) and only what we need (2GB or so).
> >
> > This mode would have two areas, it would keep a days worth of blocks to
> > make sure that any reorgs etc would not cause a re-download, but it wou=
ld
> > have additionally have an area that can be used to store historical dat=
a
> > to be shared on the network. Maybe 20 or 50GB.
> >
> > One main feature of Bitcoin is that we have massive replication. Each
> node
> > currently holds all the same data that every other node holds. But this
> > doesn't have to be the case with pruned nodes. A node itself has no nee=
d
> > for historic data at all.
> >
> > The suggestion is that a node stores a random set of blocks. Dropping
> > random blocks as the node runs out of disk-space. Additionally, we woul=
d
> > introduce a new way to download blocks from other nodes which allows th=
e
> > node to say it doesn't actually have the block requested.
> >
> > The effect of this setup is that many different nodes together end up
> > having the total amount of blocks, even though each node only has a
> > fraction of the total amount.
>
> --
> Tom Zander
> Blog: https://zander.github.io
> Vlog: https://vimeo.com/channels/tomscryptochannel
>

Your proposal has a significant disadvantage: If every peer is dropping 75%
of all blocks randomly, then you need to connect to a large number of peers
to download the whole blockchain. Any given peer has a 25% chance of
holding a block, or rather, a 75% chance of not holding a block. If you
have n peers, your probability of not being able to download a given block
is 0.75^n. If you are downloading 450,000 blocks, you will need to connect
to an expected 46 peers to download the whole blockchain.

Your proposal is also a lot less able to handle active adversaries: if
nodes are randomly dropping blocks, the probability that one block in
particular is dropped by everyone goes up significantly. And the problem
gets a lot worse in the presence of an active adversary. If there are 8000
nodes each dropping 75% of the blocks, then each block on average will only
be held by 2000 nodes. Probabilistically, some unlucky blocks will be held
by fewer than 2000 nodes. An active adversary needs only to eliminate about
2000 nodes (a chosen 2000 nodes) to knock a block off of the network. But
missing even a single block is a significant problem.

Your proposal essentially requires that archive nodes still exist and be a
part of a typical blockchain download. Given that, I don't think it would
be a sound idea to ship as a default in bitcoin core.



On Tue, Apr 18, 2017 at 9:07 AM, Tier Nolan via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> This has been discussed before.
>
> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/
> 2015-May/008101.html
>
> including a list of nice to have features by Maxwell
>
> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/
> 2015-May/008110.html
>
> You meet most of these rules, though you do have to download blocks from
> multiple peers.
>
> The suggestion in that thread were for a way to compactly indicate which
> blocks a node has.  Each node would then store a sub-set of all the
> blocks.  You just download the blocks you want from the node that has the=
m.
>
> Each node would be recommended to store the last few days worth anyway.
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
I believe that my proposal does meet all of the requirements listed by
Maxwell. Having a set of 8 random peers gives you a very high probability
of being able to recover every single block. You would need to connect to
at least 5 peers (and this is already >90% likely to be sufficient to
recover every block), but if you cannot connect to 5 random peers your node
is probably in trouble anyway. Highly parallel, high speed downloads are
just as possible with small nodes as with archive nodes. It only takes a
few bytes to indicate which part of the blockchain you have, and any 2
peers have a less than 1% chance of overlapping.

--001a114a4c868e727e054d88616c
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On T=
ue, Apr 18, 2017 at 3:43 AM, Jonas Schnelli <span dir=3D"ltr">&lt;<a target=
=3D"_blank" href=3D"mailto:dev@jonasschnelli.ch">dev@jonasschnelli.ch</a>&g=
t;</span> wrote:<br><blockquote style=3D"margin:0px 0px 0px 0.8ex;border-le=
ft:1px solid rgb(204,204,204);padding-left:1ex" class=3D"gmail_quote"><br><=
div style=3D"word-wrap:break-word">Hi Dave<span class=3D"gmail-"><br></span=
><br><b>1. I agree that we need to have a way for pruned nodes to partially=
 serve historical blocks.</b><br>My personal=C2=A0measurements told me that=
 around ~80% of historical block serving are between tip and -1=E2=80=99000=
 blocks.<br>Currently, Core nodes have only two modes of operations, =E2=80=
=9Eserver all historical blocks=E2=80=9C or =E2=80=9Enone=E2=80=9C.<br>This=
 makes little sense especially if you prune to a target size of, lets say, =
80GB (~80% of the chain).<br>Ideally,
 there would be a mode where your full node can signal a third mode =E2=80=
=9EI=20
keep the last 1000 blocks=E2=80=9C (or make this more dynamic).</div></bloc=
kquote><div><br></div><div>That probably makes sense with small nodes too. =
The past 1000 blocks are such a small footprint compared to the rest of the=
 chain.<br></div><div>=C2=A0</div><blockquote style=3D"margin:0px 0px 0px 0=
.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class=3D"gmai=
l_quote"><div style=3D"word-wrap:break-word"><div><br><b>2. Bootstrapping n=
ew peers</b><br>I=E2=80=99m
 not sure if full nodes must be the single point of historical data=20
storage. Full nodes provide a valuable service (verification, relay,=20
filtering, etc.). I=E2=80=99m not sure if serving historical blocks is one=
=20
of=C2=A0them. Historical blocks could be made available on CDN=E2=80=99s or=
 other=20
file storage networks. You are going to verify them anyways,... the=20
serving part is pure data storage.<br>I=E2=80=99m also pretty sure that som=
e=20
users have stopping running full nodes because their upstream bandwidth=20
consumption (because of serving historical blocks) was=20
getting=C2=A0intolerable.<div>Especially =E2=80=9Econsumer=E2=80=9C peers m=
ust have been hit=20
by this (little experience in how to reduce traffic, upstream in general
 is bad for consumers-connections, little resources in general).</div></div=
></div></blockquote><div><br></div><div>Perhaps it is not, but I would thin=
k that it would be pretty straightforward to configure a global bandwidth l=
imit within Bitcoin. I know many torrent clients, and clients for protocols=
 like Tor and i2p include the ability to set both speed limits and monthly =
bandwidth limits. Shipping core with sane default limits is probably suffic=
ient to solve bandwidth issues for most users. I don&#39;t know if default =
limits may result in today&#39;s archive nodes pulling less weight though -=
 changing the defaults to have limits may slow the network down as a whole.=
<br><br></div><div>In my experience (living in a city where most people hav=
e uncapped connections), disk usage is usually the bigger issue, but certai=
nly bandwidth is a known problem (especially for rural users) as well.<br><=
/div><div>=C2=A0</div><blockquote style=3D"margin:0px 0px 0px 0.8ex;border-=
left:1px solid rgb(204,204,204);padding-left:1ex" class=3D"gmail_quote"><di=
v style=3D"word-wrap:break-word"><div><div><br></div><div>Having
 a second option built into full nodes (or as an external bootstrap=20
service/app) how to download historical blocks during bootstrapping=20
could probably be a relieve for &quot;small nodes=E2=80=9C.</div><div>It co=
uld be a=20
little daemon that downloads historical blocks from CDN=E2=80=99s, etc. and=
=20
feeds them into your full node over p2p/8333 and kickstarts your=20
bootstrapping without bothering valuable peers.</div><div>Or, the alternati=
ve download, could be built into the full nodes main logic.</div><div>And, =
if it wasn=E2=80=99t obvious, this must not bypass the verification!</div><=
/div></div></blockquote><div><br></div><div>I worry about any type of CDN b=
eing a central point of failure. CDNs cost money, which means someone is fo=
oting the bill. Torrenting typically relies on a DHT, which is much easier =
to attack than Bitcoin&#39;s peer network. It&#39;s possible that a decentr=
alized CDN could be used, but I don&#39;t think any yet exist (though I am =
building one for unrelated reasons) which are both sufficiently secure and =
incentive-compatible to be considered as an alternative to using full nodes=
 to bootstrap.<br><br></div><div>I just don&#39;t want to end up in a situa=
tion where 90% of users are getting their blockchain from the same 3 websit=
es or centralized services. And I also don&#39;t want to rely on any p2p so=
lution which would not stand up to a serious adversary. Right now, I think =
the bitcoin p2p network is by significant margin the best we&#39;ve got. Th=
e landscape for decentralized data distribution is evolving rapidly though,=
 perhaps in a few years there will be a superior alternative.<br></div><div=
>=C2=A0</div><blockquote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px =
solid rgb(204,204,204);padding-left:1ex" class=3D"gmail_quote"><div style=
=3D"word-wrap:break-word"><div><div><b>To your proposal:</b><br>- Isn=E2=80=
=99t there a tiny finger-printing element if peers have to pick an segmenta=
tion index?<br>- SPV bloom filter clients can=E2=80=99t use fragmented bloc=
ks to filter txns? Right? How could they avoid connecting to those peers?<s=
pan class=3D"gmail-HOEnZb"><font color=3D"#888888"><br><br>&lt;/jonas&gt;</=
font></span></div></div></div></blockquote><div><br></div><div>Yes, there i=
s finger-print that happens if you have nodes pick an index. And the finger=
print gets a lot worse if you have a node pick multiple indexes. Though, is=
n&#39;t it already required that nodes have some sort of IP address or hidd=
en service domain? I want to say that the fingerprint created by picking an=
 index is not a big deal, because it can be separated from activity like tr=
ansaction relaying and mining. Though, I am not certain and perhaps it is a=
 problem.<br><br></div><div>To be honest, I hadn&#39;t really considered SP=
V nodes at the time of writing. Small nodes would still be seeing all of th=
e new blocks, and per your suggestion may also be storing the 1000 or so mo=
st recent blocks, but SPV nodes would still need some way to find all of th=
eir historical transactions. The problem is not fetching blocks, it&#39;s f=
iguring out which blocks are worth fetching. It may be sufficient to have n=
odes store a bloom filter for each block indicating which addresses had act=
ivity. The bloom filter would probably only need to be about 1% of the size=
 of the full block. That&#39;s not too much overhead (now you are storing 2=
1% of the block instead of just 20%), and would retain SPV compatibility.<b=
r></div></div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quot=
e">On Mon, Apr 17, 2017 at 12:17 PM, praxeology_guy <span dir=3D"ltr">&lt;<=
a href=3D"mailto:praxeology_guy@protonmail.com" target=3D"_blank">praxeolog=
y_guy@protonmail.com</a>&gt;</span> wrote:=C2=A0 <br><blockquote class=3D"g=
mail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204=
,204,204);padding-left:1ex"><div><br></div><div>FYI
 With unspent coin snapshots, needing archive data becomes less=20
important.=C2=A0 People can synchronize from a later snapshot instead of th=
e=20
genesis block.=C2=A0 See <a href=3D"https://petertodd.org/2016/delayed-txo-=
commitments" target=3D"_blank">https://petertodd.org/2016/<wbr>delayed-txo-=
commitments</a> for my current favorite idea.<br></div><br></blockquote></d=
iv></div><div class=3D"gmail_extra"><br></div>This is something that I thin=
k could help a lot too. If the build processes included verifying the chain=
 and then creating a utxo snapshot at say, block 400,000, then nodes would =
no longer need to download, store, upload, or share blocks prior to that he=
ight. It means that a reorg deeper than that point would hardfork the netwo=
rk. But a reorg 60k blocks deep is going to cause problems worse than a net=
work split. Then, the only people who would ever need to care about the ear=
ly blocks are developers, and it&#39;s more reasonable to expect developers=
 to go through a longer process and have more resources than everyday users=
.<br><div><br>On Mon, Apr 17, 2017 at 6:14 AM, Aymeric Vitte via bitcoin-de=
v <span dir=3D"ltr">&lt;<a target=3D"_blank" href=3D"mailto:bitcoin-dev@lis=
ts.linuxfoundation.org">bitcoin-dev@lists.<wbr>linuxfoundation.org</a>&gt;<=
/span> wrote:<br><div class=3D"gmail_extra"><div class=3D"gmail_quote"><blo=
ckquote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204=
,204);padding-left:1ex" class=3D"gmail_quote">
 =20
   =20
 =20
  <div bgcolor=3D"#FFFFFF">
    <p>While I fully agree with the intent (increasing full nodes so a
      big miner waking up in a bad mood can&#39;t threaten the world any
      longer every day as it is now) I am not sure to get the interest
      of this proposal, because:</p>
    <p>- it&#39;s probably not a good idea to encourage the home users to
      run full nodes, there are many people running servers far from
      their capacity that could easily run efficient full nodes<br></p></di=
v></blockquote><div>Running a full node is the only way to avoid needing to=
 trust others. It&#39;s also how you make your opinion worthwhile for event=
s like hard forks and UASFs. If decentralization is the primary motivation,=
 it absolutely makes sense to encourage people to run their own full nodes.=
 Without a full node, you are at the mercy of the governance decisions by t=
hose who do have full nodes. But if you have a full node, you can chose to =
opt-out of any upgrade (example: ethereum classic nodes).<br></div><div>=C2=
=A0</div><blockquote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px soli=
d rgb(204,204,204);padding-left:1ex" class=3D"gmail_quote"><div bgcolor=3D"=
#FFFFFF"><p>
    </p>
    <p>- if someone can&#39;t allocate 100 GB today to run a full node, the=
n
      we can&#39;t expect him to allocate more in the future</p></div></blo=
ckquote><div>That&#39;s why I&#39;m proposing something to decrease the sto=
rage requirements.<br></div><div>=C2=A0</div><blockquote style=3D"margin:0p=
x 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" cl=
ass=3D"gmail_quote"><div bgcolor=3D"#FFFFFF">
   =20
    <p>- this proposal is a kind of reinventing torrents, while limiting
      the number of connections to something not efficient at all, I
      don&#39;t see why something that is proven to be super efficient
      (torrents) would be needed to be reinvented, I am not saying that
      it should be used as the bittorrent network is doing but the
      concepts can be reused</p></div></blockquote><div>It&#39;s different =
from torrents in that it uses specialized erasure coding to make sure that =
every block is always available, even if an adversary is running around tar=
geting all the nodes with a particular piece.<br></div><div>=C2=A0</div><bl=
ockquote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,20=
4,204);padding-left:1ex" class=3D"gmail_quote"><div bgcolor=3D"#FFFFFF">
    <p>- I don&#39;t get at all the concept of &quot;archival&quot; nodes s=
ince it&#39;s
      another useless step toward centralization</p></div></blockquote><div=
>&quot;archival&quot; nodes are simply nodes with the full blockchain. Nobo=
dy can bootstrap on the network without them. Today, every bitcoin-core nod=
e is an archival node by default. <br></div><blockquote style=3D"margin:0px=
 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" cla=
ss=3D"gmail_quote"><div bgcolor=3D"#FFFFFF">
    <p>I think the only way to increase full nodes it to design an
      incentive for people to run them<br></p></div></blockquote><div>The p=
rimary incentive is the sovereignty that it gives you. Running a Bitcoin fu=
ll node gives you security and protection against political garbage that yo=
u can&#39;t get any other way. The network does currently depend on altruis=
m to allow people to download the blockchain, but as long as we can keep th=
e resource requirements of this altruism low, I think we can expect it to c=
ontinue. This proposal attempts to keep those requirements low.<br><br><br>=
<div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, Apr 18, 2=
017 at 6:50 AM, Tom Zander <span dir=3D"ltr">&lt;<a href=3D"mailto:tomz@fre=
edommail.ch" target=3D"_blank">tomz@freedommail.ch</a>&gt;</span> wrote:<br=
><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border=
-left:1px solid rgb(204,204,204);padding-left:1ex"><span class=3D"gmail-">O=
n Monday, 17 April 2017 08:54:49 CEST David Vorick via bitcoin-dev wrote:<b=
r>
&gt; The best alternative today to storing the full blockchain is to run a<=
br>
&gt; pruned node<br>
<br>
</span>The idea looks a little overly complex to me.<br>
<br>
I suggested something similar which is a much simpler version;<br>
<a href=3D"https://zander.github.io/scaling/Pruning/" rel=3D"noreferrer" ta=
rget=3D"_blank">https://zander.github.io/<wbr>scaling/Pruning/</a><br>
<br>
&gt; # Random pruning mode<br>
&gt;<br>
&gt; There is a large gap between the two current modes of everything<br>
&gt; (currently 75GB) and only what we need (2GB or so).<br>
&gt;<br>
&gt; This mode would have two areas, it would keep a days worth of blocks t=
o<br>
&gt; make sure that any reorgs etc would not cause a re-download, but it wo=
uld<br>
&gt; have additionally have an area that can be used to store historical da=
ta<br>
&gt; to be shared on the network. Maybe 20 or 50GB.<br>
&gt;<br>
&gt; One main feature of Bitcoin is that we have massive replication. Each =
node<br>
&gt; currently holds all the same data that every other node holds. But thi=
s<br>
&gt; doesn&#39;t have to be the case with pruned nodes. A node itself has n=
o need<br>
&gt; for historic data at all.<br>
&gt;<br>
&gt; The suggestion is that a node stores a random set of blocks. Dropping<=
br>
&gt; random blocks as the node runs out of disk-space. Additionally, we wou=
ld<br>
&gt; introduce a new way to download blocks from other nodes which allows t=
he<br>
&gt; node to say it doesn&#39;t actually have the block requested.<br>
&gt;<br>
&gt; The effect of this setup is that many different nodes together end up<=
br>
&gt; having the total amount of blocks, even though each node only has a<br=
>
&gt; fraction of the total amount.<br>
<span class=3D"gmail-HOEnZb"><font color=3D"#888888"><br>
--<br>
Tom Zander<br>
Blog: <a href=3D"https://zander.github.io" rel=3D"noreferrer" target=3D"_bl=
ank">https://zander.github.io</a><br>
Vlog: <a href=3D"https://vimeo.com/channels/tomscryptochannel" rel=3D"noref=
errer" target=3D"_blank">https://vimeo.com/channels/<wbr>tomscryptochannel<=
/a><br>
</font></span></blockquote></div><br></div><div class=3D"gmail_extra">Your =
proposal has a significant disadvantage: If every peer is dropping 75% of a=
ll blocks randomly, then you need to connect to a large number of peers to =
download the whole blockchain. Any given peer has a 25% chance of holding a=
 block, or rather, a 75% chance of not holding a block. If you have n peers=
, your probability of not being able to download a given block is 0.75^n. I=
f you are downloading 450,000 blocks, you will need to connect to an expect=
ed 46 peers to download the whole blockchain.<br><br>Your proposal is also =
a lot less able to handle active adversaries: if nodes are randomly droppin=
g blocks, the probability that one block in particular is dropped by everyo=
ne goes up significantly. And the problem gets a lot worse in the presence =
of an active adversary. If there are 8000 nodes each dropping 75% of the bl=
ocks, then each block on average will only be held by 2000 nodes. Probabili=
stically, some unlucky blocks will be held by fewer than 2000 nodes. An act=
ive adversary needs only to eliminate about 2000 nodes (a chosen 2000 nodes=
) to knock a block off of the network. But missing even a single block is a=
 significant problem.<br><br></div><div class=3D"gmail_extra">Your proposal=
 essentially requires that archive nodes still exist and be a part of a typ=
ical blockchain download. Given that, I don&#39;t think it would be a sound=
 idea to ship as a default in bitcoin core.<br><br><br><div class=3D"gmail_=
extra"><br><div class=3D"gmail_quote">On Tue, Apr 18, 2017 at 9:07 AM, Tier=
 Nolan via bitcoin-dev <span dir=3D"ltr">&lt;<a target=3D"_blank" href=3D"m=
ailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundat=
ion.org</a>&gt;</span> wrote:<br><blockquote style=3D"margin:0px 0px 0px 0.=
8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class=3D"gmail=
_quote"><div dir=3D"ltr"><div><div><div><div><div><div><div>This has been d=
iscussed before.<br><br><a target=3D"_blank" href=3D"https://lists.linuxfou=
ndation.org/pipermail/bitcoin-dev/2015-May/008101.html">https://lists.linux=
foundation.<wbr>org/pipermail/bitcoin-dev/<wbr>2015-May/008101.html</a><br>=
<br></div>including a list of nice to have features by Maxwell<br><br><a ta=
rget=3D"_blank" href=3D"https://lists.linuxfoundation.org/pipermail/bitcoin=
-dev/2015-May/008110.html">https://lists.linuxfoundation.<wbr>org/pipermail=
/bitcoin-dev/<wbr>2015-May/008110.html</a><br><br></div></div></div></div>Y=
ou meet most of these rules, though you do have to download blocks from mul=
tiple peers.<br><br></div>The
 suggestion in that thread were for a way to compactly indicate which=20
blocks a node has.=C2=A0 Each node would then store a sub-set of all the=20
blocks.=C2=A0 You just download the blocks you want from the node that has=
=20
them.<br><br></div>Each node would be recommended to store the last few day=
s worth anyway.<br></div>
<br>______________________________<wbr>_________________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.=
<wbr>linuxfoundation.org</a><br>
<a target=3D"_blank" rel=3D"noreferrer" href=3D"https://lists.linuxfoundati=
on.org/mailman/listinfo/bitcoin-dev">https://lists.linuxfoundation.<wbr>org=
/mailman/listinfo/bitcoin-<wbr>dev</a><br>
<br></blockquote></div><br></div><div class=3D"gmail_extra">I believe that =
my proposal does meet all of the requirements listed by Maxwell. Having a s=
et of 8 random peers gives you a very high probability of being able to rec=
over every single block. You would need to connect to at least 5 peers (and=
 this is already &gt;90% likely to be sufficient to recover every block), b=
ut if you cannot connect to 5 random peers your node is probably in trouble=
 anyway. Highly parallel, high speed downloads are just as possible with sm=
all nodes as with archive nodes. It only takes a few bytes to indicate whic=
h part of the blockchain you have, and any 2 peers have a less than 1% chan=
ce of overlapping.</div></div></div></div></div></div></div>

--001a114a4c868e727e054d88616c--