path: root/04/2a280f2799959263e8ebc3b8a4574fdac8f5a1

Return-Path: <dev@jonasschnelli.ch>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
	[172.17.192.35])
	by mail.linuxfoundation.org (Postfix) with ESMTPS id E19309C
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Tue, 18 Apr 2017 07:44:00 +0000 (UTC)
X-Greylist: from auto-whitelisted by SQLgrey-1.7.6
Received: from server3 (server3.include7.ch [144.76.194.38])
	by smtp1.linuxfoundation.org (Postfix) with ESMTP id 704E01D3
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Tue, 18 Apr 2017 07:43:59 +0000 (UTC)
Received: by server3 (Postfix, from userid 115)
	id 4CDC12D006CE; Tue, 18 Apr 2017 09:43:58 +0200 (CEST)
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
	smtp1.linux-foundation.org
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, FSL_HELO_NON_FQDN_1,
	HTML_MESSAGE autolearn=ham version=3.3.1
Received: from [192.168.1.198] (public-docking-pat-bs-mapped-0017.ethz.ch
	[195.176.111.114]) by server3 (Postfix) with ESMTPSA id 8B7822D00117;
	Tue, 18 Apr 2017 09:43:57 +0200 (CEST)
From: Jonas Schnelli <dev@jonasschnelli.ch>
Message-Id: <CA3B35F8-1E81-4202-9784-9770D22C9310@jonasschnelli.ch>
Content-Type: multipart/signed;
	boundary="Apple-Mail=_4C08EC54-E5AA-4866-A99C-65DA54F156B2";
	protocol="application/pgp-signature"; micalg=pgp-sha256
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Date: Tue, 18 Apr 2017 09:43:52 +0200
In-Reply-To: <CAFVRnypbQQ-vsSLqv48cYaqTCty4R1DmFRqfAvxe4mAqyQNXxQ@mail.gmail.com>
To: David Vorick <david.vorick@gmail.com>
References: <CAFVRnypbQQ-vsSLqv48cYaqTCty4R1DmFRqfAvxe4mAqyQNXxQ@mail.gmail.com>
X-Mailer: Apple Mail (2.3273)
Cc: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] Small Nodes: A Better Alternative to Pruned Nodes
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Tue, 18 Apr 2017 07:44:01 -0000


--Apple-Mail=_4C08EC54-E5AA-4866-A99C-65DA54F156B2
Content-Type: multipart/alternative;
	boundary="Apple-Mail=_4B1B5E29-A7B1-4A90-A621-FCA62E77278D"


--Apple-Mail=_4B1B5E29-A7B1-4A90-A621-FCA62E77278D
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

Hi Dave

> A node that stores the full blockchain (I will use the term archival =
node) requires over 100GB of disk space, which I believe is one of the =
most significant barriers to more people running full nodes. And I =
believe the ecosystem would benefit substantially if more users were =
running full nodes.


Thanks for your proposal.

I agree that 100GB of data may be cumbersome for some systems, =
especially if you target end user systems (Laptops/Desktops). Though, in =
my opinion, for those systems, CPU consumption is the biggest UX =
blocker.
Bootstrapping a full node on a decent consumer system with default =
parameters takes days, and, during this period, you probably run at full =
CPU capacity and you will be disturbed by constant fan noise. Standard =
tasks may be impossible because your system will be slowed down to a =
point where even word processing may get difficult.
This is because Core (with its default settings) is made to sync as fast =
as possible.

Once you have verified the chain and you reach the chain tip, indeed, it =
will be much better (until you shutdown for a couple of days/hours and =
have to re-sync/catch-up).

1. I agree that we need to have a way for pruned nodes to partially =
serve historical blocks.
My personal measurements told me that around ~80% of historical block =
serving are between tip and -1=E2=80=99000 blocks.
Currently, Core nodes have only two modes of operations, =E2=80=9Eserver =
all historical blocks=E2=80=9C or =E2=80=9Enone=E2=80=9C.
This makes little sense especially if you prune to a target size of, =
lets say, 80GB (~80% of the chain).
Ideally, there would be a mode where your full node can signal a third =
mode =E2=80=9EI keep the last 1000 blocks=E2=80=9C (or make this more =
dynamic).

2. Bootstrapping new peers
I=E2=80=99m not sure if full nodes must be the single point of =
historical data storage. Full nodes provide a valuable service =
(verification, relay, filtering, etc.). I=E2=80=99m not sure if serving =
historical blocks is one of them. Historical blocks could be made =
available on CDN=E2=80=99s or other file storage networks. You are going =
to verify them anyways,... the serving part is pure data storage.
I=E2=80=99m also pretty sure that some users have stopping running full =
nodes because their upstream bandwidth consumption (because of serving =
historical blocks) was getting intolerable.
Especially =E2=80=9Econsumer=E2=80=9C peers must have been hit by this =
(little experience in how to reduce traffic, upstream in general is bad =
for consumers-connections, little resources in general).

Having a second option built into full nodes (or as an external =
bootstrap service/app) how to download historical blocks during =
bootstrapping could probably be a relieve for "small nodes=E2=80=9C.
It could be a little daemon that downloads historical blocks from =
CDN=E2=80=99s, etc. and feeds them into your full node over p2p/8333 and =
kickstarts your bootstrapping without bothering valuable peers.
Or, the alternative download, could be built into the full nodes main =
logic.
And, if it wasn=E2=80=99t obvious, this must not bypass the =
verification!

I=E2=80=99m also aware of the downsides of this. This can eventually =
reduce decentralisation of the storage of historical bitcoin blockchain =
data and =E2=80=93 eventually =E2=80=93 increase the upstream bandwidth =
of peers willing to serve historical blocks (especially in a transition =
phase to a second =E2=80=9Edownload=E2=80=9C-option).
Maybe it=E2=80=99s a tradeoff between reducing decentralisation by =
killing low resource nodes because serving historical blocks is getting =
too resource-intense _or_ reducing decentralisation by moving some =
percentage of the historical data storage away from the bitcoin p2p =
network.
The later seems more promising to me.


To your proposal:
- Isn=E2=80=99t there a tiny finger-printing element if peers have to =
pick an segmentation index?
- SPV bloom filter clients can=E2=80=99t use fragmented blocks to filter =
txns? Right? How could they avoid connecting to those peers?

</jonas>

--Apple-Mail=_4B1B5E29-A7B1-4A90-A621-FCA62E77278D
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

<html><body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; =
-webkit-line-break: after-white-space;" class=3D"">Hi Dave<br =
class=3D""><br class=3D""><blockquote type=3D"cite" class=3D"">A node =
that stores the full blockchain (I will use the term archival node) =
requires over 100GB of disk space, which I believe is one of the most =
significant barriers to more&nbsp;people running full nodes.&nbsp;And I =
believe the ecosystem would benefit substantially if more users were =
running full nodes.<br class=3D""></blockquote><br class=3D""><br =
class=3D"">Thanks for your proposal.<br class=3D""><br class=3D"">I =
agree that 100GB of data may be cumbersome for some systems, especially =
if you target end user systems (Laptops/Desktops). Though, in my =
opinion, for those systems,&nbsp;CPU consumption is&nbsp;the biggest UX =
blocker.<br class=3D"">Bootstrapping a full node on a decent consumer =
system with default parameters takes days, and, during this period, you =
probably run at full CPU capacity and you will be&nbsp;disturbed by =
constant fan&nbsp;noise. Standard tasks may be impossible because your =
system will be slowed down to a point where even word processing may get =
difficult.<br class=3D"">This is because Core (with its default =
settings) is made to sync as fast as possible.<br class=3D""><br =
class=3D"">Once you have verified the chain and you reach the chain tip, =
indeed, it will be much better (until you shutdown for a couple of =
days/hours and have to re-sync/catch-up).<br class=3D""><br class=3D""><b =
class=3D"">1. I agree that we need to have a way for pruned nodes to =
partially serve historical blocks.</b><br class=3D"">My =
personal&nbsp;measurements told me that around ~80% of historical block =
serving are between tip and -1=E2=80=99000 blocks.<br =
class=3D"">Currently, Core nodes have only two modes of operations, =
=E2=80=9Eserver all historical blocks=E2=80=9C or =E2=80=9Enone=E2=80=9C.<=
br class=3D"">This makes little sense especially if you prune to a =
target size of, lets say, 80GB (~80% of the chain).<br class=3D"">Ideally,=
 there would be a mode where your full node can signal a third mode =E2=80=
=9EI keep the last 1000 blocks=E2=80=9C (or make this more dynamic).<div =
class=3D""><br class=3D""><b class=3D"">2. Bootstrapping new =
peers</b><br class=3D"">I=E2=80=99m not sure if full nodes must be the =
single point of historical data storage. Full nodes provide a valuable =
service (verification, relay, filtering, etc.). I=E2=80=99m not sure if =
serving historical blocks is one of&nbsp;them. Historical blocks could =
be made available on CDN=E2=80=99s or other file storage networks. You =
are going to verify them anyways,... the serving part is pure data =
storage.<br class=3D"">I=E2=80=99m also pretty sure that some users have =
stopping running full nodes because their upstream bandwidth consumption =
(because of serving historical blocks) was getting&nbsp;intolerable.<div =
class=3D"">Especially =E2=80=9Econsumer=E2=80=9C peers must have been =
hit by this (little experience in how to reduce traffic, upstream in =
general is bad for consumers-connections, little resources in =
general).</div><div class=3D""><br class=3D""></div><div class=3D"">Having=
 a second option built into full nodes (or as an external bootstrap =
service/app) how to download historical blocks during bootstrapping =
could probably be a relieve for "small nodes=E2=80=9C.</div><div =
class=3D"">It could be a little daemon that downloads historical blocks =
from CDN=E2=80=99s, etc. and feeds them into your full node over =
p2p/8333 and kickstarts your bootstrapping without bothering valuable =
peers.</div><div class=3D"">Or, the alternative download, could be built =
into the full nodes main logic.</div><div class=3D"">And, if it wasn=E2=80=
=99t obvious, this must not bypass the verification!</div><div =
class=3D""><br class=3D""></div><div class=3D"">I=E2=80=99m also aware =
of the downsides of this. This can eventually reduce decentralisation of =
the storage of historical bitcoin blockchain data and =E2=80=93 =
eventually =E2=80=93 increase the upstream bandwidth of peers willing to =
serve historical blocks (especially in a transition phase to a second =
=E2=80=9Edownload=E2=80=9C-option).</div><div class=3D"">Maybe it=E2=80=99=
s a tradeoff between reducing decentralisation by killing low resource =
nodes because serving historical blocks is getting too resource-intense =
_or_ reducing decentralisation by moving some percentage of the =
historical data storage away from the bitcoin p2p network.</div><div =
class=3D"">The later seems more promising to me.</div><div class=3D""><br =
class=3D""><br class=3D""><b class=3D"">To your proposal:</b><br =
class=3D"">- Isn=E2=80=99t there a tiny finger-printing element if peers =
have to pick an segmentation index?<br class=3D"">- SPV bloom filter =
clients can=E2=80=99t use fragmented blocks to filter txns? Right? How =
could they avoid connecting to those peers?<br class=3D""><br =
class=3D"">&lt;/jonas&gt;</div></div></body></html>=

--Apple-Mail=_4B1B5E29-A7B1-4A90-A621-FCA62E77278D--

--Apple-Mail=_4C08EC54-E5AA-4866-A99C-65DA54F156B2
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEyhopCNzi8TB0xizeHrd2uwPHki0FAlj1w7gACgkQHrd2uwPH
ki1E3A//SjwjLa1JcT0iB3sA9JMtBcO5+UxIaMvTdYV5Ccn28y/MHIL3Q+GlFRUH
/E/zTOIXt+pqV7/26Z04b4uXrSwJuL7rBVQMQouCdYgDux+tUxPvcvUfYhPJg3/B
otGEDXHgBGkYqPJXgbBuwxL1PLJ/56eXX+hD7Wm+BjiyQh6ioXQzozZGPNWxBIl3
IdrsZ3JqTq6dtqgJV4Nht8cJSLrXpF0841pTa/PiUW4HybY7LSZ1EMw52Gy8Ht6X
ZAWFe7C2gnjt6S2e7nvcOrG5fqM4wAXrBRx6mtuz14W+W9UpgMYuoIf4liigQwtH
b1o3jniJ647p94LftEV8bvhJUIkfFekscmQmuNQGsdAVWWmQq75/8TXew/OYWFiF
ef0JkqtQdsxopBIr2nBoHBaXXGXCao89tN7oE5c6WjkSWR79LEocAM7wT4tJWOU0
yrcq7pTfwYdoCLpoNWF8daoFVnK9Tbqd7csE3UIwrHpLRsLqH9FMRPmnqo5itwlu
1Vi6R3y/kAaM/bHCVP5bKov2+kmV33uaCQDhelb97feNLM4L8PCswG8cfgw0yoJv
c3JI9fOurQQj2IWY/3J6E2VgwUD7FkMHuiRbXvtTVRCjhoXU+/D0T1sq69CCnFEk
W4vrrtUOpZUd5n4nscQLg516yILBCfSPrKz2RhjzoOpKEnqIXXw=
=sUS0
-----END PGP SIGNATURE-----

--Apple-Mail=_4C08EC54-E5AA-4866-A99C-65DA54F156B2--