path: root/66/51ff1386eb6a9c88b314207d021baaebdc53ea

Return-Path: <tier.nolan@gmail.com>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
	[172.17.192.35])
	by mail.linuxfoundation.org (Postfix) with ESMTPS id 0BA01A92
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Wed, 13 Sep 2017 09:10:34 +0000 (UTC)
X-Greylist: whitelisted by SQLgrey-1.7.6
Received: from mail-oi0-f44.google.com (mail-oi0-f44.google.com
	[209.85.218.44])
	by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 6730B180
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Wed, 13 Sep 2017 09:10:33 +0000 (UTC)
Received: by mail-oi0-f44.google.com with SMTP id z73so39177309oia.2
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Wed, 13 Sep 2017 02:10:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
	h=mime-version:in-reply-to:references:from:date:message-id:subject:to; 
	bh=D/o8U9XkQsIvrcLABRT4qDD+1QmHZL84LK7g3/cE6ys=;
	b=fCUpJkE5fovxrfJZCcDt/5NSUJZbnYFcJNmaAiT2juqEjzM0oFpkvKIrqB8vv0/m7T
	sZ8on0MAw51IfysfbRWII6yJ3SmAFzZqMqYsgNMzEOKJmBUsU4ENe/kWEyRLALewMwY6
	9pKifzONZ0CoR5hl2RIVzsiwFUILU5E6gvXApHoNMdMB1bvFYtPXeXxAXObXyb2J0iy5
	v75dEWPoy8zedoe0Nk2XaQY2910RmtxyXvniM3h85Lxa4Ds1ECoXNNgdzYGXr5j0DEX+
	h2P/z3qKfta8BBbBruxEZ1qc5ZulrFpN6q6iVM0oCfl/Fm+kHdvBxLgt1WBVI38E2QE7
	+UIA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20161025;
	h=x-gm-message-state:mime-version:in-reply-to:references:from:date
	:message-id:subject:to;
	bh=D/o8U9XkQsIvrcLABRT4qDD+1QmHZL84LK7g3/cE6ys=;
	b=GmD6vJt2Zw5n+VXWnyKAxxXh15awsEw1Oh2L+mylCuLVeA7uzpyshzXuMzdVA8Lon6
	XEMe+9nB6PNCzeSVWyg5thb4gj4Yd5CT2LmFm+TxtqSM5ZQbg76xbt3AoZI1NU5zR7FI
	XJz8x6bN4AekiQz2zmD1yairWmeH/36USXGx/VzPUqXaMm15lX545i0xsGUF3GQoBvAv
	n3+h7pf+DzBOjObsrg3ESrWNQ72K8O9jpAXGiCf2TKhGD/ejQ4rYGbEcTG4HQg/J0skd
	j1Z8exVZV5mDaaKMGXLo6ZNvZDO4NBVXoEa59bmBM3wS0B9fw1GLWpZ/ZjEi/LAe3ckn
	0LXQ==
X-Gm-Message-State: AHPjjUhZxMrxXm04q97ojX92KfR95dWEPh/t4NqAf7HzbW/8SYL1wJp9
	/bRvKkBn1vYh+f29StryLJBUmYViqvCp0yjUkktvgg==
X-Google-Smtp-Source: ADKCNb7raxDtpr0G4Rw4hAkCK6VUDgJQ/Mn7DLnF8QYrUGVirgz84K2Bp+cgTax2SEmqK8TjMMBfvH4yJvbzA7CuM7Y=
X-Received: by 10.202.208.92 with SMTP id h89mr20870367oig.90.1505293832495;
	Wed, 13 Sep 2017 02:10:32 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.74.139.118 with HTTP; Wed, 13 Sep 2017 02:09:52 -0700 (PDT)
In-Reply-To: <351373080.1326948.1505257115533@mail.yahoo.com>
References: <351373080.1326948.1505257115533.ref@mail.yahoo.com>
	<351373080.1326948.1505257115533@mail.yahoo.com>
From: Tier Nolan <tier.nolan@gmail.com>
Date: Wed, 13 Sep 2017 10:09:52 +0100
Message-ID: <CAE-z3OUeHfPh28MqjVhJ9pRZ=P8MecaGFwR48Ggw4UaHDjNr6A@mail.gmail.com>
To: Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
Content-Type: multipart/alternative; boundary="001a113ca1de2b26a205590e8849"
X-Spam-Status: No, score=0.4 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,
	RCVD_IN_SORBS_SPAM autolearn=disabled version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
	smtp1.linux-foundation.org
Subject: Re: [bitcoin-dev] 2 softforks to cut the blockchain and IBD time
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Wed, 13 Sep 2017 09:10:34 -0000

--001a113ca1de2b26a205590e8849
Content-Type: text/plain; charset="UTF-8"

On Tue, Sep 12, 2017 at 11:58 PM, michele terzi via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

>
> Pros:
>
> you gain a much faster syncing for new nodes.
> full non pruning nodes need a lot less HD space.
> dropping old history results in more difficult future chainanalysis (at
> least by small entities)
> freezing old history in one new genesis block means the chain can no
> longer be reorged prior to that point
>

Current nodes allow pruning so you can save disk space that way.  Users
still need to download/verify the new blocks though.

Under your scheme, you don't need to throw the data away.  Nodes can decide
how far back that they want to go.

"Fast" IBD

- download header chain from genesis (~4MB per year)
- check headers against "soft" checkpoints (every 50k blocks)
- download the UTXO set of the most recent soft checkpoint (and verify
against hash)
- download blocks starting from the most recent soft checkpoint
- node is now ready to use
- [Optional] Slowly download the remaining blocks

This requires some new protocol messages to allow requesting and send the
UTXO set, though the inv and getdata messages could be used.

If you add a new services bit, NODE_NETWORK_RECENT, then nodes can find
other nodes that have the most recent blocks.  This indicates that you have
all blocks since the most recent snapshot.

The slow download doesn't have to download the blocks in order.  It can
just check against the header chain.  Once a node has all the blocks, it
would switch from NODE_NETWORK_RECENT to NODE_NETWORK.

(Multiple bits could be used to indicate that the node has 2 or more recent
time periods).

"Soft" checkpoints mean that re-orgs can't cause a network partition.  Each
soft checkpoint is a mapping of {block_hash: utxo_hash}.

A re-org of 1 year or more would be devastating so it is probably
academic.  Some people may object to centralized checkpointing and soft
checkpoints cover that objection.

full nodes with old software can no longer be fired up and sync with the
> existing network
> full nodes that went off line prior to the second fork cannot sync back
> once they turn back on line again.
>
>
This is why having archive nodes (and a way to find them) is important.

You could have a weaker requirement that nodes shouldn't delete blocks
unless they are at least 3 time periods (~3 years) old.

The software should have a setting which allows the user to specify maximum
disk space.  Disk space is cheap, so it is likely that a reasonable number
of people will leave that set to infinite.

This automatically results in lots of archive nodes.  Another setting could
decide how many time periods to download.  2-3 seem reasonable as a default
(or maybe infinite too).


> Addressing security concerns:
>
> being able to write a new genesis block means that an evil core has the
> power to steal/destroy/censor/whatever coins.
>
> this is possible only in theory, but not in practice. right now devs can
> misbehave with every softfork, but the community tests and inspects every
> new release.
>

Soft forks are inherently backward compatible.  Coins cannot be stolen
using a soft fork.  It has nothing to do with inspecting new releases.

It is possible for a majority of miners to re-write history, but that is
separate to a soft fork.

A soft fork can lock coins away.  This effectively destroys the coins, but
doesn't steal them.  It could be part of a extortion scheme I guess, but if
a majority of miners did that, then I think Bitcoin has bigger problems.


> the 2 forks will be tested and inspected as well so they are no more risky
> than other softforks.
>
>
For it to be a soft fork, you need to maintain archive nodes.  That is the
whole point.  The old network and the new network rules agree that the new
network rules are valid (and that miners only mine blocks that are valid
under the new rules).  If IBD is impossible for old nodes, then that counts
as a network split.

--001a113ca1de2b26a205590e8849
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On T=
ue, Sep 12, 2017 at 11:58 PM, michele terzi via bitcoin-dev <span dir=3D"lt=
r">&lt;<a target=3D"_blank" href=3D"mailto:bitcoin-dev@lists.linuxfoundatio=
n.org">bitcoin-dev@lists.linuxfoundation.org</a>&gt;</span> wrote:<br id=3D=
"gmail-m_4128619024572080520yui_3_16_0_ym19_1_1505235822221_3295"><blockquo=
te style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204)=
;padding-left:1ex" class=3D"gmail_quote"><div><div style=3D"color:rgb(0,0,0=
);background-color:rgb(255,255,255);font-family:Helvetica Neue,Helvetica,Ar=
ial,Lucida Grande,sans-serif;font-size:13px"><div dir=3D"ltr" id=3D"gmail-m=
_4128619024572080520yui_3_16_0_ym19_1_1505235822221_2803"><br id=3D"gmail-m=
_4128619024572080520yui_3_16_0_ym19_1_1505235822221_3296">Pros:<br id=3D"gm=
ail-m_4128619024572080520yui_3_16_0_ym19_1_1505235822221_3297"><br id=3D"gm=
ail-m_4128619024572080520yui_3_16_0_ym19_1_1505235822221_3298">you gain a m=
uch faster syncing for new nodes.<br id=3D"gmail-m_4128619024572080520yui_3=
_16_0_ym19_1_1505235822221_3299">full non pruning nodes need a lot less HD =
space.<br id=3D"gmail-m_4128619024572080520yui_3_16_0_ym19_1_1505235822221_=
3300">dropping old history results in more difficult future chainanalysis (=
at least by small entities)<br id=3D"gmail-m_4128619024572080520yui_3_16_0_=
ym19_1_1505235822221_3301">freezing old history in one new genesis block me=
ans the chain can no longer be reorged prior to that point<br id=3D"gmail-m=
_4128619024572080520yui_3_16_0_ym19_1_1505235822221_3302"></div></div></div=
></blockquote><div><br></div><div>Current nodes allow pruning so you can sa=
ve disk space that way.=C2=A0 Users still need to download/verify the new b=
locks though.<br><br></div><div>Under your scheme, you don&#39;t need to th=
row the data away.=C2=A0 Nodes can decide how far back that they want to go=
.<br></div><div><br></div><div>&quot;Fast&quot; IBD<br><br></div><div>- dow=
nload header chain from genesis (~4MB per year)<br></div><div>- check heade=
rs against &quot;soft&quot; checkpoints (every 50k blocks)<br></div><div>- =
download the UTXO set of the most recent soft checkpoint (and verify agains=
t hash)<br>- download blocks starting from the most recent soft checkpoint<=
br></div><div>- node is now ready to use<br></div><div>- [Optional] Slowly =
download the remaining blocks<br></div><div><br></div><div>This requires so=
me new protocol messages to allow requesting and send the UTXO set, though =
the inv and getdata messages could be used.<br><br></div><div>If you add a =
new services bit, NODE_NETWORK_RECENT, then nodes can find other nodes that=
 have the most recent blocks.=C2=A0 This indicates that you have all blocks=
 since the most recent snapshot.<br><br></div><div>The slow download doesn&=
#39;t have to download the blocks in order.=C2=A0 It can just check against=
 the header chain.=C2=A0 Once a node has all the blocks, it would switch fr=
om NODE_NETWORK_RECENT to NODE_NETWORK.<br><br></div><div>(Multiple bits co=
uld be used to indicate that the node has 2 or more recent time periods).<b=
r><br></div><div>&quot;Soft&quot; checkpoints mean that re-orgs can&#39;t c=
ause a network partition.=C2=A0 Each soft checkpoint is a mapping of {block=
_hash: utxo_hash}.<br><br></div><div>A re-org of 1 year or more would be de=
vastating so it is probably academic.=C2=A0 Some people may object to centr=
alized checkpointing and soft checkpoints cover that objection.<br> </div><=
div><br></div><blockquote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px=
 solid rgb(204,204,204);padding-left:1ex" class=3D"gmail_quote"><div><div s=
tyle=3D"color:rgb(0,0,0);background-color:rgb(255,255,255);font-family:Helv=
etica Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:13px"><div di=
r=3D"ltr" id=3D"gmail-m_4128619024572080520yui_3_16_0_ym19_1_1505235822221_=
2803">full nodes with old software can no longer be fired up and sync with =
the existing network<br id=3D"gmail-m_4128619024572080520yui_3_16_0_ym19_1_=
1505235822221_3322">full nodes that went off line prior to the second fork =
cannot sync back once they turn back on line again.<br id=3D"gmail-m_412861=
9024572080520yui_3_16_0_ym19_1_1505235822221_3323"><br id=3D"gmail-m_412861=
9024572080520yui_3_16_0_ym19_1_1505235822221_3324"></div></div></div></bloc=
kquote><div><br></div><div>This is why having archive nodes (and a way to f=
ind them) is important.<br></div><div><br></div><div>You could have a weake=
r requirement that nodes shouldn&#39;t delete blocks unless they are at lea=
st 3 time periods (~3 years) old.<br><br></div><div>The software should hav=
e a setting which allows the user to specify maximum disk space.=C2=A0 Disk=
 space is cheap, so it is likely that a reasonable number of people will le=
ave that set to infinite.<br></div><div><br></div><div>This automatically r=
esults in lots of archive nodes.=C2=A0 Another setting could decide how man=
y time periods to download.=C2=A0 2-3 seem reasonable as a default (or mayb=
e infinite too).<br></div><div><br></div><blockquote style=3D"margin:0px 0p=
x 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class=
=3D"gmail_quote"><div><div style=3D"color:rgb(0,0,0);background-color:rgb(2=
55,255,255);font-family:Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-s=
erif;font-size:13px"><div dir=3D"ltr" id=3D"gmail-m_4128619024572080520yui_=
3_16_0_ym19_1_1505235822221_2803"><br id=3D"gmail-m_4128619024572080520yui_=
3_16_0_ym19_1_1505235822221_3328">Addressing security concerns:<br id=3D"gm=
ail-m_4128619024572080520yui_3_16_0_ym19_1_1505235822221_3329"><br id=3D"gm=
ail-m_4128619024572080520yui_3_16_0_ym19_1_1505235822221_3330">being able t=
o write a new genesis block means that an evil core has the power to steal/=
destroy/censor/whatever coins.<br id=3D"gmail-m_4128619024572080520yui_3_16=
_0_ym19_1_1505235822221_3331"><br id=3D"gmail-m_4128619024572080520yui_3_16=
_0_ym19_1_1505235822221_3332">this is possible only in theory, but not in p=
ractice. right now devs can misbehave with every softfork, but the communit=
y tests and inspects every new release.<br id=3D"gmail-m_412861902457208052=
0yui_3_16_0_ym19_1_1505235822221_3335"></div></div></div></blockquote><div>=
<br></div><div>Soft forks are inherently backward compatible.=C2=A0 Coins c=
annot be stolen using  a soft fork.=C2=A0 It has nothing to do with inspect=
ing new releases.<br><br></div><div>It is possible for a majority of miners=
 to re-write history, but that is separate to a soft fork.<br><br></div><di=
v>A soft fork can lock coins away.=C2=A0 This effectively destroys the coin=
s, but doesn&#39;t steal them.=C2=A0 It could be part of a extortion scheme=
 I guess, but if a majority of miners did that, then I think Bitcoin has bi=
gger problems.<br></div><div><br></div><blockquote style=3D"margin:0px 0px =
0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class=3D=
"gmail_quote"><div><div style=3D"color:rgb(0,0,0);background-color:rgb(255,=
255,255);font-family:Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-seri=
f;font-size:13px"><div dir=3D"ltr" id=3D"gmail-m_4128619024572080520yui_3_1=
6_0_ym19_1_1505235822221_2803"><br id=3D"gmail-m_4128619024572080520yui_3_1=
6_0_ym19_1_1505235822221_3336">the 2 forks will be tested and inspected as =
well so they are no more risky than other softforks.<br id=3D"gmail-m_41286=
19024572080520yui_3_16_0_ym19_1_1505235822221_3337"><br id=3D"gmail-m_41286=
19024572080520yui_3_16_0_ym19_1_1505235822221_3338"></div></div></div></blo=
ckquote><div><br></div><div>For it to be a soft fork, you need to maintain =
archive nodes.=C2=A0 That is the whole point.=C2=A0 The old network and the=
 new network rules agree that the new network rules are valid (and that min=
ers only mine blocks that are valid under the new rules).=C2=A0 If IBD is i=
mpossible for old nodes, then that counts as a network split.=C2=A0 <br></d=
iv></div></div></div>

--001a113ca1de2b26a205590e8849--