Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 0BA01A92 for ; Wed, 13 Sep 2017 09:10:34 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-oi0-f44.google.com (mail-oi0-f44.google.com [209.85.218.44]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 6730B180 for ; Wed, 13 Sep 2017 09:10:33 +0000 (UTC) Received: by mail-oi0-f44.google.com with SMTP id z73so39177309oia.2 for ; Wed, 13 Sep 2017 02:10:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=D/o8U9XkQsIvrcLABRT4qDD+1QmHZL84LK7g3/cE6ys=; b=fCUpJkE5fovxrfJZCcDt/5NSUJZbnYFcJNmaAiT2juqEjzM0oFpkvKIrqB8vv0/m7T sZ8on0MAw51IfysfbRWII6yJ3SmAFzZqMqYsgNMzEOKJmBUsU4ENe/kWEyRLALewMwY6 9pKifzONZ0CoR5hl2RIVzsiwFUILU5E6gvXApHoNMdMB1bvFYtPXeXxAXObXyb2J0iy5 v75dEWPoy8zedoe0Nk2XaQY2910RmtxyXvniM3h85Lxa4Ds1ECoXNNgdzYGXr5j0DEX+ h2P/z3qKfta8BBbBruxEZ1qc5ZulrFpN6q6iVM0oCfl/Fm+kHdvBxLgt1WBVI38E2QE7 +UIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=D/o8U9XkQsIvrcLABRT4qDD+1QmHZL84LK7g3/cE6ys=; b=GmD6vJt2Zw5n+VXWnyKAxxXh15awsEw1Oh2L+mylCuLVeA7uzpyshzXuMzdVA8Lon6 XEMe+9nB6PNCzeSVWyg5thb4gj4Yd5CT2LmFm+TxtqSM5ZQbg76xbt3AoZI1NU5zR7FI XJz8x6bN4AekiQz2zmD1yairWmeH/36USXGx/VzPUqXaMm15lX545i0xsGUF3GQoBvAv n3+h7pf+DzBOjObsrg3ESrWNQ72K8O9jpAXGiCf2TKhGD/ejQ4rYGbEcTG4HQg/J0skd j1Z8exVZV5mDaaKMGXLo6ZNvZDO4NBVXoEa59bmBM3wS0B9fw1GLWpZ/ZjEi/LAe3ckn 0LXQ== X-Gm-Message-State: AHPjjUhZxMrxXm04q97ojX92KfR95dWEPh/t4NqAf7HzbW/8SYL1wJp9 /bRvKkBn1vYh+f29StryLJBUmYViqvCp0yjUkktvgg== X-Google-Smtp-Source: ADKCNb7raxDtpr0G4Rw4hAkCK6VUDgJQ/Mn7DLnF8QYrUGVirgz84K2Bp+cgTax2SEmqK8TjMMBfvH4yJvbzA7CuM7Y= X-Received: by 10.202.208.92 with SMTP id h89mr20870367oig.90.1505293832495; Wed, 13 Sep 2017 02:10:32 -0700 (PDT) MIME-Version: 1.0 Received: by 10.74.139.118 with HTTP; Wed, 13 Sep 2017 02:09:52 -0700 (PDT) In-Reply-To: <351373080.1326948.1505257115533@mail.yahoo.com> References: <351373080.1326948.1505257115533.ref@mail.yahoo.com> <351373080.1326948.1505257115533@mail.yahoo.com> From: Tier Nolan Date: Wed, 13 Sep 2017 10:09:52 +0100 Message-ID: To: Bitcoin Protocol Discussion Content-Type: multipart/alternative; boundary="001a113ca1de2b26a205590e8849" X-Spam-Status: No, score=0.4 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE, RCVD_IN_SORBS_SPAM autolearn=disabled version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: Re: [bitcoin-dev] 2 softforks to cut the blockchain and IBD time X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 09:10:34 -0000 --001a113ca1de2b26a205590e8849 Content-Type: text/plain; charset="UTF-8" On Tue, Sep 12, 2017 at 11:58 PM, michele terzi via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > > Pros: > > you gain a much faster syncing for new nodes. > full non pruning nodes need a lot less HD space. > dropping old history results in more difficult future chainanalysis (at > least by small entities) > freezing old history in one new genesis block means the chain can no > longer be reorged prior to that point > Current nodes allow pruning so you can save disk space that way. Users still need to download/verify the new blocks though. Under your scheme, you don't need to throw the data away. Nodes can decide how far back that they want to go. "Fast" IBD - download header chain from genesis (~4MB per year) - check headers against "soft" checkpoints (every 50k blocks) - download the UTXO set of the most recent soft checkpoint (and verify against hash) - download blocks starting from the most recent soft checkpoint - node is now ready to use - [Optional] Slowly download the remaining blocks This requires some new protocol messages to allow requesting and send the UTXO set, though the inv and getdata messages could be used. If you add a new services bit, NODE_NETWORK_RECENT, then nodes can find other nodes that have the most recent blocks. This indicates that you have all blocks since the most recent snapshot. The slow download doesn't have to download the blocks in order. It can just check against the header chain. Once a node has all the blocks, it would switch from NODE_NETWORK_RECENT to NODE_NETWORK. (Multiple bits could be used to indicate that the node has 2 or more recent time periods). "Soft" checkpoints mean that re-orgs can't cause a network partition. Each soft checkpoint is a mapping of {block_hash: utxo_hash}. A re-org of 1 year or more would be devastating so it is probably academic. Some people may object to centralized checkpointing and soft checkpoints cover that objection. full nodes with old software can no longer be fired up and sync with the > existing network > full nodes that went off line prior to the second fork cannot sync back > once they turn back on line again. > > This is why having archive nodes (and a way to find them) is important. You could have a weaker requirement that nodes shouldn't delete blocks unless they are at least 3 time periods (~3 years) old. The software should have a setting which allows the user to specify maximum disk space. Disk space is cheap, so it is likely that a reasonable number of people will leave that set to infinite. This automatically results in lots of archive nodes. Another setting could decide how many time periods to download. 2-3 seem reasonable as a default (or maybe infinite too). > Addressing security concerns: > > being able to write a new genesis block means that an evil core has the > power to steal/destroy/censor/whatever coins. > > this is possible only in theory, but not in practice. right now devs can > misbehave with every softfork, but the community tests and inspects every > new release. > Soft forks are inherently backward compatible. Coins cannot be stolen using a soft fork. It has nothing to do with inspecting new releases. It is possible for a majority of miners to re-write history, but that is separate to a soft fork. A soft fork can lock coins away. This effectively destroys the coins, but doesn't steal them. It could be part of a extortion scheme I guess, but if a majority of miners did that, then I think Bitcoin has bigger problems. > the 2 forks will be tested and inspected as well so they are no more risky > than other softforks. > > For it to be a soft fork, you need to maintain archive nodes. That is the whole point. The old network and the new network rules agree that the new network rules are valid (and that miners only mine blocks that are valid under the new rules). If IBD is impossible for old nodes, then that counts as a network split. --001a113ca1de2b26a205590e8849 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On T= ue, Sep 12, 2017 at 11:58 PM, michele terzi via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:

Pros:

you gain a m= uch faster syncing for new nodes.
full non pruning nodes need a lot less HD = space.
dropping old history results in more difficult future chainanalysis (= at least by small entities)
freezing old history in one new genesis block me= ans the chain can no longer be reorged prior to that point

Current nodes allow pruning so you can sa= ve disk space that way.=C2=A0 Users still need to download/verify the new b= locks though.

Under your scheme, you don't need to th= row the data away.=C2=A0 Nodes can decide how far back that they want to go= .

"Fast" IBD

- dow= nload header chain from genesis (~4MB per year)
- check heade= rs against "soft" checkpoints (every 50k blocks)
- = download the UTXO set of the most recent soft checkpoint (and verify agains= t hash)
- download blocks starting from the most recent soft checkpoint<= br>
- node is now ready to use
- [Optional] Slowly = download the remaining blocks

This requires so= me new protocol messages to allow requesting and send the UTXO set, though = the inv and getdata messages could be used.

If you add a = new services bit, NODE_NETWORK_RECENT, then nodes can find other nodes that= have the most recent blocks.=C2=A0 This indicates that you have all blocks= since the most recent snapshot.

The slow download doesn&= #39;t have to download the blocks in order.=C2=A0 It can just check against= the header chain.=C2=A0 Once a node has all the blocks, it would switch fr= om NODE_NETWORK_RECENT to NODE_NETWORK.

(Multiple bits co= uld be used to indicate that the node has 2 or more recent time periods).
"Soft" checkpoints mean that re-orgs can't c= ause a network partition.=C2=A0 Each soft checkpoint is a mapping of {block= _hash: utxo_hash}.

A re-org of 1 year or more would be de= vastating so it is probably academic.=C2=A0 Some people may object to centr= alized checkpointing and soft checkpoints cover that objection.
<= div>
full nodes with old software can no longer be fired up and sync with = the existing network
full nodes that went off line prior to the second fork = cannot sync back once they turn back on line again.


This is why having archive nodes (and a way to f= ind them) is important.

You could have a weake= r requirement that nodes shouldn't delete blocks unless they are at lea= st 3 time periods (~3 years) old.

The software should hav= e a setting which allows the user to specify maximum disk space.=C2=A0 Disk= space is cheap, so it is likely that a reasonable number of people will le= ave that set to infinite.

This automatically r= esults in lots of archive nodes.=C2=A0 Another setting could decide how man= y time periods to download.=C2=A0 2-3 seem reasonable as a default (or mayb= e infinite too).


Addressing security concerns:

being able t= o write a new genesis block means that an evil core has the power to steal/= destroy/censor/whatever coins.

this is possible only in theory, but not in p= ractice. right now devs can misbehave with every softfork, but the communit= y tests and inspects every new release.
=
Soft forks are inherently backward compatible.=C2=A0 Coins c= annot be stolen using a soft fork.=C2=A0 It has nothing to do with inspect= ing new releases.

It is possible for a majority of miners= to re-write history, but that is separate to a soft fork.

A soft fork can lock coins away.=C2=A0 This effectively destroys the coin= s, but doesn't steal them.=C2=A0 It could be part of a extortion scheme= I guess, but if a majority of miners did that, then I think Bitcoin has bi= gger problems.


the 2 forks will be tested and inspected as = well so they are no more risky than other softforks.


For it to be a soft fork, you need to maintain = archive nodes.=C2=A0 That is the whole point.=C2=A0 The old network and the= new network rules agree that the new network rules are valid (and that min= ers only mine blocks that are valid under the new rules).=C2=A0 If IBD is i= mpossible for old nodes, then that counts as a network split.=C2=A0
--001a113ca1de2b26a205590e8849--