1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
|
Received: from sog-mx-4.v43.ch3.sourceforge.com ([172.29.43.194]
helo=mx.sourceforge.net)
by sfs-ml-1.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
(envelope-from <pieter.wuille@gmail.com>) id 1Xd6B3-000278-Tx
for bitcoin-development@lists.sourceforge.net;
Sat, 11 Oct 2014 23:34:21 +0000
Received-SPF: pass (sog-mx-4.v43.ch3.sourceforge.com: domain of gmail.com
designates 209.85.213.181 as permitted sender)
client-ip=209.85.213.181; envelope-from=pieter.wuille@gmail.com;
helo=mail-ig0-f181.google.com;
Received: from mail-ig0-f181.google.com ([209.85.213.181])
by sog-mx-4.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
(Exim 4.76) id 1Xd6B2-0001Ty-Ia
for bitcoin-development@lists.sourceforge.net;
Sat, 11 Oct 2014 23:34:21 +0000
Received: by mail-ig0-f181.google.com with SMTP id r10so6753884igi.14
for <bitcoin-development@lists.sourceforge.net>;
Sat, 11 Oct 2014 16:34:15 -0700 (PDT)
MIME-Version: 1.0
X-Received: by 10.50.253.42 with SMTP id zx10mr18659696igc.43.1413070455068;
Sat, 11 Oct 2014 16:34:15 -0700 (PDT)
Received: by 10.50.65.9 with HTTP; Sat, 11 Oct 2014 16:34:15 -0700 (PDT)
Date: Sat, 11 Oct 2014 16:34:15 -0700
Message-ID: <CAPg+sBjG86ZYKZZXA4VCMiFhz0sdxr_so41Zr+DATOXyRa9_Jw@mail.gmail.com>
From: Pieter Wuille <pieter.wuille@gmail.com>
To: Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Content-Type: text/plain; charset=ISO-8859-1
X-Spam-Score: 0.1 (/)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
See http://spamassassin.org/tag/ for more details.
-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
sender-domain
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(pieter.wuille[at]gmail.com)
-0.0 SPF_PASS SPF: sender matches SPF record
1.7 URIBL_WS_SURBL Contains an URL listed in the WS SURBL blocklist
[URIs: sipa.be]
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
author's domain
0.1 DKIM_SIGNED Message has a DKIM or DK signature,
not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
X-Headers-End: 1Xd6B2-0001Ty-Ia
Subject: [Bitcoin-development] Request for review/testing: headers-first
synchronization in Bitcoin Core
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Sat, 11 Oct 2014 23:34:22 -0000
Hi all,
I believe that a large change that I've been working on for Bitcoin
Core is ready for review and testing: headers-first synchronization.
In short, it changes the way the best chain is discovered, downloaded
and verified, with several advantages:
* Parallel block downloading (much faster sync on typical network connections).
* No more stalled downloads.
* Much more robust against unresponsive or slow peers.
* Removes a class of DoS attacks related to peers feeding you
low-difficulty valid large blocks on a side branch.
* Reduces the need for checkpoints in the code.
* No orphan blocks stored in memory anymore (reducing memory usage during sync).
* A major step step towards an SPV mode using the reference codebase.
Historically, this mode of operation has been known for years (Greg
Maxwell wrote up a description of a very similar method in
https://en.bitcoin.it/wiki/User:Gmaxwell/Reverse_header-fetching_sync
in early 2012, but it was known before that), but it took a long time
to refactor these code enough to support it.
Technically, it works by replacing the single-peer blocks download by
a single-peer headers download (which typically takes seconds/minutes)
and verification, and simultaneously fetching blocks along the best
known headers chain from all peers that are known to have the relevant
blocks. Downloading is constrained to a moving window to avoid
unbounded unordering of blocks on disk (which would interfere with
pruning later).
At the protocol level, it increases the minimally supported version
for peers to 31800 (corresponding to bitcoin v3.18, released in
december 2010), as earlier versions did not support the getheaders P2P
message.
So, the code is available as a github pull request
(https://github.com/bitcoin/bitcoin/pull/4468), or packaged on
http://bitcoin.sipa.be/builds/headersfirst, where you can also find
binaries to test with.
Known issues:
* At the very start of the sync, especially before all headers are
processed, downloading is very slow due to a limited number of blocks
that are requested per peer simultaneously. The policies around this
will need some experimentation can certainly be improved.
* Blocks will be stored on disk out of order (in the order they are
received, really), which makes it incompatible with some tools or
other programs. Reindexing using earlier versions will also not work
anymore as a result of this.
* The block index database will now hold headers for which no block is
stored on disk, which earlier versions won't support. If you are fully
synced, it may still be possible to go back to an earlier version.
Unknown issues:
* Who knows, maybe it will replace your familiy pictures with Nyan
Cat? Use at your own risk.
TL;DR: Review/test https://github.com/bitcoin/bitcoin/pull/4468 or
http://bitcoin.sipa.be/builds/headersfirst.
--
Pieter
|
