1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
|
Received: from sog-mx-4.v43.ch3.sourceforge.com ([172.29.43.194]
helo=mx.sourceforge.net)
by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
(envelope-from <pieter.wuille@gmail.com>) id 1WsdQc-0002eM-2H
for bitcoin-development@lists.sourceforge.net;
Thu, 05 Jun 2014 19:34:22 +0000
Received-SPF: pass (sog-mx-4.v43.ch3.sourceforge.com: domain of gmail.com
designates 209.85.223.178 as permitted sender)
client-ip=209.85.223.178; envelope-from=pieter.wuille@gmail.com;
helo=mail-ie0-f178.google.com;
Received: from mail-ie0-f178.google.com ([209.85.223.178])
by sog-mx-4.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
(Exim 4.76) id 1WsdQb-0005PK-5I
for bitcoin-development@lists.sourceforge.net;
Thu, 05 Jun 2014 19:34:22 +0000
Received: by mail-ie0-f178.google.com with SMTP id rl12so1343102iec.9
for <bitcoin-development@lists.sourceforge.net>;
Thu, 05 Jun 2014 12:34:15 -0700 (PDT)
MIME-Version: 1.0
X-Received: by 10.50.65.3 with SMTP id t3mr23640621igs.20.1401996855225; Thu,
05 Jun 2014 12:34:15 -0700 (PDT)
Received: by 10.50.184.195 with HTTP; Thu, 5 Jun 2014 12:34:15 -0700 (PDT)
In-Reply-To: <2ADB9019-825E-410A-ADED-A7237CBC323C@ricmoo.com>
References: <34798C1C-FDA7-4A4C-B136-DBD4E59C254D@ricmoo.com>
<CANEZrP1vVY1jnV8Kdg5OWEt=Rba2PPcs3ke4tMWc6fS4wGPpOw@mail.gmail.com>
<2ADB9019-825E-410A-ADED-A7237CBC323C@ricmoo.com>
Date: Thu, 5 Jun 2014 21:34:15 +0200
Message-ID: <CAPg+sBhRQWE6_arjQi55Ars8mOCQkouCOAqVAuHrBzOD60GEqQ@mail.gmail.com>
From: Pieter Wuille <pieter.wuille@gmail.com>
To: Richard Moore <me@ricmoo.com>
Content-Type: text/plain; charset=ISO-8859-1
X-Spam-Score: -1.6 (-)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
See http://spamassassin.org/tag/ for more details.
-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
sender-domain
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(pieter.wuille[at]gmail.com)
-0.0 SPF_PASS SPF: sender matches SPF record
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
author's domain
0.1 DKIM_SIGNED Message has a DKIM or DK signature,
not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
X-Headers-End: 1WsdQb-0005PK-5I
Cc: Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Subject: Re: [Bitcoin-development] Future Feature Proposal - getgist
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Thu, 05 Jun 2014 19:34:22 -0000
On Thu, Jun 5, 2014 at 7:43 PM, Richard Moore <me@ricmoo.com> wrote:
> I was considering names like getcheckpoints() to use the terminology that
> already seemed to be in place, but they were too long :)
>
> I have been using getheaders() in my thick client to quickly grab all the
> headers before downloading the full blocks since I can grab more at a time.
> Even with getblocks(), there is the case for a getgist() call. Right now
> you call getblocks(), which can take some time to get the corresponding
> inv(), at which time you can then start the call to getdata() as well as the
> next call to getblocks().
>
> With a gist, for example of segment_count 50, you could call getgist(), then
> with the response, you could request 50 getblocks() each with a
> block_locator of 1 hash from the gist (and optimally the stop_hash of the
> next hash in the gist) to 50 different peers, providing 25,000 (50 x 500)
> block hashes.
I don't understand. If you're using getheaders(), there is no need to
use getblocks() anymore. You just do a getdata() immediately for the
block hashes you have the headers but not the transactions for.
In general, I think we should aim for as much verifiability as
possible. Much of the reference client's design is built around doing
as much validation on received data as soon as possible, to avoid
being misled by a particular peer. Getheaders() provides this: you
receive a set of headers starting from a point you already know, in
order, and can validate them syntactically and for proof-of-work
immediately. That allows building a very-hard-to-beat tree structure
locally already, at which point you can start requesting blocks along
the good branches of that tree immediately - potentially in parallel
from multiple peers. In fact, that's the planned approach for the
headers-first synchronization.
The getgist() proposed here allows the peer to basically give you
bullshit headers at the end, and you won't notice until you've
downloaded every single block (or at least every single header) up to
that point. That risks wasting time, bandwidth and diskspace,
depending on implementation.
Based on earlier experimenting with my former experimental
headersfirst branch, it's quite possible to have 2 mostly independent
synchronization mechanisms going on; 1) asking and downloading headers
from every peer, and validating them, and 2) asking and downloading
blocks from multiple peers in parallel, for blocks corresponding to
validated headers. Downloading the headers succeeds within minutes,
and within seconds you have enough to start fetching blocks. After
that point, you can keep a "download window" full with outstanding
block requests, and as blocks go much slower than headers, the headers
process never becomes a blocker for blocks to download.
Unless we're talking about a system with billions of headers to
download, I don't think this is a worthwhile optimization.
--
Pieter
|