1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
|
Received: from sog-mx-1.v43.ch3.sourceforge.com ([172.29.43.191]
helo=mx.sourceforge.net)
by sfs-ml-3.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
(envelope-from <pieter.wuille@gmail.com>) id 1Yqjdl-0006L3-II
for bitcoin-development@lists.sourceforge.net;
Fri, 08 May 2015 14:52:37 +0000
Received-SPF: pass (sog-mx-1.v43.ch3.sourceforge.com: domain of gmail.com
designates 209.85.215.53 as permitted sender)
client-ip=209.85.215.53; envelope-from=pieter.wuille@gmail.com;
helo=mail-la0-f53.google.com;
Received: from mail-la0-f53.google.com ([209.85.215.53])
by sog-mx-1.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
(Exim 4.76) id 1Yqjdj-0007g5-VD
for bitcoin-development@lists.sourceforge.net;
Fri, 08 May 2015 14:52:37 +0000
Received: by labbd9 with SMTP id bd9so54445128lab.2
for <bitcoin-development@lists.sourceforge.net>;
Fri, 08 May 2015 07:52:29 -0700 (PDT)
MIME-Version: 1.0
X-Received: by 10.112.145.136 with SMTP id su8mr3163639lbb.61.1431096749563;
Fri, 08 May 2015 07:52:29 -0700 (PDT)
Received: by 10.112.19.7 with HTTP; Fri, 8 May 2015 07:52:28 -0700 (PDT)
Received: by 10.112.19.7 with HTTP; Fri, 8 May 2015 07:52:28 -0700 (PDT)
In-Reply-To: <554CC19D.2010306@xylon.de>
References: <554CC19D.2010306@xylon.de>
Date: Fri, 8 May 2015 07:52:28 -0700
Message-ID: <CAPg+sBhR2UcpjCxOEXw0gC9scG6OXMcyPLkfjfw7fP736XbGSw@mail.gmail.com>
From: Pieter Wuille <pieter.wuille@gmail.com>
To: Arne Brutschy <abrutschy@xylon.de>
Content-Type: multipart/alternative; boundary=e89a8f502d9265a22e0515932db7
X-Spam-Score: -0.6 (/)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
See http://spamassassin.org/tag/ for more details.
-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
sender-domain
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(pieter.wuille[at]gmail.com)
-0.0 SPF_PASS SPF: sender matches SPF record
1.0 HTML_MESSAGE BODY: HTML included in message
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
author's domain
0.1 DKIM_SIGNED Message has a DKIM or DK signature,
not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
X-Headers-End: 1Yqjdj-0007g5-VD
Cc: Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Subject: Re: [Bitcoin-development] Removing transaction data from blocks
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Fri, 08 May 2015 14:52:37 -0000
--e89a8f502d9265a22e0515932db7
Content-Type: text/plain; charset=ISO-8859-1
So, there are several ideas about how to reduce the size of blocks being
sent on the network:
* Matt Corallo's relay network, which internally works by remembering the
last 5000 (i believe?) transactions sent by the peer, and allowing the peer
to backreference those rather than retransmit them inside block data. This
exists and works today.
* Gavin Andresen's IBLT based set reconciliation for blocks based on what a
peer expects the new block to contain.
* Greg Maxwell's network block coding, which is based on erasure coding,
and also supports sharding (everyone sends some block data to everyone,
rather fetching from one peer).
However, the primary purpose is not to reduce bandwidth (though that is a
nice side advantage). The purpose is reducing propagation delay. Larger
propagation delays across the network (relative to the inter-block period)
result in higher forking rates. If the forking rate gets very high, the
network may fail to converge entirely, but even long before that point, the
higher the forking rate is, the higher the advantage of larger (and better
connected) pools over smaller ones. This is why, in my opinion,
guaranteeing fast propagation is one of the most essential responsibility
of full nodes to avoid centralization pressure.
Also, none of this would let us "get rid of the block size" at all. All
transactions still have to be transferred and processed, and due to
inherent latencies of communication across the globe, the higher the
transaction rate is, the higher the number of transactions in blocks will
be that peers have not yet heard about. You can institute a policy to not
include too recent transactions in blocks, but again, this favors larger
miners over smaller ones.
Also, if the end goal is propagation delay, just minimizing the amount of
data transferred is not enough. You also need to make sure the
communication mechanism does not add huge processing overheads or adds
unnecessary roundtrips. In fact, this is the key difference between the 3
techniques listed above, and several people are working on refining and
optimizing these mechanisms to make them practically usable.
On May 8, 2015 7:23 AM, "Arne Brutschy" <abrutschy@xylon.de> wrote:
> Hello,
>
> At DevCore London, Gavin mentioned the idea that we could get rid of
> sending full blocks. Instead, newly minted blocks would only be
> distributed as block headers plus all hashes of the transactions
> included in the block. The assumption would be that nodes have already
> the majority of these transactions in their mempool.
>
> The advantages are clear: it's more efficient, as we would send
> transactions only once over the network, and it's fast as the resulting
> blocks would be small. Moreover, we would get rid of the blocksize limit
> for a long time.
>
> Unfortunately, I am too ignorant of bitcoin core's internals to judge
> the changes required to make this happen. (I guess we'd require a new
> block format and a way to bulk-request missing transactions.)
>
> However, I'm curious to hear what others with a better grasp of bitcoin
> core's internals have to say about it.
>
> Regards,
> Arne
>
>
> ------------------------------------------------------------------------------
> One dashboard for servers and applications across Physical-Virtual-Cloud
> Widest out-of-the-box monitoring support with 50+ applications
> Performance metrics, stats and reports that give you Actionable Insights
> Deep dive visibility with transaction tracing using APM Insight.
> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
--e89a8f502d9265a22e0515932db7
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<p dir=3D"ltr">So, there are several ideas about how to reduce the size of =
blocks being sent on the network:<br>
* Matt Corallo's relay network, which internally works by remembering t=
he last 5000 (i believe?) transactions sent by the peer, and allowing the p=
eer to backreference those rather than retransmit them inside block data. T=
his exists and works today.<br>
* Gavin Andresen's IBLT based set reconciliation for blocks based on wh=
at a peer expects the new block to contain.<br>
* Greg Maxwell's network block coding, which is based on erasure coding=
, and also supports sharding (everyone sends some block data to everyone, r=
ather fetching from one peer).</p>
<p dir=3D"ltr">However, the primary purpose is not to reduce bandwidth (tho=
ugh that is a nice side advantage). The purpose is reducing propagation del=
ay. Larger propagation delays across the network (relative to the inter-blo=
ck period) result in higher forking rates. If the forking rate gets very hi=
gh, the network may fail to converge entirely, but even long before that po=
int, the higher the forking rate is, the higher the advantage of larger (an=
d better connected) pools over smaller ones. This is why, in my opinion, gu=
aranteeing fast propagation is one of the most essential responsibility of =
full nodes to avoid centralization pressure.</p>
<p dir=3D"ltr">Also, none of this would let us "get rid of the block s=
ize" at all. All transactions still have to be transferred and process=
ed, and due to inherent latencies of communication across the globe, the hi=
gher the transaction rate is, the higher the number of transactions in bloc=
ks will be that peers have not yet heard about. You can institute a policy =
to not include too recent transactions in blocks, but again, this favors la=
rger miners over smaller ones.</p>
<p dir=3D"ltr">Also, if the end goal is propagation delay, just minimizing =
the amount of data transferred is not enough. You also need to make sure th=
e communication mechanism does not add huge processing overheads or adds un=
necessary roundtrips. In fact, this is the key difference between the 3 tec=
hniques listed above, and several people are working on refining and optimi=
zing these mechanisms to make them practically usable.</p>
<div class=3D"gmail_quote">On May 8, 2015 7:23 AM, "Arne Brutschy"=
; <<a href=3D"mailto:abrutschy@xylon.de">abrutschy@xylon.de</a>> wrot=
e:<br type=3D"attribution"><blockquote class=3D"gmail_quote" style=3D"margi=
n:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
<br>
At DevCore London, Gavin mentioned the idea that we could get rid of<br>
sending full blocks. Instead, newly minted blocks would only be<br>
distributed as block headers plus all hashes of the transactions<br>
included in the block. The assumption would be that nodes have already<br>
the majority of these transactions in their mempool.<br>
<br>
The advantages are clear: it's more efficient, as we would send<br>
transactions only once over the network, and it's fast as the resulting=
<br>
blocks would be small. Moreover, we would get rid of the blocksize limit<br=
>
for a long time.<br>
<br>
Unfortunately, I am too ignorant of bitcoin core's internals to judge<b=
r>
the changes required to make this happen. (I guess we'd require a new<b=
r>
block format and a way to bulk-request missing transactions.)<br>
<br>
However, I'm curious to hear what others with a better grasp of bitcoin=
<br>
core's internals have to say about it.<br>
<br>
Regards,<br>
Arne<br>
<br>
---------------------------------------------------------------------------=
---<br>
One dashboard for servers and applications across Physical-Virtual-Cloud<br=
>
Widest out-of-the-box monitoring support with 50+ applications<br>
Performance metrics, stats and reports that give you Actionable Insights<br=
>
Deep dive visibility with transaction tracing using APM Insight.<br>
<a href=3D"http://ad.doubleclick.net/ddm/clk/290420510;117567292;y" target=
=3D"_blank">http://ad.doubleclick.net/ddm/clk/290420510;117567292;y</a><br>
_______________________________________________<br>
Bitcoin-development mailing list<br>
<a href=3D"mailto:Bitcoin-development@lists.sourceforge.net">Bitcoin-develo=
pment@lists.sourceforge.net</a><br>
<a href=3D"https://lists.sourceforge.net/lists/listinfo/bitcoin-development=
" target=3D"_blank">https://lists.sourceforge.net/lists/listinfo/bitcoin-de=
velopment</a><br>
</blockquote></div>
--e89a8f502d9265a22e0515932db7--
|
