1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
|
Return-Path: <j@toom.im>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
[172.17.192.35])
by mail.linuxfoundation.org (Postfix) with ESMTPS id ECE90D2B
for <bitcoin-dev@lists.linuxfoundation.org>;
Fri, 26 Feb 2016 05:40:37 +0000 (UTC)
X-Greylist: delayed 00:05:19 by SQLgrey-1.7.6
Received: from d.spam.sonic.net (d.spam.sonic.net [69.12.208.70])
by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 3D96F145
for <bitcoin-dev@lists.linuxfoundation.org>;
Fri, 26 Feb 2016 05:40:37 +0000 (UTC)
Received: from d.mail.sonic.net (a.spam-proxy.sonic.net [69.12.221.245])
by d.spam.sonic.net (8.14.4/8.14.4) with ESMTP id u1Q5ZHN1012265
(version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-GCM-SHA256 bits=128
verify=NOT) for <bitcoin-dev@lists.linuxfoundation.org>;
Thu, 25 Feb 2016 21:35:17 -0800
Received: from [192.168.1.190] (63.135.62.197.nwinternet.com [63.135.62.197]
(may be forged)) (authenticated bits=0)
by d.mail.sonic.net (8.15.1/8.15.1) with ESMTPSA id u1Q5ZFNE017210
(version=TLSv1 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT)
for <bitcoin-dev@lists.linuxfoundation.org>;
Thu, 25 Feb 2016 21:35:16 -0800
From: Jonathan Toomim <j@toom.im>
X-Pgp-Agent: GPGMail 2.6b2
Content-Type: multipart/signed;
boundary="Apple-Mail=_9AF4F793-F5F2-48EB-9981-8B07B9B69FB3";
protocol="application/pgp-signature"; micalg=pgp-sha512
Date: Thu, 25 Feb 2016 21:35:14 -0800
Message-Id: <B186E7A6-0FD4-4C82-B42F-7EE61D420A7E@toom.im>
To: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>
Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\))
X-Mailer: Apple Mail (2.3112)
X-Sonic-CAuth: UmFuZG9tSVbba7V/pSSIdm07Pt3dg4HCvlB7gzw4St/CyplZDs6IMa4oct578S6urZ50LB5JGFjCc5WoYBIvZt2iDbCnHNEX
X-Sonic-ID: C;cNfPtkrc5RG2WE0iDwNmVA== M;SNEvt0rc5RG2WE0iDwNmVA==
X-Sonic-Spam-Details: 0.0/5.0 by cerberusd
X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,HTML_MESSAGE,
RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
smtp1.linux-foundation.org
Subject: [bitcoin-dev] INV overhead and batched INVs to reduce full node
traffic
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Fri, 26 Feb 2016 05:40:38 -0000
--Apple-Mail=_9AF4F793-F5F2-48EB-9981-8B07B9B69FB3
Content-Type: multipart/alternative;
boundary="Apple-Mail=_66A93333-4E04-49C8-BC47-ADDA071CABD1"
--Apple-Mail=_66A93333-4E04-49C8-BC47-ADDA071CABD1
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=us-ascii
The INV scheme used by Bitcoin is not very efficient at all. Once you =
take into account Bitcoin, TCP (including ACKs), IP, and ethernet =
overheads, each INV takes 193 bytes, according to wireshark. That's 127 =
bytes for the INV message and 66 bytes for the ACK. All of this is for =
32 bytes of payload, for an "efficiency" of 16.5% (i.e. 83.5% overhead). =
For a 400 byte transaction with 20 peers, this can result in 3860 bytes =
sent in INVs for only 400 bytes of actual data.
An improvement that I've been thinking about implementing (after =
Blocktorrent) is an option for batched INVs. Including the hashes for =
two txes per IP packet instead of one would increase the INV size to 229 =
bytes for 64 bytes of payload -- that is, you add 36 bytes to the packet =
for every 32 bytes of actual payload. This is a marginal efficiency of =
88.8% for each hash after the first. This is *much* better.
Waiting a short period of time to accumulate several hashes together and =
send them as a batched INV could easily reduce the traffic of running =
bitcoin nodes by a factor of 2, and possibly even more than that. =
However, if too many people used it, such a technique would slow down =
the propagation of transactions across the bitcoin network slightly, =
which might make some people unhappy. The ill effects could likely be =
mitigated by choosing a different batch size for each peer based on each =
peer's preferences. Each node could choose one or two peers to which =
they send INVs in batches of one or two, four more peers in which they =
send batches of two to four, and the rest in batches of four to eight, =
for example.
(This is a continuation of a conversation started on =
https://bitcointalk.org/index.php?topic=3D1377345 =
<https://bitcointalk.org/index.php?topic=3D1377345.new#new>.)
Jonathan
--Apple-Mail=_66A93333-4E04-49C8-BC47-ADDA071CABD1
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=us-ascii
<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><div class=3D"">The INV scheme used by Bitcoin is not very =
efficient at all. Once you take into account Bitcoin, TCP (including =
ACKs), IP, and ethernet overheads, each INV takes 193 bytes, according =
to wireshark. That's 127 bytes for the INV message and 66 bytes for the =
ACK. All of this is for 32 bytes of payload, for an "efficiency" of =
16.5% (i.e. 83.5% overhead). For a 400 byte transaction with 20 peers, =
this can result in 3860 bytes sent in INVs for only 400 bytes of actual =
data.</div><div class=3D""><br class=3D""></div><div class=3D"">An =
improvement that I've been thinking about implementing (after =
Blocktorrent) is an option for batched INVs. Including the hashes for =
two txes per IP packet instead of one would increase the INV size to 229 =
bytes for 64 bytes of payload -- that is, you add 36 bytes to the packet =
for every 32 bytes of actual payload. This is a marginal efficiency of =
88.8% for each hash after the first. This is *much* better.</div><div =
class=3D""><br class=3D""></div><div class=3D"">Waiting a short period =
of time to accumulate several hashes together and send them as a batched =
INV could easily reduce the traffic of running bitcoin nodes by a factor =
of 2, and possibly even more than that. However, if too many people used =
it, such a technique would slow down the propagation of transactions =
across the bitcoin network slightly, which might make some people =
unhappy. The ill effects could likely be mitigated by choosing a =
different batch size for each peer based on each peer's preferences. =
Each node could choose one or two peers to which they send INVs in =
batches of one or two, four more peers in which they send batches of two =
to four, and the rest in batches of four to eight, for =
example.</div><div class=3D""><br class=3D""></div><div class=3D"">(This =
is a continuation of a conversation started on <a =
href=3D"https://bitcointalk.org/index.php?topic=3D1377345.new#new" =
class=3D"">https://bitcointalk.org/index.php?topic=3D1377345</a>.)</div><d=
iv class=3D""><br class=3D""></div><div =
class=3D"">Jonathan</div></body></html>=
--Apple-Mail=_66A93333-4E04-49C8-BC47-ADDA071CABD1--
--Apple-Mail=_9AF4F793-F5F2-48EB-9981-8B07B9B69FB3
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=signature.asc
Content-Type: application/pgp-signature;
name=signature.asc
Content-Description: Message signed with OpenPGP using GPGMail
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org
iQEcBAEBCgAGBQJWz+QTAAoJEIEuMk4MG0P1Ai0H+QF4WQlpUe4j9qPfkSdCfxis
d7H2sm038ISefa0nNSztK07jczCu+uA5xB7tdotDIJ3ukFTZ1QhxZMxhcs5I0rq7
Wjc0Id/nsg0FJO5VBfI9Jg7ZrEEnwnJBud7I7oqk8bFTfEXt4xFXiIOOge7wkoeT
zOoQuzIUfSa6WKbSr7XdV1R2BvjQrfyjhdB13wrw7qie4Xt6kh5oXViTVsrIZZXS
HHSTuK0db1KbyHIsgUNHcLhbFJfzbknH7gqM7K7MJDrOkuTwXe0UYYRliJFKtyEY
+y1AZJlVNjplIrcxOTeW3hcpDv43bQuhE7pD8ZO6FrztBQBsAjG5wtN5j6IL8/g=
=Osn2
-----END PGP SIGNATURE-----
--Apple-Mail=_9AF4F793-F5F2-48EB-9981-8B07B9B69FB3--
|
