1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
|
Received: from sog-mx-4.v43.ch3.sourceforge.com ([172.29.43.194]
helo=mx.sourceforge.net)
by sfs-ml-4.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
(envelope-from <shadders.del@gmail.com>) id 1R0xWi-0004gd-CQ
for bitcoin-development@lists.sourceforge.net;
Tue, 06 Sep 2011 15:25:28 +0000
Received-SPF: pass (sog-mx-4.v43.ch3.sourceforge.com: domain of gmail.com
designates 209.85.216.182 as permitted sender)
client-ip=209.85.216.182; envelope-from=shadders.del@gmail.com;
helo=mail-qy0-f182.google.com;
Received: from mail-qy0-f182.google.com ([209.85.216.182])
by sog-mx-4.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
(Exim 4.76) id 1R0xWe-0001Gs-CF
for bitcoin-development@lists.sourceforge.net;
Tue, 06 Sep 2011 15:25:28 +0000
Received: by qyk9 with SMTP id 9so3869725qyk.13
for <bitcoin-development@lists.sourceforge.net>;
Tue, 06 Sep 2011 08:25:19 -0700 (PDT)
Received: by 10.52.177.7 with SMTP id cm7mr1150210vdc.37.1315322718894;
Tue, 06 Sep 2011 08:25:18 -0700 (PDT)
Received: from [10.1.1.50] (155.88-67-202.dynamic.dsl.syd.iprimus.net.au
[202.67.88.155])
by mx.google.com with ESMTPS id df6sm304167vdb.22.2011.09.06.08.25.15
(version=SSLv3 cipher=OTHER); Tue, 06 Sep 2011 08:25:18 -0700 (PDT)
Message-ID: <4E663B55.9050508@gmail.com>
Date: Wed, 07 Sep 2011 01:25:09 +1000
From: Steve <shadders.del@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
rv:1.9.2.21) Gecko/20110831 Thunderbird/3.1.13
MIME-Version: 1.0
To: Mike Hearn <mike@plan99.net>
References: <4E65CEE6.7030002@gmail.com> <4E65DA06.9060403@gmail.com> <CALxbBHUajARXc1oA-NjD+U8hW5uSqF=u4ZHHBfcmT_O8GjpNiA@mail.gmail.com> <CANEZrP0VXDUs_mAKCVKD1Q0ijyb989oADrCN1zTZ1nnN_JQ=cQ@mail.gmail.com> <4E661FAE.9020008@gmail.com> <CANEZrP3=UPYkBQo6b421xaMGyP4BsGiw8DBuM8pT2ow1Vom9JQ@mail.gmail.com> <4E662B79.5090303@gmail.com>
<CANEZrP2Wh82sqGjZDn_M=UPufBCU4fP9zEXV_K8JpgVF8O1FCw@mail.gmail.com>
In-Reply-To: <CANEZrP2Wh82sqGjZDn_M=UPufBCU4fP9zEXV_K8JpgVF8O1FCw@mail.gmail.com>
Content-Type: multipart/alternative;
boundary="------------060206040500050907030804"
X-Spam-Score: -0.9 (/)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
See http://spamassassin.org/tag/ for more details.
-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
sender-domain
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(shadders.del[at]gmail.com)
-0.0 SPF_PASS SPF: sender matches SPF record
1.0 HTML_MESSAGE BODY: HTML included in message
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
author's domain
0.1 DKIM_SIGNED Message has a DKIM or DK signature,
not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
-0.3 AWL AWL: From: address is in the auto white-list
X-Headers-End: 1R0xWe-0001Gs-CF
Cc: bitcoin-development@lists.sourceforge.net
Subject: Re: [Bitcoin-development] Building a node crawler to map network
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
Reply-To: shadders.del@gmail.com
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Tue, 06 Sep 2011 15:25:28 -0000
This is a multi-part message in MIME format.
--------------060206040500050907030804
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Thanks for the overview Mike. I just bailed up Gavin on IRC and between
that convo and what you've just written I'm starting to picture a plan
in my head... This sounds right up my alley, I wish I didn't have to go
to bed right now as I've got a ton of ideas buzzing around I'd like to
get started on right now. But I'll be onto it as soon as I've got a
free moment...
On 07/09/11 00:52, Mike Hearn wrote:
> On Tue, Sep 6, 2011 at 4:17 PM, Steve <shadders.del@gmail.com
> <mailto:shadders.del@gmail.com>> wrote:
>
> I'm not really understanding the use case though. I believe most
> bitcoind's have a default max connections of 8. Is the goal to
> increase this without fundamentally altering the bitcoind
> concurrency model?
>
>
> bitcoind already uses asynchronous IO. That's not the problem.
>
> The issue came up in a conversation about scalability. If Bitcoins
> popularity continues to grow, users are very likely to migrate away
> from running full verifying nodes to lightweight clients, either a
> different mode of the Satoshi client or different implementations like
> the Android Wallet or MultiBit.
>
> Lightweight clients cannot verify thus should not relay. And they'll
> be run by users who just want to send/receive coins from time to time,
> so don't leave the programs running 24/7. The result could be running
> out of sockets (like we have had problems with recently). It's
> especially true because lightweight clients cannot check transactions
> for themselves. If they want to show transactions appearing
> immediately (and they do), they have to use "heard from lots of nodes"
> as a proxy for validity. So lightweight clients are likely to be
> socket intensive.
>
> We could solve this by just hoping that lots of people run full nodes.
> The problem is that a full node is quite an intensive thing already,
> it uses lots of CPU and disk seeks, and will just get more expensive
> in future. And as transaction traffic increases, that leaves less CPU
> time available to service thousands of connected clients. The ROI of
> bringing up a new node decreases at the same time as the userbase
> increases.
>
> One traditional approach to solving this is frontend proxies.
> Jabber.com/org used this technique many years ago, and Google has also
> used it to scale up the lockservice
> <http://static.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en/us/papers/chubby-osdi06.pdf> (see
> section 3.1). It's effective because often maintaining connections to
> thousands of clients doesn't involve much brainwork, just shifting
> bytes around. This is especially true of Bitcoin. So if somebody is
> running a full node already they could increase their client capacity
> by just bringing up a frontend proxy and having it handle things like
> outbound tx broadcasts/deduping inbound broadcasts, connection setup,
> relaying recently found blocks etc. A well written proxy could
> probably support tens of thousands of simultaneous clients which frees
> up the bitcoinds time for verification and wallet manipulation.
--------------060206040500050907030804
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Thanks for the overview Mike. I just bailed up Gavin on IRC and
between that convo and what you've just written I'm starting to
picture a plan in my head... This sounds right up my alley, I wish I
didn't have to go to bed right now as I've got a ton of ideas
buzzing around I'd like to get started on right now. But I'll be
onto it as soon as I've got a free moment...<br>
<br>
On 07/09/11 00:52, Mike Hearn wrote:
<blockquote
cite="mid:CANEZrP2Wh82sqGjZDn_M=UPufBCU4fP9zEXV_K8JpgVF8O1FCw@mail.gmail.com"
type="cite">
<div class="gmail_quote">On Tue, Sep 6, 2011 at 4:17 PM, Steve <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:shadders.del@gmail.com">shadders.del@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt
0.8ex; border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;">
<div bgcolor="#ffffff" text="#000000">I'm not really
understanding the use case though. I believe most
bitcoind's have a default max connections of 8. Is the goal
to increase this without fundamentally altering the bitcoind
concurrency model?</div>
</blockquote>
<div><br>
</div>
<div>bitcoind already uses asynchronous IO. That's not the
problem.</div>
<div><br>
</div>
<div>The issue came up in a conversation about scalability. If
Bitcoins popularity continues to grow, users are very likely
to migrate away from running full verifying nodes to
lightweight clients, either a different mode of the Satoshi
client or different implementations like the Android Wallet or
MultiBit.</div>
<div><br>
</div>
<div>Lightweight clients cannot verify thus should not relay.
And they'll be run by users who just want to send/receive
coins from time to time, so don't leave the programs running
24/7. The result could be running out of sockets (like we have
had problems with recently). It's especially true because
lightweight clients cannot check transactions for themselves.
If they want to show transactions appearing immediately (and
they do), they have to use "heard from lots of nodes" as a
proxy for validity. So lightweight clients are likely to be
socket intensive.</div>
<div><br>
</div>
<div>We could solve this by just hoping that lots of people run
full nodes. The problem is that a full node is quite an
intensive thing already, it uses lots of CPU and disk seeks,
and will just get more expensive in future. And as transaction
traffic increases, that leaves less CPU time available to
service thousands of connected clients. The ROI of bringing up
a new node decreases at the same time as the userbase
increases.</div>
<div><br>
</div>
<div>One traditional approach to solving this is frontend
proxies. Jabber.com/org used this technique many years ago,
and Google has also used it to scale up <a
moz-do-not-send="true"
href="http://static.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en/us/papers/chubby-osdi06.pdf">the
lockservice</a> (see section 3.1). It's effective because
often maintaining connections to thousands of clients doesn't
involve much brainwork, just shifting bytes around. This is
especially true of Bitcoin. So if somebody is running a full
node already they could increase their client capacity by just
bringing up a frontend proxy and having it handle things like
outbound tx broadcasts/deduping inbound broadcasts, connection
setup, relaying recently found blocks etc. A well written
proxy could probably support tens of thousands of simultaneous
clients which frees up the bitcoinds time for verification and
wallet manipulation.</div>
</div>
</blockquote>
</body>
</html>
--------------060206040500050907030804--
|
