1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
|
Received: from sog-mx-2.v43.ch3.sourceforge.com ([172.29.43.192]
helo=mx.sourceforge.net)
by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
(envelope-from <rick.wesson@iidf.org>) id 1R0wlh-0007mc-68
for bitcoin-development@lists.sourceforge.net;
Tue, 06 Sep 2011 14:36:53 +0000
X-ACL-Warn:
Received: from mail-gx0-f175.google.com ([209.85.161.175])
by sog-mx-2.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-MD5:128)
(Exim 4.76) id 1R0wlf-0002Wm-Oc
for bitcoin-development@lists.sourceforge.net;
Tue, 06 Sep 2011 14:36:53 +0000
Received: by gxk3 with SMTP id 3so4577009gxk.34
for <bitcoin-development@lists.sourceforge.net>;
Tue, 06 Sep 2011 07:36:46 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.150.209.12 with SMTP id h12mr3971822ybg.395.1315319805241;
Tue, 06 Sep 2011 07:36:45 -0700 (PDT)
Received: by 10.150.186.4 with HTTP; Tue, 6 Sep 2011 07:36:45 -0700 (PDT)
In-Reply-To: <4E65CEE6.7030002@gmail.com>
References: <4E65CEE6.7030002@gmail.com>
Date: Tue, 6 Sep 2011 07:36:45 -0700
Message-ID: <CAJ1JLtsOsnsu+UTwTV5RMmco9xEAWOgoYcjT=-ZeFQ7VSJX0ZA@mail.gmail.com>
From: Rick Wesson <rick@support-intelligence.com>
To: shadders.del@gmail.com
Content-Type: multipart/alternative; boundary=00151743f90ec1e9d004ac46c16b
X-Spam-Score: 1.0 (+)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
See http://spamassassin.org/tag/ for more details.
1.0 HTML_MESSAGE BODY: HTML included in message
X-Headers-End: 1R0wlf-0002Wm-Oc
Cc: bitcoin-development@lists.sourceforge.net
Subject: Re: [Bitcoin-development] Building a node crawler to map network
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Tue, 06 Sep 2011 14:36:53 -0000
--00151743f90ec1e9d004ac46c16b
Content-Type: text/plain; charset=ISO-8859-1
I've got minna patches for nio based on bitcoinj. I've enumerated the
network a few times and am working on a DNS seed service as well as some
weather reports.
Happy to start a branch when the committers are ready.
-rick
On Tue, Sep 6, 2011 at 12:42 AM, Steve <shadders.del@gmail.com> wrote:
> Hi All,
>
> I started messing around today with building a node crawler to try and
> map out the bitcoin network and hopefully provide some useful
> statistics. It's very basic so far using a mutilated bitcoinj to
> connect (due me being java developer and not having a clue with c/c++).
> If it's worthwhile I'll hack bitcoinj some more to run on top Netty to
> take advantage of it's NIO architecture (netty's been shown to handle
> 1/2 million concurrent connections so would be ideal for the purpose).
>
> Hoping to a get a bit of input into what would be useful as well as
> strategy for getting max possible connections without distorted data. I
> seem to recall Gavin talking about the need for some kind of network
> health monitoring so I assume there's a need for something like this...
>
> Firstly at the moment basically I'm just storing version message and the
> results of getaddr for each node that I can connect to. Is there any
> other useful info that can be extracted from a node that's worth
> collecting?
>
> Second and main issue is how to connect. From my first very basic
> probing it seems the very vast majority of nodes don't accept incoming
> connections no doubt due to lack of upnp. So it seems the active crawl
> approach is not really ideal for the purpose. Even if it was used the
> resultant data would be hopelessly distorted.
>
> A honeypot approach would probably be better if there was some way to
> make a node 'attractive' to other nodes to connect to. That way it
> could capture non-listening nodes as well. If there is some way to
> influence other nodes to connect to the crawler node that solves the
> problem. If there isn't which I suspect is the case then perhaps
> another approach is to build an easy to deploy crawler node that many
> volunteers could run and that could then upload collected data to a
> central repository.
>
> While I'm asking questions I'll add one more regarding the getaddr
> message. It seems most nodes return about 1000 addresses in response to
> this message. Obviously most of these nodes haven't actually talked to
> all 1000 on the list so where does this list come from? Is it mixture
> of addresses obtained from other nodes somehow sorted by timestamp?
> Does it include some nodes discovered by IRC/DNS? Or are those only used
> to find the first nodes to connect to?
>
> Thanks for any input... Hopefully I can build something that's useful
> for the network...
>
>
> ------------------------------------------------------------------------------
> Special Offer -- Download ArcSight Logger for FREE!
> Finally, a world-class log management solution at an even better
> price-free! And you'll get a free "Love Thy Logs" t-shirt when you
> download Logger. Secure your free ArcSight Logger TODAY!
> http://p.sf.net/sfu/arcsisghtdev2dev
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
--00151743f90ec1e9d004ac46c16b
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
I've got minna patches for nio based on bitcoinj. I've enumerated t=
he network a few times and am working on a DNS seed service as well as some=
weather reports.<div><br></div><div>Happy to start a branch when the commi=
tters are ready.</div>
<div><br></div><div>-rick</div><div><br><br><div class=3D"gmail_quote">On T=
ue, Sep 6, 2011 at 12:42 AM, Steve <span dir=3D"ltr"><<a href=3D"mailto:=
shadders.del@gmail.com">shadders.del@gmail.com</a>></span> wrote:<br><bl=
ockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #=
ccc solid;padding-left:1ex;">
Hi All,<br>
<br>
I started messing around today with building a node crawler to try and<br>
map out the bitcoin network and hopefully provide some useful<br>
statistics. =A0It's very basic so far using a mutilated bitcoinj to<br>
connect (due me being java developer and not having a clue with c/c++).<br>
=A0If it's worthwhile I'll hack bitcoinj some more to run on top N=
etty to<br>
take advantage of it's NIO architecture (netty's been shown to hand=
le<br>
1/2 million concurrent connections so would be ideal for the purpose).<br>
<br>
Hoping to a get a bit of input into what would be useful as well as<br>
strategy for getting max possible connections without distorted data. =A0I<=
br>
seem to recall Gavin talking about the need for some kind of network<br>
health monitoring so I assume there's a need for something like this...=
<br>
<br>
Firstly at the moment basically I'm just storing version message and th=
e<br>
results of getaddr for each node that I can connect to. =A0Is there any<br>
other useful info that can be extracted from a node that's worth collec=
ting?<br>
<br>
Second and main issue is how to connect. =A0From my first very basic<br>
probing it seems the very vast majority of nodes don't accept incoming<=
br>
connections no doubt due to lack of upnp. =A0So it seems the active crawl<b=
r>
approach is not really ideal for the purpose. =A0Even if it was used the<br=
>
resultant data would be hopelessly distorted.<br>
<br>
A honeypot approach would probably be better if there was some way to<br>
make a node 'attractive' to other nodes to connect to. =A0That way =
it<br>
could capture non-listening nodes as well. =A0If there is some way to<br>
influence other nodes to connect to the crawler node that solves the<br>
problem. =A0If there isn't which I suspect is the case then perhaps<br>
another approach is to build an easy to deploy crawler node that many<br>
volunteers could run and that could then upload collected data to a<br>
central repository.<br>
<br>
While I'm asking questions I'll add one more regarding the getaddr<=
br>
message. =A0It seems most nodes return about 1000 addresses in response to<=
br>
this message. =A0Obviously most of these nodes haven't actually talked =
to<br>
all 1000 on the list so where does this list come from? =A0Is it mixture<br=
>
of addresses obtained from other nodes somehow sorted by timestamp?<br>
Does it include some nodes discovered by IRC/DNS? Or are those only used<br=
>
to find the first nodes to connect to?<br>
<br>
Thanks for any input... Hopefully I can build something that's useful<b=
r>
for the network...<br>
<br>
---------------------------------------------------------------------------=
---<br>
Special Offer -- Download ArcSight Logger for FREE!<br>
Finally, a world-class log management solution at an even better<br>
price-free! And you'll get a free "Love Thy Logs" t-shirt whe=
n you<br>
download Logger. Secure your free ArcSight Logger TODAY!<br>
<a href=3D"http://p.sf.net/sfu/arcsisghtdev2dev" target=3D"_blank">http://p=
.sf.net/sfu/arcsisghtdev2dev</a><br>
_______________________________________________<br>
Bitcoin-development mailing list<br>
<a href=3D"mailto:Bitcoin-development@lists.sourceforge.net">Bitcoin-develo=
pment@lists.sourceforge.net</a><br>
<a href=3D"https://lists.sourceforge.net/lists/listinfo/bitcoin-development=
" target=3D"_blank">https://lists.sourceforge.net/lists/listinfo/bitcoin-de=
velopment</a><br>
</blockquote></div><br></div>
--00151743f90ec1e9d004ac46c16b--
|
