Received: from sog-mx-2.v43.ch3.sourceforge.com ([172.29.43.192] helo=mx.sourceforge.net) by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1R0wlh-0007mc-68 for bitcoin-development@lists.sourceforge.net; Tue, 06 Sep 2011 14:36:53 +0000 X-ACL-Warn: Received: from mail-gx0-f175.google.com ([209.85.161.175]) by sog-mx-2.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-MD5:128) (Exim 4.76) id 1R0wlf-0002Wm-Oc for bitcoin-development@lists.sourceforge.net; Tue, 06 Sep 2011 14:36:53 +0000 Received: by gxk3 with SMTP id 3so4577009gxk.34 for ; Tue, 06 Sep 2011 07:36:46 -0700 (PDT) MIME-Version: 1.0 Received: by 10.150.209.12 with SMTP id h12mr3971822ybg.395.1315319805241; Tue, 06 Sep 2011 07:36:45 -0700 (PDT) Received: by 10.150.186.4 with HTTP; Tue, 6 Sep 2011 07:36:45 -0700 (PDT) In-Reply-To: <4E65CEE6.7030002@gmail.com> References: <4E65CEE6.7030002@gmail.com> Date: Tue, 6 Sep 2011 07:36:45 -0700 Message-ID: From: Rick Wesson To: shadders.del@gmail.com Content-Type: multipart/alternative; boundary=00151743f90ec1e9d004ac46c16b X-Spam-Score: 1.0 (+) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. 1.0 HTML_MESSAGE BODY: HTML included in message X-Headers-End: 1R0wlf-0002Wm-Oc Cc: bitcoin-development@lists.sourceforge.net Subject: Re: [Bitcoin-development] Building a node crawler to map network X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Sep 2011 14:36:53 -0000 --00151743f90ec1e9d004ac46c16b Content-Type: text/plain; charset=ISO-8859-1 I've got minna patches for nio based on bitcoinj. I've enumerated the network a few times and am working on a DNS seed service as well as some weather reports. Happy to start a branch when the committers are ready. -rick On Tue, Sep 6, 2011 at 12:42 AM, Steve wrote: > Hi All, > > I started messing around today with building a node crawler to try and > map out the bitcoin network and hopefully provide some useful > statistics. It's very basic so far using a mutilated bitcoinj to > connect (due me being java developer and not having a clue with c/c++). > If it's worthwhile I'll hack bitcoinj some more to run on top Netty to > take advantage of it's NIO architecture (netty's been shown to handle > 1/2 million concurrent connections so would be ideal for the purpose). > > Hoping to a get a bit of input into what would be useful as well as > strategy for getting max possible connections without distorted data. I > seem to recall Gavin talking about the need for some kind of network > health monitoring so I assume there's a need for something like this... > > Firstly at the moment basically I'm just storing version message and the > results of getaddr for each node that I can connect to. Is there any > other useful info that can be extracted from a node that's worth > collecting? > > Second and main issue is how to connect. From my first very basic > probing it seems the very vast majority of nodes don't accept incoming > connections no doubt due to lack of upnp. So it seems the active crawl > approach is not really ideal for the purpose. Even if it was used the > resultant data would be hopelessly distorted. > > A honeypot approach would probably be better if there was some way to > make a node 'attractive' to other nodes to connect to. That way it > could capture non-listening nodes as well. If there is some way to > influence other nodes to connect to the crawler node that solves the > problem. If there isn't which I suspect is the case then perhaps > another approach is to build an easy to deploy crawler node that many > volunteers could run and that could then upload collected data to a > central repository. > > While I'm asking questions I'll add one more regarding the getaddr > message. It seems most nodes return about 1000 addresses in response to > this message. Obviously most of these nodes haven't actually talked to > all 1000 on the list so where does this list come from? Is it mixture > of addresses obtained from other nodes somehow sorted by timestamp? > Does it include some nodes discovered by IRC/DNS? Or are those only used > to find the first nodes to connect to? > > Thanks for any input... Hopefully I can build something that's useful > for the network... > > > ------------------------------------------------------------------------------ > Special Offer -- Download ArcSight Logger for FREE! > Finally, a world-class log management solution at an even better > price-free! And you'll get a free "Love Thy Logs" t-shirt when you > download Logger. Secure your free ArcSight Logger TODAY! > http://p.sf.net/sfu/arcsisghtdev2dev > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > --00151743f90ec1e9d004ac46c16b Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I've got minna patches for nio based on bitcoinj. I've enumerated t= he network a few times and am working on a DNS seed service as well as some= weather reports.

Happy to start a branch when the commi= tters are ready.

-rick


On T= ue, Sep 6, 2011 at 12:42 AM, Steve <shadders.del@gmail.com> wrote:
Hi All,

I started messing around today with building a node crawler to try and
map out the bitcoin network and hopefully provide some useful
statistics. =A0It's very basic so far using a mutilated bitcoinj to
connect (due me being java developer and not having a clue with c/c++).
=A0If it's worthwhile I'll hack bitcoinj some more to run on top N= etty to
take advantage of it's NIO architecture (netty's been shown to hand= le
1/2 million concurrent connections so would be ideal for the purpose).

Hoping to a get a bit of input into what would be useful as well as
strategy for getting max possible connections without distorted data. =A0I<= br> seem to recall Gavin talking about the need for some kind of network
health monitoring so I assume there's a need for something like this...=

Firstly at the moment basically I'm just storing version message and th= e
results of getaddr for each node that I can connect to. =A0Is there any
other useful info that can be extracted from a node that's worth collec= ting?

Second and main issue is how to connect. =A0From my first very basic
probing it seems the very vast majority of nodes don't accept incoming<= br> connections no doubt due to lack of upnp. =A0So it seems the active crawl approach is not really ideal for the purpose. =A0Even if it was used the resultant data would be hopelessly distorted.

A honeypot approach would probably be better if there was some way to
make a node 'attractive' to other nodes to connect to. =A0That way = it
could capture non-listening nodes as well. =A0If there is some way to
influence other nodes to connect to the crawler node that solves the
problem. =A0If there isn't which I suspect is the case then perhaps
another approach is to build an easy to deploy crawler node that many
volunteers could run and that could then upload collected data to a
central repository.

While I'm asking questions I'll add one more regarding the getaddr<= br> message. =A0It seems most nodes return about 1000 addresses in response to<= br> this message. =A0Obviously most of these nodes haven't actually talked = to
all 1000 on the list so where does this list come from? =A0Is it mixture of addresses obtained from other nodes somehow sorted by timestamp?
Does it include some nodes discovered by IRC/DNS? Or are those only used to find the first nodes to connect to?

Thanks for any input... Hopefully I can build something that's useful for the network...

---------------------------------------------------------------------------= ---
Special Offer -- Download ArcSight Logger for FREE!
Finally, a world-class log management solution at an even better
price-free! And you'll get a free "Love Thy Logs" t-shirt whe= n you
download Logger. Secure your free ArcSight Logger TODAY!
http://p= .sf.net/sfu/arcsisghtdev2dev
_______________________________________________
Bitcoin-development mailing list
Bitcoin-develo= pment@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-de= velopment

--00151743f90ec1e9d004ac46c16b--