summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorSteve <shadders.del@gmail.com>2011-09-06 17:42:30 +1000
committerbitcoindev <bitcoindev@gnusha.org>2011-09-06 07:42:44 +0000
commitae390ee987bff58942028489774fa537e9ae1177 (patch)
tree162f61eea1719aa69c358c25779f891805259502
parent9bfe1df86fd82062c1f8f02ef22b27ef588e2b47 (diff)
downloadpi-bitcoindev-ae390ee987bff58942028489774fa537e9ae1177.tar.gz
pi-bitcoindev-ae390ee987bff58942028489774fa537e9ae1177.zip
[Bitcoin-development] Building a node crawler to map network
-rw-r--r--f5/7e4107a64217ce8b3e51a66d26b235267a646f109
1 files changed, 109 insertions, 0 deletions
diff --git a/f5/7e4107a64217ce8b3e51a66d26b235267a646f b/f5/7e4107a64217ce8b3e51a66d26b235267a646f
new file mode 100644
index 000000000..1d17aeffc
--- /dev/null
+++ b/f5/7e4107a64217ce8b3e51a66d26b235267a646f
@@ -0,0 +1,109 @@
+Received: from sog-mx-4.v43.ch3.sourceforge.com ([172.29.43.194]
+ helo=mx.sourceforge.net)
+ by sfs-ml-3.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
+ (envelope-from <shadders.del@gmail.com>) id 1R0qIu-00078Y-MD
+ for bitcoin-development@lists.sourceforge.net;
+ Tue, 06 Sep 2011 07:42:44 +0000
+Received-SPF: pass (sog-mx-4.v43.ch3.sourceforge.com: domain of gmail.com
+ designates 209.85.213.47 as permitted sender)
+ client-ip=209.85.213.47; envelope-from=shadders.del@gmail.com;
+ helo=mail-yw0-f47.google.com;
+Received: from mail-yw0-f47.google.com ([209.85.213.47])
+ by sog-mx-4.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-MD5:128)
+ (Exim 4.76) id 1R0qIt-0004i9-Vs
+ for bitcoin-development@lists.sourceforge.net;
+ Tue, 06 Sep 2011 07:42:44 +0000
+Received: by ywa12 with SMTP id 12so4343681ywa.34
+ for <bitcoin-development@lists.sourceforge.net>;
+ Tue, 06 Sep 2011 00:42:38 -0700 (PDT)
+Received: by 10.236.76.41 with SMTP id a29mr22346529yhe.40.1315294958596;
+ Tue, 06 Sep 2011 00:42:38 -0700 (PDT)
+Received: from [10.1.1.50] (155.88-67-202.dynamic.dsl.syd.iprimus.net.au
+ [202.67.88.155])
+ by mx.google.com with ESMTPS id s42sm8035331yhs.0.2011.09.06.00.42.35
+ (version=SSLv3 cipher=OTHER); Tue, 06 Sep 2011 00:42:37 -0700 (PDT)
+Message-ID: <4E65CEE6.7030002@gmail.com>
+Date: Tue, 06 Sep 2011 17:42:30 +1000
+From: Steve <shadders.del@gmail.com>
+User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
+ rv:1.9.2.21) Gecko/20110831 Thunderbird/3.1.13
+MIME-Version: 1.0
+To: bitcoin-development@lists.sourceforge.net
+Content-Type: text/plain; charset=ISO-8859-1; format=flowed
+Content-Transfer-Encoding: 7bit
+X-Spam-Score: -1.6 (-)
+X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
+ See http://spamassassin.org/tag/ for more details.
+ -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
+ sender-domain
+ 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
+ (shadders.del[at]gmail.com)
+ -0.0 SPF_PASS SPF: sender matches SPF record
+ -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
+ author's domain
+ 0.1 DKIM_SIGNED Message has a DKIM or DK signature,
+ not necessarily valid
+ -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
+ 0.0 T_TO_NO_BRKTS_FREEMAIL To: misformatted and free email service
+X-Headers-End: 1R0qIt-0004i9-Vs
+Subject: [Bitcoin-development] Building a node crawler to map network
+X-BeenThere: bitcoin-development@lists.sourceforge.net
+X-Mailman-Version: 2.1.9
+Precedence: list
+Reply-To: shadders.del@gmail.com
+List-Id: <bitcoin-development.lists.sourceforge.net>
+List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
+ <mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
+List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
+List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
+List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
+List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
+ <mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
+X-List-Received-Date: Tue, 06 Sep 2011 07:42:44 -0000
+
+Hi All,
+
+I started messing around today with building a node crawler to try and
+map out the bitcoin network and hopefully provide some useful
+statistics. It's very basic so far using a mutilated bitcoinj to
+connect (due me being java developer and not having a clue with c/c++).
+ If it's worthwhile I'll hack bitcoinj some more to run on top Netty to
+take advantage of it's NIO architecture (netty's been shown to handle
+1/2 million concurrent connections so would be ideal for the purpose).
+
+Hoping to a get a bit of input into what would be useful as well as
+strategy for getting max possible connections without distorted data. I
+seem to recall Gavin talking about the need for some kind of network
+health monitoring so I assume there's a need for something like this...
+
+Firstly at the moment basically I'm just storing version message and the
+results of getaddr for each node that I can connect to. Is there any
+other useful info that can be extracted from a node that's worth collecting?
+
+Second and main issue is how to connect. From my first very basic
+probing it seems the very vast majority of nodes don't accept incoming
+connections no doubt due to lack of upnp. So it seems the active crawl
+approach is not really ideal for the purpose. Even if it was used the
+resultant data would be hopelessly distorted.
+
+A honeypot approach would probably be better if there was some way to
+make a node 'attractive' to other nodes to connect to. That way it
+could capture non-listening nodes as well. If there is some way to
+influence other nodes to connect to the crawler node that solves the
+problem. If there isn't which I suspect is the case then perhaps
+another approach is to build an easy to deploy crawler node that many
+volunteers could run and that could then upload collected data to a
+central repository.
+
+While I'm asking questions I'll add one more regarding the getaddr
+message. It seems most nodes return about 1000 addresses in response to
+this message. Obviously most of these nodes haven't actually talked to
+all 1000 on the list so where does this list come from? Is it mixture
+of addresses obtained from other nodes somehow sorted by timestamp?
+Does it include some nodes discovered by IRC/DNS? Or are those only used
+to find the first nodes to connect to?
+
+Thanks for any input... Hopefully I can build something that's useful
+for the network...
+
+