summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorMike Hearn <mike@plan99.net>2011-09-06 16:52:20 +0200
committerbitcoindev <bitcoindev@gnusha.org>2011-09-06 14:52:27 +0000
commit327f42945c500c26120d29de2af810dd7025ce3d (patch)
tree4cac52bf39b63618eb4331795d6b79714ad2a088
parent738757c8ca53fa1680b66f99e38506676abef463 (diff)
downloadpi-bitcoindev-327f42945c500c26120d29de2af810dd7025ce3d.tar.gz
pi-bitcoindev-327f42945c500c26120d29de2af810dd7025ce3d.zip
Re: [Bitcoin-development] Building a node crawler to map network
-rw-r--r--70/e7a92bd8beed6d589f906c9837698aff343747173
1 files changed, 173 insertions, 0 deletions
diff --git a/70/e7a92bd8beed6d589f906c9837698aff343747 b/70/e7a92bd8beed6d589f906c9837698aff343747
new file mode 100644
index 000000000..6a3f06508
--- /dev/null
+++ b/70/e7a92bd8beed6d589f906c9837698aff343747
@@ -0,0 +1,173 @@
+Received: from sog-mx-3.v43.ch3.sourceforge.com ([172.29.43.193]
+ helo=mx.sourceforge.net)
+ by sfs-ml-1.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
+ (envelope-from <mh.in.england@gmail.com>) id 1R0x0l-0004pQ-He
+ for bitcoin-development@lists.sourceforge.net;
+ Tue, 06 Sep 2011 14:52:27 +0000
+Received-SPF: pass (sog-mx-3.v43.ch3.sourceforge.com: domain of gmail.com
+ designates 209.85.220.175 as permitted sender)
+ client-ip=209.85.220.175; envelope-from=mh.in.england@gmail.com;
+ helo=mail-vx0-f175.google.com;
+Received: from mail-vx0-f175.google.com ([209.85.220.175])
+ by sog-mx-3.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
+ (Exim 4.76) id 1R0x0k-0002eE-BU
+ for bitcoin-development@lists.sourceforge.net;
+ Tue, 06 Sep 2011 14:52:27 +0000
+Received: by vxj14 with SMTP id 14so6390856vxj.34
+ for <bitcoin-development@lists.sourceforge.net>;
+ Tue, 06 Sep 2011 07:52:21 -0700 (PDT)
+MIME-Version: 1.0
+Received: by 10.52.38.99 with SMTP id f3mr5094454vdk.392.1315320740917; Tue,
+ 06 Sep 2011 07:52:20 -0700 (PDT)
+Sender: mh.in.england@gmail.com
+Received: by 10.52.157.228 with HTTP; Tue, 6 Sep 2011 07:52:20 -0700 (PDT)
+In-Reply-To: <4E662B79.5090303@gmail.com>
+References: <4E65CEE6.7030002@gmail.com> <4E65DA06.9060403@gmail.com>
+ <CALxbBHUajARXc1oA-NjD+U8hW5uSqF=u4ZHHBfcmT_O8GjpNiA@mail.gmail.com>
+ <CANEZrP0VXDUs_mAKCVKD1Q0ijyb989oADrCN1zTZ1nnN_JQ=cQ@mail.gmail.com>
+ <4E661FAE.9020008@gmail.com>
+ <CANEZrP3=UPYkBQo6b421xaMGyP4BsGiw8DBuM8pT2ow1Vom9JQ@mail.gmail.com>
+ <4E662B79.5090303@gmail.com>
+Date: Tue, 6 Sep 2011 16:52:20 +0200
+X-Google-Sender-Auth: c6m0gMk9rEmJGa5GUVVFFVJN6zg
+Message-ID: <CANEZrP2Wh82sqGjZDn_M=UPufBCU4fP9zEXV_K8JpgVF8O1FCw@mail.gmail.com>
+From: Mike Hearn <mike@plan99.net>
+To: shadders.del@gmail.com
+Content-Type: multipart/alternative; boundary=bcaec51d2780872fd304ac46f916
+X-Spam-Score: -0.7 (/)
+X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
+ See http://spamassassin.org/tag/ for more details.
+ -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
+ sender-domain
+ 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
+ (mh.in.england[at]gmail.com)
+ -0.0 SPF_PASS SPF: sender matches SPF record
+ 1.0 HTML_MESSAGE BODY: HTML included in message
+ 0.1 DKIM_SIGNED Message has a DKIM or DK signature,
+ not necessarily valid
+ -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
+ 0.0 T_TO_NO_BRKTS_FREEMAIL To: misformatted and free email service
+ -0.3 AWL AWL: From: address is in the auto white-list
+X-Headers-End: 1R0x0k-0002eE-BU
+Cc: bitcoin-development@lists.sourceforge.net
+Subject: Re: [Bitcoin-development] Building a node crawler to map network
+X-BeenThere: bitcoin-development@lists.sourceforge.net
+X-Mailman-Version: 2.1.9
+Precedence: list
+List-Id: <bitcoin-development.lists.sourceforge.net>
+List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
+ <mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
+List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
+List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
+List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
+List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
+ <mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
+X-List-Received-Date: Tue, 06 Sep 2011 14:52:27 -0000
+
+--bcaec51d2780872fd304ac46f916
+Content-Type: text/plain; charset=UTF-8
+
+On Tue, Sep 6, 2011 at 4:17 PM, Steve <shadders.del@gmail.com> wrote:
+
+> **
+> I'm not really understanding the use case though. I believe most
+> bitcoind's have a default max connections of 8. Is the goal to increase
+> this without fundamentally altering the bitcoind concurrency model?
+>
+
+bitcoind already uses asynchronous IO. That's not the problem.
+
+The issue came up in a conversation about scalability. If Bitcoins
+popularity continues to grow, users are very likely to migrate away from
+running full verifying nodes to lightweight clients, either a different mode
+of the Satoshi client or different implementations like the Android Wallet
+or MultiBit.
+
+Lightweight clients cannot verify thus should not relay. And they'll be run
+by users who just want to send/receive coins from time to time, so don't
+leave the programs running 24/7. The result could be running out of sockets
+(like we have had problems with recently). It's especially true because
+lightweight clients cannot check transactions for themselves. If they want
+to show transactions appearing immediately (and they do), they have to use
+"heard from lots of nodes" as a proxy for validity. So lightweight clients
+are likely to be socket intensive.
+
+We could solve this by just hoping that lots of people run full nodes. The
+problem is that a full node is quite an intensive thing already, it uses
+lots of CPU and disk seeks, and will just get more expensive in future. And
+as transaction traffic increases, that leaves less CPU time available to
+service thousands of connected clients. The ROI of bringing up a new node
+decreases at the same time as the userbase increases.
+
+One traditional approach to solving this is frontend proxies. Jabber.com/org
+used this technique many years ago, and Google has also used it to scale up the
+lockservice<http://static.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en/us/papers/chubby-osdi06.pdf>
+(see
+section 3.1). It's effective because often maintaining connections to
+thousands of clients doesn't involve much brainwork, just shifting bytes
+around. This is especially true of Bitcoin. So if somebody is running a full
+node already they could increase their client capacity by just bringing up a
+frontend proxy and having it handle things like outbound tx
+broadcasts/deduping inbound broadcasts, connection setup, relaying recently
+found blocks etc. A well written proxy could probably support tens of
+thousands of simultaneous clients which frees up the bitcoinds time for
+verification and wallet manipulation.
+
+--bcaec51d2780872fd304ac46f916
+Content-Type: text/html; charset=UTF-8
+Content-Transfer-Encoding: quoted-printable
+
+<div class=3D"gmail_quote">On Tue, Sep 6, 2011 at 4:17 PM, Steve <span dir=
+=3D"ltr">&lt;<a href=3D"mailto:shadders.del@gmail.com">shadders.del@gmail.c=
+om</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"marg=
+in:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
+<u></u>
+
+ =20
+ =20
+ =20
+ <div bgcolor=3D"#ffffff" text=3D"#000000">I&#39;m not really understandin=
+g the use case though.=C2=A0 I believe most
+ bitcoind&#39;s have a default max connections of 8.=C2=A0 Is the goal t=
+o
+ increase this without fundamentally altering the bitcoind
+ concurrency model?</div></blockquote><div><br></div><div>bitcoind alrea=
+dy uses asynchronous IO. That&#39;s not the problem.</div><div><br></div><d=
+iv>The issue came up in a conversation about scalability. If Bitcoins popul=
+arity continues to grow, users are very likely to migrate away from running=
+ full verifying nodes to lightweight clients, either a different mode of th=
+e Satoshi client or different implementations like the Android Wallet or Mu=
+ltiBit.</div>
+<div><br></div><div>Lightweight clients cannot verify thus should not relay=
+. And they&#39;ll be run by users who just want to send/receive coins from =
+time to time, so don&#39;t leave the programs running 24/7. The result coul=
+d be running out of sockets (like we have had problems with recently). It&#=
+39;s especially true because lightweight clients cannot check transactions =
+for themselves. If they want to show transactions appearing immediately (an=
+d they do), they have to use &quot;heard from lots of nodes&quot; as a prox=
+y for validity. So lightweight clients are likely to be socket intensive.</=
+div>
+<div><br></div><div>We could solve this by just hoping that lots of people =
+run full nodes. The problem is that a full node is quite an intensive thing=
+ already, it uses lots of CPU and disk seeks, and will just get more expens=
+ive in future. And as transaction traffic increases, that leaves less CPU t=
+ime available to service thousands of connected clients. The ROI of bringin=
+g up a new node decreases at the same time as the userbase increases.</div>
+<div><br></div><div>One traditional approach to solving this is frontend pr=
+oxies. Jabber.com/org used this technique many years ago, and Google has al=
+so used it to scale up <a href=3D"http://static.googleusercontent.com/exter=
+nal_content/untrusted_dlcp/labs.google.com/en/us/papers/chubby-osdi06.pdf">=
+the lockservice</a>=C2=A0(see section 3.1). It&#39;s effective because ofte=
+n maintaining connections to thousands of clients doesn&#39;t involve much =
+brainwork, just shifting bytes around. This is especially true of Bitcoin. =
+So if somebody is running a full node already they could increase their cli=
+ent capacity by just bringing up a frontend proxy and having it handle thin=
+gs like outbound tx broadcasts/deduping inbound broadcasts, connection setu=
+p, relaying recently found blocks etc. A well written proxy could probably =
+support tens of thousands of simultaneous clients which frees up the bitcoi=
+nds time for verification and wallet manipulation.</div>
+</div>
+
+--bcaec51d2780872fd304ac46f916--
+
+