diff options
author | Mike Hearn <mike@plan99.net> | 2011-09-06 16:52:20 +0200 |
---|---|---|
committer | bitcoindev <bitcoindev@gnusha.org> | 2011-09-06 14:52:27 +0000 |
commit | 327f42945c500c26120d29de2af810dd7025ce3d (patch) | |
tree | 4cac52bf39b63618eb4331795d6b79714ad2a088 | |
parent | 738757c8ca53fa1680b66f99e38506676abef463 (diff) | |
download | pi-bitcoindev-327f42945c500c26120d29de2af810dd7025ce3d.tar.gz pi-bitcoindev-327f42945c500c26120d29de2af810dd7025ce3d.zip |
Re: [Bitcoin-development] Building a node crawler to map network
-rw-r--r-- | 70/e7a92bd8beed6d589f906c9837698aff343747 | 173 |
1 files changed, 173 insertions, 0 deletions
diff --git a/70/e7a92bd8beed6d589f906c9837698aff343747 b/70/e7a92bd8beed6d589f906c9837698aff343747 new file mode 100644 index 000000000..6a3f06508 --- /dev/null +++ b/70/e7a92bd8beed6d589f906c9837698aff343747 @@ -0,0 +1,173 @@ +Received: from sog-mx-3.v43.ch3.sourceforge.com ([172.29.43.193] + helo=mx.sourceforge.net) + by sfs-ml-1.v29.ch3.sourceforge.com with esmtp (Exim 4.76) + (envelope-from <mh.in.england@gmail.com>) id 1R0x0l-0004pQ-He + for bitcoin-development@lists.sourceforge.net; + Tue, 06 Sep 2011 14:52:27 +0000 +Received-SPF: pass (sog-mx-3.v43.ch3.sourceforge.com: domain of gmail.com + designates 209.85.220.175 as permitted sender) + client-ip=209.85.220.175; envelope-from=mh.in.england@gmail.com; + helo=mail-vx0-f175.google.com; +Received: from mail-vx0-f175.google.com ([209.85.220.175]) + by sog-mx-3.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) + (Exim 4.76) id 1R0x0k-0002eE-BU + for bitcoin-development@lists.sourceforge.net; + Tue, 06 Sep 2011 14:52:27 +0000 +Received: by vxj14 with SMTP id 14so6390856vxj.34 + for <bitcoin-development@lists.sourceforge.net>; + Tue, 06 Sep 2011 07:52:21 -0700 (PDT) +MIME-Version: 1.0 +Received: by 10.52.38.99 with SMTP id f3mr5094454vdk.392.1315320740917; Tue, + 06 Sep 2011 07:52:20 -0700 (PDT) +Sender: mh.in.england@gmail.com +Received: by 10.52.157.228 with HTTP; Tue, 6 Sep 2011 07:52:20 -0700 (PDT) +In-Reply-To: <4E662B79.5090303@gmail.com> +References: <4E65CEE6.7030002@gmail.com> <4E65DA06.9060403@gmail.com> + <CALxbBHUajARXc1oA-NjD+U8hW5uSqF=u4ZHHBfcmT_O8GjpNiA@mail.gmail.com> + <CANEZrP0VXDUs_mAKCVKD1Q0ijyb989oADrCN1zTZ1nnN_JQ=cQ@mail.gmail.com> + <4E661FAE.9020008@gmail.com> + <CANEZrP3=UPYkBQo6b421xaMGyP4BsGiw8DBuM8pT2ow1Vom9JQ@mail.gmail.com> + <4E662B79.5090303@gmail.com> +Date: Tue, 6 Sep 2011 16:52:20 +0200 +X-Google-Sender-Auth: c6m0gMk9rEmJGa5GUVVFFVJN6zg +Message-ID: <CANEZrP2Wh82sqGjZDn_M=UPufBCU4fP9zEXV_K8JpgVF8O1FCw@mail.gmail.com> +From: Mike Hearn <mike@plan99.net> +To: shadders.del@gmail.com +Content-Type: multipart/alternative; boundary=bcaec51d2780872fd304ac46f916 +X-Spam-Score: -0.7 (/) +X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. + See http://spamassassin.org/tag/ for more details. + -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for + sender-domain + 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider + (mh.in.england[at]gmail.com) + -0.0 SPF_PASS SPF: sender matches SPF record + 1.0 HTML_MESSAGE BODY: HTML included in message + 0.1 DKIM_SIGNED Message has a DKIM or DK signature, + not necessarily valid + -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature + 0.0 T_TO_NO_BRKTS_FREEMAIL To: misformatted and free email service + -0.3 AWL AWL: From: address is in the auto white-list +X-Headers-End: 1R0x0k-0002eE-BU +Cc: bitcoin-development@lists.sourceforge.net +Subject: Re: [Bitcoin-development] Building a node crawler to map network +X-BeenThere: bitcoin-development@lists.sourceforge.net +X-Mailman-Version: 2.1.9 +Precedence: list +List-Id: <bitcoin-development.lists.sourceforge.net> +List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>, + <mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe> +List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development> +List-Post: <mailto:bitcoin-development@lists.sourceforge.net> +List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help> +List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>, + <mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe> +X-List-Received-Date: Tue, 06 Sep 2011 14:52:27 -0000 + +--bcaec51d2780872fd304ac46f916 +Content-Type: text/plain; charset=UTF-8 + +On Tue, Sep 6, 2011 at 4:17 PM, Steve <shadders.del@gmail.com> wrote: + +> ** +> I'm not really understanding the use case though. I believe most +> bitcoind's have a default max connections of 8. Is the goal to increase +> this without fundamentally altering the bitcoind concurrency model? +> + +bitcoind already uses asynchronous IO. That's not the problem. + +The issue came up in a conversation about scalability. If Bitcoins +popularity continues to grow, users are very likely to migrate away from +running full verifying nodes to lightweight clients, either a different mode +of the Satoshi client or different implementations like the Android Wallet +or MultiBit. + +Lightweight clients cannot verify thus should not relay. And they'll be run +by users who just want to send/receive coins from time to time, so don't +leave the programs running 24/7. The result could be running out of sockets +(like we have had problems with recently). It's especially true because +lightweight clients cannot check transactions for themselves. If they want +to show transactions appearing immediately (and they do), they have to use +"heard from lots of nodes" as a proxy for validity. So lightweight clients +are likely to be socket intensive. + +We could solve this by just hoping that lots of people run full nodes. The +problem is that a full node is quite an intensive thing already, it uses +lots of CPU and disk seeks, and will just get more expensive in future. And +as transaction traffic increases, that leaves less CPU time available to +service thousands of connected clients. The ROI of bringing up a new node +decreases at the same time as the userbase increases. + +One traditional approach to solving this is frontend proxies. Jabber.com/org +used this technique many years ago, and Google has also used it to scale up the +lockservice<http://static.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en/us/papers/chubby-osdi06.pdf> +(see +section 3.1). It's effective because often maintaining connections to +thousands of clients doesn't involve much brainwork, just shifting bytes +around. This is especially true of Bitcoin. So if somebody is running a full +node already they could increase their client capacity by just bringing up a +frontend proxy and having it handle things like outbound tx +broadcasts/deduping inbound broadcasts, connection setup, relaying recently +found blocks etc. A well written proxy could probably support tens of +thousands of simultaneous clients which frees up the bitcoinds time for +verification and wallet manipulation. + +--bcaec51d2780872fd304ac46f916 +Content-Type: text/html; charset=UTF-8 +Content-Transfer-Encoding: quoted-printable + +<div class=3D"gmail_quote">On Tue, Sep 6, 2011 at 4:17 PM, Steve <span dir= +=3D"ltr"><<a href=3D"mailto:shadders.del@gmail.com">shadders.del@gmail.c= +om</a>></span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"marg= +in:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"> +<u></u> + + =20 + =20 + =20 + <div bgcolor=3D"#ffffff" text=3D"#000000">I'm not really understandin= +g the use case though.=C2=A0 I believe most + bitcoind's have a default max connections of 8.=C2=A0 Is the goal t= +o + increase this without fundamentally altering the bitcoind + concurrency model?</div></blockquote><div><br></div><div>bitcoind alrea= +dy uses asynchronous IO. That's not the problem.</div><div><br></div><d= +iv>The issue came up in a conversation about scalability. If Bitcoins popul= +arity continues to grow, users are very likely to migrate away from running= + full verifying nodes to lightweight clients, either a different mode of th= +e Satoshi client or different implementations like the Android Wallet or Mu= +ltiBit.</div> +<div><br></div><div>Lightweight clients cannot verify thus should not relay= +. And they'll be run by users who just want to send/receive coins from = +time to time, so don't leave the programs running 24/7. The result coul= +d be running out of sockets (like we have had problems with recently). It&#= +39;s especially true because lightweight clients cannot check transactions = +for themselves. If they want to show transactions appearing immediately (an= +d they do), they have to use "heard from lots of nodes" as a prox= +y for validity. So lightweight clients are likely to be socket intensive.</= +div> +<div><br></div><div>We could solve this by just hoping that lots of people = +run full nodes. The problem is that a full node is quite an intensive thing= + already, it uses lots of CPU and disk seeks, and will just get more expens= +ive in future. And as transaction traffic increases, that leaves less CPU t= +ime available to service thousands of connected clients. The ROI of bringin= +g up a new node decreases at the same time as the userbase increases.</div> +<div><br></div><div>One traditional approach to solving this is frontend pr= +oxies. Jabber.com/org used this technique many years ago, and Google has al= +so used it to scale up <a href=3D"http://static.googleusercontent.com/exter= +nal_content/untrusted_dlcp/labs.google.com/en/us/papers/chubby-osdi06.pdf">= +the lockservice</a>=C2=A0(see section 3.1). It's effective because ofte= +n maintaining connections to thousands of clients doesn't involve much = +brainwork, just shifting bytes around. This is especially true of Bitcoin. = +So if somebody is running a full node already they could increase their cli= +ent capacity by just bringing up a frontend proxy and having it handle thin= +gs like outbound tx broadcasts/deduping inbound broadcasts, connection setu= +p, relaying recently found blocks etc. A well written proxy could probably = +support tens of thousands of simultaneous clients which frees up the bitcoi= +nds time for verification and wallet manipulation.</div> +</div> + +--bcaec51d2780872fd304ac46f916-- + + |