diff options
| author | Olaoluwa Osuntokun <laolu32@gmail.com> | 2015-08-06 17:33:49 +0000 |
|---|---|---|
| committer | bitcoindev <bitcoindev@gnusha.org> | 2015-08-06 17:34:01 +0000 |
| commit | f869c168011401f6160ff40ccea3739f7cf9c78c (patch) | |
| tree | 05de17c029c9be29e071c94fd4854f3e3f580255 | |
| parent | ad679701da1e065bf7face38734cb0b01d38c206 (diff) | |
| download | pi-bitcoindev-f869c168011401f6160ff40ccea3739f7cf9c78c.tar.gz pi-bitcoindev-f869c168011401f6160ff40ccea3739f7cf9c78c.zip | |
Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation
| -rw-r--r-- | 2d/6ae2c9b35f4a9878e3b43b0a8e3ff856cffb61 | 324 |
1 files changed, 324 insertions, 0 deletions
diff --git a/2d/6ae2c9b35f4a9878e3b43b0a8e3ff856cffb61 b/2d/6ae2c9b35f4a9878e3b43b0a8e3ff856cffb61 new file mode 100644 index 000000000..c7059597e --- /dev/null +++ b/2d/6ae2c9b35f4a9878e3b43b0a8e3ff856cffb61 @@ -0,0 +1,324 @@ +Return-Path: <laolu32@gmail.com> +Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org + [172.17.192.35]) + by mail.linuxfoundation.org (Postfix) with ESMTPS id BABFA892 + for <bitcoin-dev@lists.linuxfoundation.org>; + Thu, 6 Aug 2015 17:34:01 +0000 (UTC) +X-Greylist: whitelisted by SQLgrey-1.7.6 +Received: from mail-qg0-f49.google.com (mail-qg0-f49.google.com + [209.85.192.49]) + by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 8B5D716A + for <bitcoin-dev@lists.linuxfoundation.org>; + Thu, 6 Aug 2015 17:34:00 +0000 (UTC) +Received: by qgj62 with SMTP id 62so32706951qgj.2 + for <bitcoin-dev@lists.linuxfoundation.org>; + Thu, 06 Aug 2015 10:33:59 -0700 (PDT) +DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; + h=mime-version:references:in-reply-to:from:date:message-id:subject:to + :content-type; bh=2rMdE9kR/9IV3I0hwoPYDNF012YiIi826qyAyagvQ/Y=; + b=OXiql6hiOFZNuCVp8e7jHiAZ6bRsb9IyH9ZhXALoq1ZsXgOVJCKnxyo+sDTF2n9JhP + QfYF3X52rv6p73rgfgnhJCUeuJw+xC2Z3d/Hy2RE/BBYQ9gK5qvG4qtKPpeonErYsigW + BO8k6lyPtscjUMRUOS3yDqIkgyz94b88FxGmXSw1bDVNTEg4gyZatMf8zL6taRLAEeBU + YJA1ud8EyxLsIb1gaw3Sy5Dv4a1sGyPn7/rq9y4DHRzo37mMN34+AsPl5tlbHRGhFvx+ + fQEFMjW1GvTeUrZFdEnhQhlJwdiljLliEfPiZtQ9u1dZMvGJIa3Gm9Dp89oSzwnLZAtU + Kecw== +X-Received: by 10.140.237.67 with SMTP id i64mr5430332qhc.5.1438882439524; + Thu, 06 Aug 2015 10:33:59 -0700 (PDT) +MIME-Version: 1.0 +References: <CALwsPgnnkbfUBhL=Qyspz13pnZ-6RHdaZOGvfLG34JjJRgt2Dw@mail.gmail.com> + <B3546CB9-6A24-474C-8B56-9B1E2D33B470@mattcorallo.com> + <CALwsPgm6xcBfLXZTNTZ40R_s3oUawE0ANZycDWpSo0cXZ+=-Vg@mail.gmail.com> + <CAAS2fgQLNH+rivNNTFz6xt_9SxO3fFj7-3z7A-_B_2X2x-6M5w@mail.gmail.com> + <CALwsPgm9S3UNd3bEuWreyGS7bcvSD+cXxueoD+F_D9fC=xLz2Q@mail.gmail.com> + <CAAS2fgSXWzCv=4cF=0bwL9+udzBHSPR7goL3U_c1NjS22dpWzQ@mail.gmail.com> + <CAKzdR-qPtPsxAXsmUX=vTkq-ro=EAmH7M8nL_Px_b4D4Z0WAXg@mail.gmail.com> +In-Reply-To: <CAKzdR-qPtPsxAXsmUX=vTkq-ro=EAmH7M8nL_Px_b4D4Z0WAXg@mail.gmail.com> +From: Olaoluwa Osuntokun <laolu32@gmail.com> +Date: Thu, 06 Aug 2015 17:33:49 +0000 +Message-ID: <CAO3Pvs-knJ5DyJou2YuB1nAQH+UXV4mcwn42KKh0Wb-YpeOm_w@mail.gmail.com> +To: Sergio Demian Lerner <sergio.d.lerner@gmail.com>, + Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev + <bitcoin-dev@lists.linuxfoundation.org> +Content-Type: multipart/alternative; boundary=001a1135c5feae7896051ca7ecfb +X-Spam-Status: No, score=-2.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, + DKIM_VALID,DKIM_VALID_AU,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM, + HTML_MESSAGE,RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 +X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on + smtp1.linux-foundation.org +Subject: Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation +X-BeenThere: bitcoin-dev@lists.linuxfoundation.org +X-Mailman-Version: 2.1.12 +Precedence: list +List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org> +List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>, + <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe> +List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/> +List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org> +List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help> +List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>, + <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe> +X-List-Received-Date: Thu, 06 Aug 2015 17:34:01 -0000 + +--001a1135c5feae7896051ca7ecfb +Content-Type: text/plain; charset=UTF-8 + +Other than the source code, the best documentation I've come across is a few +lines on IRC explaining the high-level design of the protocol: +https://botbot.me/freenode/bitcoin-wizards/2015-07-10/?msg=44146764&page=2 + +On Thu, Aug 6, 2015 at 10:18 AM Sergio Demian Lerner via bitcoin-dev < +bitcoin-dev@lists.linuxfoundation.org> wrote: + +> Is there any up to date documentation about TheBlueMatt relay network +> including what kind of block compression it is currently doing? (apart from +> the source code) +> +> Regards, Sergio. +> +> On Wed, Aug 5, 2015 at 7:14 PM, Gregory Maxwell via bitcoin-dev < +> bitcoin-dev@lists.linuxfoundation.org> wrote: +> +>> On Wed, Aug 5, 2015 at 9:19 PM, Arnoud Kouwenhoven - Pukaki Corp +>> <arnoud@pukaki.bz> wrote: +>> > Thanks for this (direct) feedback. It would make sense that if blocks +>> can be +>> > submitted using ~5kb packets, that no further optimizations would be +>> needed +>> > at this point. I will look into the relay network transmission protocol +>> to +>> > understand how it works! +>> > +>> > I hear that you are saying that this network solves speed of +>> transmission +>> > and thereby (technical) block size issues. Presumably it would solve +>> speed +>> > of block validation too by prevalidating transactions. +>> +>> +>> Correct. Bitcoin Core has cached validation for many years now... if +>> not for that and other optimizations, things would be really broken +>> right now. :) +>> +>> > Assuming this is all +>> > true, and I have no reason to doubt that at this point, I do not +>> understand +>> > why there is any discussion at all about the (technical) impact of large +>> > blocks, why there are large numbers of miners building on invalid blocks +>> > (SPV mining, https://bitcoin.org/en/alert/2015-07-04-spv-mining), or +>> why +>> > there is any discussion about the speed of block validation (cpu +>> processing +>> > time to verify blocks and transactions in blocks being a limitation). +>> +>> I'm also mystified by a lot of the large block discussion, much of it +>> is completely divorced from the technology as deployed; much less what +>> we-- in industry-- know to be possible. I don't blame you or anyone in +>> particular on this; it's a new area and we don't yet know what we need +>> to know to know what we need to know; or to the extent that we do it +>> hasn't had time to get effectively communicated. +>> +>> The technical/security implications of larger blocks are related to +>> other things than propagation time, if you assume people are using the +>> available efficient relay protocol (or better). +>> +>> SPV mining is a bit of a misnomer (If I coined the term, I'm sorry). +>> What these parties are actually doing is blinding mining on top of +>> other pools' stratum work. You can think of it as sub-pooling with +>> hopping onto whatever pool has the highest block (I'll call it VFSSP +>> in this post-- validation free stratum subpooling). It's very easy to +>> implement, and there are other considerations. +>> +>> It was initially deployed at a time when a single pool in Europe has +>> amassed more than half of the hashrate. This pool had propagation +>> problems and a very high orphan rate, it may have (perhaps +>> unintentionally) been performing a selfish mining attack; mining off +>> their stratum work was an easy fix which massively cut down the orphan +>> rates for anyone who did it. This was before the relay network +>> protocol existed (the fact that all the hashpower was consolidating on +>> a single pool was a major motivation for creating it). +>> +>> VFSSP also cuts through a number of practical issues miners have had: +>> Miners that run their own bitcoin nodes in far away colocation +>> (>100ms) due to local bandwidth or connectivity issues (censored +>> internet); relay network hubs not being anywhere near by due to +>> strange internet routing (e.g. japan to china going via the US for ... +>> reasons...); the CreateNewBlock() function being very slow and +>> unoptimized, etc. There are many other things like this-- and VFSSP +>> avoids them causing delays even when you don't understand them or know +>> about them. So even when they're easily fixed the VFSSP is a more +>> general workaround. +>> +>> Mining operations are also usually operated in a largely fire and +>> forget manner. There is a long history in (esp pooled) mining where +>> someone sets up an operation and then hardly maintains it after the +>> fact... so some of the use of VFSSP appears to just be inertia-- we +>> have better solutions now, but they they work to deploy and changing +>> things involves risk (which is heightened by a lack of good +>> monitoring-- participants learn they are too latent by observing +>> orphaned blocks at a cost of 25 BTC each). +>> +>> One of the frustrating things about incentives in this space is that +>> bad outcomes are possible even when they're not necessary. E.g. if a +>> miner can lower their orphan rate by deploying a new protocol (or +>> simply fixing some faulty hardware in their infrastructure, like +>> Bitcoin nodes running on cheap VPSes with remote storage) OR they can +>> lower their orphan rate by pointing their hashpower at a free +>> centeralized pool, they're likely to do the latter because it takes +>> less effort. +>> _______________________________________________ +>> bitcoin-dev mailing list +>> bitcoin-dev@lists.linuxfoundation.org +>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev +>> +> +> _______________________________________________ +> bitcoin-dev mailing list +> bitcoin-dev@lists.linuxfoundation.org +> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev +> + +--001a1135c5feae7896051ca7ecfb +Content-Type: text/html; charset=UTF-8 +Content-Transfer-Encoding: quoted-printable + +<div dir=3D"ltr">Other than the source code, the best documentation I'v= +e come across is a few<div>lines on=C2=A0<span style=3D"line-height:1.5;fon= +t-size:13.1999998092651px">IRC explaining the high-level design of the prot= +ocol:=C2=A0</span><div><a href=3D"https://botbot.me/freenode/bitcoin-wizard= +s/2015-07-10/?msg=3D44146764&page=3D2">https://botbot.me/freenode/bitco= +in-wizards/2015-07-10/?msg=3D44146764&page=3D2</a></div></div><br><div = +class=3D"gmail_quote"><div dir=3D"ltr">On Thu, Aug 6, 2015 at 10:18 AM Serg= +io Demian Lerner via bitcoin-dev <<a href=3D"mailto:bitcoin-dev@lists.li= +nuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a>> wrote:<br>= +</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-l= +eft:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">Is there any up to da= +te documentation about TheBlueMatt relay network including what kind of blo= +ck compression it is currently doing? (apart from the source code)<div><br>= +<div>Regards, Sergio.</div></div></div><div class=3D"gmail_extra"><br><div = +class=3D"gmail_quote">On Wed, Aug 5, 2015 at 7:14 PM, Gregory Maxwell via b= +itcoin-dev <span dir=3D"ltr"><<a href=3D"mailto:bitcoin-dev@lists.linuxf= +oundation.org" target=3D"_blank">bitcoin-dev@lists.linuxfoundation.org</a>&= +gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 = +0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On Wed, Aug 5, 20= +15 at 9:19 PM, Arnoud Kouwenhoven - Pukaki Corp<br> +<<a href=3D"mailto:arnoud@pukaki.bz" target=3D"_blank">arnoud@pukaki.bz<= +/a>> wrote:<br> +> Thanks for this (direct) feedback. It would make sense that if blocks = +can be<br> +> submitted using ~5kb packets, that no further optimizations would be n= +eeded<br> +> at this point. I will look into the relay network transmission protoco= +l to<br> +> understand how it works!<br> +><br> +> I hear that you are saying that this network solves speed of transmiss= +ion<br> +> and thereby (technical) block size issues. Presumably it would solve s= +peed<br> +> of block validation too by prevalidating transactions.<br> +<br> +<br> +</span>Correct. Bitcoin Core has cached validation for many years now... if= +<br> +not for that and other optimizations, things would be really broken<br> +right now. :)<br> +<span><br> +> Assuming this is all<br> +> true, and I have no reason to doubt that at this point, I do not under= +stand<br> +> why there is any discussion at all about the (technical) impact of lar= +ge<br> +</span>> blocks, why there are large numbers of miners building on inval= +id blocks<br> +<span>> (SPV mining, <a href=3D"https://bitcoin.org/en/alert/2015-07-04-= +spv-mining" rel=3D"noreferrer" target=3D"_blank">https://bitcoin.org/en/ale= +rt/2015-07-04-spv-mining</a>), or why<br> +> there is any discussion about the speed of block validation (cpu proce= +ssing<br> +> time to verify blocks and transactions in blocks being a limitation).<= +br> +<br> +</span>I'm also mystified by a lot of the large block discussion, much = +of it<br> +is completely divorced from the technology as deployed; much less what<br> +we-- in industry-- know to be possible. I don't blame you or anyone in<= +br> +particular on this; it's a new area and we don't yet know what we n= +eed<br> +to know to know what we need to know; or to the extent that we do it<br> +hasn't had time to get effectively communicated.<br> +<br> +The technical/security implications of larger blocks are related to<br> +other things than propagation time, if you assume people are using the<br> +available efficient relay protocol (or better).<br> +<br> +SPV mining is a bit of a misnomer (If I coined the term, I'm sorry).<br= +> +What these parties are actually doing is blinding mining on top of<br> +other pools' stratum work. You can think of it as sub-pooling with<br> +hopping onto whatever pool has the highest block (I'll call it VFSSP<br= +> +in this post-- validation free stratum subpooling).=C2=A0 It's very eas= +y to<br> +implement, and there are other considerations.<br> +<br> +It was initially deployed at a time when a single pool in Europe has<br> +amassed more than half of the hashrate. This pool had propagation<br> +problems and a very high orphan rate, it may have (perhaps<br> +unintentionally) been performing a selfish mining attack; mining off<br> +their stratum work was an easy fix which massively cut down the orphan<br> +rates for anyone who did it.=C2=A0 This was before the relay network<br> +protocol existed (the fact that all the hashpower was consolidating on<br> +a single pool was a major motivation for creating it).<br> +<br> +VFSSP also cuts through a number of practical issues miners have had:<br> +Miners that run their own bitcoin nodes in far away colocation<br> +(>100ms) due to local bandwidth or connectivity issues (censored<br> +internet); relay network hubs not being anywhere near by due to<br> +strange internet routing (e.g. japan to china going via the US for ...<br> +reasons...); the CreateNewBlock() function being very slow and<br> +unoptimized, etc.=C2=A0 =C2=A0There are many other things like this-- and V= +FSSP<br> +avoids them causing delays even when you don't understand them or know<= +br> +about them. So even when they're easily fixed the VFSSP is a more<br> +general workaround.<br> +<br> +Mining operations are also usually operated in a largely fire and<br> +forget manner. There is a long history in (esp pooled) mining where<br> +someone sets up an operation and then hardly maintains it after the<br> +fact... so some of the use of VFSSP appears to just be inertia-- we<br> +have better solutions now, but they they work to deploy and changing<br> +things involves risk (which is heightened by a lack of good<br> +monitoring-- participants learn they are too latent by observing<br> +orphaned blocks at a cost of 25 BTC each).<br> +<br> +One of the frustrating things about incentives in this space is that<br> +bad outcomes are possible even when they're not necessary. E.g. if a<br= +> +miner can lower their orphan rate by deploying a new protocol (or<br> +simply fixing some faulty hardware in their infrastructure, like<br> +Bitcoin nodes running on cheap VPSes with remote storage)=C2=A0 OR they can= +<br> +lower their orphan rate by pointing their hashpower at a free<br> +centeralized pool, they're likely to do the latter because it takes<br> +less effort.<br> +<div><div>_______________________________________________<br> +bitcoin-dev mailing list<br> +<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">= +bitcoin-dev@lists.linuxfoundation.org</a><br> +<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" = +rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail= +man/listinfo/bitcoin-dev</a><br> +</div></div></blockquote></div><br></div> +_______________________________________________<br> +bitcoin-dev mailing list<br> +<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">= +bitcoin-dev@lists.linuxfoundation.org</a><br> +<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" = +rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail= +man/listinfo/bitcoin-dev</a><br> +</blockquote></div></div> + +--001a1135c5feae7896051ca7ecfb-- + |