Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id BABFA892 for ; Thu, 6 Aug 2015 17:34:01 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-qg0-f49.google.com (mail-qg0-f49.google.com [209.85.192.49]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 8B5D716A for ; Thu, 6 Aug 2015 17:34:00 +0000 (UTC) Received: by qgj62 with SMTP id 62so32706951qgj.2 for ; Thu, 06 Aug 2015 10:33:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :content-type; bh=2rMdE9kR/9IV3I0hwoPYDNF012YiIi826qyAyagvQ/Y=; b=OXiql6hiOFZNuCVp8e7jHiAZ6bRsb9IyH9ZhXALoq1ZsXgOVJCKnxyo+sDTF2n9JhP QfYF3X52rv6p73rgfgnhJCUeuJw+xC2Z3d/Hy2RE/BBYQ9gK5qvG4qtKPpeonErYsigW BO8k6lyPtscjUMRUOS3yDqIkgyz94b88FxGmXSw1bDVNTEg4gyZatMf8zL6taRLAEeBU YJA1ud8EyxLsIb1gaw3Sy5Dv4a1sGyPn7/rq9y4DHRzo37mMN34+AsPl5tlbHRGhFvx+ fQEFMjW1GvTeUrZFdEnhQhlJwdiljLliEfPiZtQ9u1dZMvGJIa3Gm9Dp89oSzwnLZAtU Kecw== X-Received: by 10.140.237.67 with SMTP id i64mr5430332qhc.5.1438882439524; Thu, 06 Aug 2015 10:33:59 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Olaoluwa Osuntokun Date: Thu, 06 Aug 2015 17:33:49 +0000 Message-ID: To: Sergio Demian Lerner , Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev Content-Type: multipart/alternative; boundary=001a1135c5feae7896051ca7ecfb X-Spam-Status: No, score=-2.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM, HTML_MESSAGE,RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Development Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Aug 2015 17:34:01 -0000 --001a1135c5feae7896051ca7ecfb Content-Type: text/plain; charset=UTF-8 Other than the source code, the best documentation I've come across is a few lines on IRC explaining the high-level design of the protocol: https://botbot.me/freenode/bitcoin-wizards/2015-07-10/?msg=44146764&page=2 On Thu, Aug 6, 2015 at 10:18 AM Sergio Demian Lerner via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > Is there any up to date documentation about TheBlueMatt relay network > including what kind of block compression it is currently doing? (apart from > the source code) > > Regards, Sergio. > > On Wed, Aug 5, 2015 at 7:14 PM, Gregory Maxwell via bitcoin-dev < > bitcoin-dev@lists.linuxfoundation.org> wrote: > >> On Wed, Aug 5, 2015 at 9:19 PM, Arnoud Kouwenhoven - Pukaki Corp >> wrote: >> > Thanks for this (direct) feedback. It would make sense that if blocks >> can be >> > submitted using ~5kb packets, that no further optimizations would be >> needed >> > at this point. I will look into the relay network transmission protocol >> to >> > understand how it works! >> > >> > I hear that you are saying that this network solves speed of >> transmission >> > and thereby (technical) block size issues. Presumably it would solve >> speed >> > of block validation too by prevalidating transactions. >> >> >> Correct. Bitcoin Core has cached validation for many years now... if >> not for that and other optimizations, things would be really broken >> right now. :) >> >> > Assuming this is all >> > true, and I have no reason to doubt that at this point, I do not >> understand >> > why there is any discussion at all about the (technical) impact of large >> > blocks, why there are large numbers of miners building on invalid blocks >> > (SPV mining, https://bitcoin.org/en/alert/2015-07-04-spv-mining), or >> why >> > there is any discussion about the speed of block validation (cpu >> processing >> > time to verify blocks and transactions in blocks being a limitation). >> >> I'm also mystified by a lot of the large block discussion, much of it >> is completely divorced from the technology as deployed; much less what >> we-- in industry-- know to be possible. I don't blame you or anyone in >> particular on this; it's a new area and we don't yet know what we need >> to know to know what we need to know; or to the extent that we do it >> hasn't had time to get effectively communicated. >> >> The technical/security implications of larger blocks are related to >> other things than propagation time, if you assume people are using the >> available efficient relay protocol (or better). >> >> SPV mining is a bit of a misnomer (If I coined the term, I'm sorry). >> What these parties are actually doing is blinding mining on top of >> other pools' stratum work. You can think of it as sub-pooling with >> hopping onto whatever pool has the highest block (I'll call it VFSSP >> in this post-- validation free stratum subpooling). It's very easy to >> implement, and there are other considerations. >> >> It was initially deployed at a time when a single pool in Europe has >> amassed more than half of the hashrate. This pool had propagation >> problems and a very high orphan rate, it may have (perhaps >> unintentionally) been performing a selfish mining attack; mining off >> their stratum work was an easy fix which massively cut down the orphan >> rates for anyone who did it. This was before the relay network >> protocol existed (the fact that all the hashpower was consolidating on >> a single pool was a major motivation for creating it). >> >> VFSSP also cuts through a number of practical issues miners have had: >> Miners that run their own bitcoin nodes in far away colocation >> (>100ms) due to local bandwidth or connectivity issues (censored >> internet); relay network hubs not being anywhere near by due to >> strange internet routing (e.g. japan to china going via the US for ... >> reasons...); the CreateNewBlock() function being very slow and >> unoptimized, etc. There are many other things like this-- and VFSSP >> avoids them causing delays even when you don't understand them or know >> about them. So even when they're easily fixed the VFSSP is a more >> general workaround. >> >> Mining operations are also usually operated in a largely fire and >> forget manner. There is a long history in (esp pooled) mining where >> someone sets up an operation and then hardly maintains it after the >> fact... so some of the use of VFSSP appears to just be inertia-- we >> have better solutions now, but they they work to deploy and changing >> things involves risk (which is heightened by a lack of good >> monitoring-- participants learn they are too latent by observing >> orphaned blocks at a cost of 25 BTC each). >> >> One of the frustrating things about incentives in this space is that >> bad outcomes are possible even when they're not necessary. E.g. if a >> miner can lower their orphan rate by deploying a new protocol (or >> simply fixing some faulty hardware in their infrastructure, like >> Bitcoin nodes running on cheap VPSes with remote storage) OR they can >> lower their orphan rate by pointing their hashpower at a free >> centeralized pool, they're likely to do the latter because it takes >> less effort. >> _______________________________________________ >> bitcoin-dev mailing list >> bitcoin-dev@lists.linuxfoundation.org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >> > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > --001a1135c5feae7896051ca7ecfb Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Other than the source code, the best documentation I'v= e come across is a few
lines on=C2=A0IRC explaining the high-level design of the prot= ocol:=C2=A0

On Thu, Aug 6, 2015 at 10:18 AM Serg= io Demian Lerner via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
=
Is there any up to da= te documentation about TheBlueMatt relay network including what kind of blo= ck compression it is currently doing? (apart from the source code)

=
Regards, Sergio.

On Wed, Aug 5, 2015 at 7:14 PM, Gregory Maxwell via b= itcoin-dev <bitcoin-dev@lists.linuxfoundation.org&= gt; wrote:
On Wed, Aug 5, 20= 15 at 9:19 PM, Arnoud Kouwenhoven - Pukaki Corp
<arnoud@pukaki.bz<= /a>> wrote:
> Thanks for this (direct) feedback. It would make sense that if blocks = can be
> submitted using ~5kb packets, that no further optimizations would be n= eeded
> at this point. I will look into the relay network transmission protoco= l to
> understand how it works!
>
> I hear that you are saying that this network solves speed of transmiss= ion
> and thereby (technical) block size issues. Presumably it would solve s= peed
> of block validation too by prevalidating transactions.


Correct. Bitcoin Core has cached validation for many years now... if=
not for that and other optimizations, things would be really broken
right now. :)

> Assuming this is all
> true, and I have no reason to doubt that at this point, I do not under= stand
> why there is any discussion at all about the (technical) impact of lar= ge
> blocks, why there are large numbers of miners building on inval= id blocks
> (SPV mining, https://bitcoin.org/en/ale= rt/2015-07-04-spv-mining), or why
> there is any discussion about the speed of block validation (cpu proce= ssing
> time to verify blocks and transactions in blocks being a limitation).<= br>
I'm also mystified by a lot of the large block discussion, much = of it
is completely divorced from the technology as deployed; much less what
we-- in industry-- know to be possible. I don't blame you or anyone in<= br> particular on this; it's a new area and we don't yet know what we n= eed
to know to know what we need to know; or to the extent that we do it
hasn't had time to get effectively communicated.

The technical/security implications of larger blocks are related to
other things than propagation time, if you assume people are using the
available efficient relay protocol (or better).

SPV mining is a bit of a misnomer (If I coined the term, I'm sorry). What these parties are actually doing is blinding mining on top of
other pools' stratum work. You can think of it as sub-pooling with
hopping onto whatever pool has the highest block (I'll call it VFSSP in this post-- validation free stratum subpooling).=C2=A0 It's very eas= y to
implement, and there are other considerations.

It was initially deployed at a time when a single pool in Europe has
amassed more than half of the hashrate. This pool had propagation
problems and a very high orphan rate, it may have (perhaps
unintentionally) been performing a selfish mining attack; mining off
their stratum work was an easy fix which massively cut down the orphan
rates for anyone who did it.=C2=A0 This was before the relay network
protocol existed (the fact that all the hashpower was consolidating on
a single pool was a major motivation for creating it).

VFSSP also cuts through a number of practical issues miners have had:
Miners that run their own bitcoin nodes in far away colocation
(>100ms) due to local bandwidth or connectivity issues (censored
internet); relay network hubs not being anywhere near by due to
strange internet routing (e.g. japan to china going via the US for ...
reasons...); the CreateNewBlock() function being very slow and
unoptimized, etc.=C2=A0 =C2=A0There are many other things like this-- and V= FSSP
avoids them causing delays even when you don't understand them or know<= br> about them. So even when they're easily fixed the VFSSP is a more
general workaround.

Mining operations are also usually operated in a largely fire and
forget manner. There is a long history in (esp pooled) mining where
someone sets up an operation and then hardly maintains it after the
fact... so some of the use of VFSSP appears to just be inertia-- we
have better solutions now, but they they work to deploy and changing
things involves risk (which is heightened by a lack of good
monitoring-- participants learn they are too latent by observing
orphaned blocks at a cost of 25 BTC each).

One of the frustrating things about incentives in this space is that
bad outcomes are possible even when they're not necessary. E.g. if a miner can lower their orphan rate by deploying a new protocol (or
simply fixing some faulty hardware in their infrastructure, like
Bitcoin nodes running on cheap VPSes with remote storage)=C2=A0 OR they can=
lower their orphan rate by pointing their hashpower at a free
centeralized pool, they're likely to do the latter because it takes
less effort.
_______________________________________________
bitcoin-dev mailing list
= bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mail= man/listinfo/bitcoin-dev

_______________________________________________
bitcoin-dev mailing list
= bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mail= man/listinfo/bitcoin-dev
--001a1135c5feae7896051ca7ecfb--