path: root/13/5323abd2c5660562fa67e804834b67be805c70

Return-Path: <lf-lists@mattcorallo.com>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
	[172.17.192.35])
	by mail.linuxfoundation.org (Postfix) with ESMTPS id 02A3F895
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Thu,  6 Aug 2015 20:38:56 +0000 (UTC)
X-Greylist: from auto-whitelisted by SQLgrey-1.7.6
Received: from mail.bluematt.me (mail.bluematt.me [192.241.179.72])
	by smtp1.linuxfoundation.org (Postfix) with ESMTPS id E15101AD
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Thu,  6 Aug 2015 20:38:54 +0000 (UTC)
Received: from [10.64.3.246] (unknown [79.143.111.208])
	by mail.bluematt.me (Postfix) with ESMTPSA id 7EFAD55299;
	Thu,  6 Aug 2015 20:38:52 +0000 (UTC)
In-Reply-To: <CAKzdR-qPtPsxAXsmUX=vTkq-ro=EAmH7M8nL_Px_b4D4Z0WAXg@mail.gmail.com>
References: <CALwsPgnnkbfUBhL=Qyspz13pnZ-6RHdaZOGvfLG34JjJRgt2Dw@mail.gmail.com>
	<B3546CB9-6A24-474C-8B56-9B1E2D33B470@mattcorallo.com>
	<CALwsPgm6xcBfLXZTNTZ40R_s3oUawE0ANZycDWpSo0cXZ+=-Vg@mail.gmail.com>
	<CAAS2fgQLNH+rivNNTFz6xt_9SxO3fFj7-3z7A-_B_2X2x-6M5w@mail.gmail.com>
	<CALwsPgm9S3UNd3bEuWreyGS7bcvSD+cXxueoD+F_D9fC=xLz2Q@mail.gmail.com>
	<CAAS2fgSXWzCv=4cF=0bwL9+udzBHSPR7goL3U_c1NjS22dpWzQ@mail.gmail.com>
	<CAKzdR-qPtPsxAXsmUX=vTkq-ro=EAmH7M8nL_Px_b4D4Z0WAXg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----H0P36CWX80JS325N03G1FMA72BINOJ"
Content-Transfer-Encoding: 8bit
From: Matt Corallo <lf-lists@mattcorallo.com>
Date: Thu, 06 Aug 2015 20:38:41 +0000
To: Sergio Demian Lerner <sergio.d.lerner@gmail.com>,
	Sergio Demian Lerner via bitcoin-dev
	<bitcoin-dev@lists.linuxfoundation.org>,
	Arnoud Kouwenhoven - Pukaki Corp via bitcoin-dev
	<bitcoin-dev@lists.linuxfoundation.org>
Message-ID: <B839817E-07FA-4F47-B35A-CFFE60D9B87D@mattcorallo.com>
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,HTML_MESSAGE
	autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
	smtp1.linux-foundation.org
Subject: Re: [bitcoin-dev] Idea: Efficient bitcoin block propagation
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Thu, 06 Aug 2015 20:38:56 -0000

------H0P36CWX80JS325N03G1FMA72BINOJ
Content-Transfer-Encoding: 8bit
Content-Type: text/plain;
 charset=UTF-8

No, don't think so, the protocol is, essentially, relay transactions, when you get a block, send header, iterate over transactions, for each, either use two bytes for nth-recent-transaction-relayed, use 0xffff-3-byte-length-transaction-data. There are quite a few implementation details, and lots of things could be improved, but that is pretty much how it works.

Matt

On August 6, 2015 7:16:56 PM GMT+02:00, Sergio Demian Lerner via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
>Is there any up to date documentation about TheBlueMatt relay network
>including what kind of block compression it is currently doing? (apart
>from
>the source code)
>
>Regards, Sergio.
>
>On Wed, Aug 5, 2015 at 7:14 PM, Gregory Maxwell via bitcoin-dev <
>bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> On Wed, Aug 5, 2015 at 9:19 PM, Arnoud Kouwenhoven - Pukaki Corp
>> <arnoud@pukaki.bz> wrote:
>> > Thanks for this (direct) feedback. It would make sense that if
>blocks
>> can be
>> > submitted using ~5kb packets, that no further optimizations would
>be
>> needed
>> > at this point. I will look into the relay network transmission
>protocol
>> to
>> > understand how it works!
>> >
>> > I hear that you are saying that this network solves speed of
>transmission
>> > and thereby (technical) block size issues. Presumably it would
>solve
>> speed
>> > of block validation too by prevalidating transactions.
>>
>>
>> Correct. Bitcoin Core has cached validation for many years now... if
>> not for that and other optimizations, things would be really broken
>> right now. :)
>>
>> > Assuming this is all
>> > true, and I have no reason to doubt that at this point, I do not
>> understand
>> > why there is any discussion at all about the (technical) impact of
>large
>> > blocks, why there are large numbers of miners building on invalid
>blocks
>> > (SPV mining, https://bitcoin.org/en/alert/2015-07-04-spv-mining),
>or why
>> > there is any discussion about the speed of block validation (cpu
>> processing
>> > time to verify blocks and transactions in blocks being a
>limitation).
>>
>> I'm also mystified by a lot of the large block discussion, much of it
>> is completely divorced from the technology as deployed; much less
>what
>> we-- in industry-- know to be possible. I don't blame you or anyone
>in
>> particular on this; it's a new area and we don't yet know what we
>need
>> to know to know what we need to know; or to the extent that we do it
>> hasn't had time to get effectively communicated.
>>
>> The technical/security implications of larger blocks are related to
>> other things than propagation time, if you assume people are using
>the
>> available efficient relay protocol (or better).
>>
>> SPV mining is a bit of a misnomer (If I coined the term, I'm sorry).
>> What these parties are actually doing is blinding mining on top of
>> other pools' stratum work. You can think of it as sub-pooling with
>> hopping onto whatever pool has the highest block (I'll call it VFSSP
>> in this post-- validation free stratum subpooling).  It's very easy
>to
>> implement, and there are other considerations.
>>
>> It was initially deployed at a time when a single pool in Europe has
>> amassed more than half of the hashrate. This pool had propagation
>> problems and a very high orphan rate, it may have (perhaps
>> unintentionally) been performing a selfish mining attack; mining off
>> their stratum work was an easy fix which massively cut down the
>orphan
>> rates for anyone who did it.  This was before the relay network
>> protocol existed (the fact that all the hashpower was consolidating
>on
>> a single pool was a major motivation for creating it).
>>
>> VFSSP also cuts through a number of practical issues miners have had:
>> Miners that run their own bitcoin nodes in far away colocation
>> (>100ms) due to local bandwidth or connectivity issues (censored
>> internet); relay network hubs not being anywhere near by due to
>> strange internet routing (e.g. japan to china going via the US for
>...
>> reasons...); the CreateNewBlock() function being very slow and
>> unoptimized, etc.   There are many other things like this-- and VFSSP
>> avoids them causing delays even when you don't understand them or
>know
>> about them. So even when they're easily fixed the VFSSP is a more
>> general workaround.
>>
>> Mining operations are also usually operated in a largely fire and
>> forget manner. There is a long history in (esp pooled) mining where
>> someone sets up an operation and then hardly maintains it after the
>> fact... so some of the use of VFSSP appears to just be inertia-- we
>> have better solutions now, but they they work to deploy and changing
>> things involves risk (which is heightened by a lack of good
>> monitoring-- participants learn they are too latent by observing
>> orphaned blocks at a cost of 25 BTC each).
>>
>> One of the frustrating things about incentives in this space is that
>> bad outcomes are possible even when they're not necessary. E.g. if a
>> miner can lower their orphan rate by deploying a new protocol (or
>> simply fixing some faulty hardware in their infrastructure, like
>> Bitcoin nodes running on cheap VPSes with remote storage)  OR they
>can
>> lower their orphan rate by pointing their hashpower at a free
>> centeralized pool, they're likely to do the latter because it takes
>> less effort.
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>bitcoin-dev mailing list
>bitcoin-dev@lists.linuxfoundation.org
>https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

------H0P36CWX80JS325N03G1FMA72BINOJ
Content-Type: text/html;
 charset=utf-8
Content-Transfer-Encoding: 8bit

<html><head></head><body>No, don&#39;t think so, the protocol is, essentially, relay transactions, when you get a block, send header, iterate over transactions, for each, either use two bytes for nth-recent-transaction-relayed, use 0xffff-3-byte-length-transaction-data. There are quite a few implementation details, and lots of things could be improved, but that is pretty much how it works.<br>
<br>
Matt<br><br><div class="gmail_quote">On August 6, 2015 7:16:56 PM GMT+02:00, Sergio Demian Lerner via bitcoin-dev &lt;bitcoin-dev@lists.linuxfoundation.org&gt; wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div dir="ltr">Is there any up to date documentation about TheBlueMatt relay network including what kind of block compression it is currently doing? (apart from the source code)<div><br /><div>Regards, Sergio.</div></div></div><div class="gmail_extra"><br /><div class="gmail_quote">On Wed, Aug 5, 2015 at 7:14 PM, Gregory Maxwell via bitcoin-dev <span dir="ltr">&lt;<a href="mailto:bitcoin-dev@lists.linuxfoundation.org" target="_blank">bitcoin-dev@lists.linuxfoundation.org</a>&gt;</span> wrote:<br /><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Wed, Aug 5, 2015 at 9:19 PM, Arnoud Kouwenhoven - Pukaki Corp<br />
&lt;<a href="mailto:arnoud@pukaki.bz">arnoud@pukaki.bz</a>&gt; wrote:<br />
&gt; Thanks for this (direct) feedback. It would make sense that if blocks can be<br />
&gt; submitted using ~5kb packets, that no further optimizations would be needed<br />
&gt; at this point. I will look into the relay network transmission protocol to<br />
&gt; understand how it works!<br />
&gt;<br />
&gt; I hear that you are saying that this network solves speed of transmission<br />
&gt; and thereby (technical) block size issues. Presumably it would solve speed<br />
&gt; of block validation too by prevalidating transactions.<br />
<br />
<br />
</span>Correct. Bitcoin Core has cached validation for many years now... if<br />
not for that and other optimizations, things would be really broken<br />
right now. :)<br />
<span class=""><br />
&gt; Assuming this is all<br />
&gt; true, and I have no reason to doubt that at this point, I do not understand<br />
&gt; why there is any discussion at all about the (technical) impact of large<br />
</span>&gt; blocks, why there are large numbers of miners building on invalid blocks<br />
<span class="">&gt; (SPV mining, <a href="https://bitcoin.org/en/alert/2015-07-04-spv-mining" rel="noreferrer" target="_blank">https://bitcoin.org/en/alert/2015-07-04-spv-mining</a>), or why<br />
&gt; there is any discussion about the speed of block validation (cpu processing<br />
&gt; time to verify blocks and transactions in blocks being a limitation).<br />
<br />
</span>I&#39;m also mystified by a lot of the large block discussion, much of it<br />
is completely divorced from the technology as deployed; much less what<br />
we-- in industry-- know to be possible. I don&#39;t blame you or anyone in<br />
particular on this; it&#39;s a new area and we don&#39;t yet know what we need<br />
to know to know what we need to know; or to the extent that we do it<br />
hasn&#39;t had time to get effectively communicated.<br />
<br />
The technical/security implications of larger blocks are related to<br />
other things than propagation time, if you assume people are using the<br />
available efficient relay protocol (or better).<br />
<br />
SPV mining is a bit of a misnomer (If I coined the term, I&#39;m sorry).<br />
What these parties are actually doing is blinding mining on top of<br />
other pools&#39; stratum work. You can think of it as sub-pooling with<br />
hopping onto whatever pool has the highest block (I&#39;ll call it VFSSP<br />
in this post-- validation free stratum subpooling).  It&#39;s very easy to<br />
implement, and there are other considerations.<br />
<br />
It was initially deployed at a time when a single pool in Europe has<br />
amassed more than half of the hashrate. This pool had propagation<br />
problems and a very high orphan rate, it may have (perhaps<br />
unintentionally) been performing a selfish mining attack; mining off<br />
their stratum work was an easy fix which massively cut down the orphan<br />
rates for anyone who did it.  This was before the relay network<br />
protocol existed (the fact that all the hashpower was consolidating on<br />
a single pool was a major motivation for creating it).<br />
<br />
VFSSP also cuts through a number of practical issues miners have had:<br />
Miners that run their own bitcoin nodes in far away colocation<br />
(&gt;100ms) due to local bandwidth or connectivity issues (censored<br />
internet); relay network hubs not being anywhere near by due to<br />
strange internet routing (e.g. japan to china going via the US for ...<br />
reasons...); the CreateNewBlock() function being very slow and<br />
unoptimized, etc.   There are many other things like this-- and VFSSP<br />
avoids them causing delays even when you don&#39;t understand them or know<br />
about them. So even when they&#39;re easily fixed the VFSSP is a more<br />
general workaround.<br />
<br />
Mining operations are also usually operated in a largely fire and<br />
forget manner. There is a long history in (esp pooled) mining where<br />
someone sets up an operation and then hardly maintains it after the<br />
fact... so some of the use of VFSSP appears to just be inertia-- we<br />
have better solutions now, but they they work to deploy and changing<br />
things involves risk (which is heightened by a lack of good<br />
monitoring-- participants learn they are too latent by observing<br />
orphaned blocks at a cost of 25 BTC each).<br />
<br />
One of the frustrating things about incentives in this space is that<br />
bad outcomes are possible even when they&#39;re not necessary. E.g. if a<br />
miner can lower their orphan rate by deploying a new protocol (or<br />
simply fixing some faulty hardware in their infrastructure, like<br />
Bitcoin nodes running on cheap VPSes with remote storage)  OR they can<br />
lower their orphan rate by pointing their hashpower at a free<br />
centeralized pool, they&#39;re likely to do the latter because it takes<br />
less effort.<br />
<div class="HOEnZb"><div class="h5">_______________________________________________<br />
bitcoin-dev mailing list<br />
<a href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a><br />
<a href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" rel="noreferrer" target="_blank">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a><br />
</div></div></blockquote></div><br /></div>
<p style="margin-top: 2.5em; margin-bottom: 1em; border-bottom: 1px solid #000"></p><pre class="k9mail"><hr /><br />bitcoin-dev mailing list<br />bitcoin-dev@lists.linuxfoundation.org<br /><a href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a><br /></pre></blockquote></div></body></html>
------H0P36CWX80JS325N03G1FMA72BINOJ--