Return-Path: Received: from smtp2.linuxfoundation.org (smtp2.linux-foundation.org [172.17.192.36]) by mail.linuxfoundation.org (Postfix) with ESMTPS id A5D38899 for ; Mon, 27 Mar 2017 19:43:37 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mail.bluematt.me (mail.bluematt.me [192.241.179.72]) by smtp2.linuxfoundation.org (Postfix) with ESMTPS id 31B7C1DCA5 for ; Mon, 27 Mar 2017 19:43:36 +0000 (UTC) Received: from [25.39.208.129] (unknown [172.56.39.39]) by mail.bluematt.me (Postfix) with ESMTPSA id 17186139BF1; Mon, 27 Mar 2017 19:32:45 +0000 (UTC) Date: Mon, 27 Mar 2017 19:32:30 +0000 In-Reply-To: References: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----IG1U3IKD6KVHEP9LKVLCICMDMTV7UF" Content-Transfer-Encoding: 7bit To: Suhas Daftuar , Bitcoin Protocol Discussion , Suhas Daftuar via bitcoin-dev , Bitcoin Dev From: Matt Corallo Message-ID: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,HTML_MESSAGE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp2.linux-foundation.org Subject: Re: [bitcoin-dev] Segregated witness p2p layer compatibility X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Mar 2017 19:43:37 -0000 ------IG1U3IKD6KVHEP9LKVLCICMDMTV7UF Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Just to expand a tiny bit here, while the testnet setup of only a few nodes= acting as "bridges", mainnet already has many systems which act as effecti= ve bridges today - there are several relay networks in use which effectivel= y bypass the P2P network, including my legacy relay network (which many min= ers historically have used, and I'd expect those who aren't paying attentio= n and don't upgrade will not turn off, fixing the issue for them), ViaBTC's= super aggressive bandwidth-wasting block announcement network which pushes= blocks from several pools to many nodes globally, and Bitcoin=2Ecom's priv= ate relay network=2E (Of course many other miners and pools have private re= lay networks, but the several other such networks I'm aware of are already = segwit-compatible, even for pools not signaling segwit)=2E Matt On March 27, 2017 12:22:43 PM PDT, Suhas Daftuar via bitcoin-dev wrote: >Hi, > >There have been two threads recently that have made references to >peer-to-peer implementation details in Bitcoin Core's Segregated >Witness >code that I would like to clarify=2E > >In the thread "Issolated Bitcoin Nodes" ( >https://lists=2Elinuxfoundation=2Eorg/pipermail/bitcoin-dev/2017-March/01= 3765=2Ehtml), >there was some discussion about how Bitcoin Core's block download logic >behaves after segwit activation=2E After segwit activation, Bitcoin Core >nodes will not (currently) attempt to download any blocks from >non-segwit >peers (nodes that do not set the NODE WITNESS service bit)=2E This is a >bandwidth optimization to prevent a node from downloading a block that >may >be invalid only because the sender omitted the witness, requiring >re-download until the block is received with the required witness data=2E > >But to be clear, non-segwit blocks -- that is, blocks without a witness >commitment in the coinbase, and whose transactions are serialized >without >witnesses, and whose transactions are not spending segwit outputs which >require a witness -- are evaluated under the same rules as prior, >pre-segwit versions of the software=2E So such non-segwit blocks that >are >valid to older, pre-segwit nodes are also valid to segwit-nodes=2E > >In >https://lists=2Elinuxfoundation=2Eorg/pipermail/bitcoin-dev/2017-March/01= 3796=2Ehtml, >Eric Voskuil wrote: > >Given the protocol requirements of the segwit proposal this is not the >> case=2E A miner running pre-segwit code will produce blocks that no >> segwit node will ever receive=2E > > >The phrase "protocol requirements of segwit" is confusing here, because >there are two different layers that need consideration: the consensus >protocol layer and the peer-to-peer protocol layer=2E But in neither >layer >is the behavior of not downloading blocks from non-NODE WITNESS peers a >"requirement"=2E This is an implementation detail in the Bitcoin Core >code >that alternate implementations compliant with BIP 144 could implement >differently=2E > >At the consensus layer, non-segwit blocks (described above) that are >valid >to older nodes are also valid to segwit nodes=2E That means that if a >miner >was using an older, pre-segwit version of Bitcoin Core to produce >blocks >after segwit activates, that blocks they find will be valid to all >nodes=2E > >At the p2p layer, though, segwit-enabled Bitcoin Core nodes will only >try >to download those blocks if announced by a segwit-enabled peer=2E But >this >is not a protocol requirement; other implementations can remain >compatible >even they take different approaches here=2E (As an example, I could >imagine >an implementation that downloaded a new block from any peer, but if the >block has a witness commitment in the coinbase and was received from a >non-segwit peer, then the node would attempt re-download from a segwit >peer=2E I'm sure many other reasonable block download strategies could >be >devised=2E) > >Still, if a miner wants to continue mining post-segwit activation, but >using pre-segwit software, they would need a way to relay their blocks >to a >segwit-enabled peer=2E > >There are a few ways to do this that I can think of: > >- Use the RPC call "submitblock" on a segwit-enabled node=2E Calling >"submitblock" on a Bitcoin Core 0=2E13=2E1 (0=2E13=2E0 in the case of tes= tnet) >or >later node works fine as long as the block is valid (whether or not it >has >a witness commitment or witness transactions), and once a >segwit-enabled >peer has the block it will relay to other segwit peers=2E > >- Explicitly deliver the block to a segwit node over the p2p network, >even >if unrequested=2E Currently Bitcoin Core at least will process >unrequested >blocks, and valid blocks that update the tip will then be relayed to >other >peers=2E > >- Run a bridge node, which advertises NODE_WITNESS and can serialize >blocks >with witness data, which downloads blocks even from non-NODE WITNESS >peers=2E Anyone can do this to bridge the networks for the benefit of >the >whole network (I have personally been running a few nodes that do this, >for >several months now), but miners concerned about this issue for their >own >blocks could explicitly do this themselves to ensure that their own >blocks >propagate to the segwit-enabled network=2E > >- Peer directly with other miners, bypassing the p2p network=2E Many >miners >do this already, using protocols which may already serve to bridge the >network=2E > >So saying that "A miner running pre-segwit code will produce blocks >that no >segwit node will ever receive" is not really correct, in my view=2E If >the >whole network were just running Bitcoin Core software releases, and the >miner was not able/willing to deliver their block to a segwit-enabled >node >(eg by using the RPC call "submitblock", or one of the other >suggestions I >had above), then I would agree with the statement=2E But given that >there >are bridge nodes on the network, and that miners have other options to >relay their block, I think this is not an accurate portrayal of what >would >actually happen on the network -- I would expect that non-segwit >miners' >blocks would still get relayed post-segwit activation, even if only by >the >handful of bridge nodes that I expect are currently running=2E > >All that said, I do think this is an important detail to highlight and >that >this behavior should be better documented (I believe it deserves >specific >mention in a BIP), as this is an important issue for miners to be aware >of=2E ------IG1U3IKD6KVHEP9LKVLCICMDMTV7UF Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable Just to expand a tiny bit here, while the testnet = setup of only a few nodes acting as "bridges", mainnet already ha= s many systems which act as effective bridges today - there are several rel= ay networks in use which effectively bypass the P2P network, including my l= egacy relay network (which many miners historically have used, and I'd = expect those who aren't paying attention and don't upgrade will not= turn off, fixing the issue for them), ViaBTC's super aggressive bandwi= dth-wasting block announcement network which pushes blocks from several poo= ls to many nodes globally, and Bitcoin=2Ec= om's private relay network=2E (Of course many other miners and pool= s have private relay networks, but the several other such networks I'm = aware of are already segwit-compatible, even for pools not signaling segwit= )=2E

Matt

On March 27, 2017 12:22:43 PM PDT, = Suhas Daftuar via bitcoin-dev <bitcoin-dev@lists=2Elinuxfoundation=2Eorg= > wrote:
Hi,

There have been two threads rece= ntly that have made references to peer-to-peer implementation details in Bi= tcoin Core's Segregated Witness code that I would like to clarify=2E
<= div>
In the thread "Issolated Bitcoin Nodes" (https://lists=2Elinuxfoundation=2Eorg/pipermail/bitcoin-= dev/2017-March/013765=2Ehtml), there was some discussion about how Bitc= oin Core's block download logic behaves after segwit activation=2E  Af= ter segwit activation, Bitcoin Core nodes will not (currently) attempt to d= ownload any blocks from non-segwit peers (nodes that do not set the NODE WI= TNESS service bit)=2E  This is a bandwidth optimization to prevent a n= ode from downloading a block that may be invalid only because the sender om= itted the witness, requiring re-download until the block is received with t= he required witness data=2E

But to be clear, non= -segwit blocks -- that is, blocks without a witness commitment in the coinb= ase, and whose transactions are serialized without witnesses, and whose tra= nsactions are not spending segwit outputs which require a witness -- are ev= aluated under the same rules as prior, pre-segwit versions of the software= =2E  So such non-segwit blocks that are valid to older, pre-segwit nod= es are also valid to segwit-nodes=2E


Given the proto= col requirements of the segwit proposal this is not the
case=2E A mine= r running pre-segwit code will produce blocks that no
segwit node will= ever receive=2E

The phrase "protoco= l requirements of segwit" is confusing here, because there are two dif= ferent layers that need consideration: the consensus protocol layer and the= peer-to-peer protocol layer=2E  But in neither layer is the behavior = of not downloading blocks from non-NODE WITNESS peers a "requirement&q= uot;=2E  This is an implementation detail in the Bitcoin Core code tha= t alternate implementations compliant with BIP 144 could implement differen= tly=2E

At the consensus layer, non-segwit = blocks (described above) that are valid to older nodes are also valid to se= gwit nodes=2E  That means that if a miner was using an older, pre-segw= it version of Bitcoin Core to produce blocks after segwit activates, that b= locks they find will be valid to all nodes=2E

<= div>At the p2p layer, though, segwit-enabled Bitcoin Core nodes will only t= ry to download those blocks if announced by a segwit-enabled peer=2E  = But this is not a protocol requirement; other implementations can remain co= mpatible even they take different approaches here=2E  (As an example, = I could imagine an implementation that downloaded a new block from any peer= , but if the block has a witness commitment in the coinbase and was receive= d from a non-segwit peer, then the node would attempt re-download from a se= gwit peer=2E  I'm sure many other reasonable block download strategies= could be devised=2E)

Still, if a miner wants to= continue mining post-segwit activation, but using pre-segwit software, the= y would need a way to relay their blocks to a segwit-enabled peer=2E
<= div>
There are a few ways to do this that I can think of:

- Use the RPC call "submitblock" on a s= egwit-enabled node=2E  Calling "submitblock" on a Bitcoin Co= re 0=2E13=2E1 (0=2E13=2E0 in the case of testnet) or later node works fine = as long as the block is valid (whether or not it has a witness commitment o= r witness transactions), and once a segwit-enabled peer has the block it wi= ll relay to other segwit peers=2E

- Explicitly d= eliver the block to a segwit node over the p2p network, even if unrequested= =2E  Currently Bitcoin Core at least will process unrequested blocks, = and valid blocks that update the tip will then be relayed to other peers=2E=

- Run a bridge node, which advertises NODE_WITN= ESS and can serialize blocks with witness data, which downloads blocks even= from non-NODE WITNESS peers=2E  Anyone can do this to bridge the netw= orks for the benefit of the whole network (I have personally been running a= few nodes that do this, for several months now), but miners concerned abou= t this issue for their own blocks could explicitly do this themselves to en= sure that their own blocks propagate to the segwit-enabled network=2E
=

- Peer directly with other miners, bypassing the p2p = network=2E  Many miners do this already, using protocols which may alr= eady serve to bridge the network=2E

So saying th= at "A miner running pre-segwit code will produce blocks that no segwit= node will ever receive" is not really correct, in my view=2E  If= the whole network were just running Bitcoin Core software releases, and th= e miner was not able/willing to deliver their block to a segwit-enabled nod= e (eg by using the RPC call "submitblock", or one of the other su= ggestions I had above), then I would agree with the statement=2E  But = given that there are bridge nodes on the network, and that miners have othe= r options to relay their block, I think this is not an accurate portrayal o= f what would actually happen on the network -- I would expect that non-segw= it miners' blocks would still get relayed post-segwit activation, even if o= nly by the handful of bridge nodes that I expect are currently running=2E

All that said, I do think this is an importa= nt detail to highlight and that this behavior should be better documented (= I believe it deserves specific mention in a BIP), as this is an important i= ssue for miners to be aware of=2E
------IG1U3IKD6KVHEP9LKVLCICMDMTV7UF--