Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id F329F2D for ; Fri, 8 Jun 2018 05:03:18 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f52.google.com (mail-wm0-f52.google.com [74.125.82.52]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id E36EB734 for ; Fri, 8 Jun 2018 05:03:17 +0000 (UTC) Received: by mail-wm0-f52.google.com with SMTP id 69-v6so1133678wmf.3 for ; Thu, 07 Jun 2018 22:03:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=AJdMCpfqkfnHGPDr85TwIVMcRYD/GS7+KGCYk2mKR4M=; b=ThYZgKsWojwa7Ob04h2+CdCcw2WsvPfEBoNDjgTAyFWSI4E/RaxaiRg2VSPopHppRv G36/68IlPJdRv9KVDFiodCx/Eqs6+ejtp7/er4yZsbHDULxqx9kihqm6bR8LMbGYFNWd jTmnZVVUjbSHenfQLvh7FddMQ79jVdNLQNgPxcvkhbKa/yCrFdA3IKw7f6yQisGTaCL6 aj6JQi29F43RUV7gSMZ+waH26uo3zZjXa9e40MXVCrdDGBsWIOmHJK3OIny5giAw+LOR qynGMKh5JXmfJwnRAv6dlnKHCzA682pTZubM3mryoe6rCD3YugkP8JOnBCEpBn2sb1GS 8evw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=AJdMCpfqkfnHGPDr85TwIVMcRYD/GS7+KGCYk2mKR4M=; b=eZXOkICRnMiiB/6WPlCG+AtDMLUQxa3D7oDmuxkgf8T5YsGOJKkqnSqpx5mn7kEsus ULIYtz0ywrcJdAKa6FEndau+55cuG0Nok3OhQY/6VT3ICap/hIKQ6kLwcaH0B3voAG41 yqfKfeKiI5LDrGKYUEqz1FUM0Mi+jTexxgwQdAu/7VDS7qQ2Ba3FIOw7+eNuLYveGXc7 V/bJaPeqoKFNn/D9ucKuLmebRxhGNiUD3j2+6CGUmnkrKriBtjlzz+6sGGKKDa2OEqDQ 70gc799d9zQSsZOG8rq75/rqPOqgc8Z1+q1zmu2IYLJe2OHy3uaguASwn3WBao6SrvyB QZRQ== X-Gm-Message-State: APt69E10cRWkfJd+WyGdJ8niAH6ys1RvExL1EyIxEiPp68OSIxesAfDo jtpXSx8JjTGR0iSd0zaSRNiU02wHdZdaSgHqevg= X-Google-Smtp-Source: ADUXVKLFFl1GPyVqgdpqnxfp2M84MftlWRBveTHLz3Xuoi/lkvhB99FuvYPkeEZSKlUWPAf6dmIBXRQtX672WitluNY= X-Received: by 2002:a50:d311:: with SMTP id g17-v6mr5291138edh.160.1528434196334; Thu, 07 Jun 2018 22:03:16 -0700 (PDT) MIME-Version: 1.0 References: <7E4FA664-BBAF-421F-8C37-D7CE3AA5310A@gmail.com> <20180602124157.744x7j4u7dqtaa43@email> <343A3542-3103-42E9-95B7-640DFE958FFA@gmail.com> <37BECD1A-7515-4081-85AC-871B9FB57772@gmail.com> In-Reply-To: From: Olaoluwa Osuntokun Date: Thu, 7 Jun 2018 22:03:04 -0700 Message-ID: To: Pieter Wuille , Bitcoin Protocol Discussion Content-Type: multipart/alternative; boundary="00000000000055d0a0056e1a515d" X-Spam-Status: No, score=-1.7 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM, HTML_MESSAGE,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: Re: [bitcoin-dev] BIP 158 Flexibility and Filter Size X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jun 2018 05:03:19 -0000 --00000000000055d0a0056e1a515d Content-Type: text/plain; charset="UTF-8" Hi sipa, > The advantage of (a) is that it can be verified against a full block without > access to the outputs being spent by it > > The advantage of (b) is that it is more compact (scriot reuse, and outputs > spent within the same block as they are created). Thanks for this breakdown. I think you've accurately summarized the sole remaining discussing point in this thread. As someone who's written and reviews code integrating the proposal all the way up the stack (from node to wallet, to application), IMO, there's no immediate cost to deferring the inclusion/creation of a filter that includes prev scripts (b) instead of the outpoint as the "regular" filter does now. Switching to prev script in the _short term_ would be costly for the set of applications already deployed (or deployed in a minimal or flag flip gated fashion) as the move from prev script to outpoint is a cascading one that impacts wallet operation, rescans, HD seed imports, etc. Maintaining the outpoint also allows us to rely on a "single honest peer" security model in the short term. In the long term the main barrier to committing the filters isn't choosing what to place in the filters (as once you have the gcs code, adding/removing elements is a minor change), but the actual proposal to add new consensus enforced commitments to Bitcoin in the first place. Such a proposal would need to be generalized enough to allow several components to be committed, likely have versioning, and also provide the necessary extensibility to allow additional items to be committed in the future. To my knowledge no such soft-fork has yet been proposed in a serious manner, although we have years of brainstorming on the topic. The timeline of the drafting, design, review, and deployment of such a change would likely be measures in years, compared to the immediate deployment of the current p2p filter model proposed in the BIP. As a result, I see no reason to delay the p2p filter deployment (with the outpoint) in the short term, as the long lead time a soft-fork to add extensible commitments to Bitcoin would give application+wallet authors ample time to switch to the new model. Also there's no reason that full-node wallets which wish to primarily use the filters for rescan purposes can't just construct them locally for this particular use case independent of what's currently deployed on the p2p network. Finally, I've addressed the remaining comments on my PR modifying the BIP from my last message. -- Laolu On Sat, Jun 2, 2018 at 11:12 PM Pieter Wuille via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > On Sat, Jun 2, 2018, 22:56 Tamas Blummer via bitcoin-dev < > bitcoin-dev@lists.linuxfoundation.org> wrote: > >> Lighter but SPV secure nodes (filter committed) would help the network >> (esp. Layer 2) to grow mesh like, but add more user that blindly follow POW. >> >> On longer term most users' security will be determined by either trusted >> hubs or POW. >> I do not know which is worse, but we should at least offer the choice to >> the user, therefore commit filters. >> > > I don't think that's the point of discussion here. Of course, in order to > have filters that verifiably don't lie by omission, the filters need to be > committed to by blocks. > > The question is what data that filter should contain. > > There are two suggestions: > (a) The scriptPubKeys of the block's outputs, and prevouts of the block's > inputs. > (b) The scriptPubKeys of the block's outputs, and scriptPubKeys of outputs > being spent by the block's inputs. > > The advantage of (a) is that it can be verified against a full block > without access to the outputs being spent by it. This allows light clients > to ban nodes that give them incorrect filters, but they do need to actually > see the blocks (partially defeating the purpose of having filters in the > first place). > > The advantage of (b) is that it is more compact (scriot reuse, and outputs > spent within the same block as they are created). It also had the advantage > of being more easily usable for scanning of a wallet's transactions. Using > (a) for that in some cases may need to restart and refetch when an output > is discovered, to go test for its spending (whose outpoint is not known > ahead of time). Especially when fetching multiple filters at a time this > may be an issue. > > I think both of these potentially good arguments. However, once a > committed filter exists, the advantage of (a) goes away completely - > validation of committed filters is trivial and can be done without needing > the full blocks in the first place. > > So I think the question is do we aim for an uncommitted (a) first and a > committed (b) later, or go for (b) immediately? > > Cheers, > > -- > Pieter > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > --00000000000055d0a0056e1a515d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Hi sipa,

> The advant= age of (a) is that it can be verified against a full block without
> access to the outputs being spent by it
>=C2=A0
> The advantage of (b) is that it is more compact (scriot reuse, and = outputs
> spent within the same block as they are created).

Thanks for this breakdown. I think you've accura= tely summarized the sole
remaining discussing point in this threa= d.

As someone who's written and reviews code i= ntegrating the proposal all the
way up the stack (from node to wa= llet, to application), IMO, there's no
immediate cost to defe= rring the inclusion/creation of a filter that includes
prev scrip= ts (b) instead of the outpoint as the "regular" filter does now.<= /div>
Switching to prev script in the _short term_ would be costly for = the set of
applications already deployed (or deployed in a minima= l or flag flip gated
fashion) as the move from prev script to out= point is a cascading one that
impacts wallet operation, rescans, = HD seed imports, etc.

Maintaining the outpoint als= o allows us to rely on a "single honest peer"
security = model in the short term. In the long term the main barrier to
com= mitting the filters isn't choosing what to place in the filters (as onc= e
you have the gcs code, adding/removing elements is a minor chan= ge), but the
actual proposal to add new consensus enforced commit= ments to Bitcoin in the
first place. Such a proposal would need t= o be generalized enough to allow
several components to be committ= ed, likely have versioning, and also provide
the necessary extens= ibility to allow additional items to be committed in the
future. = To my knowledge no such soft-fork has yet been proposed in a serious
<= div>manner, although we have years of brainstorming on the topic. The timel= ine
of the drafting, design, review, and deployment of such a cha= nge would
likely be measures in years, compared to the immediate = deployment of the
current p2p filter model proposed in the BIP.= =C2=A0

As a result, I see no reason to delay the p= 2p filter deployment (with the
outpoint) in the short term, as th= e long lead time a soft-fork to add
extensible commitments to Bit= coin would give application+wallet authors
ample time to switch t= o the new model. Also there's no reason that full-node
wallet= s which wish to primarily use the filters for rescan purposes can't
just construct them locally for this particular use case independent= of
what's currently deployed on the p2p network.
<= br>
Finally, I've addressed the remaining comments on my PR m= odifying the BIP
from my last message.=C2=A0

=
-- Laolu

On Sat, Jun 2, 2018 at 11:12 PM Pieter Wuille via bitcoin-dev <bitcoin-dev@lists.linuxf= oundation.org> wrote:
On Sat, Jun 2,= 2018, 22:56 Tamas Blummer via bitcoin-dev <bitcoin-dev@lists.linuxfound= ation.org> wrote:
Lighter = but SPV secure nodes (filter committed) would help the network (esp. Layer = 2) to grow mesh like, but add more user that blindly follow POW.

On longer term most users' security will be determined by either truste= d hubs or POW.
I do not know which is worse, but we should at least offer the choice to th= e user, therefore commit filters.

I don't think = that's the point of discussion here. Of course, in order to have filter= s that verifiably don't lie by omission, the filters need to be committ= ed to by blocks.

The que= stion is what data that filter should contain.

<= /div>
There are two suggestions:
(a= ) The scriptPubKeys of the block's outputs, and prevouts of the block&#= 39;s inputs.
(b) The scriptPubKeys of the block'= s outputs, and scriptPubKeys of outputs being spent by the block's inpu= ts.

The advantage of (a)= is that it can be verified against a full block without access to the outp= uts being spent by it. This allows light clients to ban nodes that give the= m incorrect filters, but they do need to actually see the blocks (partially= defeating the purpose of having filters in the first place).

The advantage of (b) is that it is mo= re compact (scriot reuse, and outputs spent within the same block as they a= re created). It also had the advantage of being more easily usable for scan= ning of a wallet's transactions. Using (a) for that in some cases may n= eed to restart and refetch when an output is discovered, to go test for its= spending (whose outpoint is not known ahead of time). Especially when fetc= hing multiple filters at a time this may be an issue.

I think both of these potentially good argume= nts. However, once a committed filter exists, the advantage of (a) goes awa= y completely - validation of committed filters is trivial and can be done w= ithout needing the full blocks in the first place.
<= br>
So I think the question is do we aim for an unco= mmitted (a) first and a committed (b) later, or go for (b) immediately?

Cheers,

--=C2=A0
Pieter

_______________________________________________
bitcoin-dev mailing list
= bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mail= man/listinfo/bitcoin-dev
--00000000000055d0a0056e1a515d--