Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 25841D48 for ; Fri, 8 Jun 2018 23:35:44 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f51.google.com (mail-wm0-f51.google.com [74.125.82.51]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 77315E3 for ; Fri, 8 Jun 2018 23:35:42 +0000 (UTC) Received: by mail-wm0-f51.google.com with SMTP id j15-v6so6428487wme.0 for ; Fri, 08 Jun 2018 16:35:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=k4rZ2D7MGWoMmPcPUFs2bka4yeTasqEq8/Zi3mfDGWw=; b=IapVyg+VAU2ZYheJDENSwBW8lorphC3XSCnXCivIWDsRK3o/j2+PwFnLe07/X1H/7X lwzsd9CH/Cnmy2+W/8aX/VExU/37tFXEC+Fd5Z5Gbkg0G+SakEwK1rzQ2IbnBsbNZI3p CCFyxp8CvY2BRharvJ93T1ugEPMEg3tzCokj/k+U4LK3VUtpK5fZJ7gHPy+P5QobsgI2 lmpGoHhRJI8Tcxgh9c6zH4d/Ij7NRunTlVNedCgdPk8+auyElKbRwbKINfiv1axVmpze 30pewKgwr30Hq3UrHXksakodHm4CnARZPsGPOya5BuXqQQVdpgw15CWJGRHR6hiIU/ax OBcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=k4rZ2D7MGWoMmPcPUFs2bka4yeTasqEq8/Zi3mfDGWw=; b=U5zr3clmCoVaZQtuUVHgzl0gS7qouRMKl5eKYgAR9QQgblXuwqUDjSMsS4fZWKYcaR nagOLtB4HiEFzUDuBsEtwkJCjV+0YVg7blcOjka9bWe9fyRQ0uHr08MZg3f+Juh2DSlo pWLDHKfGf+qbIZvC0dNX3PK+MY8dU9I13+oN751ma4/uiNLAv2BUPUyPuqpbNH8k/0wc c303E1ZWGVtEgEFliPRK6XgEhrbJQAt4/b962FHJFsJYPZwF39gzFEvB8bQ9KIfXceYg 9nqwl7Le8R0t1dBJjHo4nA70uBIiV8d/PVBNW4Fd2RyZE1w8CvMXC0fdaNimxiJSLNk7 066A== X-Gm-Message-State: APt69E1SKH9VxycVbqLRR38IBQJ6HcI6ROAWDbp0oJTur1bcg/QMat87 FKJTt6Q6cD+vAdB0vNnmFtdrLDNBfIvBMafvth8= X-Google-Smtp-Source: ADUXVKJysAA8MPMW9IOjfAsk7n0tsSa5UrWi8XU6P4C8ztxqltLEplA+J3W+VitV4NNohltk3/5Cuw6DGLvXg2owyU8= X-Received: by 2002:a50:b6bc:: with SMTP id d57-v6mr8947988ede.250.1528500940937; Fri, 08 Jun 2018 16:35:40 -0700 (PDT) MIME-Version: 1.0 References: <7E4FA664-BBAF-421F-8C37-D7CE3AA5310A@gmail.com> <20180602124157.744x7j4u7dqtaa43@email> <343A3542-3103-42E9-95B7-640DFE958FFA@gmail.com> <37BECD1A-7515-4081-85AC-871B9FB57772@gmail.com> In-Reply-To: From: Olaoluwa Osuntokun Date: Fri, 8 Jun 2018 16:35:29 -0700 Message-ID: To: Gregory Maxwell Content-Type: multipart/alternative; boundary="0000000000009f9fc0056e29db9d" X-Spam-Status: No, score=-1.7 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM, HTML_MESSAGE,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] BIP 158 Flexibility and Filter Size X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jun 2018 23:35:44 -0000 --0000000000009f9fc0056e29db9d Content-Type: text/plain; charset="UTF-8" > That in argument against adopting the inferior version, as that will > contribute more momentum to doing it in a way that doesn't make sense long > term. That was moreso an attempt at a disclosure, rather than may argument. But also as noted further up in the thread, both approaches have a trade off: one is better for light clients in a p2p "one honest peer mode", while the other is more compact, but is less verifiable for the light clients. They're "inferior" in different ways. My argument goes more like: moving to prev scripts means clients cannot verify in full unless a block message is added to include the prev outs. This is a downgrade assuming a "one honest peer" model for the p2p interactions. A commitment removes this drawback, but ofc requires a soft fork. Soft forks take a "long" time to deploy. So what's the cost in using the current filter (as it lets the client verify the filter if they want to, or in an attempted "bamboozlement" scenario) in the short term (as we don't yet have a proposal for committing the filters) which would allow us to experiment more with the technique on mainnet before making the step up to committing the filter. Also, depending on the way the commitment is done, the filters themselves would need to be modified. > I don't agree at all, and I can't see why you say so. Sure it doesn't _have_ to, but from my PoV as "adding more commitments" is on the top of every developers wish list for additions to Bitcoin, it would make sense to coordinate on an "ultimate" extensible commitment once, rather than special case a bunch of distinct commitments. I can see arguments for either really. > This is inherent in how e.g. the segwit commitment is encoded, the initial > bytes are an identifying cookies. Different commitments would have different > cookies. Indeed, if the filter were to be committed, using an output on the coinbase would be a likely candidate. However, I see two issues with this: 1. The current filter format (even moving to prevouts) cannot be committed in this fashion as it indexes each of the coinbase output scripts. This creates a circular dependency: the commitment is modified by the filter, which is modified by the commitment (the filter atm indexes the commitment). So we'd need to add a special case to skip outputs with a particular witness magic. However, we don't know what that witness magic looks like (as there's no proposal). As a result, the type filters that can be served over the p2p network may be distinct from the type of filters that are to be committed, as the commitment may have an impact on the filter itself. 2. Since the coinbase transaction is the first in a block, it has the longest merkle proof path. As a result, it may be several hundred bytes (and grows with future capacity increases) to present a proof to the client. Depending on the composition of blocks, this may outweigh the gains had from taking advantage of the additional compression the prev outs allow. In regards to the second item above, what do you think of the old Tier Nolan proposal [1] to create a "constant" sized proof for future commitments by constraining the size of the block and placing the commitments within the last few transactions in the block? > but with an added advantage of permitting expirementation ahead of the > commitment. Indeed! To my knowledge, lnd is the only software deployed that even has code to experiment with the filtering proposal in general. Also, as I pointed out above, we may require an additional modification in order to be able to commit the filter. The nature of that modification may depend on how the filter is to be committed. As a result, why hinder experimentation today (since it might need to be changed anyway, and as you point out the filter being committed can even be swapped) by delaying until we know what the commitment will look like? > You can still scan blocks directly when peers disagree on the filter > content, regardless of how the filter is constructed But the difference is that one options lets you fully construct the filter from a block, while the other requires additional data. > but it makes the attack ineffective and using outpoints considerably increases > bandwidth for everyone without an attack So should we optimize for the ability to validate in a particular model (better security), or lower bandwidth in this case? It may also be the case that the overhead of receiving proofs of the commitment outweigh the savings depending on block composition (ofc entire block that re-uses the same address is super small). > It seems to me this point is being overplayed, especially considering the > current state of non-existing validation in SPV software (if SPV software > doesn't validate anything else they could be validating, why would they > implement a considerable amount of logic for this?). I don't think its fair to compare those that wish to implement this proposal (and actually do the validation) to the legacy SPV software that to my knowledge is all but abandoned. The project I work on that seeks to deploy this proposal (already has, but mainnet support is behind a flag as I anticipated further modifications) indeed has implemented the "considerable" amount of logic to check for discrepancies and ban peers trying to bamboozle the light clients. I'm confident that the other projects seeking to implement this (rust-bitcoin-spv, NBitcoin, bcoin, maybe missing a few too) won't find it too difficult to implement "full" validation, as they're bitcoin developers with quite a bit of experience. I think we've all learned from the past defects of past light clients, and don't seek to repeat history by purposefully implementing as little validation as possible. With these new projects by new authors, I think we have an opprotunity to implement light clients "correctly" this time around. [1]: https://github.com/TierNolan/bips/blob/00a8d3e1ac066ce3728658c6c40240e1c2ab859e/bip-aux-header.mediawiki -- Laolu On Fri, Jun 8, 2018 at 9:14 AM Gregory Maxwell wrote: > On Fri, Jun 8, 2018 at 5:03 AM, Olaoluwa Osuntokun via bitcoin-dev > wrote: > > As someone who's written and reviews code integrating the proposal all > the > > way up the stack (from node to wallet, to application), IMO, there's no > > immediate cost to deferring the inclusion/creation of a filter that > includes > > prev scripts (b) instead of the outpoint as the "regular" filter does > now. > > Switching to prev script in the _short term_ would be costly for the set > of > > applications already deployed (or deployed in a minimal or flag flip > gated > > fashion) as the move from prev script to outpoint is a cascading one that > > impacts wallet operation, rescans, HD seed imports, etc. > > It seems to me that you're making the argument against your own case > here: I'm reading this as a "it's hard to switch so it should be done > the inferior way". That in argument against adopting the inferior > version, as that will contribute more momentum to doing it in a way > that doesn't make sense long term. > > > Such a proposal would need to be generalized enough to allow several > components to be committed, > > I don't agree at all, and I can't see why you say so. > > > likely have versioning, > > This is inherent in how e.g. the segwit commitment is encoded, the > initial bytes are an identifying cookies. Different commitments would > have different cookies. > > > and also provide the necessary extensibility to allow additional items > to be committed in the future > > What was previously proposed is that the commitment be required to be > consistent if present but not be required to be present. This would > allow changing whats used by simply abandoning the old one. Sparsity > in an optional commitment can be addressed when there is less than > 100% participation by having each block that includes a commitment > commit to the missing filters ones from their immediate ancestors. > > Additional optionality can be provided by the other well known > mechanisms, e.g. have the soft fork expire at a block 5 years out > past deployment, and continue to soft-fork it in for a longer term so > long as its in use (or eventually without expiration if its clear that > it's not going away). > > > wallets which wish to primarily use the filters for rescan purposes can't > > just construct them locally for this particular use case independent of > > what's currently deployed on the p2p network. > > Absolutely, but given the failure of BIP37 on the network-- and the > apparent strong preference of end users for alternatives that don't > scan (e.g. electrum and web wallets)-- supporting making this > available via P2P was already only interesting to many as a nearly > free side effect of having filters for local scanning. If it's a > different filter, it's no longer attractive. > > It seems to me that some people have forgotten that this whole idea > was originally proposed to be a committed data-- but with an added > advantage of permitting expirementation ahead of the commitment. > > > Maintaining the outpoint also allows us to rely on a "single honest > peer"security model in the short term. > > You can still scan blocks directly when peers disagree on the filter > content, regardless of how the filter is constructed-- yes, it uses > more bandwidth if you're attacked, but it makes the attack ineffective > and using outpoints considerably increases bandwidth for everyone > without an attack. These ineffective (except for increasing > bandwidth) attacks would have to be common to offset the savings. It > seems to me this point is being overplayed, especially considering the > current state of non-existing validation in SPV software (if SPV > software doesn't validate anything else they could be validating, why > would they implement a considerable amount of logic for this?). > --0000000000009f9fc0056e29db9d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
> That in argument against adopting the inferior v= ersion, as that will
> contribute more momentum to doing it in= a way that doesn't make sense long
> term.

=
That was moreso an attempt at a disclosure, rather than may argu= ment. But
also as noted further up in the thread, both approaches= have a trade off:
one is better for light clients in a p2p "= ;one honest peer mode", while the
other is more compact, but= is less verifiable for the light clients. They're
"infe= rior" in different ways.

My argument goes mor= e like: moving to prev scripts means clients cannot
verify in ful= l unless a block message is added to include the prev outs.
This = is a downgrade assuming a "one honest peer" model for the p2p
interactions. A commitment removes this drawback, but ofc requires a= soft
fork. Soft forks take a "long" time to deploy. So= what's the cost in using
the current filter (as it lets the = client verify the filter if they want to,
or in an attempted &quo= t;bamboozlement" scenario) in the short term (as we don't
yet have a proposal for committing the filters) which would allow us to
experiment more with the technique on mainnet before making the st= ep up to
committing the filter. Also, depending on the way the co= mmitment is done,
the filters themselves would need to be modifie= d.=C2=A0

> I don't agree at all, and I can&= #39;t see why you say so.

Sure it doesn't _hav= e_ to, but from my PoV as "adding more commitments" is
= on the top of every developers wish list for additions to Bitcoin, it would=
make sense to coordinate on an "ultimate" extensible c= ommitment once, rather
than special case a bunch of distinct comm= itments. I can see arguments for
either really.

> This is inherent in how e.g. the segwit commitment is encoded,= the initial
> bytes are an identifying cookies. Different com= mitments would have different
> cookies.

<= div>Indeed, if the filter were to be committed, using an output on the coin= base
would be a likely candidate. However, I see two issues with = this:=C2=A0
=C2=A0=C2=A0
=C2=A0 1. The current filter f= ormat (even moving to prevouts) cannot be committed
=C2=A0 =C2=A0= =C2=A0in this fashion as it indexes each of the coinbase output scripts. T= his
=C2=A0 =C2=A0 =C2=A0creates a circular dependency: the commit= ment is modified by the
=C2=A0 =C2=A0 =C2=A0filter, which is modi= fied by the commitment (the filter atm indexes the
=C2=A0 =C2=A0 = =C2=A0commitment). So we'd need to add a special case to skip outputs w= ith a
=C2=A0 =C2=A0 =C2=A0particular witness magic. However, we d= on't know what that witness
=C2=A0 =C2=A0 =C2=A0magic looks l= ike (as there's no proposal). As a result, the type
=C2=A0 = =C2=A0 =C2=A0filters that can be served over the p2p network may be distinc= t from
=C2=A0 =C2=A0 =C2=A0the type of filters that are to be com= mitted, as the commitment may
=C2=A0 =C2=A0 =C2=A0have an impact = on the filter itself.=C2=A0

=C2=A0 2. Since the co= inbase transaction is the first in a block, it has the
=C2=A0 =C2= =A0 =C2=A0longest merkle proof path. As a result, it may be several hundred= bytes
=C2=A0 =C2=A0 =C2=A0(and grows with future capacity increa= ses) to present a proof to the
=C2=A0 =C2=A0 =C2=A0client. Depend= ing on the composition of blocks, this may outweigh the
=C2=A0 = =C2=A0 =C2=A0gains had from taking advantage of the additional compression = the prev
=C2=A0 =C2=A0 =C2=A0outs allow.

In regards to the second item above, what do you think of the old Tier Nol= an
proposal [1] to create a "constant" sized proof for = future commitments by
constraining the size of the block and plac= ing the commitments within the
last few transactions in the block= ?

> but with an added advantage of permitting e= xpirementation ahead of the
> commitment.

=
Indeed! To my knowledge, lnd is the only software deployed that even h= as
code to experiment with the filtering proposal in general. Als= o, as I
pointed out above, we may require an additional modificat= ion in order to be
able to commit the filter. The nature of that = modification may depend on how
the filter is to be committed. As = a result, why hinder experimentation today
(since it might need t= o be changed anyway, and as you point out the filter
being commit= ted can even be swapped) by delaying until we know what the
commi= tment will look like?

> You can still scan bloc= ks directly when peers disagree on the filter
> content, regar= dless of how the filter is constructed

But the dif= ference is that one options lets you fully construct the filter
f= rom a block, while the other requires additional data.

=
> but it makes the attack ineffective and using outpoints considera= bly increases
> bandwidth for everyone without an attack
=

So should we optimize for the ability to validate in a = particular model (better
security), or lower bandwidth in this ca= se? It may also be the case that the
overhead of receiving proofs= of the commitment outweigh the savings depending
on block compos= ition (ofc entire block that re-uses the same address is super
sm= all).

> It seems to me this point is being over= played, especially considering the
> current state of non-exis= ting validation in SPV software (if SPV software
> doesn't= validate anything else they could be validating, why would they
= > implement a considerable amount of logic for this?).

I don't think its fair to compare those that wish to implement = this proposal
(and actually do the validation) to the legacy SPV = software that to my
knowledge is all but abandoned. The project I= work on that seeks to deploy
this proposal (already has, but mai= nnet support is behind a flag as I
anticipated further modificati= ons) indeed has implemented the "considerable"
amount o= f logic to check for discrepancies and ban peers trying to bamboozle
<= div>the light clients. I'm confident that the other projects seeking to= implement
this (rust-bitcoin-spv, NBitcoin, bcoin, maybe missing= a few too) won't find it
too difficult to implement "fu= ll" validation, as they're bitcoin developers
with quite= a bit of experience.=C2=A0

I think we've all = learned from the past defects of past light clients, and
don'= t seek to repeat history by purposefully implementing as little validation<= /div>
as possible. With these new projects by new authors, I think we h= ave an
opprotunity to implement light clients "correctly&quo= t; this time around.


-- Laolu


On Fri, Jun 8, 2018 at 9:14 AM Gregory Maxwell <greg@xiph.org> wrote:
On Fri, Jun 8, 2018 at 5:03 AM, Olaoluwa Osuntokun via bitcoi= n-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> As someone who's written and reviews code integrating the proposal= all the
> way up the stack (from node to wallet, to application), IMO, there'= ;s no
> immediate cost to deferring the inclusion/creation of a filter that in= cludes
> prev scripts (b) instead of the outpoint as the "regular" fi= lter does now.
> Switching to prev script in the _short term_ would be costly for the s= et of
> applications already deployed (or deployed in a minimal or flag flip g= ated
> fashion) as the move from prev script to outpoint is a cascading one t= hat
> impacts wallet operation, rescans, HD seed imports, etc.

It seems to me that you're making the argument against your own case here: I'm reading this as a "it's hard to switch so it should = be done
the inferior way".=C2=A0 That in argument against adopting the inferio= r
version, as that will contribute more momentum to doing it in a way
that doesn't make sense long term.

> Such a proposal would need to be generalized enough to allow several c= omponents to be committed,

I don't agree at all, and I can't see why you say so.

> likely have versioning,

This is inherent in how e.g. the segwit commitment is encoded, the
initial bytes are an identifying cookies. Different commitments would
have different cookies.

> and also provide the necessary extensibility to allow additional items= to be committed in the future

What was previously proposed is that the commitment be required to be
consistent if present but not be required to be present.=C2=A0 This would allow changing whats used by simply abandoning the old one.=C2=A0 Sparsity<= br> in an optional commitment can be addressed when there is less than
100% participation by having each block that includes a commitment
commit to the missing filters ones from their immediate ancestors.

Additional optionality can be provided by the other well known
mechanisms,=C2=A0 e.g. have the soft fork expire at a block 5 years out
past deployment, and continue to soft-fork it in for a longer term so
long as its in use (or eventually without expiration if its clear that
it's not going away).

> wallets which wish to primarily use the filters for rescan purposes ca= n't
> just construct them locally for this particular use case independent o= f
> what's currently deployed on the p2p network.

Absolutely, but given the failure of BIP37 on the network-- and the
apparent strong preference of end users for alternatives that don't
scan (e.g. electrum and web wallets)-- supporting making this
available via P2P was already only interesting to many as a nearly
free side effect of having filters for local scanning.=C2=A0 If it's a<= br> different filter, it's no longer attractive.

It seems to me that some people have forgotten that this whole idea
was originally proposed to be a committed data-- but with an added
advantage of permitting expirementation ahead of the commitment.

> Maintaining the outpoint also allows us to rely on a "single hone= st peer"security model in the short term.

You can still scan blocks directly when peers disagree on the filter
content, regardless of how the filter is constructed-- yes, it uses
more bandwidth if you're attacked, but it makes the attack ineffective<= br> and using outpoints considerably increases bandwidth for everyone
without an attack.=C2=A0 These ineffective (except for increasing
bandwidth) attacks would have to be common to offset the savings. It
seems to me this point is being overplayed, especially considering the
current state of non-existing validation in SPV software (if SPV
software doesn't validate anything else they could be validating, why would they implement a considerable amount of logic for this?).
--0000000000009f9fc0056e29db9d--