MIME-Version: 1.0
References: <CALFqKjQkQwuxjeYkGWO_Y_HhNQmJgrjqF3m04hbORV7FSbsi3Q@mail.gmail.com>
	<nk8ihWbf71QT6w-wVbnunppF_DjS8ywDoDAugBj5mYM_LCpSzec0j6lkaTKBK4t3CsXwRSXWmzbWiW7nmqT4y0W2fn8X-3oXv-TAYXwP1R4=@protonmail.com>
	<CALFqKjQoA+4XKGePHEK9OAZv2+qg=Q669v=f=MpDtg4F3Fx4kQ@mail.gmail.com>
	<dW426iyRLdy0-CbpWL9pG7dG8qvmyQizPrwuBVHpblJBCBDSMjeIuFMiTjNHOaMfUzjaW2btTiFD9PiozOt9Cv5DQUZG0o22hYndr2wk3SI=@protonmail.com>
	<CALFqKjTfFJATAELn4F0vBGsjorH3a8nzwwtKfXNz_vgWC5XPTg@mail.gmail.com>
	<dNlv6xp0FTd29mczIiw3uOIoDYcTMV44Az0UZ53XQs_8PGzmnMIBhses0kU_SpOZGUcNzWRlerJEZm-p3YQ8USNIxPBouVZdWYCa7HcxDG0=@protonmail.com>
In-Reply-To: <dNlv6xp0FTd29mczIiw3uOIoDYcTMV44Az0UZ53XQs_8PGzmnMIBhses0kU_SpOZGUcNzWRlerJEZm-p3YQ8USNIxPBouVZdWYCa7HcxDG0=@protonmail.com>
From: Mike Brooks <m@ib.tc>
Date: Sun, 28 Jul 2019 20:07:01 -0700
Message-ID: <CALFqKjT=6aYGqTkzqcf2c5dwApPDTTVGuyGt=Vxi6Ms43m-4nQ@mail.gmail.com>
To: ZmnSCPxj <ZmnSCPxj@protonmail.com>, 
	bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>
Content-Type: multipart/alternative; boundary="0000000000003b47e4058ec930f6"
Subject: Re: [bitcoin-dev] PubRef - Script OP Code For Public Data References
Precedence: list

--0000000000003b47e4058ec930f6
Content-Type: text/plain; charset="UTF-8"

ZmnSCPxj,

> Lightning uses a "short channel ID" which is basically an index of block
number + index of transaction + index of output to refer to channels.

Oh wow, this is very similar to the PUBREF proposal. In fact the OP_PUBREF4
operation could be modified to take the tuple: (block number, index of
transaction, index of PUSHDATA) and it would be functionally equivalent.
It looks like the construction of the short channel ID was chosen for the
performance needed to resolve the lookup.

  > The problem with transaction being pruned is that the data in them
might now be used in a *future* `OP_PUBREF`.

I can see how pruning is needed for scalability, and pruning can be made
compatible with a reference to a transaction. If a transaction is pruned,
then the key material used in a prune'ed block's PUSHDATA operations are of
no value.  A user of the network shouldn't need to make this kind of
PUBREF, and if a user did want to bring a wallet back from the dead - then
the utility of PUBREF wouldn't be available to them.

Best Regards,
Mike


On Sun, Jul 28, 2019 at 7:49 PM ZmnSCPxj <ZmnSCPxj@protonmail.com> wrote:

> Good morning Mike,
>
> >  I think that this implication affects other applications built on the
> blockchain, not just the PubRef proposal:
> >
>
> I believe not?
> Current applications use txids to refer to previous transactions, so even
> a short-ranged history rewrite will mostly not affect them --- they can
> just rebroadcast the transactions they are spending and get those
> reconfirmed again.
> There is admittedly a risk of double-spending, but each individual
> application can just spend deeply-confirmed transactions, and tune what it
> considers "deeply-confirmed" depending on how large the value being spent
> is.
> The point is that history rewrites are costly, but if the value being put
> in a `scriptPubKey` that uses `OP_PUBREF` is large enough, it may justify
> the cost of history rewrites --- but if the value is small, the individual
> application (which refers to transactions by their txid anyway) can
> generally assume miners will not bother to history-rewrite.
>
> Since `OP_PUBREF` would be a consensus rule, we need to select a
> "deeply-confirmed" point that is deep enough for *all* cases, unlike
> applications **on top of the blockchain** which can tune their rule of
> "deeply-confirmed" based on value.
> Thus my suggestion to use 100, which we consider "deep enough" to risk
> allowing miners to sell their coins.
>
> Lightning uses a "short channel ID" which is basically an index of block
> number + index of transaction + index of output to refer to channels.
> This is not a problem, however, even in case of short-ranged history
> rewrites.
> The short channel ID is only used for public routing.
> Between the channel counterparties, no security is based on short channel
> ID being stable; it just loses you potential routing fees from the channel
> (and can be fixed by increasing your "deeply-confirmed" enough level before
> you announce the channel for public routing).
>
> >  > There is a potential for a targeted attack where a large payout going
> to a `scriptPubKey` that uses `OP_PUBREF` on a recently-confirmed
> transaction finds that recently-confirmed transaction is replaced with one
> that pays to a different public key, via a history-rewrite attack.
> >  > Such an attack is doable by miners, and if we consider that we accept
> 100 blocks for miner coinbase maturity as "acceptably low risk" against
> miner shenanigans, then we might consider that 100 blocks might be
> acceptable for this also.
> >  > Whether 100 is too high or not largely depends on your risk appetite.
> >
> > I agree 100% this attack is unexpected and very interesting.
>
> It is precisely because of this possibility that we tend to avoid making
> SCRIPT validity dependent on anything that is not in the transaction.
> We would have to re-evaluate the SCRIPT every time there is a chain tip
> reorganization (increasing validation CPU load), unless we do something
> like "only allow `OP_PUBREF` to data that is more than 100 blocks
> confirmed".
>
> >  However, I find the arbitrary '100' to be unsatisfying - I'll have to
> do some more digging. It would be interesting to trigger this on the
> testnet to see what happens.  Do you know if anyone has pushed these
> limits?  I am so taken by this attack I might attempt it.
> >
> >  > Data derived from > 220Gb of perpetually-growing blockchain is
> hardly, to my mind, "only needs an array".
> >
> > There are other open source projects that have to deal with larger data
> sets and have accounted for the real-world limits on computability. Apache
> HTTPD's Bucket-Brigade comes to mind, which has been well tested and can
> account for limited RAM when accessing linear data structures. For a more
> general purpose utility leveldb (bsd-license) provides random access to
> arbitrary data collections.
>
> Which is the point: we need to use something, the details need to be
> considered during implementation, implementation details may leak in the
> effective spec (e.g. DER-encoding), etc.
>
> >  Pruning can also be a real asset for PubRef.  If all transactions for a
> wallet have been pruned, then there is no need to index this PubRef - a
> validator can safely skip over it.
>
> What?
> The problem with transaction being pruned is that the data in them might
> now be used in a *future* `OP_PUBREF`.
>
> Further, pruned nodes are still full validators --- transactions may be
> pruned, but the pruned node will ***still*** validate any `OP_PUBREF` it
> uses, because it is still a full validator, it just does not archive old
> blocks in local storage.
>
> Regards,
> ZmnSCPxj
>

--0000000000003b47e4058ec930f6
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><br></div><div>ZmnSCPxj,</div><div>=C2=A0=C2=A0</div>=
<div>&gt; Lightning uses a &quot;short channel ID&quot; which is basically =
an index of block number + index of transaction + index of output to refer =
to channels.<br></div><div><br></div><div>Oh wow, this is very similar to t=
he PUBREF proposal. In fact the OP_PUBREF4 operation could be modified to t=
ake the tuple: (block number, index of transaction, index of PUSHDATA) and =
it would be functionally equivalent.=C2=A0 It looks like the construction o=
f the short channel ID was chosen for the performance needed to resolve the=
 lookup.</div><div><br></div>=C2=A0 &gt; The problem with transaction being=
 pruned is that the data in them might now be used in a *future* `OP_PUBREF=
`.<br><div><br></div><div>I can see how pruning is needed for scalability, =
and pruning=C2=A0can be made compatible with a reference to a transaction. =
If a transaction is pruned, then the key material used in a prune&#39;ed bl=
ock&#39;s PUSHDATA operations are of no value.=C2=A0 A user of the network =
shouldn&#39;t need to make this kind of PUBREF, and if a user did want to b=
ring a wallet back from the dead - then the utility of PUBREF wouldn&#39;t =
be available to them.</div><div><br></div><div>Best Regards,</div><div>Mike=
</div><div><br></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" =
class=3D"gmail_attr">On Sun, Jul 28, 2019 at 7:49 PM ZmnSCPxj &lt;<a href=
=3D"mailto:ZmnSCPxj@protonmail.com">ZmnSCPxj@protonmail.com</a>&gt; wrote:<=
br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8e=
x;border-left:1px solid rgb(204,204,204);padding-left:1ex">Good morning Mik=
e,<br>
<br>
&gt; =C2=A0I think that this implication affects other applications built o=
n the blockchain, not just the PubRef proposal:<br>
&gt;<br>
<br>
I believe not?<br>
Current applications use txids to refer to previous transactions, so even a=
 short-ranged history rewrite will mostly not affect them --- they can just=
 rebroadcast the transactions they are spending and get those reconfirmed a=
gain.<br>
There is admittedly a risk of double-spending, but each individual applicat=
ion can just spend deeply-confirmed transactions, and tune what it consider=
s &quot;deeply-confirmed&quot; depending on how large the value being spent=
 is.<br>
The point is that history rewrites are costly, but if the value being put i=
n a `scriptPubKey` that uses `OP_PUBREF` is large enough, it may justify th=
e cost of history rewrites --- but if the value is small, the individual ap=
plication (which refers to transactions by their txid anyway) can generally=
 assume miners will not bother to history-rewrite.<br>
<br>
Since `OP_PUBREF` would be a consensus rule, we need to select a &quot;deep=
ly-confirmed&quot; point that is deep enough for *all* cases, unlike applic=
ations **on top of the blockchain** which can tune their rule of &quot;deep=
ly-confirmed&quot; based on value.<br>
Thus my suggestion to use 100, which we consider &quot;deep enough&quot; to=
 risk allowing miners to sell their coins.<br>
<br>
Lightning uses a &quot;short channel ID&quot; which is basically an index o=
f block number + index of transaction + index of output to refer to channel=
s.<br>
This is not a problem, however, even in case of short-ranged history rewrit=
es.<br>
The short channel ID is only used for public routing.<br>
Between the channel counterparties, no security is based on short channel I=
D being stable; it just loses you potential routing fees from the channel (=
and can be fixed by increasing your &quot;deeply-confirmed&quot; enough lev=
el before you announce the channel for public routing).<br>
<br>
&gt; =C2=A0&gt; There is a potential for a targeted attack where a large pa=
yout going to a `scriptPubKey` that uses `OP_PUBREF` on a recently-confirme=
d transaction finds that recently-confirmed transaction is replaced with on=
e that pays to a different public key, via a history-rewrite attack.<br>
&gt; =C2=A0&gt; Such an attack is doable by miners, and if we consider that=
 we accept 100 blocks for miner coinbase maturity as &quot;acceptably low r=
isk&quot; against miner shenanigans, then we might consider that 100 blocks=
 might be acceptable for this also.<br>
&gt; =C2=A0&gt; Whether 100 is too high or not largely depends on your risk=
 appetite.<br>
&gt;<br>
&gt; I agree 100% this attack is unexpected and very interesting.<br>
<br>
It is precisely because of this possibility that we tend to avoid making SC=
RIPT validity dependent on anything that is not in the transaction.<br>
We would have to re-evaluate the SCRIPT every time there is a chain tip reo=
rganization (increasing validation CPU load), unless we do something like &=
quot;only allow `OP_PUBREF` to data that is more than 100 blocks confirmed&=
quot;.<br>
<br>
&gt;=C2=A0 However, I find the arbitrary &#39;100&#39; to be unsatisfying -=
 I&#39;ll have to do some more digging. It would be interesting to trigger =
this on the testnet to see what happens.=C2=A0 Do you know if anyone has pu=
shed these limits?=C2=A0 I am so taken by this attack I might attempt it.<b=
r>
&gt;<br>
&gt; =C2=A0&gt; Data derived from &gt; 220Gb of perpetually-growing blockch=
ain is hardly, to my mind, &quot;only needs an array&quot;.<br>
&gt;<br>
&gt; There are other open source projects that have to deal with larger dat=
a sets and have accounted for the real-world limits on computability. Apach=
e HTTPD&#39;s Bucket-Brigade comes to mind, which has been well tested and =
can account for limited RAM when accessing linear data structures. For a mo=
re general purpose utility leveldb (bsd-license) provides random access to =
arbitrary data collections.<br>
<br>
Which is the point: we need to use something, the details need to be consid=
ered during implementation, implementation details may leak in the effectiv=
e spec (e.g. DER-encoding), etc.<br>
<br>
&gt;=C2=A0 Pruning can also be a real asset for PubRef.=C2=A0 If all transa=
ctions for a wallet have been pruned, then there is no need to index this P=
ubRef - a validator can safely skip over it.<br>
<br>
What?<br>
The problem with transaction being pruned is that the data in them might no=
w be used in a *future* `OP_PUBREF`.<br>
<br>
Further, pruned nodes are still full validators --- transactions may be pru=
ned, but the pruned node will ***still*** validate any `OP_PUBREF` it uses,=
 because it is still a full validator, it just does not archive old blocks =
in local storage.<br>
<br>
Regards,<br>
ZmnSCPxj<br>
</blockquote></div>

--0000000000003b47e4058ec930f6--