Received: from sog-mx-4.v43.ch3.sourceforge.com ([172.29.43.194] helo=mx.sourceforge.net) by sfs-ml-1.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1YsYWZ-0008LZ-12 for bitcoin-development@lists.sourceforge.net; Wed, 13 May 2015 15:24:43 +0000 Received-SPF: pass (sog-mx-4.v43.ch3.sourceforge.com: domain of gmail.com designates 209.85.215.48 as permitted sender) client-ip=209.85.215.48; envelope-from=decker.christian@gmail.com; helo=mail-la0-f48.google.com; Received: from mail-la0-f48.google.com ([209.85.215.48]) by sog-mx-4.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1YsYWX-0003CL-5M for bitcoin-development@lists.sourceforge.net; Wed, 13 May 2015 15:24:42 +0000 Received: by layy10 with SMTP id y10so32080373lay.0 for ; Wed, 13 May 2015 08:24:34 -0700 (PDT) X-Received: by 10.152.44.225 with SMTP id h1mr16366496lam.5.1431530674751; Wed, 13 May 2015 08:24:34 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Christian Decker Date: Wed, 13 May 2015 15:24:34 +0000 Message-ID: To: Gavin Andresen , Tier Nolan Content-Type: multipart/alternative; boundary=089e0160b7be5a90f60515f835a5 X-Spam-Score: -0.6 (/) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (decker.christian[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record 1.0 HTML_MESSAGE BODY: HTML included in message -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1YsYWX-0003CL-5M Cc: Bitcoin Development Subject: Re: [Bitcoin-development] [BIP] Normalized Transaction IDs X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 May 2015 15:24:43 -0000 --089e0160b7be5a90f60515f835a5 Content-Type: text/plain; charset=UTF-8 Glad you like it, I was afraid that I missed something obvious :-) The points the two of you raised are valid and I will address them as soon as possible. I certainly will implement this proposal so that it becomes more concrete, but my C++ is a bit rusty and it'll take some time, so I wanted to gauge interest first. > This has the effect of doubling the size of the UTXO database. At minimum, there needs to be a legacy txid to normalized txid map in the database. > > An addition to the BIP would eliminate the need for the 2nd index. You could require a SPV proof of the spending transaction to be included with legacy transactions. This would allow clients to verify that the normalized txid matched the legacy id. > >The OutPoint would be {LegacyId | SPV Proof to spending tx | spending tx | index}. This allows a legacy transaction to be upgraded. OutPoints which use a normalized txid don't need the SPV proof. It does and I should have mentioned it in the draft, according to my calculations a mapping legacy ID -> normalized ID is about 256 MB in size, or at least it was at height 330'000, things might have changed a bit and I'll recompute that. I omitted the deprecation of legacy IDs on purpose since we don't know whether we will migrate completely or leave keep both options viable. > I think this needs more details before it gets a BIP number; for example, which opcodes does this affect, and how, exactly, does it affect them? Is the merkle root in the block header computed using normalized transaction ids or normalized ids? I think both IDs can be used in the merkle tree, since we lookup an ID in both indices we can use both to address them and we will find them either way. As for the opcodes I'll have to check, but I currently don't see how they could be affected. The OP_*SIG* codes calculate their own (more complicated) stripped transaction before hashing and checking the signature. The input of the stripped transaction simply contains whatever hash was used to reference the output, so we do not replace IDs during the operation. The stripped format used by OP_*SIG* operations does not have to adhere to the hashes used to reference a transaction in the input. > I think there might actually be two or three or four BIPs here: > > + Overall "what is trying to be accomplished" > + Changes to the OP_*SIG* opcodes > + Changes to the bloom-filtering SPV support > + ...eventually, hard fork rollout plan > > I also think that it is a good idea to have actually implemented a proposal before getting a BIP number. At least, I find that actually writing the code often turns up issues I hadn't considered when thinking about the problem at a high level. And I STRONGLY believe BIPs should be descriptive ("here is how this thing works") not proscriptive ("here's how I think we should all do it"). We can certainly split the proposal should it get too large, for now it seems manageable, since opcodes are not affected. Bloom-filtering is resolved by adding the normalized transaction IDs and checking for both IDs in the filter. Since you mention bundling the change with other changes that require a hard-fork it might be a good idea to build a separate proposal for a generic hard-fork rollout mechanism. If there are no obvious roadblocks and the change seems generally a good thing I will implement it in Bitcoin Core :-) Regards, Chris On Wed, May 13, 2015 at 3:44 PM Gavin Andresen wrote: > I think this needs more details before it gets a BIP number; for example, > which opcodes does this affect, and how, exactly, does it affect them? Is > the merkle root in the block header computed using normalized transaction > ids or normalized ids? > > I think there might actually be two or three or four BIPs here: > > + Overall "what is trying to be accomplished" > + Changes to the OP_*SIG* opcodes > + Changes to the bloom-filtering SPV support > + ...eventually, hard fork rollout plan > > I also think that it is a good idea to have actually implemented a > proposal before getting a BIP number. At least, I find that actually > writing the code often turns up issues I hadn't considered when thinking > about the problem at a high level. And I STRONGLY believe BIPs should be > descriptive ("here is how this thing works") not proscriptive ("here's how > I think we should all do it"). > > Finally: I like the idea of moving to a normalized txid. But it might make > sense to bundle that change with a bigger change to OP_CHECKSIG; see Greg > Maxwell's excellent talk about his current thoughts on that topic: > https://www.youtube.com/watch?v=Gs9lJTRZCDc > > > On Wed, May 13, 2015 at 9:12 AM, Tier Nolan wrote: > >> I think this is a good way to handle things, but as you say, it is a hard >> fork. >> >> CHECKLOCKTIMEVERIFY covers many of the use cases, but it would be nice to >> fix malleability once and for all. >> >> This has the effect of doubling the size of the UTXO database. At >> minimum, there needs to be a legacy txid to normalized txid map in the >> database. >> >> An addition to the BIP would eliminate the need for the 2nd index. You >> could require a SPV proof of the spending transaction to be included with >> legacy transactions. This would allow clients to verify that the >> normalized txid matched the legacy id. >> >> The OutPoint would be {LegacyId | SPV Proof to spending tx | spending tx >> | index}. This allows a legacy transaction to be upgraded. OutPoints >> which use a normalized txid don't need the SPV proof. >> >> The hard fork would be followed by a transitional period, in which both >> txids could be used. Afterwards, legacy transactions have to have the SPV >> proof added. This means that old transactions with locktimes years in the >> future can be upgraded for spending, without nodes needing to maintain two >> indexes. >> >> >> ------------------------------------------------------------------------------ >> One dashboard for servers and applications across Physical-Virtual-Cloud >> Widest out-of-the-box monitoring support with 50+ applications >> Performance metrics, stats and reports that give you Actionable Insights >> Deep dive visibility with transaction tracing using APM Insight. >> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y >> _______________________________________________ >> Bitcoin-development mailing list >> Bitcoin-development@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >> >> > > > -- > -- > Gavin Andresen > > ------------------------------------------------------------------------------ > One dashboard for servers and applications across Physical-Virtual-Cloud > Widest out-of-the-box monitoring support with 50+ applications > Performance metrics, stats and reports that give you Actionable Insights > Deep dive visibility with transaction tracing using APM Insight. > http://ad.doubleclick.net/ddm/clk/290420510;117567292;y > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > --089e0160b7be5a90f60515f835a5 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Glad you like it, I was afraid that I missed something obv= ious :-)

The points the two of you raised are valid and I will addre= ss them as soon as possible. I certainly will implement this proposal so th= at it becomes more concrete, but my C++ is a bit rusty and it'll take s= ome time, so I wanted to gauge interest first.

>=C2=A0This has the effect of doubling the size of the UTXO dat= abase.=C2=A0 At minimum, there needs to be a legacy txid to normalized txid= map in the database.
>
> An addition to the BIP wou= ld eliminate the need for the 2nd index.=C2=A0 You could require a SPV proo= f of the spending transaction to be included with legacy transactions.=C2= =A0 This would allow clients to verify that the normalized txid matched the= legacy id.
>
>The OutPoint would be {LegacyId | SPV Proof to= spending tx=C2=A0 | spending tx | index}.=C2=A0 This allows a legacy trans= action to be upgraded.=C2=A0 OutPoints which use a normalized txid don'= t need the SPV proof.

It does and I s= hould have mentioned it in the draft, according to my calculations a mappin= g legacy ID -> normalized ID is about 256 MB in size, or at least it was= at height 330'000, things might have changed a bit and I'll recomp= ute that. I omitted the deprecation of legacy IDs on purpose since we don&#= 39;t know whether we will migrate completely or leave keep both options via= ble.

> I think this= needs more details before it gets a BIP number; for example, which opcodes= does this affect, and how, exactly, does it affect them? Is the merkle roo= t in the block header computed using normalized transaction ids or normaliz= ed ids?

I think= both IDs can be used in the merkle tree, since we lookup an ID in both ind= ices we can use both to address them and we will find them either way.

As for the opcodes I'll have to check, but I curre= ntly don't see how they could be affected. The OP_*SIG* codes calculate= their own=C2=A0(more complicated)=C2=A0stripped transaction before hashing and c= hecking the signature. The input of the stripped transaction simply contain= s whatever hash was used to reference the output, so we do not replace IDs = during the operation. The stripped format used by OP_*SIG* operations does = not have to adhere to the hashes used to reference a transaction in the inp= ut.

> I think there might actually be tw= o or three or four BIPs here:
>
> =C2=A0+ Overall "wha= t is trying to be accomplished"
>=C2=A0=C2=A0+= Changes to the OP_*SIG* opcodes
>=C2=A0=C2=A0+ Cha= nges to the bloom-filtering SPV support
>=C2=A0=C2= =A0+ ...eventually, hard fork rollout plan
>=C2=A0
= >=C2=A0I also think that it is a good idea to have actua= lly implemented a proposal before getting a BIP number. At least, I find th= at actually writing the code often turns up issues I hadn't considered = when thinking about the problem at a high level. And I STRONGLY believe BIP= s should be descriptive ("here is how this thing works") not pros= criptive ("here's how I think we should all do it").

We can certainly split the proposal should it get t= oo large, for now it seems manageable, since opcodes are not affected. Bloo= m-filtering is resolved by adding the normalized transaction IDs and checki= ng for both IDs in the filter. Since you mention bundling the change with o= ther changes that require a hard-fork it might be a good idea to build a se= parate proposal for a generic hard-fork rollout mechanism.

=
= If there are no obvious roadblocks and the change seems generally a good th= ing I will implement it in Bitcoin Core :-)

Regards,
Ch= ris

On Wed, May 13, 2015 at 3:44 = PM Gavin Andresen <gavinandresen@gmail.com> wrote:
I think this needs more details before it gets a BIP= number; for example, which opcodes does this affect, and how, exactly, doe= s it affect them? Is the merkle root in the block header computed using nor= malized transaction ids or normalized ids?=C2=A0

I think= there might actually be two or three or four BIPs here:

=C2=A0+ Overall "what is trying to be accomplished"
<= div>=C2=A0+ Changes to the OP_*SIG* opcodes
=C2=A0+ Changes to th= e bloom-filtering SPV support
=C2=A0+ ...eventually, hard fork ro= llout plan

I also think that it is a good idea to have a= ctually implemented a proposal before getting a BIP number. At least, I fin= d that actually writing the code often turns up issues I hadn't conside= red when thinking about the problem at a high level. And I STRONGLY believe= BIPs should be descriptive ("here is how this thing works") not = proscriptive ("here's how I think we should all do it").

Finally: I like the idea of moving to a normaliz= ed txid. But it might make sense to bundle that change with a bigger change= to OP_CHECKSIG; see Greg Maxwell's excellent talk about his current th= oughts on that topic:


On Wed, May 1= 3, 2015 at 9:12 AM, Tier Nolan <tier.nolan@gmail.com> wro= te:
I think this is= a good way to handle things, but as you say, it is a hard fork.

CHECKLOCKTIMEVERIFY covers many of the use cases, but it would be nice t= o fix malleability once and for all.

This has the effect o= f doubling the size of the UTXO database.=C2=A0 At minimum, there needs to = be a legacy txid to normalized txid map in the database.

= An addition to the BIP would eliminate the need for the 2nd index.=C2=A0 Yo= u could require a SPV proof of the spending transaction to be included with= legacy transactions.=C2=A0 This would allow clients to verify that the nor= malized txid matched the legacy id.

The OutPoint would be= {LegacyId | SPV Proof to spending tx=C2=A0 | spending tx | index}.=C2=A0 T= his allows a legacy transaction to be upgraded.=C2=A0 OutPoints which use a= normalized txid don't need the SPV proof.

The hard f= ork would be followed by a transitional period, in which both txids could b= e used.=C2=A0 Afterwards, legacy transactions have to have the SPV proof ad= ded.=C2=A0 This means that old transactions with locktimes years in the fut= ure can be upgraded for spending, without nodes needing to maintain two ind= exes.

-----------------------------------------------------------------------= -------
One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
= _______________________________________________
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-de= velopment




--
--
Gavin Andresen
---------------------------------------------------------------------------= ---
One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y____= ___________________________________________
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-de= velopment
--089e0160b7be5a90f60515f835a5--