diff options
| author | Jim Posen <jim.posen@gmail.com> | 2018-06-10 16:07:07 -0700 |
|---|---|---|
| committer | bitcoindev <bitcoindev@gnusha.org> | 2018-06-10 23:07:23 +0000 |
| commit | 3b1854152607bfacefdaf03ab1db0b744bf4b22f (patch) | |
| tree | 634f3899e4ec9ebce73ceee41cb7067b4894b2ca | |
| parent | 63e3f858124acb0e06a17db8f9699071828809ac (diff) | |
| download | pi-bitcoindev-3b1854152607bfacefdaf03ab1db0b744bf4b22f.tar.gz pi-bitcoindev-3b1854152607bfacefdaf03ab1db0b744bf4b22f.zip | |
Re: [bitcoin-dev] UHS: Full-node security without maintaining a full UTXO set
| -rw-r--r-- | fe/12af2cd7d85877e48878916badc9fdf50af219 | 276 |
1 files changed, 276 insertions, 0 deletions
diff --git a/fe/12af2cd7d85877e48878916badc9fdf50af219 b/fe/12af2cd7d85877e48878916badc9fdf50af219 new file mode 100644 index 000000000..ae3c60b9a --- /dev/null +++ b/fe/12af2cd7d85877e48878916badc9fdf50af219 @@ -0,0 +1,276 @@ +Return-Path: <jim.posen@gmail.com> +Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org + [172.17.192.35]) + by mail.linuxfoundation.org (Postfix) with ESMTPS id 7D9CD147A + for <bitcoin-dev@lists.linuxfoundation.org>; + Sun, 10 Jun 2018 23:07:23 +0000 (UTC) +X-Greylist: whitelisted by SQLgrey-1.7.6 +Received: from mail-qt0-f173.google.com (mail-qt0-f173.google.com + [209.85.216.173]) + by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 6CAAC762 + for <bitcoin-dev@lists.linuxfoundation.org>; + Sun, 10 Jun 2018 23:07:21 +0000 (UTC) +Received: by mail-qt0-f173.google.com with SMTP id y20-v6so18696666qto.8 + for <bitcoin-dev@lists.linuxfoundation.org>; + Sun, 10 Jun 2018 16:07:21 -0700 (PDT) +DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; + h=mime-version:references:in-reply-to:from:date:message-id:subject:to; + bh=F4gyOt0UlKhKYKAMuiC1zcX7GtcCviMPsAvEsUL1c/k=; + b=BfDdks37QizRaLNO2O28lXcxMj558kkI5I2zyTsvspZV/AfIih/GTAESJXsKh2XK7K + OIv20F0j4Vy0BVaom+zC3H5XkShRaZ98jXlClYFrOQ4Ffyj8e65VxbVQllb3MsKaykKl + zFyHkrKTPSjoBJoa/mTPfBvkRtToTSNwEZh/HbUQKEcqLYiowMN5G3DJ0lcZ19BnlQZc + 82tvviIAZnfFUCUJJz4cebwzXwRnABi2caU2I5OuTVJ1mGr7VkvvTfQmM35nLpZPBELs + Ya3STCu+mjMzQggIJSOsA3tOq3xPtqZhMCM3T+hUvjoL5C4zDYn66T9Jk4z7Y/uBtOdH + RNtQ== +X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; + d=1e100.net; s=20161025; + h=x-gm-message-state:mime-version:references:in-reply-to:from:date + :message-id:subject:to; + bh=F4gyOt0UlKhKYKAMuiC1zcX7GtcCviMPsAvEsUL1c/k=; + b=SFyDqcEP+ncJy+XllRwzN4CNMyScqzUsi3rXPJqRz0bpQ3KoO3jX6S3FLR+4q2IJdk + rlUCMQSFDjOaykBgHlgQ2yKqspIC2QrMo9g8GrUx3/+UOwSqMtOWUsUyUkdmCKqjl2U1 + hJXg6xQVmlgPf8Voy7wW1Cik5zovfm7pZimBG57X/KA8OCoNJ8GhRCJ30m68kRnY5iNS + P7DKAtVMQa8le4ePMaPxN+lnJBqcDxBnMXhEzwFYV94IscPHKnkHv7uEwI31Oo+Ekgcc + /cKz3V/SzLPvfW9cMjSEe0Bv5kuLmHvibp/adqVymgVthZHvf1ChryGimB78hkTdDtQ+ + jcMQ== +X-Gm-Message-State: APt69E3ZcwXLyzwlfY/NZfZhids8vRG4fx1gLbaiEb1bulgJjWT3uGrb + gmw8jdaM9x3LM6/tSPZp0Pgg6EWZOks+4xaOwnBwdw== +X-Google-Smtp-Source: ADUXVKI23umF2WChjKsQUWTNnW5VQ5pRVYTrhekQB5c/TuexApYFFuDVAw4mfue5sm+6hvhi8G8JwtWynGAtQIAkG+U= +X-Received: by 2002:ac8:2836:: with SMTP id + 51-v6mr14332101qtq.131.1528672040252; + Sun, 10 Jun 2018 16:07:20 -0700 (PDT) +MIME-Version: 1.0 +References: <CAApLimjfPKDxmiy_SHjuOKbfm6HumFPjc9EFKvw=3NwZO8JcmQ@mail.gmail.com> + <CAAS2fgTHTK8Dve9xHW9yULa1yObWtmwmeKKcD_BMjON=RAw8Sg@mail.gmail.com> + <E5BD6DC6-281B-46E5-ABD3-71B2D5549902@sprovoost.nl> +In-Reply-To: <E5BD6DC6-281B-46E5-ABD3-71B2D5549902@sprovoost.nl> +From: Jim Posen <jim.posen@gmail.com> +Date: Sun, 10 Jun 2018 16:07:07 -0700 +Message-ID: <CADZtCSiPkah3sX5yw_29mcnFxFm_SrgzFxOuaQwffMHhRDaobg@mail.gmail.com> +To: Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org> +Content-Type: multipart/alternative; boundary="000000000000f00327056e51b1ee" +X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, + DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, HTML_MESSAGE, + RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 +X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on + smtp1.linux-foundation.org +X-Mailman-Approved-At: Sun, 10 Jun 2018 23:15:36 +0000 +Subject: Re: [bitcoin-dev] UHS: Full-node security without maintaining a + full UTXO set +X-BeenThere: bitcoin-dev@lists.linuxfoundation.org +X-Mailman-Version: 2.1.12 +Precedence: list +List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org> +List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>, + <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe> +List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/> +List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org> +List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help> +List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>, + <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe> +X-List-Received-Date: Sun, 10 Jun 2018 23:07:23 -0000 + +--000000000000f00327056e51b1ee +Content-Type: text/plain; charset="UTF-8" +Content-Transfer-Encoding: quoted-printable + +I generally like the direction of this proposal in terms of allowing full +nodes to run with a different storage/bandwidth tradeoff. Cory, were this +implemented, would you expect Core to support both operating modes (full +UTXO set and UHS) depending on user configuration, or would UHS be +mandatory? + +Also, given that Bram Cohen's TXO bitfield proposal was an inspiration for +this, could you comment on why the UHS is preferable to that approach? An +alternative that goes even further in the direction of more bandwidth, less +storage, would be for nodes to simply maintain a Merkle Mountain Range over +all TXOs in order of creation and a spentness bitfield. Blocks could be +requested with the prev outputs and a Merkle proof linking them into the +MMR root. Since the Merkle proof is deterministic, it could be computed by +archive nodes and miners and saved alongside the block data for relay. +Another benefit of this is the TXO MMR root may be independently useful if +committed into the coinbase transaction. + +On Thu, Jun 7, 2018 at 7:02 AM Sjors Provoost via bitcoin-dev < +bitcoin-dev@lists.linuxfoundation.org> wrote: + +> eMMC storage, which low end devices often use, come in 2x increments. +> Running a pruned full node on 8 GB is difficult if not impossible (the UT= +XO +> set peaked at 3.5 GB in January, but a full node stores additional stuff)= +. +> +> However, 16 GB is only =E2=82=AC10 more expensive and presumably standard= + by the +> time this would be rolled out. +> +> On AWS every GB of SSD storage avoided saves $1 per year, not end of the +> world stuff, but not negligible either. Outbound traffic costs $0.10 / GB +> (ignoring free allowance), so when uploading 200 GB per year, the 5% woul= +d +> offset $1 of storage cost savings. +> +> The above seems marginal, probably not worth it unless there=E2=80=99s re= +ally no +> downside. +> +> What I find attractive about this proposal is the ability to squeeze more +> out of limited RAM (typically only 1 or 2 GB on these low end devices). I= +=E2=80=99d +> have to test Cory=E2=80=99s branch to see if that actually matters in pra= +ctice. +> +> It=E2=80=99s also useful to distinguish benefits during initial sync from= + ongoing +> operation. The former I=E2=80=99ve almost given up on for low end device= +s (can +> take weeks), in favor of doing it on a faster computer and copying the +> result. The latter needs far less RAM, so perhaps this proposal doesn=E2= +=80=99t +> help much there, but that would be useful to measure. +> +> Did you try the recent SHA256 optimizations on your branch? +> +> Sjors +> +> > Op 17 mei 2018, om 18:56 heeft Gregory Maxwell via bitcoin-dev < +> bitcoin-dev@lists.linuxfoundation.org> het volgende geschreven: +> > +> > On Wed, May 16, 2018 at 4:36 PM, Cory Fields via bitcoin-dev +> > <bitcoin-dev@lists.linuxfoundation.org> wrote: +> >> Tl;dr: Rather than storing all unspent outputs, store their hashes. +> > +> > My initial thoughts are it's not _completely_ obvious to me that a 5% +> > ongoing bandwidth increase is actually a win to get something like a +> > 40% reduction in the size of a pruned node (and less than a 1% +> > reduction in an archive node) primarily because I've not seen size of +> > a pruned node cited as a usage limiting factor basically anywhere. I +> > would assume it is a win but wouldn't be shocked to see a careful +> > analysis that concluded it wasn't. +> > +> > But perhaps more interestingly, I think the overhead is not really 5%, +> > but it's 5% measured in the context of the phenomenally inefficient tx +> > mechanisms ( https://bitcointalk.org/index.php?topic=3D1377345.0 ). +> > Napkin math on the size of a txn alone tells me it's more like a 25% +> > increase if you just consider size of tx vs size of +> > tx+scriptpubkeys,amounts. If I'm not missing something there, I think +> > that would get in into a very clear not-win range. +> > +> > On the positive side is that it doesn't change the blockchain +> > datastructure, so it's something implementations could do without +> > marrying the network to it forever. +> > +> +> _______________________________________________ +> bitcoin-dev mailing list +> bitcoin-dev@lists.linuxfoundation.org +> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev +> + +--000000000000f00327056e51b1ee +Content-Type: text/html; charset="UTF-8" +Content-Transfer-Encoding: quoted-printable + +<div dir=3D"ltr">I generally like the direction of this proposal in terms o= +f allowing full nodes to run with a different storage/bandwidth tradeoff. C= +ory, were this implemented, would you expect Core to support both operating= + modes (full UTXO set and UHS) depending on user configuration, or would UH= +S be mandatory?<div><br></div><div>Also, given that Bram Cohen's TXO bi= +tfield proposal was an inspiration for this, could you comment on why the U= +HS is preferable to that approach? An alternative that goes even further in= + the direction of more bandwidth, less storage, would be for nodes to simpl= +y maintain a Merkle Mountain Range over all TXOs in order of creation and a= + spentness bitfield. Blocks could be requested with the prev outputs and a = +Merkle proof linking them into the MMR root. Since the Merkle proof is dete= +rministic, it could be computed by archive nodes and miners and saved along= +side the block data for relay. Another benefit of this is the TXO MMR root = +may be independently useful if committed into the coinbase transaction.</di= +v></div><br><div class=3D"gmail_quote"><div dir=3D"ltr">On Thu, Jun 7, 2018= + at 7:02 AM Sjors Provoost via bitcoin-dev <<a href=3D"mailto:bitcoin-de= +v@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a>> = +wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8e= +x;border-left:1px #ccc solid;padding-left:1ex">eMMC storage, which low end = +devices often use, come in 2x increments. Running a pruned full node on 8 G= +B is difficult if not impossible (the UTXO set peaked at 3.5 GB in January,= + but a full node stores additional stuff).<br> +<br> +However, 16 GB is only =E2=82=AC10 more expensive and presumably standard b= +y the time this would be rolled out.<br> +<br> +On AWS every GB of SSD storage avoided saves $1 per year, not end of the wo= +rld stuff, but not negligible either. Outbound traffic costs $0.10 / GB (ig= +noring free allowance), so when uploading 200 GB per year, the 5% would off= +set $1 of storage cost savings.<br> +<br> +The above seems marginal, probably not worth it unless there=E2=80=99s real= +ly no downside.<br> +<br> +What I find attractive about this proposal is the ability to squeeze more o= +ut of limited RAM (typically only 1 or 2 GB on these low end devices). I=E2= +=80=99d have to test Cory=E2=80=99s branch to see if that actually matters = +in practice.<br> +<br> +It=E2=80=99s also useful to distinguish benefits during initial sync from o= +ngoing operation. The former I=E2=80=99ve almost given up on for=C2=A0 low = +end devices (can take weeks), in favor of doing it on a faster computer and= + copying the result. The latter needs far less RAM, so perhaps this proposa= +l doesn=E2=80=99t help much there, but that would be useful to measure.<br> +<br> +Did you try the recent SHA256 optimizations on your branch?<br> +<br> +Sjors<br> +<br> +> Op 17 mei 2018, om 18:56 heeft Gregory Maxwell via bitcoin-dev <<a = +href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">bit= +coin-dev@lists.linuxfoundation.org</a>> het volgende geschreven:<br> +> <br> +> On Wed, May 16, 2018 at 4:36 PM, Cory Fields via bitcoin-dev<br> +> <<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D= +"_blank">bitcoin-dev@lists.linuxfoundation.org</a>> wrote:<br> +>> Tl;dr: Rather than storing all unspent outputs, store their hashes= +.<br> +> <br> +> My initial thoughts are it's not _completely_ obvious to me that a= + 5%<br> +> ongoing bandwidth increase is actually a win to get something like a<b= +r> +> 40% reduction in the size of a pruned node (and less than a 1%<br> +> reduction in an archive node) primarily because I've not seen size= + of<br> +> a pruned node cited as a usage limiting factor basically anywhere. I<b= +r> +> would assume it is a win but wouldn't be shocked to see a careful<= +br> +> analysis that concluded it wasn't.<br> +> <br> +> But perhaps more interestingly, I think the overhead is not really 5%,= +<br> +> but it's 5% measured in the context of the phenomenally inefficien= +t tx<br> +> mechanisms ( <a href=3D"https://bitcointalk.org/index.php?topic=3D1377= +345.0" rel=3D"noreferrer" target=3D"_blank">https://bitcointalk.org/index.p= +hp?topic=3D1377345.0</a> ).<br> +> Napkin math on the size of a txn alone tells me it's more like a 2= +5%<br> +> increase if you just consider size of tx vs size of<br> +> tx+scriptpubkeys,amounts.=C2=A0 If I'm not missing something there= +, I think<br> +> that would get in into a very clear not-win range.<br> +> <br> +> On the positive side is that it doesn't change the blockchain<br> +> datastructure, so it's something implementations could do without<= +br> +> marrying the network to it forever.<br> +> <br> +<br> +_______________________________________________<br> +bitcoin-dev mailing list<br> +<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">= +bitcoin-dev@lists.linuxfoundation.org</a><br> +<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" = +rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail= +man/listinfo/bitcoin-dev</a><br> +</blockquote></div> + +--000000000000f00327056e51b1ee-- + |