MIME-Version: 1.0
References: <CALeFGL1WSSA69ARvJW3di-UC_gz7NV9q7=6zd7s=CHnmttdQFg@mail.gmail.com>
In-Reply-To: <CALeFGL1WSSA69ARvJW3di-UC_gz7NV9q7=6zd7s=CHnmttdQFg@mail.gmail.com>
From: Igor Cota <igor@codexapertus.com>
Date: Sat, 27 Feb 2021 08:10:39 +0100
Message-ID: <CAJx8jdz3uOCpwed3MZkf1ghkvaZMfy-+vvOCVZdvhz2KAn38DQ@mail.gmail.com>
To: Keagan McClelland <keagan.mcclelland@gmail.com>, 
 Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>,
 chike@sfc.wide.ad.jp, Artur Brugeman <brugeman.artur@gmail.com>
Content-Type: multipart/alternative; boundary="0000000000009f13fa05bc4c1565"
Subject: Re: [bitcoin-dev] A design for Probabilistic Partial Pruning
Precedence: list

--0000000000009f13fa05bc4c1565
Content-Type: text/plain; charset="UTF-8"

Hi Keagan,

I had a very similar idea. The only difference being for the node to decide
on a range of blocks to keep beforehand, rather than making the decision
block-by-block like you suggest.

I felt the other nodes would be better served by ranges due to the
sequential nature of IBD. Perhaps this would be computationally lighter as
well.

I also encourage you to read Ryosuke Abe's paper [1] that proposes a DHT
scheme to solve this same problem.

Cheers,
Igor

[1] https://arxiv.org/abs/1902.02174

On Fri, 26 Feb 2021 at 21:57, Keagan McClelland via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Hi all,
>
> I've been thinking for quite some time about the problem of pruned nodes
> and ongoing storage costs for full nodes. One of the things that strikes me
> as odd is that we only really have two settings.
>
> A. Prune everything except the most recent blocks, down to the cache size
> B. Keep everything since genesis
>
> From my observations and conversations with various folks in the
> community, they would like to be able to run a "partially" pruned node to
> help bear the load of bootstrapping other nodes and helping with data
> redundancy in the network, but would prefer to not dedicate hundreds of
> Gigabytes of storage space to the cause.
>
> This led me to the idea that a node could randomly prune some of the
> blocks from history if it passed some predicate. A rough sketch of this
> would look as follows.
>
> 1. At node startup, it would generate a random seed, this would be unique
> to the node but not necessary that it be cryptographically secure.
> 2. In the node configuration it would also carry a "threshold" expressed
> as some percentage of blocks it wanted to keep.
> 3. As IBD occurs, based off of the threshold, the block hash, and the
> node's unique seed, the node would either decide to prune the data or keep
> it. The uniqueness of the node's hash should ensure that no block is
> systematically overrepresented in the set of nodes choosing this storage
> scheme.
> 4. Once the node's IBD is complete it would advertise this as a peer
> service, advertising its seed and threshold, so that nodes could
> deterministically deduce which of its peers had which blocks.
>
> The goals are to increase data redundancy in a way that more uniformly
> shares the load across nodes, alleviating some of the pressure of full
> archive nodes on the IBD problem. I am working on a draft BIP for this
> proposal but figured I would submit it as a high level idea in case anyone
> had any feedback on the initial design before I go into specification
> levels of detail.
>
> If you have thoughts on
>
> A. The protocol design itself
> B. The barriers to put this kind of functionality into Core
>
> I would love to hear from you,
>
> Cheers,
> Keagan
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>


-- 
*Igor Cota*
Codex Apertus d.o.o.

--0000000000009f13fa05bc4c1565
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Keagan,<div><br></div><div>I had a very similar idea. T=
he only difference being for the node to decide on a range of blocks to kee=
p beforehand, rather than making the decision block-by-block like you sugge=
st.</div><div><br></div><div>I felt the other nodes would be better served =
by ranges due to the sequential nature of IBD. Perhaps this would be comput=
ationally lighter as well.</div><div><br></div><div>I also encourage=C2=A0y=
ou to read=C2=A0Ryosuke Abe&#39;s paper [1] that proposes a=C2=A0DHT scheme=
 to solve this same problem.</div><div><br></div><div>Cheers,</div><div>Igo=
r</div><div><br></div><div>[1]=C2=A0<a href=3D"https://arxiv.org/abs/1902.0=
2174">https://arxiv.org/abs/1902.02174</a></div></div><br><div class=3D"gma=
il_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Fri, 26 Feb 2021 at 21:5=
7, Keagan McClelland via bitcoin-dev &lt;<a href=3D"mailto:bitcoin-dev@list=
s.linuxfoundation.org" target=3D"_blank">bitcoin-dev@lists.linuxfoundation.=
org</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"marg=
in:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1e=
x"><div dir=3D"ltr">Hi all,<div><br></div><div>I&#39;ve been thinking for q=
uite some time about the problem of pruned nodes and ongoing storage costs =
for full nodes. One of the things that strikes me as odd is that we only re=
ally have two settings.</div><div><br></div><div>A. Prune everything except=
 the most recent blocks, down to the cache size</div><div>B. Keep everythin=
g since genesis</div><div><br></div><div>From my observations and conversat=
ions with various folks in the community, they would like to be able to run=
 a &quot;partially&quot; pruned node to help bear the load of bootstrapping=
 other nodes and helping with data redundancy in the network, but would pre=
fer to not dedicate hundreds of Gigabytes of storage space to the cause.</d=
iv><div><br></div><div>This led me to the idea that a node could randomly p=
rune some of the blocks from history if it passed some predicate. A rough s=
ketch of this would look as follows.</div><div><br></div><div>1. At node st=
artup, it would generate a random seed, this would be unique to the node bu=
t not necessary that it be cryptographically secure.</div><div>2. In the no=
de configuration it would also carry a &quot;threshold&quot; expressed as s=
ome percentage of blocks it wanted to keep.</div><div>3. As IBD occurs, bas=
ed off of the threshold, the block hash, and the node&#39;s unique seed, th=
e node would either decide to prune the data or keep it. The uniqueness of =
the node&#39;s hash should ensure that no block is systematically overrepre=
sented in the set of nodes choosing this storage scheme.</div><div>4. Once =
the node&#39;s IBD is complete it would advertise this as a peer service, a=
dvertising its seed and threshold, so that nodes could deterministically de=
duce which of its peers had which blocks.</div><div><br></div><div>The goal=
s are to increase data redundancy in a way that more uniformly shares the l=
oad across nodes, alleviating some of the pressure of full archive nodes on=
 the IBD problem. I am working on a draft BIP for this proposal but figured=
 I would submit it as a high level idea in case anyone had any feedback on =
the initial design before I go into specification levels of detail.</div><d=
iv><br></div><div>If you have thoughts on</div><div><br></div><div>A. The p=
rotocol design itself</div><div>B. The barriers to put this kind of functio=
nality into Core</div><div><br></div><div>I would love to hear from you,</d=
iv><div><br></div><div>Cheers,</div><div>Keagan</div></div>
_______________________________________________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">=
bitcoin-dev@lists.linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" =
rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail=
man/listinfo/bitcoin-dev</a><br>
</blockquote></div><br clear=3D"all"><div><br></div>-- <br><div dir=3D"ltr"=
><div dir=3D"ltr"><div><div dir=3D"ltr"><b>Igor Cota</b><div>Codex Apertus =
d.o.o.<br></div></div></div></div></div>

--0000000000009f13fa05bc4c1565--