path: root/b8/93c6048b898463fa4af9907ff99fd5fd4d26ea

Return-Path: <igor.cota.hr@gmail.com>
Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138])
 by lists.linuxfoundation.org (Postfix) with ESMTP id 192BAC0001
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Sat, 27 Feb 2021 07:10:54 +0000 (UTC)
Received: from localhost (localhost [127.0.0.1])
 by smtp1.osuosl.org (Postfix) with ESMTP id E859D842FC
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Sat, 27 Feb 2021 07:10:53 +0000 (UTC)
X-Virus-Scanned: amavisd-new at osuosl.org
X-Spam-Flag: NO
X-Spam-Score: 1.3
X-Spam-Level: *
X-Spam-Status: No, score=1.3 tagged_above=-999 required=5 tests=[BAYES_50=0.8, 
 DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
 FREEMAIL_FORGED_FROMDOMAIN=0.249, 
 FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25,
 HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001,
 RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001]
 autolearn=no autolearn_force=no
Authentication-Results: smtp1.osuosl.org (amavisd-new);
 dkim=pass (2048-bit key)
 header.d=codexapertus-com.20150623.gappssmtp.com
Received: from smtp1.osuosl.org ([127.0.0.1])
 by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id WCTb3dtYA3iz
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Sat, 27 Feb 2021 07:10:52 +0000 (UTC)
X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0
Received: from mail-yb1-f170.google.com (mail-yb1-f170.google.com
 [209.85.219.170])
 by smtp1.osuosl.org (Postfix) with ESMTPS id 8892B842FB
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Sat, 27 Feb 2021 07:10:52 +0000 (UTC)
Received: by mail-yb1-f170.google.com with SMTP id c131so11284867ybf.7
 for <bitcoin-dev@lists.linuxfoundation.org>;
 Fri, 26 Feb 2021 23:10:52 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=codexapertus-com.20150623.gappssmtp.com; s=20150623;
 h=mime-version:references:in-reply-to:from:date:message-id:subject:to;
 bh=n/ht1PcMwBb+RstIxCx6iVJxZKk2ayvn7+TU1GPbDp4=;
 b=e+kVkufVBFt8I+zIRJmadJg5Onr1FcY6MySa4FHNQ8KdTlnsg6H9aD5yJhu7aciy74
 KC3orlHpezl968qu1nn0AAw7P4+pllwUSZ2glFyazCKDdrsxSBDBQGiERwsqVJFepecl
 BJBxswzN8h2RSqiyteSKkS6ehLW30TsJhFKlezxteNa1i8AZ7Ovbp88bOIzMbJ4NqAoM
 LfxfEdJeHpClZ40EAjZuUDf2h9RUBSUt55jFNWAU/offVfPai0zJ2pX5d6i9ruOC3mcy
 m8frybNohImoS5aPwNrp1Euz3ram3MxKPFdx3/UrjHSiA1L+VYfM3+/ju+l+H6BYI/W/
 zELg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:references:in-reply-to:from:date
 :message-id:subject:to;
 bh=n/ht1PcMwBb+RstIxCx6iVJxZKk2ayvn7+TU1GPbDp4=;
 b=oK41sWwF67mzlsxJ+qH5Tr2/nsKGo5xTbaSNhcb3pqQDy+KmcEIm38aUSfOUE2cxqm
 8tMW2AUtDCsj1+kLGHi2PEju/KUUbouAOJsMsRfU1ZiJtCZRS3N2TJOIPasbg7QbmydA
 HDHeQO2xxUW8k9/fiT26yKUwAD4tlfn/KeMLrHYcvhs/ab1Ui726pyyhx5WTVuqj3GYm
 QcIMAJbF2Fq0E5UKUbxhAM+psyK1GLzlpGcRO2fqAJBz9lWxDx92mRr8xhojQozh9HzR
 O5CwGpxQJIWl9LcKFMOwCmvoVgJ4CxAsx/Wa8pxPlbSAKgNc5Hrh5W2NyZ4YLRvcHoPT
 ++SQ==
X-Gm-Message-State: AOAM531LFiES1Z10mAt6/ce1g5lNIzSWW28PY1QupRVhtI8vT72l9jtO
 4NbhNr/J5kV/yDFceZkq/fu73GR8nonl0YUjA0U=
X-Google-Smtp-Source: ABdhPJxQf93Hmq8RbilEXj+BZDmspFJkcuIo891kYDurzipk/CiIvOBHsabFyZ3wfkbddSExUK9cq4D5ctWdeuyMYeo=
X-Received: by 2002:a5b:847:: with SMTP id v7mr9526758ybq.354.1614409849835;
 Fri, 26 Feb 2021 23:10:49 -0800 (PST)
MIME-Version: 1.0
References: <CALeFGL1WSSA69ARvJW3di-UC_gz7NV9q7=6zd7s=CHnmttdQFg@mail.gmail.com>
In-Reply-To: <CALeFGL1WSSA69ARvJW3di-UC_gz7NV9q7=6zd7s=CHnmttdQFg@mail.gmail.com>
From: Igor Cota <igor@codexapertus.com>
Date: Sat, 27 Feb 2021 08:10:39 +0100
Message-ID: <CAJx8jdz3uOCpwed3MZkf1ghkvaZMfy-+vvOCVZdvhz2KAn38DQ@mail.gmail.com>
To: Keagan McClelland <keagan.mcclelland@gmail.com>, 
 Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>,
 chike@sfc.wide.ad.jp, Artur Brugeman <brugeman.artur@gmail.com>
Content-Type: multipart/alternative; boundary="0000000000009f13fa05bc4c1565"
X-Mailman-Approved-At: Sat, 27 Feb 2021 11:49:53 +0000
Subject: Re: [bitcoin-dev] A design for Probabilistic Partial Pruning
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>, 
 <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>, 
 <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Sat, 27 Feb 2021 07:10:54 -0000

--0000000000009f13fa05bc4c1565
Content-Type: text/plain; charset="UTF-8"

Hi Keagan,

I had a very similar idea. The only difference being for the node to decide
on a range of blocks to keep beforehand, rather than making the decision
block-by-block like you suggest.

I felt the other nodes would be better served by ranges due to the
sequential nature of IBD. Perhaps this would be computationally lighter as
well.

I also encourage you to read Ryosuke Abe's paper [1] that proposes a DHT
scheme to solve this same problem.

Cheers,
Igor

[1] https://arxiv.org/abs/1902.02174

On Fri, 26 Feb 2021 at 21:57, Keagan McClelland via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Hi all,
>
> I've been thinking for quite some time about the problem of pruned nodes
> and ongoing storage costs for full nodes. One of the things that strikes me
> as odd is that we only really have two settings.
>
> A. Prune everything except the most recent blocks, down to the cache size
> B. Keep everything since genesis
>
> From my observations and conversations with various folks in the
> community, they would like to be able to run a "partially" pruned node to
> help bear the load of bootstrapping other nodes and helping with data
> redundancy in the network, but would prefer to not dedicate hundreds of
> Gigabytes of storage space to the cause.
>
> This led me to the idea that a node could randomly prune some of the
> blocks from history if it passed some predicate. A rough sketch of this
> would look as follows.
>
> 1. At node startup, it would generate a random seed, this would be unique
> to the node but not necessary that it be cryptographically secure.
> 2. In the node configuration it would also carry a "threshold" expressed
> as some percentage of blocks it wanted to keep.
> 3. As IBD occurs, based off of the threshold, the block hash, and the
> node's unique seed, the node would either decide to prune the data or keep
> it. The uniqueness of the node's hash should ensure that no block is
> systematically overrepresented in the set of nodes choosing this storage
> scheme.
> 4. Once the node's IBD is complete it would advertise this as a peer
> service, advertising its seed and threshold, so that nodes could
> deterministically deduce which of its peers had which blocks.
>
> The goals are to increase data redundancy in a way that more uniformly
> shares the load across nodes, alleviating some of the pressure of full
> archive nodes on the IBD problem. I am working on a draft BIP for this
> proposal but figured I would submit it as a high level idea in case anyone
> had any feedback on the initial design before I go into specification
> levels of detail.
>
> If you have thoughts on
>
> A. The protocol design itself
> B. The barriers to put this kind of functionality into Core
>
> I would love to hear from you,
>
> Cheers,
> Keagan
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>


-- 
*Igor Cota*
Codex Apertus d.o.o.

--0000000000009f13fa05bc4c1565
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Keagan,<div><br></div><div>I had a very similar idea. T=
he only difference being for the node to decide on a range of blocks to kee=
p beforehand, rather than making the decision block-by-block like you sugge=
st.</div><div><br></div><div>I felt the other nodes would be better served =
by ranges due to the sequential nature of IBD. Perhaps this would be comput=
ationally lighter as well.</div><div><br></div><div>I also encourage=C2=A0y=
ou to read=C2=A0Ryosuke Abe&#39;s paper [1] that proposes a=C2=A0DHT scheme=
 to solve this same problem.</div><div><br></div><div>Cheers,</div><div>Igo=
r</div><div><br></div><div>[1]=C2=A0<a href=3D"https://arxiv.org/abs/1902.0=
2174">https://arxiv.org/abs/1902.02174</a></div></div><br><div class=3D"gma=
il_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Fri, 26 Feb 2021 at 21:5=
7, Keagan McClelland via bitcoin-dev &lt;<a href=3D"mailto:bitcoin-dev@list=
s.linuxfoundation.org" target=3D"_blank">bitcoin-dev@lists.linuxfoundation.=
org</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"marg=
in:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1e=
x"><div dir=3D"ltr">Hi all,<div><br></div><div>I&#39;ve been thinking for q=
uite some time about the problem of pruned nodes and ongoing storage costs =
for full nodes. One of the things that strikes me as odd is that we only re=
ally have two settings.</div><div><br></div><div>A. Prune everything except=
 the most recent blocks, down to the cache size</div><div>B. Keep everythin=
g since genesis</div><div><br></div><div>From my observations and conversat=
ions with various folks in the community, they would like to be able to run=
 a &quot;partially&quot; pruned node to help bear the load of bootstrapping=
 other nodes and helping with data redundancy in the network, but would pre=
fer to not dedicate hundreds of Gigabytes of storage space to the cause.</d=
iv><div><br></div><div>This led me to the idea that a node could randomly p=
rune some of the blocks from history if it passed some predicate. A rough s=
ketch of this would look as follows.</div><div><br></div><div>1. At node st=
artup, it would generate a random seed, this would be unique to the node bu=
t not necessary that it be cryptographically secure.</div><div>2. In the no=
de configuration it would also carry a &quot;threshold&quot; expressed as s=
ome percentage of blocks it wanted to keep.</div><div>3. As IBD occurs, bas=
ed off of the threshold, the block hash, and the node&#39;s unique seed, th=
e node would either decide to prune the data or keep it. The uniqueness of =
the node&#39;s hash should ensure that no block is systematically overrepre=
sented in the set of nodes choosing this storage scheme.</div><div>4. Once =
the node&#39;s IBD is complete it would advertise this as a peer service, a=
dvertising its seed and threshold, so that nodes could deterministically de=
duce which of its peers had which blocks.</div><div><br></div><div>The goal=
s are to increase data redundancy in a way that more uniformly shares the l=
oad across nodes, alleviating some of the pressure of full archive nodes on=
 the IBD problem. I am working on a draft BIP for this proposal but figured=
 I would submit it as a high level idea in case anyone had any feedback on =
the initial design before I go into specification levels of detail.</div><d=
iv><br></div><div>If you have thoughts on</div><div><br></div><div>A. The p=
rotocol design itself</div><div>B. The barriers to put this kind of functio=
nality into Core</div><div><br></div><div>I would love to hear from you,</d=
iv><div><br></div><div>Cheers,</div><div>Keagan</div></div>
_______________________________________________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">=
bitcoin-dev@lists.linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" =
rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail=
man/listinfo/bitcoin-dev</a><br>
</blockquote></div><br clear=3D"all"><div><br></div>-- <br><div dir=3D"ltr"=
><div dir=3D"ltr"><div><div dir=3D"ltr"><b>Igor Cota</b><div>Codex Apertus =
d.o.o.<br></div></div></div></div></div>

--0000000000009f13fa05bc4c1565--