Return-Path: Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 944F5C0001 for ; Mon, 1 Mar 2021 06:22:14 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 830636066B for ; Mon, 1 Mar 2021 06:22:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org X-Spam-Flag: NO X-Spam-Score: -2.099 X-Spam-Level: X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id w8W7TBiLM-ZW for ; Mon, 1 Mar 2021 06:22:13 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mail-io1-f43.google.com (mail-io1-f43.google.com [209.85.166.43]) by smtp3.osuosl.org (Postfix) with ESMTPS id 24CB26066A for ; Mon, 1 Mar 2021 06:22:13 +0000 (UTC) Received: by mail-io1-f43.google.com with SMTP id p16so16527688ioj.4 for ; Sun, 28 Feb 2021 22:22:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=I7QMTZgDXW6/5y9zhc/IGUJqHIayOn1F773/C7WN3Kw=; b=qOCYrihb9xM8Y6T/M8GysLUY9ydOMp2Fm4p4GCInwm5pYRJG/ZXj7jUf94JoqMpcjc 5L3WlFc5du+cWwpOw6faENPm30WALjc6PgN0G1CMU5EJMS2kN5+Ro36tK07m2WA8uPCO 38NDH+Fa7HqokCuKaNzvmtlaZMAJvYbUQYh5+dCj+AlfERiU5d8kd1TglJ+sjR0LkUMJ RY7/0Ino684f7Uuly6NQv2w7VBb1mH/uwCguJboq6PGLH/t5ynf+Qv73cBzX3VzFxYdU mbC8+DyAypDXJgs7yyhOiNcliBSjzXYimN56zGTEgh3zryxxhECp93URMFt+YwUrSGgs J/4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=I7QMTZgDXW6/5y9zhc/IGUJqHIayOn1F773/C7WN3Kw=; b=D8Nw7FGdyarY+0qV770rtNleN0Vyxd9nsok5SYLrxXZou5/WrtZxyWrNkplTzQfd4y ckt1QKbm+Y3osuFZUxTC4RpIdej15Lus1S7bduDM1eHVedWa1Ko4oJ4C3uPJ9p+jQaE2 9p7+4YjYErqWolsQW3Z9ac63w/TNyEC1KTFIEKbjjJKQZtCvcU4W/QUGbaUkJ9BB0L3a bUPSjjz7bzWqBGpx1CxD0Opior5xL/12i6qB4szi/csXCQMfvPrJm23N/syDlksMfaaP HEh/KI0Lv9K3q32qk8mHVEcoFAf+K0IAZPMGRw1gr443VefHGk7urwK1MNNQnV6yDqL0 NXfw== X-Gm-Message-State: AOAM532afq3o5CLckWLlNz1RPDhdZ5cWfGG/bnN55lfvBVUDKal4ROHu Z7dAc8wG8tTkbemJ/wLb+HCsz62Gwj0Xv/mfdIu9JZ0D X-Google-Smtp-Source: ABdhPJz0YuY4wgOxlNSLkWrZjWVE+gf/NrLEI0EQXsoAtEvAfufvAtwmCOihui+Iw65ZHhincWSq2t5WWGONv1nZ7A0= X-Received: by 2002:a05:6638:3285:: with SMTP id f5mr14475038jav.37.1614579732002; Sun, 28 Feb 2021 22:22:12 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Craig Raw Date: Mon, 1 Mar 2021 08:22:01 +0200 Message-ID: To: Bitcoin Protocol Discussion Content-Type: multipart/alternative; boundary="000000000000633ba805bc73a356" X-Mailman-Approved-At: Mon, 01 Mar 2021 08:51:28 +0000 Subject: Re: [bitcoin-dev] A design for Probabilistic Partial Pruning X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Mar 2021 06:22:14 -0000 --000000000000633ba805bc73a356 Content-Type: text/plain; charset="UTF-8" Something to consider adding to this proposal is to keep the idea of pruning - i.e. retain a sequentially uninterrupted number of the most recent blocks. Many users do not run a node for entirely altruistic reasons - they do so, at least in part, because it allows them to use their wallets privately. Without this ability, I think the number of users willing to run their node in this configuration might be reduced. Another related thought is to have a decreasing density over blocks over time as you go backwards towards genesis, in order for the data density of the storage to match the actual usage of the network, in which (I would imagine) more recent blocks are more heavily requested than early ones. Craig On Sun, Feb 28, 2021 at 10:18 AM Leo Wandersleb via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > Only headers need to be downloaded sequentially so downloading relevant > blocks > from one node is totally possible with gaps in between. > > On 2/27/21 4:10 AM, Igor Cota via bitcoin-dev wrote: > > Hi Keagan, > > > > I had a very similar idea. The only difference being for the node to > decide on > > a range of blocks to keep beforehand, rather than making the decision > > block-by-block like you suggest. > > > > I felt the other nodes would be better served by ranges due to the > sequential > > nature of IBD. Perhaps this would be computationally lighter as well. > > > > I also encourage you to read Ryosuke Abe's paper [1] that proposes a DHT > > scheme to solve this same problem. > > > > Cheers, > > Igor > > > > [1] https://arxiv.org/abs/1902.02174 > > > > On Fri, 26 Feb 2021 at 21:57, Keagan McClelland via bitcoin-dev > > > > wrote: > > > > Hi all, > > > > I've been thinking for quite some time about the problem of pruned > nodes > > and ongoing storage costs for full nodes. One of the things that > strikes > > me as odd is that we only really have two settings. > > > > A. Prune everything except the most recent blocks, down to the cache > size > > B. Keep everything since genesis > > > > From my observations and conversations with various folks in the > > community, they would like to be able to run a "partially" pruned > node to > > help bear the load of bootstrapping other nodes and helping with data > > redundancy in the network, but would prefer to not dedicate hundreds > of > > Gigabytes of storage space to the cause. > > > > This led me to the idea that a node could randomly prune some of the > > blocks from history if it passed some predicate. A rough sketch of > this > > would look as follows. > > > > 1. At node startup, it would generate a random seed, this would be > unique > > to the node but not necessary that it be cryptographically secure. > > 2. In the node configuration it would also carry a "threshold" > expressed > > as some percentage of blocks it wanted to keep. > > 3. As IBD occurs, based off of the threshold, the block hash, and the > > node's unique seed, the node would either decide to prune the data > or keep > > it. The uniqueness of the node's hash should ensure that no block is > > systematically overrepresented in the set of nodes choosing this > storage > > scheme. > > 4. Once the node's IBD is complete it would advertise this as a peer > > service, advertising its seed and threshold, so that nodes could > > deterministically deduce which of its peers had which blocks. > > > > The goals are to increase data redundancy in a way that more > uniformly > > shares the load across nodes, alleviating some of the pressure of > full > > archive nodes on the IBD problem. I am working on a draft BIP for > this > > proposal but figured I would submit it as a high level idea in case > anyone > > had any feedback on the initial design before I go into specification > > levels of detail. > > > > If you have thoughts on > > > > A. The protocol design itself > > B. The barriers to put this kind of functionality into Core > > > > I would love to hear from you, > > > > Cheers, > > Keagan > > _______________________________________________ > > bitcoin-dev mailing list > > bitcoin-dev@lists.linuxfoundation.org > > > > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > > > > > > > > -- > > *Igor Cota* > > Codex Apertus d.o.o. > > > > _______________________________________________ > > bitcoin-dev mailing list > > bitcoin-dev@lists.linuxfoundation.org > > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > --000000000000633ba805bc73a356 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Something to consider adding to this proposal is to keep t= he idea of pruning - i.e. retain a sequentially uninterrupted number of the= most recent blocks.

Many users do not run a node for en= tirely altruistic reasons - they do so, at least in part, because it allows= them to use their wallets privately. Without this ability, I think the num= ber of users willing to run their node in this configuration might be reduc= ed.

Another related thought is to have a decreasin= g density over blocks over time as you go backwards towards genesis, in ord= er for the data density of the storage to match the actual usage of the net= work, in which (I would imagine) more recent blocks are more heavily reques= ted than early ones.

Craig

On Sun, Feb 28, 20= 21 at 10:18 AM Leo Wandersleb via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org&g= t; wrote:
Only h= eaders need to be downloaded sequentially so downloading relevant blocks from one node is totally possible with gaps in between.

On 2/27/21 4:10 AM, Igor Cota via bitcoin-dev wrote:
> Hi Keagan,
>
> I had a very similar idea. The only difference being for the node to d= ecide on
> a range of blocks to keep beforehand, rather than making the decision<= br> > block-by-block like you suggest.
>
> I felt the other nodes would be better served by ranges due to the seq= uential
> nature of IBD. Perhaps this would be computationally lighter as well.<= br> >
> I also encourage=C2=A0you to read=C2=A0Ryosuke Abe's paper [1] tha= t proposes a=C2=A0DHT
> scheme to solve this same problem.
>
> Cheers,
> Igor
>
> [1]=C2=A0https://arxiv.org/abs/1902.02174
>
> On Fri, 26 Feb 2021 at 21:57, Keagan McClelland via bitcoin-dev
> <bitcoin-dev@lists.linuxfoundation.org
> <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote: >
>=C2=A0 =C2=A0 =C2=A0Hi all,
>
>=C2=A0 =C2=A0 =C2=A0I've been thinking for quite some time about th= e problem of pruned nodes
>=C2=A0 =C2=A0 =C2=A0and ongoing storage costs for full nodes. One of th= e things that strikes
>=C2=A0 =C2=A0 =C2=A0me as odd is that we only really have two settings.=
>
>=C2=A0 =C2=A0 =C2=A0A. Prune everything except the most recent blocks, = down to the cache size
>=C2=A0 =C2=A0 =C2=A0B. Keep everything since genesis
>
>=C2=A0 =C2=A0 =C2=A0From my observations and conversations with various= folks in the
>=C2=A0 =C2=A0 =C2=A0community, they would like to be able to run a &quo= t;partially" pruned node to
>=C2=A0 =C2=A0 =C2=A0help bear the load of bootstrapping other nodes and= helping with data
>=C2=A0 =C2=A0 =C2=A0redundancy in the network, but would prefer to not = dedicate hundreds of
>=C2=A0 =C2=A0 =C2=A0Gigabytes of storage space to the cause.
>
>=C2=A0 =C2=A0 =C2=A0This led me to the idea that a node could randomly = prune some of the
>=C2=A0 =C2=A0 =C2=A0blocks from history if it passed some predicate. A = rough sketch of this
>=C2=A0 =C2=A0 =C2=A0would look as follows.
>
>=C2=A0 =C2=A0 =C2=A01. At node startup, it would generate a random seed= , this would be unique
>=C2=A0 =C2=A0 =C2=A0to the node but not necessary that it be cryptograp= hically secure.
>=C2=A0 =C2=A0 =C2=A02. In the node configuration it would also carry a = "threshold" expressed
>=C2=A0 =C2=A0 =C2=A0as some percentage of blocks it wanted to keep.
>=C2=A0 =C2=A0 =C2=A03. As IBD occurs, based off of the threshold, the b= lock hash, and the
>=C2=A0 =C2=A0 =C2=A0node's unique seed, the node would either decid= e to prune the data or keep
>=C2=A0 =C2=A0 =C2=A0it. The uniqueness of the node's hash should en= sure that no block is
>=C2=A0 =C2=A0 =C2=A0systematically overrepresented in the set of nodes = choosing this storage
>=C2=A0 =C2=A0 =C2=A0scheme.
>=C2=A0 =C2=A0 =C2=A04. Once the node's IBD is complete it would adv= ertise this as a peer
>=C2=A0 =C2=A0 =C2=A0service, advertising its seed and threshold, so tha= t nodes could
>=C2=A0 =C2=A0 =C2=A0deterministically deduce which of its peers had whi= ch blocks.
>
>=C2=A0 =C2=A0 =C2=A0The goals are to increase data redundancy in a way = that more uniformly
>=C2=A0 =C2=A0 =C2=A0shares the load across nodes, alleviating some of t= he pressure of full
>=C2=A0 =C2=A0 =C2=A0archive nodes on the IBD problem. I am working on a= draft BIP for this
>=C2=A0 =C2=A0 =C2=A0proposal but figured I would submit it as a high le= vel idea in case anyone
>=C2=A0 =C2=A0 =C2=A0had any feedback on the initial design before I go = into specification
>=C2=A0 =C2=A0 =C2=A0levels of detail.
>
>=C2=A0 =C2=A0 =C2=A0If you have thoughts on
>
>=C2=A0 =C2=A0 =C2=A0A. The protocol design itself
>=C2=A0 =C2=A0 =C2=A0B. The barriers to put this kind of functionality i= nto Core
>
>=C2=A0 =C2=A0 =C2=A0I would love to hear from you,
>
>=C2=A0 =C2=A0 =C2=A0Cheers,
>=C2=A0 =C2=A0 =C2=A0Keagan
>=C2=A0 =C2=A0 =C2=A0_______________________________________________
>=C2=A0 =C2=A0 =C2=A0bitcoin-dev mailing list
>=C2=A0 =C2=A0 =C2=A0bitcoin-dev@lists.linuxfoundation.org
>=C2=A0 =C2=A0 =C2=A0<mailto:bitcoin-dev@lists.linuxfoundation.org>
>=C2=A0 =C2=A0 =C2=A0https://lists.= linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
>
> --
> *Igor Cota*
> Codex Apertus d.o.o.
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org= /mailman/listinfo/bitcoin-dev

_______________________________________________
bitcoin-dev mailing list
= bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mail= man/listinfo/bitcoin-dev
--000000000000633ba805bc73a356--