Return-Path: Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id B05BFC0001 for ; Mon, 1 Mar 2021 20:55:24 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 9E2984EC14 for ; Mon, 1 Mar 2021 20:55:24 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org X-Spam-Flag: NO X-Spam-Score: -0.2 X-Spam-Level: X-Spam-Status: No, score=-0.2 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MB09ykEDiqyt for ; Mon, 1 Mar 2021 20:55:23 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) by smtp4.osuosl.org (Postfix) with ESMTPS id 114414EC0D for ; Mon, 1 Mar 2021 20:55:22 +0000 (UTC) Received: by mail-wr1-f52.google.com with SMTP id h98so17578446wrh.11 for ; Mon, 01 Mar 2021 12:55:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=W8CHiQv7S2RR8jRdJAIAF73fpdaGNdLrkVTCvpnT8lk=; b=fX3NDwT1/WqhBXGH/kwz7v+XmuezAwgousr5Z/AHmXJKOV+ADij30stGQ1THvUTKhh dBnn/M4RfQD64WfWPjdNUG0yt3LrJKJgT2pcMGPUh9bbusCIbu5pFdxVxz2vB/qZM7Nr P1/oCM+BX/eJ6ZtSkeRh6vrITReukGbol5e0k6mbxS/ddd92wJxo4VJ8IxxytukaZwJj X+QobceAR6kUgbngvbLwyPU25znTOUgb+plqnMigxti3vSRe+hgN7XYSMQEfXlCdGWpt 1vCWuYma/b1Cy5pkMAC3zlC3ZLldcrPW2rNWFYn3G5N83ItaPhmfeDET3PrBmFVbzmxR jguQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=W8CHiQv7S2RR8jRdJAIAF73fpdaGNdLrkVTCvpnT8lk=; b=ZDoXWoKywMS1ruweQAhHCnVicdDclRnRjwOHQCkbwPQq111BrpNO+ZQB26FR99TlPA fpIVw1EtbCJWpf7KI7BOXzQwSPFkYdz16yikIytV3nnob8czAwJe2M+EY0sWHHptymxE axklR4v+pTDS3JY0mAbG3VNLwH5MAP95HJwXGL7tVefKk5UA+mkzxtjAjDK5Sz/QrLqv HxUWl9dMPrLf9cfpZKHL7PewYWC0x83lfxON1dia754f08TofOrYoRx6Z9AnJbICcpaA G+zlUYIffPHG4rV0gDmrO88A9+dptX2dXS3EBK197QfJbNi64W3akLJpxFaXJxgPFThx SD0A== X-Gm-Message-State: AOAM530tAp8KWhFz9P7B73wmg2K3ydszLOM73HOPQPS7zVK/c+GB50yG Q2xv/bTvb2S7Yn7rmP2fPRYW9vTu67OfzHUD9vUPMHonUnNgJQ== X-Google-Smtp-Source: ABdhPJw660+u4kMrJZAQEDx6v52aZeTBSdoR/sYxWp3xhjpBvxneu9biH4EKb6VmmChZ1agDSOmDv05d0NbV3j5baQI= X-Received: by 2002:a5d:4443:: with SMTP id x3mr3220394wrr.49.1614632121231; Mon, 01 Mar 2021 12:55:21 -0800 (PST) MIME-Version: 1.0 References: <00b001d70e7e$7204e2f0$560ea8d0$@voskuil.org> In-Reply-To: <00b001d70e7e$7204e2f0$560ea8d0$@voskuil.org> From: Keagan McClelland Date: Mon, 1 Mar 2021 13:55:10 -0700 Message-ID: To: eric@voskuil.org, Bitcoin Protocol Discussion Content-Type: multipart/alternative; boundary="000000000000076db405bc7fd653" X-Mailman-Approved-At: Mon, 01 Mar 2021 21:06:49 +0000 Subject: Re: [bitcoin-dev] A design for Probabilistic Partial Pruning X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Mar 2021 20:55:24 -0000 --000000000000076db405bc7fd653 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable > Personally I consider this counterproductive. Apart from the complexity, it=E2=80=99s not healthy. And the chain grows linearly with storage cost fa= lling exponentially, leading to a straightforward conclusion. The motivation for this change is not to encourage full archival nodes to prune, but to make it possible for pruned nodes to beef up what kind of archive they retain. Personally I think using the falling storage costs as a means of providing access to more users is more important than using it to justify requiring higher node requirements. > Something to consider adding to this proposal is to keep the idea of pruning - i.e. retain a sequentially uninterrupted number of the most recent blocks. > > Many users do not run a node for entirely altruistic reasons - they do so, at least in part, because it allows them to use their wallets privately. Without this ability, I think the number of users willing to run their node in this configuration might be reduced. > > Another related thought is to have a decreasing density over blocks over time as you go backwards towards genesis, in order for the data density of the storage to match the actual usage of the network, in which (I would imagine) more recent blocks are more heavily requested than early ones. Per my above comments, this change is actually capitalizing primarily upon those who wish to do it for more altruistic reasons. Furthermore, doing linear block scans when you need to query blocks that you don't keep does not leak privacy details in the same way that bloom filters do. You are not signaling to the peer that there is something specific in that block that you care about, because you don't actually know. You are signalling only that you do not have that block right now, which from the other parts of the design you are already leaking. In light of this, I don't think that it is necessary for the blocks to be in sequential sets at all. If there is no requirement on them being sequential, uniform randomness will take care of the density problem automatically. Keagan On Mon, Mar 1, 2021 at 4:20 AM Eric Voskuil via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > On Sun, Feb 28, 2021 at 10:18 AM Leo Wandersleb via bitcoin-dev < > bitcoin-dev@lists.linuxfoundation.org> wrote: > > > > > Only headers need to be downloaded sequentially so downloading relevant > blocks from one node is totally possible with gaps in between. > > > > In fact this is exactly how libbitcoin v4 works. We download and store > blocks in parallel. In the case of a restart block gaps are repopulated. > Given that headers are validated, we go after the most responsive nodes. > Based on standard deviation, we drop the slowest peers and rebalance load > to new/empty channels. We make ordered but not necessarily sequential > requests. There is no distinction between =E2=80=9Cinitial=E2=80=9D block= download, a > restart, or a single or few blocks at the top. So it=E2=80=99s referred t= o as > continuous parallel block download. > > > > But we don=E2=80=99t prune. Personally I consider this counterproductive.= Apart > from the complexity, it=E2=80=99s not healthy. And the chain grows linear= ly with > storage cost falling exponentially, leading to a straightforward conclusi= on. > > > > e > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > --000000000000076db405bc7fd653 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
> Personally I consider this counterproductive. Apart f= rom the complexity, it=E2=80=99s not healthy. And the chain grows linearly = with storage cost falling exponentially, leading to a straightforward concl= usion.

The motivation for this change is not to encourag= e full archival nodes to prune, but to make it possible for pruned nodes to= beef up what kind of archive they retain. Personally I think using the fal= ling storage costs as a means of providing access to more users is more imp= ortant than using it to justify requiring higher node requirements.

> Something to consider adding to this proposal is to = keep the idea of pruning - i.e. retain a sequentially uninterrupted number = of the most recent blocks.
>
> Many users do not run a = node for entirely altruistic reasons - they do so, at least in part, becaus= e it allows them to use their wallets privately. Without this ability, I th= ink the number of users willing to run their node in this configuration mig= ht be reduced.
>
> Another related thought is to = have a decreasing density over blocks over time as you go backwards towards= genesis, in order for the data density of the storage to match the actual = usage of the network, in which (I would imagine) more recent blocks are mor= e heavily requested than early ones.

Per my above comment= s, this change is actually capitalizing primarily upon those who wish to do= it for more altruistic reasons. Furthermore, doing linear block scans when= you need to query blocks that you don't keep does not leak privacy det= ails in the same way that bloom filters do. You are not signaling to the pe= er that there is something specific in that block that you care about, beca= use you don't actually know. You are signalling only that you do not ha= ve that block right now, which from the other parts of the design you are a= lready leaking. In light of this, I don't think that it is necessary fo= r the blocks to be in sequential sets at all. If there is no requirement on= them being sequential, uniform randomness will take care of the density pr= oblem automatically.
Keagan
<= div style=3D"color:rgb(136,136,136)">

On Mon, Mar 1, 2021 = at 4:20 AM Eric Voskuil via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wro= te:

On Sun, Feb 28, 2= 021 at 10:18 AM Leo Wandersleb via bitcoin-dev <bitcoin-dev@lists.linuxf= oundation.org> wrote:

=C2=A0

> Only headers need to be downl= oaded sequentially so downloading relevant blocks from one node is totally = possible with gaps in between.

<= /u>=C2=A0

In fact this is exactly how libb= itcoin v4 works. We download and store blocks in parallel. In the case of a= restart block gaps are repopulated. Given that headers are validated, we g= o after the most responsive nodes. Based on standard deviation, we drop the= slowest peers and rebalance load to new/empty channels. We make ordered bu= t not necessarily sequential requests. There is no distinction between =E2= =80=9Cinitial=E2=80=9D block download, a restart, or a single or few blocks= at the top. So it=E2=80=99s referred to as continuous parallel block downl= oad.

=C2=A0

But we don=E2=80=99t prune. Personally I consider this cou= nterproductive. Apart from the complexity, it=E2=80=99s not healthy. And th= e chain grows linearly with storage cost falling exponentially, leading to = a straightforward conclusion.

=C2=A0

e

_______________________________________________
bitcoin-dev mailing list
= bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mail= man/listinfo/bitcoin-dev
--000000000000076db405bc7fd653--