To: bitcoin-dev@lists.linuxfoundation.org
References: <CALeFGL1WSSA69ARvJW3di-UC_gz7NV9q7=6zd7s=CHnmttdQFg@mail.gmail.com>
 <CAJx8jdz3uOCpwed3MZkf1ghkvaZMfy-+vvOCVZdvhz2KAn38DQ@mail.gmail.com>
From: Leo Wandersleb <leo@LeoWandersleb.de>
Autocrypt: addr=leo@LeoWandersleb.de; prefer-encrypt=mutual; keydata=
 mQENBE3ePdoBCADZmTvPyledh9wmNRE+i5k7dUmegSmBpe1kMgzlt3nxwzyE6l3wzT9wen8E
 clmA4ZCLn77DjxF3Dg9X0sRio9No5r4u6luI0CzM7axvTi+5DWU25b+JGGdhKTMkyfyKBZdT
 4xvjLzW9tT7upUuAh9nTcz9MBZS3vhKB1SqiBUdYqj33+2erIQzaYMIZ/crR6FbgDT5dI3gg
 2YKJn4rdFy9saA0qngoDPkyU9u5TYtFk8zOreQrFXmXpLiptoxRolF9tqHyn51H4uV8ggIGc
 xJWRW/i2vBFHSM6ZD7PceWCRU5ehf3h/wat1xtn/H7AB+wcywkYrdg1TLifftwcKYBlzABEB
 AAG0JUxlbyBXYW5kZXJzbGViIDxMZW9ATGVvV2FuZGVyc2xlYi5kZT6JATgEEwECACIFAk3e
 PdoCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEGuaHwy3wggSrqQH/Amh6+kYnIUM
 8X5yicpXd37h/Nqnrs6KIE0nYCZe03tkQytndbWXmf2pBth6CEzqrS+wivU7L8dPdiFQtdEW
 dDe8bjJ993TsVSyYrPHv549vD4DbjtB8WS7KEkMhhl+uVaC07YXNwOTAkshv4l3G2ll6dmBn
 LDeF8kSU+DZ1eNzUYHjuUcQYl0XAK5wE7rVpzZUlR0Z5bKNm8QmUPgrZPu6cVRZ6tOD0EMR6
 3dcGI5x0fB35gEUasgNYxQ/g2arh66Pg45olKT1bTcrTJYrE8IePElSCIFDrzrZMhStY3TbQ
 qI3T0vsFGzEeU9M6LmKV8aii4g8o7af0kr4vYS3Nsp65AQ0ETd492gEIAMnahwe+MNliWgpF
 B+UfrwzfgiLjtizH307HYB67THS4h2+MoBJUAqrV7CQkZ8kMYLBl4D+hvxrXDYte6wTb0kcY
 X4NTGof8EDBPxUERYJlWbTbLccbA8d3ia++aTtqd9yRHobK8AO4R22cslYV/nQChHc+1/NFV
 8V1O67U7XpnxQMUzYIhhvVcUwj+8bw/F3zPzS03PWnDv+gmwHt63w72Dth+P0F93/rI+OiEA
 kC/7//1ngQ7auqOeWVdqR8Z1s7tI4kxsVkJfqj3wVu+qaso8FpjNGs888xH8xhgXlll8APWj
 V9Rl3GtpRTdobzNh58jUfGmHh5bcRrYTrtldDwkAEQEAAYkBHwQYAQIACQUCTd492gIbDAAK
 CRBrmh8Mt8IIEi3MB/0UlpEORi7qBOH9egn/8E9fXj3QGpmQ7IAvzvLjkp3cArerBhObhGJ+
 lZ679/9KZ4FxWLDStzGJH3thQstZjcUZzLwTP1sNdY+uPkvOqrzclfPK13z5hJhORD8W0brB
 UptTvyZthIxNIbY4uSbrvaAJmcPFi3cdKOjZz9XyH8AAxXcWGUjQuqxEc0GuM8h8asaUJesB
 bE4J+HPZXcAOC3D+qIqvM438Nbb8Pu7yYWrdq5kmbDMg2nPHXUOOovEsXWBGjWEiXWVUei75
 5Fumqg5mz4P8/Ry3DwZBuHN3Op77gXogvyeE/B5O6dZzWWpSqBIS6MQcLD1divwi/2vNS4c4
Message-ID: <b895f2e4-513f-0c0d-91ac-52af055f332c@LeoWandersleb.de>
Date: Sun, 28 Feb 2021 00:41:06 -0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: <CAJx8jdz3uOCpwed3MZkf1ghkvaZMfy-+vvOCVZdvhz2KAn38DQ@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Content-Language: en-US
Subject: Re: [bitcoin-dev] A design for Probabilistic Partial Pruning
Precedence: list

Only headers need to be downloaded sequentially so downloading relevant blocks
from one node is totally possible with gaps in between.

On 2/27/21 4:10 AM, Igor Cota via bitcoin-dev wrote:
> Hi Keagan,
>
> I had a very similar idea. The only difference being for the node to decide on
> a range of blocks to keep beforehand, rather than making the decision
> block-by-block like you suggest.
>
> I felt the other nodes would be better served by ranges due to the sequential
> nature of IBD. Perhaps this would be computationally lighter as well.
>
> I also encourage you to read Ryosuke Abe's paper [1] that proposes a DHT
> scheme to solve this same problem.
>
> Cheers,
> Igor
>
> [1] https://arxiv.org/abs/1902.02174
>
> On Fri, 26 Feb 2021 at 21:57, Keagan McClelland via bitcoin-dev
> <bitcoin-dev@lists.linuxfoundation.org
> <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
>
>     Hi all,
>
>     I've been thinking for quite some time about the problem of pruned nodes
>     and ongoing storage costs for full nodes. One of the things that strikes
>     me as odd is that we only really have two settings.
>
>     A. Prune everything except the most recent blocks, down to the cache size
>     B. Keep everything since genesis
>
>     From my observations and conversations with various folks in the
>     community, they would like to be able to run a "partially" pruned node to
>     help bear the load of bootstrapping other nodes and helping with data
>     redundancy in the network, but would prefer to not dedicate hundreds of
>     Gigabytes of storage space to the cause.
>
>     This led me to the idea that a node could randomly prune some of the
>     blocks from history if it passed some predicate. A rough sketch of this
>     would look as follows.
>
>     1. At node startup, it would generate a random seed, this would be unique
>     to the node but not necessary that it be cryptographically secure.
>     2. In the node configuration it would also carry a "threshold" expressed
>     as some percentage of blocks it wanted to keep.
>     3. As IBD occurs, based off of the threshold, the block hash, and the
>     node's unique seed, the node would either decide to prune the data or keep
>     it. The uniqueness of the node's hash should ensure that no block is
>     systematically overrepresented in the set of nodes choosing this storage
>     scheme.
>     4. Once the node's IBD is complete it would advertise this as a peer
>     service, advertising its seed and threshold, so that nodes could
>     deterministically deduce which of its peers had which blocks.
>
>     The goals are to increase data redundancy in a way that more uniformly
>     shares the load across nodes, alleviating some of the pressure of full
>     archive nodes on the IBD problem. I am working on a draft BIP for this
>     proposal but figured I would submit it as a high level idea in case anyone
>     had any feedback on the initial design before I go into specification
>     levels of detail.
>
>     If you have thoughts on
>
>     A. The protocol design itself
>     B. The barriers to put this kind of functionality into Core
>
>     I would love to hear from you,
>
>     Cheers,
>     Keagan
>     _______________________________________________
>     bitcoin-dev mailing list
>     bitcoin-dev@lists.linuxfoundation.org
>     <mailto:bitcoin-dev@lists.linuxfoundation.org>
>     https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
>
> -- 
> *Igor Cota*
> Codex Apertus d.o.o.
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev