Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 8548089B for ; Wed, 6 Jun 2018 01:13:09 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f50.google.com (mail-wm0-f50.google.com [74.125.82.50]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 8CB9E71A for ; Wed, 6 Jun 2018 01:13:08 +0000 (UTC) Received: by mail-wm0-f50.google.com with SMTP id j15-v6so8685032wme.0 for ; Tue, 05 Jun 2018 18:13:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=0W5Eyv3IYPZ52+lKYVEamWjuKQvNz7Iy58fwaUeyIfY=; b=UrVDxN2ZSuG8e2xqcm42dryTLgJF148c4PPuL8LNeCI68TD2oc9H+vy7p03EIc1Ona sX+NgxyN25pJAUoRip+gXNfNRHClqXYp7lz/wLTFK9vScQ/gubb2PjNYbpl0Ex2qIk1E pz5WBM/1/kfCvJC0GyiiESGhtBHnECdpf1OZXqLZ+bFmNt0WQyKcEAqrVkgxSU+qc3WJ ySSJfOCAt7kO3OIOLiHna6C5zqrsi1xhWVf7FoOKW3oKIl9OjdkCADWhZ3kFLBzwE362 dURTNwPxmIef181cFDSvYq4T53zYWVIdzjJen/pXvNYjyZSehCRolf2eXwoDlnqPIQXh SIrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=0W5Eyv3IYPZ52+lKYVEamWjuKQvNz7Iy58fwaUeyIfY=; b=OeuJMSXi4jWYf+e4QzZIJ2SYoAxcdLJFfdTiUdXL/wPDAtGfvaa1fy0np4qHqlFk2G e4MaGodd8S9K9FDDu1QqbFyTzcYswC9DxOl3ZtFa3lvpQ2BOm2tU1Df/T9ZoJNUlo7nW 6jeJeTmghrylBFEhwLZFdpUjHeaQUj1OI9omoBXqqHeSQL60xnpvaETDn8L2046yRsOL tP9W+fcAn4F0icJPRnIi+3c4wweYC8eoityVT+pT/168Q/t/bUkL9gvWLffUrn3srsCj BQIgqpkA15pfoTuGgUaNTZIAKwVVtRxy1tkvSHV/7+Q3JJlVc46/+NeEgTu+LVJ/5NVi i8Fw== X-Gm-Message-State: APt69E16q0bX7aC/smW82RwNFbO9BZkDpSjp6Sympn+ZFheK1PJQpWOn BBo57bEOL/EA2pZtTGl2zSa/b8iee9h02UHBvQA= X-Google-Smtp-Source: ADUXVKJp/cwpwo52/PiCO2otiURzPprXlUrMSUhfwmtZb2N7INfY5wSBex4dr4eXWr4I8JglzhAbfnNmXQ6SlTfan8I= X-Received: by 2002:a50:b6bc:: with SMTP id d57-v6mr1310845ede.250.1528247587152; Tue, 05 Jun 2018 18:13:07 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Olaoluwa Osuntokun Date: Tue, 5 Jun 2018 18:12:55 -0700 Message-ID: To: Riccardo Casatta , Bitcoin Protocol Discussion Content-Type: multipart/alternative; boundary="0000000000008fa85b056deede81" X-Spam-Status: No, score=-1.7 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM, HTML_MESSAGE,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: Re: [bitcoin-dev] BIP 158 Flexibility and Filter Size X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jun 2018 01:13:09 -0000 --0000000000008fa85b056deede81 Content-Type: text/plain; charset="UTF-8" It isn't being discussed atm (but was discussed 1 year ago when the BIP draft was originally published), as we're in the process of removing items or filters that aren't absolutely necessary. We're now at the point where there're no longer any items we can remove w/o making the filters less generally useful which signals a stopping point so we can begin widespread deployment. In terms of a future extension, BIP 158 already defines custom filter types, and BIP 157 allows filters to be fetched in batch based on the block height and numerical range. The latter feature can later be modified to return a single composite filter rather than several individual filters. -- Laolu On Mon, Jun 4, 2018 at 7:28 AM Riccardo Casatta via bitcoin-dev < bitcoin-dev@lists.linuxfoundation.org> wrote: > I was wondering why this multi-layer multi-block filter proposal isn't > getting any comment, > is it because not asking all filters is leaking information? > > Thanks > > Il giorno ven 18 mag 2018 alle ore 08:29 Karl-Johan Alm via bitcoin-dev < > bitcoin-dev@lists.linuxfoundation.org> ha scritto: > >> On Fri, May 18, 2018 at 12:25 AM, Matt Corallo via bitcoin-dev >> wrote: >> > In general, I'm concerned about the size of the filters making existing >> > SPV clients less willing to adopt BIP 158 instead of the existing bloom >> > filter garbage and would like to see a further exploration of ways to >> > split out filters to make them less bandwidth intensive. Some further >> > ideas we should probably play with before finalizing moving forward is >> > providing filters for certain script templates, eg being able to only >> > get outputs that are segwit version X or other similar ideas. >> >> There is also the idea of multi-block filters. The idea is that light >> clients would download a pair of filters for blocks X..X+255 and >> X+256..X+511, check if they have any matches and then grab pairs for >> any that matched, e.g. X..X+127 & X+128..X+255 if left matched, and >> iterate down until it ran out of hits-in-a-row or it got down to >> single-block level. >> >> This has an added benefit where you can accept a slightly higher false >> positive rate for bigger ranges, because the probability of a specific >> entry having a false positive in each filter is (empirically speaking) >> independent. I.e. with a FP probability of 1% in the 256 range block >> and a FP probability of 0.1% in the 128 range block would mean the >> probability is actually 0.001%. >> >> Wrote about this here: https://bc-2.jp/bfd-profile.pdf (but the filter >> type is different in my experiments) >> _______________________________________________ >> bitcoin-dev mailing list >> bitcoin-dev@lists.linuxfoundation.org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >> > > > -- > Riccardo Casatta - @RCasatta > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > --0000000000008fa85b056deede81 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
It isn't being discussed atm (but was discussed 1= year ago when the BIP
draft was originally published), as we'= ;re in the process of removing items
or filters that aren't a= bsolutely necessary. We're now at the point where
there'r= e no longer any items we can remove w/o making the filters less
g= enerally useful which signals a stopping point so we can begin widespread
deployment.=C2=A0=C2=A0

In terms of a fut= ure extension, BIP 158 already defines custom filter types,
and B= IP 157 allows filters to be fetched in batch based on the block height
and numerical range. The latter feature can later be modified to retu= rn a
single composite filter rather than several individual filte= rs.

-- Laolu


On Mon, Jun 4, 2018 at 7:28 AM Riccardo Casa= tta via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
I was wondering why this multi-layer multi-block filte= r proposal isn't getting any comment,
is it because not asking all filters is leaking = information?
Thanks
=

Il giorno ven 18 mag = 2018 alle ore 08:29 Karl-Johan Alm via bitcoin-dev <bitcoin-dev@lists.li= nuxfoundation.org> ha scritto:
On Fri, May 18, 2018 at 12:25 AM, Matt Corallo via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> In general, I'm concerned about the size of the filters making exi= sting
> SPV clients less willing to adopt BIP 158 instead of the existing bloo= m
> filter garbage and would like to see a further exploration of ways to<= br> > split out filters to make them less bandwidth intensive. Some further<= br> > ideas we should probably play with before finalizing moving forward is=
> providing filters for certain script templates, eg being able to only<= br> > get outputs that are segwit version X or other similar ideas.

There is also the idea of multi-block filters. The idea is that light
clients would download a pair of filters for blocks X..X+255 and
X+256..X+511, check if they have any matches and then grab pairs for
any that matched, e.g. X..X+127 & X+128..X+255 if left matched, and
iterate down until it ran out of hits-in-a-row or it got down to
single-block level.

This has an added benefit where you can accept a slightly higher false
positive rate for bigger ranges, because the probability of a specific
entry having a false positive in each filter is (empirically speaking)
independent. I.e. with a FP probability of 1% in the 256 range block
and a FP probability of 0.1% in the 128 range block would mean the
probability is actually 0.001%.

Wrote about this here: https://bc-2.jp/bfd-profile.pdf (but the f= ilter
type is different in my experiments)
_______________________________________________
bitcoin-dev mailing list
= bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mail= man/listinfo/bitcoin-dev


--
Riccardo Casatta - @RCasatta
_______________________________________________
bitcoin-dev mailing list
= bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mail= man/listinfo/bitcoin-dev
--0000000000008fa85b056deede81--