Received-SPF: pass (sog-mx-3.v43.ch3.sourceforge.com: domain of gmail.com
	designates 209.85.214.170 as permitted sender)
	client-ip=209.85.214.170; envelope-from=keziahw@gmail.com;
	helo=mail-ob0-f170.google.com; 
MIME-Version: 1.0
In-Reply-To: <CAAS2fgR32qBtAjYNMduHTjz7ae2TSVms-2O53uTgZqtZxX+fqQ@mail.gmail.com>
References: <CA+iPb=EaX=bvOjNtZ+LnYTMRLQQ9nFcrefAkBdv8eActoX_b8A@mail.gmail.com>
	<CABsx9T2PSa3MpfMMDCb8ACVF5vDOZOFLEK9zfP9PakgHA4U16w@mail.gmail.com>
	<CAPkFh0vKFnKRE-sd-Z9t1zB73VLPsiaQ3o=OYgBqqtUE4_rTaw@mail.gmail.com>
	<CA+iPb=GC7iw1LP6boyfX22oMO2k2=YcAuRhE0E3OzzJHYapsow@mail.gmail.com>
	<CAAS2fgS-KiP-tiy91Ah2hJ0pepA0OJDCG+Bv+redFtsqrUTevQ@mail.gmail.com>
	<CA+iPb=Fa4YSTjPuCfyWy0wB2XBV=Mi99G3Hb84gjy+muNDin+g@mail.gmail.com>
	<CAAS2fgSObqk=rD1vtV6LZzxUuyQMh+nwGuatOq1hUaQz2od0sg@mail.gmail.com>
	<CA+iPb=FV1_0SCzcqCz+2eeQW6L18c2O2aKW4zusgNKBYirqHcA@mail.gmail.com>
	<CAAS2fgR32qBtAjYNMduHTjz7ae2TSVms-2O53uTgZqtZxX+fqQ@mail.gmail.com>
From: Kaz Wesley <keziahw@gmail.com>
Date: Thu, 31 Jul 2014 18:00:06 -0700
Message-ID: <CA+iPb=HioTPDNjmq2WPa0ObR+2epN2efnJHaB9YCXynhiewtaw@mail.gmail.com>
To: Gregory Maxwell <gmaxwell@gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Cc: Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Subject: Re: [Bitcoin-development] Squashing redundant tx data in blocks on
 the wire
Precedence: list

On Thu, Jul 31, 2014 at 4:18 PM, Gregory Maxwell <gmaxwell@gmail.com> wrote=
:
> On Thu, Jul 31, 2014 at 3:27 PM, Kaz Wesley <keziahw@gmail.com> wrote:
>>> the FEC still lets you fill in the missing transactions without knowing=
 in advance all that will be missing.
>>
>> I don't see why we need to solve that problem, since the protocol
>> already involves exchanging the information necessary to determine
>> (with some false positives) what a peer is missing, and needs to
>> continue doing so regardless of how blocks are transmitted.
>
> False positives, and if you have more than one peer=E2=80=94 false negati=
ves.
> (or a rule for what you must keep which is conservative in order to
> avoid creating huge storage requirements=E2=80=94 but then also has false
> negatives).

I think a rule for what to keep (along with the false-positive rate
associated with the rule's conservativeness) is preferable to false
negatives, since round-trip cost is potentially very high. The
prototype I'm working on will be able to provide data on what the
false-positive-missing-tx rate would look like with something like
remember-last-N.

There are various ways that rule could be upgraded to nearly eliminate
the false-positive-missing rate, including learning what txes a peer
has dropped via periodic mempool syncing, or specifying the rule
explicitly with priority scripts, both of which have other benefits of
their own. All of these changes synergize, but as long as the
simplistic remembering rule yields worthwhile improvement over the
current approach they can all be done independently as incremental
improvements.

I also really like the idea of the referring to transactions by ids
that can themselves provide part of the tx data; I think that could
also be added separately, on top of these other changes. (Gregory,
your various wiki pages are basically my to-do list of things I'd like
to work on.)

The idea of mempool synchronization brings up the issue of transaction
expiration, since lack of mempool syncing is currently the mechanism
for tx expiry. I'm starting a discussion about how to address that in
a separate thread; see "deterministic transaction expiration".

>> As far as I can tell, channel memory sparseblocks achieve most of the
>> possible bandwidth savings, and when FEC-based mempool synchronization
>> is implemented its benefits can be applied to the sparseblocks by
>> resetting the channel memory to the mutual mempool state each time
>> mempool differences are exchanged. Am I missing a benefit to doing FEC
>> at block forwarding time that can't be realized by FEC-based mempool
>> synchronization, implemented separately from channel-memory based
>> index-coding?
>
> Yes, minimizing latency in the face of multiple peers.
>
> Otherwise no. And certantly no reason to to implement something simple fi=
rst.