Received-SPF: pass (sog-mx-1.v43.ch3.sourceforge.com: domain of gmail.com
	designates 209.85.219.48 as permitted sender)
	client-ip=209.85.219.48; envelope-from=keziahw@gmail.com;
	helo=mail-oa0-f48.google.com; 
MIME-Version: 1.0
In-Reply-To: <CAJHLa0O=eCoyvV19dWgTnYd9Di0wLLZtWmCPidc-dWqPNQv_oQ@mail.gmail.com>
References: <CA+iPb=EaX=bvOjNtZ+LnYTMRLQQ9nFcrefAkBdv8eActoX_b8A@mail.gmail.com>
	<CABsx9T0ag_o_mu=5Q7Ju7s2hO3jz-o5g9FihE9h4B6+ednd2Pg@mail.gmail.com>
	<CAJHLa0NZRF+1QjSwtwjaTE07NWJ_U-O-DE24=P5eSAutMqTupg@mail.gmail.com>
	<CABsx9T2BDBNqvinVNk3FmBRWU7R8jf6Vm6NaH74te0FRCh1O-w@mail.gmail.com>
	<CAJHLa0O=eCoyvV19dWgTnYd9Di0wLLZtWmCPidc-dWqPNQv_oQ@mail.gmail.com>
From: Kaz Wesley <keziahw@gmail.com>
Date: Fri, 18 Jul 2014 10:39:38 -0700
Message-ID: <CA+iPb=H2fkjCxS7-hYqHjFzfMh6onk5RqZMxa8zsXeTn6pQMpA@mail.gmail.com>
To: Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Content-Type: text/plain; charset=UTF-8
Subject: Re: [Bitcoin-development] Squashing redundant tx data in blocks on
 the wire
Precedence: list

Peers exchanging mempool priority policies is great; that accomplishes
the flexibility in what txes to remember that I was going for with the
forget-filters, but much more neatly, with less overhead and some side
benefits.

Here's what I'm picturing now:
- exchange priority policies in peer introductions
- assign unique sequential IDs in the order the transactions were
inved (per peer)
- receiving a getdata for a tx updates last-known-peer-received inv to
all invs up to the one referenced
- include ID-last-received, last-known-peer-received in sparse block
- reference txes in sparse block by index in receiver's
prioritiziation with peer's sent invs up to ID-last-received and
sender's prior invs up to last-known-peer-received

Possible new messages:
- sparseblock
- invack message a node can send at times when it's received a bunch
of invs it already has, so it hasn't acked with a getdata in a while
- gettx: getdata, but using new sequential ID to save 28 bytes per tx

It seems important for ordering policies to be able to be specified in
as much detail as possible. Parameters that should be available:
- total inputs
- total outputs
- bytes
- coin days destroyed
- net UTXO size change
- sigops
- is data carrier
- is output raw multisig
- age in mempool
- what else?
This parameter set should be extensible to allow for unforeseen future factors.

Ordering policies should allow arbitrary algebraic combinations of
their parameters, as well as thresholds. Boolean combinations of
sub-policies would also be desirable. This could be implemented with a
tx-script-like stack-based language, in which each supported tx
property is pushed onto the stack by a particular opcode, and
+-*//min/max/boolean operators combine them to yield the sort key.

Difficult parameters:
* Coin-days-destroyed: changes, peers need agreement on when (if?)
it's recalculated. Probably can just not recalculate, but peers still
need agreement on "time seen" to get CDD.
* Age in mempool: seems intractable in terms of time, but could be
done easily in terms of "how many txes old is this sequential ID"

One potential pitfall: this allows for an environment of completely
heterogeneous mempool policies. I think that's a good thing, but we
need to avoid a situation where only least-common-denominator
transactions make it farther than a hop or two, and we don't want
nodes to have a strong preference for connecting to like-minded peers
since clustering reduces overall connectivity. It may be worthwhile to
add a parallel mechanism for relay policies, to differentiate between
what a node would keep in its mempool vs. what it wouldn't even relay
and doesn't want to see at all. Relay policies could be specified just
like prioritization policies, but with the final stack value evaluated
in a boolean context.

An interesting additional use of policy-scripts would be a
standardized way for miners to include a policy script in a coinbase,
allowing miners a mechanism to advertise things like their relative
price of sigops vs bytes. Nodes may then choose to take this
information into account in order to optimize their mempool policies
for likelihood of consistency with future blocks. Since policy scripts
provide only relative information on prices of different transaction
properties rather than an absolute fee, this should not allow miners
to "vote fees up", although care would need to be taken they wouldn't
be able to drive up prices by claiming common transaction types are at
the high end of the fee scale.