Received: from sog-mx-1.v43.ch3.sourceforge.com ([172.29.43.191] helo=mx.sourceforge.net) by sfs-ml-3.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1X8C8b-0000rQ-68 for bitcoin-development@lists.sourceforge.net; Fri, 18 Jul 2014 17:40:05 +0000 Received-SPF: pass (sog-mx-1.v43.ch3.sourceforge.com: domain of gmail.com designates 209.85.219.48 as permitted sender) client-ip=209.85.219.48; envelope-from=keziahw@gmail.com; helo=mail-oa0-f48.google.com; Received: from mail-oa0-f48.google.com ([209.85.219.48]) by sog-mx-1.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1X8C8a-0006og-7z for bitcoin-development@lists.sourceforge.net; Fri, 18 Jul 2014 17:40:05 +0000 Received: by mail-oa0-f48.google.com with SMTP id m1so3742768oag.21 for ; Fri, 18 Jul 2014 10:39:58 -0700 (PDT) X-Received: by 10.182.60.42 with SMTP id e10mr9162281obr.33.1405705198292; Fri, 18 Jul 2014 10:39:58 -0700 (PDT) MIME-Version: 1.0 Received: by 10.202.98.11 with HTTP; Fri, 18 Jul 2014 10:39:38 -0700 (PDT) In-Reply-To: References: From: Kaz Wesley Date: Fri, 18 Jul 2014 10:39:38 -0700 Message-ID: To: Bitcoin Dev Content-Type: text/plain; charset=UTF-8 X-Spam-Score: -1.6 (-) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (keziahw[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1X8C8a-0006og-7z Subject: Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jul 2014 17:40:05 -0000 Peers exchanging mempool priority policies is great; that accomplishes the flexibility in what txes to remember that I was going for with the forget-filters, but much more neatly, with less overhead and some side benefits. Here's what I'm picturing now: - exchange priority policies in peer introductions - assign unique sequential IDs in the order the transactions were inved (per peer) - receiving a getdata for a tx updates last-known-peer-received inv to all invs up to the one referenced - include ID-last-received, last-known-peer-received in sparse block - reference txes in sparse block by index in receiver's prioritiziation with peer's sent invs up to ID-last-received and sender's prior invs up to last-known-peer-received Possible new messages: - sparseblock - invack message a node can send at times when it's received a bunch of invs it already has, so it hasn't acked with a getdata in a while - gettx: getdata, but using new sequential ID to save 28 bytes per tx It seems important for ordering policies to be able to be specified in as much detail as possible. Parameters that should be available: - total inputs - total outputs - bytes - coin days destroyed - net UTXO size change - sigops - is data carrier - is output raw multisig - age in mempool - what else? This parameter set should be extensible to allow for unforeseen future factors. Ordering policies should allow arbitrary algebraic combinations of their parameters, as well as thresholds. Boolean combinations of sub-policies would also be desirable. This could be implemented with a tx-script-like stack-based language, in which each supported tx property is pushed onto the stack by a particular opcode, and +-*//min/max/boolean operators combine them to yield the sort key. Difficult parameters: * Coin-days-destroyed: changes, peers need agreement on when (if?) it's recalculated. Probably can just not recalculate, but peers still need agreement on "time seen" to get CDD. * Age in mempool: seems intractable in terms of time, but could be done easily in terms of "how many txes old is this sequential ID" One potential pitfall: this allows for an environment of completely heterogeneous mempool policies. I think that's a good thing, but we need to avoid a situation where only least-common-denominator transactions make it farther than a hop or two, and we don't want nodes to have a strong preference for connecting to like-minded peers since clustering reduces overall connectivity. It may be worthwhile to add a parallel mechanism for relay policies, to differentiate between what a node would keep in its mempool vs. what it wouldn't even relay and doesn't want to see at all. Relay policies could be specified just like prioritization policies, but with the final stack value evaluated in a boolean context. An interesting additional use of policy-scripts would be a standardized way for miners to include a policy script in a coinbase, allowing miners a mechanism to advertise things like their relative price of sigops vs bytes. Nodes may then choose to take this information into account in order to optimize their mempool policies for likelihood of consistency with future blocks. Since policy scripts provide only relative information on prices of different transaction properties rather than an absolute fee, this should not allow miners to "vote fees up", although care would need to be taken they wouldn't be able to drive up prices by claiming common transaction types are at the high end of the fee scale.