Received: from sog-mx-1.v43.ch3.sourceforge.com ([172.29.43.191] helo=mx.sourceforge.net) by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1VZrgp-0005OM-KX for bitcoin-development@lists.sourceforge.net; Sat, 26 Oct 2013 00:25:15 +0000 Received-SPF: pass (sog-mx-1.v43.ch3.sourceforge.com: domain of gmail.com designates 74.125.82.169 as permitted sender) client-ip=74.125.82.169; envelope-from=gavinandresen@gmail.com; helo=mail-we0-f169.google.com; Received: from mail-we0-f169.google.com ([74.125.82.169]) by sog-mx-1.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1VZrgn-0002oS-7C for bitcoin-development@lists.sourceforge.net; Sat, 26 Oct 2013 00:25:15 +0000 Received: by mail-we0-f169.google.com with SMTP id q58so4578680wes.14 for ; Fri, 25 Oct 2013 17:25:07 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.180.101.134 with SMTP id fg6mr612777wib.9.1382747106991; Fri, 25 Oct 2013 17:25:06 -0700 (PDT) Received: by 10.194.156.163 with HTTP; Fri, 25 Oct 2013 17:25:06 -0700 (PDT) In-Reply-To: References: <20131024143043.GA12658@savin> <20131024144358.GA17142@savin> <20131024145447.GA19949@savin> Date: Sat, 26 Oct 2013 10:25:06 +1000 Message-ID: From: Gavin Andresen To: Jeremy Spilman Content-Type: multipart/alternative; boundary=f46d04462e5620652504e999e66d X-Spam-Score: -0.6 (/) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. 0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [URIs: github.com] -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (gavinandresen[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record 1.0 HTML_MESSAGE BODY: HTML included in message -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1VZrgn-0002oS-7C Cc: Bitcoin Dev Subject: Re: [Bitcoin-development] Making fee estimation better X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Oct 2013 00:25:15 -0000 --f46d04462e5620652504e999e66d Content-Type: text/plain; charset=ISO-8859-1 On Fri, Oct 25, 2013 at 5:51 PM, Jeremy Spilman wrote: > ** > Gavin, can you confirm the best place to read up on the discuss fee > estimation changes for v0.9? > The blog post is the best place for high-level overview. The (closed for now, but it will come back) pull request is the best place for low-level details and nit-picking discussion: https://github.com/bitcoin/bitcoin/pull/3024 > I think fee estimation at its core is about providing a data point, or > even call it an API, which can be used however you see fit. > > What parameters do I want to see in a 'fee estimation' API? > > - 30 minutes vs 24 hours processing time > - Confidence Levels (50%/90%) > The pull request adds an 'estimatefees' JSON-RPC api call: ==== estimatefees [prioritymedian=0.1] [feemedian=0.5] Estimates the priority or fee a transaction needs to be relayed across the network and included in the block chain. prioritymedian and feemedian are values from 0.0 to 1.0, where 0.0 will return the smallest recently-included-in-a-block priority (or fee) seen, 1.0 the largest, and 0.5 the median priority (or fee) for transactions that were broadcast on the network and included in a block. The default value for prioritymedian (0.1) is chosen to return a priority for free transactions that will eventually be confirmed, but might take several hours. The default value for feemedian (0.5) returns how much fee you should include to have your transactions confirmed in an average amount of time. Values returned are: freepriority : priority needed to out-compete a prioritymedian fraction of free transactions to be relayed and included in blocks. feeperbyte : fee, in satoshis/byte, needed to out-compete a feemedian fraction of fee-paying transactions. Values of -1.0 are returned if not enough transactions have been seen to make a good estimate. ==== That API doesn't give "30 minute versus 24 hour" confirmation time or confidence intervals. I've always regretted not taking a statistics class; if you want to help write code that estimates confidence intervals send me an email. The API certainly isn't set in stone. - Is it globally consistent? > Ummm.... roughly, yes, it will be. Nodes that have just joined the network and haven't seen enough transactions enter and leave the memory pool will have a different estimate than long-running nodes, but in my testing the estimate narrows down very quickly (with three or four blocks enough fee-paying transactions have been seen to make a reasonable estimate; it takes longer to see enough free transactions to get a good estimate of the priority needed to get into the free space of a block). RE: lots of other comments: I feel like there is a lot of "in the weeds" discussion here about theoretical, what-if-this-and-that-happens-in-the-future scenarios. I would just like to point out (again) that this is not intended to be The One True Solution For Transaction Fees And Transaction Prioritization. If you've got a better mechanism for estimating fees, fantastic! If it turns out estimates are often-enough wrong to be a problem and you've got a solution for that, fantastic! RE: are we already seeing pressure on transaction fees: I believe we are, yes. As part of the prep work for the smart fee work I spent some time plotting priority (for zero-fee transactions) and transaction fee (for zero-priority transactions) versus confirmation time, and it looks to me like people/services are starting to include more than the hard-coded fees in the reference implementation-- I assume because they want their transactions to be confirmed more quickly. There is definitely already competition among zero-fee transactions for the "free" block space. One of the reasons I'm comfortable with the fee changes I'm proposing is if the estimation code gets it very wrong we'll see that first as free transactions taking "too long" to confirm, but they'll confirm eventually because priority increases over time. -- -- Gavin Andresen --f46d04462e5620652504e999e66d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
On Fri, Oct 25, 2013 at 5:51 PM, Jeremy Spilman <jeremy@= taplink.co> wrote:
Gavin, can you confirm the best place to =A0read =A0up on the dis= cuss fee estimation changes for v0.9?

The blog post is the best place for high-level overview.

The (closed for now, but it will come back) pull request is = the best place for low-level details and nit-picking discussion:

=A0
I think = fee estimation at its core is about providing a data point, or even call it= an API, which can be used however you see fit.

What parameters do I want to see in a 'fee estimati= on' API?

=A0- 30 minutes vs 24 hours processin= g time
=A0- Confidence Levels (50%/90%)
<= div>
The pull request adds an 'estimatefees' JSON-RPC api= call:

=3D=3D=3D=3D
estimatefees [p= rioritymedian=3D0.1] [feemedian=3D0.5]
Estimates the priority or = fee a transaction needs
to be relayed across the network and included in
the block c= hain.

prioritymedian and feemedian are values from= 0.0
to 1.0, where 0.0 will return the smallest
recentl= y-included-in-a-block priority (or fee) seen,
1.0 the largest, and 0.5 the median priority (or fee)
for tr= ansactions that were broadcast on the network and
included in a b= lock.

The default value for prioritymedian (0.1) i= s
chosen to return a priority for free transactions that
will = eventually be confirmed, but might take several hours.
The defaul= t value for feemedian (0.5) returns how much
fee you should inclu= de to have your transactions confirmed
in an average amount of time.

Values returned= are:
=A0freepriority : priority needed to out-compete a priority= median
=A0 fraction of free transactions to be relayed and includ= ed in blocks.
=A0feeperbyte : fee, in satoshis/byte, needed to out-compete a
=A0 feemedian fraction of fee-paying transactions.

<= div>Values of -1.0 are returned if not enough transactions
have b= een seen to make a good estimate.
=3D=3D=3D=3D

That API doesn't give = "30 minute versus 24 hour" confirmation time or confidence interv= als. I've always regretted not taking a statistics class; if you want t= o help write code that estimates confidence intervals send me an email. The= API certainly isn't set in stone.

=A0 - Is it globally consistent?<= /div>

Ummm.... roughly, yes, it will be. N= odes that have just joined the network and haven't seen enough transact= ions enter and leave the memory pool will have a different estimate than lo= ng-running nodes, but in my testing the estimate narrows down very quickly = (with three or four blocks enough fee-paying transactions have been seen to= make a reasonable estimate; it takes longer to see enough free transaction= s to get a good estimate of the priority needed to get into the free space = of a block).

RE: lots of other comments:

I feel like there is a lot of &q= uot;in the weeds" discussion here about theoretical, what-if-this-and-= that-happens-in-the-future scenarios.

I would jus= t like to point out (again) that this is not intended to be The One True So= lution For Transaction Fees And Transaction Prioritization. If you've g= ot a better mechanism for estimating fees, fantastic! If it turns out estim= ates are often-enough wrong to be a problem and you've got a solution f= or that, fantastic!

RE: are we = already seeing pressure on transaction fees:

I believe we are, yes. As part of th= e prep work for the smart fee work I spent some time plotting priority (for= zero-fee transactions) and transaction fee (for zero-priority transactions= ) versus confirmation time, and it looks to me like people/services are sta= rting to include more than the hard-coded fees in the reference implementat= ion-- I assume because they want their transactions to be confirmed more qu= ickly.

There is de= finitely already competition among zero-fee transactions for the "free= " block space. One of the reasons I'm comfortable with the fee cha= nges I'm proposing is if the estimation code gets it very wrong we'= ll see that first as free transactions taking "too long" to confi= rm, but they'll confirm eventually because priority increases over time= .

--
--
Gavin Andresen
--f46d04462e5620652504e999e66d--