path: root/a0/256144c4f02acc0beff8261fb4a8e776a89929

Received: from sog-mx-1.v43.ch3.sourceforge.com ([172.29.43.191]
	helo=mx.sourceforge.net)
	by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
	(envelope-from <gavinandresen@gmail.com>) id 1VZrgp-0005OM-KX
	for bitcoin-development@lists.sourceforge.net;
	Sat, 26 Oct 2013 00:25:15 +0000
Received-SPF: pass (sog-mx-1.v43.ch3.sourceforge.com: domain of gmail.com
	designates 74.125.82.169 as permitted sender)
	client-ip=74.125.82.169; envelope-from=gavinandresen@gmail.com;
	helo=mail-we0-f169.google.com; 
Received: from mail-we0-f169.google.com ([74.125.82.169])
	by sog-mx-1.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
	(Exim 4.76) id 1VZrgn-0002oS-7C
	for bitcoin-development@lists.sourceforge.net;
	Sat, 26 Oct 2013 00:25:15 +0000
Received: by mail-we0-f169.google.com with SMTP id q58so4578680wes.14
	for <bitcoin-development@lists.sourceforge.net>;
	Fri, 25 Oct 2013 17:25:07 -0700 (PDT)
MIME-Version: 1.0
X-Received: by 10.180.101.134 with SMTP id fg6mr612777wib.9.1382747106991;
	Fri, 25 Oct 2013 17:25:06 -0700 (PDT)
Received: by 10.194.156.163 with HTTP; Fri, 25 Oct 2013 17:25:06 -0700 (PDT)
In-Reply-To: <op.w5h2rwhcyldrnw@laptop-air>
References: <20131024143043.GA12658@savin>
	<CANEZrP100Lg_1LcFMKx1yWrGTSFb5GZmLmXNbZjPGaiEgOeuwA@mail.gmail.com>
	<20131024144358.GA17142@savin>
	<CANEZrP1TfM+wYbGjUk3+8JJZs6cKZXdb57xGMc=hDr9dQjMMZA@mail.gmail.com>
	<20131024145447.GA19949@savin>
	<CABsx9T0T0v=HnRRr6BLKNQOFMBJWrhF4G4SOCJ9DidGJBB8Eow@mail.gmail.com>
	<op.w5h2rwhcyldrnw@laptop-air>
Date: Sat, 26 Oct 2013 10:25:06 +1000
Message-ID: <CABsx9T0nc-TO1_=n47UnYHiWKSNvci9Xyhni9PQa=DRo1B7FDg@mail.gmail.com>
From: Gavin Andresen <gavinandresen@gmail.com>
To: Jeremy Spilman <jeremy@taplink.co>
Content-Type: multipart/alternative; boundary=f46d04462e5620652504e999e66d
X-Spam-Score: -0.6 (/)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
	See http://spamassassin.org/tag/ for more details.
	0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked.
	See
	http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
	for more information. [URIs: github.com]
	-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
	sender-domain
	0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
	(gavinandresen[at]gmail.com)
	-0.0 SPF_PASS               SPF: sender matches SPF record
	1.0 HTML_MESSAGE           BODY: HTML included in message
	-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
	author's domain
	0.1 DKIM_SIGNED            Message has a DKIM or DK signature,
	not necessarily valid
	-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
X-Headers-End: 1VZrgn-0002oS-7C
Cc: Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Subject: Re: [Bitcoin-development] Making fee estimation better
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
	<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
	<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Sat, 26 Oct 2013 00:25:15 -0000

--f46d04462e5620652504e999e66d
Content-Type: text/plain; charset=ISO-8859-1

On Fri, Oct 25, 2013 at 5:51 PM, Jeremy Spilman <jeremy@taplink.co> wrote:

> **
> Gavin, can you confirm the best place to  read  up on the discuss fee
> estimation changes for v0.9?
>

The blog post is the best place for high-level overview.

The (closed for now, but it will come back) pull request is the best place
for low-level details and nit-picking discussion:
  https://github.com/bitcoin/bitcoin/pull/3024



> I think fee estimation at its core is about providing a data point, or
> even call it an API, which can be used however you see fit.
>
> What parameters do I want to see in a 'fee estimation' API?
>
>  - 30 minutes vs 24 hours processing time
>  - Confidence Levels (50%/90%)
>

The pull request adds an 'estimatefees' JSON-RPC api call:

====
estimatefees [prioritymedian=0.1] [feemedian=0.5]
Estimates the priority or fee a transaction needs
to be relayed across the network and included in
the block chain.

prioritymedian and feemedian are values from 0.0
to 1.0, where 0.0 will return the smallest
recently-included-in-a-block priority (or fee) seen,
1.0 the largest, and 0.5 the median priority (or fee)
for transactions that were broadcast on the network and
included in a block.

The default value for prioritymedian (0.1) is
chosen to return a priority for free transactions that
will eventually be confirmed, but might take several hours.
The default value for feemedian (0.5) returns how much
fee you should include to have your transactions confirmed
in an average amount of time.

Values returned are:
 freepriority : priority needed to out-compete a prioritymedian
  fraction of free transactions to be relayed and included in blocks.
 feeperbyte : fee, in satoshis/byte, needed to out-compete a
  feemedian fraction of fee-paying transactions.

Values of -1.0 are returned if not enough transactions
have been seen to make a good estimate.
====

That API doesn't give "30 minute versus 24 hour" confirmation time or
confidence intervals. I've always regretted not taking a statistics class;
if you want to help write code that estimates confidence intervals send me
an email. The API certainly isn't set in stone.

  - Is it globally consistent?
>

Ummm.... roughly, yes, it will be. Nodes that have just joined the network
and haven't seen enough transactions enter and leave the memory pool will
have a different estimate than long-running nodes, but in my testing the
estimate narrows down very quickly (with three or four blocks enough
fee-paying transactions have been seen to make a reasonable estimate; it
takes longer to see enough free transactions to get a good estimate of the
priority needed to get into the free space of a block).

RE: lots of other comments:

I feel like there is a lot of "in the weeds" discussion here about
theoretical, what-if-this-and-that-happens-in-the-future scenarios.

I would just like to point out (again) that this is not intended to be The
One True Solution For Transaction Fees And Transaction Prioritization. If
you've got a better mechanism for estimating fees, fantastic! If it turns
out estimates are often-enough wrong to be a problem and you've got a
solution for that, fantastic!

RE: are we already seeing pressure on transaction fees:

I believe we are, yes. As part of the prep work for the smart fee work I
spent some time plotting priority (for zero-fee transactions) and
transaction fee (for zero-priority transactions) versus confirmation time,
and it looks to me like people/services are starting to include more than
the hard-coded fees in the reference implementation-- I assume because they
want their transactions to be confirmed more quickly.

There is definitely already competition among zero-fee transactions for the
"free" block space. One of the reasons I'm comfortable with the fee changes
I'm proposing is if the estimation code gets it very wrong we'll see that
first as free transactions taking "too long" to confirm, but they'll
confirm eventually because priority increases over time.

-- 
--
Gavin Andresen

--f46d04462e5620652504e999e66d
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">On Fri, Oct 25, 2013 at 5:51 PM, Jeremy Spilman <span dir=
=3D"ltr">&lt;<a href=3D"mailto:jeremy@taplink.co" target=3D"_blank">jeremy@=
taplink.co</a>&gt;</span> wrote:<br><div class=3D"gmail_extra"><div class=
=3D"gmail_quote">
<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-=
left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p=
adding-left:1ex"><u></u>


<div><div>Gavin, can you confirm the best place to =A0read =A0up on the dis=
cuss fee estimation changes for v0.9?</div></div></blockquote><div><br></di=
v><div>The blog post is the best place for high-level overview.</div><div>
<br></div><div>The (closed for now, but it will come back) pull request is =
the best place for low-level details and nit-picking discussion:</div><div>=
=A0=A0<a href=3D"https://github.com/bitcoin/bitcoin/pull/3024">https://gith=
ub.com/bitcoin/bitcoin/pull/3024</a></div>
<div><br></div><div>=A0</div><blockquote class=3D"gmail_quote" style=3D"mar=
gin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,2=
04);border-left-style:solid;padding-left:1ex"><div><div></div><div>I think =
fee estimation at its core is about providing a data point, or even call it=
 an API, which can be used however you see fit.</div>
<div><br></div><div>What parameters do I want to see in a &#39;fee estimati=
on&#39; API?</div><div><br></div><div>=A0- 30 minutes vs 24 hours processin=
g time</div><div>=A0- Confidence Levels (50%/90%)</div></div></blockquote><=
div>
<br></div><div>The pull request adds an &#39;estimatefees&#39; JSON-RPC api=
 call:</div><div><br></div><div>=3D=3D=3D=3D</div><div><div>estimatefees [p=
rioritymedian=3D0.1] [feemedian=3D0.5]</div><div>Estimates the priority or =
fee a transaction needs</div>
<div>to be relayed across the network and included in</div><div>the block c=
hain.</div><div><br></div><div>prioritymedian and feemedian are values from=
 0.0</div><div>to 1.0, where 0.0 will return the smallest</div><div>recentl=
y-included-in-a-block priority (or fee) seen,</div>
<div>1.0 the largest, and 0.5 the median priority (or fee)</div><div>for tr=
ansactions that were broadcast on the network and</div><div>included in a b=
lock.</div><div><br></div><div>The default value for prioritymedian (0.1) i=
s</div>
<div>chosen to return a priority for free transactions that</div><div>will =
eventually be confirmed, but might take several hours.</div><div>The defaul=
t value for feemedian (0.5) returns how much</div><div>fee you should inclu=
de to have your transactions confirmed</div>
<div>in an average amount of time.</div><div><br></div><div>Values returned=
 are:</div><div>=A0freepriority : priority needed to out-compete a priority=
median</div><div>=A0 fraction of free transactions to be relayed and includ=
ed in blocks.</div>
<div>=A0feeperbyte : fee, in satoshis/byte, needed to out-compete a</div><d=
iv>=A0 feemedian fraction of fee-paying transactions.</div><div><br></div><=
div>Values of -1.0 are returned if not enough transactions</div><div>have b=
een seen to make a good estimate.</div>
</div><div>=3D=3D=3D=3D</div><div><br></div><div>That API doesn&#39;t give =
&quot;30 minute versus 24 hour&quot; confirmation time or confidence interv=
als. I&#39;ve always regretted not taking a statistics class; if you want t=
o help write code that estimates confidence intervals send me an email. The=
 API certainly isn&#39;t set in stone.</div>
<div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0p=
x 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-lef=
t-style:solid;padding-left:1ex"><div><div>=A0 - Is it globally consistent?<=
/div>
</div></blockquote><div><br></div><div>Ummm.... roughly, yes, it will be. N=
odes that have just joined the network and haven&#39;t seen enough transact=
ions enter and leave the memory pool will have a different estimate than lo=
ng-running nodes, but in my testing the estimate narrows down very quickly =
(with three or four blocks enough fee-paying transactions have been seen to=
 make a reasonable estimate; it takes longer to see enough free transaction=
s to get a good estimate of the priority needed to get into the free space =
of a block).</div>
<div><br></div></div>RE: lots of other comments:</div><div class=3D"gmail_e=
xtra"><br></div><div class=3D"gmail_extra">I feel like there is a lot of &q=
uot;in the weeds&quot; discussion here about theoretical, what-if-this-and-=
that-happens-in-the-future scenarios.</div>
<div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">I would jus=
t like to point out (again) that this is not intended to be The One True So=
lution For Transaction Fees And Transaction Prioritization. If you&#39;ve g=
ot a better mechanism for estimating fees, fantastic! If it turns out estim=
ates are often-enough wrong to be a problem and you&#39;ve got a solution f=
or that, fantastic!</div>
<div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">RE: are we =
already seeing pressure on transaction fees:</div><div class=3D"gmail_extra=
"><br></div><div class=3D"gmail_extra">I believe we are, yes. As part of th=
e prep work for the smart fee work I spent some time plotting priority (for=
 zero-fee transactions) and transaction fee (for zero-priority transactions=
) versus confirmation time, and it looks to me like people/services are sta=
rting to include more than the hard-coded fees in the reference implementat=
ion-- I assume because they want their transactions to be confirmed more qu=
ickly.</div>
<div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">There is de=
finitely already competition among zero-fee transactions for the &quot;free=
&quot; block space. One of the reasons I&#39;m comfortable with the fee cha=
nges I&#39;m proposing is if the estimation code gets it very wrong we&#39;=
ll see that first as free transactions taking &quot;too long&quot; to confi=
rm, but they&#39;ll confirm eventually because priority increases over time=
.<br clear=3D"all">
<div><br></div>-- <br>--<br>Gavin Andresen<br>
</div></div>

--f46d04462e5620652504e999e66d--