1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
|
Received: from sog-mx-3.v43.ch3.sourceforge.com ([172.29.43.193]
helo=mx.sourceforge.net)
by sfs-ml-4.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
(envelope-from <morcos@gmail.com>) id 1Xj8B3-00022V-BA
for bitcoin-development@lists.sourceforge.net;
Tue, 28 Oct 2014 14:55:17 +0000
Received-SPF: pass (sog-mx-3.v43.ch3.sourceforge.com: domain of gmail.com
designates 209.85.213.182 as permitted sender)
client-ip=209.85.213.182; envelope-from=morcos@gmail.com;
helo=mail-ig0-f182.google.com;
Received: from mail-ig0-f182.google.com ([209.85.213.182])
by sog-mx-3.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
(Exim 4.76) id 1Xj8B2-0008VR-7U
for bitcoin-development@lists.sourceforge.net;
Tue, 28 Oct 2014 14:55:17 +0000
Received: by mail-ig0-f182.google.com with SMTP id hn18so1168234igb.15
for <bitcoin-development@lists.sourceforge.net>;
Tue, 28 Oct 2014 07:55:11 -0700 (PDT)
MIME-Version: 1.0
X-Received: by 10.42.190.196 with SMTP id dj4mr4286412icb.35.1414508110899;
Tue, 28 Oct 2014 07:55:10 -0700 (PDT)
Received: by 10.50.223.146 with HTTP; Tue, 28 Oct 2014 07:55:10 -0700 (PDT)
In-Reply-To: <CAPWm=eX0MMBOPvugETxq+pyDzZ00xc90hZAJe8qgg4Shftm-9w@mail.gmail.com>
References: <CAPWm=eXxs=AfFhaT2EeGFsR+2r96WcaOeWL_Z59-6LixH+=4AQ@mail.gmail.com>
<CABsx9T35NdEkFmdVDX19gOO1p0h1M_ZDK1iXxTFNLHE9dtC3hQ@mail.gmail.com>
<CAPWm=eX0MMBOPvugETxq+pyDzZ00xc90hZAJe8qgg4Shftm-9w@mail.gmail.com>
Date: Tue, 28 Oct 2014 10:55:10 -0400
Message-ID: <CAPWm=eWA7ZnpE-zQ8snKnSiA0YtBowUzibtWW7fRhvoRoVDeLg@mail.gmail.com>
From: Alex Morcos <morcos@gmail.com>
To: Gavin Andresen <gavinandresen@gmail.com>
Content-Type: multipart/alternative; boundary=20cf303f6e0c7b9b3005067cd516
X-Spam-Score: -0.6 (/)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
See http://spamassassin.org/tag/ for more details.
-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
sender-domain
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(morcos[at]gmail.com)
-0.0 SPF_PASS SPF: sender matches SPF record
1.0 HTML_MESSAGE BODY: HTML included in message
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
author's domain
0.1 DKIM_SIGNED Message has a DKIM or DK signature,
not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
X-Headers-End: 1Xj8B2-0008VR-7U
Cc: Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Subject: Re: [Bitcoin-development] Reworking the policy estimation code (fee
estimates)
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Tue, 28 Oct 2014 14:55:17 -0000
--20cf303f6e0c7b9b3005067cd516
Content-Type: text/plain; charset=UTF-8
Sorry, perhaps I misinterpreted that question. The estimates will be
dominated by the prevailing transaction rates initially, so the estimates
you get for something like "what is the least I can pay and still be 90%
sure I get confirmed in 20 blocks" won't be insane, but they will still be
way too conservative. I'm not sure what you meant by reasonable. You
won't get the "correct" answer of something significantly less than 40k
sat/kB for quite some time. Given that the half-life of the decay is 2.5
days, then within a couple of days. And in fact even in the steady state,
the new code will still return a much higher rate than the existing code,
say 10k sat/kB instead of 1k sat/kB, but that's just a result of the
sorting the existing code does and the fact that no one places transactions
with that small fee. To correctly give such low answers, the new code
will require that those super low feerate transactions are occurring
frequently enough, but the bar for enough datapoints in a feerate bucket is
pretty low, an average of 1 tx per block. The bar can be made lower at the
expense of a bit of noisiness in the answers, for instance for priorities I
had to make the bar significantly lower because there are so many fewer
transactions confirmed because of priorities. I'm certainly open to tuning
some of these variables.
On Tue, Oct 28, 2014 at 10:30 AM, Alex Morcos <morcos@gmail.com> wrote:
> Oh in just a couple of blocks, it'll give you a somewhat reasonable
> estimate for asking about every confirmation count other than 1, but it
> could take several hours for it to have enough data points to give you a
> good estimate for getting confirmed in one block (because the prevalent
> feerate is not always confirmed in 1 block >80% of the time) Essentially
> what it does is just combine buckets until it has enough data points, so
> after the first block it might be treating all of the txs as belonging to
> the same feerate bucket, but since the answer it returns is the "median"*
> fee rate for that bucket, its a reasonable answer right off the get go.
>
> Do you think it would make sense to make that 90% number an argument to
> rpc call? For instance there could be a default (I would use 80%) but then
> you could specify if you required a different certainty. It wouldn't
> require any code changes and might make it easier for people to build more
> complicated logic on top of it.
>
> *It can't actually track the median, but it identifies which of the
> smaller actual buckets the median would have fallen into and returns the
> average feerate for that median bucket.
>
>
>
>
>
> On Tue, Oct 28, 2014 at 9:59 AM, Gavin Andresen <gavinandresen@gmail.com>
> wrote:
>
>> I think Alex's approach is better; I don't think we can know how much
>> better until we have a functioning fee market.
>>
>> We don't have a functioning fee market now, because fees are hard-coded.
>> So we get "pay the hard-coded fee and you'll get confirmed in one or two or
>> three blocks, depending on which miners mine the next three blocks and what
>> time of day it is."
>>
>> git HEAD code says you need a fee of 10,0000 satoshis/kb to be pretty
>> sure you'll get confirmed in the next block. That looks about right with
>> Alex's real-world data (if we take "90% chance" as 'pretty sure you'll get
>> confirmed'):
>>
>> Fee rate 100000 Avg blocks to confirm 1.09 NumBlocks:% confirmed 1: 0.901
>> 2: 1.0 3: 1.0
>>
>> My only concern with Alex's code is that it takes much longer to get
>> 'primed' -- Alex, if I started with no data about fees, how long would it
>> take to be able to get enough data for a reasonable estimate of "what is
>> the least I can pay and still be 90% sure I get confirmed in 20 blocks" ?
>> Hours? Days? Weeks?
>>
>> --
>> --
>> Gavin Andresen
>>
>
>
--20cf303f6e0c7b9b3005067cd516
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Sorry, perhaps I misinterpreted that question.=C2=A0 The e=
stimates will be dominated by the prevailing transaction rates initially, s=
o the estimates you get for something like=C2=A0<font face=3D"arial, sans-s=
erif">"what is the least I can pay and still be 90% sure I get confirm=
ed in 20 blocks"=C2=A0 won't be insane, but they will still be way=
too conservative.=C2=A0 I'm not sure what you meant by reasonable.=C2=
=A0 You won't get the "correct" answer of something significa=
ntly less than 40k sat/kB for quite some time.=C2=A0 Given that the half-li=
fe of the decay is 2.5 days, then within a couple of days.=C2=A0 And in fac=
t even in the steady state, the new code will still return a much higher ra=
te than the existing code, say 10k sat/kB instead of 1k sat/kB, but that=
9;s just a result of the sorting the existing code does and the fact that n=
o one places transactions with that small fee. =C2=A0 To correctly give suc=
h low answers, the new code will require that those super low feerate trans=
actions are occurring frequently enough, but the bar for enough datapoints =
in a feerate bucket is pretty low, an average of 1 tx per block.=C2=A0 The =
bar can be made lower at the expense of a bit of noisiness in the answers, =
for instance for priorities I had to make the bar=C2=A0significantly=C2=A0l=
ower because there are so many fewer transactions confirmed=C2=A0because of=
priorities.=C2=A0 I'm certainly open to tuning some of these variables=
.</font><div><font face=3D"arial, sans-serif"><br></font></div><div><font f=
ace=3D"arial, sans-serif"><br></font><div><span style=3D"font-family:arial,=
sans-serif;font-size:13px"><br></span></div><div><span style=3D"font-family=
:arial,sans-serif;font-size:13px"><br></span></div></div></div><div class=
=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, Oct 28, 2014 at 10:=
30 AM, Alex Morcos <span dir=3D"ltr"><<a href=3D"mailto:morcos@gmail.com=
" target=3D"_blank">morcos@gmail.com</a>></span> wrote:<br><blockquote c=
lass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;=
padding-left:1ex"><div dir=3D"ltr">Oh in just a couple of blocks, it'll=
give you a somewhat reasonable estimate for asking about every confirmatio=
n count other than 1, but it could take several hours for it to have enough=
data points to give you a good estimate for getting confirmed in one block=
(because the prevalent feerate is not always confirmed in 1 block >80% =
of the time) =C2=A0 Essentially what it does is just combine buckets until =
it has enough data points, so after the first block it might be treating al=
l of the txs as belonging to the same feerate bucket, but since the answer =
it returns is the "median"* fee rate for that bucket, its a reaso=
nable answer right off the get go.=C2=A0<div><br></div><div>Do you think it=
would make sense to make that 90% number an argument to rpc call?=C2=A0 Fo=
r instance there could be a default (I would use 80%) but then you could sp=
ecify if you required a different certainty.=C2=A0 It wouldn't require =
any code changes and might make it easier for people to build more complica=
ted logic on top of it.<br><div><br></div><div>*It can't actually track=
the median, but it identifies which of the smaller actual buckets the medi=
an would have fallen into and returns the average feerate for that median b=
ucket.</div><div><br></div><div><br></div><div><br></div><div><br></div></d=
iv></div><div class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail_extra"=
><br><div class=3D"gmail_quote">On Tue, Oct 28, 2014 at 9:59 AM, Gavin Andr=
esen <span dir=3D"ltr"><<a href=3D"mailto:gavinandresen@gmail.com" targe=
t=3D"_blank">gavinandresen@gmail.com</a>></span> wrote:<br><blockquote c=
lass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;=
padding-left:1ex"><div dir=3D"ltr"><div class=3D"gmail_extra">I think Alex&=
#39;s approach is better; I don't think we can know how much better unt=
il we have a functioning fee market.</div><div class=3D"gmail_extra"><br></=
div><div class=3D"gmail_extra">We don't have a functioning fee market n=
ow, because fees are hard-coded. So we get "pay the hard-coded fee and=
you'll get confirmed in one or two or three blocks, depending on which=
miners mine the next three blocks and what time of day it is."</div><=
div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">git HEAD cod=
e says you need a fee of 10,0000 satoshis/kb to be pretty sure you'll g=
et confirmed in the next block. That looks about right with Alex's real=
-world data (if we take "90% chance" as 'pretty sure you'=
ll get confirmed'):</div><span><div class=3D"gmail_extra"><br></div><di=
v class=3D"gmail_extra"><div style=3D"font-family:arial,sans-serif;font-siz=
e:13px"><font face=3D"courier new, monospace">Fee rate 100000 Avg blocks to=
confirm 1.09 NumBlocks:% confirmed 1: 0.901 2: 1.0 =C2=A0 3: 1.0</font></d=
iv><div><font face=3D"courier new, monospace"><br></font></div></div></span=
><div class=3D"gmail_extra">My only concern with Alex's code is that it=
takes much longer to get 'primed' -- Alex, if I started with no da=
ta about fees, how long would it take to be able to get enough data for a r=
easonable estimate of "what is the least I can pay and still be 90% su=
re I get confirmed in 20 blocks" ? Hours? Days? Weeks?<span><font colo=
r=3D"#888888"><br clear=3D"all"><div><br></div>-- <br>--<br>Gavin Andresen<=
br>
</font></span></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
--20cf303f6e0c7b9b3005067cd516--
|
