MIME-Version: 1.0
In-Reply-To: <CAPWm=eVVdyYxePrXur17P=FdMpUvNmByz30hey5=R46PQPhf-Q@mail.gmail.com>
References: <CAPWm=eWuvC8zYM_ipAnaQttKQQG2Vas6np_bAFkxG31eR5w=xQ@mail.gmail.com>
	<55D77A7F.40402@mattcorallo.com>
	<CAJN5wHVzzo-dD6FFyaydEDm27HK2OkWxC0o0Pxcy-N9wTfv8Gw@mail.gmail.com>
	<CAPWm=eW-g9F5YZ9EdqXGzpzvs2mQJ8N5wKG15Ofz4cWGaHQ0BQ@mail.gmail.com>
	<CAPWm=eVVdyYxePrXur17P=FdMpUvNmByz30hey5=R46PQPhf-Q@mail.gmail.com>
Date: Mon, 5 Oct 2015 11:51:23 -0700
Message-ID: <CAJN5wHXJkFefminbnY6O93PUkQKCyLpvM57TCp+4q6p6CEzhzg@mail.gmail.com>
From: Danny Thorpe <danny.thorpe@gmail.com>
To: Alex Morcos <morcos@gmail.com>
Content-Type: multipart/alternative; boundary=001a1147a3b0f0debc05215fff92
Cc: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] Proposed new policy for transactions that depend
 on other unconfirmed transactions
Precedence: list

--001a1147a3b0f0debc05215fff92
Content-Type: text/plain; charset=UTF-8

What does "package" mean here?

When you say 25 txs, does that mean maximum linked chain depth, or total
number of dependent transactions regardless of chain depth?

Thanks,
-Danny



On Mon, Oct 5, 2015 at 11:45 AM, Alex Morcos via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> I'd like to propose updates to the new policy limits on unconfirmed
> transaction chains.
>
> The existing limits in master and scheduled for release in 0.12 are:
> Ancestor packages = 100 txs and 900kb total size
> Descendant packages = 1000 txs and 2500kb total size
>
> Before 0.12 is released I would like to propose a significant reduction in
> these limits. In the course of analyzing algorithms for mempool limiting,
> it became clear that large packages of unconfirmed transactions were the
> primary vector for mempool clogging or relay fee boosting attacks. Feedback
> from the initial proposed limits was that they were too generous anyway.
>
> The proposed new limits are:
> Ancestor packages = 25 txs and 100kb total size
> Descendant packages = 25 txs and 100kb total size
>
> Based on historical transaction data, the most restrictive of these limits
> is the 25 transaction count on descendant packages. Over the period of
> April and May of this year (before stress tests), 5.8% of transactions
> would have violated this limit alone. Applying all the limits together
> would have affected 6.1% of transactions.
>
> Please keep in mind these are policy limits that affect transactions which
> depend on other unconfirmed transactions only. They are not a change to
> consensus rules and do not affect how many chained txs a valid block may
> contain. Furthermore, any transaction that was unable to be relayed due to
> these limits need only wait for some of its unconfirmed ancestors to be
> included in a block and then it could be successfully broadcast. This is
> unlikely to affect the total time from creation to inclusion in a block.
> Finally, these limits are command line arguments that can easily be changed
> on an individual node basis in Bitcoin Core.
>
> Please give your feedback if you know of legitimate use cases that would
> be hindered by these limits.
>
> Thanks,
> Alex
>
> On Mon, Sep 21, 2015 at 11:02 AM, Alex Morcos <morcos@gmail.com> wrote:
>
>> Thanks for everyone's review.  These policy changes have been merged in
>> to master in 6654 <https://github.com/bitcoin/bitcoin/pull/6654>, which
>> just implements these limits and no mempool limiting yet.  The default
>> ancestor package size limit is 900kb not 1MB.
>>
>> Yes I think these limits are generous, but they were designed to be as
>> generous as was computationally feasible so they were unobjectionable
>> (since the existing policy was no limits).  This does not preclude future
>> changes to policy that would reduce these limits.
>>
>>
>>
>>
>>
>> On Fri, Aug 21, 2015 at 3:52 PM, Danny Thorpe <danny.thorpe@gmail.com>
>> wrote:
>>
>>> The limits Alex proposed are generous (bordering on obscene!), but
>>> dropping that down to allowing only two levels of chained unconfirmed
>>> transactions is too tight.
>>>
>>> Use case: Brokered asset transfers may require sets of transactions with
>>> a dependency tree depth of 3 to be published together. ( N seller txs, 1
>>> broker bridge tx, M buyer txs )
>>>
>>> If the originally proposed depth limit of 100 does not provide a
>>> sufficient cap on memory consumption or loop/recursion depth, a depth limit
>>> of 10 would provide plenty of headroom for this 3 level use case and
>>> similar patterns.
>>>
>>> -Danny
>>>
>>> On Fri, Aug 21, 2015 at 12:22 PM, Matt Corallo via bitcoin-dev <
>>> bitcoin-dev@lists.linuxfoundation.org> wrote:
>>>
>>>> I dont see any problem with such limits. Though, hell, if you limited
>>>> entire tx dependency trees (ie transactions and all required unconfirmed
>>>> transactions for them) to something like 10 txn, maximum two levels
>>>> deep, I also wouldnt have a problem.
>>>>
>>>> Matt
>>>>
>>>> On 08/14/15 19:33, Alex Morcos via bitcoin-dev wrote:
>>>> > Hi everyone,
>>>> >
>>>> >
>>>> > I'd like to propose a new set of requirements as a policy on when to
>>>> > accept new transactions into the mempool and relay them.  This policy
>>>> > would affect transactions which have as inputs other transactions
>>>> which
>>>> > are not yet confirmed in the blockchain.
>>>> >
>>>> > The motivation for this policy is 6470
>>>> > <https://github.com/bitcoin/bitcoin/pull/6470> which aims to limit
>>>> the
>>>> > size of a mempool.  As discussed in that pull
>>>> > <https://github.com/bitcoin/bitcoin/pull/6470#issuecomment-125324736
>>>> >,
>>>> > once the mempool is full a new transaction must be able to pay not
>>>> only
>>>> > for the transaction it would evict, but any dependent transactions
>>>> that
>>>> > would be removed from the mempool as well.  In order to make sure this
>>>> > is always feasible, I'm proposing 4 new policy limits.
>>>> >
>>>> > All limits are command line configurable.
>>>> >
>>>> > The first two limits are required to make sure no chain of
>>>> transactions
>>>> > will be too large for the eviction code to handle:
>>>> >
>>>> > Max number of descendant txs : No transaction shall be accepted if it
>>>> > would cause another transaction in the mempool to have too many
>>>> > descendant transactions (all of which would have to be evicted if the
>>>> > ancestor transaction was evicted).  Default: 1000
>>>> >
>>>> > Max descendant size : No transaction shall be accepted if it would
>>>> cause
>>>> > another transaction in the mempool to have the total size of all its
>>>> > descendant transactions be too great.  Default : maxmempool / 200  =
>>>> 2.5MB
>>>> >
>>>> > The third limit is required to make sure calculating the state
>>>> required
>>>> > for sorting and limiting the mempool and enforcing the first 2 limits
>>>> is
>>>> > computationally feasible:
>>>> >
>>>> > Max number of ancestor txs:  No transaction shall be accepted if it
>>>> has
>>>> > too many ancestor transactions which are not yet confirmed (ie, in the
>>>> > mempool). Default: 100
>>>> >
>>>> > The fourth limit is required to maintain the pre existing policy goal
>>>> > that all transactions in the mempool should be mineable in the next
>>>> block.
>>>> >
>>>> > Max ancestor size: No transaction shall be accepted if the total size
>>>> of
>>>> > all its unconfirmed ancestor transactions is too large.  Default: 1MB
>>>> >
>>>> > (All limits include the transaction itself.)
>>>> >
>>>> > For reference, these limits would have affected less than 2% of
>>>> > transactions entering the mempool in April or May of this year.
>>>> During
>>>> > the period of 7/6 through 7/14, while the network was under stress
>>>> test,
>>>> > as many as 25% of the transactions would have been affected.
>>>> >
>>>> > The code to implement the descendant package tracking and new policy
>>>> > limits can be found in 6557
>>>> > <https://github.com/bitcoin/bitcoin/pull/6557> which is built off of
>>>> 6470.
>>>> >
>>>> > Thanks,
>>>> > Alex
>>>> >
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > bitcoin-dev mailing list
>>>> > bitcoin-dev@lists.linuxfoundation.org
>>>> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>> >
>>>> _______________________________________________
>>>> bitcoin-dev mailing list
>>>> bitcoin-dev@lists.linuxfoundation.org
>>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>>
>>>
>>>
>>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>

--001a1147a3b0f0debc05215fff92
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">What does &quot;package&quot; mean here? =C2=A0<div><br></=
div><div>When you say 25 txs, does that mean maximum linked chain depth, or=
 total number of dependent transactions regardless of chain depth?</div><di=
v><br></div><div>Thanks,</div><div>-Danny<br><div><br></div><div><br></div>=
</div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Mo=
n, Oct 5, 2015 at 11:45 AM, Alex Morcos via bitcoin-dev <span dir=3D"ltr">&=
lt;<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blan=
k">bitcoin-dev@lists.linuxfoundation.org</a>&gt;</span> wrote:<br><blockquo=
te class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc so=
lid;padding-left:1ex"><div dir=3D"ltr">I&#39;d like to propose updates to t=
he new policy limits on unconfirmed transaction chains.=C2=A0<div><br></div=
><div>The existing limits in master and scheduled for release in 0.12 are:=
=C2=A0</div><div>Ancestor packages =3D 100 txs and 900kb total size</div><d=
iv>Descendant packages =3D 1000 txs and 2500kb total size=C2=A0</div><div><=
br></div><div>Before 0.12 is released I would like to propose a significant=
 reduction in these limits. In the course of analyzing algorithms for mempo=
ol limiting, it became clear that large packages of unconfirmed transaction=
s were the primary vector for mempool clogging or relay fee boosting attack=
s. Feedback from the initial proposed limits was that they were too generou=
s anyway.=C2=A0</div><div><br></div><div>The proposed new limits are:=C2=A0=
</div><div>Ancestor packages =3D 25 txs and 100kb total size</div><div>Desc=
endant packages =3D 25 txs and 100kb total size=C2=A0</div><div><br></div><=
div>Based on historical transaction data, the most restrictive of these lim=
its is the 25 transaction count on descendant packages. Over the period of =
April and May of this year (before stress tests), 5.8% of transactions woul=
d have violated this limit alone. Applying all the limits together would ha=
ve affected 6.1% of transactions.=C2=A0</div><div><br></div><div>Please kee=
p in mind these are policy limits that affect transactions which depend on =
other unconfirmed transactions only. They are not a change to consensus rul=
es and do not affect how many chained txs a valid block may contain. Furthe=
rmore, any transaction that was unable to be relayed due to these limits ne=
ed only wait for some of its unconfirmed ancestors to be included in a bloc=
k and then it could be successfully broadcast. This is unlikely to affect t=
he total time from creation to inclusion in a block. Finally, these limits =
are command line arguments that can easily be changed on an individual node=
 basis in Bitcoin Core.=C2=A0</div><div><br></div><div>Please give your fee=
dback if you know of legitimate use cases that would be hindered by these l=
imits.=C2=A0</div><div><br></div><div>Thanks,=C2=A0</div><span class=3D"HOE=
nZb"><font color=3D"#888888"><div>Alex</div></font></span></div><div class=
=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail_extra"><br><div class=3D"=
gmail_quote">On Mon, Sep 21, 2015 at 11:02 AM, Alex Morcos <span dir=3D"ltr=
">&lt;<a href=3D"mailto:morcos@gmail.com" target=3D"_blank">morcos@gmail.co=
m</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margi=
n:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">=
Thanks for everyone&#39;s review.=C2=A0 These policy changes have been merg=
ed in to master in <a href=3D"https://github.com/bitcoin/bitcoin/pull/6654"=
 target=3D"_blank">6654</a>, which just implements these limits and no memp=
ool limiting yet.=C2=A0 The default ancestor package size limit is 900kb no=
t 1MB.<div><br></div><div>Yes I think these limits are generous, but they w=
ere designed to be as generous as was computationally feasible so they were=
 unobjectionable (since the existing policy was no limits).=C2=A0 This does=
 not preclude future changes to policy that would reduce these limits.</div=
><div><br></div><div><br></div><div><br></div><div><br></div></div><div><di=
v><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Fri, Aug 21,=
 2015 at 3:52 PM, Danny Thorpe <span dir=3D"ltr">&lt;<a href=3D"mailto:dann=
y.thorpe@gmail.com" target=3D"_blank">danny.thorpe@gmail.com</a>&gt;</span>=
 wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bor=
der-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">The limits Alex =
proposed are generous (bordering on obscene!), but dropping that down to al=
lowing only two levels of chained unconfirmed transactions is too tight. =
=C2=A0<div><br></div><div>Use case: Brokered asset transfers may require se=
ts of transactions with a dependency tree depth of 3 to be published togeth=
er. ( N seller txs, 1 broker bridge tx, M buyer txs )</div><div><br></div><=
div>If the originally proposed depth limit of 100 does not provide a suffic=
ient cap on memory consumption or loop/recursion depth, a depth limit of 10=
 would provide plenty of headroom for this 3 level use case and similar pat=
terns.</div><span><font color=3D"#888888"><div><br></div><div>-Danny</div><=
/font></span></div><div><div><div class=3D"gmail_extra"><br><div class=3D"g=
mail_quote">On Fri, Aug 21, 2015 at 12:22 PM, Matt Corallo via bitcoin-dev =
<span dir=3D"ltr">&lt;<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.o=
rg" target=3D"_blank">bitcoin-dev@lists.linuxfoundation.org</a>&gt;</span> =
wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bord=
er-left:1px #ccc solid;padding-left:1ex">I dont see any problem with such l=
imits. Though, hell, if you limited<br>
entire tx dependency trees (ie transactions and all required unconfirmed<br=
>
transactions for them) to something like 10 txn, maximum two levels<br>
deep, I also wouldnt have a problem.<br>
<br>
Matt<br>
<br>
On 08/14/15 19:33, Alex Morcos via bitcoin-dev wrote:<br>
&gt; Hi everyone,<br>
&gt;<br>
&gt;<br>
&gt; I&#39;d like to propose a new set of requirements as a policy on when =
to<br>
&gt; accept new transactions into the mempool and relay them.=C2=A0 This po=
licy<br>
&gt; would affect transactions which have as inputs other transactions whic=
h<br>
&gt; are not yet confirmed in the blockchain.<br>
&gt;<br>
&gt; The motivation for this policy is 6470<br>
&gt; &lt;<a href=3D"https://github.com/bitcoin/bitcoin/pull/6470" rel=3D"no=
referrer" target=3D"_blank">https://github.com/bitcoin/bitcoin/pull/6470</a=
>&gt; which aims to limit the<br>
&gt; size of a mempool.=C2=A0 As discussed in that pull<br>
&gt; &lt;<a href=3D"https://github.com/bitcoin/bitcoin/pull/6470#issuecomme=
nt-125324736" rel=3D"noreferrer" target=3D"_blank">https://github.com/bitco=
in/bitcoin/pull/6470#issuecomment-125324736</a>&gt;,<br>
&gt; once the mempool is full a new transaction must be able to pay not onl=
y<br>
&gt; for the transaction it would evict, but any dependent transactions tha=
t<br>
&gt; would be removed from the mempool as well.=C2=A0 In order to make sure=
 this<br>
&gt; is always feasible, I&#39;m proposing 4 new policy limits.<br>
&gt;<br>
&gt; All limits are command line configurable.<br>
&gt;<br>
&gt; The first two limits are required to make sure no chain of transaction=
s<br>
&gt; will be too large for the eviction code to handle:<br>
&gt;<br>
&gt; Max number of descendant txs : No transaction shall be accepted if it<=
br>
&gt; would cause another transaction in the mempool to have too many<br>
&gt; descendant transactions (all of which would have to be evicted if the<=
br>
&gt; ancestor transaction was evicted).=C2=A0 Default: 1000<br>
&gt;<br>
&gt; Max descendant size : No transaction shall be accepted if it would cau=
se<br>
&gt; another transaction in the mempool to have the total size of all its<b=
r>
&gt; descendant transactions be too great.=C2=A0 Default : maxmempool / 200=
=C2=A0 =3D=C2=A0 2.5MB<br>
&gt;<br>
&gt; The third limit is required to make sure calculating the state require=
d<br>
&gt; for sorting and limiting the mempool and enforcing the first 2 limits =
is<br>
&gt; computationally feasible:<br>
&gt;<br>
&gt; Max number of ancestor txs:=C2=A0 No transaction shall be accepted if =
it has<br>
&gt; too many ancestor transactions which are not yet confirmed (ie, in the=
<br>
&gt; mempool). Default: 100<br>
&gt;<br>
&gt; The fourth limit is required to maintain the pre existing policy goal<=
br>
&gt; that all transactions in the mempool should be mineable in the next bl=
ock.<br>
&gt;<br>
&gt; Max ancestor size: No transaction shall be accepted if the total size =
of<br>
&gt; all its unconfirmed ancestor transactions is too large.=C2=A0 Default:=
 1MB<br>
&gt;<br>
&gt; (All limits include the transaction itself.)<br>
&gt;<br>
&gt; For reference, these limits would have affected less than 2% of<br>
&gt; transactions entering the mempool in April or May of this year.=C2=A0 =
During<br>
&gt; the period of 7/6 through 7/14, while the network was under stress tes=
t,<br>
&gt; as many as 25% of the transactions would have been affected.<br>
&gt;<br>
&gt; The code to implement the descendant package tracking and new policy<b=
r>
&gt; limits can be found in 6557<br>
&gt; &lt;<a href=3D"https://github.com/bitcoin/bitcoin/pull/6557" rel=3D"no=
referrer" target=3D"_blank">https://github.com/bitcoin/bitcoin/pull/6557</a=
>&gt; which is built off of 6470.<br>
&gt;<br>
&gt; Thanks,<br>
&gt; Alex<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; _______________________________________________<br>
&gt; bitcoin-dev mailing list<br>
&gt; <a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_bl=
ank">bitcoin-dev@lists.linuxfoundation.org</a><br>
&gt; <a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-=
dev" rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org=
/mailman/listinfo/bitcoin-dev</a><br>
&gt;<br>
_______________________________________________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">=
bitcoin-dev@lists.linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" =
rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail=
man/listinfo/bitcoin-dev</a><br>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.=
linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" =
rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail=
man/listinfo/bitcoin-dev</a><br>
<br></blockquote></div><br></div>

--001a1147a3b0f0debc05215fff92--