Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id B895FAAC for ; Mon, 5 Oct 2015 20:02:43 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-io0-f180.google.com (mail-io0-f180.google.com [209.85.223.180]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id ACAC5168 for ; Mon, 5 Oct 2015 20:02:41 +0000 (UTC) Received: by ioiz6 with SMTP id z6so198171827ioi.2 for ; Mon, 05 Oct 2015 13:02:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=A7+niDi1wfTk5qhzSR0z/oxwi6zX1i0U2QA68L6pHfg=; b=WPSX6XcMMimgIIx6c16TaNyx6kelSNy3QTj5a3SWgaiCtLlwrNJbzyrERUEF89LsMp HZpL0miFbcwNwpnVYPHxcANflViecFNTaSlvOzllTJxo3SsB/tiYTdEBdPeXoykkKCSI s6d0+9ynG3bxkXgZaj9ruB/JRNAO4wfVj8jKlfw55JzFZbyEuAYGHEiONrP+YbInKPJ1 ippVI1n9zqNpUds1tcl+JAr68aWRONRGQXhqAmGykBP7E8jwEu2mbdB6KJVBznP1lJz1 WUhJcLK5scMpQFVC2WAhjX2u7LHkhFb0wUb0dY7HODBW29c/fKzdZkD5Fma/2tDgHbNj pOjQ== MIME-Version: 1.0 X-Received: by 10.107.10.140 with SMTP id 12mr31498238iok.160.1444075360976; Mon, 05 Oct 2015 13:02:40 -0700 (PDT) Received: by 10.64.106.103 with HTTP; Mon, 5 Oct 2015 13:02:40 -0700 (PDT) In-Reply-To: References: <55D77A7F.40402@mattcorallo.com> Date: Mon, 5 Oct 2015 16:02:40 -0400 Message-ID: From: Alex Morcos To: Danny Thorpe Content-Type: multipart/alternative; boundary=001a113eeccaeb8799052160feaf X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Bitcoin Dev Subject: Re: [bitcoin-dev] Proposed new policy for transactions that depend on other unconfirmed transactions X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Development Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2015 20:02:43 -0000 --001a113eeccaeb8799052160feaf Content-Type: text/plain; charset=UTF-8 Yes, total number of dependent transactions regardless of chain depth. A descendant package means all the transactions that can not be included in a block before the transaction in question. An ancestor package means all the transactions that are required to be included in a block before the transaction in question can be. On Mon, Oct 5, 2015 at 2:51 PM, Danny Thorpe wrote: > What does "package" mean here? > > When you say 25 txs, does that mean maximum linked chain depth, or total > number of dependent transactions regardless of chain depth? > > Thanks, > -Danny > > > > On Mon, Oct 5, 2015 at 11:45 AM, Alex Morcos via bitcoin-dev < > bitcoin-dev@lists.linuxfoundation.org> wrote: > >> I'd like to propose updates to the new policy limits on unconfirmed >> transaction chains. >> >> The existing limits in master and scheduled for release in 0.12 are: >> Ancestor packages = 100 txs and 900kb total size >> Descendant packages = 1000 txs and 2500kb total size >> >> Before 0.12 is released I would like to propose a significant reduction >> in these limits. In the course of analyzing algorithms for mempool >> limiting, it became clear that large packages of unconfirmed transactions >> were the primary vector for mempool clogging or relay fee boosting attacks. >> Feedback from the initial proposed limits was that they were too generous >> anyway. >> >> The proposed new limits are: >> Ancestor packages = 25 txs and 100kb total size >> Descendant packages = 25 txs and 100kb total size >> >> Based on historical transaction data, the most restrictive of these >> limits is the 25 transaction count on descendant packages. Over the period >> of April and May of this year (before stress tests), 5.8% of transactions >> would have violated this limit alone. Applying all the limits together >> would have affected 6.1% of transactions. >> >> Please keep in mind these are policy limits that affect transactions >> which depend on other unconfirmed transactions only. They are not a change >> to consensus rules and do not affect how many chained txs a valid block may >> contain. Furthermore, any transaction that was unable to be relayed due to >> these limits need only wait for some of its unconfirmed ancestors to be >> included in a block and then it could be successfully broadcast. This is >> unlikely to affect the total time from creation to inclusion in a block. >> Finally, these limits are command line arguments that can easily be changed >> on an individual node basis in Bitcoin Core. >> >> Please give your feedback if you know of legitimate use cases that would >> be hindered by these limits. >> >> Thanks, >> Alex >> >> On Mon, Sep 21, 2015 at 11:02 AM, Alex Morcos wrote: >> >>> Thanks for everyone's review. These policy changes have been merged in >>> to master in 6654 , which >>> just implements these limits and no mempool limiting yet. The default >>> ancestor package size limit is 900kb not 1MB. >>> >>> Yes I think these limits are generous, but they were designed to be as >>> generous as was computationally feasible so they were unobjectionable >>> (since the existing policy was no limits). This does not preclude future >>> changes to policy that would reduce these limits. >>> >>> >>> >>> >>> >>> On Fri, Aug 21, 2015 at 3:52 PM, Danny Thorpe >>> wrote: >>> >>>> The limits Alex proposed are generous (bordering on obscene!), but >>>> dropping that down to allowing only two levels of chained unconfirmed >>>> transactions is too tight. >>>> >>>> Use case: Brokered asset transfers may require sets of transactions >>>> with a dependency tree depth of 3 to be published together. ( N seller txs, >>>> 1 broker bridge tx, M buyer txs ) >>>> >>>> If the originally proposed depth limit of 100 does not provide a >>>> sufficient cap on memory consumption or loop/recursion depth, a depth limit >>>> of 10 would provide plenty of headroom for this 3 level use case and >>>> similar patterns. >>>> >>>> -Danny >>>> >>>> On Fri, Aug 21, 2015 at 12:22 PM, Matt Corallo via bitcoin-dev < >>>> bitcoin-dev@lists.linuxfoundation.org> wrote: >>>> >>>>> I dont see any problem with such limits. Though, hell, if you limited >>>>> entire tx dependency trees (ie transactions and all required >>>>> unconfirmed >>>>> transactions for them) to something like 10 txn, maximum two levels >>>>> deep, I also wouldnt have a problem. >>>>> >>>>> Matt >>>>> >>>>> On 08/14/15 19:33, Alex Morcos via bitcoin-dev wrote: >>>>> > Hi everyone, >>>>> > >>>>> > >>>>> > I'd like to propose a new set of requirements as a policy on when to >>>>> > accept new transactions into the mempool and relay them. This policy >>>>> > would affect transactions which have as inputs other transactions >>>>> which >>>>> > are not yet confirmed in the blockchain. >>>>> > >>>>> > The motivation for this policy is 6470 >>>>> > which aims to limit >>>>> the >>>>> > size of a mempool. As discussed in that pull >>>>> > >>>> >, >>>>> > once the mempool is full a new transaction must be able to pay not >>>>> only >>>>> > for the transaction it would evict, but any dependent transactions >>>>> that >>>>> > would be removed from the mempool as well. In order to make sure >>>>> this >>>>> > is always feasible, I'm proposing 4 new policy limits. >>>>> > >>>>> > All limits are command line configurable. >>>>> > >>>>> > The first two limits are required to make sure no chain of >>>>> transactions >>>>> > will be too large for the eviction code to handle: >>>>> > >>>>> > Max number of descendant txs : No transaction shall be accepted if it >>>>> > would cause another transaction in the mempool to have too many >>>>> > descendant transactions (all of which would have to be evicted if the >>>>> > ancestor transaction was evicted). Default: 1000 >>>>> > >>>>> > Max descendant size : No transaction shall be accepted if it would >>>>> cause >>>>> > another transaction in the mempool to have the total size of all its >>>>> > descendant transactions be too great. Default : maxmempool / 200 >>>>> = 2.5MB >>>>> > >>>>> > The third limit is required to make sure calculating the state >>>>> required >>>>> > for sorting and limiting the mempool and enforcing the first 2 >>>>> limits is >>>>> > computationally feasible: >>>>> > >>>>> > Max number of ancestor txs: No transaction shall be accepted if it >>>>> has >>>>> > too many ancestor transactions which are not yet confirmed (ie, in >>>>> the >>>>> > mempool). Default: 100 >>>>> > >>>>> > The fourth limit is required to maintain the pre existing policy goal >>>>> > that all transactions in the mempool should be mineable in the next >>>>> block. >>>>> > >>>>> > Max ancestor size: No transaction shall be accepted if the total >>>>> size of >>>>> > all its unconfirmed ancestor transactions is too large. Default: 1MB >>>>> > >>>>> > (All limits include the transaction itself.) >>>>> > >>>>> > For reference, these limits would have affected less than 2% of >>>>> > transactions entering the mempool in April or May of this year. >>>>> During >>>>> > the period of 7/6 through 7/14, while the network was under stress >>>>> test, >>>>> > as many as 25% of the transactions would have been affected. >>>>> > >>>>> > The code to implement the descendant package tracking and new policy >>>>> > limits can be found in 6557 >>>>> > which is built off >>>>> of 6470. >>>>> > >>>>> > Thanks, >>>>> > Alex >>>>> > >>>>> > >>>>> > >>>>> > _______________________________________________ >>>>> > bitcoin-dev mailing list >>>>> > bitcoin-dev@lists.linuxfoundation.org >>>>> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >>>>> > >>>>> _______________________________________________ >>>>> bitcoin-dev mailing list >>>>> bitcoin-dev@lists.linuxfoundation.org >>>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >>>>> >>>> >>>> >>> >> >> _______________________________________________ >> bitcoin-dev mailing list >> bitcoin-dev@lists.linuxfoundation.org >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >> >> > --001a113eeccaeb8799052160feaf Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Yes, total number of dependent transactions regardless of = chain depth. =C2=A0

A descendant package means all the t= ransactions that can not be included in a block before the transaction in q= uestion.

An ancestor package means all the transac= tions that are required to be included in a block before the transaction in= question can be.



<= div class=3D"gmail_extra">
On Mon, Oct 5, 201= 5 at 2:51 PM, Danny Thorpe <danny.thorpe@gmail.com> wro= te:
What does "pack= age" mean here? =C2=A0

When you say 25 txs, does th= at mean maximum linked chain depth, or total number of dependent transactio= ns regardless of chain depth?

Thanks,
-D= anny



On= Mon, Oct 5, 2015 at 11:45 AM, Alex Morcos via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org> wrote:
I'd like to propose updates t= o the new policy limits on unconfirmed transaction chains.=C2=A0

The existing limits in master and scheduled for release in 0.12 ar= e:=C2=A0
Ancestor packages =3D 100 txs and 900kb total size
=
Descendant packages =3D 1000 txs and 2500kb total size=C2=A0

Before 0.12 is released I would like to propose a significa= nt reduction in these limits. In the course of analyzing algorithms for mem= pool limiting, it became clear that large packages of unconfirmed transacti= ons were the primary vector for mempool clogging or relay fee boosting atta= cks. Feedback from the initial proposed limits was that they were too gener= ous anyway.=C2=A0

The proposed new limits are:=C2= =A0
Ancestor packages =3D 25 txs and 100kb total size
D= escendant packages =3D 25 txs and 100kb total size=C2=A0

Based on historical transaction data, the most restrictive of these = limits is the 25 transaction count on descendant packages. Over the period = of April and May of this year (before stress tests), 5.8% of transactions w= ould have violated this limit alone. Applying all the limits together would= have affected 6.1% of transactions.=C2=A0

Please = keep in mind these are policy limits that affect transactions which depend = on other unconfirmed transactions only. They are not a change to consensus = rules and do not affect how many chained txs a valid block may contain. Fur= thermore, any transaction that was unable to be relayed due to these limits= need only wait for some of its unconfirmed ancestors to be included in a b= lock and then it could be successfully broadcast. This is unlikely to affec= t the total time from creation to inclusion in a block. Finally, these limi= ts are command line arguments that can easily be changed on an individual n= ode basis in Bitcoin Core.=C2=A0

Please give your = feedback if you know of legitimate use cases that would be hindered by thes= e limits.=C2=A0

Thanks,=C2=A0
Alex

On Mon, Sep 21, 2015 at 11:02 AM= , Alex Morcos <morcos@gmail.com> wrote:
Thanks for everyone's review.=C2=A0 Thes= e policy changes have been merged in to master in 6654, which just impl= ements these limits and no mempool limiting yet.=C2=A0 The default ancestor= package size limit is 900kb not 1MB.

Yes I think these = limits are generous, but they were designed to be as generous as was comput= ationally feasible so they were unobjectionable (since the existing policy = was no limits).=C2=A0 This does not preclude future changes to policy that = would reduce these limits.





On Fri, Aug 21, 2015 at 3:52 PM, Danny Thorpe <danny= .thorpe@gmail.com> wrote:
<= div dir=3D"ltr">The limits Alex proposed are generous (bordering on obscene= !), but dropping that down to allowing only two levels of chained unconfirm= ed transactions is too tight. =C2=A0

Use case: Brokered = asset transfers may require sets of transactions with a dependency tree dep= th of 3 to be published together. ( N seller txs, 1 broker bridge tx, M buy= er txs )

If the originally proposed depth limit of= 100 does not provide a sufficient cap on memory consumption or loop/recurs= ion depth, a depth limit of 10 would provide plenty of headroom for this 3 = level use case and similar patterns.

-Danny

On Fri, Aug 21, 2015 at 12:22 PM,= Matt Corallo via bitcoin-dev <bitcoin-dev@lists.linux= foundation.org> wrote:
I do= nt see any problem with such limits. Though, hell, if you limited
entire tx dependency trees (ie transactions and all required unconfirmed transactions for them) to something like 10 txn, maximum two levels
deep, I also wouldnt have a problem.

Matt

On 08/14/15 19:33, Alex Morcos via bitcoin-dev wrote:
> Hi everyone,
>
>
> I'd like to propose a new set of requirements as a policy on when = to
> accept new transactions into the mempool and relay them.=C2=A0 This po= licy
> would affect transactions which have as inputs other transactions whic= h
> are not yet confirmed in the blockchain.
>
> The motivation for this policy is 6470
> <https://github.com/bitcoin/bitcoin/pull/6470> which aims to limit the
> size of a mempool.=C2=A0 As discussed in that pull
> <https://github.com/bitco= in/bitcoin/pull/6470#issuecomment-125324736>,
> once the mempool is full a new transaction must be able to pay not onl= y
> for the transaction it would evict, but any dependent transactions tha= t
> would be removed from the mempool as well.=C2=A0 In order to make sure= this
> is always feasible, I'm proposing 4 new policy limits.
>
> All limits are command line configurable.
>
> The first two limits are required to make sure no chain of transaction= s
> will be too large for the eviction code to handle:
>
> Max number of descendant txs : No transaction shall be accepted if it<= br> > would cause another transaction in the mempool to have too many
> descendant transactions (all of which would have to be evicted if the<= br> > ancestor transaction was evicted).=C2=A0 Default: 1000
>
> Max descendant size : No transaction shall be accepted if it would cau= se
> another transaction in the mempool to have the total size of all its > descendant transactions be too great.=C2=A0 Default : maxmempool / 200= =C2=A0 =3D=C2=A0 2.5MB
>
> The third limit is required to make sure calculating the state require= d
> for sorting and limiting the mempool and enforcing the first 2 limits = is
> computationally feasible:
>
> Max number of ancestor txs:=C2=A0 No transaction shall be accepted if = it has
> too many ancestor transactions which are not yet confirmed (ie, in the=
> mempool). Default: 100
>
> The fourth limit is required to maintain the pre existing policy goal<= br> > that all transactions in the mempool should be mineable in the next bl= ock.
>
> Max ancestor size: No transaction shall be accepted if the total size = of
> all its unconfirmed ancestor transactions is too large.=C2=A0 Default:= 1MB
>
> (All limits include the transaction itself.)
>
> For reference, these limits would have affected less than 2% of
> transactions entering the mempool in April or May of this year.=C2=A0 = During
> the period of 7/6 through 7/14, while the network was under stress tes= t,
> as many as 25% of the transactions would have been affected.
>
> The code to implement the descendant package tracking and new policy > limits can be found in 6557
> <https://github.com/bitcoin/bitcoin/pull/6557> which is built off of 6470.
>
> Thanks,
> Alex
>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org= /mailman/listinfo/bitcoin-dev
>
_______________________________________________
bitcoin-dev mailing list
= bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mail= man/listinfo/bitcoin-dev




_______________________________________________
bitcoin-dev mailing list
= bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mail= man/listinfo/bitcoin-dev



--001a113eeccaeb8799052160feaf--