Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 3D2B567 for ; Sun, 18 Oct 2015 15:59:16 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from d.mail.sonic.net (d.mail.sonic.net [64.142.111.50]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 8BAB2F5 for ; Sun, 18 Oct 2015 15:59:15 +0000 (UTC) Received: from [192.168.1.190] (63.135.62.197.nwinternet.com [63.135.62.197] (may be forged)) (authenticated bits=0) by d.mail.sonic.net (8.15.1/8.15.1) with ESMTPSA id t9IFxBdk025659 (version=TLSv1 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Sun, 18 Oct 2015 08:59:12 -0700 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Content-Type: multipart/signed; boundary="Apple-Mail=_818AD798-4BDF-41A6-A9DB-8EF5B45DC674"; protocol="application/pgp-signature"; micalg=pgp-sha512 X-Pgp-Agent: GPGMail 2.5.2 From: Jonathan Toomim In-Reply-To: <20151014093913.GB19607@amethyst.visucore.com> Date: Sun, 18 Oct 2015 08:59:11 -0700 Message-Id: References: <99C42DE7-814A-48F8-AB28-A5ADD77A9FD9@toom.im> <20151014093913.GB19607@amethyst.visucore.com> To: Bitcoin Dev X-Mailer: Apple Mail (2.1878.6) X-Sonic-CAuth: UmFuZG9tSVaH4ltZBozHiZKlWjc4I41+A1rFRRQziasL5jdC+2o2HOpvfsnWXbPXiZJwLiimOjO4IPaC0iqbCP6zBGLeFfhV X-Sonic-ID: C;RGdqLLF15RGrb+K7sH9FTg== M;5mbSLLF15RGrb+K7sH9FTg== X-Sonic-Spam-Details: 1.7/5.0 by cerberusd X-Spam-Status: No, score=-0.9 required=5.0 tests=BAYES_00,HTML_MESSAGE, RCVD_IN_DNSWL_LOW,URIBL_BLACK autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: bitcoin-xt Subject: Re: [bitcoin-dev] Memory leaks? X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Development Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Oct 2015 15:59:16 -0000 --Apple-Mail=_818AD798-4BDF-41A6-A9DB-8EF5B45DC674 Content-Type: multipart/alternative; boundary="Apple-Mail=_9C963CE6-6CA2-4ABE-9D61-E57718ABA125" --Apple-Mail=_9C963CE6-6CA2-4ABE-9D61-E57718ABA125 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan = wrote: > This is *most likely* the mempool, but is just not reported correctly. I did some testing with PR #6410's better mempool reporting. The = improved reporting suggests that actual in-memory usage ("usage":) by = CTxMemPool is about 2.5x to 3x higher than the serialized transaction = sizes ("bytes":). The excess memory usage that I'm seeing is on the = order of 100x higher than the mempool "bytes": value. As such, I think = it's unlikely that this is the mempool, or at least not normal/correct = mempool behavior. Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm = guessing his bitcoind has been running longer than any of mine. His = server definitely has more RAM. I don't know which email list he is = subscribed to (probably XT), so I'm sharing it with both lists to make = sure you're all aware of how big an issue this can be. > In the meantime you can mitigate the mempool growth by setting = `-mintxfee`, see > = https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#trans= action-flooding I have mintxfee and minrelaytxfee set to about 0.00003, which is high = enough to exclude essentially all of the of the 14700-14800 byte flood = transactions. My nodes' mempools only contain about one or two blocks' = worth of transactions. So I don't think this is correct either. Some additional notes on this issue: 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran = a Core bitcoind process (commit d78a880) overnight with no mining = connected to it, and (IIRC -- my memory is fuzzy) when I woke up it was = using around 400 MB of RSS and the mempool was at around "bytes":10MB, = "usage": 25MB. I ran ./bitcoin-cli getblocktemplate once, and IIRC the = RSS shot up to around 800 MB. I then ran getblocktemplate every 5 = seconds for about 30 minutes, and RSS climbed to 1180 MB. An hour after = that with more getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 = MB about 30 minutes later.] getmempoolinfo is still showing "usage" = around 25MB or less. I'll do some more testing with this and see if I can make it repeatable, = and record the results more carefully. Expect a follow-up from me in a = day or two. 2. valgrind did not show anything super promising. It did report this: =3D=3D6880=3D=3D LEAK SUMMARY: =3D=3D6880=3D=3D definitely lost: 0 bytes in 0 blocks =3D=3D6880=3D=3D indirectly lost: 0 bytes in 0 blocks =3D=3D6880=3D=3D possibly lost: 288 bytes in 1 blocks =3D=3D6880=3D=3D still reachable: 10,552 bytes in 39 blocks =3D=3D6880=3D=3D suppressed: 0 bytes in 0 blocks (Bitcoin Core commit d78a880) and this: =3D=3D6778=3D=3D LEAK SUMMARY: =3D=3D6778=3D=3D definitely lost: 0 bytes in 0 blocks =3D=3D6778=3D=3D indirectly lost: 0 bytes in 0 blocks =3D=3D6778=3D=3D possibly lost: 320 bytes in 1 blocks =3D=3D6778=3D=3D still reachable: 10,080 bytes in 32 blocks =3D=3D6778=3D=3D suppressed: 0 bytes in 0 blocks (Bitcoin XT commit fe446d) I haven't found anything in there yet that I think would produce the = multi-GB memory usage after running for a few days, but I could be = missing it. Email me if you want the full log. I did not try running getblocktemplate while valgrind was running. I'll = have to try that. I also have not let valgrind run for more than an = hour. P.S.: Sorry for all the cross-post confusion and consequent flamewar = fallout. While it's probably too late for this thread, I'll make sure to = post in a manner that keeps the threads clearly separate in the future = (e.g. different subject lines). --Apple-Mail=_9C963CE6-6CA2-4ABE-9D61-E57718ABA125 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
On Oct 14, 2015, at 2:39 AM, Wladimir J. = van der Laan <laanwj@gmail.com> = wrote:
This is *most likely* the mempool, = but is just not reported = correctly.

I did some testing with = PR #6410's better mempool reporting. The improved reporting suggests = that actual in-memory usage ("usage":) by CTxMemPool is about 2.5x to 3x = higher than the serialized transaction sizes ("bytes":). The excess = memory usage that I'm seeing is on the order of 100x higher than the = mempool "bytes": value. As such, I think it's unlikely that this is the = mempool, or at least not normal/correct mempool = behavior. 

Another user (admin@multipool.us) reported 35 = GB of RSS usage. I'm guessing his bitcoind has been running longer than = any of mine. His server definitely has more RAM. I don't know which = email list he is subscribed to (probably XT), so I'm sharing it with = both lists to make sure you're all aware of how big an issue this can = be.

In the meantime you can mitigate = the mempool growth by setting `-mintxfee`, see
https://github.com/bitcoin/bitcoin/blob/v0.11.0/d= oc/release-notes.md#transaction-flooding

I have mintxfee and minrelaytxfee set to about 0.00003, which is = high enough to exclude essentially all of the of the 14700-14800 byte = flood transactions. My nodes' mempools only contain about one or two = blocks' worth of transactions. So I don't think this is correct = either.



Some = additional notes on this issue:

1. I think it's = related to CreateNewBlock() and getblocktemplate. I ran a Core bitcoind = process (commit d78a880) overnight with no mining connected to it, and = (IIRC -- my memory is fuzzy) when I woke up it was using around 400 MB = of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I ran = ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around = 800 MB. I then ran getblocktemplate every 5 seconds for about 30 = minutes, and RSS climbed to 1180 MB. An hour after that with more = getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 MB about 30 = minutes later.] getmempoolinfo is still showing "usage" around 25MB or = less.

I'll do some more testing with this and = see if I can make it repeatable, and record the results more carefully. = Expect a follow-up from me in a day or two.

2. = valgrind did not show anything super promising. It did report = this:

=3D=3D6880=3D=3D LEAK SUMMARY:
=3D=3D6880=3D=3D    definitely lost: 0 bytes in = 0 blocks
=3D=3D6880=3D=3D=     indirectly lost: 0 bytes in 0 blocks
=3D=3D6880=3D=3D    =   possibly lost: 288 bytes in 1 blocks
=3D=3D6880=3D=3D    still = reachable: 10,552 bytes in 39 blocks
=3D=3D6880=3D=3D         suppressed: = 0 bytes in 0 blocks
(Bitcoin Core commit = d78a880)

and this:
=3D=3D6778=3D=3D LEAK = SUMMARY:
=3D=3D6778=3D=3D=     definitely lost: 0 bytes in 0 blocks
=3D=3D6778=3D=3D    = indirectly lost: 0 bytes in 0 blocks
=3D=3D6778=3D=3D      possibly lost: 320 = bytes in 1 blocks
=3D=3D6778=3D=3D    still reachable: 10,080 bytes in 32 = blocks
=3D=3D6778=3D=3D=         suppressed: 0 bytes in 0 = blocks
(Bitcoin XT = commit fe446d)

I haven't found anything in = there yet that I think would produce the multi-GB memory usage after = running for a few days, but I could be missing it. Email me if you want = the full log.

I did not try running = getblocktemplate while valgrind was running. I'll have to try that. I = also have not let valgrind run for more than an = hour. 



P.S.: = Sorry for all the cross-post confusion and consequent flamewar fallout. = While it's probably too late for this thread, I'll make sure to post in = a manner that keeps the threads clearly separate in the future (e.g. = different subject lines).
= --Apple-Mail=_9C963CE6-6CA2-4ABE-9D61-E57718ABA125-- --Apple-Mail=_818AD798-4BDF-41A6-A9DB-8EF5B45DC674 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJWI8HPAAoJEIEuMk4MG0P1G8wIAI9e4zumzWl23AHnBGyEqrAr tQ7aR+2WDTpUrNY2ELl874ZYtbLiF+joLrq+kfV1qNnygTjBaBs2Kn93yAT6/qXb 1RnwoEw9/q2YptP6A2RUOz1L3LDtmxhiZmrluZAPVIX7+OhdJsfj5dR/UUnMsJqF tWIfoi9iO14BRCEfDi44OT/BWXbUTRQfgnrfvMjCulRIEGzKtAK+SGONL8tJGdD7 bjZ33WTP23sOuLX7b3udbdmO1ywCwAgQob6H7QIU8m2kL6A/59d2kKIVtUcnjQSm rZQyHcySVkb01kmcdBzEuI8f4I782G7aMmWgf5XLnF0eodhnIub2J9PP1QjXIhg= =4O5/ -----END PGP SIGNATURE----- --Apple-Mail=_818AD798-4BDF-41A6-A9DB-8EF5B45DC674--