Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 93409410 for ; Tue, 20 Oct 2015 12:39:11 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from d.mail.sonic.net (d.mail.sonic.net [64.142.111.50]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 4DEB6135 for ; Tue, 20 Oct 2015 12:39:10 +0000 (UTC) Received: from [192.168.1.190] (63.135.62.197.nwinternet.com [63.135.62.197] (may be forged)) (authenticated bits=0) by d.mail.sonic.net (8.15.1/8.15.1) with ESMTPSA id t9KCd1km003265 (version=TLSv1 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 20 Oct 2015 05:39:02 -0700 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Content-Type: multipart/signed; boundary="Apple-Mail=_3A7E479B-CFF5-476D-A1D3-1607249350EA"; protocol="application/pgp-signature"; micalg=pgp-sha512 X-Pgp-Agent: GPGMail 2.5.2 From: Jonathan Toomim In-Reply-To: Date: Tue, 20 Oct 2015 05:39:01 -0700 Message-Id: <1FE17DEB-8F77-4A60-A644-46A4F97D0E24@toom.im> References: <99C42DE7-814A-48F8-AB28-A5ADD77A9FD9@toom.im> <20151014093913.GB19607@amethyst.visucore.com> To: Mike Hearn X-Mailer: Apple Mail (2.1878.6) X-Sonic-CAuth: UmFuZG9tSVYBJD8cXmKuv3Q4F1P0BEb879tPes+c88946MxDxDws0oQaend4KNsceg5bmfixlu8a1wY0nAs1KBs/uU54ja4V X-Sonic-ID: C;9Ftoiid35RGSRuK7sH9FTg== M;Uo3xiid35RGSRuK7sH9FTg== X-Sonic-Spam-Details: 0.0/5.0 by cerberusd X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,HTML_MESSAGE, RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Bitcoin Dev , bitcoin-xt Subject: Re: [bitcoin-dev] Memory leaks? X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Development Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2015 12:39:11 -0000 --Apple-Mail=_3A7E479B-CFF5-476D-A1D3-1607249350EA Content-Type: multipart/alternative; boundary="Apple-Mail=_1FD40EC5-F3CD-400F-B839-4A4AC80B44ED" --Apple-Mail=_1FD40EC5-F3CD-400F-B839-4A4AC80B44ED Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii I did that Sunday twice. I'll report the results soon. Short version is = that it looks like valgrind is just finding 200 kB to 600 kB of = pblocktemplate, which is declared as a static pointer. Not exactly the = multi-GB leak I'm looking for, but possibly related. I've also got two bitcoind processes running on the same machine that I = started at the same time, running on different ports, all with the same = settings, but one of which is serving getblocktemplate every 5-6 seconds = and the other is not, while logging RSS on both every 6 seconds. RSS for = the non-serving node is now 734 MB, and for the serving node 1997 MB. = Graphs coming soon. On Oct 20, 2015, at 3:12 AM, Mike Hearn wrote: > OK, then running under Valgrind whilst sending gbt RPCs would be the = next step. >=20 > On Mon, Oct 19, 2015 at 9:17 PM, Multipool Admin = wrote: > My nodes are continuously running getblocktemplate and getinfo, and I = also suspected the issue is in either gbt or the rpc server. >=20 > The instance only takes a few hours to get up to that memory usage. >=20 > On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" = wrote: > On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan = wrote: >> This is *most likely* the mempool, but is just not reported = correctly. >=20 > I did some testing with PR #6410's better mempool reporting. The = improved reporting suggests that actual in-memory usage ("usage":) by = CTxMemPool is about 2.5x to 3x higher than the serialized transaction = sizes ("bytes":). The excess memory usage that I'm seeing is on the = order of 100x higher than the mempool "bytes": value. As such, I think = it's unlikely that this is the mempool, or at least not normal/correct = mempool behavior. >=20 > Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm = guessing his bitcoind has been running longer than any of mine. His = server definitely has more RAM. I don't know which email list he is = subscribed to (probably XT), so I'm sharing it with both lists to make = sure you're all aware of how big an issue this can be. >=20 >> In the meantime you can mitigate the mempool growth by setting = `-mintxfee`, see >> = https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#trans= action-flooding >=20 > I have mintxfee and minrelaytxfee set to about 0.00003, which is high = enough to exclude essentially all of the of the 14700-14800 byte flood = transactions. My nodes' mempools only contain about one or two blocks' = worth of transactions. So I don't think this is correct either. >=20 >=20 >=20 > Some additional notes on this issue: >=20 > 1. I think it's related to CreateNewBlock() and getblocktemplate. I = ran a Core bitcoind process (commit d78a880) overnight with no mining = connected to it, and (IIRC -- my memory is fuzzy) when I woke up it was = using around 400 MB of RSS and the mempool was at around "bytes":10MB, = "usage": 25MB. I ran ./bitcoin-cli getblocktemplate once, and IIRC the = RSS shot up to around 800 MB. I then ran getblocktemplate every 5 = seconds for about 30 minutes, and RSS climbed to 1180 MB. An hour after = that with more getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 = MB about 30 minutes later.] getmempoolinfo is still showing "usage" = around 25MB or less. >=20 > I'll do some more testing with this and see if I can make it = repeatable, and record the results more carefully. Expect a follow-up = from me in a day or two. >=20 > 2. valgrind did not show anything super promising. It did report this: >=20 > =3D=3D6880=3D=3D LEAK SUMMARY: > =3D=3D6880=3D=3D definitely lost: 0 bytes in 0 blocks > =3D=3D6880=3D=3D indirectly lost: 0 bytes in 0 blocks > =3D=3D6880=3D=3D possibly lost: 288 bytes in 1 blocks > =3D=3D6880=3D=3D still reachable: 10,552 bytes in 39 blocks > =3D=3D6880=3D=3D suppressed: 0 bytes in 0 blocks > (Bitcoin Core commit d78a880) >=20 > and this: > =3D=3D6778=3D=3D LEAK SUMMARY: > =3D=3D6778=3D=3D definitely lost: 0 bytes in 0 blocks > =3D=3D6778=3D=3D indirectly lost: 0 bytes in 0 blocks > =3D=3D6778=3D=3D possibly lost: 320 bytes in 1 blocks > =3D=3D6778=3D=3D still reachable: 10,080 bytes in 32 blocks > =3D=3D6778=3D=3D suppressed: 0 bytes in 0 blocks > (Bitcoin XT commit fe446d) >=20 > I haven't found anything in there yet that I think would produce the = multi-GB memory usage after running for a few days, but I could be = missing it. Email me if you want the full log. >=20 > I did not try running getblocktemplate while valgrind was running. = I'll have to try that. I also have not let valgrind run for more than an = hour. >=20 >=20 >=20 > P.S.: Sorry for all the cross-post confusion and consequent flamewar = fallout. While it's probably too late for this thread, I'll make sure to = post in a manner that keeps the threads clearly separate in the future = (e.g. different subject lines). >=20 > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev >=20 >=20 > -- > You received this message because you are subscribed to the Google = Groups "bitcoin-xt" group. > To unsubscribe from this group and stop receiving emails from it, send = an email to bitcoin-xt+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. >=20 --Apple-Mail=_1FD40EC5-F3CD-400F-B839-4A4AC80B44ED Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii I did = that Sunday twice. I'll report the results soon. Short version is that = it looks like valgrind is just finding 200 kB to 600 kB of = pblocktemplate, which is declared as a static pointer. Not exactly the = multi-GB leak I'm looking for, but possibly = related.

I've also got two bitcoind processes running = on the same machine that I started at the same time, running on = different ports, all with the same settings, but one of which is serving = getblocktemplate every 5-6 seconds and the other is not, while logging = RSS on both every 6 seconds. RSS for the non-serving node is now 734 MB, = and for the serving node 1997 MB. Graphs coming = soon.


On Oct 20, 2015, at = 3:12 AM, Mike Hearn <hearn@vinumeris.com> = wrote:

OK, then running under Valgrind whilst = sending gbt RPCs would be the next step.

On Mon, Oct 19, = 2015 at 9:17 PM, Multipool Admin <admin@multipool.us> = wrote:

My = nodes are continuously running getblocktemplate and getinfo, and I also = suspected the issue is in either gbt or the rpc server.

The instance only takes a few hours to get up to that memory = usage.

On Oct 18, 2015 8:59 = AM, "Jonathan Toomim via bitcoin-dev" <bitcoin-dev@lists.linuxfoundation.org> = wrote:
On Oct 14, 2015, at 2:39 AM, = Wladimir J. van der Laan <laanwj@gmail.com> wrote:
This is *most likely* the mempool, but is just not = reported correctly.

I did some = testing with PR #6410's better mempool reporting. The improved reporting = suggests that actual in-memory usage ("usage":) by CTxMemPool is about = 2.5x to 3x higher than the serialized transaction sizes ("bytes":). The = excess memory usage that I'm seeing is on the order of 100x higher than = the mempool "bytes": value. As such, I think it's unlikely that this is = the mempool, or at least not normal/correct mempool = behavior. 

Another user (admin@multipool.us) reported 35 GB of RSS usage. = I'm guessing his bitcoind has been running longer than any of mine. His = server definitely has more RAM. I don't know which email list he is = subscribed to (probably XT), so I'm sharing it with both lists to make = sure you're all aware of how big an issue this can = be.

In the meantime you can mitigate = the mempool growth by setting `-mintxfee`, see
https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/rele= ase-notes.md#transaction-flooding

I = have mintxfee and minrelaytxfee set to about 0.00003, which is high = enough to exclude essentially all of the of the 14700-14800 byte flood = transactions. My nodes' mempools only contain about one or two blocks' = worth of transactions. So I don't think this is correct = either.



Some = additional notes on this issue:

1. I think it's = related to CreateNewBlock() and getblocktemplate. I ran a Core bitcoind = process (commit d78a880) overnight with no mining connected to it, and = (IIRC -- my memory is fuzzy) when I woke up it was using around 400 MB = of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I ran = ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around = 800 MB. I then ran getblocktemplate every 5 seconds for about 30 = minutes, and RSS climbed to 1180 MB. An hour after that with more = getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 MB about 30 = minutes later.] getmempoolinfo is still showing "usage" around 25MB or = less.

I'll do some more testing with this and = see if I can make it repeatable, and record the results more carefully. = Expect a follow-up from me in a day or two.

2. = valgrind did not show anything super promising. It did report = this:

=3D=3D6880=3D=3D = LEAK SUMMARY:
=3D=3D6880=3D=3D&n= bsp;   definitely lost: 0 bytes in 0 blocks
=3D=3D6880=3D=3D&n= bsp;   indirectly lost: 0 bytes in 0 blocks
=3D=3D6880=3D=3D&n= bsp;     possibly lost: 288 bytes in 1 blocks
=3D=3D6880=3D=3D&n= bsp;   still reachable: 10,552 bytes in 39 blocks
=3D=3D6880=3D=3D =         suppressed: 0 bytes in 0 = blocks
(Bitcoin Core commit = d78a880)

and this:
=3D=3D6778=3D=3D = LEAK SUMMARY:
=3D=3D6778=3D=3D&n= bsp;   definitely lost: 0 bytes in 0 blocks
=3D=3D6778=3D=3D&n= bsp;   indirectly lost: 0 bytes in 0 blocks
=3D=3D6778=3D=3D&n= bsp;     possibly lost: 320 bytes in 1 blocks
=3D=3D6778=3D=3D&n= bsp;   still reachable: 10,080 bytes in 32 blocks
=3D=3D6778=3D=3D =         suppressed: 0 bytes in 0 = blocks
(Bitcoin XT = commit fe446d)

I haven't found anything in = there yet that I think would produce the multi-GB memory usage after = running for a few days, but I could be missing it. Email me if you want = the full log.

I did not try running = getblocktemplate while valgrind was running. I'll have to try that. I = also have not let valgrind run for more than an = hour. 



P.S.: = Sorry for all the cross-post confusion and consequent flamewar fallout. = While it's probably too late for this thread, I'll make sure to post in = a manner that keeps the threads clearly separate in the future (e.g. = different subject lines).

_______________________________________________
bitcoin-dev mailing list
bitcoin-dev@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/bitco= in-dev


--
You received this message because you are subscribed to the Google = Groups "bitcoin-xt" group.
To unsubscribe from this group and stop receiving emails from it, send = an email to bitcoin-xt+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


= --Apple-Mail=_1FD40EC5-F3CD-400F-B839-4A4AC80B44ED-- --Apple-Mail=_3A7E479B-CFF5-476D-A1D3-1607249350EA Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJWJjXmAAoJEIEuMk4MG0P1Oz8H/irPp4Qz5FoDl7mmgDeixzwW rU1drVfjgfumfViwMM5TleWspjb2L9GJCpYZprIOH9rL/ndXujcr63zEwyX6OSFX 2KVZm2YpWWYReq2RNVLzZU/d995AEY2JfFTrYOosBui3hfxXOpca8RStAdy1jrAM IdUEz4fCaMhvwl1kbm5gNSHAQFwlNQdhCDdI+1qxggF6ghri3+vBJebELTdmnn68 6SevVuxqMkQPsVQ10s+mVIb5YbdzQQa2CokKt3p6Hlo1LlVr53vMw6pbhxWm6glo SZXd7e8eDXItBhiU2ZGEaHe2ab9vIM+ykr7g9rS603ne3qclcjPUKaHFq6QgaDY= =RHiU -----END PGP SIGNATURE----- --Apple-Mail=_3A7E479B-CFF5-476D-A1D3-1607249350EA--