path: root/22/c63bd14c5a2ba79012844d4650e03dd2294acd

Return-Path: <j@toom.im>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
	[172.17.192.35])
	by mail.linuxfoundation.org (Postfix) with ESMTPS id 3D2B567
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Sun, 18 Oct 2015 15:59:16 +0000 (UTC)
X-Greylist: from auto-whitelisted by SQLgrey-1.7.6
Received: from d.mail.sonic.net (d.mail.sonic.net [64.142.111.50])
	by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 8BAB2F5
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Sun, 18 Oct 2015 15:59:15 +0000 (UTC)
Received: from [192.168.1.190] (63.135.62.197.nwinternet.com [63.135.62.197]
	(may be forged)) (authenticated bits=0)
	by d.mail.sonic.net (8.15.1/8.15.1) with ESMTPSA id t9IFxBdk025659
	(version=TLSv1 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT);
	Sun, 18 Oct 2015 08:59:12 -0700
Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\))
Content-Type: multipart/signed;
	boundary="Apple-Mail=_818AD798-4BDF-41A6-A9DB-8EF5B45DC674";
	protocol="application/pgp-signature"; micalg=pgp-sha512
X-Pgp-Agent: GPGMail 2.5.2
From: Jonathan Toomim <j@toom.im>
In-Reply-To: <20151014093913.GB19607@amethyst.visucore.com>
Date: Sun, 18 Oct 2015 08:59:11 -0700
Message-Id: <F938BD02-3D80-4E99-BD1C-490543187895@toom.im>
References: <99C42DE7-814A-48F8-AB28-A5ADD77A9FD9@toom.im>
	<20151014093913.GB19607@amethyst.visucore.com>
To: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>
X-Mailer: Apple Mail (2.1878.6)
X-Sonic-CAuth: UmFuZG9tSVaH4ltZBozHiZKlWjc4I41+A1rFRRQziasL5jdC+2o2HOpvfsnWXbPXiZJwLiimOjO4IPaC0iqbCP6zBGLeFfhV
X-Sonic-ID: C;RGdqLLF15RGrb+K7sH9FTg== M;5mbSLLF15RGrb+K7sH9FTg==
X-Sonic-Spam-Details: 1.7/5.0 by cerberusd
X-Spam-Status: No, score=-0.9 required=5.0 tests=BAYES_00,HTML_MESSAGE,
	RCVD_IN_DNSWL_LOW,URIBL_BLACK autolearn=no version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
	smtp1.linux-foundation.org
Cc: bitcoin-xt <bitcoin-xt@googlegroups.com>
Subject: Re: [bitcoin-dev] Memory leaks?
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Sun, 18 Oct 2015 15:59:16 -0000


--Apple-Mail=_818AD798-4BDF-41A6-A9DB-8EF5B45DC674
Content-Type: multipart/alternative;
	boundary="Apple-Mail=_9C963CE6-6CA2-4ABE-9D61-E57718ABA125"


--Apple-Mail=_9C963CE6-6CA2-4ABE-9D61-E57718ABA125
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail.com> =
wrote:
> This is *most likely* the mempool, but is just not reported correctly.

I did some testing with PR #6410's better mempool reporting. The =
improved reporting suggests that actual in-memory usage ("usage":) by =
CTxMemPool is about 2.5x to 3x higher than the serialized transaction =
sizes ("bytes":). The excess memory usage that I'm seeing is on the =
order of 100x higher than the mempool "bytes": value. As such, I think =
it's unlikely that this is the mempool, or at least not normal/correct =
mempool behavior.

Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm =
guessing his bitcoind has been running longer than any of mine. His =
server definitely has more RAM. I don't know which email list he is =
subscribed to (probably XT), so I'm sharing it with both lists to make =
sure you're all aware of how big an issue this can be.

> In the meantime you can mitigate the mempool growth by setting =
`-mintxfee`, see
> =
https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#trans=
action-flooding

I have mintxfee and minrelaytxfee set to about 0.00003, which is high =
enough to exclude essentially all of the of the 14700-14800 byte flood =
transactions. My nodes' mempools only contain about one or two blocks' =
worth of transactions. So I don't think this is correct either.



Some additional notes on this issue:

1. I think it's related to CreateNewBlock() and getblocktemplate. I ran =
a Core bitcoind process (commit d78a880) overnight with no mining =
connected to it, and (IIRC -- my memory is fuzzy) when I woke up it was =
using around 400 MB of RSS and the mempool was at around "bytes":10MB, =
"usage": 25MB. I ran ./bitcoin-cli getblocktemplate once, and IIRC the =
RSS shot up to around 800 MB. I then ran getblocktemplate every 5 =
seconds for about 30 minutes, and RSS climbed to 1180 MB. An hour after =
that with more getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 =
MB about 30 minutes later.] getmempoolinfo is still showing "usage" =
around 25MB or less.

I'll do some more testing with this and see if I can make it repeatable, =
and record the results more carefully. Expect a follow-up from me in a =
day or two.

2. valgrind did not show anything super promising. It did report this:

=3D=3D6880=3D=3D LEAK SUMMARY:
=3D=3D6880=3D=3D    definitely lost: 0 bytes in 0 blocks
=3D=3D6880=3D=3D    indirectly lost: 0 bytes in 0 blocks
=3D=3D6880=3D=3D      possibly lost: 288 bytes in 1 blocks
=3D=3D6880=3D=3D    still reachable: 10,552 bytes in 39 blocks
=3D=3D6880=3D=3D         suppressed: 0 bytes in 0 blocks
(Bitcoin Core commit d78a880)

and this:
=3D=3D6778=3D=3D LEAK SUMMARY:
=3D=3D6778=3D=3D    definitely lost: 0 bytes in 0 blocks
=3D=3D6778=3D=3D    indirectly lost: 0 bytes in 0 blocks
=3D=3D6778=3D=3D      possibly lost: 320 bytes in 1 blocks
=3D=3D6778=3D=3D    still reachable: 10,080 bytes in 32 blocks
=3D=3D6778=3D=3D         suppressed: 0 bytes in 0 blocks
(Bitcoin XT commit fe446d)

I haven't found anything in there yet that I think would produce the =
multi-GB memory usage after running for a few days, but I could be =
missing it. Email me if you want the full log.

I did not try running getblocktemplate while valgrind was running. I'll =
have to try that. I also have not let valgrind run for more than an =
hour.



P.S.: Sorry for all the cross-post confusion and consequent flamewar =
fallout. While it's probably too late for this thread, I'll make sure to =
post in a manner that keeps the threads clearly separate in the future =
(e.g. different subject lines).

--Apple-Mail=_9C963CE6-6CA2-4ABE-9D61-E57718ABA125
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=us-ascii

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: =
after-white-space;"><div><div>On Oct 14, 2015, at 2:39 AM, Wladimir J. =
van der Laan &lt;<a =
href=3D"mailto:laanwj@gmail.com">laanwj@gmail.com</a>&gt; =
wrote:</div><blockquote type=3D"cite">This is *most likely* the mempool, =
but is just not reported =
correctly.<br></blockquote><div><br></div><div>I did some testing with =
PR #6410's better mempool reporting. The improved reporting suggests =
that actual in-memory usage ("usage":) by CTxMemPool is about 2.5x to 3x =
higher than the serialized transaction sizes ("bytes":). The excess =
memory usage that I'm seeing is on the order of 100x higher than the =
mempool "bytes": value. As such, I think it's unlikely that this is the =
mempool, or at least not normal/correct mempool =
behavior.&nbsp;</div><div><br></div><div>Another user (<a =
href=3D"mailto:admin@multipool.us">admin@multipool.us</a>) reported 35 =
GB of RSS usage. I'm guessing his bitcoind has been running longer than =
any of mine. His server definitely has more RAM. I don't know which =
email list he is subscribed to (probably XT), so I'm sharing it with =
both lists to make sure you're all aware of how big an issue this can =
be.</div><br><blockquote type=3D"cite">In the meantime you can mitigate =
the mempool growth by setting `-mintxfee`, see<br><a =
href=3D"https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.=
md#transaction-flooding">https://github.com/bitcoin/bitcoin/blob/v0.11.0/d=
oc/release-notes.md#transaction-flooding</a><br></blockquote><div><br></di=
v>I have mintxfee and minrelaytxfee set to about&nbsp;0.00003, which is =
high enough to exclude essentially all of the of the 14700-14800 byte =
flood transactions. My nodes' mempools only contain about one or two =
blocks' worth of transactions. So I don't think this is correct =
either.</div><div><br></div><div><br></div><div><br></div><div>Some =
additional notes on this issue:</div><div><br></div><div>1. I think it's =
related to CreateNewBlock() and getblocktemplate. I ran a Core bitcoind =
process (commit d78a880) overnight with no mining connected to it, and =
(IIRC -- my memory is fuzzy) when I woke up it was using around 400 MB =
of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I ran =
./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around =
800 MB. I then ran getblocktemplate every 5 seconds for about 30 =
minutes, and RSS climbed to 1180 MB. An hour after that with more =
getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 MB about 30 =
minutes later.] getmempoolinfo is still showing "usage" around 25MB or =
less.</div><div><br></div><div>I'll do some more testing with this and =
see if I can make it repeatable, and record the results more carefully. =
Expect a follow-up from me in a day or two.</div><div><br></div><div>2. =
valgrind did not show anything super promising. It did report =
this:</div><div><br></div><div><div style=3D"margin: 0px; font-family: =
'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, =
0);">=3D=3D6880=3D=3D LEAK SUMMARY:</div><div style=3D"margin: 0px; =
font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: =
rgb(0, 0, 0);">=3D=3D6880=3D=3D&nbsp; &nbsp; definitely lost: 0 bytes in =
0 blocks</div><div style=3D"margin: 0px; font-family: 'Andale Mono'; =
color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);">=3D=3D6880=3D=3D=
&nbsp; &nbsp; indirectly lost: 0 bytes in 0 blocks</div><div =
style=3D"margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, =
20); background-color: rgb(0, 0, 0);">=3D=3D6880=3D=3D&nbsp; &nbsp; =
&nbsp; possibly lost: 288 bytes in 1 blocks</div><div style=3D"margin: =
0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); =
background-color: rgb(0, 0, 0);">=3D=3D6880=3D=3D&nbsp; &nbsp; still =
reachable: 10,552 bytes in 39 blocks</div><div style=3D"margin: 0px; =
font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: =
rgb(0, 0, 0);">=3D=3D6880=3D=3D &nbsp; &nbsp; &nbsp; &nbsp; suppressed: =
0 bytes in 0 blocks</div></div><div>(Bitcoin Core commit =
d78a880)</div><div><br></div><div>and this:</div><div><div =
style=3D"margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, =
20); background-color: rgb(0, 0, 0);">=3D=3D6778=3D=3D LEAK =
SUMMARY:</div><div style=3D"margin: 0px; font-family: 'Andale Mono'; =
color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);">=3D=3D6778=3D=3D=
&nbsp; &nbsp; definitely lost: 0 bytes in 0 blocks</div><div =
style=3D"margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, =
20); background-color: rgb(0, 0, 0);">=3D=3D6778=3D=3D&nbsp; &nbsp; =
indirectly lost: 0 bytes in 0 blocks</div><div style=3D"margin: 0px; =
font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: =
rgb(0, 0, 0);">=3D=3D6778=3D=3D&nbsp; &nbsp; &nbsp; possibly lost: 320 =
bytes in 1 blocks</div><div style=3D"margin: 0px; font-family: 'Andale =
Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, =
0);">=3D=3D6778=3D=3D&nbsp; &nbsp; still reachable: 10,080 bytes in 32 =
blocks</div><div style=3D"margin: 0px; font-family: 'Andale Mono'; =
color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);">=3D=3D6778=3D=3D=
 &nbsp; &nbsp; &nbsp; &nbsp; suppressed: 0 bytes in 0 =
blocks</div></div><div>(Bitcoin XT =
commit&nbsp;fe446d)</div><div><br></div><div>I haven't found anything in =
there yet that I think would produce the multi-GB memory usage after =
running for a few days, but I could be missing it. Email me if you want =
the full log.</div><div><br></div><div>I did not try running =
getblocktemplate while valgrind was running. I'll have to try that. I =
also have not let valgrind run for more than an =
hour.&nbsp;</div><div><br></div><div><br></div><div><br></div><div>P.S.: =
Sorry for all the cross-post confusion and consequent flamewar fallout. =
While it's probably too late for this thread, I'll make sure to post in =
a manner that keeps the threads clearly separate in the future (e.g. =
different subject lines).</div></body></html>=

--Apple-Mail=_9C963CE6-6CA2-4ABE-9D61-E57718ABA125--

--Apple-Mail=_818AD798-4BDF-41A6-A9DB-8EF5B45DC674
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP using GPGMail

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iQEcBAEBCgAGBQJWI8HPAAoJEIEuMk4MG0P1G8wIAI9e4zumzWl23AHnBGyEqrAr
tQ7aR+2WDTpUrNY2ELl874ZYtbLiF+joLrq+kfV1qNnygTjBaBs2Kn93yAT6/qXb
1RnwoEw9/q2YptP6A2RUOz1L3LDtmxhiZmrluZAPVIX7+OhdJsfj5dR/UUnMsJqF
tWIfoi9iO14BRCEfDi44OT/BWXbUTRQfgnrfvMjCulRIEGzKtAK+SGONL8tJGdD7
bjZ33WTP23sOuLX7b3udbdmO1ywCwAgQob6H7QIU8m2kL6A/59d2kKIVtUcnjQSm
rZQyHcySVkb01kmcdBzEuI8f4I782G7aMmWgf5XLnF0eodhnIub2J9PP1QjXIhg=
=4O5/
-----END PGP SIGNATURE-----

--Apple-Mail=_818AD798-4BDF-41A6-A9DB-8EF5B45DC674--