Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\))
Content-Type: multipart/signed;
	boundary="Apple-Mail=_3A7E479B-CFF5-476D-A1D3-1607249350EA";
	protocol="application/pgp-signature"; micalg=pgp-sha512
From: Jonathan Toomim <j@toom.im>
In-Reply-To: <CA+w+GKTU6C7KKFx9dDd_--s1DQCO15n=034Lku2-kTYKf96XYw@mail.gmail.com>
Date: Tue, 20 Oct 2015 05:39:01 -0700
Message-Id: <1FE17DEB-8F77-4A60-A644-46A4F97D0E24@toom.im>
References: <99C42DE7-814A-48F8-AB28-A5ADD77A9FD9@toom.im>
	<20151014093913.GB19607@amethyst.visucore.com>
	<F938BD02-3D80-4E99-BD1C-490543187895@toom.im>
	<CAP3QyGJNBdsBtxYjOprRJ=YW2v-N_CopVQeSgDs6J4J8LMWuxA@mail.gmail.com>
	<CA+w+GKTU6C7KKFx9dDd_--s1DQCO15n=034Lku2-kTYKf96XYw@mail.gmail.com>
To: Mike Hearn <hearn@vinumeris.com>
Cc: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>,
	bitcoin-xt <bitcoin-xt@googlegroups.com>
Subject: Re: [bitcoin-dev] Memory leaks?
Precedence: list


--Apple-Mail=_3A7E479B-CFF5-476D-A1D3-1607249350EA
Content-Type: multipart/alternative;
	boundary="Apple-Mail=_1FD40EC5-F3CD-400F-B839-4A4AC80B44ED"


--Apple-Mail=_1FD40EC5-F3CD-400F-B839-4A4AC80B44ED
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

I did that Sunday twice. I'll report the results soon. Short version is =
that it looks like valgrind is just finding 200 kB to 600 kB of =
pblocktemplate, which is declared as a static pointer. Not exactly the =
multi-GB leak I'm looking for, but possibly related.

I've also got two bitcoind processes running on the same machine that I =
started at the same time, running on different ports, all with the same =
settings, but one of which is serving getblocktemplate every 5-6 seconds =
and the other is not, while logging RSS on both every 6 seconds. RSS for =
the non-serving node is now 734 MB, and for the serving node 1997 MB. =
Graphs coming soon.


On Oct 20, 2015, at 3:12 AM, Mike Hearn <hearn@vinumeris.com> wrote:

> OK, then running under Valgrind whilst sending gbt RPCs would be the =
next step.
>=20
> On Mon, Oct 19, 2015 at 9:17 PM, Multipool Admin <admin@multipool.us> =
wrote:
> My nodes are continuously running getblocktemplate and getinfo, and I =
also suspected the issue is in either gbt or the rpc server.
>=20
> The instance only takes a few hours to get up to that memory usage.
>=20
> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" =
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan =
<laanwj@gmail.com> wrote:
>> This is *most likely* the mempool, but is just not reported =
correctly.
>=20
> I did some testing with PR #6410's better mempool reporting. The =
improved reporting suggests that actual in-memory usage ("usage":) by =
CTxMemPool is about 2.5x to 3x higher than the serialized transaction =
sizes ("bytes":). The excess memory usage that I'm seeing is on the =
order of 100x higher than the mempool "bytes": value. As such, I think =
it's unlikely that this is the mempool, or at least not normal/correct =
mempool behavior.
>=20
> Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm =
guessing his bitcoind has been running longer than any of mine. His =
server definitely has more RAM. I don't know which email list he is =
subscribed to (probably XT), so I'm sharing it with both lists to make =
sure you're all aware of how big an issue this can be.
>=20
>> In the meantime you can mitigate the mempool growth by setting =
`-mintxfee`, see
>> =
https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#trans=
action-flooding
>=20
> I have mintxfee and minrelaytxfee set to about 0.00003, which is high =
enough to exclude essentially all of the of the 14700-14800 byte flood =
transactions. My nodes' mempools only contain about one or two blocks' =
worth of transactions. So I don't think this is correct either.
>=20
>=20
>=20
> Some additional notes on this issue:
>=20
> 1. I think it's related to CreateNewBlock() and getblocktemplate. I =
ran a Core bitcoind process (commit d78a880) overnight with no mining =
connected to it, and (IIRC -- my memory is fuzzy) when I woke up it was =
using around 400 MB of RSS and the mempool was at around "bytes":10MB, =
"usage": 25MB. I ran ./bitcoin-cli getblocktemplate once, and IIRC the =
RSS shot up to around 800 MB. I then ran getblocktemplate every 5 =
seconds for about 30 minutes, and RSS climbed to 1180 MB. An hour after =
that with more getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 =
MB about 30 minutes later.] getmempoolinfo is still showing "usage" =
around 25MB or less.
>=20
> I'll do some more testing with this and see if I can make it =
repeatable, and record the results more carefully. Expect a follow-up =
from me in a day or two.
>=20
> 2. valgrind did not show anything super promising. It did report this:
>=20
> =3D=3D6880=3D=3D LEAK SUMMARY:
> =3D=3D6880=3D=3D    definitely lost: 0 bytes in 0 blocks
> =3D=3D6880=3D=3D    indirectly lost: 0 bytes in 0 blocks
> =3D=3D6880=3D=3D      possibly lost: 288 bytes in 1 blocks
> =3D=3D6880=3D=3D    still reachable: 10,552 bytes in 39 blocks
> =3D=3D6880=3D=3D         suppressed: 0 bytes in 0 blocks
> (Bitcoin Core commit d78a880)
>=20
> and this:
> =3D=3D6778=3D=3D LEAK SUMMARY:
> =3D=3D6778=3D=3D    definitely lost: 0 bytes in 0 blocks
> =3D=3D6778=3D=3D    indirectly lost: 0 bytes in 0 blocks
> =3D=3D6778=3D=3D      possibly lost: 320 bytes in 1 blocks
> =3D=3D6778=3D=3D    still reachable: 10,080 bytes in 32 blocks
> =3D=3D6778=3D=3D         suppressed: 0 bytes in 0 blocks
> (Bitcoin XT commit fe446d)
>=20
> I haven't found anything in there yet that I think would produce the =
multi-GB memory usage after running for a few days, but I could be =
missing it. Email me if you want the full log.
>=20
> I did not try running getblocktemplate while valgrind was running. =
I'll have to try that. I also have not let valgrind run for more than an =
hour.
>=20
>=20
>=20
> P.S.: Sorry for all the cross-post confusion and consequent flamewar =
fallout. While it's probably too late for this thread, I'll make sure to =
post in a manner that keeps the threads clearly separate in the future =
(e.g. different subject lines).
>=20
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>=20
>=20
> --
> You received this message because you are subscribed to the Google =
Groups "bitcoin-xt" group.
> To unsubscribe from this group and stop receiving emails from it, send =
an email to bitcoin-xt+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>=20


--Apple-Mail=_1FD40EC5-F3CD-400F-B839-4A4AC80B44ED
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=us-ascii

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">I did =
that Sunday twice. I'll report the results soon. Short version is that =
it looks like valgrind is just finding 200 kB to 600 kB of =
pblocktemplate, which is declared as a static pointer. Not exactly the =
multi-GB leak I'm looking for, but possibly =
related.<div><br></div><div>I've also got two bitcoind processes running =
on the same machine that I started at the same time, running on =
different ports, all with the same settings, but one of which is serving =
getblocktemplate every 5-6 seconds and the other is not, while logging =
RSS on both every 6 seconds. RSS for the non-serving node is now 734 MB, =
and for the serving node 1997 MB. Graphs coming =
soon.</div><div><div><br></div><div><br><div><div>On Oct 20, 2015, at =
3:12 AM, Mike Hearn &lt;<a =
href=3D"mailto:hearn@vinumeris.com">hearn@vinumeris.com</a>&gt; =
wrote:</div><br class=3D"Apple-interchange-newline"><blockquote =
type=3D"cite"><div dir=3D"ltr">OK, then running under Valgrind whilst =
sending gbt RPCs would be the next step.</div><div =
class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Mon, Oct 19, =
2015 at 9:17 PM, Multipool Admin <span dir=3D"ltr">&lt;<a =
href=3D"mailto:admin@multipool.us" =
target=3D"_blank">admin@multipool.us</a>&gt;</span> =
wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir=3D"ltr">My =
nodes are continuously running getblocktemplate and getinfo, and I also =
suspected the issue is in either gbt or the rpc server.</p><p =
dir=3D"ltr">The instance only takes a few hours to get up to that memory =
usage.</p>
<div class=3D"gmail_quote"><div><div class=3D"h5">On Oct 18, 2015 8:59 =
AM, "Jonathan Toomim via bitcoin-dev" &lt;<a =
href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" =
target=3D"_blank">bitcoin-dev@lists.linuxfoundation.org</a>&gt; =
wrote:<br type=3D"attribution"></div></div><blockquote =
class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc =
solid;padding-left:1ex"><div><div class=3D"h5"><div =
style=3D"word-wrap:break-word"><div><div>On Oct 14, 2015, at 2:39 AM, =
Wladimir J. van der Laan &lt;<a href=3D"mailto:laanwj@gmail.com" =
target=3D"_blank">laanwj@gmail.com</a>&gt; wrote:</div><blockquote =
type=3D"cite">This is *most likely* the mempool, but is just not =
reported correctly.<br></blockquote><div><br></div><div>I did some =
testing with PR #6410's better mempool reporting. The improved reporting =
suggests that actual in-memory usage ("usage":) by CTxMemPool is about =
2.5x to 3x higher than the serialized transaction sizes ("bytes":). The =
excess memory usage that I'm seeing is on the order of 100x higher than =
the mempool "bytes": value. As such, I think it's unlikely that this is =
the mempool, or at least not normal/correct mempool =
behavior.&nbsp;</div><div><br></div><div>Another user (<a =
href=3D"mailto:admin@multipool.us" =
target=3D"_blank">admin@multipool.us</a>) reported 35 GB of RSS usage. =
I'm guessing his bitcoind has been running longer than any of mine. His =
server definitely has more RAM. I don't know which email list he is =
subscribed to (probably XT), so I'm sharing it with both lists to make =
sure you're all aware of how big an issue this can =
be.</div><br><blockquote type=3D"cite">In the meantime you can mitigate =
the mempool growth by setting `-mintxfee`, see<br><a =
href=3D"https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.=
md#transaction-flooding" =
target=3D"_blank">https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/rele=
ase-notes.md#transaction-flooding</a><br></blockquote><div><br></div>I =
have mintxfee and minrelaytxfee set to about&nbsp;0.00003, which is high =
enough to exclude essentially all of the of the 14700-14800 byte flood =
transactions. My nodes' mempools only contain about one or two blocks' =
worth of transactions. So I don't think this is correct =
either.</div><div><br></div><div><br></div><div><br></div><div>Some =
additional notes on this issue:</div><div><br></div><div>1. I think it's =
related to CreateNewBlock() and getblocktemplate. I ran a Core bitcoind =
process (commit d78a880) overnight with no mining connected to it, and =
(IIRC -- my memory is fuzzy) when I woke up it was using around 400 MB =
of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I ran =
./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around =
800 MB. I then ran getblocktemplate every 5 seconds for about 30 =
minutes, and RSS climbed to 1180 MB. An hour after that with more =
getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 MB about 30 =
minutes later.] getmempoolinfo is still showing "usage" around 25MB or =
less.</div><div><br></div><div>I'll do some more testing with this and =
see if I can make it repeatable, and record the results more carefully. =
Expect a follow-up from me in a day or two.</div><div><br></div><div>2. =
valgrind did not show anything super promising. It did report =
this:</div><div><br></div><div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=3D =
LEAK SUMMARY:</div><div style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=3D&n=
bsp; &nbsp; definitely lost: 0 bytes in 0 blocks</div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=3D&n=
bsp; &nbsp; indirectly lost: 0 bytes in 0 blocks</div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=3D&n=
bsp; &nbsp; &nbsp; possibly lost: 288 bytes in 1 blocks</div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=3D&n=
bsp; &nbsp; still reachable: 10,552 bytes in 39 blocks</div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=3D =
&nbsp; &nbsp; &nbsp; &nbsp; suppressed: 0 bytes in 0 =
blocks</div></div><div>(Bitcoin Core commit =
d78a880)</div><div><br></div><div>and this:</div><div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D =
LEAK SUMMARY:</div><div style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D&n=
bsp; &nbsp; definitely lost: 0 bytes in 0 blocks</div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D&n=
bsp; &nbsp; indirectly lost: 0 bytes in 0 blocks</div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D&n=
bsp; &nbsp; &nbsp; possibly lost: 320 bytes in 1 blocks</div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D&n=
bsp; &nbsp; still reachable: 10,080 bytes in 32 blocks</div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D =
&nbsp; &nbsp; &nbsp; &nbsp; suppressed: 0 bytes in 0 =
blocks</div></div><div>(Bitcoin XT =
commit&nbsp;fe446d)</div><div><br></div><div>I haven't found anything in =
there yet that I think would produce the multi-GB memory usage after =
running for a few days, but I could be missing it. Email me if you want =
the full log.</div><div><br></div><div>I did not try running =
getblocktemplate while valgrind was running. I'll have to try that. I =
also have not let valgrind run for more than an =
hour.&nbsp;</div><div><br></div><div><br></div><div><br></div><div>P.S.: =
Sorry for all the cross-post confusion and consequent flamewar fallout. =
While it's probably too late for this thread, I'll make sure to post in =
a manner that keeps the threads clearly separate in the future (e.g. =
different subject lines).</div></div><br></div></div><span =
class=3D"">_______________________________________________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" =
target=3D"_blank">bitcoin-dev@lists.linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev"=
 rel=3D"noreferrer" =
target=3D"_blank">https://lists.linuxfoundation.org/mailman/listinfo/bitco=
in-dev</a><br>
<br></span></blockquote></div><div class=3D"HOEnZb"><div =
class=3D"h5"><div><br class=3D"webkit-block-placeholder"></div>

-- <br>
You received this message because you are subscribed to the Google =
Groups "bitcoin-xt" group.<br>
To unsubscribe from this group and stop receiving emails from it, send =
an email to <a href=3D"mailto:bitcoin-xt+unsubscribe@googlegroups.com" =
target=3D"_blank">bitcoin-xt+unsubscribe@googlegroups.com</a>.<br>
For more options, visit <a href=3D"https://groups.google.com/d/optout" =
target=3D"_blank">https://groups.google.com/d/optout</a>.<br>
</div></div></blockquote></div><br></div>
</blockquote></div><br></div></div></body></html>=

--Apple-Mail=_1FD40EC5-F3CD-400F-B839-4A4AC80B44ED--

--Apple-Mail=_3A7E479B-CFF5-476D-A1D3-1607249350EA
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP using GPGMail

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iQEcBAEBCgAGBQJWJjXmAAoJEIEuMk4MG0P1Oz8H/irPp4Qz5FoDl7mmgDeixzwW
rU1drVfjgfumfViwMM5TleWspjb2L9GJCpYZprIOH9rL/ndXujcr63zEwyX6OSFX
2KVZm2YpWWYReq2RNVLzZU/d995AEY2JfFTrYOosBui3hfxXOpca8RStAdy1jrAM
IdUEz4fCaMhvwl1kbm5gNSHAQFwlNQdhCDdI+1qxggF6ghri3+vBJebELTdmnn68
6SevVuxqMkQPsVQ10s+mVIb5YbdzQQa2CokKt3p6Hlo1LlVr53vMw6pbhxWm6glo
SZXd7e8eDXItBhiU2ZGEaHe2ab9vIM+ykr7g9rS603ne3qclcjPUKaHFq6QgaDY=
=RHiU
-----END PGP SIGNATURE-----

--Apple-Mail=_3A7E479B-CFF5-476D-A1D3-1607249350EA--