path: root/ea/5f2bd38909bda68889e707e127035e3a57090a

Return-Path: <admin@multipool.us>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
	[172.17.192.35])
	by mail.linuxfoundation.org (Postfix) with ESMTPS id 9D7251BB
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Mon, 19 Oct 2015 19:17:16 +0000 (UTC)
X-Greylist: whitelisted by SQLgrey-1.7.6
Received: from mail-oi0-f47.google.com (mail-oi0-f47.google.com
	[209.85.218.47])
	by smtp1.linuxfoundation.org (Postfix) with ESMTPS id B9147128
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Mon, 19 Oct 2015 19:17:15 +0000 (UTC)
Received: by oiev17 with SMTP id v17so57219144oie.2
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Mon, 19 Oct 2015 12:17:15 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20130820;
	h=x-gm-message-state:mime-version:in-reply-to:references:date
	:message-id:subject:from:to:cc:content-type;
	bh=A1AoA1WoG4ZQNpbVIgH+aqD4g+SJ7lt88jLEvw7V9pw=;
	b=ERLDw+VDeXH5ZPZ8KbXv3YrJF/fzPsA+BRzJ8IuiuR9BidVDVKkEGMeB4bWqms7qbl
	i80esks5nzegMaum7CxgrzSABe15DCekcupL65LZfDave7WMZIC8tK5q+DU7bXyChPbP
	FlEekHm1MaguJdhbywJqNqMvj3lP6MEVA+AXCbmay2NvNFmuJtBQfX7y6uIUSZmB9/mH
	X18o4qfy+u4EgIno7Rc4ErLEYjdvw7Ne0NnoFImZNuX/TQWWFH90S86EapYTcYYgVi6n
	6Bq0npTh9KAPZDjgvOMmu/eTRadacSWrsBW5iKBPo/dAP3A5BvRWxMRTdY1u0wsmzo4j
	vZHA==
X-Gm-Message-State: ALoCoQkZx1fv8+eE56GeQPh/1LbHNs5qhOYp7hNI6B/OAlsUSXhjlyh3//V+oalodBo0J8vYTF/B
MIME-Version: 1.0
X-Received: by 10.202.86.86 with SMTP id k83mr18476971oib.68.1445282235063;
	Mon, 19 Oct 2015 12:17:15 -0700 (PDT)
Received: by 10.182.220.193 with HTTP; Mon, 19 Oct 2015 12:17:14 -0700 (PDT)
Received: by 10.182.220.193 with HTTP; Mon, 19 Oct 2015 12:17:14 -0700 (PDT)
In-Reply-To: <F938BD02-3D80-4E99-BD1C-490543187895@toom.im>
References: <99C42DE7-814A-48F8-AB28-A5ADD77A9FD9@toom.im>
	<20151014093913.GB19607@amethyst.visucore.com>
	<F938BD02-3D80-4E99-BD1C-490543187895@toom.im>
Date: Mon, 19 Oct 2015 12:17:14 -0700
Message-ID: <CAP3QyGJNBdsBtxYjOprRJ=YW2v-N_CopVQeSgDs6J4J8LMWuxA@mail.gmail.com>
From: Multipool Admin <admin@multipool.us>
To: Jonathan Toomim <j@toom.im>
Content-Type: multipart/alternative; boundary=001a113dd26838b8c7052279fec9
X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,HTML_MESSAGE,
	RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
	smtp1.linux-foundation.org
Cc: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>,
	bitcoin-xt <bitcoin-xt@googlegroups.com>
Subject: Re: [bitcoin-dev] Memory leaks?
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Mon, 19 Oct 2015 19:17:16 -0000

--001a113dd26838b8c7052279fec9
Content-Type: text/plain; charset=UTF-8

My nodes are continuously running getblocktemplate and getinfo, and I also
suspected the issue is in either gbt or the rpc server.

The instance only takes a few hours to get up to that memory usage.
On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail.com>
> wrote:
>
> This is *most likely* the mempool, but is just not reported correctly.
>
>
> I did some testing with PR #6410's better mempool reporting. The improved
> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
> The excess memory usage that I'm seeing is on the order of 100x higher than
> the mempool "bytes": value. As such, I think it's unlikely that this is the
> mempool, or at least not normal/correct mempool behavior.
>
> Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm
> guessing his bitcoind has been running longer than any of mine. His server
> definitely has more RAM. I don't know which email list he is subscribed to
> (probably XT), so I'm sharing it with both lists to make sure you're all
> aware of how big an issue this can be.
>
> In the meantime you can mitigate the mempool growth by setting
> `-mintxfee`, see
>
> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>
>
> I have mintxfee and minrelaytxfee set to about 0.00003, which is high
> enough to exclude essentially all of the of the 14700-14800 byte flood
> transactions. My nodes' mempools only contain about one or two blocks'
> worth of transactions. So I don't think this is correct either.
>
>
>
> Some additional notes on this issue:
>
> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
> Core bitcoind process (commit d78a880) overnight with no mining connected
> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
> getmempoolinfo is still showing "usage" around 25MB or less.
>
> I'll do some more testing with this and see if I can make it repeatable,
> and record the results more carefully. Expect a follow-up from me in a day
> or two.
>
> 2. valgrind did not show anything super promising. It did report this:
>
> ==6880== LEAK SUMMARY:
> ==6880==    definitely lost: 0 bytes in 0 blocks
> ==6880==    indirectly lost: 0 bytes in 0 blocks
> ==6880==      possibly lost: 288 bytes in 1 blocks
> ==6880==    still reachable: 10,552 bytes in 39 blocks
> ==6880==         suppressed: 0 bytes in 0 blocks
> (Bitcoin Core commit d78a880)
>
> and this:
> ==6778== LEAK SUMMARY:
> ==6778==    definitely lost: 0 bytes in 0 blocks
> ==6778==    indirectly lost: 0 bytes in 0 blocks
> ==6778==      possibly lost: 320 bytes in 1 blocks
> ==6778==    still reachable: 10,080 bytes in 32 blocks
> ==6778==         suppressed: 0 bytes in 0 blocks
> (Bitcoin XT commit fe446d)
>
> I haven't found anything in there yet that I think would produce the
> multi-GB memory usage after running for a few days, but I could be missing
> it. Email me if you want the full log.
>
> I did not try running getblocktemplate while valgrind was running. I'll
> have to try that. I also have not let valgrind run for more than an hour.
>
>
>
> P.S.: Sorry for all the cross-post confusion and consequent flamewar
> fallout. While it's probably too late for this thread, I'll make sure to
> post in a manner that keeps the threads clearly separate in the future
> (e.g. different subject lines).
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>

--001a113dd26838b8c7052279fec9
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<p dir=3D"ltr">My nodes are continuously running getblocktemplate and getin=
fo, and I also suspected the issue is in either gbt or the rpc server.</p>
<p dir=3D"ltr">The instance only takes a few hours to get up to that memory=
 usage.</p>
<div class=3D"gmail_quote">On Oct 18, 2015 8:59 AM, &quot;Jonathan Toomim v=
ia bitcoin-dev&quot; &lt;<a href=3D"mailto:bitcoin-dev@lists.linuxfoundatio=
n.org">bitcoin-dev@lists.linuxfoundation.org</a>&gt; wrote:<br type=3D"attr=
ibution"><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;borde=
r-left:1px #ccc solid;padding-left:1ex"><div style=3D"word-wrap:break-word"=
><div><div>On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan &lt;<a hre=
f=3D"mailto:laanwj@gmail.com" target=3D"_blank">laanwj@gmail.com</a>&gt; wr=
ote:</div><blockquote type=3D"cite">This is *most likely* the mempool, but =
is just not reported correctly.<br></blockquote><div><br></div><div>I did s=
ome testing with PR #6410&#39;s better mempool reporting. The improved repo=
rting suggests that actual in-memory usage (&quot;usage&quot;:) by CTxMemPo=
ol is about 2.5x to 3x higher than the serialized transaction sizes (&quot;=
bytes&quot;:). The excess memory usage that I&#39;m seeing is on the order =
of 100x higher than the mempool &quot;bytes&quot;: value. As such, I think =
it&#39;s unlikely that this is the mempool, or at least not normal/correct =
mempool behavior.=C2=A0</div><div><br></div><div>Another user (<a href=3D"m=
ailto:admin@multipool.us" target=3D"_blank">admin@multipool.us</a>) reporte=
d 35 GB of RSS usage. I&#39;m guessing his bitcoind has been running longer=
 than any of mine. His server definitely has more RAM. I don&#39;t know whi=
ch email list he is subscribed to (probably XT), so I&#39;m sharing it with=
 both lists to make sure you&#39;re all aware of how big an issue this can =
be.</div><br><blockquote type=3D"cite">In the meantime you can mitigate the=
 mempool growth by setting `-mintxfee`, see<br><a href=3D"https://github.co=
m/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding" t=
arget=3D"_blank">https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/releas=
e-notes.md#transaction-flooding</a><br></blockquote><div><br></div>I have m=
intxfee and minrelaytxfee set to about=C2=A00.00003, which is high enough t=
o exclude essentially all of the of the 14700-14800 byte flood transactions=
. My nodes&#39; mempools only contain about one or two blocks&#39; worth of=
 transactions. So I don&#39;t think this is correct either.</div><div><br><=
/div><div><br></div><div><br></div><div>Some additional notes on this issue=
:</div><div><br></div><div>1. I think it&#39;s related to CreateNewBlock() =
and getblocktemplate. I ran a Core bitcoind process (commit d78a880) overni=
ght with no mining connected to it, and (IIRC -- my memory is fuzzy) when I=
 woke up it was using around 400 MB of RSS and the mempool was at around &q=
uot;bytes&quot;:10MB, &quot;usage&quot;: 25MB. I ran ./bitcoin-cli getblock=
template once, and IIRC the RSS shot up to around 800 MB. I then ran getblo=
cktemplate every 5 seconds for about 30 minutes, and RSS climbed to 1180 MB=
. An hour after that with more getblocktemplates, and now RSS is at 1350 MB=
. [Edit: 1490 MB about 30 minutes later.] getmempoolinfo is still showing &=
quot;usage&quot; around 25MB or less.</div><div><br></div><div>I&#39;ll do =
some more testing with this and see if I can make it repeatable, and record=
 the results more carefully. Expect a follow-up from me in a day or two.</d=
iv><div><br></div><div>2. valgrind did not show anything super promising. I=
t did report this:</div><div><br></div><div><div style=3D"margin:0px;font-f=
amily:&#39;Andale Mono&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0=
)">=3D=3D6880=3D=3D LEAK SUMMARY:</div><div style=3D"margin:0px;font-family=
:&#39;Andale Mono&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)">=
=3D=3D6880=3D=3D=C2=A0 =C2=A0 definitely lost: 0 bytes in 0 blocks</div><di=
v style=3D"margin:0px;font-family:&#39;Andale Mono&#39;;color:rgb(41,249,20=
);background-color:rgb(0,0,0)">=3D=3D6880=3D=3D=C2=A0 =C2=A0 indirectly los=
t: 0 bytes in 0 blocks</div><div style=3D"margin:0px;font-family:&#39;Andal=
e Mono&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=
=3D=C2=A0 =C2=A0 =C2=A0 possibly lost: 288 bytes in 1 blocks</div><div styl=
e=3D"margin:0px;font-family:&#39;Andale Mono&#39;;color:rgb(41,249,20);back=
ground-color:rgb(0,0,0)">=3D=3D6880=3D=3D=C2=A0 =C2=A0 still reachable: 10,=
552 bytes in 39 blocks</div><div style=3D"margin:0px;font-family:&#39;Andal=
e Mono&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=
=3D =C2=A0 =C2=A0 =C2=A0 =C2=A0 suppressed: 0 bytes in 0 blocks</div></div>=
<div>(Bitcoin Core commit d78a880)</div><div><br></div><div>and this:</div>=
<div><div style=3D"margin:0px;font-family:&#39;Andale Mono&#39;;color:rgb(4=
1,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D LEAK SUMMARY:</div>=
<div style=3D"margin:0px;font-family:&#39;Andale Mono&#39;;color:rgb(41,249=
,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D=C2=A0 =C2=A0 definitely =
lost: 0 bytes in 0 blocks</div><div style=3D"margin:0px;font-family:&#39;An=
dale Mono&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=
=3D=3D=C2=A0 =C2=A0 indirectly lost: 0 bytes in 0 blocks</div><div style=3D=
"margin:0px;font-family:&#39;Andale Mono&#39;;color:rgb(41,249,20);backgrou=
nd-color:rgb(0,0,0)">=3D=3D6778=3D=3D=C2=A0 =C2=A0 =C2=A0 possibly lost: 32=
0 bytes in 1 blocks</div><div style=3D"margin:0px;font-family:&#39;Andale M=
ono&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D=
=C2=A0 =C2=A0 still reachable: 10,080 bytes in 32 blocks</div><div style=3D=
"margin:0px;font-family:&#39;Andale Mono&#39;;color:rgb(41,249,20);backgrou=
nd-color:rgb(0,0,0)">=3D=3D6778=3D=3D =C2=A0 =C2=A0 =C2=A0 =C2=A0 suppresse=
d: 0 bytes in 0 blocks</div></div><div>(Bitcoin XT commit=C2=A0fe446d)</div=
><div><br></div><div>I haven&#39;t found anything in there yet that I think=
 would produce the multi-GB memory usage after running for a few days, but =
I could be missing it. Email me if you want the full log.</div><div><br></d=
iv><div>I did not try running getblocktemplate while valgrind was running. =
I&#39;ll have to try that. I also have not let valgrind run for more than a=
n hour.=C2=A0</div><div><br></div><div><br></div><div><br></div><div>P.S.: =
Sorry for all the cross-post confusion and consequent flamewar fallout. Whi=
le it&#39;s probably too late for this thread, I&#39;ll make sure to post i=
n a manner that keeps the threads clearly separate in the future (e.g. diff=
erent subject lines).</div></div><br>______________________________________=
_________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.=
linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" =
rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail=
man/listinfo/bitcoin-dev</a><br>
<br></blockquote></div>

--001a113dd26838b8c7052279fec9--