path: root/70/7408c705c123db9d3b82e39a54bd532bb02961

Return-Path: <j@toom.im>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
	[172.17.192.35])
	by mail.linuxfoundation.org (Postfix) with ESMTPS id D0FC64D3
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Wed, 21 Oct 2015 03:01:24 +0000 (UTC)
X-Greylist: from auto-whitelisted by SQLgrey-1.7.6
Received: from c.mail.sonic.net (c.mail.sonic.net [64.142.111.80])
	by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 30707E1
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Wed, 21 Oct 2015 03:01:22 +0000 (UTC)
Received: from [192.168.1.190] (63.135.62.197.nwinternet.com [63.135.62.197]
	(may be forged)) (authenticated bits=0)
	by c.mail.sonic.net (8.15.1/8.15.1) with ESMTPSA id t9L31H9j002198
	(version=TLSv1 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT);
	Tue, 20 Oct 2015 20:01:18 -0700
Content-Type: multipart/signed;
	boundary="Apple-Mail=_83BDB11D-1F73-4B97-9508-FF4D7B4BDA07";
	protocol="application/pgp-signature"; micalg=pgp-sha512
Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\))
X-Pgp-Agent: GPGMail 2.5.2
From: Jonathan Toomim <j@toom.im>
In-Reply-To: <1FE17DEB-8F77-4A60-A644-46A4F97D0E24@toom.im>
Date: Tue, 20 Oct 2015 20:01:16 -0700
Message-Id: <984D5FD5-9871-43FC-BD44-5F2E6EFD0671@toom.im>
References: <99C42DE7-814A-48F8-AB28-A5ADD77A9FD9@toom.im>
	<20151014093913.GB19607@amethyst.visucore.com>
	<F938BD02-3D80-4E99-BD1C-490543187895@toom.im>
	<CAP3QyGJNBdsBtxYjOprRJ=YW2v-N_CopVQeSgDs6J4J8LMWuxA@mail.gmail.com>
	<CA+w+GKTU6C7KKFx9dDd_--s1DQCO15n=034Lku2-kTYKf96XYw@mail.gmail.com>
	<1FE17DEB-8F77-4A60-A644-46A4F97D0E24@toom.im>
To: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>,
	bitcoin-xt <bitcoin-xt@googlegroups.com>
X-Mailer: Apple Mail (2.1878.6)
X-Sonic-CAuth: UmFuZG9tSVYREh/4W0hJpIYP7Lo7CrnE7dvhQXDd/nuaPuwSqQR46/Qy1DDPgBnlC8yq37Yfku+Yn0uWu0sHYBNBdYJjHfaR
X-Sonic-ID: C;Yja0/5935RGSWL0U9jFv0A== M;vAM3AKB35RGSWL0U9jFv0A==
X-Sonic-Spam-Details: 0.0/5.0 by cerberusd
X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,HTML_MESSAGE,
	RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
	smtp1.linux-foundation.org
Subject: Re: [bitcoin-dev] Memory leaks?
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Wed, 21 Oct 2015 03:01:24 -0000


--Apple-Mail=_83BDB11D-1F73-4B97-9508-FF4D7B4BDA07
Content-Type: multipart/alternative;
	boundary="Apple-Mail=_AD29A490-0C81-493D-A9F4-B7389A1D22F4"


--Apple-Mail=_AD29A490-0C81-493D-A9F4-B7389A1D22F4
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

More notes:

1. I ran a side-by-side comparison with two bitcoind processes (Core, =
same recent git commit as before) on the same computer with the same =
settings running on different ports. With both processes, I logged RSS =
(via /proc/$pid/status) every 6 seconds. With one of those processes, I =
also ran bitcoin-cli getblocktemplate > /dev/null every 6 seconds. I let =
that run for about 30 hours. A graph and links to the CSVs of raw data =
are below. Results seem pretty clear: the getblocktemplate RPC is =
implicated in this issue.


http://toom.im/files/memlog8518.csv
http://toom.im/files/memlog-nogbt-8503.csv
http://toom.im/files/bitcoind_memory_usage_gbt.png


2. I ran valgrind twice, for about 6 hours each, on bitcoind while =
hitting it with getblocktemplate every 6 hours. Full valgrind output can =
be found at these two URLs:

http://toom.im/files/valgrind-gbt-1.log
http://toom.im/files/valgrind-gbt-2.log

The summary:

=3D=3D4064=3D=3D LEAK SUMMARY:
=3D=3D4064=3D=3D    definitely lost: 0 bytes in 0 blocks
=3D=3D4064=3D=3D    indirectly lost: 0 bytes in 0 blocks
=3D=3D4064=3D=3D      possibly lost: 288 bytes in 1 blocks
=3D=3D4064=3D=3D    still reachable: 527,594 bytes in 4,367 blocks
=3D=3D4064=3D=3D         suppressed: 0 bytes in 0 blocks
The main components of that still reachable section seem to just be one =
output of CreateNewBlock that's cached in case another getblocktemplate =
request is received before any new transactions come in:

=3D=3D4064=3D=3D 98,304 bytes in 1 blocks are still reachable in loss =
record 39 of 40
=3D=3D4064=3D=3D    at 0x4C29180: operator new(unsigned long) =
(vg_replace_malloc.c:324)
=3D=3D4064=3D=3D    by 0x28EAA1: =
__gnu_cxx::new_allocator<CTransaction>::allocate(unsigned long, void =
const*) (new_allocator.h:104)
=3D=3D4064=3D=3D    by 0x27EE44: =
__gnu_cxx::__alloc_traits<std::allocator<CTransaction> =
>::allocate(std::allocator<CTransaction>&, unsigned long) =
(alloc_traits.h:182)
=3D=3D4064=3D=3D    by 0x26DFB0: std::_Vector_base<CTransaction, =
std::allocator<CTransaction> >::_M_allocate(unsigned long) =
(stl_vector.h:170)
=3D=3D4064=3D=3D    by 0x2D5BDE: std::vector<CTransaction, =
std::allocator<CTransaction> =
>::_M_insert_aux(__gnu_cxx::__normal_iterator<CTransaction*, =
std::vector<CTransaction, std::allocator<CTransaction> > >, CTransaction =
const&) (vector.tcc:353)
=3D=3D4064=3D=3D    by 0x2D3FF8: std::vector<CTransaction, =
std::allocator<CTransaction> >::push_back(CTransaction const&) =
(stl_vector.h:925)
=3D=3D4064=3D=3D    by 0x2D113E: CreateNewBlock(CScript const&) =
(miner.cpp:298)
=3D=3D4064=3D=3D    by 0x442D78: getblocktemplate(UniValue const&, bool) =
(rpcmining.cpp:513)
=3D=3D4064=3D=3D    by 0x390CEB: CRPCTable::execute(std::string const&, =
UniValue const&) const (rpcserver.cpp:526)
=3D=3D4064=3D=3D    by 0x41C5AB: HTTPReq_JSONRPC(HTTPRequest*, =
std::string const&) (httprpc.cpp:125)
=3D=3D4064=3D=3D    by 0x3559BD: =
boost::detail::function::void_function_invoker2<bool (*)(HTTPRequest*, =
std::string const&), void, HTTPRequest*, std::string =
const&>::invoke(boost::detail::function::function_buffer&, HTTPRequest*, =
std::string const&) (function_template.hpp:112)
=3D=3D4064=3D=3D    by 0x422520: boost::function2<void, HTTPRequest*, =
std::string const&>::operator()(HTTPRequest*, std::string const&) const =
(function_template.hpp:767)

There are a few other similar loss records (mostly referring to pblock =
or pblocktemplate in CreateNewBlock(...), but I see nothing that can =
explain the multi-GB memory consumption.

3. One user on the bitcointalk p2pool thread =
(https://bitcointalk.org/index.php?topic=3D18313.msg12733791#msg12733791) =
claimed that he had this memory usage issue on Linux, but not on Mac OS =
X, under a GBT workload in both situations. If this is true, that would =
suggest this might be a fragmentation issue due to poor memory =
allocation. The other likely hypothesis is bloated caches. Looking into =
those two possibilities will be my next steps.



On Oct 20, 2015, at 5:39 AM, Jonathan Toomim <j@toom.im> wrote:

> I did that Sunday twice. I'll report the results soon. Short version =
is that it looks like valgrind is just finding 200 kB to 600 kB of =
pblocktemplate, which is declared as a static pointer. Not exactly the =
multi-GB leak I'm looking for, but possibly related.
>=20
> I've also got two bitcoind processes running on the same machine that =
I started at the same time, running on different ports, all with the =
same settings, but one of which is serving getblocktemplate every 5-6 =
seconds and the other is not, while logging RSS on both every 6 seconds. =
RSS for the non-serving node is now 734 MB, and for the serving node =
1997 MB. Graphs coming soon.
>=20
>=20
> On Oct 20, 2015, at 3:12 AM, Mike Hearn <hearn@vinumeris.com> wrote:
>=20
>> OK, then running under Valgrind whilst sending gbt RPCs would be the =
next step.
>>=20
>> On Mon, Oct 19, 2015 at 9:17 PM, Multipool Admin <admin@multipool.us> =
wrote:
>> My nodes are continuously running getblocktemplate and getinfo, and I =
also suspected the issue is in either gbt or the rpc server.
>>=20
>> The instance only takes a few hours to get up to that memory usage.
>>=20
>> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" =
<bitcoin-dev@lists.linuxfoundation.org> wrote:
>> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan =
<laanwj@gmail.com> wrote:
>>> This is *most likely* the mempool, but is just not reported =
correctly.
>>=20
>> I did some testing with PR #6410's better mempool reporting. The =
improved reporting suggests that actual in-memory usage ("usage":) by =
CTxMemPool is about 2.5x to 3x higher than the serialized transaction =
sizes ("bytes":). The excess memory usage that I'm seeing is on the =
order of 100x higher than the mempool "bytes": value. As such, I think =
it's unlikely that this is the mempool, or at least not normal/correct =
mempool behavior.
>>=20
>> Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm =
guessing his bitcoind has been running longer than any of mine. His =
server definitely has more RAM. I don't know which email list he is =
subscribed to (probably XT), so I'm sharing it with both lists to make =
sure you're all aware of how big an issue this can be.
>>=20
>>> In the meantime you can mitigate the mempool growth by setting =
`-mintxfee`, see
>>> =
https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#trans=
action-flooding
>>=20
>> I have mintxfee and minrelaytxfee set to about 0.00003, which is high =
enough to exclude essentially all of the of the 14700-14800 byte flood =
transactions. My nodes' mempools only contain about one or two blocks' =
worth of transactions. So I don't think this is correct either.
>>=20
>>=20
>>=20
>> Some additional notes on this issue:
>>=20
>> 1. I think it's related to CreateNewBlock() and getblocktemplate. I =
ran a Core bitcoind process (commit d78a880) overnight with no mining =
connected to it, and (IIRC -- my memory is fuzzy) when I woke up it was =
using around 400 MB of RSS and the mempool was at around "bytes":10MB, =
"usage": 25MB. I ran ./bitcoin-cli getblocktemplate once, and IIRC the =
RSS shot up to around 800 MB. I then ran getblocktemplate every 5 =
seconds for about 30 minutes, and RSS climbed to 1180 MB. An hour after =
that with more getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 =
MB about 30 minutes later.] getmempoolinfo is still showing "usage" =
around 25MB or less.
>>=20
>> I'll do some more testing with this and see if I can make it =
repeatable, and record the results more carefully. Expect a follow-up =
from me in a day or two.
>>=20
>> 2. valgrind did not show anything super promising. It did report =
this:
>>=20
>> =3D=3D6880=3D=3D LEAK SUMMARY:
>> =3D=3D6880=3D=3D    definitely lost: 0 bytes in 0 blocks
>> =3D=3D6880=3D=3D    indirectly lost: 0 bytes in 0 blocks
>> =3D=3D6880=3D=3D      possibly lost: 288 bytes in 1 blocks
>> =3D=3D6880=3D=3D    still reachable: 10,552 bytes in 39 blocks
>> =3D=3D6880=3D=3D         suppressed: 0 bytes in 0 blocks
>> (Bitcoin Core commit d78a880)
>>=20
>> and this:
>> =3D=3D6778=3D=3D LEAK SUMMARY:
>> =3D=3D6778=3D=3D    definitely lost: 0 bytes in 0 blocks
>> =3D=3D6778=3D=3D    indirectly lost: 0 bytes in 0 blocks
>> =3D=3D6778=3D=3D      possibly lost: 320 bytes in 1 blocks
>> =3D=3D6778=3D=3D    still reachable: 10,080 bytes in 32 blocks
>> =3D=3D6778=3D=3D         suppressed: 0 bytes in 0 blocks
>> (Bitcoin XT commit fe446d)
>>=20
>> I haven't found anything in there yet that I think would produce the =
multi-GB memory usage after running for a few days, but I could be =
missing it. Email me if you want the full log.
>>=20
>> I did not try running getblocktemplate while valgrind was running. =
I'll have to try that. I also have not let valgrind run for more than an =
hour.
>>=20
>>=20
>>=20
>> P.S.: Sorry for all the cross-post confusion and consequent flamewar =
fallout. While it's probably too late for this thread, I'll make sure to =
post in a manner that keeps the threads clearly separate in the future =
(e.g. different subject lines).
>>=20
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>=20
>>=20
>> --
>> You received this message because you are subscribed to the Google =
Groups "bitcoin-xt" group.
>> To unsubscribe from this group and stop receiving emails from it, =
send an email to bitcoin-xt+unsubscribe@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>=20
>=20


--Apple-Mail=_AD29A490-0C81-493D-A9F4-B7389A1D22F4
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=us-ascii

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: =
after-white-space;"><div>More notes:</div><div><br></div><div>1. I ran a =
side-by-side comparison with two bitcoind processes (Core, same recent =
git commit as before) on the same computer with the same settings =
running on different ports. With both processes, I logged RSS (via =
/proc/$pid/status) every 6 seconds. With one of those processes, I also =
ran bitcoin-cli getblocktemplate &gt; /dev/null every 6 seconds. I let =
that run for about 30 hours. A graph and links to the CSVs of raw data =
are below. Results seem pretty clear: the getblocktemplate RPC is =
implicated in this issue.</div><blockquote style=3D"margin: 0 0 0 40px; =
border: none; padding: 0px;"><div><p style=3D"font-family: Times; =
font-size: medium; widows: 1;"><a =
href=3D"http://toom.im/files/bitcoind_memory_usage_gbt.png"><img =
width=3D"800" =
src=3D"http://toom.im/files/bitcoind_memory_usage_gbt.png"></a></p></div><=
div><a =
href=3D"http://toom.im/files/memlog8518.csv">http://toom.im/files/memlog85=
18.csv</a></div><div><a =
href=3D"http://toom.im/files/memlog-nogbt-8503.csv">http://toom.im/files/m=
emlog-nogbt-8503.csv</a></div><div><a =
href=3D"http://toom.im/files/bitcoind_memory_usage_gbt.png">http://toom.im=
/files/bitcoind_memory_usage_gbt.png</a></div></blockquote><div><br></div>=
<div><br></div><div>2. I ran valgrind twice, for about 6 hours each, on =
bitcoind while hitting it with getblocktemplate every 6 hours. Full =
valgrind output can be found at these two =
URLs:</div><div><br></div><div><a =
href=3D"http://toom.im/files/valgrind-gbt-1.log">http://toom.im/files/valg=
rind-gbt-1.log</a></div><div><a =
href=3D"http://toom.im/files/valgrind-gbt-2.log">http://toom.im/files/valg=
rind-gbt-2.log</a></div><div><br></div><div>The =
summary:</div><div><br></div><div><blockquote style=3D"margin: 0 0 0 =
40px; border: none; padding: 0px;"><div><pre style=3D"widows: 1; =
word-wrap: break-word; white-space: pre-wrap;"><div style=3D"margin: =
0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); =
background-color: rgb(0, 0, 0);">=3D=3D4064=3D=3D LEAK =
SUMMARY:</div><div style=3D"margin: 0px; font-family: 'Andale Mono'; =
color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);">=3D=3D4064=3D=3D=
&nbsp; &nbsp; definitely lost: 0 bytes in 0 blocks</div><div =
style=3D"margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, =
20); background-color: rgb(0, 0, 0);">=3D=3D4064=3D=3D&nbsp; &nbsp; =
indirectly lost: 0 bytes in 0 blocks</div><div style=3D"margin: 0px; =
font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: =
rgb(0, 0, 0);">=3D=3D4064=3D=3D&nbsp; &nbsp; &nbsp; possibly lost: 288 =
bytes in 1 blocks</div><div style=3D"margin: 0px; font-family: 'Andale =
Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, =
0);">=3D=3D4064=3D=3D&nbsp; &nbsp; still reachable: 527,594 bytes in =
4,367 blocks</div><div style=3D"margin: 0px; font-family: 'Andale Mono'; =
color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);">=3D=3D4064=3D=3D=
 &nbsp; &nbsp; &nbsp; &nbsp; suppressed: 0 bytes in 0 =
blocks</div></pre></div></blockquote>The main components of that still =
reachable section seem to just be one output of CreateNewBlock that's =
cached in case another getblocktemplate request is received before any =
new transactions come in:</div><div><br></div><blockquote style=3D"margin:=
 0 0 0 40px; border: none; padding: 0px;"><pre style=3D"widows: 1; =
word-wrap: break-word; white-space: pre-wrap;"><div style=3D"margin: =
0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); =
background-color: rgb(0, 0, 0); position: static; z-index: auto;"><div =
style=3D"margin: 0px;">=3D=3D4064=3D=3D 98,304 bytes in 1 blocks are =
still reachable in loss record 39 of 40</div><div style=3D"margin: =
0px;">=3D=3D4064=3D=3D&nbsp; &nbsp; at 0x4C29180: operator new(unsigned =
long) (vg_replace_malloc.c:324)</div><div style=3D"margin: =
0px;">=3D=3D4064=3D=3D&nbsp; &nbsp; by 0x28EAA1: =
__gnu_cxx::new_allocator&lt;CTransaction&gt;::allocate(unsigned long, =
void const*) (new_allocator.h:104)</div><div style=3D"margin: =
0px;">=3D=3D4064=3D=3D&nbsp; &nbsp; by 0x27EE44: =
__gnu_cxx::__alloc_traits&lt;std::allocator&lt;CTransaction&gt; =
&gt;::allocate(std::allocator&lt;CTransaction&gt;&amp;, unsigned long) =
(alloc_traits.h:182)</div><div style=3D"margin: 0px;">=3D=3D4064=3D=3D&nbs=
p; &nbsp; by 0x26DFB0: std::_Vector_base&lt;CTransaction, =
std::allocator&lt;CTransaction&gt; &gt;::_M_allocate(unsigned long) =
(stl_vector.h:170)</div><div style=3D"margin: 0px;">=3D=3D4064=3D=3D&nbsp;=
 &nbsp; by 0x2D5BDE: std::vector&lt;CTransaction, =
std::allocator&lt;CTransaction&gt; =
&gt;::_M_insert_aux(__gnu_cxx::__normal_iterator&lt;CTransaction*, =
std::vector&lt;CTransaction, std::allocator&lt;CTransaction&gt; &gt; =
&gt;, CTransaction const&amp;) (vector.tcc:353)</div><div style=3D"margin:=
 0px;">=3D=3D4064=3D=3D&nbsp; &nbsp; by 0x2D3FF8: =
std::vector&lt;CTransaction, std::allocator&lt;CTransaction&gt; =
&gt;::push_back(CTransaction const&amp;) (stl_vector.h:925)</div><div =
style=3D"margin: 0px;">=3D=3D4064=3D=3D&nbsp; &nbsp; by 0x2D113E: =
CreateNewBlock(CScript const&amp;) (miner.cpp:298)</div><div =
style=3D"margin: 0px;">=3D=3D4064=3D=3D&nbsp; &nbsp; by 0x442D78: =
getblocktemplate(UniValue const&amp;, bool) =
(rpcmining.cpp:513)</div><div style=3D"margin: 0px;">=3D=3D4064=3D=3D&nbsp=
; &nbsp; by 0x390CEB: CRPCTable::execute(std::string const&amp;, =
UniValue const&amp;) const (rpcserver.cpp:526)</div><div style=3D"margin: =
0px;">=3D=3D4064=3D=3D&nbsp; &nbsp; by 0x41C5AB: =
HTTPReq_JSONRPC(HTTPRequest*, std::string const&amp;) =
(httprpc.cpp:125)</div><div style=3D"margin: 0px;">=3D=3D4064=3D=3D&nbsp; =
&nbsp; by 0x3559BD: =
boost::detail::function::void_function_invoker2&lt;bool =
(*)(HTTPRequest*, std::string const&amp;), void, HTTPRequest*, =
std::string =
const&amp;&gt;::invoke(boost::detail::function::function_buffer&amp;, =
HTTPRequest*, std::string const&amp;) =
(function_template.hpp:112)</div><div style=3D"margin: =
0px;">=3D=3D4064=3D=3D&nbsp; &nbsp; by 0x422520: =
boost::function2&lt;void, HTTPRequest*, std::string =
const&amp;&gt;::operator()(HTTPRequest*, std::string const&amp;) const =
(function_template.hpp:767)</div>
</div></pre></blockquote><div>There are a few other similar loss records =
(mostly referring to pblock or pblocktemplate in CreateNewBlock(...), =
but I see nothing that can explain the multi-GB memory =
consumption.&nbsp;</div><div><br></div><div>3. One user on the =
bitcointalk p2pool thread (<a =
href=3D"https://bitcointalk.org/index.php?topic=3D18313.msg12733791#msg127=
33791">https://bitcointalk.org/index.php?topic=3D18313.msg12733791#msg1273=
3791</a>) claimed that he had this memory usage issue on Linux, but not =
on Mac OS X, under a GBT workload in both situations. If this is true, =
that would suggest this might be a fragmentation issue due to poor =
memory allocation. The other likely hypothesis is bloated caches. =
Looking into those two possibilities will be my next =
steps.</div><div><br></div><div><br></div><br><div><div>On Oct 20, 2015, =
at 5:39 AM, Jonathan Toomim &lt;<a =
href=3D"mailto:j@toom.im">j@toom.im</a>&gt; wrote:</div><br =
class=3D"Apple-interchange-newline"><blockquote type=3D"cite"><meta =
http-equiv=3D"Content-Type" content=3D"text/html charset=3Dus-ascii"><div =
style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; =
-webkit-line-break: after-white-space;">I did that Sunday twice. I'll =
report the results soon. Short version is that it looks like valgrind is =
just finding 200 kB to 600 kB of pblocktemplate, which is declared as a =
static pointer. Not exactly the multi-GB leak I'm looking for, but =
possibly related.<div><br></div><div>I've also got two bitcoind =
processes running on the same machine that I started at the same time, =
running on different ports, all with the same settings, but one of which =
is serving getblocktemplate every 5-6 seconds and the other is not, =
while logging RSS on both every 6 seconds. RSS for the non-serving node =
is now 734 MB, and for the serving node 1997 MB. Graphs coming =
soon.</div><div><div><br></div><div><br><div><div>On Oct 20, 2015, at =
3:12 AM, Mike Hearn &lt;<a =
href=3D"mailto:hearn@vinumeris.com">hearn@vinumeris.com</a>&gt; =
wrote:</div><br class=3D"Apple-interchange-newline"><blockquote =
type=3D"cite"><div dir=3D"ltr">OK, then running under Valgrind whilst =
sending gbt RPCs would be the next step.</div><div =
class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Mon, Oct 19, =
2015 at 9:17 PM, Multipool Admin <span dir=3D"ltr">&lt;<a =
href=3D"mailto:admin@multipool.us" =
target=3D"_blank">admin@multipool.us</a>&gt;</span> =
wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir=3D"ltr">My =
nodes are continuously running getblocktemplate and getinfo, and I also =
suspected the issue is in either gbt or the rpc server.</p><p =
dir=3D"ltr">The instance only takes a few hours to get up to that memory =
usage.</p>
<div class=3D"gmail_quote"><div><div class=3D"h5">On Oct 18, 2015 8:59 =
AM, "Jonathan Toomim via bitcoin-dev" &lt;<a =
href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" =
target=3D"_blank">bitcoin-dev@lists.linuxfoundation.org</a>&gt; =
wrote:<br type=3D"attribution"></div></div><blockquote =
class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc =
solid;padding-left:1ex"><div><div class=3D"h5"><div =
style=3D"word-wrap:break-word"><div><div>On Oct 14, 2015, at 2:39 AM, =
Wladimir J. van der Laan &lt;<a href=3D"mailto:laanwj@gmail.com" =
target=3D"_blank">laanwj@gmail.com</a>&gt; wrote:</div><blockquote =
type=3D"cite">This is *most likely* the mempool, but is just not =
reported correctly.<br></blockquote><div><br></div><div>I did some =
testing with PR #6410's better mempool reporting. The improved reporting =
suggests that actual in-memory usage ("usage":) by CTxMemPool is about =
2.5x to 3x higher than the serialized transaction sizes ("bytes":). The =
excess memory usage that I'm seeing is on the order of 100x higher than =
the mempool "bytes": value. As such, I think it's unlikely that this is =
the mempool, or at least not normal/correct mempool =
behavior.&nbsp;</div><div><br></div><div>Another user (<a =
href=3D"mailto:admin@multipool.us" =
target=3D"_blank">admin@multipool.us</a>) reported 35 GB of RSS usage. =
I'm guessing his bitcoind has been running longer than any of mine. His =
server definitely has more RAM. I don't know which email list he is =
subscribed to (probably XT), so I'm sharing it with both lists to make =
sure you're all aware of how big an issue this can =
be.</div><br><blockquote type=3D"cite">In the meantime you can mitigate =
the mempool growth by setting `-mintxfee`, see<br><a =
href=3D"https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.=
md#transaction-flooding" =
target=3D"_blank">https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/rele=
ase-notes.md#transaction-flooding</a><br></blockquote><div><br></div>I =
have mintxfee and minrelaytxfee set to about&nbsp;0.00003, which is high =
enough to exclude essentially all of the of the 14700-14800 byte flood =
transactions. My nodes' mempools only contain about one or two blocks' =
worth of transactions. So I don't think this is correct =
either.</div><div><br></div><div><br></div><div><br></div><div>Some =
additional notes on this issue:</div><div><br></div><div>1. I think it's =
related to CreateNewBlock() and getblocktemplate. I ran a Core bitcoind =
process (commit d78a880) overnight with no mining connected to it, and =
(IIRC -- my memory is fuzzy) when I woke up it was using around 400 MB =
of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I ran =
./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around =
800 MB. I then ran getblocktemplate every 5 seconds for about 30 =
minutes, and RSS climbed to 1180 MB. An hour after that with more =
getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 MB about 30 =
minutes later.] getmempoolinfo is still showing "usage" around 25MB or =
less.</div><div><br></div><div>I'll do some more testing with this and =
see if I can make it repeatable, and record the results more carefully. =
Expect a follow-up from me in a day or two.</div><div><br></div><div>2. =
valgrind did not show anything super promising. It did report =
this:</div><div><br></div><div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=3D =
LEAK SUMMARY:</div><div style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=3D&n=
bsp; &nbsp; definitely lost: 0 bytes in 0 blocks</div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=3D&n=
bsp; &nbsp; indirectly lost: 0 bytes in 0 blocks</div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=3D&n=
bsp; &nbsp; &nbsp; possibly lost: 288 bytes in 1 blocks</div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=3D&n=
bsp; &nbsp; still reachable: 10,552 bytes in 39 blocks</div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=3D =
&nbsp; &nbsp; &nbsp; &nbsp; suppressed: 0 bytes in 0 =
blocks</div></div><div>(Bitcoin Core commit =
d78a880)</div><div><br></div><div>and this:</div><div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D =
LEAK SUMMARY:</div><div style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D&n=
bsp; &nbsp; definitely lost: 0 bytes in 0 blocks</div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D&n=
bsp; &nbsp; indirectly lost: 0 bytes in 0 blocks</div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D&n=
bsp; &nbsp; &nbsp; possibly lost: 320 bytes in 1 blocks</div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D&n=
bsp; &nbsp; still reachable: 10,080 bytes in 32 blocks</div><div =
style=3D"margin:0px;font-family:'Andale =
Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D =
&nbsp; &nbsp; &nbsp; &nbsp; suppressed: 0 bytes in 0 =
blocks</div></div><div>(Bitcoin XT =
commit&nbsp;fe446d)</div><div><br></div><div>I haven't found anything in =
there yet that I think would produce the multi-GB memory usage after =
running for a few days, but I could be missing it. Email me if you want =
the full log.</div><div><br></div><div>I did not try running =
getblocktemplate while valgrind was running. I'll have to try that. I =
also have not let valgrind run for more than an =
hour.&nbsp;</div><div><br></div><div><br></div><div><br></div><div>P.S.: =
Sorry for all the cross-post confusion and consequent flamewar fallout. =
While it's probably too late for this thread, I'll make sure to post in =
a manner that keeps the threads clearly separate in the future (e.g. =
different subject lines).</div></div><br></div></div><span =
class=3D"">_______________________________________________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" =
target=3D"_blank">bitcoin-dev@lists.linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev"=
 rel=3D"noreferrer" =
target=3D"_blank">https://lists.linuxfoundation.org/mailman/listinfo/bitco=
in-dev</a><br>
<br></span></blockquote></div><div class=3D"HOEnZb"><div =
class=3D"h5"><div><br class=3D"webkit-block-placeholder"></div>

-- <br>
You received this message because you are subscribed to the Google =
Groups "bitcoin-xt" group.<br>
To unsubscribe from this group and stop receiving emails from it, send =
an email to <a href=3D"mailto:bitcoin-xt+unsubscribe@googlegroups.com" =
target=3D"_blank">bitcoin-xt+unsubscribe@googlegroups.com</a>.<br>
For more options, visit <a href=3D"https://groups.google.com/d/optout" =
target=3D"_blank">https://groups.google.com/d/optout</a>.<br>
</div></div></blockquote></div><br></div>
=
</blockquote></div><br></div></div></div></blockquote></div><br></body></h=
tml>=

--Apple-Mail=_AD29A490-0C81-493D-A9F4-B7389A1D22F4--

--Apple-Mail=_83BDB11D-1F73-4B97-9508-FF4D7B4BDA07
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP using GPGMail

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iQEcBAEBCgAGBQJWJv/8AAoJEIEuMk4MG0P1Pl8IAKQjDuS1NPXKCxJ87HH2hGtI
GC7Ei31/bYE0lxtKMYsN/AG3gI8JYpXRXEH1+A6Xn/H5lcfh1sl5iOibrxqCsyst
46Zps6/9ibdsSWuX1RQCARUf21jDpUytPssQm5AiOptlSLcsDdiGENvzuNYvIAT6
favryjktnHtQIJM4SD5BVS52lym5rt/9BKLSXH5BaXYyWPbrBy0CDMQAholxomiO
Rj2xSwNwwe+yEn/kwIqQSQGOtjsPUTQBZPVcZPUP23gxk0O56GqTAjYt6wvOyKX3
jMoSF4yGSSggcFFGmIQc6pm/Cfr5g+uSmzNW7eI48R2irJaa3NN+SSDHns0ck14=
=ZNB0
-----END PGP SIGNATURE-----

--Apple-Mail=_83BDB11D-1F73-4B97-9508-FF4D7B4BDA07--