path: root/22/3512a5f872bc5ac8a97dd882ebf40019b217a4

Return-Path: <admin@multipool.us>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
	[172.17.192.35])
	by mail.linuxfoundation.org (Postfix) with ESMTPS id B99AF410
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Mon, 19 Oct 2015 22:24:12 +0000 (UTC)
X-Greylist: whitelisted by SQLgrey-1.7.6
Received: from mail-ob0-f176.google.com (mail-ob0-f176.google.com
	[209.85.214.176])
	by smtp1.linuxfoundation.org (Postfix) with ESMTPS id DE18E122
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Mon, 19 Oct 2015 22:24:11 +0000 (UTC)
Received: by obbda8 with SMTP id da8so151538162obb.1
	for <bitcoin-dev@lists.linuxfoundation.org>;
	Mon, 19 Oct 2015 15:24:11 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20130820;
	h=x-gm-message-state:mime-version:in-reply-to:references:date
	:message-id:subject:from:to:cc:content-type;
	bh=dH56yv/sQF2QQQ3QKj83S2DTGVO5+ILjnNggaiY4KBE=;
	b=fFDWVrntA3wEYj6om3oYQF4Yz1y+wroSglQN24pSFeQnA0sm9TA8hrYe0uRfOkxfMf
	L8FbS5DQQLEeMKOwfoxcMp95YCY+7frjzQ+98NrsO2MkIPd59cc7zKAiotyy5cnCYgAF
	szDGwZoqS6N1o4cJd6bdFgfDkjFM3EdWW5RfI5tDdBYio+o9QDsFHEZrjzOllHm0BvR1
	Nycogfbfcj1pyryzTaldl81k73x4P4d1YfnNLZlyZhtm7Z0z2dqo796AcZ7EY4T3HrL5
	fgbbswgsg7Y0xIhqjc2/+MNc1VzZ0JpvSBXWoQQ81UybUVZqOr1A3v/nBxeVQJXDwaIa
	tA1A==
X-Gm-Message-State: ALoCoQkwCvRwQ6BA/WTZsRGbgoFHo7YKHfSbW1I3HQXwZq/P/Wx4yUdaev9FAV+31E8wiaeHOmVY
MIME-Version: 1.0
X-Received: by 10.182.133.9 with SMTP id oy9mr20196412obb.41.1445293451318;
	Mon, 19 Oct 2015 15:24:11 -0700 (PDT)
Received: by 10.182.220.193 with HTTP; Mon, 19 Oct 2015 15:24:11 -0700 (PDT)
In-Reply-To: <CAP3QyGJNBdsBtxYjOprRJ=YW2v-N_CopVQeSgDs6J4J8LMWuxA@mail.gmail.com>
References: <99C42DE7-814A-48F8-AB28-A5ADD77A9FD9@toom.im>
	<20151014093913.GB19607@amethyst.visucore.com>
	<F938BD02-3D80-4E99-BD1C-490543187895@toom.im>
	<CAP3QyGJNBdsBtxYjOprRJ=YW2v-N_CopVQeSgDs6J4J8LMWuxA@mail.gmail.com>
Date: Mon, 19 Oct 2015 15:24:11 -0700
Message-ID: <CAP3QyGLGeDc2qfzmGxmQcTLSJzXL0_uQrY4ab7EjBbQS2fxgug@mail.gmail.com>
From: Multipool Admin <admin@multipool.us>
To: Jonathan Toomim <j@toom.im>
Content-Type: multipart/alternative; boundary=e89a8ff1ccc2c32ff605227c9a7d
X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,HTML_MESSAGE,
	RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
	smtp1.linux-foundation.org
Cc: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>,
	bitcoin-xt <bitcoin-xt@googlegroups.com>
Subject: Re: [bitcoin-dev] Memory leaks?
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Mon, 19 Oct 2015 22:24:12 -0000

--e89a8ff1ccc2c32ff605227c9a7d
Content-Type: text/plain; charset=UTF-8

I should also mention that this is definitely not an attack coming from
connected nodes.  My node experiencing the issue is only connected to 3
other nodes, all of which I control (via connect=).

--Adam

On Mon, Oct 19, 2015 at 12:17 PM, Multipool Admin <admin@multipool.us>
wrote:

> My nodes are continuously running getblocktemplate and getinfo, and I also
> suspected the issue is in either gbt or the rpc server.
>
> The instance only takes a few hours to get up to that memory usage.
> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail.com>
>> wrote:
>>
>> This is *most likely* the mempool, but is just not reported correctly.
>>
>>
>> I did some testing with PR #6410's better mempool reporting. The improved
>> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
>> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
>> The excess memory usage that I'm seeing is on the order of 100x higher than
>> the mempool "bytes": value. As such, I think it's unlikely that this is the
>> mempool, or at least not normal/correct mempool behavior.
>>
>> Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm
>> guessing his bitcoind has been running longer than any of mine. His server
>> definitely has more RAM. I don't know which email list he is subscribed to
>> (probably XT), so I'm sharing it with both lists to make sure you're all
>> aware of how big an issue this can be.
>>
>> In the meantime you can mitigate the mempool growth by setting
>> `-mintxfee`, see
>>
>> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>>
>>
>> I have mintxfee and minrelaytxfee set to about 0.00003, which is high
>> enough to exclude essentially all of the of the 14700-14800 byte flood
>> transactions. My nodes' mempools only contain about one or two blocks'
>> worth of transactions. So I don't think this is correct either.
>>
>>
>>
>> Some additional notes on this issue:
>>
>> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
>> Core bitcoind process (commit d78a880) overnight with no mining connected
>> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
>> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
>> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
>> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
>> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
>> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
>> getmempoolinfo is still showing "usage" around 25MB or less.
>>
>> I'll do some more testing with this and see if I can make it repeatable,
>> and record the results more carefully. Expect a follow-up from me in a day
>> or two.
>>
>> 2. valgrind did not show anything super promising. It did report this:
>>
>> ==6880== LEAK SUMMARY:
>> ==6880==    definitely lost: 0 bytes in 0 blocks
>> ==6880==    indirectly lost: 0 bytes in 0 blocks
>> ==6880==      possibly lost: 288 bytes in 1 blocks
>> ==6880==    still reachable: 10,552 bytes in 39 blocks
>> ==6880==         suppressed: 0 bytes in 0 blocks
>> (Bitcoin Core commit d78a880)
>>
>> and this:
>> ==6778== LEAK SUMMARY:
>> ==6778==    definitely lost: 0 bytes in 0 blocks
>> ==6778==    indirectly lost: 0 bytes in 0 blocks
>> ==6778==      possibly lost: 320 bytes in 1 blocks
>> ==6778==    still reachable: 10,080 bytes in 32 blocks
>> ==6778==         suppressed: 0 bytes in 0 blocks
>> (Bitcoin XT commit fe446d)
>>
>> I haven't found anything in there yet that I think would produce the
>> multi-GB memory usage after running for a few days, but I could be missing
>> it. Email me if you want the full log.
>>
>> I did not try running getblocktemplate while valgrind was running. I'll
>> have to try that. I also have not let valgrind run for more than an hour.
>>
>>
>>
>> P.S.: Sorry for all the cross-post confusion and consequent flamewar
>> fallout. While it's probably too late for this thread, I'll make sure to
>> post in a manner that keeps the threads clearly separate in the future
>> (e.g. different subject lines).
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>

--e89a8ff1ccc2c32ff605227c9a7d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I should also mention that this is definitely not an attac=
k coming from connected nodes.=C2=A0 My node experiencing the issue is only=
 connected to 3 other nodes, all of which I control (via connect=3D).<div><=
br></div><div>--Adam</div></div><div class=3D"gmail_extra"><br><div class=
=3D"gmail_quote">On Mon, Oct 19, 2015 at 12:17 PM, Multipool Admin <span di=
r=3D"ltr">&lt;<a href=3D"mailto:admin@multipool.us" target=3D"_blank">admin=
@multipool.us</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" st=
yle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p di=
r=3D"ltr">My nodes are continuously running getblocktemplate and getinfo, a=
nd I also suspected the issue is in either gbt or the rpc server.</p>
<p dir=3D"ltr">The instance only takes a few hours to get up to that memory=
 usage.</p>
<div class=3D"gmail_quote"><div><div class=3D"h5">On Oct 18, 2015 8:59 AM, =
&quot;Jonathan Toomim via bitcoin-dev&quot; &lt;<a href=3D"mailto:bitcoin-d=
ev@lists.linuxfoundation.org" target=3D"_blank">bitcoin-dev@lists.linuxfoun=
dation.org</a>&gt; wrote:<br type=3D"attribution"></div></div><blockquote c=
lass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;=
padding-left:1ex"><div><div class=3D"h5"><div style=3D"word-wrap:break-word=
"><div><div>On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan &lt;<a hr=
ef=3D"mailto:laanwj@gmail.com" target=3D"_blank">laanwj@gmail.com</a>&gt; w=
rote:</div><blockquote type=3D"cite">This is *most likely* the mempool, but=
 is just not reported correctly.<br></blockquote><div><br></div><div>I did =
some testing with PR #6410&#39;s better mempool reporting. The improved rep=
orting suggests that actual in-memory usage (&quot;usage&quot;:) by CTxMemP=
ool is about 2.5x to 3x higher than the serialized transaction sizes (&quot=
;bytes&quot;:). The excess memory usage that I&#39;m seeing is on the order=
 of 100x higher than the mempool &quot;bytes&quot;: value. As such, I think=
 it&#39;s unlikely that this is the mempool, or at least not normal/correct=
 mempool behavior.=C2=A0</div><div><br></div><div>Another user (<a href=3D"=
mailto:admin@multipool.us" target=3D"_blank">admin@multipool.us</a>) report=
ed 35 GB of RSS usage. I&#39;m guessing his bitcoind has been running longe=
r than any of mine. His server definitely has more RAM. I don&#39;t know wh=
ich email list he is subscribed to (probably XT), so I&#39;m sharing it wit=
h both lists to make sure you&#39;re all aware of how big an issue this can=
 be.</div><br><blockquote type=3D"cite">In the meantime you can mitigate th=
e mempool growth by setting `-mintxfee`, see<br><a href=3D"https://github.c=
om/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding" =
target=3D"_blank">https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/relea=
se-notes.md#transaction-flooding</a><br></blockquote><div><br></div>I have =
mintxfee and minrelaytxfee set to about=C2=A00.00003, which is high enough =
to exclude essentially all of the of the 14700-14800 byte flood transaction=
s. My nodes&#39; mempools only contain about one or two blocks&#39; worth o=
f transactions. So I don&#39;t think this is correct either.</div><div><br>=
</div><div><br></div><div><br></div><div>Some additional notes on this issu=
e:</div><div><br></div><div>1. I think it&#39;s related to CreateNewBlock()=
 and getblocktemplate. I ran a Core bitcoind process (commit d78a880) overn=
ight with no mining connected to it, and (IIRC -- my memory is fuzzy) when =
I woke up it was using around 400 MB of RSS and the mempool was at around &=
quot;bytes&quot;:10MB, &quot;usage&quot;: 25MB. I ran ./bitcoin-cli getbloc=
ktemplate once, and IIRC the RSS shot up to around 800 MB. I then ran getbl=
ocktemplate every 5 seconds for about 30 minutes, and RSS climbed to 1180 M=
B. An hour after that with more getblocktemplates, and now RSS is at 1350 M=
B. [Edit: 1490 MB about 30 minutes later.] getmempoolinfo is still showing =
&quot;usage&quot; around 25MB or less.</div><div><br></div><div>I&#39;ll do=
 some more testing with this and see if I can make it repeatable, and recor=
d the results more carefully. Expect a follow-up from me in a day or two.</=
div><div><br></div><div>2. valgrind did not show anything super promising. =
It did report this:</div><div><br></div><div><div style=3D"margin:0px;font-=
family:&#39;Andale Mono&#39;;color:rgb(41,249,20);background-color:rgb(0,0,=
0)">=3D=3D6880=3D=3D LEAK SUMMARY:</div><div style=3D"margin:0px;font-famil=
y:&#39;Andale Mono&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)">=
=3D=3D6880=3D=3D=C2=A0 =C2=A0 definitely lost: 0 bytes in 0 blocks</div><di=
v style=3D"margin:0px;font-family:&#39;Andale Mono&#39;;color:rgb(41,249,20=
);background-color:rgb(0,0,0)">=3D=3D6880=3D=3D=C2=A0 =C2=A0 indirectly los=
t: 0 bytes in 0 blocks</div><div style=3D"margin:0px;font-family:&#39;Andal=
e Mono&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=
=3D=C2=A0 =C2=A0 =C2=A0 possibly lost: 288 bytes in 1 blocks</div><div styl=
e=3D"margin:0px;font-family:&#39;Andale Mono&#39;;color:rgb(41,249,20);back=
ground-color:rgb(0,0,0)">=3D=3D6880=3D=3D=C2=A0 =C2=A0 still reachable: 10,=
552 bytes in 39 blocks</div><div style=3D"margin:0px;font-family:&#39;Andal=
e Mono&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=
=3D =C2=A0 =C2=A0 =C2=A0 =C2=A0 suppressed: 0 bytes in 0 blocks</div></div>=
<div>(Bitcoin Core commit d78a880)</div><div><br></div><div>and this:</div>=
<div><div style=3D"margin:0px;font-family:&#39;Andale Mono&#39;;color:rgb(4=
1,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D LEAK SUMMARY:</div>=
<div style=3D"margin:0px;font-family:&#39;Andale Mono&#39;;color:rgb(41,249=
,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D=C2=A0 =C2=A0 definitely =
lost: 0 bytes in 0 blocks</div><div style=3D"margin:0px;font-family:&#39;An=
dale Mono&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=
=3D=3D=C2=A0 =C2=A0 indirectly lost: 0 bytes in 0 blocks</div><div style=3D=
"margin:0px;font-family:&#39;Andale Mono&#39;;color:rgb(41,249,20);backgrou=
nd-color:rgb(0,0,0)">=3D=3D6778=3D=3D=C2=A0 =C2=A0 =C2=A0 possibly lost: 32=
0 bytes in 1 blocks</div><div style=3D"margin:0px;font-family:&#39;Andale M=
ono&#39;;color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D=
=C2=A0 =C2=A0 still reachable: 10,080 bytes in 32 blocks</div><div style=3D=
"margin:0px;font-family:&#39;Andale Mono&#39;;color:rgb(41,249,20);backgrou=
nd-color:rgb(0,0,0)">=3D=3D6778=3D=3D =C2=A0 =C2=A0 =C2=A0 =C2=A0 suppresse=
d: 0 bytes in 0 blocks</div></div><div>(Bitcoin XT commit=C2=A0fe446d)</div=
><div><br></div><div>I haven&#39;t found anything in there yet that I think=
 would produce the multi-GB memory usage after running for a few days, but =
I could be missing it. Email me if you want the full log.</div><div><br></d=
iv><div>I did not try running getblocktemplate while valgrind was running. =
I&#39;ll have to try that. I also have not let valgrind run for more than a=
n hour.=C2=A0</div><div><br></div><div><br></div><div><br></div><div>P.S.: =
Sorry for all the cross-post confusion and consequent flamewar fallout. Whi=
le it&#39;s probably too late for this thread, I&#39;ll make sure to post i=
n a manner that keeps the threads clearly separate in the future (e.g. diff=
erent subject lines).</div></div><br></div></div><span class=3D"">_________=
______________________________________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">=
bitcoin-dev@lists.linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" =
rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail=
man/listinfo/bitcoin-dev</a><br>
<br></span></blockquote></div>
</blockquote></div><br></div>

--e89a8ff1ccc2c32ff605227c9a7d--