1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
|
Return-Path: <admin@multipool.us>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
[172.17.192.35])
by mail.linuxfoundation.org (Postfix) with ESMTPS id B99AF410
for <bitcoin-dev@lists.linuxfoundation.org>;
Mon, 19 Oct 2015 22:24:12 +0000 (UTC)
X-Greylist: whitelisted by SQLgrey-1.7.6
Received: from mail-ob0-f176.google.com (mail-ob0-f176.google.com
[209.85.214.176])
by smtp1.linuxfoundation.org (Postfix) with ESMTPS id DE18E122
for <bitcoin-dev@lists.linuxfoundation.org>;
Mon, 19 Oct 2015 22:24:11 +0000 (UTC)
Received: by obbda8 with SMTP id da8so151538162obb.1
for <bitcoin-dev@lists.linuxfoundation.org>;
Mon, 19 Oct 2015 15:24:11 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20130820;
h=x-gm-message-state:mime-version:in-reply-to:references:date
:message-id:subject:from:to:cc:content-type;
bh=dH56yv/sQF2QQQ3QKj83S2DTGVO5+ILjnNggaiY4KBE=;
b=fFDWVrntA3wEYj6om3oYQF4Yz1y+wroSglQN24pSFeQnA0sm9TA8hrYe0uRfOkxfMf
L8FbS5DQQLEeMKOwfoxcMp95YCY+7frjzQ+98NrsO2MkIPd59cc7zKAiotyy5cnCYgAF
szDGwZoqS6N1o4cJd6bdFgfDkjFM3EdWW5RfI5tDdBYio+o9QDsFHEZrjzOllHm0BvR1
Nycogfbfcj1pyryzTaldl81k73x4P4d1YfnNLZlyZhtm7Z0z2dqo796AcZ7EY4T3HrL5
fgbbswgsg7Y0xIhqjc2/+MNc1VzZ0JpvSBXWoQQ81UybUVZqOr1A3v/nBxeVQJXDwaIa
tA1A==
X-Gm-Message-State: ALoCoQkwCvRwQ6BA/WTZsRGbgoFHo7YKHfSbW1I3HQXwZq/P/Wx4yUdaev9FAV+31E8wiaeHOmVY
MIME-Version: 1.0
X-Received: by 10.182.133.9 with SMTP id oy9mr20196412obb.41.1445293451318;
Mon, 19 Oct 2015 15:24:11 -0700 (PDT)
Received: by 10.182.220.193 with HTTP; Mon, 19 Oct 2015 15:24:11 -0700 (PDT)
In-Reply-To: <CAP3QyGJNBdsBtxYjOprRJ=YW2v-N_CopVQeSgDs6J4J8LMWuxA@mail.gmail.com>
References: <99C42DE7-814A-48F8-AB28-A5ADD77A9FD9@toom.im>
<20151014093913.GB19607@amethyst.visucore.com>
<F938BD02-3D80-4E99-BD1C-490543187895@toom.im>
<CAP3QyGJNBdsBtxYjOprRJ=YW2v-N_CopVQeSgDs6J4J8LMWuxA@mail.gmail.com>
Date: Mon, 19 Oct 2015 15:24:11 -0700
Message-ID: <CAP3QyGLGeDc2qfzmGxmQcTLSJzXL0_uQrY4ab7EjBbQS2fxgug@mail.gmail.com>
From: Multipool Admin <admin@multipool.us>
To: Jonathan Toomim <j@toom.im>
Content-Type: multipart/alternative; boundary=e89a8ff1ccc2c32ff605227c9a7d
X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,HTML_MESSAGE,
RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
smtp1.linux-foundation.org
Cc: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>,
bitcoin-xt <bitcoin-xt@googlegroups.com>
Subject: Re: [bitcoin-dev] Memory leaks?
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Mon, 19 Oct 2015 22:24:12 -0000
--e89a8ff1ccc2c32ff605227c9a7d
Content-Type: text/plain; charset=UTF-8
I should also mention that this is definitely not an attack coming from
connected nodes. My node experiencing the issue is only connected to 3
other nodes, all of which I control (via connect=).
--Adam
On Mon, Oct 19, 2015 at 12:17 PM, Multipool Admin <admin@multipool.us>
wrote:
> My nodes are continuously running getblocktemplate and getinfo, and I also
> suspected the issue is in either gbt or the rpc server.
>
> The instance only takes a few hours to get up to that memory usage.
> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail.com>
>> wrote:
>>
>> This is *most likely* the mempool, but is just not reported correctly.
>>
>>
>> I did some testing with PR #6410's better mempool reporting. The improved
>> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
>> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
>> The excess memory usage that I'm seeing is on the order of 100x higher than
>> the mempool "bytes": value. As such, I think it's unlikely that this is the
>> mempool, or at least not normal/correct mempool behavior.
>>
>> Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm
>> guessing his bitcoind has been running longer than any of mine. His server
>> definitely has more RAM. I don't know which email list he is subscribed to
>> (probably XT), so I'm sharing it with both lists to make sure you're all
>> aware of how big an issue this can be.
>>
>> In the meantime you can mitigate the mempool growth by setting
>> `-mintxfee`, see
>>
>> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>>
>>
>> I have mintxfee and minrelaytxfee set to about 0.00003, which is high
>> enough to exclude essentially all of the of the 14700-14800 byte flood
>> transactions. My nodes' mempools only contain about one or two blocks'
>> worth of transactions. So I don't think this is correct either.
>>
>>
>>
>> Some additional notes on this issue:
>>
>> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
>> Core bitcoind process (commit d78a880) overnight with no mining connected
>> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
>> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
>> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
>> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
>> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
>> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
>> getmempoolinfo is still showing "usage" around 25MB or less.
>>
>> I'll do some more testing with this and see if I can make it repeatable,
>> and record the results more carefully. Expect a follow-up from me in a day
>> or two.
>>
>> 2. valgrind did not show anything super promising. It did report this:
>>
>> ==6880== LEAK SUMMARY:
>> ==6880== definitely lost: 0 bytes in 0 blocks
>> ==6880== indirectly lost: 0 bytes in 0 blocks
>> ==6880== possibly lost: 288 bytes in 1 blocks
>> ==6880== still reachable: 10,552 bytes in 39 blocks
>> ==6880== suppressed: 0 bytes in 0 blocks
>> (Bitcoin Core commit d78a880)
>>
>> and this:
>> ==6778== LEAK SUMMARY:
>> ==6778== definitely lost: 0 bytes in 0 blocks
>> ==6778== indirectly lost: 0 bytes in 0 blocks
>> ==6778== possibly lost: 320 bytes in 1 blocks
>> ==6778== still reachable: 10,080 bytes in 32 blocks
>> ==6778== suppressed: 0 bytes in 0 blocks
>> (Bitcoin XT commit fe446d)
>>
>> I haven't found anything in there yet that I think would produce the
>> multi-GB memory usage after running for a few days, but I could be missing
>> it. Email me if you want the full log.
>>
>> I did not try running getblocktemplate while valgrind was running. I'll
>> have to try that. I also have not let valgrind run for more than an hour.
>>
>>
>>
>> P.S.: Sorry for all the cross-post confusion and consequent flamewar
>> fallout. While it's probably too late for this thread, I'll make sure to
>> post in a manner that keeps the threads clearly separate in the future
>> (e.g. different subject lines).
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
--e89a8ff1ccc2c32ff605227c9a7d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">I should also mention that this is definitely not an attac=
k coming from connected nodes.=C2=A0 My node experiencing the issue is only=
connected to 3 other nodes, all of which I control (via connect=3D).<div><=
br></div><div>--Adam</div></div><div class=3D"gmail_extra"><br><div class=
=3D"gmail_quote">On Mon, Oct 19, 2015 at 12:17 PM, Multipool Admin <span di=
r=3D"ltr"><<a href=3D"mailto:admin@multipool.us" target=3D"_blank">admin=
@multipool.us</a>></span> wrote:<br><blockquote class=3D"gmail_quote" st=
yle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p di=
r=3D"ltr">My nodes are continuously running getblocktemplate and getinfo, a=
nd I also suspected the issue is in either gbt or the rpc server.</p>
<p dir=3D"ltr">The instance only takes a few hours to get up to that memory=
usage.</p>
<div class=3D"gmail_quote"><div><div class=3D"h5">On Oct 18, 2015 8:59 AM, =
"Jonathan Toomim via bitcoin-dev" <<a href=3D"mailto:bitcoin-d=
ev@lists.linuxfoundation.org" target=3D"_blank">bitcoin-dev@lists.linuxfoun=
dation.org</a>> wrote:<br type=3D"attribution"></div></div><blockquote c=
lass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;=
padding-left:1ex"><div><div class=3D"h5"><div style=3D"word-wrap:break-word=
"><div><div>On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <<a hr=
ef=3D"mailto:laanwj@gmail.com" target=3D"_blank">laanwj@gmail.com</a>> w=
rote:</div><blockquote type=3D"cite">This is *most likely* the mempool, but=
is just not reported correctly.<br></blockquote><div><br></div><div>I did =
some testing with PR #6410's better mempool reporting. The improved rep=
orting suggests that actual in-memory usage ("usage":) by CTxMemP=
ool is about 2.5x to 3x higher than the serialized transaction sizes ("=
;bytes":). The excess memory usage that I'm seeing is on the order=
of 100x higher than the mempool "bytes": value. As such, I think=
it's unlikely that this is the mempool, or at least not normal/correct=
mempool behavior.=C2=A0</div><div><br></div><div>Another user (<a href=3D"=
mailto:admin@multipool.us" target=3D"_blank">admin@multipool.us</a>) report=
ed 35 GB of RSS usage. I'm guessing his bitcoind has been running longe=
r than any of mine. His server definitely has more RAM. I don't know wh=
ich email list he is subscribed to (probably XT), so I'm sharing it wit=
h both lists to make sure you're all aware of how big an issue this can=
be.</div><br><blockquote type=3D"cite">In the meantime you can mitigate th=
e mempool growth by setting `-mintxfee`, see<br><a href=3D"https://github.c=
om/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding" =
target=3D"_blank">https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/relea=
se-notes.md#transaction-flooding</a><br></blockquote><div><br></div>I have =
mintxfee and minrelaytxfee set to about=C2=A00.00003, which is high enough =
to exclude essentially all of the of the 14700-14800 byte flood transaction=
s. My nodes' mempools only contain about one or two blocks' worth o=
f transactions. So I don't think this is correct either.</div><div><br>=
</div><div><br></div><div><br></div><div>Some additional notes on this issu=
e:</div><div><br></div><div>1. I think it's related to CreateNewBlock()=
and getblocktemplate. I ran a Core bitcoind process (commit d78a880) overn=
ight with no mining connected to it, and (IIRC -- my memory is fuzzy) when =
I woke up it was using around 400 MB of RSS and the mempool was at around &=
quot;bytes":10MB, "usage": 25MB. I ran ./bitcoin-cli getbloc=
ktemplate once, and IIRC the RSS shot up to around 800 MB. I then ran getbl=
ocktemplate every 5 seconds for about 30 minutes, and RSS climbed to 1180 M=
B. An hour after that with more getblocktemplates, and now RSS is at 1350 M=
B. [Edit: 1490 MB about 30 minutes later.] getmempoolinfo is still showing =
"usage" around 25MB or less.</div><div><br></div><div>I'll do=
some more testing with this and see if I can make it repeatable, and recor=
d the results more carefully. Expect a follow-up from me in a day or two.</=
div><div><br></div><div>2. valgrind did not show anything super promising. =
It did report this:</div><div><br></div><div><div style=3D"margin:0px;font-=
family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,=
0)">=3D=3D6880=3D=3D LEAK SUMMARY:</div><div style=3D"margin:0px;font-famil=
y:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=
=3D=3D6880=3D=3D=C2=A0 =C2=A0 definitely lost: 0 bytes in 0 blocks</div><di=
v style=3D"margin:0px;font-family:'Andale Mono';color:rgb(41,249,20=
);background-color:rgb(0,0,0)">=3D=3D6880=3D=3D=C2=A0 =C2=A0 indirectly los=
t: 0 bytes in 0 blocks</div><div style=3D"margin:0px;font-family:'Andal=
e Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=
=3D=C2=A0 =C2=A0 =C2=A0 possibly lost: 288 bytes in 1 blocks</div><div styl=
e=3D"margin:0px;font-family:'Andale Mono';color:rgb(41,249,20);back=
ground-color:rgb(0,0,0)">=3D=3D6880=3D=3D=C2=A0 =C2=A0 still reachable: 10,=
552 bytes in 39 blocks</div><div style=3D"margin:0px;font-family:'Andal=
e Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6880=3D=
=3D =C2=A0 =C2=A0 =C2=A0 =C2=A0 suppressed: 0 bytes in 0 blocks</div></div>=
<div>(Bitcoin Core commit d78a880)</div><div><br></div><div>and this:</div>=
<div><div style=3D"margin:0px;font-family:'Andale Mono';color:rgb(4=
1,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D LEAK SUMMARY:</div>=
<div style=3D"margin:0px;font-family:'Andale Mono';color:rgb(41,249=
,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D=C2=A0 =C2=A0 definitely =
lost: 0 bytes in 0 blocks</div><div style=3D"margin:0px;font-family:'An=
dale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=
=3D=3D=C2=A0 =C2=A0 indirectly lost: 0 bytes in 0 blocks</div><div style=3D=
"margin:0px;font-family:'Andale Mono';color:rgb(41,249,20);backgrou=
nd-color:rgb(0,0,0)">=3D=3D6778=3D=3D=C2=A0 =C2=A0 =C2=A0 possibly lost: 32=
0 bytes in 1 blocks</div><div style=3D"margin:0px;font-family:'Andale M=
ono';color:rgb(41,249,20);background-color:rgb(0,0,0)">=3D=3D6778=3D=3D=
=C2=A0 =C2=A0 still reachable: 10,080 bytes in 32 blocks</div><div style=3D=
"margin:0px;font-family:'Andale Mono';color:rgb(41,249,20);backgrou=
nd-color:rgb(0,0,0)">=3D=3D6778=3D=3D =C2=A0 =C2=A0 =C2=A0 =C2=A0 suppresse=
d: 0 bytes in 0 blocks</div></div><div>(Bitcoin XT commit=C2=A0fe446d)</div=
><div><br></div><div>I haven't found anything in there yet that I think=
would produce the multi-GB memory usage after running for a few days, but =
I could be missing it. Email me if you want the full log.</div><div><br></d=
iv><div>I did not try running getblocktemplate while valgrind was running. =
I'll have to try that. I also have not let valgrind run for more than a=
n hour.=C2=A0</div><div><br></div><div><br></div><div><br></div><div>P.S.: =
Sorry for all the cross-post confusion and consequent flamewar fallout. Whi=
le it's probably too late for this thread, I'll make sure to post i=
n a manner that keeps the threads clearly separate in the future (e.g. diff=
erent subject lines).</div></div><br></div></div><span class=3D"">_________=
______________________________________<br>
bitcoin-dev mailing list<br>
<a href=3D"mailto:bitcoin-dev@lists.linuxfoundation.org" target=3D"_blank">=
bitcoin-dev@lists.linuxfoundation.org</a><br>
<a href=3D"https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" =
rel=3D"noreferrer" target=3D"_blank">https://lists.linuxfoundation.org/mail=
man/listinfo/bitcoin-dev</a><br>
<br></span></blockquote></div>
</blockquote></div><br></div>
--e89a8ff1ccc2c32ff605227c9a7d--
|