1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
|
Return-Path: <j@toom.im>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
[172.17.192.35])
by mail.linuxfoundation.org (Postfix) with ESMTPS id 3D2B567
for <bitcoin-dev@lists.linuxfoundation.org>;
Sun, 18 Oct 2015 15:59:16 +0000 (UTC)
X-Greylist: from auto-whitelisted by SQLgrey-1.7.6
Received: from d.mail.sonic.net (d.mail.sonic.net [64.142.111.50])
by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 8BAB2F5
for <bitcoin-dev@lists.linuxfoundation.org>;
Sun, 18 Oct 2015 15:59:15 +0000 (UTC)
Received: from [192.168.1.190] (63.135.62.197.nwinternet.com [63.135.62.197]
(may be forged)) (authenticated bits=0)
by d.mail.sonic.net (8.15.1/8.15.1) with ESMTPSA id t9IFxBdk025659
(version=TLSv1 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT);
Sun, 18 Oct 2015 08:59:12 -0700
Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\))
Content-Type: multipart/signed;
boundary="Apple-Mail=_818AD798-4BDF-41A6-A9DB-8EF5B45DC674";
protocol="application/pgp-signature"; micalg=pgp-sha512
X-Pgp-Agent: GPGMail 2.5.2
From: Jonathan Toomim <j@toom.im>
In-Reply-To: <20151014093913.GB19607@amethyst.visucore.com>
Date: Sun, 18 Oct 2015 08:59:11 -0700
Message-Id: <F938BD02-3D80-4E99-BD1C-490543187895@toom.im>
References: <99C42DE7-814A-48F8-AB28-A5ADD77A9FD9@toom.im>
<20151014093913.GB19607@amethyst.visucore.com>
To: Bitcoin Dev <bitcoin-dev@lists.linuxfoundation.org>
X-Mailer: Apple Mail (2.1878.6)
X-Sonic-CAuth: UmFuZG9tSVaH4ltZBozHiZKlWjc4I41+A1rFRRQziasL5jdC+2o2HOpvfsnWXbPXiZJwLiimOjO4IPaC0iqbCP6zBGLeFfhV
X-Sonic-ID: C;RGdqLLF15RGrb+K7sH9FTg== M;5mbSLLF15RGrb+K7sH9FTg==
X-Sonic-Spam-Details: 1.7/5.0 by cerberusd
X-Spam-Status: No, score=-0.9 required=5.0 tests=BAYES_00,HTML_MESSAGE,
RCVD_IN_DNSWL_LOW,URIBL_BLACK autolearn=no version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
smtp1.linux-foundation.org
Cc: bitcoin-xt <bitcoin-xt@googlegroups.com>
Subject: Re: [bitcoin-dev] Memory leaks?
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Development Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Sun, 18 Oct 2015 15:59:16 -0000
--Apple-Mail=_818AD798-4BDF-41A6-A9DB-8EF5B45DC674
Content-Type: multipart/alternative;
boundary="Apple-Mail=_9C963CE6-6CA2-4ABE-9D61-E57718ABA125"
--Apple-Mail=_9C963CE6-6CA2-4ABE-9D61-E57718ABA125
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=us-ascii
On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj@gmail.com> =
wrote:
> This is *most likely* the mempool, but is just not reported correctly.
I did some testing with PR #6410's better mempool reporting. The =
improved reporting suggests that actual in-memory usage ("usage":) by =
CTxMemPool is about 2.5x to 3x higher than the serialized transaction =
sizes ("bytes":). The excess memory usage that I'm seeing is on the =
order of 100x higher than the mempool "bytes": value. As such, I think =
it's unlikely that this is the mempool, or at least not normal/correct =
mempool behavior.
Another user (admin@multipool.us) reported 35 GB of RSS usage. I'm =
guessing his bitcoind has been running longer than any of mine. His =
server definitely has more RAM. I don't know which email list he is =
subscribed to (probably XT), so I'm sharing it with both lists to make =
sure you're all aware of how big an issue this can be.
> In the meantime you can mitigate the mempool growth by setting =
`-mintxfee`, see
> =
https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#trans=
action-flooding
I have mintxfee and minrelaytxfee set to about 0.00003, which is high =
enough to exclude essentially all of the of the 14700-14800 byte flood =
transactions. My nodes' mempools only contain about one or two blocks' =
worth of transactions. So I don't think this is correct either.
Some additional notes on this issue:
1. I think it's related to CreateNewBlock() and getblocktemplate. I ran =
a Core bitcoind process (commit d78a880) overnight with no mining =
connected to it, and (IIRC -- my memory is fuzzy) when I woke up it was =
using around 400 MB of RSS and the mempool was at around "bytes":10MB, =
"usage": 25MB. I ran ./bitcoin-cli getblocktemplate once, and IIRC the =
RSS shot up to around 800 MB. I then ran getblocktemplate every 5 =
seconds for about 30 minutes, and RSS climbed to 1180 MB. An hour after =
that with more getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 =
MB about 30 minutes later.] getmempoolinfo is still showing "usage" =
around 25MB or less.
I'll do some more testing with this and see if I can make it repeatable, =
and record the results more carefully. Expect a follow-up from me in a =
day or two.
2. valgrind did not show anything super promising. It did report this:
=3D=3D6880=3D=3D LEAK SUMMARY:
=3D=3D6880=3D=3D definitely lost: 0 bytes in 0 blocks
=3D=3D6880=3D=3D indirectly lost: 0 bytes in 0 blocks
=3D=3D6880=3D=3D possibly lost: 288 bytes in 1 blocks
=3D=3D6880=3D=3D still reachable: 10,552 bytes in 39 blocks
=3D=3D6880=3D=3D suppressed: 0 bytes in 0 blocks
(Bitcoin Core commit d78a880)
and this:
=3D=3D6778=3D=3D LEAK SUMMARY:
=3D=3D6778=3D=3D definitely lost: 0 bytes in 0 blocks
=3D=3D6778=3D=3D indirectly lost: 0 bytes in 0 blocks
=3D=3D6778=3D=3D possibly lost: 320 bytes in 1 blocks
=3D=3D6778=3D=3D still reachable: 10,080 bytes in 32 blocks
=3D=3D6778=3D=3D suppressed: 0 bytes in 0 blocks
(Bitcoin XT commit fe446d)
I haven't found anything in there yet that I think would produce the =
multi-GB memory usage after running for a few days, but I could be =
missing it. Email me if you want the full log.
I did not try running getblocktemplate while valgrind was running. I'll =
have to try that. I also have not let valgrind run for more than an =
hour.
P.S.: Sorry for all the cross-post confusion and consequent flamewar =
fallout. While it's probably too late for this thread, I'll make sure to =
post in a manner that keeps the threads clearly separate in the future =
(e.g. different subject lines).
--Apple-Mail=_9C963CE6-6CA2-4ABE-9D61-E57718ABA125
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=us-ascii
<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: =
after-white-space;"><div><div>On Oct 14, 2015, at 2:39 AM, Wladimir J. =
van der Laan <<a =
href=3D"mailto:laanwj@gmail.com">laanwj@gmail.com</a>> =
wrote:</div><blockquote type=3D"cite">This is *most likely* the mempool, =
but is just not reported =
correctly.<br></blockquote><div><br></div><div>I did some testing with =
PR #6410's better mempool reporting. The improved reporting suggests =
that actual in-memory usage ("usage":) by CTxMemPool is about 2.5x to 3x =
higher than the serialized transaction sizes ("bytes":). The excess =
memory usage that I'm seeing is on the order of 100x higher than the =
mempool "bytes": value. As such, I think it's unlikely that this is the =
mempool, or at least not normal/correct mempool =
behavior. </div><div><br></div><div>Another user (<a =
href=3D"mailto:admin@multipool.us">admin@multipool.us</a>) reported 35 =
GB of RSS usage. I'm guessing his bitcoind has been running longer than =
any of mine. His server definitely has more RAM. I don't know which =
email list he is subscribed to (probably XT), so I'm sharing it with =
both lists to make sure you're all aware of how big an issue this can =
be.</div><br><blockquote type=3D"cite">In the meantime you can mitigate =
the mempool growth by setting `-mintxfee`, see<br><a =
href=3D"https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.=
md#transaction-flooding">https://github.com/bitcoin/bitcoin/blob/v0.11.0/d=
oc/release-notes.md#transaction-flooding</a><br></blockquote><div><br></di=
v>I have mintxfee and minrelaytxfee set to about 0.00003, which is =
high enough to exclude essentially all of the of the 14700-14800 byte =
flood transactions. My nodes' mempools only contain about one or two =
blocks' worth of transactions. So I don't think this is correct =
either.</div><div><br></div><div><br></div><div><br></div><div>Some =
additional notes on this issue:</div><div><br></div><div>1. I think it's =
related to CreateNewBlock() and getblocktemplate. I ran a Core bitcoind =
process (commit d78a880) overnight with no mining connected to it, and =
(IIRC -- my memory is fuzzy) when I woke up it was using around 400 MB =
of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I ran =
./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around =
800 MB. I then ran getblocktemplate every 5 seconds for about 30 =
minutes, and RSS climbed to 1180 MB. An hour after that with more =
getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 MB about 30 =
minutes later.] getmempoolinfo is still showing "usage" around 25MB or =
less.</div><div><br></div><div>I'll do some more testing with this and =
see if I can make it repeatable, and record the results more carefully. =
Expect a follow-up from me in a day or two.</div><div><br></div><div>2. =
valgrind did not show anything super promising. It did report =
this:</div><div><br></div><div><div style=3D"margin: 0px; font-family: =
'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, =
0);">=3D=3D6880=3D=3D LEAK SUMMARY:</div><div style=3D"margin: 0px; =
font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: =
rgb(0, 0, 0);">=3D=3D6880=3D=3D definitely lost: 0 bytes in =
0 blocks</div><div style=3D"margin: 0px; font-family: 'Andale Mono'; =
color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);">=3D=3D6880=3D=3D=
indirectly lost: 0 bytes in 0 blocks</div><div =
style=3D"margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, =
20); background-color: rgb(0, 0, 0);">=3D=3D6880=3D=3D =
possibly lost: 288 bytes in 1 blocks</div><div style=3D"margin: =
0px; font-family: 'Andale Mono'; color: rgb(41, 249, 20); =
background-color: rgb(0, 0, 0);">=3D=3D6880=3D=3D still =
reachable: 10,552 bytes in 39 blocks</div><div style=3D"margin: 0px; =
font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: =
rgb(0, 0, 0);">=3D=3D6880=3D=3D suppressed: =
0 bytes in 0 blocks</div></div><div>(Bitcoin Core commit =
d78a880)</div><div><br></div><div>and this:</div><div><div =
style=3D"margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, =
20); background-color: rgb(0, 0, 0);">=3D=3D6778=3D=3D LEAK =
SUMMARY:</div><div style=3D"margin: 0px; font-family: 'Andale Mono'; =
color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);">=3D=3D6778=3D=3D=
definitely lost: 0 bytes in 0 blocks</div><div =
style=3D"margin: 0px; font-family: 'Andale Mono'; color: rgb(41, 249, =
20); background-color: rgb(0, 0, 0);">=3D=3D6778=3D=3D =
indirectly lost: 0 bytes in 0 blocks</div><div style=3D"margin: 0px; =
font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: =
rgb(0, 0, 0);">=3D=3D6778=3D=3D possibly lost: 320 =
bytes in 1 blocks</div><div style=3D"margin: 0px; font-family: 'Andale =
Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, =
0);">=3D=3D6778=3D=3D still reachable: 10,080 bytes in 32 =
blocks</div><div style=3D"margin: 0px; font-family: 'Andale Mono'; =
color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);">=3D=3D6778=3D=3D=
suppressed: 0 bytes in 0 =
blocks</div></div><div>(Bitcoin XT =
commit fe446d)</div><div><br></div><div>I haven't found anything in =
there yet that I think would produce the multi-GB memory usage after =
running for a few days, but I could be missing it. Email me if you want =
the full log.</div><div><br></div><div>I did not try running =
getblocktemplate while valgrind was running. I'll have to try that. I =
also have not let valgrind run for more than an =
hour. </div><div><br></div><div><br></div><div><br></div><div>P.S.: =
Sorry for all the cross-post confusion and consequent flamewar fallout. =
While it's probably too late for this thread, I'll make sure to post in =
a manner that keeps the threads clearly separate in the future (e.g. =
different subject lines).</div></body></html>=
--Apple-Mail=_9C963CE6-6CA2-4ABE-9D61-E57718ABA125--
--Apple-Mail=_818AD798-4BDF-41A6-A9DB-8EF5B45DC674
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=signature.asc
Content-Type: application/pgp-signature;
name=signature.asc
Content-Description: Message signed with OpenPGP using GPGMail
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org
iQEcBAEBCgAGBQJWI8HPAAoJEIEuMk4MG0P1G8wIAI9e4zumzWl23AHnBGyEqrAr
tQ7aR+2WDTpUrNY2ELl874ZYtbLiF+joLrq+kfV1qNnygTjBaBs2Kn93yAT6/qXb
1RnwoEw9/q2YptP6A2RUOz1L3LDtmxhiZmrluZAPVIX7+OhdJsfj5dR/UUnMsJqF
tWIfoi9iO14BRCEfDi44OT/BWXbUTRQfgnrfvMjCulRIEGzKtAK+SGONL8tJGdD7
bjZ33WTP23sOuLX7b3udbdmO1ywCwAgQob6H7QIU8m2kL6A/59d2kKIVtUcnjQSm
rZQyHcySVkb01kmcdBzEuI8f4I782G7aMmWgf5XLnF0eodhnIub2J9PP1QjXIhg=
=4O5/
-----END PGP SIGNATURE-----
--Apple-Mail=_818AD798-4BDF-41A6-A9DB-8EF5B45DC674--
|