1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
|
Received: from sog-mx-3.v43.ch3.sourceforge.com ([172.29.43.193]
helo=mx.sourceforge.net)
by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
(envelope-from <pieter.wuille@gmail.com>) id 1SuhKm-0005Uw-Sn
for bitcoin-development@lists.sourceforge.net;
Fri, 27 Jul 2012 09:59:48 +0000
Received-SPF: pass (sog-mx-3.v43.ch3.sourceforge.com: domain of gmail.com
designates 209.85.214.175 as permitted sender)
client-ip=209.85.214.175; envelope-from=pieter.wuille@gmail.com;
helo=mail-ob0-f175.google.com;
Received: from mail-ob0-f175.google.com ([209.85.214.175])
by sog-mx-3.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
(Exim 4.76) id 1SuhKm-0008Ts-4Z
for bitcoin-development@lists.sourceforge.net;
Fri, 27 Jul 2012 09:59:48 +0000
Received: by obcva7 with SMTP id va7so4215558obc.34
for <bitcoin-development@lists.sourceforge.net>;
Fri, 27 Jul 2012 02:59:42 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.60.12.8 with SMTP id u8mr2675239oeb.46.1343383182623; Fri, 27
Jul 2012 02:59:42 -0700 (PDT)
Received: by 10.182.10.3 with HTTP; Fri, 27 Jul 2012 02:59:42 -0700 (PDT)
Received: by 10.182.10.3 with HTTP; Fri, 27 Jul 2012 02:59:42 -0700 (PDT)
In-Reply-To: <CAPg+sBjpqqxL_GxYc+8Ry7DcXMkouO9bYi5VqOyEjj_5s0x06Q@mail.gmail.com>
References: <CAD2Ti29dqCYoOMcX0zcOQnpLGCxnCjYHHqMzyyq+hvcVQ2c7nQ@mail.gmail.com>
<CAAS2fgTHrWdXHbX8oiLgnrhdC+yxL4QyPd7S4iB8RMNip_sUGg@mail.gmail.com>
<A428177D-62AD-4712-9053-47B7ED5DBC84@mac.com>
<CAD2Ti2-3sR_qusfmiStb8pzaxaB8DsPaK8a2+LWm_uL+DwvzeA@mail.gmail.com>
<CAAS2fgREzk_dU0ie+YvDdRwKcTk6tk_i=a2Bb74w9uF=EwYhGA@mail.gmail.com>
<CAD2Ti2_Pz9-SsHP49+6MKnM0er9zdAaFKDQaOgqDpju1_igd_g@mail.gmail.com>
<CAD2Ti28snGOZn9mCSALZ341TNCex23zxKHCKYztnMK3cF=jaTQ@mail.gmail.com>
<CAPg+sBjpqqxL_GxYc+8Ry7DcXMkouO9bYi5VqOyEjj_5s0x06Q@mail.gmail.com>
Date: Fri, 27 Jul 2012 11:59:42 +0200
Message-ID: <CAPg+sBiT15KfRyBSeH15RKTcAPbq3Q12+8=w2dK8QncrWbD=sQ@mail.gmail.com>
From: Pieter Wuille <pieter.wuille@gmail.com>
To: Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Content-Type: multipart/alternative; boundary=e89a8fb1ef5665c26c04c5ccc58f
X-Spam-Score: -0.6 (/)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
See http://spamassassin.org/tag/ for more details.
-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
sender-domain
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(pieter.wuille[at]gmail.com)
-0.0 SPF_PASS SPF: sender matches SPF record
1.0 HTML_MESSAGE BODY: HTML included in message
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
author's domain
0.1 DKIM_SIGNED Message has a DKIM or DK signature,
not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
X-Headers-End: 1SuhKm-0008Ts-4Z
Subject: Re: [Bitcoin-development] Scalability issues
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Fri, 27 Jul 2012 09:59:49 -0000
--e89a8fb1ef5665c26c04c5ccc58f
Content-Type: text/plain; charset=ISO-8859-1
And now to the list...
On Jul 27, 2012 6:21 AM, "grarpamp" <grarpamp@gmail.com> wrote:
>
> Update: this class of machine just became useless for bitcoin.
> When blk0002.dat was created to store more blocks, all forward
> progress processing blocks turned into losing ground by 20 or so
> a day. Guessing both datfiles were being accessed at once resulting
> in disk based overload. I've not seen any other mentions of crypto
> in this thread so I'm not sure how well new hardware would perform.
> Going shopping I guess.
I doubt the encryption is the problem. I have a much more recent machine
(i7 with 8 GiB RAM), and my blockchain is on a (userspace!) encrypted
filesystem. However, I do not notice any measurable slowdown from doing so.
You are however using a filesystem (ZFS) that is uses its own filesystem
caching implementation to reach some performance, and is known to be very
memory-hungry at that. Furthermore, I believe it is known to have
performance issues on 32-bit architectures. The bdb backend Bitcoin uses
does many I/O operations, and writes them synchronously to disk, killing
whatever caching your filesystem provides. For those who run a large
database on ZFS, I believe it is advised to put ZFS's intent log on a
separate SSD-backed device, to get fast synchronous writes.
That said, some improvememts may be coming. Mike has been working on
changing the backend from bdb to leveldb, which may (or may not) result in
a very different performance profile on your machine. Also, I've been
working on switching the bitcoin block verifier to use a different style
database layout ("ultraprune"), which is smaller, faster, and will support
pruning. A recent test on my own machine synced the blockchain up to the
latest checkpoint in 7 minutes (6 minutes when stored in RAM instead of
disk).
--
Pieter
--e89a8fb1ef5665c26c04c5ccc58f
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<p>And now to the list...</p>
<p>On Jul 27, 2012 6:21 AM, "grarpamp" <<a href=3D"mailto:grar=
pamp@gmail.com">grarpamp@gmail.com</a>> wrote:<br>
><br>
> Update: this class of machine just became useless for bitcoin.<br>
> When blk0002.dat was created to store more blocks, all forward<br>
> progress processing blocks turned into losing ground by 20 or so<br>
> a day. Guessing both datfiles were being accessed at once resulting<br=
>
> in disk based overload. I've not seen any other mentions of crypto=
<br>
> in this thread so I'm not sure how well new hardware would perform=
.<br>
> Going shopping I guess.</p>
<p>I doubt the encryption is the problem. I have a much more recent machine=
(i7 with 8 GiB RAM), and my blockchain is on a (userspace!) encrypted file=
system. However, I do not notice any measurable slowdown from doing so.</p>
<p>You are however using a filesystem (ZFS) that is uses its own filesystem=
caching implementation to reach some performance, and is known to be very =
memory-hungry at that. Furthermore, I believe it is known to have performan=
ce issues on 32-bit architectures. The bdb backend Bitcoin uses does many I=
/O operations, and writes them synchronously to disk, killing whatever cach=
ing your filesystem provides. For those who run a large database on ZFS, I =
believe it is advised to put ZFS's intent log on a separate SSD-backed =
device, to get fast synchronous writes.</p>
<p>That said, some improvememts may be coming. Mike has been working on cha=
nging the backend from bdb to leveldb, which may (or may not) result in a v=
ery different performance profile on your machine. Also, I've been work=
ing on switching the bitcoin block verifier to use a different style databa=
se layout ("ultraprune"), which is smaller, faster, and will supp=
ort pruning. A recent test on my own machine synced the blockchain up to th=
e latest checkpoint in 7 minutes (6 minutes when stored in RAM instead of d=
isk).</p>
<p>-- <br>
Pieter<br>
</p>
--e89a8fb1ef5665c26c04c5ccc58f--
|
