1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
|
Received: from sog-mx-1.v43.ch3.sourceforge.com ([172.29.43.191]
helo=mx.sourceforge.net)
by sfs-ml-3.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
(envelope-from <grarpamp@gmail.com>) id 1Rv1qr-0006eN-1J
for bitcoin-development@lists.sourceforge.net;
Wed, 08 Feb 2012 07:22:01 +0000
Received-SPF: pass (sog-mx-1.v43.ch3.sourceforge.com: domain of gmail.com
designates 209.85.212.175 as permitted sender)
client-ip=209.85.212.175; envelope-from=grarpamp@gmail.com;
helo=mail-wi0-f175.google.com;
Received: from mail-wi0-f175.google.com ([209.85.212.175])
by sog-mx-1.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
(Exim 4.76) id 1Rv1qn-0004PB-3q
for bitcoin-development@lists.sourceforge.net;
Wed, 08 Feb 2012 07:22:00 +0000
Received: by wibhq7 with SMTP id hq7so190868wib.34
for <bitcoin-development@lists.sourceforge.net>;
Tue, 07 Feb 2012 23:21:51 -0800 (PST)
MIME-Version: 1.0
Received: by 10.180.78.98 with SMTP id a2mr38924382wix.17.1328685711074; Tue,
07 Feb 2012 23:21:51 -0800 (PST)
Received: by 10.180.103.227 with HTTP; Tue, 7 Feb 2012 23:21:51 -0800 (PST)
In-Reply-To: <CAD2Ti2_vGc+SJX_+uTz4ZVk1r5DhCOm6n3yKW16o9QaPKTQkHQ@mail.gmail.com>
References: <CAD2Ti2_vGc+SJX_+uTz4ZVk1r5DhCOm6n3yKW16o9QaPKTQkHQ@mail.gmail.com>
Date: Wed, 8 Feb 2012 02:21:51 -0500
Message-ID: <CAD2Ti29UvdVevKvccbA6PbUMcmrgM4LDnL+cxVSaU4Zt3P-mJw@mail.gmail.com>
From: grarpamp <grarpamp@gmail.com>
To: bitcoin-development@lists.sourceforge.net
Content-Type: text/plain; charset=UTF-8
X-Spam-Score: -1.2 (-)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
See http://spamassassin.org/tag/ for more details.
-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
sender-domain
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(grarpamp[at]gmail.com)
-0.0 SPF_PASS SPF: sender matches SPF record
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
author's domain
0.1 DKIM_SIGNED Message has a DKIM or DK signature,
not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
0.4 AWL AWL: From: address is in the auto white-list
X-Headers-End: 1Rv1qn-0004PB-3q
Subject: Re: [Bitcoin-development] Scaling at the end user level
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Wed, 08 Feb 2012 07:22:01 -0000
> I never did track down this exact issue but it's an artificial
> slowdown.. meaning compression and whatever else wouldn't help much.
I meant for anyone who wanted to distribute the dataset as a project.
> It has something to do with the database file locking and flushing..
> on some systems I've seen the block chain get fully done in 10-20
> mins and on others it slows down to the point where it will never
> catch up.. but not in a way that's related to the age of the computer
> or anything. You might want to experiment if you want to track this
> down.. try building your own libs
Rather than use dated/modified packages, I compiled current versions
of all component sources manually.
> and compare different operating
> systems, on the same hardware to get a more 'true' comparison maybe.
True. Used them all before, happy with BSD for now. Just knowing
what the general setup is on those zippy systems should suffice.
ie: blindly fishing for such a zippy system to compare through various
install tests doesn't sound too appealing. It's different than benchmarking.
Datapoint: The system below is not zippy.
> I think everyone is vaguely aware of the problem but it has not
> been tracked down and eliminated. I don't think the problem is
> within bitcoin itself but in how truthfully the database file is
> actually written to disk.
Am I correct in guessing that, given a certain height, the data
in blkindex and blk0001 should be the same across instances?
# file blk*
blk0001.dat: data
blkindex.dat: Berkeley DB (Btree, version 9, native byte-order)
Pursuant to comparison, what is the format of blk0001.dat?
> If it really gets flushed to disk every
> block like bitcoin wants it to be, then there is no way that you
> could get more than 50-60 blocks per second through it (due to
> rotational latency), but on some operating systems and versions/options
> it seems to end up caching the writes and flies through it at
> thousands of blocks per second. The problem is similar to what's
> mentioned here: http://www.sqlite.org/faq.html#q19
I'm not running Linux with asynchronous data and metadata
turned on by default if that's what you mean :) ZFS, disk crypto,
standard drive write cache. Looking at it, I'm largely buried in
that crypto at 8MB/sec or so.
> Perhaps it's as simple as some default in the db lib.. and it seems
> to default to different things on different version/operating
> systems/filesystems.
Hmm, I compiled everything with the defaults. Will go back and
look at bdb options. I don't think there was anything interesting
there. I'd bet a lot is tied to the fs and cpu.
Single core p4@1.8 512k/2g isn't much up against ZFS+disk crypto.
It seems to take its time and roll up all but the last database file (of
a hundred or more) on receiving sigterm. Is it supposed to roll
and delete the last log too? Can I safely delete everything but
the blk* files? (wallet excepted of course :)
Currently, in KiB...
running:
853716 database
747881 blk0001.dat
290601 blkindex.dat
4361 addr.dat
137 __db.005
137 __db.004
137 __db.003
137 __db.002
41 __db.006
25 __db.001
sigterm:
750569 blk0001.dat
291497 blkindex.dat
8465 database/log.0000000nnn
4361 addr.dat
database/log.0000000133: Berkeley DB (Log, version 16, native byte-order)
|
