1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
|
Return-Path: <eric@voskuil.org>
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
[172.17.192.35])
by mail.linuxfoundation.org (Postfix) with ESMTPS id 80339BAE
for <bitcoin-dev@lists.linuxfoundation.org>;
Fri, 7 Apr 2017 23:50:36 +0000 (UTC)
X-Greylist: whitelisted by SQLgrey-1.7.6
Received: from mail-pg0-f41.google.com (mail-pg0-f41.google.com [74.125.83.41])
by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 1AB06139
for <bitcoin-dev@lists.linuxfoundation.org>;
Fri, 7 Apr 2017 23:50:36 +0000 (UTC)
Received: by mail-pg0-f41.google.com with SMTP id x125so79427662pgb.0
for <bitcoin-dev@lists.linuxfoundation.org>;
Fri, 07 Apr 2017 16:50:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=voskuil-org.20150623.gappssmtp.com; s=20150623;
h=subject:to:references:from:message-id:date:user-agent:mime-version
:in-reply-to:content-transfer-encoding;
bh=CKuCErc5xcq5/wh2ePvs98RJWMzZS6dooXIEP6C5BqU=;
b=BqKyMt1UQZf4u3Wnv0EYRdDXxB3a9c+7qw1Wv60BtaIw2iXIjcQBoo5oX8ZJ5aPKIN
c4JZHW337mLHOpsJypyHQQlsf32RuKc1JQgSd3+aDFcowUSdeF8WyxAQi27CeWaJIy7o
QPI2X2z3GjTgYqJbgHAtyBBndz217MouCRgqfGKtOUlssUyoKG2Fn7vXtwkxtWHfj/iV
PYLZ23tL4/Y3+8t39YBm6HbJR4dZZwY2Og8p2vf1/ovnChZ9ILcTC9A5YKW8zlaDMn8k
MljQwdmF8wc0xOcjMHoAqpWmMXeol7RBfJYJXSABHGJwXH5RIe+wO117YW5U8uJkj1GR
lxlQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:subject:to:references:from:message-id:date
:user-agent:mime-version:in-reply-to:content-transfer-encoding;
bh=CKuCErc5xcq5/wh2ePvs98RJWMzZS6dooXIEP6C5BqU=;
b=NWB7VvR4/8RVnK45FBjnPQ9DhpoRawCwhHCmC7fDSsWCiagj+RNXuYEMbI4r4cFIS7
NsKkJFFaw7uLdwOZhlAc2mgLxNsCwhAdmRB8xiFPZMCgIEKm9dTgqsn5W6E3CdnFf683
7Tjv4fWQIKgcIlhTKyFjfnyuzp3IzNDPvAIHm7AqeHuOrM4JyuJZAJl5wTc91t+26G5V
z5vgXDry02gqaD3KMRKBTeVCTg1nt0EfubJjwFmWxpuDGOddpyGCS6rgIy9rIguzeLrZ
wm2AFUDX1LjxADNWhMUtX+QjVC0qzReIcmP6QXJjST9sxSOBVpPFDn8yX2i9cbG7rotf
VEtA==
X-Gm-Message-State: AFeK/H0YCjLUrcehwY6ff0tJH/RWcpHjvgl2U8pYk3fpE584LqTynecMAARPe0RG0JYsMg==
X-Received: by 10.84.224.12 with SMTP id r12mr13602568plj.69.1491609035670;
Fri, 07 Apr 2017 16:50:35 -0700 (PDT)
Received: from ?IPv6:2601:600:9000:d69e:7543:944:9329:6f9?
([2601:600:9000:d69e:7543:944:9329:6f9])
by smtp.gmail.com with ESMTPSA id
b8sm11545798pgn.51.2017.04.07.16.50.34
(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Fri, 07 Apr 2017 16:50:34 -0700 (PDT)
To: Tomas <tomas@tomasvdw.nl>,
Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
References: <1491516747.3791700.936828232.69F82904@webmail.messagingengine.com>
<CAAS2fgTJ8xOj8zCmBq1LN9OdMV-tDfSjVUPhLpO98cR1w-QAoA@mail.gmail.com>
<CA+KqGko0cDY29bhznMxJJ7yAUTuB6GaDDNGBRwzssJUxM_53xQ@mail.gmail.com>
<cb45ab6e-f3d0-85f2-4b41-8c7bc2bdf399@voskuil.org>
<1491601483.1253274.937963928.1B1D9B41@webmail.messagingengine.com>
From: Eric Voskuil <eric@voskuil.org>
X-Enigmail-Draft-Status: N1110
Message-ID: <03ab63f2-d4b0-33d4-db73-1cf5a94592ba@voskuil.org>
Date: Fri, 7 Apr 2017 16:51:08 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
Thunderbird/45.5.1
MIME-Version: 1.0
In-Reply-To: <1491601483.1253274.937963928.1B1D9B41@webmail.messagingengine.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,DKIM_SIGNED,
DKIM_VALID, RCVD_IN_DNSWL_NONE,
RCVD_IN_SORBS_SPAM autolearn=no version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
smtp1.linux-foundation.org
X-Mailman-Approved-At: Sat, 08 Apr 2017 00:39:08 +0000
Subject: Re: [bitcoin-dev] Using a storage engine without UTXO-index
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Fri, 07 Apr 2017 23:50:36 -0000
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On 04/07/2017 02:44 PM, Tomas via bitcoin-dev wrote:
> Hi Eric,
>
> On Fri, Apr 7, 2017, at 21:55, Eric Voskuil via bitcoin-dev wrote:
>> Optimization for lower memory platforms then becomes a process
>> of reducing the need for paging. This is the purpose of a cache.
>> The seam between disk and memory can be filled quite nicely by a
>> small amount of cache. On high RAM systems any cache is actually
>> a de-optimization but on low RAM systems it can prevent excessive
>> paging. This is directly analogous to a CPU cache.
>
>
> I am not entirely sure I agree with that, or understand it
> correctly.
>
> If -for example - the data of some application is a set of
> records which can be sorted from least frequently used to most
> frequently used then doing just that sort will beat any
> application-layer cache. Regardless of size of data and size of
> RAM, you simply allow the OS to use disk caching or memory map
> caching to work its magic .
It's a reasonable assumption, and given that the no-explicit-cache
implementation is a subset of the optionally-cached implementation,
was of course the initial implementation.
> In fact, I would argue that an application-layer cache *only*
> makes sense if the data model shows a *hard* distinction between
> often and not often used data. If usage-frequency is a continuous
> line, caching is best left to the OS by focussing on proper spatial
> and temporal locality of reference of your data, because the OS has
> much more information to make the right decision.
In practice this is not the case. The Bitcoin data model is neither
continuous nor strictly segregated by usage.
It is true that with sufficient RAM a cache is totally
counterproductive. It is also my experience that an independent UTXO
store is not a reasonable/necessary trade of disk space, memory
scalability, and/or code complexity in exchange for speed.
But on lower memory systems a explicit cache is beneficial. The
difference is clearly measurable in production code by simply changing
the cache limit and testing on various configurations.
e
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQEcBAEBCAAGBQJY6CXnAAoJEDzYwH8LXOFOf0YH/2qk3hYC6iEDW/DWM2ffkdb9
QM7A29Pvbfw9Wjr5Xx+ugIQvlAr4T+nByOCT6AnrqNU5K3UUmbC0KIB1rEL94hsK
QYVlLs0cOrjg8qKJpck+wcgiWw3VbEa/Y44hK7NLUxoy2HsLYaxPhqFH3GGgowqR
syga626jf2YUyudZxj1gFuqn7grkwghnzdrEUJMcqQo8IvCqjftGXlKxBGyB/AIs
Dx+5EWO9Q9IxrNpg/fsKKB6xkMxkmSx2hbD7dmEBvi/afbVF66rDTinjInG/LCju
pV7kT/GAWqGQGku6sQyAOexsxVhWA8EA/QEjvbyyGb+3YnR0s6nPk+CxO+RkOgo=
=e+Pr
-----END PGP SIGNATURE-----
|
