1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
|
Received: from sog-mx-2.v43.ch3.sourceforge.com ([172.29.43.192]
helo=mx.sourceforge.net)
by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76)
(envelope-from <runesvend@gmail.com>) id 1UiOb3-0004pt-VP
for bitcoin-development@lists.sourceforge.net;
Fri, 31 May 2013 12:38:18 +0000
Received-SPF: pass (sog-mx-2.v43.ch3.sourceforge.com: domain of gmail.com
designates 74.125.82.51 as permitted sender)
client-ip=74.125.82.51; envelope-from=runesvend@gmail.com;
helo=mail-wg0-f51.google.com;
Received: from mail-wg0-f51.google.com ([74.125.82.51])
by sog-mx-2.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128)
(Exim 4.76) id 1UiOb2-0003DS-P2
for bitcoin-development@lists.sourceforge.net;
Fri, 31 May 2013 12:38:17 +0000
Received: by mail-wg0-f51.google.com with SMTP id b13so1185079wgh.30
for <bitcoin-development@lists.sourceforge.net>;
Fri, 31 May 2013 05:38:10 -0700 (PDT)
X-Received: by 10.194.158.102 with SMTP id wt6mr9318511wjb.5.1370003890665;
Fri, 31 May 2013 05:38:10 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.194.161.227 with HTTP; Fri, 31 May 2013 05:37:50 -0700 (PDT)
In-Reply-To: <CAFHuXuZaSNkUUX8o_k=wSf5MeQ3tAj6yD8DJuLwbyA0C3xRQuQ@mail.gmail.com>
References: <CAH2=CKzW41TYbX6c1F8oknA_LttOaA8vmDPmojuowXgEADY61g@mail.gmail.com>
<CAFHuXuZaSNkUUX8o_k=wSf5MeQ3tAj6yD8DJuLwbyA0C3xRQuQ@mail.gmail.com>
From: =?ISO-8859-1?Q?Rune_Kj=E6r_Svendsen?= <runesvend@gmail.com>
Date: Fri, 31 May 2013 14:37:50 +0200
Message-ID: <CAH2=CKxBX6BcLS+CSKxZstut+uhHPQUY7HnLHSoMjZACgd4v1g@mail.gmail.com>
To: Bitcoin Dev <bitcoin-development@lists.sourceforge.net>
Content-Type: multipart/alternative; boundary=089e013c63e83e7f9b04de02e359
X-Spam-Score: -0.6 (/)
X-Spam-Report: Spam Filtering performed by mx.sourceforge.net.
See http://spamassassin.org/tag/ for more details.
-1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for
sender-domain
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(runesvend[at]gmail.com)
-0.0 SPF_PASS SPF: sender matches SPF record
1.0 HTML_MESSAGE BODY: HTML included in message
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
author's domain
0.1 DKIM_SIGNED Message has a DKIM or DK signature,
not necessarily valid
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
X-Headers-End: 1UiOb2-0003DS-P2
Subject: Re: [Bitcoin-development] Implementing batch processing for
-blocknotify
X-BeenThere: bitcoin-development@lists.sourceforge.net
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <bitcoin-development.lists.sourceforge.net>
List-Unsubscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=unsubscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-development>
List-Post: <mailto:bitcoin-development@lists.sourceforge.net>
List-Help: <mailto:bitcoin-development-request@lists.sourceforge.net?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/bitcoin-development>,
<mailto:bitcoin-development-request@lists.sourceforge.net?subject=subscribe>
X-List-Received-Date: Fri, 31 May 2013 12:38:18 -0000
--089e013c63e83e7f9b04de02e359
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
On Fri, May 31, 2013 at 2:10 PM, Michael Hendricks <michael@ndrix.org>wrote=
:
> On Fri, May 31, 2013 at 5:56 AM, Rune Kj=E6r Svendsen <runesvend@gmail.co=
m>wrote:
>
>> I have an application that wants to keep up with new blocks as they come
>> in. For that I can use the -blocknotify option with bitcoind, which will
>> execute my application for each new block.
>>
>> The problem is that my app isn't necessarily quick enough to finish its
>> work before a new block comes in and the app is executed again.
>>
>
> In a similar circumstance, I changed my -blocknotify script to quickly
> append necessary information to a queue and immediately exit. A separate
> script runs at all times monitoring this queue for work and performs the
> labor intensive calculations.
>
I've thought about this as well. It just seems somewhat clunky to me. I'd
really prefer having bitcoind put out messages in batches, if it's doable,
that is.
I'd run into a lot of concurrency issues, as far as I can see, where I
can't be sure that the queue isn't written to while, for example, it is
opened by the program that needs to process the queue items.
What if a disk operation takes a long time to finish, and a two queue
operations want to add to the queue simultaneously? This really brings
forward all the horrors of concurrent programming.
On Fri, May 31, 2013 at 2:17 PM, Jeremy Spilman <jeremy@taplink.co> wrote:
> Would it work to just block the bitcoind thread until your process exits?
>
I don't think that's optimal, no. That would slow down synchronization
drastically.
It would be really nimble for bitcoind to be able to synchronize at full
speed, and only send out events when necessary, batching together
previously queued items.
--089e013c63e83e7f9b04de02e359
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">On Fri, May 31, 2013 at 2:10 PM, Michael Hendricks <span d=
ir=3D"ltr"><<a href=3D"mailto:michael@ndrix.org" target=3D"_blank">micha=
el@ndrix.org</a>></span> wrote:<br><div class=3D"gmail_extra"><div class=
=3D"gmail_quote">
<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-=
left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p=
adding-left:1ex"><div dir=3D"ltr"><div class=3D"im">On Fri, May 31, 2013 at=
5:56 AM, Rune Kj=E6r Svendsen <span dir=3D"ltr"><<a href=3D"mailto:rune=
svend@gmail.com" target=3D"_blank">runesvend@gmail.com</a>></span> wrote=
:<br>
</div><div class=3D"gmail_extra"><div class=3D"gmail_quote"><div class=3D"i=
m">
<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-=
left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p=
adding-left:1ex"><div dir=3D"ltr"><div>I have an application that wants to =
keep up with new blocks as they come in. For that I can use the -blocknotif=
y option with bitcoind, which will execute my application for each new bloc=
k.</div>
<div><br>
</div><div>The problem is that my app isn't necessarily quick enough to=
finish its work before a new block comes in and the app is executed again.=
</div></div></blockquote><div><br></div></div><div>In a similar circumstanc=
e, I changed my -blocknotify script to quickly append necessary information=
to a queue and immediately exit. =A0A separate script runs at all times mo=
nitoring this queue for work and performs the labor intensive calculations.=
</div>
</div></div></div></blockquote><div><br></div><div>I've thought about t=
his as well. It just seems somewhat clunky to me. I'd really prefer hav=
ing bitcoind put out messages in batches, if it's doable, that is.</div=
>
<div><br></div><div>I'd run into a lot of concurrency issues, as far as=
I can see, where I can't be sure that the queue isn't written to w=
hile, for example, it is opened by the program that needs to process the qu=
eue items.</div>
<div><br></div><div>What if a disk operation takes a long time to finish, a=
nd a two queue operations want to add to the queue simultaneously? This rea=
lly brings forward all the horrors of concurrent programming.</div><div>
<br></div><div><br><div class=3D"gmail_quote">On Fri, May 31, 2013 at 2:17 =
PM, Jeremy Spilman=A0<span dir=3D"ltr"><<a href=3D"mailto:jeremy@taplink=
.co" target=3D"_blank">jeremy@taplink.co</a>></span>=A0wrote:<br><blockq=
uote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-wi=
dth:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-=
left:1ex">
Would it work to just block the bitcoind thread until your process exits?<b=
r><div></div></blockquote></div></div></div><br></div><div class=3D"gmail_e=
xtra">I don't think that's optimal, no. That would slow down synchr=
onization drastically.</div>
<div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">It would be=
really nimble for bitcoind to be able to synchronize at full speed, and on=
ly send out events when necessary, batching together previously queued item=
s.</div>
</div>
--089e013c63e83e7f9b04de02e359--
|
