Received: from sog-mx-2.v43.ch3.sourceforge.com ([172.29.43.192] helo=mx.sourceforge.net) by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1UiOb3-0004pt-VP for bitcoin-development@lists.sourceforge.net; Fri, 31 May 2013 12:38:18 +0000 Received-SPF: pass (sog-mx-2.v43.ch3.sourceforge.com: domain of gmail.com designates 74.125.82.51 as permitted sender) client-ip=74.125.82.51; envelope-from=runesvend@gmail.com; helo=mail-wg0-f51.google.com; Received: from mail-wg0-f51.google.com ([74.125.82.51]) by sog-mx-2.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1UiOb2-0003DS-P2 for bitcoin-development@lists.sourceforge.net; Fri, 31 May 2013 12:38:17 +0000 Received: by mail-wg0-f51.google.com with SMTP id b13so1185079wgh.30 for ; Fri, 31 May 2013 05:38:10 -0700 (PDT) X-Received: by 10.194.158.102 with SMTP id wt6mr9318511wjb.5.1370003890665; Fri, 31 May 2013 05:38:10 -0700 (PDT) MIME-Version: 1.0 Received: by 10.194.161.227 with HTTP; Fri, 31 May 2013 05:37:50 -0700 (PDT) In-Reply-To: References: From: =?ISO-8859-1?Q?Rune_Kj=E6r_Svendsen?= Date: Fri, 31 May 2013 14:37:50 +0200 Message-ID: To: Bitcoin Dev Content-Type: multipart/alternative; boundary=089e013c63e83e7f9b04de02e359 X-Spam-Score: -0.6 (/) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (runesvend[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record 1.0 HTML_MESSAGE BODY: HTML included in message -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1UiOb2-0003DS-P2 Subject: Re: [Bitcoin-development] Implementing batch processing for -blocknotify X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 May 2013 12:38:18 -0000 --089e013c63e83e7f9b04de02e359 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Fri, May 31, 2013 at 2:10 PM, Michael Hendricks wrote= : > On Fri, May 31, 2013 at 5:56 AM, Rune Kj=E6r Svendsen wrote: > >> I have an application that wants to keep up with new blocks as they come >> in. For that I can use the -blocknotify option with bitcoind, which will >> execute my application for each new block. >> >> The problem is that my app isn't necessarily quick enough to finish its >> work before a new block comes in and the app is executed again. >> > > In a similar circumstance, I changed my -blocknotify script to quickly > append necessary information to a queue and immediately exit. A separate > script runs at all times monitoring this queue for work and performs the > labor intensive calculations. > I've thought about this as well. It just seems somewhat clunky to me. I'd really prefer having bitcoind put out messages in batches, if it's doable, that is. I'd run into a lot of concurrency issues, as far as I can see, where I can't be sure that the queue isn't written to while, for example, it is opened by the program that needs to process the queue items. What if a disk operation takes a long time to finish, and a two queue operations want to add to the queue simultaneously? This really brings forward all the horrors of concurrent programming. On Fri, May 31, 2013 at 2:17 PM, Jeremy Spilman wrote: > Would it work to just block the bitcoind thread until your process exits? > I don't think that's optimal, no. That would slow down synchronization drastically. It would be really nimble for bitcoind to be able to synchronize at full speed, and only send out events when necessary, batching together previously queued items. --089e013c63e83e7f9b04de02e359 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
On Fri, May 31, 2013 at 2:10 PM, Michael Hendricks <micha= el@ndrix.org> wrote:
On Fri, May 31, 2013 at= 5:56 AM, Rune Kj=E6r Svendsen <runesvend@gmail.com> wrote= :
I have an application that wants to = keep up with new blocks as they come in. For that I can use the -blocknotif= y option with bitcoind, which will execute my application for each new bloc= k.

The problem is that my app isn't necessarily quick enough to= finish its work before a new block comes in and the app is executed again.=

In a similar circumstanc= e, I changed my -blocknotify script to quickly append necessary information= to a queue and immediately exit. =A0A separate script runs at all times mo= nitoring this queue for work and performs the labor intensive calculations.=

I've thought about t= his as well. It just seems somewhat clunky to me. I'd really prefer hav= ing bitcoind put out messages in batches, if it's doable, that is.

I'd run into a lot of concurrency issues, as far as= I can see, where I can't be sure that the queue isn't written to w= hile, for example, it is opened by the program that needs to process the qu= eue items.

What if a disk operation takes a long time to finish, a= nd a two queue operations want to add to the queue simultaneously? This rea= lly brings forward all the horrors of concurrent programming.


On Fri, May 31, 2013 at 2:17 = PM, Jeremy Spilman=A0<jeremy@taplink.co>=A0wrote:
Would it work to just block the bitcoind thread until your process exits?

I don't think that's optimal, no. That would slow down synchr= onization drastically.

It would be= really nimble for bitcoind to be able to synchronize at full speed, and on= ly send out events when necessary, batching together previously queued item= s.
--089e013c63e83e7f9b04de02e359--