Received: from sog-mx-4.v43.ch3.sourceforge.com ([172.29.43.194] helo=mx.sourceforge.net) by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1RbuSp-0004KI-T7 for bitcoin-development@lists.sourceforge.net; Sat, 17 Dec 2011 13:38:11 +0000 Received-SPF: pass (sog-mx-4.v43.ch3.sourceforge.com: domain of gmail.com designates 209.85.212.175 as permitted sender) client-ip=209.85.212.175; envelope-from=decker.christian@gmail.com; helo=mail-wi0-f175.google.com; Received: from mail-wi0-f175.google.com ([209.85.212.175]) by sog-mx-4.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1RbuSo-0007Im-AU for bitcoin-development@lists.sourceforge.net; Sat, 17 Dec 2011 13:38:11 +0000 Received: by wibhq7 with SMTP id hq7so625539wib.34 for ; Sat, 17 Dec 2011 05:38:04 -0800 (PST) Received: by 10.180.106.104 with SMTP id gt8mr18182109wib.6.1324129082244; Sat, 17 Dec 2011 05:38:02 -0800 (PST) MIME-Version: 1.0 Received: by 10.227.152.10 with HTTP; Sat, 17 Dec 2011 05:37:21 -0800 (PST) In-Reply-To: <82659F61-0449-47BB-88DC-497E0D02F8A1@ceptacle.com> References: <82659F61-0449-47BB-88DC-497E0D02F8A1@ceptacle.com> From: Christian Decker Date: Sat, 17 Dec 2011 14:37:21 +0100 Message-ID: To: =?ISO-8859-1?Q?Michael_Gr=F8nager?= Content-Type: multipart/alternative; boundary=f46d04428ee2957e6404b449d39a X-Spam-Score: -0.6 (/) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (decker.christian[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record 1.0 HTML_MESSAGE BODY: HTML included in message -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1RbuSo-0007Im-AU Cc: bitcoin-development@lists.sourceforge.net Subject: Re: [Bitcoin-development] Protocol extensions X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Dec 2011 13:38:12 -0000 --f46d04428ee2957e6404b449d39a Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable A while back I had proposed a similar idea to the DHT, although my main goal was to reduce the need for broadcasts. My idea was to structure the network in a hypercube and use prefixes to address different parts of the network, and use those prefixes also to find the location where an item (transaction, block, ...) should be stored. Each vertex in the hypercube is a small, highly connected, cluster of nodes. The storage would be distributed, messages are routed on behalf of others, which makes finding the origin of the query hard to find (think Tor), each node would have to store only O(log(p)) items, with p being the prefix length, maximum number of hops is equal to the dimension of the hypercube O(log(n)). Newly created transaction will be sent directly to the location they'll be stored and miners retrieve new transactions at regular intervals. It might increase delays to the confirmations, but it reduces the number of broadcasts and storage requirements on nodes greatly. Regards, Chris On Sat, Dec 17, 2011 at 2:13 PM, Michael Gr=F8nager = wrote: > Hey Eric, > > Two comments. > > 1. > The ability to query for transactions belonging to pubkeys or bitcoin > addresses is supported today by several implementations: > * blockexplorer.com > * bitcoin-js > * my own libBTC (will more on this soon) > > To query for transactions you need to use json-rpc and not the bitcoin > protocol, however. But still the purpose is the same: to be able to build > thin clients that can rely on a server for storing the blockchain and > keeping connected on the p2p network. > > The reason for not having these queries part of the standard protocol (I > think) are as they breaks anonymity, and that you would actually encourag= e > people to participate in the p2p. > > 2. The second part you mention, to some how move the storage of the > blockchain into a DHT based storage would be quite nice. The benefit of > this is that it could be a way to integrate the smaller clients into the > network without breaking the anonymity. But it should be thought out quit= e > carefully. Further, if each client only store a fraction of the blockchai= n > we should work out what fraction that need to be in order to ensure a > similar service level. I would be happy to work with you on this. > > Cheers, > > Michael > > On 17/12/2011, at 08:41, Eric Lombrozo wrote: > > > Hey, guys. > > > > I haven't posted here before so I'll introduce myself. My name's Eric, > > I've been developing cryptocurrency-related > > software for several months now, I've implemented some libraries for > > dealing with core bitcoin datastructures, made > > some custom builds of bitcoind and interfaced it with a few apps I've > written. > > > > In doing so, I've come to appreciate just how little of the potential > > for the bitcoin protocol is being exploited right now... > > not only in terms of the script features but in terms of the potential > > commands and node types that could exist. > > > > For instance, the protocol spec at > > https://en.bitcoin.it/wiki/Protocol_specification only has 16 commands > > listed and > > only one service type...despite having a full 12 bytes for a command > > code and a full eight bytes for a services > > type. > > > > The fact that only one node service type is specified is probably due > > to the fact that the satoshi client was written > > to be a standalone monolithic app that took care of all the essential > > needs for a network of peers. > > i.e. block chain storage/management, transaction signing/verification, > > key generation/wallet management, block mining, etc... > > However, I think there's an urgent need for breaking up all these > > different tasks into separate components that can run as independent > > services on different types of devices. > > > > One of the big issues I'm dealing with now pertains to block chain > > storage. As of right now, it is implemented as sequential > > disk files using Berkeley DB in the satoshi client. Then you have > > other projects that have been using SQL tables, etc... > > But I believe the direction this really needs to move towards is some > > sort of distributed hash table...and the database queries > > should be performed using the bitcoin protocol itself. Perhaps adding > > a few more commands. As things stand right now, > > the only way to query for transactions or blocks is by their hash. And > > once a transaction gets incorporated into a block and > > removed from the transaction pool, one can no longer query it by the > > transaction hash without stepping outside the bitcoin protocol. > > We need access to the disk file that stores the blocks whether it be > > via Berkeley DB or SQL or whatever. > > > > I propose an extension to the bitcoin protocol to provide methods for > > performing more sophisticated queries, such as "Give me > > an inventory of transactions involving this particular public key" or > > "Give me an inventory all transactions in the last n blocks with > > unredeemed outputs." This could be done by adding a few more commands. > > > > Furthermore, I propose a new network services type for nodes that > > serve as block chain/transaction pool storage. > > > > Of couse, any peer that wishes to verify the integrity of the block > > chain would still have to download at the very least > > all the block headers...and to be completely sure, also all the blocks > > themselves...and verify everything. But it would be > > very nice to be able to run thin services that can rely on other > > network peers to do this work. It is still possible to attain > > a high level of confidence in the integrity by querying multiple peers > > for similar objects and comparing. It is also possible > > to run your own dedicated block chain storage servers which you trust. > > > > There are other ideas I have for other types of services, too. > > > > Anyhow, I'm just throwing this out there...if anyone's interested I'd > > love to develop these ideas further and help put together some > > specs. > > > > -Eric Lombrozo > > > > > -------------------------------------------------------------------------= ----- > > Learn Windows Azure Live! Tuesday, Dec 13, 2011 > > Microsoft is holding a special Learn Windows Azure training event for > > developers. It will provide a great way to learn Windows Azure and what > it > > provides. You can attend the event by watching it streamed LIVE online. > > Learn more at http://p.sf.net/sfu/ms-windowsazure > > _______________________________________________ > > Bitcoin-development mailing list > > Bitcoin-development@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > > > > -------------------------------------------------------------------------= ----- > Learn Windows Azure Live! Tuesday, Dec 13, 2011 > Microsoft is holding a special Learn Windows Azure training event for > developers. It will provide a great way to learn Windows Azure and what i= t > provides. You can attend the event by watching it streamed LIVE online. > Learn more at http://p.sf.net/sfu/ms-windowsazure > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > --f46d04428ee2957e6404b449d39a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable A while back I had proposed a similar idea to the DHT, although my main goa= l was to reduce the need for broadcasts.

My idea was to structure th= e network in a hypercube and use prefixes to address different parts of the= network, and use those prefixes also to find the location where an item (t= ransaction, block, ...) should be stored. Each vertex in the hypercube is a= small, highly connected, cluster of nodes. The storage would be distribute= d, messages are routed on behalf of others, which makes finding the origin = of the query hard to find (think Tor), each node would have to store only O= (log(p)) items, with p being the prefix length, maximum number of hops is e= qual to the dimension of the hypercube O(log(n)).

Newly created transaction will be sent directly to the location they= 9;ll be stored and miners retrieve new transactions at regular intervals. I= t might increase delays to the confirmations, but it reduces the number of = broadcasts and storage requirements on nodes greatly.

Regards,
Chris


O= n Sat, Dec 17, 2011 at 2:13 PM, Michael Gr=F8nager <gronager@ceptacle.com>= wrote:
Hey Eric,

Two comments.

1.
The ability to query for transactions belonging to pubkeys or bitcoin addre= sses is supported today by several implementations:
* blockexplorer.com<= /a>
* bitcoin-js
* my own libBTC (will more on this soon)

To query for transactions you need to use json-rpc and not the bitcoin prot= ocol, however. But still the purpose is the same: to be able to build thin = clients that can rely on a server for storing the blockchain and keeping co= nnected on the p2p network.

The reason for not having these queries part of the standard protocol (I th= ink) are as they breaks anonymity, and that you would actually encourage pe= ople to participate in the p2p.

2. The second part you mention, to some how move the storage of the blockch= ain into a DHT based storage would be quite nice. The benefit of this is th= at it could be a way to integrate the smaller clients into the network with= out breaking the anonymity. But it should be thought out quite carefully. F= urther, if each client only store a fraction of the blockchain we should wo= rk out what fraction that need to be in order to ensure a similar service l= evel. I would be happy to work with you on this.

Cheers,

Michael

On 17/12/2011, at 08:41, Eric Lombrozo wrote:

> Hey, guys.
>
> I haven't posted here before so I'll introduce myself. My name= 's Eric,
> I've been developing cryptocurrency-related
> software for several months now, I've implemented some libraries f= or
> dealing with core bitcoin datastructures, made
> some custom builds of bitcoind and interfaced it with a few apps I'= ;ve written.
>
> In doing so, I've come to appreciate just how little of the potent= ial
> for the bitcoin protocol is being exploited right now...
> not only in terms of the script features but in terms of the potential=
> commands and node types that could exist.
>
> For instance, the protocol spec at
> https://en.bitcoin.it/wiki/Protocol_specification only has = 16 commands
> listed and
> only one service type...despite having a full 12 bytes for a command > code and a full eight bytes for a services
> type.
>
> The fact that only one node service type is specified is probably due<= br> > to the fact that the satoshi client was written
> to be a standalone monolithic app that took care of all the essential<= br> > needs for a network of peers.
> i.e. block chain storage/management, transaction signing/verification,=
> key generation/wallet management, block mining, etc...
> However, I think there's an urgent need for breaking up all these<= br> > different tasks into separate components that can run as independent > services on different types of devices.
>
> One of the big issues I'm dealing with now pertains to block chain=
> storage. As of right now, it is implemented as sequential
> disk files using Berkeley DB in the satoshi client. Then you have
> other projects that have been using SQL tables, etc...
> But I believe the direction this really needs to move towards is some<= br> > sort of distributed hash table...and the database queries
> should be performed using the bitcoin protocol itself. Perhaps adding<= br> > a few more commands. As things stand right now,
> the only way to query for transactions or blocks is by their hash. And=
> once a transaction gets incorporated into a block and
> removed from the transaction pool, one can no longer query it by the > transaction hash without stepping outside the bitcoin protocol.
> We need access to the disk file that stores the blocks whether it be > via Berkeley DB or SQL or whatever.
>
> I propose an extension to the bitcoin protocol to provide methods for<= br> > performing more sophisticated queries, such as "Give me
> an inventory of transactions involving this particular public key"= ; or
> "Give me an inventory all transactions in the last n blocks with<= br> > unredeemed outputs." This could be done by adding a few more comm= ands.
>
> Furthermore, I propose a new network services type for nodes that
> serve as block chain/transaction pool storage.
>
> Of couse, any peer that wishes to verify the integrity of the block > chain would still have to download at the very least
> all the block headers...and to be completely sure, also all the blocks=
> themselves...and verify everything. But it would be
> very nice to be able to run thin services that can rely on other
> network peers to do this work. It is still possible to attain
> a high level of confidence in the integrity by querying multiple peers=
> for similar objects and comparing. It is also possible
> to run your own dedicated block chain storage servers which you trust.=
>
> There are other ideas I have for other types of services, too.
>
> Anyhow, I'm just throwing this out there...if anyone's interes= ted I'd
> love to develop these ideas further and help put together some
> specs.
>
> -Eric Lombrozo
>
> ----------------------------------------------------------------------= --------
> Learn Windows Azure Live! =A0Tuesday, Dec 13, 2011
> Microsoft is holding a special Learn Windows Azure training event for<= br> > developers. It will provide a great way to learn Windows Azure and wha= t it
> provides. You can attend the event by watching it streamed LIVE online= .
> Learn more at http://p.sf.net/sfu/ms-windowsazure
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-d= evelopment@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitco= in-development



---------------------------------------------------------------------------= ---
Learn Windows Azure Live! =A0Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it<= br> provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure
_______________________________________________
Bitcoin-development mailing list
Bitcoin-develo= pment@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-de= velopment

--f46d04428ee2957e6404b449d39a--