From dcarboni at gmail.com Thu Feb 2 17:57:09 2006 From: dcarboni at gmail.com (Davide "dada" Carboni) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] overlay netowork Message-ID: <71b79fa90602020957n28557e11j7f0b4cbbd1a22b35@mail.gmail.com> Hi, I'm searching a definition for 'overlay network' and how this concept is related to P2P computing. Thanx. -- La gente vuole le manopole (Anonimo inventore) -- http://people.crs4.it/dcarboni From trep at cs.ucr.edu Fri Feb 3 00:08:08 2006 From: trep at cs.ucr.edu (Thomas Repantis) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] overlay netowork In-Reply-To: <71b79fa90602020957n28557e11j7f0b4cbbd1a22b35@mail.gmail.com> References: <71b79fa90602020957n28557e11j7f0b4cbbd1a22b35@mail.gmail.com> Message-ID: <20060203000808.GA2888@angeldust.chaos> http://en.wikipedia.org/wiki/Overlay_network Thomas On Thu, Feb 02, 2006 at 06:57:09PM +0100, Davide dada Carboni wrote: > Hi, > I'm searching a definition for 'overlay network' and how this concept > is related to P2P computing. > Thanx. -- http://www.cs.ucr.edu/~trep From dcarboni at gmail.com Fri Feb 3 11:59:15 2006 From: dcarboni at gmail.com (Davide "dada" Carboni) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] overlay netowork In-Reply-To: <20060203000808.GA2888@angeldust.chaos> References: <71b79fa90602020957n28557e11j7f0b4cbbd1a22b35@mail.gmail.com> <20060203000808.GA2888@angeldust.chaos> Message-ID: <71b79fa90602030359q22d5d55bp10d26052c6291908@mail.gmail.com> On 2/3/06, Thomas Repantis wrote: > http://en.wikipedia.org/wiki/Overlay_network > > Thomas Oops. So easy?! Thanx From matthew at allpeers.com Wed Feb 8 20:03:50 2006 From: matthew at allpeers.com (Matthew Gertner) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Keep Alive and network performance In-Reply-To: <71b79fa90602030359q22d5d55bp10d26052c6291908@mail.gmail.com> References: <71b79fa90602020957n28557e11j7f0b4cbbd1a22b35@mail.gmail.com> <20060203000808.GA2888@angeldust.chaos> <71b79fa90602030359q22d5d55bp10d26052c6291908@mail.gmail.com> Message-ID: <43EA4EA6.8040803@allpeers.com> Hi, We have a straightforward P2P topology with each peer in the network attached to hub. We want to detect efficiently that a peer has disconnected from the hub. The obvious way to do this would be to use our current TCP connection with a relatively short Keep Alive value (a few seconds so we could see the peer's presence updated quickly if they don't disconnect cleanly). Does anyone have a feeling for the performance implications of this approach if many peers are attached to a single hub? Is there another approach that is recommend (such as using a custom UDP-based keep alive protocol)? Regards, Matt From alenlpeacock at gmail.com Thu Feb 9 04:46:33 2006 From: alenlpeacock at gmail.com (Alen Peacock) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Petnames Go Mainstream Message-ID: First Schneier linked to Mark Stiegler's paper in his blog today ( http://www.schneier.com/blog/archives/2006/02/petnames.html), then boingboing picked up the story ( http://www.boingboing.net/2006/02/08/fascinating_solution.html). I imagine you are getting a nice little surge in traffic at your site today, Zooko? Nice to see some of this stuff being explained to the masses (even if those masses are mostly made up of technical folk anyway) -- it makes proselytizing this type of technology more feasible when a general audience has some grasp of the concepts. Alen -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20060208/d9f82be9/attachment.htm From lgonze at panix.com Thu Feb 9 05:14:52 2006 From: lgonze at panix.com (Lucas Gonze) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Petnames Go Mainstream In-Reply-To: References: Message-ID: Popularization is always nice, I guess, but I can't believe that neither Schneier nor Cory knew about this stuff, which is as basic as air in the p2p subculture. On Wed, 8 Feb 2006, Alen Peacock wrote: > First Schneier linked to Mark Stiegler's paper in his blog today ( > http://www.schneier.com/blog/archives/2006/02/petnames.html), then > boingboing picked up the story ( > http://www.boingboing.net/2006/02/08/fascinating_solution.html). > > I imagine you are getting a nice little surge in traffic at your site today, > Zooko? > > Nice to see some of this stuff being explained to the masses (even if those > masses are mostly made up of technical folk anyway) -- it makes > proselytizing this type of technology more feasible when a general audience > has some grasp of the concepts. > > Alen > From turbogeek at cluck.com Thu Feb 9 07:34:53 2006 From: turbogeek at cluck.com (Daniel B.) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Keep Alive and network performance In-Reply-To: <43EA4EA6.8040803@allpeers.com> References: <71b79fa90602020957n28557e11j7f0b4cbbd1a22b35@mail.gmail.com> <20060203000808.GA2888@angeldust.chaos> <71b79fa90602030359q22d5d55bp10d26052c6291908@mail.gmail.com> <43EA4EA6.8040803@allpeers.com> Message-ID: <5037146A-FF9F-4786-BD8A-200BABF490DD@cluck.com> Keep alive is both a missing component in practical P2P and a reason why P2P is also hard to make practical. The trap is that the keep-alive is between two machines. Imagine hundreds of thousands of machines!!!! Even with small signatures matching each machine, we are talking about a flooded network. A secondary trap is managing reliability. Reliability in P2P is not moving bytes back and forth, but also taking into account the machine may be a laptop or other personal computer that has a pathetic uptime. The better choice is online/offline signaling and optimistic communications with good error control. On Feb 8, 2006, at 12:03 PM, Matthew Gertner wrote: > Hi, > > We have a straightforward P2P topology with each peer in the > network attached to hub. We want to detect efficiently that a peer > has disconnected from the hub. The obvious way to do this would be > to use our current TCP connection with a relatively short Keep > Alive value (a few seconds so we could see the peer's presence > updated quickly if they don't disconnect cleanly). Does anyone have > a feeling for the performance implications of this approach if many > peers are attached to a single hub? Is there another approach that > is recommend (such as using a custom UDP-based keep alive protocol)? > > Regards, > Matt > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From matthew at allpeers.com Thu Feb 9 09:34:19 2006 From: matthew at allpeers.com (Matthew Gertner) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Keep Alive and network performance In-Reply-To: <5037146A-FF9F-4786-BD8A-200BABF490DD@cluck.com> References: <71b79fa90602020957n28557e11j7f0b4cbbd1a22b35@mail.gmail.com> <20060203000808.GA2888@angeldust.chaos> <71b79fa90602030359q22d5d55bp10d26052c6291908@mail.gmail.com> <43EA4EA6.8040803@allpeers.com> <5037146A-FF9F-4786-BD8A-200BABF490DD@cluck.com> Message-ID: <43EB0C9B.7090506@allpeers.com> Daniel B. wrote: > Keep alive is both a missing component in practical P2P and a reason > why P2P is also hard to make practical. > > The trap is that the keep-alive is between two machines. Imagine > hundreds of thousands of machines!!!! Even with small signatures > matching each machine, we are talking about a flooded network. A > secondary trap is managing reliability. Reliability in P2P is not > moving bytes back and forth, but also taking into account the machine > may be a laptop or other personal computer that has a pathetic uptime. > > The better choice is online/offline signaling and optimistic > communications with good error control. I assume that by "optimistic communications" you mean that we should assume a connection exists and confirm this using periodic pings? Isn't this basically the equivalent of KA? It seems to me that the fundamental tradeoff is speed of notification vs. extra network traffic (i.e. you know more quickly when someone's network connection is cut if you ping more often), and this is the case both for KA and for a custom optimistic approach that uses pings. Am I missing something? Cheers, Matt From turbogeek at cluck.com Thu Feb 9 15:30:44 2006 From: turbogeek at cluck.com (Daniel B.) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Keep Alive and network performance In-Reply-To: <43EB0C9B.7090506@allpeers.com> References: <71b79fa90602020957n28557e11j7f0b4cbbd1a22b35@mail.gmail.com> <20060203000808.GA2888@angeldust.chaos> <71b79fa90602030359q22d5d55bp10d26052c6291908@mail.gmail.com> <43EA4EA6.8040803@allpeers.com> <5037146A-FF9F-4786-BD8A-200BABF490DD@cluck.com> <43EB0C9B.7090506@allpeers.com> Message-ID: In an optimistic communications should not use pings per say. My preference is just to write the data. To deal with optimism, in some cases, you do need feedback. A couple of choices are to use checksums or last-heard-from in your data. There are many complex issues here. Many of them depend on the P2P network and your application. Speed of notification is only sometimes important. If you are sending data via relayed machines, it would be better to stop a send soon, when the destination disappears. If the relays store for a later forward, the source does not care that the destination is up or down because the destination will eventually get the message. If we assume infinite bandwidth and computers that can never be flooded by bandwidth, its all cool. If we assume spotty wifi and 5 year old computers, the solutions need to be carefully implemented to avoid failures. On Feb 9, 2006, at 1:34 AM, Matthew Gertner wrote: > Daniel B. wrote: > >> Keep alive is both a missing component in practical P2P and a >> reason why P2P is also hard to make practical. >> >> The trap is that the keep-alive is between two machines. Imagine >> hundreds of thousands of machines!!!! Even with small signatures >> matching each machine, we are talking about a flooded network. A >> secondary trap is managing reliability. Reliability in P2P is not >> moving bytes back and forth, but also taking into account the >> machine may be a laptop or other personal computer that has a >> pathetic uptime. >> >> The better choice is online/offline signaling and optimistic >> communications with good error control. > > I assume that by "optimistic communications" you mean that we > should assume a connection exists and confirm this using periodic > pings? Isn't this basically the equivalent of KA? It seems to me > that the fundamental tradeoff is speed of notification vs. extra > network traffic (i.e. you know more quickly when someone's network > connection is cut if you ping more often), and this is the case > both for KA and for a custom optimistic approach that uses pings. > Am I missing something? > > Cheers, > Matt > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From dbarrett at quinthar.com Thu Feb 9 17:53:14 2006 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Keep Alive and network performance In-Reply-To: Message-ID: <20060209175320.9A6923FCA7@capsicum.zgp.org> Well, in practice keep-alives seem to do well despite the theoretical worst case. Really it's a balance between the importance of fast dirty-disconnect detection and bandwidth. If the feature is critical to your application, the bandwidth is worth it. If the feature isn't, then the bandwidth is wasted. If you're writing an IM-like tool, the general standard seems to be to detect coming online instantly (eg, you contact everyone on your buddy list the moment you come online), detect clean disconnects instantly (eg, notify everyone before you go offline), and then notice dirty disconnects after several seconds or more (eg, using keepalives). Thus IM-like presence in a world where most clients disconnect in a clean fashion (which, incidentally, is the world we live in) doesn't take much keep-alive traffic at all because the ping frequencies are so low (10+ seconds?). But another entirely separate (and more valuable?) use of keepalives is to keep NAT mappings alive in a hole-punching scenario. With iGlance (which is being presented at CodeCon on Saturday) I use an adaptive approach where I look for the slowest keepalive frequency that works. Adaptiveness is critical, as I've seen some NATs that start with 5-second windows, but grow up to 256-second windows over time. -david > -----Original Message----- > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On > Behalf Of Daniel B. > Sent: Thursday, February 09, 2006 7:31 AM > To: Peer-to-peer development. > Subject: Re: [p2p-hackers] Keep Alive and network performance > > In an optimistic communications should not use pings per say. My > preference is just to write the data. To deal with optimism, in some > cases, you do need feedback. A couple of choices are to use checksums > or last-heard-from in your data. > > There are many complex issues here. Many of them depend on the P2P > network and your application. Speed of notification is only sometimes > important. If you are sending data via relayed machines, it would be > better to stop a send soon, when the destination disappears. If the > relays store for a later forward, the source does not care that the > destination is up or down because the destination will eventually get > the message. > > If we assume infinite bandwidth and computers that can never be > flooded by bandwidth, its all cool. If we assume spotty wifi and 5 > year old computers, the solutions need to be carefully implemented to > avoid failures. > > On Feb 9, 2006, at 1:34 AM, Matthew Gertner wrote: > > > Daniel B. wrote: > > > >> Keep alive is both a missing component in practical P2P and a > >> reason why P2P is also hard to make practical. > >> > >> The trap is that the keep-alive is between two machines. Imagine > >> hundreds of thousands of machines!!!! Even with small signatures > >> matching each machine, we are talking about a flooded network. A > >> secondary trap is managing reliability. Reliability in P2P is not > >> moving bytes back and forth, but also taking into account the > >> machine may be a laptop or other personal computer that has a > >> pathetic uptime. > >> > >> The better choice is online/offline signaling and optimistic > >> communications with good error control. > > > > I assume that by "optimistic communications" you mean that we > > should assume a connection exists and confirm this using periodic > > pings? Isn't this basically the equivalent of KA? It seems to me > > that the fundamental tradeoff is speed of notification vs. extra > > network traffic (i.e. you know more quickly when someone's network > > connection is cut if you ping more often), and this is the case > > both for KA and for a custom optimistic approach that uses pings. > > Am I missing something? > > > > Cheers, > > Matt > > _______________________________________________ > > p2p-hackers mailing list > > p2p-hackers@zgp.org > > http://zgp.org/mailman/listinfo/p2p-hackers > > _______________________________________________ > > Here is a web page listing P2P Conferences: > > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From brturn at cisco.com Thu Feb 9 18:02:44 2006 From: brturn at cisco.com (Bryan Turner (brturn)) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Keep Alive and network performance In-Reply-To: <43EA4EA6.8040803@allpeers.com> Message-ID: <000301c62da3$09905190$b5576640@amer.cisco.com> Matt, TCP already uses keep-alives (typically set to 3-minutes). Adding additional keep-alives on top of this should be carefully deployed. For instance, if you send a keep-alive 3 minutes after the last packet received, you will be coinciding with the "natural" TCP keep-alive and not gaining any new information. Overhead: TCP messages over the wire typically cost Ethernet + IP + TCP header sizes, plus the size of your keep alive (the TCP keep alive on Windows was 64 bytes last I checked). A good overview of packet overhead is [1]. So a typical keep-alive is: ~16 + 20 + 20 + 64 bytes = 120 bytes Plus the ACK response - also 120 bytes (although the ACK may be acking other packets too, so this is a conservative estimate). Thus the TCP keep-alives consume: 240 bytes/3 min = 80 bytes/min = 4 bytes/sec = 32 bits/sec If your link capacity is a conservative 256 Kbit/sec broadband, and you wanted to use 1% for keep-alives, that gives you a budget of: ~2600 bits/sec = 325 bytes/sec. That's about one keep-alive per second for 1% overhead. If you have control over the hardware abstraction layer in your app, I would suggest piggybacking the socket write of the keep-alive with an outgoing data payload so the OS will send them out as one packet on the wire, saving the protocol overhead. --Bryan bryan.turner@pobox.com [1] http://sd.wareonearth.com/~phil/net/overhead/ -----Original Message----- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On Behalf Of Matthew Gertner Sent: Wednesday, February 08, 2006 3:04 PM To: Peer-to-peer development. Subject: [p2p-hackers] Keep Alive and network performance Hi, We have a straightforward P2P topology with each peer in the network attached to hub. We want to detect efficiently that a peer has disconnected from the hub. The obvious way to do this would be to use our current TCP connection with a relatively short Keep Alive value (a few seconds so we could see the peer's presence updated quickly if they don't disconnect cleanly). Does anyone have a feeling for the performance implications of this approach if many peers are attached to a single hub? Is there another approach that is recommend (such as using a custom UDP-based keep alive protocol)? Regards, Matt _______________________________________________ p2p-hackers mailing list p2p-hackers@zgp.org http://zgp.org/mailman/listinfo/p2p-hackers _______________________________________________ Here is a web page listing P2P Conferences: http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From gbildson at limepeer.com Thu Feb 9 18:04:23 2006 From: gbildson at limepeer.com (Greg Bildson) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Keep Alive and network performance In-Reply-To: <43EA4EA6.8040803@allpeers.com> Message-ID: Rather than using a dedicated message for keepalives, LimeWire first looks for message traffic of any kind on a connection. If there is traffic on the connection during a time window, then the connection is assumed to still be alive. If there has been no natural traffic, we will send a keepalive ping to it from the hub/ultrapeer. If there is still no network traffic coming from the host after a recheck time, we assume the connection is dead and kill it on the ultrapeer. We were concerned both about unclean disconnects and connections where the bandwidth was getting so choked that it was effectively dead. Thanks -greg > -----Original Message----- > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On > Behalf Of Matthew Gertner > Sent: Wednesday, February 08, 2006 3:04 PM > To: Peer-to-peer development. > Subject: [p2p-hackers] Keep Alive and network performance > > > Hi, > > We have a straightforward P2P topology with each peer in the network > attached to hub. We want to detect efficiently that a peer has > disconnected from the hub. The obvious way to do this would be to use > our current TCP connection with a relatively short Keep Alive value (a > few seconds so we could see the peer's presence updated quickly if they > don't disconnect cleanly). Does anyone have a feeling for the > performance implications of this approach if many peers are attached to > a single hub? Is there another approach that is recommend (such as using > a custom UDP-based keep alive protocol)? > > Regards, > Matt > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From gwendal.simon at francetelecom.com Thu Feb 9 19:15:55 2006 From: gwendal.simon at francetelecom.com (SIMON Gwendal RD-MAPS-ISS) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Keep Alive and network performance Message-ID: In Solipsis, we also use a unique message for both "heartbeat" (confirm aliveness) and "keepalive" (maintain NAT hole-punching). A french saying is "One stone, two hits !" :-) We also intended to adjust heartbeat periodicity to the "link value". Some links are essential to the network connectivity, so they deserve a small periodicity. Some links are valuable because of some criteria (in our case, the peer is graphically displayed). A "NAT hole-punching" periodicity is enough. Finally, some links are optional. We just keep them alive with a long periodicity because (i) why not, (ii) handshaking may be costly. This may be applied in other p2p networks : adjust keepalive periodicity to the number of shared files, to the redudancy (in DHT), to the estimated half-life of peer... -- Gwendal http://solipsis.netofpeers.net > -----Message d'origine----- > De : p2p-hackers-bounces@zgp.org > [mailto:p2p-hackers-bounces@zgp.org] De la part de Greg Bildson > Envoy? : jeudi 9 f?vrier 2006 19:04 > ? : Peer-to-peer development. > Objet : RE: [p2p-hackers] Keep Alive and network performance > > Rather than using a dedicated message for keepalives, > LimeWire first looks > for message traffic of any kind on a connection. If there is > traffic on the > connection during a time window, then the connection is > assumed to still be > alive. If there has been no natural traffic, we will send > a keepalive > ping to it from the hub/ultrapeer. If there is still no > network traffic > coming from the host after a recheck time, we assume the > connection is dead > and kill it on the ultrapeer. > > We were concerned both about unclean disconnects and > connections where the > bandwidth was getting so choked that it was effectively dead. > > Thanks > -greg > > > -----Original Message----- > > From: p2p-hackers-bounces@zgp.org > [mailto:p2p-hackers-bounces@zgp.org]On > > Behalf Of Matthew Gertner > > Sent: Wednesday, February 08, 2006 3:04 PM > > To: Peer-to-peer development. > > Subject: [p2p-hackers] Keep Alive and network performance > > > > > > Hi, > > > > We have a straightforward P2P topology with each peer in the network > > attached to hub. We want to detect efficiently that a peer has > > disconnected from the hub. The obvious way to do this would > be to use > > our current TCP connection with a relatively short Keep > Alive value (a > > few seconds so we could see the peer's presence updated > quickly if they > > don't disconnect cleanly). Does anyone have a feeling for the > > performance implications of this approach if many peers are > attached to > > a single hub? Is there another approach that is recommend > (such as using > > a custom UDP-based keep alive protocol)? > > > > Regards, > > Matt > > _______________________________________________ > > p2p-hackers mailing list > > p2p-hackers@zgp.org > > http://zgp.org/mailman/listinfo/p2p-hackers > > _______________________________________________ > > Here is a web page listing P2P Conferences: > > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From alain.damiral at student.uclouvain.be Mon Feb 13 10:36:16 2006 From: alain.damiral at student.uclouvain.be (Alain Damiral) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Secure communication in p2p networks Message-ID: <43F06120.50902@student.uclouvain.be> Hello list, This is my first post so please let me know if I'm already doing it wrong. I am a student and am currently working on establishing SSL/TLS enabled communication channels in a peer-to-peer network (for one-to-one secure communication). In my report, I would also like to mention some current research that might be in progress on secure protocols specifically designed for one to many communication. According to what I last heard, hardly anything has been done in this regard (creating shared secrets between more than 2 entities). I was hoping that maybe some of the members of this fine community could direct me to some documentation on current research on this topic. Well that's it for now, thank you all for your time and if you never received this message, please disregard it. -- Alain Damiral, I hope this message makes me look like a very intelligent person Universit? Catholique de Louvain - student alain.damiral'at'student.info.ucl.ac.be From fis at wiwi.hu-berlin.de Mon Feb 13 13:54:03 2006 From: fis at wiwi.hu-berlin.de (Matthias Fischmann) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Secure communication in p2p networks In-Reply-To: <43F06120.50902@student.uclouvain.be> References: <43F06120.50902@student.uclouvain.be> Message-ID: <20060213135403.GD6557@localhost.localdomain> hi alain, the reason there is not much research might be that solutions better than the trivial ones are hard to find. with trivial i mean that one of the communicating group broadcasts a symmetric key over ssl-secured channels to all other group members and then everybody uses symmetric crypto. slightly more efficiently, i think DH key agreement [1] can be adapted for more than two parties. one property of these approaches is that the receiver of a secured package only knows that *some* member of the group sent it, not which one. is that helpful? cheers, matthias [1] http://www.rsasecurity.com/rsalabs/node.asp?id=2248 On Mon, Feb 13, 2006 at 11:36:16AM +0100, Alain Damiral wrote: > To: p2p-hackers@zgp.org > From: Alain Damiral > Date: Mon, 13 Feb 2006 11:36:16 +0100 > Subject: [p2p-hackers] Secure communication in p2p networks > > Hello list, > > This is my first post so please let me know if I'm already doing it wrong. > > I am a student and am currently working on establishing SSL/TLS enabled > communication channels in a peer-to-peer network (for one-to-one secure > communication). In my report, I would also like to mention some current > research that might be in progress on secure protocols specifically > designed for one to many communication. According to what I last heard, > hardly anything has been done in this regard (creating shared secrets > between more than 2 entities). > > I was hoping that maybe some of the members of this fine community could > direct me to some documentation on current research on this topic. > > Well that's it for now, thank you all for your time and if you never > received this message, please disregard it. > > -- > Alain Damiral, > > I hope this message makes me look like a very intelligent person > > Universit? Catholique de Louvain - student > alain.damiral'at'student.info.ucl.ac.be > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences -- Institute of Information Systems, Humboldt-Universitaet zu Berlin web: http://www.wiwi.hu-berlin.de/~fis/ e-mail: fis@wiwi.hu-berlin.de tel: +49 30 2093-5742 fax: +49 30 2093-5741 office: Spandauer Strasse 1, R.324, 10178 Berlin, Germany -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://zgp.org/pipermail/p2p-hackers/attachments/20060213/51378d4f/attachment.pgp From networksimulator at gmail.com Mon Feb 13 16:46:57 2006 From: networksimulator at gmail.com (Ranus) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Secure communication in p2p networks In-Reply-To: <43F06120.50902@student.uclouvain.be> Message-ID: <000c01c630bd$1cce4470$ccc96fa6@thinkingfish> End-to-End encryption is tested by (mu)torrent (the greek letter \mu), they use DH public-key scheme to exchange the secret and symmetric key encryption for data transfer. read this interview http://www.slyck.com/news.php?story=1083 I don't know if it helps, are you trying to find a way to achieve something like group signature or merely secure communication for each connection? Hope you can tell more about your project, sounds interesting. -- Ranus Yue Tsinghua University ( -----Original Mail----- Sender: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] Alain Damiral time: 2006/02/13 18:36 receiver: p2p-hackers@zgp.org Subject: [p2p-hackers] Secure communication in p2p networks Hello list, This is my first post so please let me know if I'm already doing it wrong. I am a student and am currently working on establishing SSL/TLS enabled communication channels in a peer-to-peer network (for one-to-one secure communication). In my report, I would also like to mention some current research that might be in progress on secure protocols specifically designed for one to many communication. According to what I last heard, hardly anything has been done in this regard (creating shared secrets between more than 2 entities). I was hoping that maybe some of the members of this fine community could direct me to some documentation on current research on this topic. Well that's it for now, thank you all for your time and if you never received this message, please disregard it. -- Alain Damiral, I hope this message makes me look like a very intelligent person Universit? Catholique de Louvain - student alain.damiral'at'student.info.ucl.ac.be _______________________________________________ p2p-hackers mailing list p2p-hackers@zgp.org http://zgp.org/mailman/listinfo/p2p-hackers _______________________________________________ Here is a web page listing P2P Conferences: http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From alain.damiral at student.uclouvain.be Mon Feb 13 17:51:13 2006 From: alain.damiral at student.uclouvain.be (Alain Damiral) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Secure communication in p2p networks In-Reply-To: <000c01c630bd$1cce4470$ccc96fa6@thinkingfish> References: <000c01c630bd$1cce4470$ccc96fa6@thinkingfish> Message-ID: <43F0C711.505@student.uclouvain.be> Thanks, (and to Matthias also) So here are some extra details about my project... What I am trying to achieve is rather simple. I'm using OpenSSL but instead of letting it send data directly over a TCP stream, I feed the data back to the peer-to-peer middleware. I'm basically moving SSL/TLS's transport layer one level up. This can be done as long as the actual transport of messages accross the network is reliable. This allows for secure communication between two peers much in the same way SSL usually does, without having to trust any intermediate peer in the network. The most exotic aspect of my work is that the p2p middleware I am working with is written in Oz (hello Oz coders !) while the OpenSSL API is defined in C. Simple end-to-end TLS enabled channels can already give the user some modest solutions for group communication, if a symmetric key algorithm is available. This can be done by having a service responsible for providing peers (if they can be authentified) with a shared secret key, that he can update regularly. However I believe that it is possible to design a protocol that would achieve something more efficient and maybe stronger. I do not intend to support such a protocol in my work, but if something exists or is being researched today, I would like to mention it in my written report :) So it is absolutely not critical for my project. Once again, thanks for the help ! -Alain Ranus wrote: >End-to-End encryption is tested by (mu)torrent (the greek letter \mu), they >use DH public-key scheme to exchange the secret and symmetric key encryption >for data transfer. read this interview >http://www.slyck.com/news.php?story=1083 > >I don't know if it helps, are you trying to find a way to achieve something >like group signature or merely secure communication for each connection? >Hope you can tell more about your project, sounds interesting. > > >-- >Ranus Yue >Tsinghua University > >( >-----Original Mail----- >Sender: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] >Alain Damiral >time: 2006/02/13 18:36 >receiver: p2p-hackers@zgp.org >Subject: [p2p-hackers] Secure communication in p2p networks > >Hello list, > >This is my first post so please let me know if I'm already doing it wrong. > >I am a student and am currently working on establishing SSL/TLS enabled >communication channels in a peer-to-peer network (for one-to-one secure >communication). In my report, I would also like to mention some current >research that might be in progress on secure protocols specifically designed >for one to many communication. According to what I last heard, hardly >anything has been done in this regard (creating shared secrets between more >than 2 entities). > >I was hoping that maybe some of the members of this fine community could >direct me to some documentation on current research on this topic. > >Well that's it for now, thank you all for your time and if you never >received this message, please disregard it. > >-- >Alain Damiral, > >I hope this message makes me look like a very intelligent person > >Universit? Catholique de Louvain - student >alain.damiral'at'student.info.ucl.ac.be > >_______________________________________________ >p2p-hackers mailing list >p2p-hackers@zgp.org >http://zgp.org/mailman/listinfo/p2p-hackers >_______________________________________________ >Here is a web page listing P2P Conferences: >http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > >_______________________________________________ >p2p-hackers mailing list >p2p-hackers@zgp.org >http://zgp.org/mailman/listinfo/p2p-hackers >_______________________________________________ >Here is a web page listing P2P Conferences: >http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > From coderman at gmail.com Mon Feb 13 20:48:17 2006 From: coderman at gmail.com (coderman) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Secure communication in p2p networks In-Reply-To: <43F06120.50902@student.uclouvain.be> References: <43F06120.50902@student.uclouvain.be> Message-ID: <4ef5fec60602131248m6f348876xdb44758b0a4bdfc6@mail.gmail.com> On 2/13/06, Alain Damiral wrote: > ... > I am a student and am currently working on establishing SSL/TLS enabled > communication channels in a peer-to-peer network (for one-to-one secure > communication). you might find openvpn and the latest 4.3 openssh with tun/tap relevant/interesting then. there are some other tun/tap p2p projects in progress if you search around a bit as well. > In my report, I would also like to mention some current > research that might be in progress on secure protocols specifically > designed for one to many communication. According to what I last heard, > hardly anything has been done in this regard (creating shared secrets > between more than 2 entities). are you thinking of group key distribution and management? this is tricky, but there is also a lot of work being done in this regard, from efficient pre distribution methods to n-party public key mechanisms. From greg at capepaterson.com Mon Feb 13 21:20:43 2006 From: greg at capepaterson.com (Greg Rolan) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] gig announcement In-Reply-To: <4ef5fec60602131248m6f348876xdb44758b0a4bdfc6@mail.gmail.com> References: <43F06120.50902@student.uclouvain.be> <4ef5fec60602131248m6f348876xdb44758b0a4bdfc6@mail.gmail.com> Message-ID: <43F0F82B.2080101@capepaterson.com> Hi people, I'm not sure if this sort of post is allowed on this list so apologies if this contravenes the etiquette conventions here. We're a small startup developing a broadly distributed application which requires an dht overlay network for information lookup. I've been playing around with various open-source dht implementations, lurking in this list and trying to find a good solution for the NAT traversal issues. Many members have posted good suggestions for approaches to a number of problems and it appears that there is the makings of a fairly good (and I acknowledge that good is not perfect or even great) solution. What we would like to do is engage someone to do some integration work based on an existing code-base or even buy a commercial ready-to-go implementation of dht node code which addresses NAT/Firewall traversal issues using a mix of Firewall port-forwarding registration, hole punching and proxy nodes (super-node) strategies. We would possibly be interested in working with another organisations to achieve common goals in this area. I appreciate that the search for a solution will be an ongoing activity. For questions, offers, suggestions or more information (a specification document) please reply off-list (unless you think it is relevant to the list). Thanks - and apologies again if this post was inappropriate. Greg -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.375 / Virus Database: 267.15.6/257 - Release Date: 2/10/2006 From agthorr at cs.uoregon.edu Mon Feb 13 23:51:10 2006 From: agthorr at cs.uoregon.edu (Daniel Stutzbach) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Secure communication in p2p networks In-Reply-To: <4ef5fec60602131248m6f348876xdb44758b0a4bdfc6@mail.gmail.com> References: <43F06120.50902@student.uclouvain.be> <4ef5fec60602131248m6f348876xdb44758b0a4bdfc6@mail.gmail.com> Message-ID: <20060213235110.GB5752@cs.uoregon.edu> On 2/13/06, Alain Damiral wrote: > In my report, I would also like to mention some current > research that might be in progress on secure protocols specifically > designed for one to many communication. According to what I last heard, > hardly anything has been done in this regard (creating shared secrets > between more than 2 entities). Another group at my University is working on something like that: http://www.acsa-admin.org/2005/abstracts/166.html -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon From supercanadian at gmail.com Tue Feb 14 08:35:32 2006 From: supercanadian at gmail.com (Charles Iliya Krempeaux) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? Message-ID: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> Hello, (First, I should say that I'm writing this around midnight, so let me first apologise for all the spelling and grammar mistakes I know I'll probably make but won't notice until I read this tomorrow. But anyways....) One thing that's been on my mind for a while is making "tighter integration" between HTTP and P2P technologies. (I've finally been motivated to write this e-mail since I need to put this to practice now. Although I really wanted to write a longer and more detailed blog post on this. And do more research first.... Maybe later.) The current methods that bring some integration between HTTP and P2P technologies include the magnet-uri and BitTorrent (via BitTorrent's "application/x-bittorrent" MIME type). However, one could get even tighter integration. (Here's a few different ways that may work... I'd like to get people's opinions on these proposals.) One possible avenue for getting even tighter integration is the creation of something like a "conditional get" for P2P technologies. (For those unfamiliar with the HTTP "conditional get", look for information of the "If-None-Match" and "If-Modified-Since" HTTP headers.) A "P2P conditional get" (sent by the client) would tell the server NOT to send the requested document if there is an alternative way of getting the document via a P2P technology supported by the client. (The client would tell the server which P2P technologies it supports.) (This "P2P conditional get" would work similar to the "If-None-Match" and "If-Modified-Since" "conditional get" headers.). For example, it might be something like: X-If-No-Alternate: bittorrent, something-else, magnet First, this says that the client supports the protocols: "bittorrent", "something-else", and "magnet". It also says it prefers "bittorrent" first, then "something-else", and then "magnet". (Really, there's nothing that mandates that these protocols be for P2P technologies. But that is the use case that is motivating this.) This could also include "q" factors. Perhaps something like: X-If-No-Alternate: protocol1; q=1.0, protocol2; q=0.7, protocol3; q=0.5 And there could be ways of including more information too. (Which could be used to add protocol specific information.) For example: X-If-No-Alternate: protocol1; q=1.0; a=b;c="d", magnet; q=0.5;urn-allowed="sha1 md5" Note, I'm NOT trying to propose the exact format here or the names used. Just trying to illustrate what I'm talking about through examples. (I've tried to follow similar formats as other HTTP headers though.) The server might then respond with an "HTTP 302" (Found) and then redirect the user to a P2P version using a URI for the P2P technology. This "redirect" would be done via the HTTP "Location" header. For example: HTTP/1.1 302 Found Location: magnet:?xt=urn:sha1:WRCIRZV5ZO56CWMNHFV4FRGNPWPPDVKT or: HTTP/1.1 302 Found Location: x-some-new-p2p-protocol:etc/etc/etc However, I'm not sure if "302" is the correct HTTP response code to use. (Anyone have any suggestions?) Another possible solution would be to use the "Accept" header (sent by the client). Now, this would only be useful for P2P technologies that are shunted into HTTP via a MIME type (like BitTorrent and the "application/x-bittorrent" MIME type.). But for technologies which have this, the client could list the P2P shunt MIME types first, in the HTTP "Accept" header. For example: Accept: application/x-bittorrent, application/ogg However, with BitTorrent's case, this brings the problem of confusing the torrent and files in the torrent. So perhaps a "accept extension" is needed: Accept: application/x-bittorrent; enclosure-content-type="application/ogg", application/ogg Although I still think this still conveys the wrong semantics. So perhaps this really isn't a good solution. Another alternative is to use the "Upgrade" header (from the client) and the "HTTP 101" (Switching Protocols) (from the server). This would let you switch to a P2P protocol. However, this seems to somewhat imply that whatever protocol is "upgraded to" will have communication going from between the same client and server. (At least that's my feeling.) Any Comments? Anyone have any suggestions or advice? Also, are there already any proposals or specs along these lines? (No sense in reinventing the wheel.) TIA See ya -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ From kmagdsick at limewire.com Tue Feb 14 16:07:34 2006 From: kmagdsick at limewire.com (Karl A. Magdsick) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> Message-ID: <43F20046.4040709@limewire.com> Hello Charles, A couple of weeks ago, there was some work done to integrate both BitTorrent and Gnutella technologies with the Plone CMS/Zope webserver. http://plone.org/events/sprints/snow-sprint3 I've done some preliminary work (along with Matt Hamilton from NetSight) using an Apache plugin to add the X-Alt and X-Node HTTP headers that allow Gnutella clients downloading the same file to find each other and take load off the server. I haven't gone past proof-of-concept, but I have confirmed that two LimeWire clients are able to find each other through an Apache server using a plugin I helped write. The two LimeWire clients then share portions of the file with each other and thereby reduce load on the server. Note that Apache 2.2 on OS X 10.4 (Tiger) deals very poorly with a single connection utilizing multiple 128 kb GET requests in random order. We got very high CPU useage on the PowerBook being used as the webserver, and even got a kernel panic. We were forced to use the latest Apache 2.0, which seemed to work fine. (This was the weekend of Feb 4th, 2006.) Good Luck, Karl Magdsick Software Engineer, Lime Wire LLC Charles Iliya Krempeaux wrote: >Hello, > >(First, I should say that I'm writing this around midnight, so let me >first apologise for all the spelling and grammar mistakes I know I'll >probably make but won't notice until I read this tomorrow. But >anyways....) > >One thing that's been on my mind for a while is making "tighter >integration" between HTTP and P2P technologies. (I've finally been >motivated to write this e-mail since I need to put this to practice >now. Although I really wanted to write a longer and more detailed >blog post on this. And do more research first.... Maybe later.) > >The current methods that bring some integration between HTTP and P2P >technologies include the magnet-uri and BitTorrent (via BitTorrent's >"application/x-bittorrent" MIME type). > >However, one could get even tighter integration. (Here's a few >different ways that may work... I'd like to get people's opinions on >these proposals.) > >One possible avenue for getting even tighter integration is the >creation of something like a "conditional get" for P2P technologies. >(For those unfamiliar with the HTTP "conditional get", look for >information of the "If-None-Match" and "If-Modified-Since" HTTP >headers.) > >A "P2P conditional get" (sent by the client) would tell the server NOT >to send the requested document if there is an alternative way of >getting the document via a P2P technology supported by the client. >(The client would tell the server which P2P technologies it supports.) > (This "P2P conditional get" would work similar to the "If-None-Match" >and "If-Modified-Since" "conditional get" headers.). > >For example, it might be something like: > > X-If-No-Alternate: bittorrent, something-else, magnet > >First, this says that the client supports the protocols: "bittorrent", >"something-else", and "magnet". It also says it prefers "bittorrent" >first, then "something-else", and then "magnet". > >(Really, there's nothing that mandates that these protocols be for P2P >technologies. But that is the use case that is motivating this.) > >This could also include "q" factors. Perhaps something like: > > X-If-No-Alternate: protocol1; q=1.0, protocol2; q=0.7, protocol3; q=0.5 > >And there could be ways of including more information too. (Which >could be used to add protocol specific information.) For example: > > X-If-No-Alternate: protocol1; q=1.0; a=b;c="d", magnet; >q=0.5;urn-allowed="sha1 md5" > >Note, I'm NOT trying to propose the exact format here or the names >used. Just trying to illustrate what I'm talking about through >examples. (I've tried to follow similar formats as other HTTP headers >though.) > >The server might then respond with an "HTTP 302" (Found) and then >redirect the user to a P2P version using a URI for the P2P technology. > This "redirect" would be done via the HTTP "Location" header. For >example: > > HTTP/1.1 302 Found > Location: magnet:?xt=urn:sha1:WRCIRZV5ZO56CWMNHFV4FRGNPWPPDVKT > >or: > > HTTP/1.1 302 Found > Location: x-some-new-p2p-protocol:etc/etc/etc > >However, I'm not sure if "302" is the correct HTTP response code to >use. (Anyone have any suggestions?) > > >Another possible solution would be to use the "Accept" header (sent by >the client). Now, this would only be useful for P2P technologies that >are shunted into HTTP via a MIME type (like BitTorrent and the >"application/x-bittorrent" MIME type.). But for technologies which >have this, the client could list the P2P shunt MIME types first, in >the HTTP "Accept" header. For example: > > Accept: application/x-bittorrent, application/ogg > >However, with BitTorrent's case, this brings the problem of confusing >the torrent and files in the torrent. So perhaps a "accept extension" >is needed: > > Accept: application/x-bittorrent; >enclosure-content-type="application/ogg", application/ogg > >Although I still think this still conveys the wrong semantics. So >perhaps this really isn't a good solution. > > >Another alternative is to use the "Upgrade" header (from the client) >and the "HTTP 101" (Switching Protocols) (from the server). This would >let you switch to a P2P protocol. However, this seems to somewhat >imply that whatever protocol is "upgraded to" will have communication >going from between the same client and server. (At least that's my >feeling.) > > >Any Comments? Anyone have any suggestions or advice? > >Also, are there already any proposals or specs along these lines? (No >sense in reinventing the wheel.) > > >TIA > > >See ya > >-- > Charles Iliya Krempeaux, B.Sc. > > charles @ reptile.ca > supercanadian @ gmail.com > > developer weblog: http://ChangeLog.ca/ >___________________________________________________________________________ > Make Television http://maketelevision.com/ >_______________________________________________ >p2p-hackers mailing list >p2p-hackers@zgp.org >http://zgp.org/mailman/listinfo/p2p-hackers >_______________________________________________ >Here is a web page listing P2P Conferences: >http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > From travis at redswoosh.net Tue Feb 14 18:35:53 2006 From: travis at redswoosh.net (Travis Kalanick) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <43F20046.4040709@limewire.com> Message-ID: <200602141841.k1EIf9qx019157@be9.noc0.redswoosh.com> Web integration has been key to Red Swoosh's architecture. We have been doing a more advanced version of your "conditional get" where clients simply look at http as another source. In addition we have several parameters that are embedded in the links that allow for different streaming/download methods (peers only, peers and server, peers but fallback on server, etc.). You can get a Basic version of the API (a limited set of the parameters available in the advanced API) emailed to you here: http://redswoosh.net/tryus_step_by_step.php We've had live customers (i.e. websites) up on this web-integration architecture since 2001 (Deviantart.com launched with us in October 2001) and have found it to be quite a powerful way to employ P2P tech. Thanks, Travis -----Original Message----- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On Behalf Of Karl A. Magdsick Sent: Tuesday, February 14, 2006 8:08 AM To: charles@reptile.ca; Peer-to-peer development. Subject: Re: [p2p-hackers] Tighter HTTP and P2P integration?? Hello Charles, A couple of weeks ago, there was some work done to integrate both BitTorrent and Gnutella technologies with the Plone CMS/Zope webserver. http://plone.org/events/sprints/snow-sprint3 I've done some preliminary work (along with Matt Hamilton from NetSight) using an Apache plugin to add the X-Alt and X-Node HTTP headers that allow Gnutella clients downloading the same file to find each other and take load off the server. I haven't gone past proof-of-concept, but I have confirmed that two LimeWire clients are able to find each other through an Apache server using a plugin I helped write. The two LimeWire clients then share portions of the file with each other and thereby reduce load on the server. Note that Apache 2.2 on OS X 10.4 (Tiger) deals very poorly with a single connection utilizing multiple 128 kb GET requests in random order. We got very high CPU useage on the PowerBook being used as the webserver, and even got a kernel panic. We were forced to use the latest Apache 2.0, which seemed to work fine. (This was the weekend of Feb 4th, 2006.) Good Luck, Karl Magdsick Software Engineer, Lime Wire LLC Charles Iliya Krempeaux wrote: >Hello, > >(First, I should say that I'm writing this around midnight, so let me >first apologise for all the spelling and grammar mistakes I know I'll >probably make but won't notice until I read this tomorrow. But >anyways....) > >One thing that's been on my mind for a while is making "tighter >integration" between HTTP and P2P technologies. (I've finally been >motivated to write this e-mail since I need to put this to practice >now. Although I really wanted to write a longer and more detailed >blog post on this. And do more research first.... Maybe later.) > >The current methods that bring some integration between HTTP and P2P >technologies include the magnet-uri and BitTorrent (via BitTorrent's >"application/x-bittorrent" MIME type). > >However, one could get even tighter integration. (Here's a few >different ways that may work... I'd like to get people's opinions on >these proposals.) > >One possible avenue for getting even tighter integration is the >creation of something like a "conditional get" for P2P technologies. >(For those unfamiliar with the HTTP "conditional get", look for >information of the "If-None-Match" and "If-Modified-Since" HTTP >headers.) > >A "P2P conditional get" (sent by the client) would tell the server NOT >to send the requested document if there is an alternative way of >getting the document via a P2P technology supported by the client. >(The client would tell the server which P2P technologies it supports.) > (This "P2P conditional get" would work similar to the "If-None-Match" >and "If-Modified-Since" "conditional get" headers.). > >For example, it might be something like: > > X-If-No-Alternate: bittorrent, something-else, magnet > >First, this says that the client supports the protocols: "bittorrent", >"something-else", and "magnet". It also says it prefers "bittorrent" >first, then "something-else", and then "magnet". > >(Really, there's nothing that mandates that these protocols be for P2P >technologies. But that is the use case that is motivating this.) > >This could also include "q" factors. Perhaps something like: > > X-If-No-Alternate: protocol1; q=1.0, protocol2; q=0.7, protocol3; q=0.5 > >And there could be ways of including more information too. (Which >could be used to add protocol specific information.) For example: > > X-If-No-Alternate: protocol1; q=1.0; a=b;c="d", magnet; >q=0.5;urn-allowed="sha1 md5" > >Note, I'm NOT trying to propose the exact format here or the names >used. Just trying to illustrate what I'm talking about through >examples. (I've tried to follow similar formats as other HTTP headers >though.) > >The server might then respond with an "HTTP 302" (Found) and then >redirect the user to a P2P version using a URI for the P2P technology. > This "redirect" would be done via the HTTP "Location" header. For >example: > > HTTP/1.1 302 Found > Location: magnet:?xt=urn:sha1:WRCIRZV5ZO56CWMNHFV4FRGNPWPPDVKT > >or: > > HTTP/1.1 302 Found > Location: x-some-new-p2p-protocol:etc/etc/etc > >However, I'm not sure if "302" is the correct HTTP response code to >use. (Anyone have any suggestions?) > > >Another possible solution would be to use the "Accept" header (sent by >the client). Now, this would only be useful for P2P technologies that >are shunted into HTTP via a MIME type (like BitTorrent and the >"application/x-bittorrent" MIME type.). But for technologies which >have this, the client could list the P2P shunt MIME types first, in >the HTTP "Accept" header. For example: > > Accept: application/x-bittorrent, application/ogg > >However, with BitTorrent's case, this brings the problem of confusing >the torrent and files in the torrent. So perhaps a "accept extension" >is needed: > > Accept: application/x-bittorrent; >enclosure-content-type="application/ogg", application/ogg > >Although I still think this still conveys the wrong semantics. So >perhaps this really isn't a good solution. > > >Another alternative is to use the "Upgrade" header (from the client) >and the "HTTP 101" (Switching Protocols) (from the server). This would >let you switch to a P2P protocol. However, this seems to somewhat >imply that whatever protocol is "upgraded to" will have communication >going from between the same client and server. (At least that's my >feeling.) > > >Any Comments? Anyone have any suggestions or advice? > >Also, are there already any proposals or specs along these lines? (No >sense in reinventing the wheel.) > > >TIA > > >See ya > >-- > Charles Iliya Krempeaux, B.Sc. > > charles @ reptile.ca > supercanadian @ gmail.com > > developer weblog: http://ChangeLog.ca/ >___________________________________________________________________________ > Make Television http://maketelevision.com/ >_______________________________________________ >p2p-hackers mailing list >p2p-hackers@zgp.org >http://zgp.org/mailman/listinfo/p2p-hackers >_______________________________________________ >Here is a web page listing P2P Conferences: >http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > _______________________________________________ p2p-hackers mailing list p2p-hackers@zgp.org http://zgp.org/mailman/listinfo/p2p-hackers _______________________________________________ Here is a web page listing P2P Conferences: http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From ian at locut.us Tue Feb 14 18:43:58 2006 From: ian at locut.us (Ian Clarke) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> Message-ID: <41879D03-2A20-4D3B-80EF-724C4A1B6629@locut.us> You should take a look at Dijjer - http://dijjer.org/. Dijjer is a Java client that runs on your computer, and emulates a local HTTP server. You can request (almost) any file through Dijjer that is available on the WWW by requesting http://127.0.0.1:9115/http://the.url.of/the/ file.mov . Unlike BitTorrent, files are downloaded from Dijjer starting at the beginning, and will be streamed to the browser over the local HTTP connection as they are downloaded. This allows people to do neat things like embed Quicktime movies in web browsers and have the video play as it is downloaded from the P2P network. This will appear to your web browser as if you are downloading the file from an ordinary HTTP server, but in fact, the file will be downloaded through a distributed P2P cache. The first time the file is requested it will be pulled into the cache, subsequent requests will come from other nodes in the P2P network. Other Dijjer features: * NAT hole-punching (see http://en.wikipedia.org/wiki/ UDP_hole_punching) * RESTful design (see above) * Decentralized algorithm (based on small-world principle) * Two complimentary forms of integrity checking (http://dijjer.org/ wiki/IntegrityChecking) * TCP friendly data transfer (http://dijjer.org/wiki/DataTransmission) * Neat internal network abstraction layer that has formed the basis for the next major release of Freenet (http://dijjer.org/wiki/MessagingLayer) * Free as in speech (GPL license) If this sounds like advocacy, that's because it is :-) Dijjer needs developers, the code is relatively clean, and relatively well documented, but right now I am basically the only active developer, and I haven't had much free time lately with the next version of Freenet, not to mention my day job. In particular, it needs work on deployment, there is no OSX installer right now, and Windows version relies on Excelsior Jet - an easy to install but pure Java version would be nice. If you are a Java programmer and looking for something interesting to do, check out the Development section of the site and join our mailing list. Ian. On 14 Feb 2006, at 00:35, Charles Iliya Krempeaux wrote: > Hello, > > (First, I should say that I'm writing this around midnight, so let me > first apologise for all the spelling and grammar mistakes I know I'll > probably make but won't notice until I read this tomorrow. But > anyways....) > > One thing that's been on my mind for a while is making "tighter > integration" between HTTP and P2P technologies. (I've finally been > motivated to write this e-mail since I need to put this to practice > now. Although I really wanted to write a longer and more detailed > blog post on this. And do more research first.... Maybe later.) > > The current methods that bring some integration between HTTP and P2P > technologies include the magnet-uri and BitTorrent (via BitTorrent's > "application/x-bittorrent" MIME type). > > However, one could get even tighter integration. (Here's a few > different ways that may work... I'd like to get people's opinions on > these proposals.) > > One possible avenue for getting even tighter integration is the > creation of something like a "conditional get" for P2P technologies. > (For those unfamiliar with the HTTP "conditional get", look for > information of the "If-None-Match" and "If-Modified-Since" HTTP > headers.) > > A "P2P conditional get" (sent by the client) would tell the server NOT > to send the requested document if there is an alternative way of > getting the document via a P2P technology supported by the client. > (The client would tell the server which P2P technologies it supports.) > (This "P2P conditional get" would work similar to the "If-None-Match" > and "If-Modified-Since" "conditional get" headers.). > > For example, it might be something like: > > X-If-No-Alternate: bittorrent, something-else, magnet > > First, this says that the client supports the protocols: "bittorrent", > "something-else", and "magnet". It also says it prefers "bittorrent" > first, then "something-else", and then "magnet". > > (Really, there's nothing that mandates that these protocols be for P2P > technologies. But that is the use case that is motivating this.) > > This could also include "q" factors. Perhaps something like: > > X-If-No-Alternate: protocol1; q=1.0, protocol2; q=0.7, > protocol3; q=0.5 > > And there could be ways of including more information too. (Which > could be used to add protocol specific information.) For example: > > X-If-No-Alternate: protocol1; q=1.0; a=b;c="d", magnet; > q=0.5;urn-allowed="sha1 md5" > > Note, I'm NOT trying to propose the exact format here or the names > used. Just trying to illustrate what I'm talking about through > examples. (I've tried to follow similar formats as other HTTP headers > though.) > > The server might then respond with an "HTTP 302" (Found) and then > redirect the user to a P2P version using a URI for the P2P technology. > This "redirect" would be done via the HTTP "Location" header. For > example: > > HTTP/1.1 302 Found > Location: magnet:?xt=urn:sha1:WRCIRZV5ZO56CWMNHFV4FRGNPWPPDVKT > > or: > > HTTP/1.1 302 Found > Location: x-some-new-p2p-protocol:etc/etc/etc > > However, I'm not sure if "302" is the correct HTTP response code to > use. (Anyone have any suggestions?) > > > Another possible solution would be to use the "Accept" header (sent by > the client). Now, this would only be useful for P2P technologies that > are shunted into HTTP via a MIME type (like BitTorrent and the > "application/x-bittorrent" MIME type.). But for technologies which > have this, the client could list the P2P shunt MIME types first, in > the HTTP "Accept" header. For example: > > Accept: application/x-bittorrent, application/ogg > > However, with BitTorrent's case, this brings the problem of confusing > the torrent and files in the torrent. So perhaps a "accept extension" > is needed: > > Accept: application/x-bittorrent; > enclosure-content-type="application/ogg", application/ogg > > Although I still think this still conveys the wrong semantics. So > perhaps this really isn't a good solution. > > > Another alternative is to use the "Upgrade" header (from the client) > and the "HTTP 101" (Switching Protocols) (from the server). This would > let you switch to a P2P protocol. However, this seems to somewhat > imply that whatever protocol is "upgraded to" will have communication > going from between the same client and server. (At least that's my > feeling.) > > > Any Comments? Anyone have any suggestions or advice? > > Also, are there already any proposals or specs along these lines? (No > sense in reinventing the wheel.) > > > TIA > > > See ya > > -- > Charles Iliya Krempeaux, B.Sc. > > charles @ reptile.ca > supercanadian @ gmail.com > > developer weblog: http://ChangeLog.ca/ > ______________________________________________________________________ > _____ > Make Television http:// > maketelevision.com/ > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From supercanadian at gmail.com Tue Feb 14 23:49:02 2006 From: supercanadian at gmail.com (Charles Iliya Krempeaux) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <43F20046.4040709@limewire.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <43F20046.4040709@limewire.com> Message-ID: <84ce626f0602141549l2d994826g84b01eafb61da5bd@mail.gmail.com> Hello Karl, On 2/14/06, Karl A. Magdsick wrote: > Hello Charles, > > A couple of weeks ago, there was some work done to integrate > both BitTorrent and Gnutella technologies with the Plone > CMS/Zope webserver. http://plone.org/events/sprints/snow-sprint3 Would this be a more direct link?: http://plone.org/products/plonetramline > I've done some preliminary work (along with Matt Hamilton from NetSight) > using an Apache plugin to add the X-Alt and X-Node HTTP headers > that allow Gnutella clients downloading the same file to find each other > and take load off the server. (I know I could probably read the code to get this info, but I thought it might be easier to just ask, so....) Could you explain the semantics and usage of X-Alt and X-Node more. As well as elaborate more on how all this works, please. (My guess is that the client sends an X-Alt... and then the server responds with a X-Node, probably giving a list of nodes. But please elaborate.) > I haven't gone past proof-of-concept, but I have confirmed that two > LimeWire clients are able to find each other through an Apache > server using a plugin I helped write. The two LimeWire clients > then share portions of the file with each other and thereby reduce > load on the server. If I remember the Gnutella protocol correctly... Do the 2 clients still share this data via HTTP? And it is a normally HTTP download?... or is BitTorrent being employed for this? > Note that Apache 2.2 on OS X 10.4 (Tiger) deals very poorly with a single > connection utilizing multiple 128 kb GET requests in random order. We > got very high CPU useage on the PowerBook being used as the webserver, > and even got a kernel panic. We were forced to use the latest Apache 2.0, > which seemed to work fine. (This was the weekend of Feb 4th, 2006.) Is this for the clients. I.e., is this for the client to client transfer of file pieces? [...] Thanks. See ya -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ From supercanadian at gmail.com Wed Feb 15 00:38:25 2006 From: supercanadian at gmail.com (Charles Iliya Krempeaux) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <41879D03-2A20-4D3B-80EF-724C4A1B6629@locut.us> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <41879D03-2A20-4D3B-80EF-724C4A1B6629@locut.us> Message-ID: <84ce626f0602141638r1550bf0aq8fdd0e0b8114ba7a@mail.gmail.com> Hello, I've been actually considering using a local HTTP server to implement things. (Which is something you are doing too.) Although I thought some extra "sugar" was warranted, to help with (user) usabiliy and developer usability. I skimmed through the stuff on dijjer.org, and I have a questions. Does digger pay attention to HTTP headers? And act basically as a web cache? For example, here's some (but not all) "rules" that I was planning on following.... If HTTP headers "say" that something should NOT be cached, then it should NOT be on the P2P network. Stuff on the P2P network should be removed after the date for the "Expires" header passes? Things from non-HTTP GET methods should NOT be put onto the P2P network? Etc. Does dijjer follow these web caching rules? See ya On 2/14/06, Ian Clarke wrote: > You should take a look at Dijjer - http://dijjer.org/. > > Dijjer is a Java client that runs on your computer, and emulates a > local HTTP server. > > You can request (almost) any file through Dijjer that is available on > the WWW by requesting http://127.0.0.1:9115/http://the.url.of/the/ > file.mov . Unlike BitTorrent, files are downloaded from Dijjer > starting at the beginning, and will be streamed to the browser over > the local HTTP connection as they are downloaded. This allows people > to do neat things like embed Quicktime movies in web browsers and > have the video play as it is downloaded from the P2P network. > > This will appear to your web browser as if you are downloading the > file from an ordinary HTTP server, but in fact, the file will be > downloaded through a distributed P2P cache. The first time the file > is requested it will be pulled into the cache, subsequent requests > will come from other nodes in the P2P network. > > Other Dijjer features: > > * NAT hole-punching (see http://en.wikipedia.org/wiki/ > UDP_hole_punching) > * RESTful design (see above) > * Decentralized algorithm (based on small-world principle) > * Two complimentary forms of integrity checking (http://dijjer.org/ > wiki/IntegrityChecking) > * TCP friendly data transfer (http://dijjer.org/wiki/DataTransmission) > * Neat internal network abstraction layer that has formed the basis > for the next > major release of Freenet (http://dijjer.org/wiki/MessagingLayer) > * Free as in speech (GPL license) > > If this sounds like advocacy, that's because it is :-) Dijjer needs > developers, the code is relatively clean, and relatively well > documented, but right now I am basically the only active developer, > and I haven't had much free time lately with the next version of > Freenet, not to mention my day job. > > In particular, it needs work on deployment, there is no OSX installer > right now, and Windows version relies on Excelsior Jet - an easy to > install but pure Java version would be nice. > > If you are a Java programmer and looking for something interesting to > do, check out the Development section of the site and join our > mailing list. > > Ian. > > On 14 Feb 2006, at 00:35, Charles Iliya Krempeaux wrote: > > > Hello, > > > > (First, I should say that I'm writing this around midnight, so let me > > first apologise for all the spelling and grammar mistakes I know I'll > > probably make but won't notice until I read this tomorrow. But > > anyways....) > > > > One thing that's been on my mind for a while is making "tighter > > integration" between HTTP and P2P technologies. (I've finally been > > motivated to write this e-mail since I need to put this to practice > > now. Although I really wanted to write a longer and more detailed > > blog post on this. And do more research first.... Maybe later.) > > > > The current methods that bring some integration between HTTP and P2P > > technologies include the magnet-uri and BitTorrent (via BitTorrent's > > "application/x-bittorrent" MIME type). > > > > However, one could get even tighter integration. (Here's a few > > different ways that may work... I'd like to get people's opinions on > > these proposals.) > > > > One possible avenue for getting even tighter integration is the > > creation of something like a "conditional get" for P2P technologies. > > (For those unfamiliar with the HTTP "conditional get", look for > > information of the "If-None-Match" and "If-Modified-Since" HTTP > > headers.) > > > > A "P2P conditional get" (sent by the client) would tell the server NOT > > to send the requested document if there is an alternative way of > > getting the document via a P2P technology supported by the client. > > (The client would tell the server which P2P technologies it supports.) > > (This "P2P conditional get" would work similar to the "If-None-Match" > > and "If-Modified-Since" "conditional get" headers.). > > > > For example, it might be something like: > > > > X-If-No-Alternate: bittorrent, something-else, magnet > > > > First, this says that the client supports the protocols: "bittorrent", > > "something-else", and "magnet". It also says it prefers "bittorrent" > > first, then "something-else", and then "magnet". > > > > (Really, there's nothing that mandates that these protocols be for P2P > > technologies. But that is the use case that is motivating this.) > > > > This could also include "q" factors. Perhaps something like: > > > > X-If-No-Alternate: protocol1; q=1.0, protocol2; q=0.7, > > protocol3; q=0.5 > > > > And there could be ways of including more information too. (Which > > could be used to add protocol specific information.) For example: > > > > X-If-No-Alternate: protocol1; q=1.0; a=b;c="d", magnet; > > q=0.5;urn-allowed="sha1 md5" > > > > Note, I'm NOT trying to propose the exact format here or the names > > used. Just trying to illustrate what I'm talking about through > > examples. (I've tried to follow similar formats as other HTTP headers > > though.) > > > > The server might then respond with an "HTTP 302" (Found) and then > > redirect the user to a P2P version using a URI for the P2P technology. > > This "redirect" would be done via the HTTP "Location" header. For > > example: > > > > HTTP/1.1 302 Found > > Location: magnet:?xt=urn:sha1:WRCIRZV5ZO56CWMNHFV4FRGNPWPPDVKT > > > > or: > > > > HTTP/1.1 302 Found > > Location: x-some-new-p2p-protocol:etc/etc/etc > > > > However, I'm not sure if "302" is the correct HTTP response code to > > use. (Anyone have any suggestions?) > > > > > > Another possible solution would be to use the "Accept" header (sent by > > the client). Now, this would only be useful for P2P technologies that > > are shunted into HTTP via a MIME type (like BitTorrent and the > > "application/x-bittorrent" MIME type.). But for technologies which > > have this, the client could list the P2P shunt MIME types first, in > > the HTTP "Accept" header. For example: > > > > Accept: application/x-bittorrent, application/ogg > > > > However, with BitTorrent's case, this brings the problem of confusing > > the torrent and files in the torrent. So perhaps a "accept extension" > > is needed: > > > > Accept: application/x-bittorrent; > > enclosure-content-type="application/ogg", application/ogg > > > > Although I still think this still conveys the wrong semantics. So > > perhaps this really isn't a good solution. > > > > > > Another alternative is to use the "Upgrade" header (from the client) > > and the "HTTP 101" (Switching Protocols) (from the server). This would > > let you switch to a P2P protocol. However, this seems to somewhat > > imply that whatever protocol is "upgraded to" will have communication > > going from between the same client and server. (At least that's my > > feeling.) > > > > > > Any Comments? Anyone have any suggestions or advice? > > > > Also, are there already any proposals or specs along these lines? (No > > sense in reinventing the wheel.) -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ From supercanadian at gmail.com Wed Feb 15 00:59:36 2006 From: supercanadian at gmail.com (Charles Iliya Krempeaux) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <200602141841.k1EIf9qx019157@be9.noc0.redswoosh.com> References: <43F20046.4040709@limewire.com> <200602141841.k1EIf9qx019157@be9.noc0.redswoosh.com> Message-ID: <84ce626f0602141659r75488e35tf90a27dbb7b11a64@mail.gmail.com> Hello, I took a look at the site, but I had trouble finding any technical information. Is there any available? (Or is everything proprietary?) See ya On 2/14/06, Travis Kalanick wrote: > Web integration has been key to Red Swoosh's architecture. > > We have been doing a more advanced version of your "conditional get" where > clients simply look at http as another source. In addition we have several > parameters that are embedded in the links that allow for different > streaming/download methods (peers only, peers and server, peers but fallback > on server, etc.). > > You can get a Basic version of the API (a limited set of the parameters > available in the advanced API) emailed to you here: > > http://redswoosh.net/tryus_step_by_step.php > > > We've had live customers (i.e. websites) up on this web-integration > architecture since 2001 (Deviantart.com launched with us in October 2001) > and have found it to be quite a powerful way to employ P2P tech. > > > Thanks, > > Travis > > -----Original Message----- > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On > Behalf Of Karl A. Magdsick > Sent: Tuesday, February 14, 2006 8:08 AM > To: charles@reptile.ca; Peer-to-peer development. > Subject: Re: [p2p-hackers] Tighter HTTP and P2P integration?? > > Hello Charles, > > A couple of weeks ago, there was some work done to integrate > both BitTorrent and Gnutella technologies with the Plone > CMS/Zope webserver. http://plone.org/events/sprints/snow-sprint3 > > I've done some preliminary work (along with Matt Hamilton from NetSight) > using an Apache plugin to add the X-Alt and X-Node HTTP headers > that allow Gnutella clients downloading the same file to find each other > and take load off the server. > > I haven't gone past proof-of-concept, but I have confirmed that two > LimeWire clients are able to find each other through an Apache > server using a plugin I helped write. The two LimeWire clients > then share portions of the file with each other and thereby reduce > load on the server. > > Note that Apache 2.2 on OS X 10.4 (Tiger) deals very poorly with a single > connection utilizing multiple 128 kb GET requests in random order. We > got very high CPU useage on the PowerBook being used as the webserver, > and even got a kernel panic. We were forced to use the latest Apache 2.0, > which seemed to work fine. (This was the weekend of Feb 4th, 2006.) > > > Good Luck, > > Karl Magdsick > Software Engineer, Lime Wire LLC > > Charles Iliya Krempeaux wrote: > > >Hello, > > > >(First, I should say that I'm writing this around midnight, so let me > >first apologise for all the spelling and grammar mistakes I know I'll > >probably make but won't notice until I read this tomorrow. But > >anyways....) > > > >One thing that's been on my mind for a while is making "tighter > >integration" between HTTP and P2P technologies. (I've finally been > >motivated to write this e-mail since I need to put this to practice > >now. Although I really wanted to write a longer and more detailed > >blog post on this. And do more research first.... Maybe later.) > > > >The current methods that bring some integration between HTTP and P2P > >technologies include the magnet-uri and BitTorrent (via BitTorrent's > >"application/x-bittorrent" MIME type). > > > >However, one could get even tighter integration. (Here's a few > >different ways that may work... I'd like to get people's opinions on > >these proposals.) > > > >One possible avenue for getting even tighter integration is the > >creation of something like a "conditional get" for P2P technologies. > >(For those unfamiliar with the HTTP "conditional get", look for > >information of the "If-None-Match" and "If-Modified-Since" HTTP > >headers.) > > > >A "P2P conditional get" (sent by the client) would tell the server NOT > >to send the requested document if there is an alternative way of > >getting the document via a P2P technology supported by the client. > >(The client would tell the server which P2P technologies it supports.) > > (This "P2P conditional get" would work similar to the "If-None-Match" > >and "If-Modified-Since" "conditional get" headers.). > > > >For example, it might be something like: > > > > X-If-No-Alternate: bittorrent, something-else, magnet > > > >First, this says that the client supports the protocols: "bittorrent", > >"something-else", and "magnet". It also says it prefers "bittorrent" > >first, then "something-else", and then "magnet". > > > >(Really, there's nothing that mandates that these protocols be for P2P > >technologies. But that is the use case that is motivating this.) > > > >This could also include "q" factors. Perhaps something like: > > > > X-If-No-Alternate: protocol1; q=1.0, protocol2; q=0.7, protocol3; q=0.5 > > > >And there could be ways of including more information too. (Which > >could be used to add protocol specific information.) For example: > > > > X-If-No-Alternate: protocol1; q=1.0; a=b;c="d", magnet; > >q=0.5;urn-allowed="sha1 md5" > > > >Note, I'm NOT trying to propose the exact format here or the names > >used. Just trying to illustrate what I'm talking about through > >examples. (I've tried to follow similar formats as other HTTP headers > >though.) > > > >The server might then respond with an "HTTP 302" (Found) and then > >redirect the user to a P2P version using a URI for the P2P technology. > > This "redirect" would be done via the HTTP "Location" header. For > >example: > > > > HTTP/1.1 302 Found > > Location: magnet:?xt=urn:sha1:WRCIRZV5ZO56CWMNHFV4FRGNPWPPDVKT > > > >or: > > > > HTTP/1.1 302 Found > > Location: x-some-new-p2p-protocol:etc/etc/etc > > > >However, I'm not sure if "302" is the correct HTTP response code to > >use. (Anyone have any suggestions?) > > > > > >Another possible solution would be to use the "Accept" header (sent by > >the client). Now, this would only be useful for P2P technologies that > >are shunted into HTTP via a MIME type (like BitTorrent and the > >"application/x-bittorrent" MIME type.). But for technologies which > >have this, the client could list the P2P shunt MIME types first, in > >the HTTP "Accept" header. For example: > > > > Accept: application/x-bittorrent, application/ogg > > > >However, with BitTorrent's case, this brings the problem of confusing > >the torrent and files in the torrent. So perhaps a "accept extension" > >is needed: > > > > Accept: application/x-bittorrent; > >enclosure-content-type="application/ogg", application/ogg > > > >Although I still think this still conveys the wrong semantics. So > >perhaps this really isn't a good solution. > > > > > >Another alternative is to use the "Upgrade" header (from the client) > >and the "HTTP 101" (Switching Protocols) (from the server). This would > >let you switch to a P2P protocol. However, this seems to somewhat > >imply that whatever protocol is "upgraded to" will have communication > >going from between the same client and server. (At least that's my > >feeling.) > > > > > >Any Comments? Anyone have any suggestions or advice? > > > >Also, are there already any proposals or specs along these lines? (No > >sense in reinventing the wheel.) -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ From kmagdsick at limewire.com Wed Feb 15 02:00:34 2006 From: kmagdsick at limewire.com (Karl A. Magdsick) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <84ce626f0602141549l2d994826g84b01eafb61da5bd@mail.gmail.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <43F20046.4040709@limewire.com> <84ce626f0602141549l2d994826g84b01eafb61da5bd@mail.gmail.com> Message-ID: <43F28B42.8070009@limewire.com> Charles Iliya Krempeaux wrote: [snip] >Would this be a more direct link?: http://plone.org/products/plonetramline > > No. Tramline is just a somewhat clever hack to prevent the Zope webserver from storing large files as objects in its object database. If you're not familiar with Zope, looking at Tramilne will likely just confuse you. Tramline was involved with my work only in a round-about way. >>I've done some preliminary work (along with Matt Hamilton from NetSight) >>using an Apache plugin to add the X-Alt and X-Node HTTP headers >>that allow Gnutella clients downloading the same file to find each other >>and take load off the server. >> >> > >(I know I could probably read the code to get this info, but I thought >it might be easier to just ask, so....) Could you explain the >semantics and usage of X-Alt and X-Node more. As well as elaborate >more on how all this works, please. > > First, a short bit about the relationship between Gnutella and HTTP: Gnutella uses HTTP to transfer all file data. This is a huge advantage for integration with webservers, as Gnutella clients can treat the webserver as just another Gnutella client. Many P2P networks invent their own file-transfer protocols, but the overhead of HTTP isn't very large, and the ability to pretend that regular webservers are peers is a huge win. Optional HTTP headers are used to exchange information so that Gnutella clients that are downloading the same file can form a "download mesh" -- a set of clients that share and share alike chunks of a file they are all trying to download. Once you've installed a webserver plugin to allow the webserver to understand and send a very small number of optional headers, the webserver can be thought of as a special case of Gnutella client: one that can't search or download, but one that is capable of uploading files, helping coordinate download meshes, and participating in download meshes. It's all just HTTP with a handful of optional headers. The Gnutella protocol itself is only necessary for searching, and for getting files from hosts that are unable to punch holes in NAT. It's a fair assumption that your webserver isn't behind NAT, or you've punched a hole in NAT for your webserver. In this case, the webserver only needs to speak the Gnutella protocol if you want your webserver to respond to Gnutella searches or you want some kind of web interface for searching Gnutella. Facilitating a Gnutella mesh through a webserver is very simple, much simpler than implementing a BitTorrent tracker. There is nearly no extra intelligence required in the webserver, and the clients all treat the webserver as just another member of the download mesh. Next, a bit about the headers: X-Node: a Gnutella client that's able to punch through NAT/firewalls and get an externally contactable IP address will send its external IP and Gnutella port number in this header. X-Alt: a Gnutella client will send IP:port pairs of Gnutella clients in the download mesh using the X-Alt header. If a client is able to punch through NAT, it'll include itself in the X-Alt list once it is sharing part of the file. X-NAlt: this header contains a list of "bad" alternate locations. In essence, this header says "you gave me some bad entries in an X-Alt header. stop sending out the following IP:port pairs". X-FAlt: this is the X-Alt header, but for mesh members that are unable to punch through NAT/firewalls. The entries contain information about which proxies to use to contact these firewalled clients. X-NFAlt: this is the firewalled version of X-NAlt. How this all works: For each file to be downloaded, the webserver remembers information that it gets in X-Alt headers, and spits the same information back out in X-Alt headers. It purges X-Alt entries that it sees in X-NAlt headers. It does the same thing for X-FAlt and X-NFAlt headers. The X-Node information may be useful in deciding which entries in its internal X-Alt pool should be sent out. >(My guess is that the client sends an X-Alt... and then the server >responds with a X-Node, probably giving a list of nodes. But please >elaborate.) > > Unlike semi-centralized systems like Napster or BitTorrent, there is no distinction between client and server in Gnutella. X-Node is for "this is me", and X-Alt is for "you may want to talk to these". Both the "client" and "server" side send the X-Alt headers. Presumably, the "client" side knows how it contacted the "server" and therefore knows how the "server" is externally contactable, so it's useless for the "server" to send X-Node. >>I haven't gone past proof-of-concept, but I have confirmed that two >>LimeWire clients are able to find each other through an Apache >>server using a plugin I helped write. The two LimeWire clients >>then share portions of the file with each other and thereby reduce >>load on the server. >> >> > >If I remember the Gnutella protocol correctly... Do the 2 clients >still share this data via HTTP? And it is a normally HTTP >download?... or is BitTorrent being employed for this? > > It's pure HTTP. There's no need to thrown in some other protocol. I hope this helps, Karl Magdsick Software Engineer, Lime Wire LLC From ian at locut.us Wed Feb 15 02:13:31 2006 From: ian at locut.us (Ian Clarke) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <84ce626f0602141638r1550bf0aq8fdd0e0b8114ba7a@mail.gmail.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <41879D03-2A20-4D3B-80EF-724C4A1B6629@locut.us> <84ce626f0602141638r1550bf0aq8fdd0e0b8114ba7a@mail.gmail.com> Message-ID: <097AC0C1-CFE5-46A4-A49C-42691CA3F69D@locut.us> On 14 Feb 2006, at 16:38, Charles Iliya Krempeaux wrote: > I've been actually considering using a local HTTP server to implement > things. (Which is something you are doing too.) Although I thought > some extra "sugar" was warranted, to help with (user) usabiliy and > developer usability. We have added some sugar too. For example, we have javascript available on our site that can detect whether Dijjer is present on the user's machine and redirect the user to a download page if it isn't. > I skimmed through the stuff on dijjer.org, and I have a questions. > Does digger pay attention to HTTP headers? And act basically as a web > cache? Yes, we have attempted to stick as closely to the HTTP protocol as possible. > For example, here's some (but not all) "rules" that I was planning on > following.... If HTTP headers "say" that something should NOT be > cached, then it should NOT be on the P2P network. Yes, we obey the various HTTP no-cache headers. We consider it important for Dijjer to act as much like a HTTP cache as possible so that we are afforded the protection the DMCA gives to caches for copyright infringement. > Stuff on the P2P > network should be removed after the date for the "Expires" header > passes? We don't obey "Expires" explicitly, but since Dijjer pings the original web server each-time something is downloaded to ensure that the file is still there and hasn't changed, if it is removed from the web server, it will effectively be removed from Dijjer at the same instant. > Things from non-HTTP GET methods should NOT be put onto the > P2P network? Etc. We attempt to be smart about our interaction with HTTP, if there are shortcomings in this regard they will be easy to fix. Ian. From travis at redswoosh.net Wed Feb 15 08:40:33 2006 From: travis at redswoosh.net (Travis Kalanick) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <84ce626f0602141659r75488e35tf90a27dbb7b11a64@mail.gmail.com> Message-ID: <200602150845.k1F8jnqx028119@be9.noc0.redswoosh.com> An email with the Basic API is available at the link I provided. http://redswoosh.net/tryus_step_by_step.php Check it out, let me know what you think. Also, keep in mind that our installation system will be re-vamped within 45 days. We'll have a much cleaner, tighter .exe based installation. T -----Original Message----- From: Charles Iliya Krempeaux [mailto:supercanadian@gmail.com] Sent: Tuesday, February 14, 2006 5:00 PM To: Peer-to-peer development. Cc: Travis Kalanick Subject: Re: [p2p-hackers] Tighter HTTP and P2P integration?? Hello, I took a look at the site, but I had trouble finding any technical information. Is there any available? (Or is everything proprietary?) See ya On 2/14/06, Travis Kalanick wrote: > Web integration has been key to Red Swoosh's architecture. > > We have been doing a more advanced version of your "conditional get" where > clients simply look at http as another source. In addition we have several > parameters that are embedded in the links that allow for different > streaming/download methods (peers only, peers and server, peers but fallback > on server, etc.). > > You can get a Basic version of the API (a limited set of the parameters > available in the advanced API) emailed to you here: > > http://redswoosh.net/tryus_step_by_step.php > > > We've had live customers (i.e. websites) up on this web-integration > architecture since 2001 (Deviantart.com launched with us in October 2001) > and have found it to be quite a powerful way to employ P2P tech. > > > Thanks, > > Travis > > -----Original Message----- > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On > Behalf Of Karl A. Magdsick > Sent: Tuesday, February 14, 2006 8:08 AM > To: charles@reptile.ca; Peer-to-peer development. > Subject: Re: [p2p-hackers] Tighter HTTP and P2P integration?? > > Hello Charles, > > A couple of weeks ago, there was some work done to integrate > both BitTorrent and Gnutella technologies with the Plone > CMS/Zope webserver. http://plone.org/events/sprints/snow-sprint3 > > I've done some preliminary work (along with Matt Hamilton from NetSight) > using an Apache plugin to add the X-Alt and X-Node HTTP headers > that allow Gnutella clients downloading the same file to find each other > and take load off the server. > > I haven't gone past proof-of-concept, but I have confirmed that two > LimeWire clients are able to find each other through an Apache > server using a plugin I helped write. The two LimeWire clients > then share portions of the file with each other and thereby reduce > load on the server. > > Note that Apache 2.2 on OS X 10.4 (Tiger) deals very poorly with a single > connection utilizing multiple 128 kb GET requests in random order. We > got very high CPU useage on the PowerBook being used as the webserver, > and even got a kernel panic. We were forced to use the latest Apache 2.0, > which seemed to work fine. (This was the weekend of Feb 4th, 2006.) > > > Good Luck, > > Karl Magdsick > Software Engineer, Lime Wire LLC > > Charles Iliya Krempeaux wrote: > > >Hello, > > > >(First, I should say that I'm writing this around midnight, so let me > >first apologise for all the spelling and grammar mistakes I know I'll > >probably make but won't notice until I read this tomorrow. But > >anyways....) > > > >One thing that's been on my mind for a while is making "tighter > >integration" between HTTP and P2P technologies. (I've finally been > >motivated to write this e-mail since I need to put this to practice > >now. Although I really wanted to write a longer and more detailed > >blog post on this. And do more research first.... Maybe later.) > > > >The current methods that bring some integration between HTTP and P2P > >technologies include the magnet-uri and BitTorrent (via BitTorrent's > >"application/x-bittorrent" MIME type). > > > >However, one could get even tighter integration. (Here's a few > >different ways that may work... I'd like to get people's opinions on > >these proposals.) > > > >One possible avenue for getting even tighter integration is the > >creation of something like a "conditional get" for P2P technologies. > >(For those unfamiliar with the HTTP "conditional get", look for > >information of the "If-None-Match" and "If-Modified-Since" HTTP > >headers.) > > > >A "P2P conditional get" (sent by the client) would tell the server NOT > >to send the requested document if there is an alternative way of > >getting the document via a P2P technology supported by the client. > >(The client would tell the server which P2P technologies it supports.) > > (This "P2P conditional get" would work similar to the "If-None-Match" > >and "If-Modified-Since" "conditional get" headers.). > > > >For example, it might be something like: > > > > X-If-No-Alternate: bittorrent, something-else, magnet > > > >First, this says that the client supports the protocols: "bittorrent", > >"something-else", and "magnet". It also says it prefers "bittorrent" > >first, then "something-else", and then "magnet". > > > >(Really, there's nothing that mandates that these protocols be for P2P > >technologies. But that is the use case that is motivating this.) > > > >This could also include "q" factors. Perhaps something like: > > > > X-If-No-Alternate: protocol1; q=1.0, protocol2; q=0.7, protocol3; q=0.5 > > > >And there could be ways of including more information too. (Which > >could be used to add protocol specific information.) For example: > > > > X-If-No-Alternate: protocol1; q=1.0; a=b;c="d", magnet; > >q=0.5;urn-allowed="sha1 md5" > > > >Note, I'm NOT trying to propose the exact format here or the names > >used. Just trying to illustrate what I'm talking about through > >examples. (I've tried to follow similar formats as other HTTP headers > >though.) > > > >The server might then respond with an "HTTP 302" (Found) and then > >redirect the user to a P2P version using a URI for the P2P technology. > > This "redirect" would be done via the HTTP "Location" header. For > >example: > > > > HTTP/1.1 302 Found > > Location: magnet:?xt=urn:sha1:WRCIRZV5ZO56CWMNHFV4FRGNPWPPDVKT > > > >or: > > > > HTTP/1.1 302 Found > > Location: x-some-new-p2p-protocol:etc/etc/etc > > > >However, I'm not sure if "302" is the correct HTTP response code to > >use. (Anyone have any suggestions?) > > > > > >Another possible solution would be to use the "Accept" header (sent by > >the client). Now, this would only be useful for P2P technologies that > >are shunted into HTTP via a MIME type (like BitTorrent and the > >"application/x-bittorrent" MIME type.). But for technologies which > >have this, the client could list the P2P shunt MIME types first, in > >the HTTP "Accept" header. For example: > > > > Accept: application/x-bittorrent, application/ogg > > > >However, with BitTorrent's case, this brings the problem of confusing > >the torrent and files in the torrent. So perhaps a "accept extension" > >is needed: > > > > Accept: application/x-bittorrent; > >enclosure-content-type="application/ogg", application/ogg > > > >Although I still think this still conveys the wrong semantics. So > >perhaps this really isn't a good solution. > > > > > >Another alternative is to use the "Upgrade" header (from the client) > >and the "HTTP 101" (Switching Protocols) (from the server). This would > >let you switch to a P2P protocol. However, this seems to somewhat > >imply that whatever protocol is "upgraded to" will have communication > >going from between the same client and server. (At least that's my > >feeling.) > > > > > >Any Comments? Anyone have any suggestions or advice? > > > >Also, are there already any proposals or specs along these lines? (No > >sense in reinventing the wheel.) -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ From supercanadian at gmail.com Wed Feb 15 22:39:29 2006 From: supercanadian at gmail.com (Charles Iliya Krempeaux) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <43F28B42.8070009@limewire.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <43F20046.4040709@limewire.com> <84ce626f0602141549l2d994826g84b01eafb61da5bd@mail.gmail.com> <43F28B42.8070009@limewire.com> Message-ID: <84ce626f0602151439w2031da3cl966c0d0a08aefa67@mail.gmail.com> Hello Karl, On 2/14/06, Karl A. Magdsick wrote: > Charles Iliya Krempeaux wrote: [...] > >>I've done some preliminary work (along with Matt Hamilton from NetSight) > >>using an Apache plugin to add the X-Alt and X-Node HTTP headers > >>that allow Gnutella clients downloading the same file to find each other > >>and take load off the server. > >> > >> > > > >(I know I could probably read the code to get this info, but I thought > >it might be easier to just ask, so....) Could you explain the > >semantics and usage of X-Alt and X-Node more. As well as elaborate > >more on how all this works, please. > > > > > > First, a short bit about the relationship between Gnutella and HTTP: > > Gnutella uses HTTP to transfer all file data. This is a huge advantage > for integration with webservers, as Gnutella clients can treat the webserver > as just another Gnutella client. Many P2P networks invent their own > file-transfer protocols, but the overhead of HTTP isn't very large, and > the ability to pretend that regular webservers are peers is a huge win. > > Optional HTTP headers are used to exchange information so that Gnutella > clients that are downloading the same file can form a "download mesh" -- a > set of clients that share and share alike chunks of a file they are all > trying to download. > > Once you've installed a webserver plugin to allow the webserver to > understand > and send a very small number of optional headers, the webserver can be > thought of as a special case of Gnutella client: one that can't search > or download, > but one that is capable of uploading files, helping coordinate download > meshes, > and participating in download meshes. It's all just HTTP with a handful of > optional headers. > > The Gnutella protocol itself is only necessary for searching, and for > getting files from hosts that are unable to punch holes in NAT. It's a fair > assumption that your webserver isn't behind NAT, or you've punched a hole > in NAT for your webserver. In this case, the webserver only needs to speak > the Gnutella protocol if you want your webserver to respond to Gnutella > searches or you want some kind of web interface for searching Gnutella. > > Facilitating a Gnutella mesh through a webserver is very simple, much > simpler than implementing a BitTorrent tracker. There is nearly no > extra intelligence > required in the webserver, and the clients all treat the webserver as > just another > member of the download mesh. > > > Next, a bit about the headers: > > X-Node: a Gnutella client that's able to punch through NAT/firewalls > and get > an externally contactable IP address will send its external IP and > Gnutella port > number in this header. > > X-Alt: a Gnutella client will send IP:port pairs of Gnutella clients in > the download > mesh using the X-Alt header. If a client is able to punch through NAT, > it'll > include itself in the X-Alt list once it is sharing part of the file. > > X-NAlt: this header contains a list of "bad" alternate locations. In > essence, this > header says "you gave me some bad entries in an X-Alt header. stop sending > out the following IP:port pairs". > > X-FAlt: this is the X-Alt header, but for mesh members that are unable > to punch > through NAT/firewalls. The entries contain information about which proxies > to use to contact these firewalled clients. > > X-NFAlt: this is the firewalled version of X-NAlt. > > > How this all works: > > For each file to be downloaded, the webserver remembers information that > it gets > in X-Alt headers, and spits the same information back out in X-Alt > headers. It purges > X-Alt entries that it sees in X-NAlt headers. It does the same thing > for X-FAlt and > X-NFAlt headers. The X-Node information may be useful in deciding which > entries in its internal X-Alt pool should be sent out. That illuminates things a bit. I guess my knowledge of the Gnutella protocol is a bit lacking :-) (It's been a while since I looked at the protocol :-) ) I did a bit of searching (after you wrote this), and came to these 2 documents that were helpful too: http://www.the-gdf.org/wiki/index.php?title=Known_HTTP_Download_Headers http://www.the-gdf.org/wiki/index.php?title=The_Download_Mesh The "X-Alt" headers seem to be pretty close to what I was thinking. Although (in my original post) I had this type of information communicated with the "Location" header. And I was also thinking of somehow "encoding" all the different alternate locations into one URL. But, the "X-Alt" header seems to be more flexible, in that you can list all the alternate sources easily. One problem I see with it (at least for my purposes) is that it only let's you list a IP address with a TCP port number. I guess because Gnutella has a special format for requesting files; as documented here: http://www.the-gdf.org/wiki/index.php?title=File_Transfer I think the "X-Alt" would have been able to be used for more things if you were able to use full URLs or full URIs. But anyways, something like "X-Alt" seems better than using the "Location" header. In the HTTP 1.1 spec, there is mention of a "Link" header. (See section 19.6.2.4 of RFC 2068.) This could probably be used. The "Link" header basically has the same semantics as the HTML element. So, something like the following could be done. Link: ; rel="alternate" Link: ; rel="alternate" Link: ; rel="alternate" Link: ; rel="alternate" (Note, in the example above, the server is sending multiple "Link" headers in one response.) With the link you can add other "attributes" too. (Think "attributes" in terms of HTML "attribute"... on the element.) You can have "title", "class", "media", "type", etc. (Just like the HTML element.) As a side note, Gnutella could even use the "Link" header instead of the "X-Alt" header. For example, the following: X-Alt: 192.0.2.17:6347, 192.0.2.44 could be translated into: Link: ; rel="alternate" Link: ; rel="alternate" (Where "FILE_INDEX" and "FILE_NAME" would be replaced with whatever they actually are.) So, having said all that, I'm still thinking that a "P2P conditional get" is a good idea. (For reason I'll explain in the next paragraph. But please argue if you disagree.) And that, the shunting of the client (or servent or whatever) onto the P2P network could be done via those "Link" headers. Now, if some of those "alternate" locations are Gnutella servents, then you'd use the Gnutella extension to HTTP. And if they are using a different protocol, the you do whatever is necessary for it. The reason I'm still inclined to go with a "P2P conditional get" is because it gives some level of control to the publisher. That way they are in some sense sanctioning it. (Which under some situations matters.) Now, like with web caching, one could always go ahead and use a P2P method of getting the file unilaterally -- without asking or needing the publisher to get involved. So right now my thinking is something like the following example. (Please argue if you disagree.) Client: GET /some/file HTTP/1.1 Host: example.com X-If-No-Alternate: ## PUT STUFF HERE ABOUT WHAT I CAN ACCEPT ## Server: HTTP/1.1 204 No Content Link: ; rel="alternate" Link: ; rel="alternate"; Link: ; rel="alternate", class="gnutella" (Note sure if the "204" HTTP result code is the correct one to use though. But anyways....) Note, I'm using a "URI reference" on the BitTorrent URI to access the file "within" the torrent. I'm not sure if anyone has used that kind of notation before. But it seems like a good way to "give a link to" what's inside of the torrent. Also note, that I've maked the Gnutella link with a "class" parameters... so that the client (or servernt or whatever) can identify it as being Gnutella. Thoughts? Comments? [...] See ya -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ From coderman at gmail.com Thu Feb 16 00:34:16 2006 From: coderman at gmail.com (coderman) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> Message-ID: <4ef5fec60602151634w2f37e25dhb338c9af2e7b50c1@mail.gmail.com> On 2/14/06, Charles Iliya Krempeaux wrote: > ... > One thing that's been on my mind for a while is making "tighter > integration" between HTTP and P2P technologies. > ... > A "P2P conditional get" (sent by the client) would tell the server NOT > to send the requested document if there is an alternative way of > getting the document via a P2P technology supported by the client. > ... > Also, are there already any proposals or specs along these lines? another route to http / p2p integration is through the file system with each peer running a local web server instance for UI on top of a shared fs tree / archive. [decentralized and/or distributed file systems are another topic, so i'll assume the presence of a robust shared file system view among peers in this explanation] our context is a bit different, as this mainly applies to private group networks with known client configurations (everyone running off a live CD / hdd install of the same shared OS) the desired handlers for various MIME types / filters / views can be accomplished per user with custom browser configurations and plugins. for interactive / collaborative / real time functions the private group network endpoints can be used to host services. consider: a pre-distributed archive / fs tree associated with the group: .../mygroup/public/music/ video/ docs/ all peers can reference these resources via local methods like file:///... or aliases in a drupal instance on localhost - http://127.0.0.1/video/favorites. aliases/vhosts for 127.0.0.1 (localhost, home, mygroup, etc) can also be used to provide distinct shared group apps/instances with distinct document roots as desired. (again, simply apache and drupal with a common httpd.conf bound to 127.0.0.1:80 using vservers for names in /etc/hosts) services and other resources are bound to peer identities and IPv6, for example a private IPv6 network using feed:7ba3:c779:e5f5:cc1a:8bc7:5fe1:ed08 where feed: is the common subnet and the remaining identifier space is based off a hash of public node identity. feed:7ba3:c779:e5f5:cc1a:8bc7:5fe1:ed08 can then be aliased like 127.0.0.1 and will remain valid as long as communication (direct or via proxy) with that peer can be accomplished. using private networks also means that UDP|TCP/IPv4/v6 aware applications can be hosted on these endpoints and used as private p2p group services. so while this approach requires a heavy endpoint (custom live CD / hdd OS linux image with group key material and base|cached fs distributed securely out of band) the benefits from a security|privacy and flexibility viewpoint are significant. this moves a lot of the complexity out of application space (where it usually resides for p2p) and into the networking / file system layers, where it is more appropriate and can be handled much more robustly. (think access restrictions, network and distributed filesystems, netfilter hooks, security modules / caps, etc rather than trying to reproduce a subset of this poorly in user space applications) this is one of the many reasons i am so fond of the tun/tap based networking software (openvpn, openssh 4.3), IPsec, and virtual machines (Xen, VMware player, etc). From supercanadian at gmail.com Thu Feb 16 06:39:31 2006 From: supercanadian at gmail.com (Charles Iliya Krempeaux) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <84ce626f0602151439w2031da3cl966c0d0a08aefa67@mail.gmail.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <43F20046.4040709@limewire.com> <84ce626f0602141549l2d994826g84b01eafb61da5bd@mail.gmail.com> <43F28B42.8070009@limewire.com> <84ce626f0602151439w2031da3cl966c0d0a08aefa67@mail.gmail.com> Message-ID: <84ce626f0602152239t451de6b6of0b7d603a827b510@mail.gmail.com> Hello, On 2/15/06, Charles Iliya Krempeaux wrote: [...] > So right now my thinking is something like the following example. > (Please argue if you disagree.) > > Client: > > GET /some/file HTTP/1.1 > Host: example.com > X-If-No-Alternate: ## PUT STUFF HERE ABOUT WHAT I CAN ACCEPT ## > > > Server: > > HTTP/1.1 204 No Content > Link: ; rel="alternate" > Link: ; rel="alternate"; > Link: ; rel="alternate", class="gnutella" I suppose if the "Link" header is being used to return information, this could be generalized to make this "conditional get" be a way to probe for "Link" data. So, instead of using "X-If-No-Alternate" it could be "X-If-No-Link" (and then specify what relation or other meta data you want). For example, I could get a blogs RSS feed with: X-If-No-Link: rel="alternate"; type="application/rss+xml" Although the exact notation may need to be different since using the "=" sign may not be enough. (For example, "rel" is a space separated list. And "type" is a content type, not a MIME type... and we're really trying to match a MIME type.) Maybe using CSS selector style notation. And in that case, the above example would be: X-If-No-Link: [rel~="alternate"][type="application/rss+xml"] (The "~=" operator is like "element of".) But anways, back to the original example, it might then be: Client: GET /some/file HTTP/1.1 Host: example.com X-If-No-Link: [rel~="alternate"][href^="http:"][class~="gnutella"] X-If-No-Link: [rel~="alternate"][type="application/x-bittorrent"] Server: HTTP/1.1 204 No Content Link: ; rel="alternate"; class="gnutella" Link: ; rel="alternate"; class="gnutella" Link: ; rel="alternate"; Which says, don't give me any content if I can get this over gnutella or bittorrent. (And give me links to those.) (The syntax for "X-If-No-Link" probably needs to be improved though, to make it more inline with the syntax of other HTTP headers.) Comments? Suggestions? [...] See ya -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ From kmagdsick at limewire.com Thu Feb 16 15:41:24 2006 From: kmagdsick at limewire.com (Karl A. Magdsick) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <84ce626f0602152239t451de6b6of0b7d603a827b510@mail.gmail.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <43F20046.4040709@limewire.com> <84ce626f0602141549l2d994826g84b01eafb61da5bd@mail.gmail.com> <43F28B42.8070009@limewire.com> <84ce626f0602151439w2031da3cl966c0d0a08aefa67@mail.gmail.com> <84ce626f0602152239t451de6b6of0b7d603a827b510@mail.gmail.com> Message-ID: <43F49D24.60209@limewire.com> Charles Iliya Krempeaux wrote: [snip] >I suppose if the "Link" header is being used to return information, >this could be generalized to make this "conditional get" be a way to >probe for "Link" data. > >So, instead of using "X-If-No-Alternate" it could be "X-If-No-Link" >(and then specify what relation or other meta data you want). > >For example, I could get a blogs RSS feed with: > > X-If-No-Link: rel="alternate"; type="application/rss+xml" > >Although the exact notation may need to be different since using the > > Why add an extra header in this case? It would be less intrusive to the webserver code if the client issues a HEAD request and checks for Link headers, rather than add a completely new X-If-No-Link header. -Karl From supercanadian at gmail.com Thu Feb 16 16:22:04 2006 From: supercanadian at gmail.com (Charles Iliya Krempeaux) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <43F49D24.60209@limewire.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <43F20046.4040709@limewire.com> <84ce626f0602141549l2d994826g84b01eafb61da5bd@mail.gmail.com> <43F28B42.8070009@limewire.com> <84ce626f0602151439w2031da3cl966c0d0a08aefa67@mail.gmail.com> <84ce626f0602152239t451de6b6of0b7d603a827b510@mail.gmail.com> <43F49D24.60209@limewire.com> Message-ID: <84ce626f0602160822q24b79acw79d5e09c23b1eb1f@mail.gmail.com> Hello Karl, On 2/16/06, Karl A. Magdsick wrote: > > Charles Iliya Krempeaux wrote: > [snip] > > >I suppose if the "Link" header is being used to return information, > >this could be generalized to make this "conditional get" be a way to > >probe for "Link" data. > > > >So, instead of using "X-If-No-Alternate" it could be "X-If-No-Link" > >(and then specify what relation or other meta data you want). > > > >For example, I could get a blogs RSS feed with: > > > > X-If-No-Link: rel="alternate"; type="application/rss+xml" > > > >Although the exact notation may need to be different since using the > > > > > Why add an extra header in this case? It would be less intrusive to the > webserver > code if the client issues a HEAD request and checks for Link headers, > rather > than add a completely new X-If-No-Link header. > > There's 2 reasons. The first is that it would be for the same reason that we have the other "conditional gets"... speed. The time it takes to "create" and "tear down" a TCP connection is significant. And, from a users point of view, will impact usability, in that it will make things seem slower (if we have to create 2 separate TCP connections for every request we want to do). If you could assume that ever server always had a "persistent connection" (or "keep alive" connection, or whatever you want to call them), then this wouldn't be an issue. But, you can't make that assumption. So that's why "conditional gets" are used. And the second (and probably more important reason) is that I'm making the assumption that the server would NOT always give all these Link headers. (And that some kind of "probing request", like via "X-If-No-Link", is required.) Comments? Suggestions? See ya -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20060216/9ad58d30/attachment.htm From gbildson at limepeer.com Thu Feb 16 16:56:37 2006 From: gbildson at limepeer.com (Greg Bildson) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> Message-ID: > -----Original Message----- > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On > Behalf Of Charles Iliya Krempeaux > > A "P2P conditional get" (sent by the client) would tell the server NOT > to send the requested document if there is an alternative way of > getting the document via a P2P technology supported by the client. > (The client would tell the server which P2P technologies it supports.) > (This "P2P conditional get" would work similar to the "If-None-Match" > and "If-Modified-Since" "conditional get" headers.). > > For example, it might be something like: > > X-If-No-Alternate: bittorrent, something-else, magnet > > First, this says that the client supports the protocols: "bittorrent", > "something-else", and "magnet". It also says it prefers "bittorrent" > first, then "something-else", and then "magnet". Are these hypothetical options for future web browsers? I take it that there would be no way to make a current browser actually communicate what the computer it's running on supports in this way right now? > > The server might then respond with an "HTTP 302" (Found) and then > redirect the user to a P2P version using a URI for the P2P technology. > This "redirect" would be done via the HTTP "Location" header. For > example: > > HTTP/1.1 302 Found > Location: magnet:?xt=urn:sha1:WRCIRZV5ZO56CWMNHFV4FRGNPWPPDVKT For what it's worth, a Location redirect will trigger magnet handling. We do something similar now with javascript. I wish magnets embedded in emails worked so easily. Thanks -greg From supercanadian at gmail.com Thu Feb 16 19:44:01 2006 From: supercanadian at gmail.com (Charles Iliya Krempeaux) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> Message-ID: <84ce626f0602161144h43b2b13clf42d351ee6389fed@mail.gmail.com> Hello Greg, On 2/16/06, Greg Bildson wrote: > > > > > -----Original Message----- > > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On > > Behalf Of Charles Iliya Krempeaux > > > > A "P2P conditional get" (sent by the client) would tell the server NOT > > to send the requested document if there is an alternative way of > > getting the document via a P2P technology supported by the client. > > (The client would tell the server which P2P technologies it supports.) > > (This "P2P conditional get" would work similar to the "If-None-Match" > > and "If-Modified-Since" "conditional get" headers.). > > > > For example, it might be something like: > > > > X-If-No-Alternate: bittorrent, something-else, magnet > > > > First, this says that the client supports the protocols: "bittorrent", > > "something-else", and "magnet". It also says it prefers "bittorrent" > > first, then "something-else", and then "magnet". > > > Are these hypothetical options for future web browsers? I take it that > there would be no way to make a current browser actually communicate what > the computer it's running on supports in this way right now? First let me say that my proposal actually changed a bit. As I discussed things with others on this mailing list (in other messages in this thread/tree of e-mails) I modified my proposal. Right now (in my proposal) the information that I previously had being returned with the "Location" header is now being returned with the "Link" header. (See section 19.6.2.4of RFC 2068 for more information on the "Link" header.) Basically though the "Link" header basically has the same semantics as the HTML element. Given that,... the header I was proposing is now called "X-If-No-Link". So, getting back to you question... Yes, these are ideas for future browsers. My project (that is motivating this work) is being built on the Mozilla application framework (using XULRunner). So I could imagine it very easily getting into Firefox via an extention. (And if people find it useful and it becomes popular I could see it getting into the mainstream web browsers. But as far as my work is concerned it doesn't have to.) Now, as far as current browsers are concerned, if you mandate that you don't want to install anything, then you could do some of this via an XmlHttpRequest (or a GM_xmlHttpRequest is you are using Greasemonkey). You could make a request and set all the "X-If-No-Link" headers that you want. And then when you get the response, you could check for the "Link" headers. See ya -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20060216/b633c8e8/attachment.html From supercanadian at gmail.com Fri Feb 17 08:09:13 2006 From: supercanadian at gmail.com (Charles Iliya Krempeaux) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <097AC0C1-CFE5-46A4-A49C-42691CA3F69D@locut.us> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <41879D03-2A20-4D3B-80EF-724C4A1B6629@locut.us> <84ce626f0602141638r1550bf0aq8fdd0e0b8114ba7a@mail.gmail.com> <097AC0C1-CFE5-46A4-A49C-42691CA3F69D@locut.us> Message-ID: <84ce626f0602170009m39272f99u5d2c9b98dbadcadb@mail.gmail.com> Hello Ian, Is there a way to get at dijjer's cached files besides going through it's local HTTP server? (I.e., with direct file I/O.) See ya On 2/14/06, Ian Clarke wrote: > > On 14 Feb 2006, at 16:38, Charles Iliya Krempeaux wrote: > > I've been actually considering using a local HTTP server to implement > > things. (Which is something you are doing too.) Although I thought > > some extra "sugar" was warranted, to help with (user) usabiliy and > > developer usability. > > We have added some sugar too. For example, we have javascript > available on our site that can detect whether Dijjer is present on > the user's machine and redirect the user to a download page if it isn't. > > > I skimmed through the stuff on dijjer.org, and I have a questions. > > Does digger pay attention to HTTP headers? And act basically as a web > > cache? > > Yes, we have attempted to stick as closely to the HTTP protocol as > possible. > > > For example, here's some (but not all) "rules" that I was planning on > > following.... If HTTP headers "say" that something should NOT be > > cached, then it should NOT be on the P2P network. > > Yes, we obey the various HTTP no-cache headers. We consider it > important for Dijjer to act as much like a HTTP cache as possible so > that we are afforded the protection the DMCA gives to caches for > copyright infringement. > > > Stuff on the P2P > > network should be removed after the date for the "Expires" header > > passes? > > We don't obey "Expires" explicitly, but since Dijjer pings the > original web server each-time something is downloaded to ensure that > the file is still there and hasn't changed, if it is removed from the > web server, it will effectively be removed from Dijjer at the same > instant. > > > Things from non-HTTP GET methods should NOT be put onto the > > P2P network? Etc. > > We attempt to be smart about our interaction with HTTP, if there are > shortcomings in this regard they will be easy to fix. > > Ian. > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20060217/2d2280b9/attachment.htm From ian at locut.us Fri Feb 17 18:33:44 2006 From: ian at locut.us (Ian Clarke) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <84ce626f0602170009m39272f99u5d2c9b98dbadcadb@mail.gmail.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <41879D03-2A20-4D3B-80EF-724C4A1B6629@locut.us> <84ce626f0602141638r1550bf0aq8fdd0e0b8114ba7a@mail.gmail.com> <097AC0C1-CFE5-46A4-A49C-42691CA3F69D@locut.us> <84ce626f0602170009m39272f99u5d2c9b98dbadcadb@mail.gmail.com> Message-ID: <0772B130-ABD1-4971-9E65-76EE0CB5531E@locut.us> No, Dijjer peers don't cache entire files, rather they just cache 256k blocks of files, and these are stored in a single binary file. Typically any given Dijjer peer would probably only be caching one block from any given file. Can you elaborate on why you might want to do this? Ian. On 17 Feb 2006, at 00:09, Charles Iliya Krempeaux wrote: > Hello Ian, > > Is there a way to get at dijjer's cached files besides going > through it's local HTTP server? (I.e., with direct file I/O.) > > > See ya > > On 2/14/06, Ian Clarke wrote: > On 14 Feb 2006, at 16:38, Charles Iliya Krempeaux wrote: > > I've been actually considering using a local HTTP server to > implement > > things. (Which is something you are doing too.) Although I thought > > some extra "sugar" was warranted, to help with (user) usabiliy and > > developer usability. > > We have added some sugar too. For example, we have javascript > available on our site that can detect whether Dijjer is present on > the user's machine and redirect the user to a download page if it > isn't. > > > I skimmed through the stuff on dijjer.org, and I have a questions. > > Does digger pay attention to HTTP headers? And act basically as > a web > > cache? > > Yes, we have attempted to stick as closely to the HTTP protocol as > possible. > > > For example, here's some (but not all) "rules" that I was > planning on > > following.... If HTTP headers "say" that something should NOT be > > cached, then it should NOT be on the P2P network. > > Yes, we obey the various HTTP no-cache headers. We consider it > important for Dijjer to act as much like a HTTP cache as possible so > that we are afforded the protection the DMCA gives to caches for > copyright infringement. > > > Stuff on the P2P > > network should be removed after the date for the "Expires" header > > passes? > > We don't obey "Expires" explicitly, but since Dijjer pings the > original web server each-time something is downloaded to ensure that > the file is still there and hasn't changed, if it is removed from the > web server, it will effectively be removed from Dijjer at the same > instant. > > > Things from non-HTTP GET methods should NOT be put onto the > > P2P network? Etc. > > We attempt to be smart about our interaction with HTTP, if there are > shortcomings in this regard they will be easy to fix. > > Ian. > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > > > -- > Charles Iliya Krempeaux, B.Sc. > > charles @ reptile.ca > supercanadian @ gmail.com > > developer weblog: http://ChangeLog.ca/ > ______________________________________________________________________ > _____ > Make Television http:// > maketelevision.com/ > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20060217/81b24c73/attachment.html From supercanadian at gmail.com Sat Feb 18 00:17:40 2006 From: supercanadian at gmail.com (Charles Iliya Krempeaux) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <0772B130-ABD1-4971-9E65-76EE0CB5531E@locut.us> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <41879D03-2A20-4D3B-80EF-724C4A1B6629@locut.us> <84ce626f0602141638r1550bf0aq8fdd0e0b8114ba7a@mail.gmail.com> <097AC0C1-CFE5-46A4-A49C-42691CA3F69D@locut.us> <84ce626f0602170009m39272f99u5d2c9b98dbadcadb@mail.gmail.com> <0772B130-ABD1-4971-9E65-76EE0CB5531E@locut.us> Message-ID: <84ce626f0602171617v64562f7fm2545e6e8ec5b22d7@mail.gmail.com> Hello Ian, I was thinking about how to integrate digger with my "P2P conditional get" idea. The steps when making certain kinds of requests for a URL (like for RSS/Atom enclosures) would roughtly be llike the followig. (Although configuration options would allow you to change this.) 1. Look in "cache" for URL. If in there, then get "ETag" or "Last-Modified" value. 2. Make an HTTP request on the server. Send a "If-None-Match" or "If-Modified-Since" "conditional get" if found "ETag" or "Last-Modified" from previous step. Also, send a "X-If-No-Link" for P2P protocols we understand. 3. If the server sends back a "304 Not Modified" then we have the up to date file. (So we're done. Else....) If the server sends back a "204 No Content" then get the "Link" headers (the server returned) and shunt over to the "preferred" P2P network. Else get dijjer to get it. Step #1 is where I wanted to look into dijjer's cache. (Step #2 is in there so that the publisher can provide "sanctioned" ways of getting the file off of P2P networks.) P.S. I don't think that the mailing list is sending all the messages/replies I've sent. Although the messages are in the mailing list archive, mailman -- the mailing list software -- sent 3 messages to me says it wasn't able to send (3) messages. See ya On 2/17/06, Ian Clarke wrote: > > No, Dijjer peers don't cache entire files, rather they just cache 256k > blocks of files, and these are stored in a single binary file. Typically > any given Dijjer peer would probably only be caching one block from any > given file. > > Can you elaborate on why you might want to do this? > > Ian. > > On 17 Feb 2006, at 00:09, Charles Iliya Krempeaux wrote: > > Hello Ian, > > Is there a way to get at dijjer's cached files besides going through it's > local HTTP server? (I.e., with direct file I/O.) > > > See ya > > On 2/14/06, Ian Clarke wrote: > > > > On 14 Feb 2006, at 16:38, Charles Iliya Krempeaux wrote: > > > I've been actually considering using a local HTTP server to implement > > > things. (Which is something you are doing too.) Although I thought > > > some extra "sugar" was warranted, to help with (user) usabiliy and > > > developer usability. > > > > We have added some sugar too. For example, we have javascript > > available on our site that can detect whether Dijjer is present on > > the user's machine and redirect the user to a download page if it isn't. > > > > > > > I skimmed through the stuff on dijjer.org, and I have a questions. > > > Does digger pay attention to HTTP headers? And act basically as a web > > > cache? > > > > Yes, we have attempted to stick as closely to the HTTP protocol as > > possible. > > > > > For example, here's some (but not all) "rules" that I was planning on > > > following.... If HTTP headers "say" that something should NOT be > > > cached, then it should NOT be on the P2P network. > > > > Yes, we obey the various HTTP no-cache headers. We consider it > > important for Dijjer to act as much like a HTTP cache as possible so > > that we are afforded the protection the DMCA gives to caches for > > copyright infringement. > > > > > Stuff on the P2P > > > network should be removed after the date for the "Expires" header > > > passes? > > > > We don't obey "Expires" explicitly, but since Dijjer pings the > > original web server each-time something is downloaded to ensure that > > the file is still there and hasn't changed, if it is removed from the > > web server, it will effectively be removed from Dijjer at the same > > instant. > > > > > Things from non-HTTP GET methods should NOT be put onto the > > > P2P network? Etc. > > > > We attempt to be smart about our interaction with HTTP, if there are > > shortcomings in this regard they will be easy to fix. > > > > -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20060217/ecb844bb/attachment.htm From jonetony at gmail.com Mon Feb 20 18:36:41 2006 From: jonetony at gmail.com (Jonetony) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Adding simple and effective trust measurements to F2F networks Message-ID: Hi! Here is a new paper in the GNUnet forum about "Adding simple and effective trust measurements to F2F networks" https://gnunet.org/drupal/?q=node/163 Further comments are welcome. I'd like also some advice in order to code what is inside the paper, especially what would be the easiest F2F P2P software to modify, and which source files in it I should modify. Thanks. Introduction: The Friend-to-Friend P2P networks already provide a simple way for friends to connect to each other securely. By adding simple features like document prices, refunds, and personalised "goodwill" credits between peers, we will show that we can bring trust management to an existing F2F network. http://gnunet.org/drupal/?q=node/163 From Raphael_Manfredi at pobox.com Tue Feb 21 10:22:23 2006 From: Raphael_Manfredi at pobox.com (Raphael Manfredi) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Keep Alive and network performance In-Reply-To: Your article in ml.p2p.hackers of Thu, 9 Feb 2006 20:15:55 +0100 Message-ID: <3128.1140517343@nice> You wrote in ml.p2p.hackers: :In Solipsis, we also use a unique message for both "heartbeat" (confirm :aliveness) and "keepalive" (maintain NAT hole-punching). A french saying is "One :stone, two hits !" :-) La traduction anglaise de la locution "faire d'une pierre deux coups" est "catch two birds with the same stone". La meme a l'envers, comme d'habitude avec les anglo-saxons. Raphael From Raphael_Manfredi at pobox.com Tue Feb 21 10:28:43 2006 From: Raphael_Manfredi at pobox.com (Raphael Manfredi) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Keep Alive and network performance In-Reply-To: <3128.1140517343@nice> References: <3128.1140517343@nice> Message-ID: My apologies. This was meant to be a private reply, but someone must have messed up with Reply-To headers, and the message made it back to the list. Raphael ::In Solipsis, we also use a unique message for both "heartbeat" (confirm ::aliveness) and "keepalive" (maintain NAT hole-punching). A french saying is "One ::stone, two hits !" :-) : :La traduction anglaise de la locution "faire d'une pierre deux coups" est :"catch two birds with the same stone". La meme a l'envers, comme d'habitude :avec les anglo-saxons. : :Raphael :_______________________________________________ :p2p-hackers mailing list :p2p-hackers@zgp.org :http://zgp.org/mailman/listinfo/p2p-hackers :_______________________________________________ :Here is a web page listing P2P Conferences: :http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences : : From matthew at allpeers.com Tue Feb 21 15:58:05 2006 From: matthew at allpeers.com (Matthew Gertner) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Keep Alive and network performance In-Reply-To: References: <3128.1140517343@nice> Message-ID: <43FB388D.3090701@allpeers.com> For the record, I believe "kill two birds with one stone" is the standard version of the English idiom. Raphael Manfredi wrote: >My apologies. This was meant to be a private reply, but someone must >have messed up with Reply-To headers, and the message made it back to the list. > >Raphael > >::In Solipsis, we also use a unique message for both "heartbeat" (confirm >::aliveness) and "keepalive" (maintain NAT hole-punching). A french saying is "One >::stone, two hits !" :-) >: >:La traduction anglaise de la locution "faire d'une pierre deux coups" est >:"catch two birds with the same stone". La meme a l'envers, comme d'habitude >:avec les anglo-saxons. >: >:Raphael >:_______________________________________________ >:p2p-hackers mailing list >:p2p-hackers@zgp.org >:http://zgp.org/mailman/listinfo/p2p-hackers >:_______________________________________________ >:Here is a web page listing P2P Conferences: >:http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences >: >: > > >_______________________________________________ >p2p-hackers mailing list >p2p-hackers@zgp.org >http://zgp.org/mailman/listinfo/p2p-hackers >_______________________________________________ >Here is a web page listing P2P Conferences: >http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > > > From slavitch at gmail.com Tue Feb 21 16:20:47 2006 From: slavitch at gmail.com (Michael Slavitch) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Keep Alive and network performance In-Reply-To: <43FB388D.3090701@allpeers.com> References: <3128.1140517343@nice> <43FB388D.3090701@allpeers.com> Message-ID: <1fc32c8f0602210820x35611c34h@mail.gmail.com> No need to apologize, Raphael. It adds to the context. On 21/02/06, Matthew Gertner wrote: > > For the record, I believe "kill two birds with one stone" is the > standard version of the English idiom. > > Raphael Manfredi wrote: > > >My apologies. This was meant to be a private reply, but someone must > >have messed up with Reply-To headers, and the message made it back to the > list. > > > >Raphael > > > >::In Solipsis, we also use a unique message for both "heartbeat" (confirm > >::aliveness) and "keepalive" (maintain NAT hole-punching). A french > saying is "One > >::stone, two hits !" :-) > >: > >:La traduction anglaise de la locution "faire d'une pierre deux coups" > est > >:"catch two birds with the same stone". La meme a l'envers, comme > d'habitude > >:avec les anglo-saxons. > >: > >:Raphael > >:_______________________________________________ > >:p2p-hackers mailing list > >:p2p-hackers@zgp.org > >:http://zgp.org/mailman/listinfo/p2p-hackers > >:_______________________________________________ > >:Here is a web page listing P2P Conferences: > >:http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > >: > >: > > > > > >_______________________________________________ > >p2p-hackers mailing list > >p2p-hackers@zgp.org > >http://zgp.org/mailman/listinfo/p2p-hackers > >_______________________________________________ > >Here is a web page listing P2P Conferences: > >http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > > > > > > > > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > -- Michael Slavitch Ottawa, Ontario Canada -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20060221/be00befd/attachment.html From googaya at gmail.com Wed Feb 22 19:49:43 2006 From: googaya at gmail.com (googaya) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Overlay Weaver: An Overlay Construction Toolkit Message-ID: <43FCC057.6060207@gmail.com> I read with much enthusiam your report about "Overlay Construction Toolkit" My question can the toolkit be use to create a P2P SIP network or do need to use CHORD / BAMBOO / KADEMLIA ? I am trying to add DHT to OpenSER , any tips would be appreciated. Thanks From mac.ibook at gmail.com Thu Feb 23 09:32:19 2006 From: mac.ibook at gmail.com (Yifei Zeng) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Control network file transfer rate Message-ID: Hi all. I am writing a peer-to-peer file sharing application in Java. Could someone help me with how to control the download/upload rate of the file transfer (practically) through internet? just like the BitTorrent clients (Bitcomet, Azureus), most of them have the function for the user to set the download / upload rate limit. How should I do in java to achieve this? Can someone please give me some ideas or hints? I am currently using non-blocking IO and socket and for each file transfer between two peers there is a non-blocking socket associated with it. And my application only has single thread to achieve all socket read / write request. Thanks Yifei From prakash_sabapathy at hotmail.com Thu Feb 23 10:17:49 2006 From: prakash_sabapathy at hotmail.com (Prakash Sabapathy) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] OmilyX: Secure VoIP Message-ID: After looking around the web without success for a secure VoIP application, I decided to build one: OmilyX. For the sake of brevity, the key features of OmilyX are: - Perfect forward secrecy (Diffie-Hellman key exchange for every call) - Anonymity (contact details, e.g. email address etc., not required) - NAT traversal (UDP hole punching) - Low bandwidth (6kb/s total including all overheads) - Great sound quality (Speex) - Works with PocketPC and Windows To try the beta release and for more details visit http://www.omilyx.com Prakash From coderman at gmail.com Thu Feb 23 12:32:31 2006 From: coderman at gmail.com (coderman) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] OmilyX: Secure VoIP In-Reply-To: References: Message-ID: <4ef5fec60602230432o6d720cc4n4bbea301e610998@mail.gmail.com> On 2/23/06, Prakash Sabapathy wrote: > After looking around the web without success for a secure VoIP application, > I decided to build one: OmilyX. no sources? no security... looks interesting otherwise though. From ramo at skuff-band.de Thu Feb 23 13:58:51 2006 From: ramo at skuff-band.de (ramo@skuff-band.de) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Identify "defective" nodes Message-ID: <2742.84.169.87.19.1140703131.squirrel@webmail-panel.de> Hi List, i'm a student of Computer Science and trying to search for some good Help's but don't know where to start. So what i want to know are there some techniques for finding defective peers in a , let me say small p2p network with 50 peers. With "defective" i mean , that a peer distribute malicious manipulated services in a service-orientated network, so that the processes on that peers work wrong and is there a technique to eliminate automatically such a peer from the network, i mean something like: put it in a dmz so that it can't do actual bad things. Is there any literature about that or does anyone of the Listpeople know some more about that part? I would be thankful for any Information. Thank you for your thoughts. Ramo From googaya at gmail.com Thu Feb 23 14:19:22 2006 From: googaya at gmail.com (googaya) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] OmilyX: Secure VoIP In-Reply-To: References: Message-ID: <43FDC46A.40600@gmail.com> Prakash Sabapathy wrote: >After looking around the web without success for a secure VoIP application, >I decided to build one: OmilyX. > >For the sake of brevity, the key features of OmilyX are: >- Perfect forward secrecy (Diffie-Hellman key exchange for every call) >- Anonymity (contact details, e.g. email address etc., not required) >- NAT traversal (UDP hole punching) >- Low bandwidth (6kb/s total including all overheads) >- Great sound quality (Speex) >- Works with PocketPC and Windows > >To try the beta release and for more details visit http://www.omilyx.com > >Prakash > >_______________________________________________ >p2p-hackers mailing list >p2p-hackers@zgp.org >http://zgp.org/mailman/listinfo/p2p-hackers >_______________________________________________ >Here is a web page listing P2P Conferences: >http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > > Did not read if it was SIP based. is it? -E From yfttyp2p at gmail.com Thu Feb 23 15:05:40 2006 From: yfttyp2p at gmail.com (y f) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Control network file transfer rate In-Reply-To: References: Message-ID: <734557130602230705g642d819ej6bf8fa02cbf20330@mail.gmail.com> is this the simplist way: while (1) { old_time = gettimeofday( ); select(fds); if (cur_time = gettimeofday( ) - old_time > 1 sec) destroy ignore_list; for (all fd whose bit is set) { if (fd in ignore_list) mark it and contiue; do read/write; add the data_size to this fd's data_total_within_this_second; /* which is hashed link list */ if (data_total_within_this_second > max_data_within_a_second) add it to ignore_list; } } and with the tcp semantic, if we slow down the receiver speed, then the sender's sending speed is slowed down accordingly. of course it will eat much memory as we get a full socket receiving-queue at the beginning time. I hope I say some right and can-do words. or there is a simple ioclt(sd, ...), or fcntl(sd, ...) can tell the sender what speed we like ? Wish you a nice day ! On 2/23/06, Yifei Zeng wrote: > > Hi all. > > I am writing a peer-to-peer file sharing application in Java. > > Could someone help me with how to control the download/upload rate of > the file transfer (practically) through internet? > > just like the BitTorrent clients (Bitcomet, Azureus), most of them > have the function for the user to set the download / upload rate limit. > > How should I do in java to achieve this? Can someone please give me > some ideas or hints? > > I am currently using non-blocking IO and socket and for each file > transfer between two peers there is a non-blocking socket associated > with it. And my application only has single thread to achieve all > socket read / write request. > > Thanks > > Yifei > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20060223/43de8c2e/attachment.htm From jonetony at gmail.com Thu Feb 23 15:30:34 2006 From: jonetony at gmail.com (Jonetony) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: <2742.84.169.87.19.1140703131.squirrel@webmail-panel.de> References: <2742.84.169.87.19.1140703131.squirrel@webmail-panel.de> Message-ID: On 2/23/06, ramo@skuff-band.de wrote: >So what i want to know are there some techniques for finding defective peers try this website about trust in p2p: http://p2ptrust.org/. I already posted a paper about trust in the p2p-hackers list: http://gnunet.org/drupal/?q=node/163 "Adding simple and effective trust measurements to F2F networks" From jonetony at gmail.com Thu Feb 23 15:56:05 2006 From: jonetony at gmail.com (Jonetony) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: References: <2742.84.169.87.19.1140703131.squirrel@webmail-panel.de> Message-ID: I forgot this workshop with lots of papers about trust metrics in p2p: http://p2p.ingce.unibo.it/2005/ From gbildson at limepeer.com Thu Feb 23 16:28:21 2006 From: gbildson at limepeer.com (Greg Bildson) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Control network file transfer rate In-Reply-To: Message-ID: You might want to look at our BandwidthThrottle, ThrottledOutputStream classes and the use of BandwidthThrottle elsewhere (HTTPDownloader, NormalUploadState, etc). We built off of some earlier FreeNet techniques. There is some added complexity along the way but I believe we handle things elegantly overall. http://tinyurl.com/l86ch http://tinyurl.com/qluxj ... Thanks -greg > -----Original Message----- > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On > Behalf Of Yifei Zeng > Sent: Thursday, February 23, 2006 4:32 AM > To: p2p-hackers@zgp.org > Subject: [p2p-hackers] Control network file transfer rate > > > Hi all. > > I am writing a peer-to-peer file sharing application in Java. > > Could someone help me with how to control the download/upload rate of > the file transfer (practically) through internet? > > just like the BitTorrent clients (Bitcomet, Azureus), most of them > have the function for the user to set the download / upload rate limit. > > How should I do in java to achieve this? Can someone please give me > some ideas or hints? > > I am currently using non-blocking IO and socket and for each file > transfer between two peers there is a non-blocking socket associated > with it. And my application only has single thread to achieve all > socket read / write request. > > Thanks > > Yifei > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From prakash_sabapathy at hotmail.com Thu Feb 23 16:32:53 2006 From: prakash_sabapathy at hotmail.com (Prakash Sabapathy) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] OmilyX: Secure VoIP References: <43FDC46A.40600@gmail.com> Message-ID: ----- Original Message ----- From: "googaya" To: "Peer-to-peer development." Sent: Thursday, February 23, 2006 2:19 PM Subject: Re: [p2p-hackers] OmilyX: Secure VoIP > > > Did not read if it was SIP based. is it? > -E It does not use SIP as I wanted to get the bandwidth down to the level of a 56k dialup modem and therefore decided to start with a custom message protocol. From Raphael_Manfredi at pobox.com Thu Feb 23 16:53:25 2006 From: Raphael_Manfredi at pobox.com (Raphael Manfredi) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Control network file transfer rate In-Reply-To: References: Message-ID: Quoting Greg Bildson from ml.p2p.hackers: :You might want to look at our BandwidthThrottle, ThrottledOutputStream :classes and the use of BandwidthThrottle elsewhere (HTTPDownloader, :NormalUploadState, etc). We built off of some earlier FreeNet techniques. :There is some added complexity along the way but I believe we handle things :elegantly overall. Doing b/w allocation on a per-stream basis is sub-optimal and may lead to non-full utilization of the configured total b/w. If you can read C, thre's a more complex bandwidth scheduler at: http://tinyurl.com/q3toe Start with bsched_heartbeat(). The idea is the scheduler knows the whole set of file descriptors that want to emit. But not all can emit at some time (kernel buffers full or flow-controlled), so the b/w is more evenly distributed among the other files. Also, from the application layer, you can control how much you write, but you have really no control on what TCP will do (retransmits, ACKs, etc...). So all you can do is try to emit *on average* what the user has configured. Hence the use of EMA in the code trying to balance things to absorb peaks. Raphael From sberlin at gmail.com Thu Feb 23 17:10:00 2006 From: sberlin at gmail.com (Sam Berlin) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Control network file transfer rate In-Reply-To: References: Message-ID: <19196d860602230910h5ff88541lf46acbda697ff07c@mail.gmail.com> You may also want to take a look at http://tinyurl.com/la4aj . It's similar to Raphael's global bandwidth scheduler, but implemented in Java and designed for non-blocking I/O. LimeWire uses it for Gnutella upload traffic bandwidth throttling (whereas the classes Greg described earlier are used for file upload/download bandwidth throttling, since they have not switched to NIO yet). Sam On 23 Feb 2006 16:53:25 GMT, Raphael Manfredi wrote: > Quoting Greg Bildson from ml.p2p.hackers: > :You might want to look at our BandwidthThrottle, ThrottledOutputStream > :classes and the use of BandwidthThrottle elsewhere (HTTPDownloader, > :NormalUploadState, etc). We built off of some earlier FreeNet techniques. > :There is some added complexity along the way but I believe we handle things > :elegantly overall. > > Doing b/w allocation on a per-stream basis is sub-optimal and may lead > to non-full utilization of the configured total b/w. > > If you can read C, thre's a more complex bandwidth scheduler at: > > http://tinyurl.com/q3toe > > Start with bsched_heartbeat(). The idea is the scheduler knows > the whole set of file descriptors that want to emit. But not all > can emit at some time (kernel buffers full or flow-controlled), so > the b/w is more evenly distributed among the other files. > > Also, from the application layer, you can control how much you > write, but you have really no control on what TCP will do (retransmits, > ACKs, etc...). So all you can do is try to emit *on average* what > the user has configured. > > Hence the use of EMA in the code trying to balance things to absorb > peaks. > > Raphael > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From michaelhartm at gmail.com Thu Feb 23 17:19:34 2006 From: michaelhartm at gmail.com (Michael Hartmann) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Symmetric Replication for Chord Message-ID: Anyone actually tried implementing the replication scheme suggested by this paper: Symmetric Replication for Structured Peer-to-Peer Systems, A. Ghodsi, L-O Alima, S. Haridi, DBISP2P2005, 2005. http://dks.sics.se/pub/replication.pdf I am trying to implement it for Chord, it seems better than the successor-list approach, but I am wondering if it is possible to make it work for replication degrees that do not divide the size of the key space? Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20060223/d546d547/attachment.html From dbarrett at quinthar.com Thu Feb 23 17:26:35 2006 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] OmilyX: Secure VoIP In-Reply-To: Message-ID: <20060223172649.2838D3FD3B@capsicum.zgp.org> > ----- Original Message ----- > From: "googaya" > Subject: Re: [p2p-hackers] OmilyX: Secure VoIP > > > Did not read if it was SIP based. is it? > > -E > > It does not use SIP as I wanted to get the bandwidth down to the level of > a > 56k dialup modem and therefore decided to start with a custom message > protocol. Wise move -- I modeled after SIP for iGlance (www.iglance.com) and largely regretted it. You might also consider IAX2 for a tight binary protocol that still has wide (and increasing) adoption between SIP phones, both soft and hard. -david From alenlpeacock at gmail.com Thu Feb 23 17:43:44 2006 From: alenlpeacock at gmail.com (Alen Peacock) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Identify "defective" nodes In-Reply-To: <2742.84.169.87.19.1140703131.squirrel@webmail-panel.de> References: <2742.84.169.87.19.1140703131.squirrel@webmail-panel.de> Message-ID: On 2/23/06, ramo@skuff-band.de wrote: > > > Is there any literature about that or does anyone of the Listpeople know > some more about that part? > Search for 'trust' or 'reputation' systems. There's quite a bit of published work in these areas. A couple of my favorite papers: An Excess-based Economic Model for Resource Allocation in Peer-to-Peer Networks: http://www.freehaven.net/anonbib/cache/ebe2003.ps Samsara: Honor Among Theives in a Peer-to-Peer System: http://www.cs.rochester.edu/sosp2003/papers/p135-cox.pdf Another short list of papers at: http://www.cs.umd.edu/~slee/p2p-trust-paper-list.html Alen -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20060223/7fc2f35e/attachment.htm From dbarrett at quinthar.com Thu Feb 23 17:55:35 2006 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Control network file transfer rate In-Reply-To: Message-ID: <20060223175552.EB0F03FD2C@capsicum.zgp.org> If you're using UDP I'd recommend just modeling yourself after TCP as it works pretty darn well (and ensures your application is friendly with other TCP applications out there). Though C++, the relevant class in iGlance is: http://svn.iglance.com/svn/trunk/iglance/client/GTCP.h http://svn.iglance.com/svn/trunk/iglance/client/GTCP.cpp And a sample that uses it to do congestion-controlled, non-lossy file transfers using a custom RTP profile: http://svn.iglance.com/svn/trunk/iglance/client/GFileService.h http://svn.iglance.com/svn/trunk/iglance/client/GFileService.cpp This'll figure out the fastest you can possibly send to saturate an unknown link speed. Granted, this isn't quite what you're asking for (you want to achieve a fixed, predefined throughput), but it's something you'll need if the specified "maximum throughput" is greater than the actual throughput. Furthermore, to limit at a maximum throughput you might enforce a ceiling on 'cwnd' (the maximum number of unacknowledged packets you can have "in flight" on the network at one point in time). I'm not precisely sure how that'd work (congestion control full of notoriously tricky bugs) but it might be something like: MAX_CWND = (MAX_BYTES_PER_S * RTT_UNITS) / (PACKET_SIZE * latestRTT) So if your RTT (round trip time) is measured in microseconds, you're targeting 1KBps, you have 1KB packet size, and a 1000ms round-trip latency: MAX_CWND = (1024 * 1000*1000) / (1024 * 1000*1000) = 1 In other words, you can send at most one 1KB packet before waiting for acknowledgement. And because you can expect your acknowledgement in one second, you can send another 1KB packet in another second, for a total throughput of 1KB/s -- aka 1KBps. Or something like that. -david > -----Original Message----- > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On > Behalf Of Yifei Zeng > Sent: Thursday, February 23, 2006 1:32 AM > To: p2p-hackers@zgp.org > Subject: [p2p-hackers] Control network file transfer rate > > Hi all. > > I am writing a peer-to-peer file sharing application in Java. > > Could someone help me with how to control the download/upload rate of > the file transfer (practically) through internet? > > just like the BitTorrent clients (Bitcomet, Azureus), most of them > have the function for the user to set the download / upload rate limit. > > How should I do in java to achieve this? Can someone please give me > some ideas or hints? > > I am currently using non-blocking IO and socket and for each file > transfer between two peers there is a non-blocking socket associated > with it. And my application only has single thread to achieve all > socket read / write request. > > Thanks > > Yifei > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From dcarboni at gmail.com Thu Feb 23 18:12:39 2006 From: dcarboni at gmail.com (Davide "dada" Carboni) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] On the number of users in P2P networks Message-ID: <71b79fa90602231012r3d86a444p81286ecb35085161@mail.gmail.com> Hi, I'm looking for a to-date survey on P2P networks. I'd like to know how many nodes are connected to eDonkey2k, Gnutella, etc. I've found this one (1) which is quite biased on the amount of traffic and gives Bittorrent as the winner. I would object that Bittorrent is not a single network but there is a BT P2P network for any content published. About the other networks, I see a dominance of ed2k over gnutella and fasttrack. Even here I've read the ed2k protocol specs (2) and found that there is no server-to-server communication so de facto each server is a sort of old-good-napster network and nodes connected to different servers cannot be considered part of the same community (unless these results come from other mechanisms that are not part of the ed2k protocol like kademlia, or emule extensions that I'm not aware) (1) http://www.cachelogic.com/research/2005_slide01.php (2) http://hydranode.com/docs/ed2k/ed2kproto.php -- Prima il 30% poi Barbolomeo. -- http://people.crs4.it/dcarboni From mgp at ucla.edu Thu Feb 23 19:05:51 2006 From: mgp at ucla.edu (Michael Parker) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Control network file transfer rate In-Reply-To: References: Message-ID: <20060223110551.d9pq5rnwg0ko08w4@mail.ucla.edu> IIRC, a long time ago while looking through the Azureus source code I saw a token bucket implementation to control the upload rate. See here: http://en.wikipedia.org/wiki/Token_bucket Although if you're just interested in ready-to-use source code, the previous replies are much more helpful :) - Mike Parker Quoting Yifei Zeng : > Hi all. > > I am writing a peer-to-peer file sharing application in Java. > > Could someone help me with how to control the download/upload rate of > the file transfer (practically) through internet? > > just like the BitTorrent clients (Bitcomet, Azureus), most of them > have the function for the user to set the download / upload rate > limit. > > How should I do in java to achieve this? Can someone please give me > some ideas or hints? > > I am currently using non-blocking IO and socket and for each file > transfer between two peers there is a non-blocking socket associated > with it. And my application only has single thread to achieve all > socket read / write request. > > Thanks > > Yifei > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From googaya at gmail.com Thu Feb 23 19:52:22 2006 From: googaya at gmail.com (googaya) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] OmilyX: Secure VoIP In-Reply-To: <20060223172649.2838D3FD3B@capsicum.zgp.org> References: <20060223172649.2838D3FD3B@capsicum.zgp.org> Message-ID: <43FE1276.1000809@gmail.com> David Barrett wrote: >>----- Original Message ----- >>From: "googaya" >>Subject: Re: [p2p-hackers] OmilyX: Secure VoIP >> >> >> >>>Did not read if it was SIP based. is it? >>>-E >>> >>> >>It does not use SIP as I wanted to get the bandwidth down to the level of >>a >>56k dialup modem and therefore decided to start with a custom message >>protocol. >> >> > >Wise move -- I modeled after SIP for iGlance (www.iglance.com) and largely >regretted it. You might also consider IAX2 for a tight binary protocol that >still has wide (and increasing) adoption between SIP phones, both soft and >hard. > >-david > > > No argument there, IAX2 is certainly more efficient. Do you have plans in adding IAX2 in the near future? I personally looking for a CHORD / KADEMLIA/BAMBOO type overlay network with SIP support so I can use it with OpenSER. Regards -E From dbarrett at quinthar.com Thu Feb 23 20:05:05 2006 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] OmilyX: Secure VoIP In-Reply-To: <43FE1276.1000809@gmail.com> Message-ID: <20060223200516.C3C7B3FD23@capsicum.zgp.org> > -----Original Message----- > From: googaya > > No argument there, IAX2 is certainly more efficient. Do you have plans > in adding IAX2 in the near future? > I personally looking for a CHORD / KADEMLIA/BAMBOO type overlay network > with SIP support so I can use it with OpenSER. > Regards > -E I'm sticking with SIP for now. If you're looking for SIP integrated with a DHT, I highly recommend participating in the P2P-SIP working group. -david From ap at hamachi.cc Thu Feb 23 22:38:28 2006 From: ap at hamachi.cc (Alex Pankratov) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] OmilyX: Secure VoIP In-Reply-To: <4ef5fec60602230432o6d720cc4n4bbea301e610998@mail.gmail.com> References: <4ef5fec60602230432o6d720cc4n4bbea301e610998@mail.gmail.com> Message-ID: <43FE3964.4040207@hamachi.cc> coderman wrote: > On 2/23/06, Prakash Sabapathy wrote: > >>After looking around the web without success for a secure VoIP application, >>I decided to build one: OmilyX. > > > no sources? no security... Sources are not needed. Security spec is. From coderman at gmail.com Thu Feb 23 23:15:56 2006 From: coderman at gmail.com (coderman) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] OmilyX: Secure VoIP In-Reply-To: <43FE3964.4040207@hamachi.cc> References: <4ef5fec60602230432o6d720cc4n4bbea301e610998@mail.gmail.com> <43FE3964.4040207@hamachi.cc> Message-ID: <4ef5fec60602231515p6858f38erc0e96a3c01c789dd@mail.gmail.com> On 2/23/06, Alex Pankratov wrote: > ... > Sources are not needed. Security spec is. i have a hard time accepting code quality, adherence to spec, and implementation correctness on faith when dealing with "secure software" provided by strangers in binary form only. do you see the problem here? From ap at hamachi.cc Thu Feb 23 23:52:17 2006 From: ap at hamachi.cc (Alex Pankratov) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] OmilyX: Secure VoIP In-Reply-To: <4ef5fec60602231515p6858f38erc0e96a3c01c789dd@mail.gmail.com> References: <4ef5fec60602230432o6d720cc4n4bbea301e610998@mail.gmail.com> <43FE3964.4040207@hamachi.cc> <4ef5fec60602231515p6858f38erc0e96a3c01c789dd@mail.gmail.com> Message-ID: <43FE4AB1.2010204@hamachi.cc> coderman wrote: > On 2/23/06, Alex Pankratov wrote: > >>... >>Sources are not needed. Security spec is. > > > i have a hard time accepting code quality, adherence to spec, and > implementation correctness on faith when dealing with "secure > software" provided by strangers in binary form only. > > do you see the problem here? I see the problem in assuming that their binary distro will be assembled from the same sources they release to the public. If they are out to get users, they are perfectly able to do it with their full source open. I would however feel more confident in their binaries if I can verify that they adhere to the protocol spec. See my point ? From coderman at gmail.com Fri Feb 24 00:17:57 2006 From: coderman at gmail.com (coderman) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] OmilyX: Secure VoIP In-Reply-To: <43FE4AB1.2010204@hamachi.cc> References: <4ef5fec60602230432o6d720cc4n4bbea301e610998@mail.gmail.com> <43FE3964.4040207@hamachi.cc> <4ef5fec60602231515p6858f38erc0e96a3c01c789dd@mail.gmail.com> <43FE4AB1.2010204@hamachi.cc> Message-ID: <4ef5fec60602231617h3ae6f434x4af6970f51299196@mail.gmail.com> On 2/23/06, Alex Pankratov wrote: > ... > I would however feel more confident in their binaries if I can > verify that they adhere to the protocol spec. glad to know such verification is trivial with binary products. (i'd love to know how you trigger every possible error/boundary/edge condition that may lead to remote exploitation or unintentional disclosure in such protocols.) > See my point ? yes. you possess skills far beyond most. now if only i could take your word for it when you tell me $app follows spec precisely under all conditions with no exceptions... [i don't mean to be so flippant, but really, you have a steep hill to climb when proclaiming some closed source networked application a piece of "security software" with no evidence of the process or internals that comprise it] see https://buildsecurityin.us-cert.gov/portal/ for one example of the requisite best practices for "secure software", whether closed or open source. From ap at hamachi.cc Fri Feb 24 00:42:01 2006 From: ap at hamachi.cc (Alex Pankratov) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] OmilyX: Secure VoIP In-Reply-To: <4ef5fec60602231617h3ae6f434x4af6970f51299196@mail.gmail.com> References: <4ef5fec60602230432o6d720cc4n4bbea301e610998@mail.gmail.com> <43FE3964.4040207@hamachi.cc> <4ef5fec60602231515p6858f38erc0e96a3c01c789dd@mail.gmail.com> <43FE4AB1.2010204@hamachi.cc> <4ef5fec60602231617h3ae6f434x4af6970f51299196@mail.gmail.com> Message-ID: <43FE5659.3090708@hamachi.cc> coderman wrote: > On 2/23/06, Alex Pankratov wrote: > >>... >>I would however feel more confident in their binaries if I can >>verify that they adhere to the protocol spec. > > glad to know such verification is trivial with binary products. (i'd > love to know how you trigger every possible error/boundary/edge > condition that may lead to remote exploitation or unintentional > disclosure in such protocols.) You are quoting out of the context. Nowhere I said it was a trivial task nor was it the primary point. >>See my point ? > > yes. you possess skills far beyond most. now if only i could take > your word for it when you tell me $app follows spec precisely under > all conditions with no exceptions... We are talking about different things it seems. The binary might be buggy or it might be bugged. To me the former is bad, but latter is far worse. Having a source at hand does nothing to establish the trust in a binaries. > [i don't mean to be so flippant, but really, you have a steep hill to > climb when proclaiming some closed source networked application a > piece of "security software" with no evidence of the process or > internals that comprise it] Would Cisco IPsec client or SSH SecureCRT qualify as 'some closed source networked application' ? From coderman at gmail.com Fri Feb 24 01:10:37 2006 From: coderman at gmail.com (coderman) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] OmilyX: Secure VoIP In-Reply-To: <43FE5659.3090708@hamachi.cc> References: <4ef5fec60602230432o6d720cc4n4bbea301e610998@mail.gmail.com> <43FE3964.4040207@hamachi.cc> <4ef5fec60602231515p6858f38erc0e96a3c01c789dd@mail.gmail.com> <43FE4AB1.2010204@hamachi.cc> <4ef5fec60602231617h3ae6f434x4af6970f51299196@mail.gmail.com> <43FE5659.3090708@hamachi.cc> Message-ID: <4ef5fec60602231710i3e0f7346if1df115e9dc540e7@mail.gmail.com> On 2/23/06, Alex Pankratov wrote: > ... > We are talking about different things it seems. The binary > might be buggy or it might be bugged. To me the former is > bad, but latter is far worse. Having a source at hand does > nothing to establish the trust in a binaries. different things indeed. i agree that source does not imply security, see the recent bsd beacon debacle as evidence [1]. what i am saying is that source goes a long way to clearing the "low hanging fruit" that is extremely difficult to validate in binary applications and gives much more oversight with less effort. really bad code is easy to spot in source, but much more difficult in resulting binaries. if you are referring to a disconnect between source and binary (regarding the trust comment), see the source checksum in linux kernel modules feature as a way to bind these if you don't want to build from sources yourself. obviously you should always obtain binaries from someone you trust who has built from source in such a situation. > Would Cisco IPsec client or SSH SecureCRT qualify as 'some > closed source networked application' ? not in my book, but in others perhaps. remember cisco gate? not remotely exploitable? while this code is much better than most (again, due to the process of development that is visible, even if the sources are not widely so) it could be more reputable with visible sources providing additional oversight and making any security claims easier to verify. enjoy your security theater... 1. http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/net80211/ieee80211_ioctl.c.diff?r1=1.41&r2=1.42&f=h P.S. visible sources does not mean open source (as in freedom). you can retain tight ownership ala M$ shared source. see also: http://www.dwheeler.com/secure-programs/Secure-Programs-HOWTO/open-source-security.html "Is Open Source Good for Security?" From coderman at gmail.com Fri Feb 24 02:35:46 2006 From: coderman at gmail.com (coderman) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] OmilyX: Secure VoIP In-Reply-To: <4ef5fec60602231710i3e0f7346if1df115e9dc540e7@mail.gmail.com> References: <4ef5fec60602230432o6d720cc4n4bbea301e610998@mail.gmail.com> <43FE3964.4040207@hamachi.cc> <4ef5fec60602231515p6858f38erc0e96a3c01c789dd@mail.gmail.com> <43FE4AB1.2010204@hamachi.cc> <4ef5fec60602231617h3ae6f434x4af6970f51299196@mail.gmail.com> <43FE5659.3090708@hamachi.cc> <4ef5fec60602231710i3e0f7346if1df115e9dc540e7@mail.gmail.com> Message-ID: <4ef5fec60602231835p5955a74agc0e5408d958de90e@mail.gmail.com> On 2/23/06, coderman wrote: > [poor display of social communication skill] i apologize for the tone of this thread; this happens to be one of my hot buttons as too many people consider "adding encryption" the path to strong application security as this is really just presenting a false sense of it when the other requisite development and UI practices are ignored. thank you Prakash for sharing this despite my criticisms; an interesting implementation regardless. From enzomich at gmail.com Fri Feb 24 08:18:26 2006 From: enzomich at gmail.com (Enzo Michelangeli) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] OmilyX: Secure VoIP References: <4ef5fec60602230432o6d720cc4n4bbea301e610998@mail.gmail.com><43FE3964.4040207@hamachi.cc><4ef5fec60602231515p6858f38erc0e96a3c01c789dd@mail.gmail.com><43FE4AB1.2010204@hamachi.cc><4ef5fec60602231617h3ae6f434x4af6970f51299196@mail.gmail.com><43FE5659.3090708@hamachi.cc><4ef5fec60602231710i3e0f7346if1df115e9dc540e7@mail.gmail.com> <4ef5fec60602231835p5955a74agc0e5408d958de90e@mail.gmail.com> Message-ID: <080601c6391b$1ccec150$0200a8c0@EMNB> ----- Original Message ----- From: "coderman" To: "Peer-to-peer development." Sent: Friday, February 24, 2006 10:35 AM Subject: Re: [p2p-hackers] OmilyX: Secure VoIP > On 2/23/06, coderman wrote: > > [poor display of social communication skill] > > i apologize for the tone of this thread; this happens to be one of my > hot buttons as too many people consider "adding encryption" the path > to strong application security as this is really just presenting a > false sense of it when the other requisite development and UI > practices are ignored. No, coderman, you were absolutely right, and I think that ruffling some feathers every now and then is a minor sin, considering the damage that misplaced trust in the security of a flawed application could make in some cases. And to Alex: having the source code allows to build fresh binaries and disregard the standard binary distribution. Granted, that's not by itself sufficient to build trust in a complex application: undocumented spaghetti code would be almost impossible to subject to proper analysis and peer review. But the best design document is meaningless if you don't know if the implementation really follows it. Enzo From lemonobrien at yahoo.com Sat Feb 25 22:17:42 2006 From: lemonobrien at yahoo.com (Lemon Obrien) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Optimal UDP Message size for data streaming... Message-ID: <20060225221742.53946.qmail@web53603.mail.yahoo.com> I've tried to get TCP-ASYNC Connect to work for a while; i think its basicaly a ail marry pass; so i'm going to implement data streaming over udp....i jst have one questions...what is a good size to make the udp messages (data) when sending over the internet...i know most MTUs are about 1500K; is it better to chunk data at say 32 or 64K or keep the size smaller. My application works over the internet...not a corporate Internet...so bandwidth an all that plays a role. thank for the help. You don't get no juice unless you squeeze Lemon Obrien, the Third. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20060225/a65bac7f/attachment.html From dbarrett at quinthar.com Sat Feb 25 22:35:49 2006 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Optimal UDP Message size for data streaming... In-Reply-To: <20060225221742.53946.qmail@web53603.mail.yahoo.com> Message-ID: <20060225223553.54EAA3FD10@capsicum.zgp.org> UDP MTU is around 1500 *bytes* not *kilobytes*. The actual MTU depends on a zillion factors (a wide ranging discussion was on the list a while back); I eventually settled on 1472 bytes for iGlance and it seems to work. -david _____ From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On Behalf Of Lemon Obrien Sent: Saturday, February 25, 2006 2:18 PM To: Peer-to-peer development. Subject: [p2p-hackers] Optimal UDP Message size for data streaming... I've tried to get TCP-ASYNC Connect to work for a while; i think its basicaly a ail marry pass; so i'm going to implement data streaming over udp....i jst have one questions...what is a good size to make the udp messages (data) when sending over the internet...i know most MTUs are about 1500K; is it better to chunk data at say 32 or 64K or keep the size smaller. My application works over the internet...not a corporate Internet...so bandwidth an all that plays a role. thank for the help. You don't get no juice unless you squeeze Lemon Obrien, the Third. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20060225/0f5d2ed7/attachment.htm From agthorr at cs.uoregon.edu Sat Feb 25 22:50:56 2006 From: agthorr at cs.uoregon.edu (Daniel Stutzbach) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Optimal UDP Message size for data streaming... In-Reply-To: <20060225221742.53946.qmail@web53603.mail.yahoo.com> References: <20060225221742.53946.qmail@web53603.mail.yahoo.com> Message-ID: <20060225225055.GA3916@cs.uoregon.edu> On Sat, Feb 25, 2006 at 02:17:42PM -0800, Lemon Obrien wrote: > I've tried to get TCP-ASYNC Connect to work for a while; i think its > basicaly a ail marry pass; so i'm going to implement data streaming > over udp....i jst have one questions...what is a good size to make > the udp messages (data) when sending over the internet...i know most > MTUs are about 1500K; is it better to chunk data at say 32 or 64K or > keep the size smaller. My application works over the internet...not > a corporate Internet...so bandwidth an all that plays a role. Keeping the size below the path-MTU is important. If you use larger datagrams, they'll become fragmented by IP. If any of the fragments is lost, the whole datagram is lost, meaning you have to retransmit the whole datagram even though only one fragment was lost (or if you don't need retransmits, you're losing more data than necessary). Additionally, since your datagram is now made of several fragments, there's a greater chance that at least one of them will be lost. The right solution is to use path-MTU discovery. Initially set your datagram size to be your local MTU, and set the Don't-Fragment bit in all your packets. If you get back ICMP Needs Fragmentation messages, then you gradually lower the datagram size until they fit. You can also include an extra field in your protocol's initial handshake so each side learns the other's local MTU (since *typically* the lowest MTU is at one of the two edges). TCP does all of this for you behind-the-scenes. Or you can just set it to something like 1400 and cross your fingers. 1500 is often just a little too high because a significant minority have extra headers due to PPP, PPPoE, and/or VPNs. -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon From dbarrett at quinthar.com Sun Feb 26 01:08:39 2006 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Optimal UDP Message size for data streaming... In-Reply-To: <20060225225055.GA3916@cs.uoregon.edu> Message-ID: <20060226010848.6D12D3FD10@capsicum.zgp.org> > -----Original Message----- > From: Daniel Stutzbach > > The right solution is to use path-MTU discovery. Initially set your > datagram size to be your local MTU, and set the Don't-Fragment bit in > all your packets. If you get back ICMP Needs Fragmentation messages, > then you gradually lower the datagram size until they fit. You can > also include an extra field in your protocol's initial handshake so > each side learns the other's local MTU (since *typically* the lowest > MTU is at one of the two edges). TCP does all of this for you > behind-the-scenes. Has anyone on this list gotten MTU discovery over UDP to work in the real world? And if so, is there truly such variance that it's worth it? If the range is 1472-1500 bytes, it seems like a pointless exercise. If it's 1472-65536 bytes, then it's more interesting. Basically, given that you've got to make your application work well on small-MTU paths, is it worth the effort to find and use big MTUs or just more practical to find the lowest common denominator? -david From saikat at cs.cornell.edu Sun Feb 26 01:22:59 2006 From: saikat at cs.cornell.edu (Saikat Guha) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Optimal UDP Message size for data streaming... In-Reply-To: <20060226010848.6D12D3FD10@capsicum.zgp.org> References: <20060226010848.6D12D3FD10@capsicum.zgp.org> Message-ID: <1140916979.2958.5.camel@localhost.localdomain> On Sat, 2006-02-25 at 17:08 -0800, David Barrett wrote: > And if so, is there truly such variance that it's worth it? If the > range is 1472-1500 bytes, it seems like a pointless exercise. If it's > 1472-65536 bytes, then it's more interesting. The problem with MTU is that if you are off-by-1, you risk cutting your effective bandwidth by a factor of 2, or more. If you peg your app at 1472, and the real MTU is 1471 - every alternate IP packet has 1 byte of data, and in some cases (certain NATs that don't support fragmented packets), you lose all connectivity. Underestimating MTU isn't as bad as overestimating though. -- Saikat -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://zgp.org/pipermail/p2p-hackers/attachments/20060225/03bef25c/attachment.pgp From matthew at matthew.at Sun Feb 26 01:35:14 2006 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Optimal UDP Message size for data streaming... In-Reply-To: <20060226010848.6D12D3FD10@capsicum.zgp.org> Message-ID: <00b901c63a74$e72951e0$0202fea9@matthewdesk> > Basically, given that you've got to make your application > work well on small-MTU paths, is it worth the effort to find > and use big MTUs or just more practical to find the lowest > common denominator? I know too much about how the Internet is designed (see my bio) to believe that trying to detect >~1500-byte MTU is a useful exercise to engage in, at least for several more years, unless you are designing a demonstration for a very specific closed environment (and your application cares about total throughput, rather than, say, head-of-line blocking issues on slow links) And, as has been pointed out, overestimating is worse than underestimating. My personal choice is 1424 for the data portion of a UDP packet... (802.3 with LLC (1492 bytes instead of 1500) minus a GRE tunnel overhead (24 bytes, one of the worst tunnel overheads there is) minus a standard IP header with no options minus a UDP header, rounded down to the block size of my default symmetric crypto algorithm) Matthew Kaufman matthew@matthew.at http://www.amicima.com From agthorr at cs.uoregon.edu Sun Feb 26 01:46:39 2006 From: agthorr at cs.uoregon.edu (Daniel Stutzbach) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Optimal UDP Message size for data streaming... In-Reply-To: <20060226010848.6D12D3FD10@capsicum.zgp.org> References: <20060225225055.GA3916@cs.uoregon.edu> <20060226010848.6D12D3FD10@capsicum.zgp.org> Message-ID: <20060226014639.GB3916@cs.uoregon.edu> On Sat, Feb 25, 2006 at 05:08:39PM -0800, David Barrett wrote: > Has anyone on this list gotten MTU discovery over UDP to work in the real > world? And if so, is there truly such variance that it's worth it? If the > range is 1472-1500 bytes, it seems like a pointless exercise. If it's > 1472-65536 bytes, then it's more interesting. Most connections will have a path-MTU of 1500 bytes. Always using larger datagrams is definitely a Bad Idea. You might find larger MTUs in parts of the Internet backbone core or on finely-tuned gigabit-ethernet/FDDI/ATM networks, but that's about it. Regular users have ethernet (wired or wireless) in their path, which means their MTU is at most 1500 bytes. A significant minority of users will have a path-MTU slightly less than 1500 bytes. Basically, these users have a real MTU of 1500, but they're doing some kind of tunneling (PPP, PPPoE, VPN, etc.) which adds a few headers and eats into the MTU available to your application. PPPoE is an important one because some broadband access providers use it to deliver service to their users (I'm not sure what percentage, though). For a very tiny minority of users, the MTU will be even lower because they're using some funky access technology or doing something else strange. If you send datagrams that are slightly too big, they'll be fragmented and the performance hit is pretty bad. If you send datagrams that are slightly too small, you lose a little efficiency because a larger fraction of your throughput is being eaten up by packet headers. (As an aside: remember that for a datagram of x bytes, you only get x-28 bytes of payload since a 20-byte IP header and an 8 byte UDP header need to fit in there, too) > Basically, given that you've got to make your application work well on > small-MTU paths, is it worth the effort to find and use big MTUs or just > more practical to find the lowest common denominator? Well, the problem is that "the lowest common denominator" is not well defined. There may be one guy somewhere who is still using a SLIP link with a 256-byte MTU. If you use that as your baseline, all your other users are going to take a performance hit. On the other hand, if you use a fixed MTU of around 1400--1500, anyone with a lower MTU is going to have especially bad performance. Anyone with a higher MTU is going to have only slightly worse performance (compared to doing Path-MTU discovery). Unfortunately, you always risk somebody having a slightly lower MTU than the line you draw in the sand. I suggest just picking a reasonable value that works for you and your users. You can always add Path-MTU discovery latter if you want to squeeze out a small performance improvement and/or cater to the small fraction of users with a lower MTU and/or if the networks of the future have a more diverse set of commonly-used MTUs than today's Internet. -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon From saikat at cs.cornell.edu Sun Feb 26 02:01:40 2006 From: saikat at cs.cornell.edu (Saikat Guha) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Optimal UDP Message size for data streaming... In-Reply-To: <20060226014639.GB3916@cs.uoregon.edu> References: <20060225225055.GA3916@cs.uoregon.edu> <20060226010848.6D12D3FD10@capsicum.zgp.org> <20060226014639.GB3916@cs.uoregon.edu> Message-ID: <1140919300.2958.12.camel@localhost.localdomain> On Sat, 2006-02-25 at 17:46 -0800, Daniel Stutzbach wrote: > Well, the problem is that "the lowest common denominator" is not well > defined. There may be one guy somewhere who is still using a SLIP > link with a 256-byte MTU. If you use that as your baseline, all your > other users are going to take a performance hit. Well, technically, the minimum MTU is specified to be 576 in the RFC's. So if you really really need a lowest common denominator, there is one. But your argument stands that using any sort of lower bound will cause a performance hit. -- Saikat -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://zgp.org/pipermail/p2p-hackers/attachments/20060225/1bd2a8a0/attachment.pgp From agthorr at cs.uoregon.edu Sun Feb 26 02:17:13 2006 From: agthorr at cs.uoregon.edu (Daniel Stutzbach) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Optimal UDP Message size for data streaming... In-Reply-To: <1140919300.2958.12.camel@localhost.localdomain> References: <20060225225055.GA3916@cs.uoregon.edu> <20060226010848.6D12D3FD10@capsicum.zgp.org> <20060226014639.GB3916@cs.uoregon.edu> <1140919300.2958.12.camel@localhost.localdomain> Message-ID: <20060226021712.GC3916@cs.uoregon.edu> On Sat, Feb 25, 2006 at 09:01:40PM -0500, Saikat Guha wrote: > Well, technically, the minimum MTU is specified to be 576 in the RFC's. > So if you really really need a lowest common denominator, there is one. > But your argument stands that using any sort of lower bound will cause a > performance hit. You're thinking of the minimum datagram size the receiving IP implementation must support *after* reassembly. The official minimum MTU is a measly 68 bytes (RFC 1191, page 17). -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon From saikat at cs.cornell.edu Sun Feb 26 02:59:03 2006 From: saikat at cs.cornell.edu (Saikat Guha) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Optimal UDP Message size for data streaming... In-Reply-To: <20060226021712.GC3916@cs.uoregon.edu> References: <20060225225055.GA3916@cs.uoregon.edu> <20060226010848.6D12D3FD10@capsicum.zgp.org> <20060226014639.GB3916@cs.uoregon.edu> <1140919300.2958.12.camel@localhost.localdomain> <20060226021712.GC3916@cs.uoregon.edu> Message-ID: <1140922743.2958.21.camel@localhost.localdomain> On Sat, 2006-02-25 at 18:17 -0800, Daniel Stutzbach wrote: > You're thinking of the minimum datagram size the receiving IP > implementation must support *after* reassembly. > > The official minimum MTU is a measly 68 bytes (RFC 1191, page 17). Ah. Indeed. You're right. Interestingly, RFC 1122 seems to imply that "all networks in the Internet currently support an MTU of 576 or greater" and that it is strongly recommended (but not "RECOMMENDED"); regardless the official MTU of 68 still stands. cheers, -- Saikat -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://zgp.org/pipermail/p2p-hackers/attachments/20060225/12690625/attachment.pgp From kwalsh at cs.cornell.edu Mon Feb 27 14:03:18 2006 From: kwalsh at cs.cornell.edu (Kevin Walsh) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes Message-ID: <6.2.3.4.0.20060227090313.03ad6848@mail.cs.cornell.edu> Hi, There are a lot of proposals for peer-to-peer reputation systems of one kind or another, many of them targeted at finding peers that have good performance. Unfortunately very few systems have been implemented and deployed. Credence (http://www.cs.cornell.edu/People/egs/credence) is one system I have worked on, that has been deployed pretty widely in the Gnutella network. The goal of this project is to let clients identify spam, decoys, and polluted files, and it can also identify "credible" peers in the network. Please feel free to download the source code or binary package and play with it, of course. On the site you will also find some of our papers giving details. The papers also have related work and bibliographies listing at least a half dozen other proposals that have come up recently in the literature. This should let you get a start on peer-to-peer trust, reputation, and security literature, in any case. -Kevin > From: ramo@skuff-band.de > Subject: [p2p-hackers] Identify "defective" nodes > > Hi List, > > > i'm a student of Computer Science and trying to search for some good > Help's but don't know where to start. So what i want to know are there > some techniques for finding defective peers in a , let me say small p2p > network with 50 peers. With "defective" i mean , that a peer distribute > malicious manipulated services in a service-orientated network, so that > the processes on that peers work wrong and is there a technique to > eliminate automatically such a peer from the network, i mean something > like: put it in a dmz so that it can't do actual bad things. > > Is there any literature about that or does anyone of the Listpeople know > some more about that part? > > I would be thankful for any Information. > > Thank you for your thoughts. > > Ramo From Raphael_Manfredi at pobox.com Mon Feb 27 14:44:16 2006 From: Raphael_Manfredi at pobox.com (Raphael Manfredi) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: <6.2.3.4.0.20060227090313.03ad6848@mail.cs.cornell.edu> References: <6.2.3.4.0.20060227090313.03ad6848@mail.cs.cornell.edu> Message-ID: Quoting Kevin Walsh from ml.p2p.hackers: :Credence (http://www.cs.cornell.edu/People/egs/credence) is one :system I have worked on, that has been deployed pretty widely in the :Gnutella network. Gnutella, really? I've never heard about such a system being implemented. Do you have pointers to which servents support this, and documentation about what it means for Gnutella to support this system (header exchanges, messages, etc...). Thanks, Raphael From gbildson at limepeer.com Mon Feb 27 17:20:50 2006 From: gbildson at limepeer.com (Greg Bildson) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: Message-ID: You must have missed the train then. It was available about a year ago based off the LimeWire open source code. Kevin and company also found some security issues that we had to scramble to fix. Thanks -greg > -----Original Message----- > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On > Behalf Of Raphael Manfredi > Sent: Monday, February 27, 2006 9:44 AM > To: p2p-hackers@zgp.org > Subject: [p2p-hackers] Re: Identify "defective" nodes > > > Quoting Kevin Walsh from ml.p2p.hackers: > :Credence (http://www.cs.cornell.edu/People/egs/credence) is one > :system I have worked on, that has been deployed pretty widely in the > :Gnutella network. > > Gnutella, really? I've never heard about such a system being implemented. > > Do you have pointers to which servents support this, and documentation > about what it means for Gnutella to support this system (header exchanges, > messages, etc...). > > Thanks, > Raphael > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From coderman at gmail.com Mon Feb 27 18:14:49 2006 From: coderman at gmail.com (coderman) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Fwd: [funsec] Re: AT&T's database of 1.92 trillion phone calls (Sprint does it too, and i'm sure they aren't the only ones) In-Reply-To: <4ef5fec60602270905h1d4626b0scc67e79e6a2f751c@mail.gmail.com> References: <4ef5fec60602250852t4a1eaa5v96c70ebc95d9234d@mail.gmail.com> <3c4611bc0602270742x4fe47927q8b11a8a7826cf22b@mail.gmail.com> <4ef5fec60602270905h1d4626b0scc67e79e6a2f751c@mail.gmail.com> Message-ID: <4ef5fec60602271014m4385f392j7db601b1ad812e08@mail.gmail.com> Fwd: discussion on enabling and motivating end users to assume responsbility for their own information security/privacy over the communication and computing resources they use. can we go ahead and state as fact that a capability model tied to a pet name pattern / sticky note metaphor is required for strong least privelege which in turn is mandatory for the secure user interface / interaction requirements mentioned below? [if you don't think caps and pet names should be mandatory, can you provide a reasonable explanation of how key based capabilities and pets names are less secure than the alternative you are describing?] wow, that's a lot work to describe in detail (design) let alone even attempt to implement. (at least if you designed and implemented it right once you should never need to implement again) ---------- Forwarded message ---------- From: coderman Date: Feb 27, 2006 9:05 AM Subject: Re: [funsec] Re: AT&T's database of 1.92 trillion phone calls (Sprint does it too, and i'm sure they aren't the only ones) To: Brian Loe Cc: funsec@linuxbox.org, cypherpunks@jfet.org On 2/27/06, Brian Loe wrote: > ... > This kind of thing doesn't scare me. What they wind up doing with it, > at times, does scare me. What scares me even more is that no one will > ever do anything about it on a scale that matters. i'd have to agree, with regards to corporate or government entities making strong individual privacy a priority _on their own accord_ with this kind of applied information technology.[1] so the only feasible solution is empowering users to take responsibility for their own information security and privacy. if "johnny can't encrypt"[2] this is a very tall order indeed[3]. what would the ideal minimum amount of information exposed consist of if you could apply usable security/encryption and privacy enhancing technologies to the usual communications today (voice, text, video, data)? - no content of payloads, due to end to end encryption - strong anonymous mix networks for non interactive messaging - weakly anonymous low latency onion/relay networks for near real time messaging - seamless wireless and sneaker net support to offload locally/out of band whenever possible you'd still be exposing: - location of endpoints used (except in the strong and latent mix scenario perhaps) - distinct parties involved (social network analysis) - volume of encrypted traffic exchanged i suppose the real question is how long would it take to design and implement (and the hardware to support it prevalent for all users). 5 years seems extremely optimistic given the difficulties involved. [and i suppose this also means the paranoid will all become proficient TSCM technicians.] ah, we can dream :) until then, the fraction of unusual end lusers making use of strong privacy enhancing technologies will be a function of how annoying they are to use vs. how annoying the government privacy invasion programs become. single digits for the near future... --- [1.] "DoJ strikes back against Google (your privacy concerns are unfounded (lol))" http://www.theinquirer.net/?article=29918 [2.] "NPR : E-Mail Encryption Rare in Everyday Use" http://www.mail-archive.com/cryptography@metzdowd.com/msg05769.html [3.] " User Interaction Design for Secure Systems" http://www.cs.berkeley.edu/~pingster/sec/uid/ MANDATORY REQUIREMENTS:: A. Path of Least Resistance. The most natural way to do any task should also be the most secure way. B. Appropriate Boundaries. The interface should expose, and the system should enforce, distinctions between objects and between actions along boundaries that matter to the user. C. Explicit Authorization. A user's authorities must only be provided to other actors as a result of an explicit user action that is understood to imply granting. D. Visibility. The interface should allow the user to easily review any active actors and authority relationships that would affect security-relevant decisions. E. Revocability. The interface should allow the user to easily revoke authorities that the user has granted, wherever revocation is possible. F. Trusted Path. The interface must provide an unspoofable and faithful communication channel between the user and any entity trusted to manipulate authorities on the user's behalf. G. Identifiability. The interface should enforce that distinct objects and distinct actions have unspoofably identifiable and distinguishable representations. From coderman at gmail.com Mon Feb 27 19:56:48 2006 From: coderman at gmail.com (coderman) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] hamachi p2p vpn nat-friendly protocol details Message-ID: <4ef5fec60602271156h7978e66fr2557460962601bbc@mail.gmail.com> --- various interesting forwards related to decentralized (p2p) authenticity and privacy favorite quote: ">> Designing security protocols is hard... Yes, it is. This is why I like it." --- 'Cypherpunks Write [Secure] Code!' An open question for anyone reading this: is the critically wounded, barely beating cypherpunks list languishing in such a sad state due to apathy or impending irrelevance/death? i see embers of life awaiting some minimal votes of confidence (i'd go so far as to offer sexual favors for a toad node back* :) but any kind of renewed interest is meager at best. [would a public list archive help? an rss feed? a abridged list / digest? [ala kernel trap]] such volatile times, so little interest... ---- http://www.mail-archive.com/cryptography@metzdowd.com/msg05790.html "" Alex Pankratov Sun, 26 Feb 2006 07:18:15 -0800 ... Tero Kivinen wrote: >> Travis H. writes: >> > >>>>http://www.hamachi.cc/security >>>> >>>>Based on a cursory look over this, I'm impressed by both the level of >>>>detail and the level of security apparently afforded. Too bad I can't >>>>see the source code. > >> >> >> I can see couple of problems in the system. Firstly it seems it uses >> same key for both directions for the encryption and for >> authentication, i.e. the KEYMAT is only split to Ke and Ka keys, which >> are used for encryption and authentication. In general using same keys >> for different directions is bad. The description on a page was not updated properly. Recent clients use per-direction keys after they complete P2P KE. >> Secondly I cannot find where it >> authenticates the crypto suite used at all (it is not included in the >> signature of the AUTH message). Crypto suite is essentially just a protocol number. It requires no authentication. If the server side responds with HELO.OK, it means that it can comprehend specified protocol revision. Similar to what happens during the SSH handshake. >> Also it seems that the identity itself >> is not authenticated at all, as it (or it's MACed form) is not >> included in the signature. It is not. >> There might be (I am not sure whether AUTH >> packet is encrypted and MACed) a MAC over it, but the MAC key is not >> yet authenticated as it is generated from the anonymous >> Diffie-Hellman. That might give it some protection, but I am not sure >> if that is enough. A protection against what kind of attack ? Identity is used to specify which public key the client wants to be authenticated with on the server side. Assuming it is swapped in transition by a man in the middle, it would still require an attacker to re-sign authentication hash in the message. Assuming he has a private key to do that, he will effectively succeed in authenticating under substituted ID. He then will need to re-sign server's auth hash to complete the attack, which is not going to happen. There is an off chance that the attacker might swapped the identity to one that has the same public key. The chances of this happening are infinitely small unless an attacker also has an access to victim's keypair, which becomes a trivial attack case. . . . >> The protocol is also tied to use SHA1. If you are referring to HMAC-SHA1 for authentication hashes, it is a part of a crypto suite (protocol revision) spec. >> In general it would be much better to use standard protocol, instead >> of generating your own. This is the second revision of Hamachi system. First revision was using SSL for cli-srv and IKE/ESP for p2p security. It was a prototype and it soon become obvious that both SSL and IKE were overkills for our purposes. We did not need certificate authentication of SSL, we did not want to run our own auth protocol over SSL/AnonDH, which would've increased the number of packets per login sequence. We didn't need the flexibility (ie complexity) of IKE either. After stripping down IKE (ie removing SA negotiation, reworking ID payloads and not doing quick mode), we essentially ended up with a protocol that was also fit for securing cli-srv session. It was further tweaked and replaced SSL. I should probably add that I implemented IKE (v1) keying daemon from scratch with all bells and wistles (NATT, extended MODP groups, etc) at some point in the past. Some remnants of it are still floating around, the library name was libike. >> Designing security protocols is hard... Yes, it is. This is why I like it. "" ===== http://www.mail-archive.com/cryptography@metzdowd.com/msg05796.html Travis H. Sun, 26 Feb 2006 07:22:06 -0800 > Crypto suite is essentially just a protocol number. It requires > no authentication. If the server side responds with HELO.OK, it > means that it can comprehend specified protocol revision. Similar > to what happens during the SSH handshake. In SSL, the lack of authentication of the cryptosuite could be used to convince a v3 client that it is communicating with a v2 server, and the v3 server that it is communicating with a v2 client, causing them to communicate using SSL v2, which is called the "version rollback attack". This is not relevant to the hamachi protocol because there is no negotiation. Nevertheless, authenticating the previous plaintext fields once a secure channel is established is considered good form. In Schneier's "Practical Cryptography", he suggests computing the MAC over the entire history of sent messages, which ensures that any tampering is detected at the next MAC. This is eventually what was done in SSLv3, for reasons Tero alluded to and which are successfully thwarted for the reasons you describe. . . . I sort of wonder at the utility of a TCP implementation of the p2p VPN... tunnelling TCP over TCP is well known to be a Bad Thing with regard to interaction of the TCP timeouts. Aside: Can anyone tell me why the constants used in ipad and opad for HMAC were chosen? If they're not arbitrary, I'd like to know the rationale behind them. ===== http://www.mail-archive.com/cryptography@metzdowd.com/msg05801.html Alex Pankratov Sun, 26 Feb 2006 07:24:20 -0800 > Presumably he wants to make sure that the messages like the following > have an unambiguous interpretation: > AUTH Identity Signature(Ni | Nr | Gi | Gr, Kpri_cli) > Merely concatenating them is insufficient unless all but one have a > fixed length. > I think a terse "unambiguous representation" rationale is the whole > reason for ASN.1, although it seems awfully complex for such a simple > goal. Nonces and DH exponents are serialized using PER-style ASN.1 encoding. So the whole concatenation is unambigious. > I sort of wonder at the utility of a TCP implementation of the p2p > VPN... tunnelling TCP over TCP is well known to be a Bad Thing with > regard to interaction of the TCP timeouts. Just to be clear, Hamachi tunnels VPN/P2P traffic over UDP. TCP is used for client-server session only. VPN over TCP is bad for two reasons. One you listed, and another is that it becomes trivial to DoS this kind of VPN. TCP packets are not authenticated (unless MD5/BGP extension is used, which is unlikely), so the state of VPN transport layer and consequently the state of a tunnel can be altered by 3rd party. That's why SSL VPNs make very little sense in non-proxied setups and that's why (I'd guess) OpenVPN 'tweaked' SSL to run over UDP instead. ---end-cut--- From coderman at gmail.com Mon Feb 27 21:03:21 2006 From: coderman at gmail.com (coderman) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] FAQ: How to subscribe and or contribute to cypherpunks Message-ID: <4ef5fec60602271303g45d7064bsc128c3550b38e446@mail.gmail.com> by popular demand (for some degree of popular that includes a few emails to my inbox seeking additional information :) ::Question:: How do I observe/participate in cypherpunks list discussions? subscription requests consist of a simple text body with the list name and your email address sent to a special email list manager account. a subscribe request typically looks like: "subscribe cypherpunks coderman@gmail.com" for example. some popular/common list administration services/hosts are: majordomo@jfet.org majordomo@al-qaeda.net [ maybe majordomo@toad.com ? :P ] a full example just for clarification: ---begin--- To: majordomo@jfet.org Subject: subscribe cypherpunks Body: subscribe cypherpunks coderman@gmail.com ---end--- ::Question:: what can i do to contribute? anything that would be helpful / interesting to facilitate easier and most useful conversations via this list: - more nodes and more contributors, be that code, design or philosophical tangents of tenuous relation to encryption or privacy. - a public web based subscription interface for users - a RSS/Atom/feed based distribution of list contents - a sister best of / abridged / digest version of list traffic distributed in daily and monthly digests. - write secure code to protect your privacy and facilitate trusted social interactions with your peers and associates. - SEND ADDITIONAL COMMENTS / SUGGESTIONS TO THIS LIST :) From Raphael_Manfredi at pobox.com Mon Feb 27 22:38:20 2006 From: Raphael_Manfredi at pobox.com (Raphael Manfredi) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: References: Message-ID: Quoting Greg Bildson from ml.p2p.hackers: :You must have missed the train then. I must have missed the specs, yes. :It was available about a year ago based off the LimeWire open source code. :Kevin and company also found some security issues that we had to scramble to :fix. Where can I find the specs so that I can implement the protocol in gtk-gnutella? Raphael From sberlin at gmail.com Mon Feb 27 22:55:10 2006 From: sberlin at gmail.com (Sam Berlin) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: References: Message-ID: <19196d860602271455p11947229v33cc0139043c0ac2@mail.gmail.com> I don't think Credence has released any specs yet? If they have, that'd be nice. I'm sure they have the code available on their site though. Sam On 27 Feb 2006 22:38:20 GMT, Raphael Manfredi wrote: > Quoting Greg Bildson from ml.p2p.hackers: > :You must have missed the train then. > > I must have missed the specs, yes. > > :It was available about a year ago based off the LimeWire open source code. > :Kevin and company also found some security issues that we had to scramble to > :fix. > > Where can I find the specs so that I can implement the protocol in gtk-gnutella? > > Raphael > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From Raphael_Manfredi at pobox.com Mon Feb 27 23:34:56 2006 From: Raphael_Manfredi at pobox.com (Raphael Manfredi) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: <19196d860602271455p11947229v33cc0139043c0ac2@mail.gmail.com> References: <19196d860602271455p11947229v33cc0139043c0ac2@mail.gmail.com> Message-ID: Quoting Sam Berlin from ml.p2p.hackers: :I don't think Credence has released any specs yet? If they have, :that'd be nice. I'm sure they have the code available on their site :though. OK, then it explains why I "missed the train", as Greg said. No specs, no inter-operable implementations accross Gnutella servents. Although I can reverse-engineer Java, there's no way I'm going to do it on code that could evolve at any time in the future, since no specs are published. Raphael From gbildson at limepeer.com Tue Feb 28 15:38:25 2006 From: gbildson at limepeer.com (Greg Bildson) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: Message-ID: The train was full of ideas and enlightenment. It wasn't necessarily full of an implementation imperative. :-) Thanks -greg > -----Original Message----- > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On > Behalf Of Raphael Manfredi > Sent: Monday, February 27, 2006 6:35 PM > To: p2p-hackers@zgp.org > Subject: [p2p-hackers] Re: Identify "defective" nodes > > > Quoting Sam Berlin from ml.p2p.hackers: > :I don't think Credence has released any specs yet? If they have, > :that'd be nice. I'm sure they have the code available on their site > :though. > > OK, then it explains why I "missed the train", as Greg said. > No specs, no inter-operable implementations accross Gnutella servents. > > Although I can reverse-engineer Java, there's no way I'm going to > do it on code that could evolve at any time in the future, since no > specs are published. > > Raphael From kwalsh at cs.cornell.edu Tue Feb 28 15:56:07 2006 From: kwalsh at cs.cornell.edu (Kevin Walsh) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: References: Message-ID: <6.2.3.4.0.20060228105303.01f980a0@mail.cs.cornell.edu> > > OK, then it explains why I "missed the train", as Greg said. The train would be happy to come back... > > No specs, no inter-operable implementations accross Gnutella servents. We currently have an implementation built as an add-on to the LimeWire client. It is in Java, of course, and the source has been available for about a year now (see http://www.cs.cornell.edu/People/egs/credence/). Our implementation rides on top of LimeWire, basically, and doesn't depend on any limewire specific features, and (so far) I have not needed to make any changes to the line protocol in any way. > > Although I can reverse-engineer Java, there's no way I'm going to > > do it on code that could evolve at any time in the future, since no > > specs are published. If you are interested in a second implementation, that would be great! Reverse engineering isn't necessary -- I'd be happy to work with you directly to make it happen. As for details about the protocol, a good place to start is our published papers on Credence (also linked off the site above), which lay out the rational and design. I'd also be happy to draft a spec of the currently implemented protocol, if that would be helpful. If anyone wants to suggest a "model" gnutella spec, I will make an effort to follow its style and conventions. At the same time, if there are going to be multiple implementations, it may make sense to have a wider discussion about the spec and interoperability as that happens. For instance, some things would definitely be more efficient if Credence were integrated a bit tighter with the underlying gnutella protocol. Anyone interested in following up can contact me directly any time. -Kevin From sberlin at gmail.com Tue Feb 28 16:00:39 2006 From: sberlin at gmail.com (Sam Berlin) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: <6.2.3.4.0.20060228105303.01f980a0@mail.cs.cornell.edu> References: <6.2.3.4.0.20060228105303.01f980a0@mail.cs.cornell.edu> Message-ID: <19196d860602280800n2674cc64s4a07565d5651fd4b@mail.gmail.com> I've been meaning for a very long time to add a plugin architecture to LimeWire, for things such as Credence and the like. If anyone here is interested in such an addition, so they can provide their own add-ons to the program (and be able to ship just the plugin, not the whole program), please let me know. I'll attempt to gather up what folks are looking for in an architecture and write something suitable. Thanks. Sam On 2/28/06, Kevin Walsh wrote: > > > > OK, then it explains why I "missed the train", as Greg said. > > The train would be happy to come back... > > > > No specs, no inter-operable implementations accross Gnutella servents. > > We currently have an implementation built as an add-on to the > LimeWire client. It is in Java, of course, and the source has been > available for about a year now (see > http://www.cs.cornell.edu/People/egs/credence/). Our implementation > rides on top of LimeWire, basically, and doesn't depend on any > limewire specific features, and (so far) I have not needed to make > any changes to the line protocol in any way. > > > > Although I can reverse-engineer Java, there's no way I'm going to > > > do it on code that could evolve at any time in the future, since no > > > specs are published. > > If you are interested in a second implementation, that would be > great! Reverse engineering isn't necessary -- I'd be happy to work > with you directly to make it happen. As for details about the > protocol, a good place to start is our published papers on Credence > (also linked off the site above), which lay out the rational and design. > > I'd also be happy to draft a spec of the currently implemented > protocol, if that would be helpful. If anyone wants to suggest a > "model" gnutella spec, I will make an effort to follow its style and > conventions. At the same time, if there are going to be multiple > implementations, it may make sense to have a wider discussion about > the spec and interoperability as > that happens. For instance, some things would definitely be more > efficient if Credence were integrated a bit tighter with the > underlying gnutella protocol. > > Anyone interested in following up can contact me directly any time. > > -Kevin > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From agthorr at cs.uoregon.edu Tue Feb 28 16:14:19 2006 From: agthorr at cs.uoregon.edu (Daniel Stutzbach) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: <19196d860602280800n2674cc64s4a07565d5651fd4b@mail.gmail.com> References: <6.2.3.4.0.20060228105303.01f980a0@mail.cs.cornell.edu> <19196d860602280800n2674cc64s4a07565d5651fd4b@mail.gmail.com> Message-ID: <20060228161418.GN5704@cs.uoregon.edu> On Tue, Feb 28, 2006 at 11:00:39AM -0500, Sam Berlin wrote: > I've been meaning for a very long time to add a plugin architecture to > LimeWire, for things such as Credence and the like. If anyone here is > interested in such an addition, so they can provide their own add-ons > to the program (and be able to ship just the plugin, not the whole > program), please let me know. I'll attempt to gather up what folks > are looking for in an architecture and write something suitable. Wouldn't the API needed by plug-ins vary dramatically by what the plug-in is designed to do? Or do you have a particular group of plug-in applications in mind that would interact with LimeWire in a similar way? I'd love it if the plug-ins ran as separate processes and communicated with LimeWire over a local-only TCP port. That way the plug-ins wouldn't have to be written in a particular language. -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon From sberlin at gmail.com Tue Feb 28 16:49:02 2006 From: sberlin at gmail.com (Sam Berlin) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: <20060228161418.GN5704@cs.uoregon.edu> References: <6.2.3.4.0.20060228105303.01f980a0@mail.cs.cornell.edu> <19196d860602280800n2674cc64s4a07565d5651fd4b@mail.gmail.com> <20060228161418.GN5704@cs.uoregon.edu> Message-ID: <19196d860602280849v2d9c9c6am671a020881f425a3@mail.gmail.com> I'm looking for exactly these kinds of ideas. Basically, I'd like to flesh out the range (however dramatic it may be) of what plug-ins would like to do with LimeWire. Then we can trim it down from there as to what's reasonable to do within a plug-in. Communication over a local-only TCP port is good, although I suspect the API available would be slightly more limited if that route was taken. Sam On 2/28/06, Daniel Stutzbach wrote: > On Tue, Feb 28, 2006 at 11:00:39AM -0500, Sam Berlin wrote: > > I've been meaning for a very long time to add a plugin architecture to > > LimeWire, for things such as Credence and the like. If anyone here is > > interested in such an addition, so they can provide their own add-ons > > to the program (and be able to ship just the plugin, not the whole > > program), please let me know. I'll attempt to gather up what folks > > are looking for in an architecture and write something suitable. > > Wouldn't the API needed by plug-ins vary dramatically by what the > plug-in is designed to do? Or do you have a particular group of > plug-in applications in mind that would interact with LimeWire in a > similar way? > > I'd love it if the plug-ins ran as separate processes and communicated > with LimeWire over a local-only TCP port. That way the plug-ins > wouldn't have to be written in a particular language. > > -- > Daniel Stutzbach Computer Science Ph.D Student > http://www.barsoom.org/~agthorr University of Oregon > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From lemonobrien at yahoo.com Tue Feb 28 18:06:49 2006 From: lemonobrien at yahoo.com (Lemon Obrien) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: Message-ID: <20060228180649.18774.qmail@web53603.mail.yahoo.com> i do this by using info from ping/pong....u have to ping nodes to kep the udp channel open.../even relay stations have to ping nodes behind firewalls....so...say, after X number of pings and no pong from the application....the reference is removed. Greg Bildson wrote: The train was full of ideas and enlightenment. It wasn't necessarily full of an implementation imperative. :-) Thanks -greg > -----Original Message----- > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On > Behalf Of Raphael Manfredi > Sent: Monday, February 27, 2006 6:35 PM > To: p2p-hackers@zgp.org > Subject: [p2p-hackers] Re: Identify "defective" nodes > > > Quoting Sam Berlin from ml.p2p.hackers: > :I don't think Credence has released any specs yet? If they have, > :that'd be nice. I'm sure they have the code available on their site > :though. > > OK, then it explains why I "missed the train", as Greg said. > No specs, no inter-operable implementations accross Gnutella servents. > > Although I can reverse-engineer Java, there's no way I'm going to > do it on code that could evolve at any time in the future, since no > specs are published. > > Raphael _______________________________________________ p2p-hackers mailing list p2p-hackers@zgp.org http://zgp.org/mailman/listinfo/p2p-hackers _______________________________________________ Here is a web page listing P2P Conferences: http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences You don't get no juice unless you squeeze Lemon Obrien, the Third. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20060228/150a5a15/attachment.htm From agthorr at cs.uoregon.edu Tue Feb 28 18:25:32 2006 From: agthorr at cs.uoregon.edu (Daniel Stutzbach) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: <19196d860602280849v2d9c9c6am671a020881f425a3@mail.gmail.com> References: <6.2.3.4.0.20060228105303.01f980a0@mail.cs.cornell.edu> <19196d860602280800n2674cc64s4a07565d5651fd4b@mail.gmail.com> <20060228161418.GN5704@cs.uoregon.edu> <19196d860602280849v2d9c9c6am671a020881f425a3@mail.gmail.com> Message-ID: <20060228182532.GQ5704@cs.uoregon.edu> On Tue, Feb 28, 2006 at 11:49:02AM -0500, Sam Berlin wrote: > Basically, I'd like to flesh out the range (however dramatic it may > be) of what plug-ins would like to do with LimeWire. Then we can trim > it down from there as to what's reasonable to do within a plug-in. My pet plug-in idea is a little library called "Stitch" that takes over writing downloaded bytes to disk. It'd be a multiple-app plug-in, so that I could run LimeWire and (for example) BitTorrent at the same time and if I happen to be downloading the same file in both P2P systems, it will figure that out and merge the results into one file. Naturally, it would also need a way to report back to the apps "I got these segments elsewhere" and to make use of the app-specific integrity checking. (this would also expand the market for P2P file-sharing applications by making it more useful for individuals to use multiple applications instead of just one) Unfortunately, I can't think of any *other* reason a plug-in would need that particular kind of API, so it's kind of a one-trick pony. Certainly it would be useful for a plug-in to rate or categorize files and have LimeWire display this information (for both shared files and query-hits). There's a wide range of things a plug-in could do here, from integrity checking to license checking to recommendations based on files the user already has. -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon From gbildson at limepeer.com Tue Feb 28 18:42:14 2006 From: gbildson at limepeer.com (Greg Bildson) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: <20060228182532.GQ5704@cs.uoregon.edu> Message-ID: > > My pet plug-in idea is a little library called "Stitch" that takes > over writing downloaded bytes to disk. It'd be a multiple-app > plug-in, so that I could run LimeWire and (for example) BitTorrent at > the same time and if I happen to be downloading the same file in both > P2P systems, it will figure that out and merge the results into one > file. Naturally, it would also need a way to report back to the apps > "I got these segments elsewhere" and to make use of the app-specific > integrity checking. > > (this would also expand the market for P2P file-sharing applications > by making it more useful for individuals to use multiple applications > instead of just one) > > Unfortunately, I can't think of any *other* reason a plug-in would > need that particular kind of API, so it's kind of a one-trick pony. > That would be extremely hard to do efficiently with the internal algorithms used to grab chunks of files. If you don't want to redownload overlapping regions, you would have to control the internals of both algorithms very precisely. The best reason I could see for doing this would be for failure recovery. In that case, you just need to restart one download type using another protocol. Presumably, either protocol can handle the download of a specific file well. Thanks -greg From agthorr at cs.uoregon.edu Tue Feb 28 18:47:40 2006 From: agthorr at cs.uoregon.edu (Daniel Stutzbach) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: References: <20060228182532.GQ5704@cs.uoregon.edu> Message-ID: <20060228184738.GS5704@cs.uoregon.edu> On Tue, Feb 28, 2006 at 01:42:14PM -0500, Greg Bildson wrote: > That would be extremely hard to do efficiently with the internal algorithms > used to grab chunks of files. If you don't want to redownload overlapping > regions, you would have to control the internals of both algorithms very > precisely. Yes, true, it would also need to be able to tell the application "go fetch this chunk" rather than letting the application choose. -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon From gbildson at limepeer.com Tue Feb 28 18:56:43 2006 From: gbildson at limepeer.com (Greg Bildson) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: <20060228184738.GS5704@cs.uoregon.edu> Message-ID: > > Yes, true, it would also need to be able to tell the application "go > fetch this chunk" rather than letting the application choose. > Which would likely undermine any algorithm trying to improve availability by downloading random or most rare chunks. Thanks -greg From agthorr at cs.uoregon.edu Tue Feb 28 19:04:24 2006 From: agthorr at cs.uoregon.edu (Daniel Stutzbach) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: References: <20060228184738.GS5704@cs.uoregon.edu> Message-ID: <20060228190423.GT5704@cs.uoregon.edu> On Tue, Feb 28, 2006 at 01:56:43PM -0500, Greg Bildson wrote: > > Yes, true, it would also need to be able to tell the application "go > > fetch this chunk" rather than letting the application choose. > > Which would likely undermine any algorithm trying to improve availability by > downloading random or most rare chunks. Well, the plug-in could take over doing that part, too. I did mention this was my pipe-dream plug-in, right? :-) -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon From coderman at gmail.com Tue Feb 28 19:26:10 2006 From: coderman at gmail.com (coderman) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Interoperable distribution networks, was Re: Identify "defective" nodes Message-ID: <4ef5fec60602281126l514789a0v67f0f4bb957137d5@mail.gmail.com> On 2/28/06, Daniel Stutzbach wrote: > On Tue, Feb 28, 2006 at 01:56:43PM -0500, Greg Bildson wrote: > > > Yes, true, it would also need to be able to tell the application "go > > > fetch this chunk" rather than letting the application choose. > > > > Which would likely undermine any algorithm trying to improve availability by > > downloading random or most rare chunks. > > Well, the plug-in could take over doing that part, too. > > I did mention this was my pipe-dream plug-in, right? :-) one way in which this might be useful is download queues, where a number of files are scheduled for downloading over a number of distribution/transport networks. we all know how often failures occur leaving you with a partial transfer. Stitch could then identify a partial transfer of one resource and copy the data into a format (torrent?) that would be resumable over a different network. this way you are not running two competing transfers for the same resource at once, but rather managing two pools of dowload queues and cooperating between them when partial failures occur. From Raphael_Manfredi at pobox.com Tue Feb 28 19:33:13 2006 From: Raphael_Manfredi at pobox.com (Raphael Manfredi) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Interoperable distribution networks, was Re: Identify "defective" nodes In-Reply-To: <4ef5fec60602281126l514789a0v67f0f4bb957137d5@mail.gmail.com> References: <4ef5fec60602281126l514789a0v67f0f4bb957137d5@mail.gmail.com> Message-ID: Quoting coderman from ml.p2p.hackers: :one way in which this might be useful is download queues, where a :number of files are scheduled for downloading over a number of :distribution/transport networks. To have inter-operable distribution networks, it would help if all said distribution networks used the same identification for files and the same download protocol. SHA1 and HTTP come to mind, but of course some people will prefer MD5 and whatever. :we all know how often failures occur leaving you with a partial :transfer. Stitch could then identify a partial transfer of one :resource and copy the data into a format (torrent?) that would be :resumable over a different network. Ideally, in a distribution network, given a file hash (SHA1, say) you should be able to get it. All you need is a magnet URI with the SHA1. Raphael From coderman at gmail.com Tue Feb 28 19:42:47 2006 From: coderman at gmail.com (coderman) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Interoperable distribution networks, was Re: Identify "defective" nodes In-Reply-To: References: <4ef5fec60602281126l514789a0v67f0f4bb957137d5@mail.gmail.com> Message-ID: <4ef5fec60602281142n25939015s2eb28aa92751ed19@mail.gmail.com> On 28 Feb 2006 19:33:13 GMT, Raphael Manfredi wrote: > Quoting coderman from ml.p2p.hackers: > :one way in which this might be useful is download queues, where a > :number of files are scheduled for downloading over a number of > :distribution/transport networks. > > To have inter-operable distribution networks, it would help if all > said distribution networks used the same identification for files and > the same download protocol. a bitzi like service might be helpful when resource identifiers were not shared; you could use the bitzi service to locate other identifiers given the one you know already. (http://bitzi.com) but yes, it would be much better to simply pick a common identifier (i'd highly recommend SHA2-256 or 512 at this point) universally supported by all networks. From m.rogers at cs.ucl.ac.uk Tue Feb 28 20:04:53 2006 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Interoperable distribution networks, was Re: Identify "defective" nodes In-Reply-To: References: <4ef5fec60602281126l514789a0v67f0f4bb957137d5@mail.gmail.com> Message-ID: <4404ACE5.5080107@cs.ucl.ac.uk> Raphael Manfredi wrote: > To have inter-operable distribution networks, it would help if all > said distribution networks used the same identification for files and > the same download protocol. There seem to be at least five ways to identify files: SHA1 of the entire file, THEX root hash, MD4 hash of the entire file (ed2k), kzhash, and BitTorrent infohash. It seems unlikely that any of these will go away, so I guess interoperability will depend on apps supporting all five methods, despite the overhead... or we could work on a new standard and then everyone would have six methods to support. ;-) Cheers, Michael From gojomo at bitzi.com Tue Feb 28 20:28:38 2006 From: gojomo at bitzi.com (Gordon Mohr) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: <20060228161418.GN5704@cs.uoregon.edu> References: <6.2.3.4.0.20060228105303.01f980a0@mail.cs.cornell.edu> <19196d860602280800n2674cc64s4a07565d5651fd4b@mail.gmail.com> <20060228161418.GN5704@cs.uoregon.edu> Message-ID: <4404B276.2070000@bitzi.com> Daniel Stutzbach wrote: > On Tue, Feb 28, 2006 at 11:00:39AM -0500, Sam Berlin wrote: >> I've been meaning for a very long time to add a plugin architecture to >> LimeWire, for things such as Credence and the like. If anyone here is >> interested in such an addition, so they can provide their own add-ons >> to the program (and be able to ship just the plugin, not the whole >> program), please let me know. I'll attempt to gather up what folks >> are looking for in an architecture and write something suitable. > > Wouldn't the API needed by plug-ins vary dramatically by what the > plug-in is designed to do? Or do you have a particular group of > plug-in applications in mind that would interact with LimeWire in a > similar way? Yes, the API designer has to have a fair idea of the universe of potential plug-in capabilities, and what hooks will be necessary to enable those. But, there may be good models to emulate. I'm not familiar with the Eclipse plug-in API, but it seems to allow a lot of deep, varied improvements via drop-in modules. This project could be of interest for Limewire: Java Plug-in Framework (JPF) Project http://jpf.sourceforge.net/ It's an Eclipse-inspired framework for adding plug-in capabilities to other Java applications. - Gordon From sberlin at gmail.com Tue Feb 28 20:36:20 2006 From: sberlin at gmail.com (Sam Berlin) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: <4404B276.2070000@bitzi.com> References: <6.2.3.4.0.20060228105303.01f980a0@mail.cs.cornell.edu> <19196d860602280800n2674cc64s4a07565d5651fd4b@mail.gmail.com> <20060228161418.GN5704@cs.uoregon.edu> <4404B276.2070000@bitzi.com> Message-ID: <19196d860602281236g2c4a469bu486d61ed85df76af@mail.gmail.com> Thanks very much for pointing that out, Gordon. The Stitch plug-in sounds like a very neat idea. Has anyone else had a pipe-dream plug-in they dreamed of being added to a p2p application? Sam On 2/28/06, Gordon Mohr wrote: > Daniel Stutzbach wrote: > > On Tue, Feb 28, 2006 at 11:00:39AM -0500, Sam Berlin wrote: > >> I've been meaning for a very long time to add a plugin architecture to > >> LimeWire, for things such as Credence and the like. If anyone here is > >> interested in such an addition, so they can provide their own add-ons > >> to the program (and be able to ship just the plugin, not the whole > >> program), please let me know. I'll attempt to gather up what folks > >> are looking for in an architecture and write something suitable. > > > > Wouldn't the API needed by plug-ins vary dramatically by what the > > plug-in is designed to do? Or do you have a particular group of > > plug-in applications in mind that would interact with LimeWire in a > > similar way? > > Yes, the API designer has to have a fair idea of the universe of > potential plug-in capabilities, and what hooks will be necessary > to enable those. But, there may be good models to emulate. I'm not > familiar with the Eclipse plug-in API, but it seems to allow a lot > of deep, varied improvements via drop-in modules. > > This project could be of interest for Limewire: > > Java Plug-in Framework (JPF) Project > http://jpf.sourceforge.net/ > > It's an Eclipse-inspired framework for adding plug-in capabilities > to other Java applications. > > - Gordon > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From agthorr at cs.uoregon.edu Tue Feb 28 21:09:51 2006 From: agthorr at cs.uoregon.edu (Daniel Stutzbach) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Interoperable distribution networks, was Re: Identify "defective" nodes In-Reply-To: <4404ACE5.5080107@cs.ucl.ac.uk> References: <4ef5fec60602281126l514789a0v67f0f4bb957137d5@mail.gmail.com> <4404ACE5.5080107@cs.ucl.ac.uk> Message-ID: <20060228210951.GW5704@cs.uoregon.edu> On Tue, Feb 28, 2006 at 08:04:53PM +0000, Michael Rogers wrote: > There seem to be at least five ways to identify files: SHA1 of the > entire file, THEX root hash, MD4 hash of the entire file (ed2k), kzhash, > and BitTorrent infohash. It seems unlikely that any of these will go > away, so I guess interoperability will depend on apps supporting all > five methods, despite the overhead... or we could work on a new standard > and then everyone would have six methods to support. ;-) No, just add all the methods to my magic Stitch library and it will take care of everything. Then they all need to be supported in only one place. ;) -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon From kmagdsick at limewire.com Tue Feb 28 22:04:33 2006 From: kmagdsick at limewire.com (Karl A. Magdsick) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: <20060228190423.GT5704@cs.uoregon.edu> References: <20060228184738.GS5704@cs.uoregon.edu> <20060228190423.GT5704@cs.uoregon.edu> Message-ID: <4404C8F1.2010005@limewire.com> You want to minimize the amount that you hurt network performance through ignorance of protocol nuances and application internals. It would be much better to be able to ask each application for sets of byte ranges that the application has downloaded/is downloading and also be able to offer each application byte ranges that it is allowed to download next. You would also want a way to ask each application for which byte ranges of the download have been verified by the application, as well as a way to notify each application of the set of bytes that have been verified by any application. With two clients, presumably you would tell one that it is responsible for the first half of the download, and tell the other application that it is allowed to fetch the second half of the download. As the download progresses, you would modify the boundary between the two programs, making sure not to allow any program to download bytes that have been downloaded by the other. You'd still have to be very careful, or you could end up really hurting network performance. You'd probably want a preview-accommodating client (such as LimeWire) to fetch the first half of the download, and have a preview-indifferent client (such as most/all BitTorrent clients) fetch the second half of the download so that the user could still preview the download. -Karl Daniel Stutzbach wrote: >On Tue, Feb 28, 2006 at 01:56:43PM -0500, Greg Bildson wrote: > > >>>Yes, true, it would also need to be able to tell the application "go >>>fetch this chunk" rather than letting the application choose. >>> >>> >>Which would likely undermine any algorithm trying to improve availability by >>downloading random or most rare chunks. >> >> > >Well, the plug-in could take over doing that part, too. > >I did mention this was my pipe-dream plug-in, right? :-) > > > From lally at vt.edu Tue Feb 28 23:37:03 2006 From: lally at vt.edu (H. Lally Singh) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Re: Identify "defective" nodes In-Reply-To: <20060228161418.GN5704@cs.uoregon.edu> References: <6.2.3.4.0.20060228105303.01f980a0@mail.cs.cornell.edu> <19196d860602280800n2674cc64s4a07565d5651fd4b@mail.gmail.com> <20060228161418.GN5704@cs.uoregon.edu> Message-ID: <7FE01058-EEAF-4602-B0C7-57B9E48B8984@vt.edu> Any java-based plugin API would still be open to you opening up a TCP connection in your plugin, as a stub back to your C++ app. Hell you could go as far as a CORBA. -- H. Lally Singh Ph.D. Candidate, Computer Science Virginia Tech lallysingh@mac.com On Feb 28, 2006, at 11:14 AM, Daniel Stutzbach wrote: > On Tue, Feb 28, 2006 at 11:00:39AM -0500, Sam Berlin wrote: >> I've been meaning for a very long time to add a plugin >> architecture to >> LimeWire, for things such as Credence and the like. If anyone >> here is >> interested in such an addition, so they can provide their own add-ons >> to the program (and be able to ship just the plugin, not the whole >> program), please let me know. I'll attempt to gather up what folks >> are looking for in an architecture and write something suitable. > > Wouldn't the API needed by plug-ins vary dramatically by what the > plug-in is designed to do? Or do you have a particular group of > plug-in applications in mind that would interact with LimeWire in a > similar way? > > I'd love it if the plug-ins ran as separate processes and communicated > with LimeWire over a local-only TCP port. That way the plug-ins > wouldn't have to be written in a particular language. > > -- > Daniel Stutzbach Computer Science Ph.D > Student > http://www.barsoom.org/~agthorr University of > Oregon > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences