From jim at at.org Mon Sep 2 08:22:01 2002 From: jim at at.org (Jim Carrico) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] p2p for indymedia Message-ID: apologies if this is perceived to be off-topic. It's not about development of technology but its deployment. I've been hosting a number of musician's sites for a few years, and recently took responsibility for an indymedia box - needless to say my bandwidth charges have gone from bad to worse. In both cases, the major culprits are a few large media files, 5 or 10 files accounting for the bulk of the traffic - something like $500 worth in July, August will be higher. Obviously we need to offload this traffic onto an 'edge network' of some kind. The subject has come up here before now, but I believe it was more of a general invitation to participate in a process whereby the IMC tech collective designs and builds their own "dream system." (See http://docs.indymedia.org/twiki/bin/view/Devel/ActiVe2/ - I'm not involved in that project at the moment so I can't really comment on their goals or progress.) I'm sure that this invitation is still open, but financial pressure forces me to look for more immediate relief. Question: does it exist? BitTorrent looks interesting, although the "upload while downloading" architecture isn't very suitable for the density of requests we're getting at the moment - the likelihood of simultaneous downloads of any one file is not particularly high (it may get to that but we'll go broke before then.) If I understand the BT docs correctly, this problem can be alleviated by convincing downloaders to leave the download "open" after completion so that others may also get the file. It's quite likely that the good will of the indymedia community can be relied upon to provide this service - any info out there on BT deployment and the general willingness of end-users to cooperate in this way? (or am I way of base here?) This sort of 'conscious background mirroring' of media files reminds me of Justin's OCN project - not sure where that's at the moment, or whether or not it's being field-tested ( - maybe we could provide a pilot project.) A point to consider is that the content that we're talking about is, or could be, released under an Open Content / Creative Commons license - this is, presumably, material that the creators want people to copy and distribute. We would be in a position to say that anyone who wants to upload audio or video to the newswire must publish it according to a license which, at minimum, explicitly allows copying/mirroring/distribution - at least on a non-commercial basis. This seems like a reasonable way to bootstrap the "provisioning" (to use Lessig's term) of the network, as part of the process of educating the public and "normalizing" the process of creating a non-scarcity-based culture. So - any thoughts on this? I'll be in SF next week and would be happy to meet and discuss options with any interested parties. - Jim Carrico From bram at gawth.com Mon Sep 2 18:13:01 2002 From: bram at gawth.com (Bram Cohen) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] next meeting on sunday Message-ID: Well, this month the 1st was a sunday, so the monthly p2p-hackers meeting is coming up soon, all the usual info. When: sunday, the 8th of september, 3pm Where: SONY metreon As usual, we'll mostly just hang out and chat. -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From bram at gawth.com Mon Sep 2 18:18:01 2002 From: bram at gawth.com (Bram Cohen) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] p2p for indymedia In-Reply-To: Message-ID: Jim Carrico wrote: > apologies if this is perceived to be off-topic. It's not about > development of technology but its deployment. That's quite on topic. > I've been hosting a number of musician's sites for a few years, and > recently took responsibility for an indymedia box - needless to say my > bandwidth charges have gone from bad to worse. In both cases, the major > culprits are a few large media files, 5 or 10 files accounting for the > bulk of the traffic - something like $500 worth in July, August will be > higher. Obviously we need to offload this traffic onto an 'edge > network' of some kind. [snip] > BitTorrent looks interesting, although the "upload while downloading" > architecture isn't very suitable for the density of requests we're > getting at the moment - the likelihood of simultaneous downloads of any > one file is not particularly high That seems to contradict what you said earlier about a handful of files taking up most of the bandwidth. Could you give real numbers of the size of those files, the number of downloads they get and their average rate of download? I'd expect putting up just the most popular files using BitTorrent would work quite well. > http://docs.indymedia.org/twiki/bin/view/Devel/ActiVe2/ That's apparently an invalid url -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From jim at at.org Mon Sep 2 22:43:01 2002 From: jim at at.org (Jim Carrico) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] p2p for indymedia In-Reply-To: Message-ID: On Monday, September 2, 2002, at 06:17 PM, Bram Cohen wrote: > Jim Carrico wrote: >> the likelihood of simultaneous downloads of any >> one file is not particularly high > > That seems to contradict what you said earlier about a handful of files > taking up most of the bandwidth. Could you give real numbers of the size > of those files, the number of downloads they get and their average rate > of > download? > here's some relevant stats (one month): Top 10 of 6762 Total URLs By KBytes # Hits KBytes URL 1 1110 0.13% 11047818 29.46% /uploads/bushpdx.rm 2 314 0.04% 1671779 4.46% /uploads/greg_palast_part_1.mov 3 278 0.03% 1481739 3.95% /uploads/the_wholesome_undie.mov 4 38819 4.57% 1437272 3.83% / 5 328 0.04% 1204566 3.21% /uploads/chrisitine_maggiore.mp3 6 186 0.02% 1148474 3.06% /uploads/extinction_stinks-hi.rm 7 452 0.05% 1031280 2.75% /uploads/anti-poverty.mov 8 38519 4.54% 907150 2.42% /news/ 9 128 0.02% 741278 1.98% /uploads/greg_palast3.mov 10 73 0.01% 702909 1.87% /uploads/neil_brooks.mov that's an average of only 37 d/l's per day for the *most popular* file - the others are in the 5-10/day range or less - the transfers overlap occasionally but usually not. in this case, 8 files make up over 50% of the traffic - it adds up... in peak periods bitTorrent would certainly help, and i'm willing to give it a try, but appealing to downloaders to mirror the files *after* they've got the whole thing seems like the way to go. If it's as easy as leaving a window open I think there's a good chance for a reasonable number of users to comply. > >> http://docs.indymedia.org/twiki/bin/view/Devel/ActiVe2/ > > That's apparently an invalid url > sorry - no trailing slash. -JC From bram at gawth.com Mon Sep 2 23:15:02 2002 From: bram at gawth.com (Bram Cohen) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] p2p for indymedia In-Reply-To: Message-ID: Jim Carrico wrote: > here's some relevant stats (one month): > > Top 10 of 6762 Total URLs By KBytes > # Hits KBytes URL > 1 1110 0.13% 11047818 29.46% /uploads/bushpdx.rm > 2 314 0.04% 1671779 4.46% /uploads/greg_palast_part_1.mov > 3 278 0.03% 1481739 3.95% /uploads/the_wholesome_undie.mov > 4 38819 4.57% 1437272 3.83% / > 5 328 0.04% 1204566 3.21% /uploads/chrisitine_maggiore.mp3 > 6 186 0.02% 1148474 3.06% /uploads/extinction_stinks-hi.rm > 7 452 0.05% 1031280 2.75% /uploads/anti-poverty.mov > 8 38519 4.54% 907150 2.42% /news/ > 9 128 0.02% 741278 1.98% /uploads/greg_palast3.mov > 10 73 0.01% 702909 1.87% /uploads/neil_brooks.mov Multiplying your hit counts by file sizes gives very different relative percentages of total than you have there, especially for numbers 4 and 8, but glossing over that for the moment... Even the smallest of those is over 700 megs, which is about a 10 hour download going at a respectable clip of 20 k/sec. Of your eight most popular files, I think all probably have 10 or more concurrent downloaders at most times. Since the sum of your top eight percentages is about 57%, this would probably wind up saving about half your bandwidth costs, which is, ahem, not bad. > in peak periods bitTorrent would certainly help, and i'm willing to give > it a try, but appealing to downloaders to mirror the files *after* > they've got the whole thing seems like the way to go. If it's as easy > as leaving a window open I think there's a good chance for a reasonable > number of users to comply. BitTorrent does, indeed, do that. In practice about one in ten downloaders is left connected for an extended time after it's done downloading. You can often stop serving a particular file after a few downloads are complete and downloaders can't even tell. > >> http://docs.indymedia.org/twiki/bin/view/Devel/ActiVe2/ > > > > That's apparently an invalid url > > sorry - no trailing slash. http://docs.indymedia.org/twiki/bin/view/Devel/ActiVe2 from the page - "Active2 (codenamed "CNNSlayer") - a free, open-source codebase project for the Indymedia network and its allies that would provide 'next-generation' trusted multi-language open publishing including enhanced security & privacy, customization, syndication, threaded discussions and distributed (p2p) media storage, searching and streaming..." With only a bit of cynicism, I'm going to guess that they've bit off more than they can chew. -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From jim at at.org Tue Sep 3 06:47:01 2002 From: jim at at.org (Jim Carrico) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] p2p for indymedia In-Reply-To: Message-ID: <9395B014-BF43-11D6-B4CE-0003935AE71A@at.org> On Monday, September 2, 2002, at 11:14 PM, Bram Cohen wrote: > Jim Carrico wrote: >> 1 1110 0.13% 11047818 29.46% /uploads/bushpdx.rm > > > Even the smallest of those is over 700 megs, which is about a 10 hour > download going at a respectable clip of 20 k/sec. actually the 4th column is total traffic for the file in k - the top file is only about 10 megs in size. >> in peak periods bitTorrent would certainly help, and i'm willing to >> give >> it a try, but appealing to downloaders to mirror the files *after* >> they've got the whole thing seems like the way to go. If it's as easy >> as leaving a window open I think there's a good chance for a reasonable >> number of users to comply. > > BitTorrent does, indeed, do that. In practice about one in ten > downloaders > is left connected for an extended time after it's done downloading. You > can often stop serving a particular file after a few downloads are > complete and downloaders can't even tell. > alrighty then! > > "Active2 (codenamed "CNNSlayer") - a free, open-source codebase project > for the Indymedia network and its allies that would provide > 'next-generation' trusted multi-language open publishing including > enhanced security & privacy, customization, syndication, threaded > discussions and distributed (p2p) media storage, searching and > streaming..." > > With only a bit of cynicism, I'm going to guess that they've bit off > more > than they can chew. > precisely my point in asking you guys what's already working... -J From bram at gawth.com Tue Sep 3 09:20:01 2002 From: bram at gawth.com (Bram Cohen) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] p2p for indymedia In-Reply-To: <9395B014-BF43-11D6-B4CE-0003935AE71A@at.org> Message-ID: Jim Carrico wrote: > On Monday, September 2, 2002, at 11:14 PM, Bram Cohen wrote: > > > Jim Carrico wrote: > >> 1 1110 0.13% 11047818 29.46% /uploads/bushpdx.rm > > > > Even the smallest of those is over 700 megs, which is about a 10 hour > > download going at a respectable clip of 20 k/sec. > > actually the 4th column is total traffic for the file in k - the top > file is only about 10 megs in size. Ah, never mind the analysis I gave then - only your top file is likely to have much in the way of concurrent downloaders, and even that one not too much. -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From Franck.Cappello at lri.fr Tue Sep 3 09:56:01 2002 From: Franck.Cappello at lri.fr (fci) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] International Workshop on Global and P2P Computing 2003 - Call For Papers Message-ID: <000001c25368$70fc1cb0$cc07af81@Vaio2> Dear Colleagues, Please find bellow the call for papers of the "International Workshop on Global and Peer-to-Peer Computing" which will take place along with the CCGRID 2003 conference in Tokyo, Japan, May 2003. We take this opportunity to invite you to submit scientific papers to this workshop. Please excuse use if you receive several copies of this message. Feel free to forward this message to your colleagues. Best Regards. Please accept our apologies, if you have received multiple copies of this e-mail --------------------------------------------------------------------- CALL FOR PAPERS Third International Scientific Workshop on GLOBAL and PEER-TO-PEER Computing (http://www.lri.fr/~fci/GP2PC) organized at the IEEE/ACM International Symposium on Cluster Computing and the Grid 2003 IEEE/ACM CCGRID 2003 Toshi Center Hotel Tokyo, Japan, May 2003 In cooperation with the IEEE Task Force on Cluster Computing (TFCC) --------------------------------------------------------------------- SCOPE Global Computing and Peer-to-Peer systems allow running distributed computations and exchange document/data files on many devices, which can be flexibly and dynamically connected with each other via industry and university level networks or via the Internet. Because of size, autonomy and high volatility of their resources, Global Computing and P2P platforms provide the opportunity for researchers to revisit major fields of Distributed computing: protocols, infrastructures, security, certification, fault tolerance, scheduling, performance, etc. Moreover, new issues concerning installation, utilization, flexibility and scalability are raised by the current trend for the convergence of GRID and P2P systems. Authors are invited to submit original, unpublished work describing current research and novel ideas in the area of Global and P2P Computing as well as applications from science and industry that demonstrate how Global and Peer-to-Peer Computing technology can be effectively deployed. TOPICS Topics of interest include, but are not limited to: * Global Computing and Peer-to-Peer computing platforms * P2P Merging / Interoperability with GRID systems (OGSA/I) * Software technologies evaluation: Web Services, GRID services * Middleware, programming models, environments and toolkits * Protocols for resource management/discovery/reservation/scheduling * Economic considerations of resource usage (protocols, accounting) * Storage in Global Computing Infrastructures (strategies, protocols) * Performance monitoring, benchmarking, evaluation and modelling of Global Computing and Peer-to-Peer systems and/or components thereof * Security, management and monitoring of resources * Result certification (detection/tolerance of corrupted results) * Parallel computing on large scale distributed systems * Compute & I/O driven applications (scientific, engineering, business) * Global and P2P computing applications (programmed from scratch, ported from sequential, or parallel version, adaptations to fit a global computing environment) PAPER SUBMISSION Authors are encouraged to: Submit a full paper (max: 6 pages, IEEE format). Use a minimum of 10pt font, and printable on A4 paper. Please visit the Workshop web page at http://www.lri.fr/~fci/GP2PC for paper submission procedure. IMPORTANT DATES Papers submition: November 24, 2002 Notification to authors: January 10, 2003 Final version of papers due: February 21, 2003 PROGRAM COMMITTEE (tentative): David Anderson, University of Berkeley, SETI@home project, USA Mark Baker, Division of Computer Science, Univ. of Portsmouth, UK Micah Beck, University of Tennessee, USA Franck Cappello, CNRS, Paris-South University, France Henri Casanova, SDSC, California, USA Ian Foster, University of Chicago and ANL, USA Christian Huitema, Microsoft, USA Spyros Lalis, ICS-FORTH, University of Thessaly, Greece Olivier Richard, INRIA ID, Grenoble, France Mitsuhisa Sato, CCP, University of Tsukuba, Japan Juan Carlos Soto, SUN, Jxta, USA David Skillicorn, Queen's University, Canada WORKSHOP CHAIRS (and contact for info) Franck Cappello, Spyros Lalis, CNRS, Universite Paris-Sud University of Thessaly and ICS-FORTH France, Greece, fci@lri.lri.fr lalis@ics.forth.gr --------------------------------------------------------------------- --- Franck Cappello fci@lri.fr Head Cluster & GRID Group www.lri.fr/~fci LRI, Universit? Paris Sud tel:+33 1 691 570 91 91405, ORsay Cedex fax:+33 1 691 565 86 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20020903/c013148b/attachment.html From kedarb at cse.iitb.ac.in Tue Sep 3 13:57:01 2002 From: kedarb at cse.iitb.ac.in (Kedar Bellare) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Quota and Reputation in P2P In-Reply-To: Message-ID: Hi I am designing a P2P system for distributed backup of files on an untrusted network. I want to limit the size a person can store on the network to be equal or a litter more than the space he is giving to the system. Is there a method of implementing this? Also can I design a reputation system that gives me a great deal of reliability? Thanx --kedar f u cn rd ths, u r prbbly a lsy spllr From bram at gawth.com Tue Sep 3 14:15:01 2002 From: bram at gawth.com (Bram Cohen) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Quota and Reputation in P2P In-Reply-To: Message-ID: Kedar Bellare wrote: > I am designing a P2P system for distributed backup of files on an > untrusted network. I want to limit the size a person can store on the > network to be equal or a litter more than the space he is giving to the > system. Is there a method of implementing this? The best way is to make each peer directly trade storage space with a few specific other peers it has long-term relationships with. > Also can I design a reputation system that gives me a great deal of > reliability? Reputation is a very hard problem. Your best bet is to redundantly store data to a few places, so that if one goes down the others are still up. -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From arma at mit.edu Tue Sep 3 14:21:01 2002 From: arma at mit.edu (Roger Dingledine) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Quota and Reputation in P2P In-Reply-To: ; from kedarb@cse.iitb.ac.in on Wed, Sep 04, 2002 at 02:29:37AM +0530 References: Message-ID: <20020903172021.N3264@moria.seul.org> On Wed, Sep 04, 2002 at 02:29:37AM +0530, Kedar Bellare wrote: > Hi > I am designing a P2P system for distributed backup of files on an > untrusted network. I want to limit the size a person can store on the > network to be equal or a litter more than the space he is giving to the > system. Is there a method of implementing this? Also can I design a > reputation system that gives me a great deal of reliability? This is a simpler version of the Free Haven [1] problem. Basically, in Free Haven we want to do this while maintaining censorship-resistance. This implies hiding the location of each server. In order to have any hope of solving this, you want to start with the following: * Figure out a mechanism to verify that somebody is following through with a promise to store a file. Make sure this mechanism is publicly verifiable in some way, and that "enough" people periodically verify it that you're comfortable. Our paper at CFP2002 [2] has a section on Free Haven that describes overall pitfalls with using a 'gossip' system rather than verifiable transactions in such a setting. * Figure out how to maintain strong identities for your participants. They may be untrusted, but are there few enough that you can keep track of the actual human beings involved? The worry here is that people will join the network, take some resources, and leave without paying them back. A post on the freehaven-dev list from April [3] provides a short overview of the "reputation vs risk management" issue. (On the other hand, if you require new users to provide lots of resources before you trust them, a) the best strategy is to screw new users, and b) it's very hard to get critical mass of users.) There are several other approaches people have taken to this problem. A common one is micropayments, a la what Mojo Nation used to be. The basic idea is that you trade around tokens for work, and if you've got a token then presumably you are owed some work. You still need to address the two points above, though. If you want an academic side on this idea, take a look at the Fileteller paper from [4], plus the things in its bibliography. The Tangler paper [5] includes a nice discussion of enforcing quota on pseudonymous untrusted peers by requiring people to periodically republish. Hope this helps as a first step. This is a field that is picking up speed these days. --Roger [1] http://freehaven.net/ -- you can get a reasonably good overview at http://freehaven.net/doc/oreilly/freehaven-ch12.html [2] http://freehaven.net/doc/cfp02/cfp02.html [3] http://archives.seul.org//freehaven/dev/Apr-2002/msg00000.html [4] http://ifca.ai/fc02/program.html, and google for Fileteller [5] http://www.scs.cs.nyu.edu/tangler/ From jeff at platypus.ro Tue Sep 3 14:55:01 2002 From: jeff at platypus.ro (Jeff Darcy) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Quota and Reputation in P2P References: Message-ID: <005b01c25394$80ade360$6601a8c0@DEAGOL> > I am designing a P2P system for distributed backup of files on an > untrusted network. I want to limit the size a person can store on the > network to be equal or a litter more than the space he is giving to the > system. Is there a method of implementing this? Also can I design a > reputation system that gives me a great deal of reliability? The eventual answer will probably be related to how you find peers to store stuff for you, which you don't specify. Your basic problem is that when you want someone else to store stuff for you - whether known to you or otherwise - you need to have them know that you have sufficient credit/mojo/whatever to justify their doing so. The simple case, which I'd almost call a degenerate case, would be to establish credit directly with the node providing the storage as Bram suggests. If you want to get fancier, you need some indirect way to "convince" them that you're good for it, and two general approaches suggest themselves: (1) Have them ask someone else that they trust and who can vouch for you, in classic "web of trust" style. (2) Provide some unforgeable "payment" along with the data to be stored. The two options actually end up being very similar in some ways. Establishing the validity of a payment ultimately involves contacting one or more trusted nodes, rejection of forgery and duplicates works much the same way, etc. However, validating the payment rather than the node offering payment offers a little bit more flexibility with regard to preserving anonymity, avoiding single points of failure, etc. Another possibility, which would be a little bit more difficult under a strong-identity validate-the-user approach, would be to support that little bit of free quota for new users by having a "generous" but generally-trusted node provide those first few tokens out of its own store. It's kind of like co-signing a loan; other nodes don't need to know that the actual newcomer itself is good for the credit, because they're effectively backed up by the generous node. As I said, though, a lot depends on what your distribution methods (and anonymity goals) look like. Maybe if you could describe those a little we could brainstorm more about how specific solutions might apply to your situation. From justin at chapweske.com Tue Sep 3 19:02:01 2002 From: justin at chapweske.com (Justin Chapweske) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] p2p for indymedia References: Message-ID: <3D756981.40401@chapweske.com> The OCN will be made available on a wider scale Real Soon Now. The remaining hurdles are policy/legal rather than technical. The next step is the public release of the CAW Gateway Server which will make it trivial to P2P-enable arbitrary web-content. Also, there are currently a couple of THEX implementations in progress which should make it easier to CAW-enable third party applications such as the various Gnutella clients. In the mean time, something like Shareaza which supports the Content-Addressable Web's sister, HUGE, and MAGNET-URI might be able to help you out in the short term. Hope this helps, -Justin > > This sort of 'conscious background mirroring' of media files reminds me > of Justin's OCN project - not sure where that's at the moment, or > whether or not it's being field-tested ( - maybe we could provide a > pilot project.) A point to consider is that the content that we're > talking about is, or could be, released under an Open Content / Creative > Commons license - this is, presumably, material that the creators want > people to copy and distribute. We would be in a position to say that > anyone who wants to upload audio or video to the newswire must publish > it according to a license which, at minimum, explicitly allows > copying/mirroring/distribution - at least on a non-commercial basis. > This seems like a reasonable way to bootstrap the "provisioning" (to use > Lessig's term) of the network, as part of the process of educating the > public and "normalizing" the process of creating a non-scarcity-based > culture. > > So - any thoughts on this? I'll be in SF next week and would be happy to > meet and discuss options with any interested parties. > > - Jim Carrico > > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers -- Justin Chapweske, Onion Networks http://onionnetworks.com/ From gojomo at usa.net Tue Sep 3 20:14:01 2002 From: gojomo at usa.net (Gordon Mohr) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] p2p for indymedia References: <3D756981.40401@chapweske.com> Message-ID: <020c01c253c1$1300d390$640a000a@golden> Justin writes: > The OCN will be made available on a wider scale Real Soon Now. The > remaining hurdles are policy/legal rather than technical. The next step > is the public release of the CAW Gateway Server which will make it > trivial to P2P-enable arbitrary web-content. Great to hear the public release is coming soon! Indymedia seems like one example of akind of project where spirited community members might just want to donate some % of their disk/bandwidth for helping to mirror whatever's popular. How possible would it be for someone throw together a "slaved" CAW cache utility, to deploy as a CGI on a remote web space or as a background process on a home machine, that would take a cure from a "master" location to mirror content on demand? (I'm thinking: slave informs master that it's got X MB of space available. Master, when stressed, tells fetchers to get material from slave -- even though slave doesn't yet really have it. Slave gets request, pretends it has it, proxies to master on the fly, caching material for subsequent requests.) > Also, there are currently a couple of THEX implementations in progress > which should make it easier to CAW-enable third party applications such > as the various Gnutella clients. > > In the mean time, something like Shareaza which supports the > Content-Addressable Web's sister, HUGE, and MAGNET-URI might be able to > help you out in the short term. There's also some work to add a Chord find-by-hash resolving system to ultrapeers, in the Limewire code base, which should help turbocharge hash-based queries between Gnutella nodes. - Gojomo ____________________ Gordon Mohr Bitzi CTO . . . describe and discover files of every kind. _ http://bitzi.com _ . . . Bitzi knows bits -- because you teach it! From lgonze at panix.com Wed Sep 4 08:27:02 2002 From: lgonze at panix.com (Lucas Gonze) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] p2p for indymedia In-Reply-To: <020c01c253c1$1300d390$640a000a@golden> Message-ID: Justin -- > > Also, there are currently a couple of THEX implementations in progress > > which should make it easier to CAW-enable third party applications such > > as the various Gnutella clients. > > > > In the mean time, something like Shareaza which supports the > > Content-Addressable Web's sister, HUGE, and MAGNET-URI might be able to > > help you out in the short term. Looks like I'm falling behind on jargon. What's THEX? What's MAGNET-URI? Thanks. - Lucas From gojomo at usa.net Wed Sep 4 09:35:01 2002 From: gojomo at usa.net (Gordon Mohr) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] p2p for indymedia References: Message-ID: <005501c25430$f72bc740$640a000a@golden> Lucas Gonze writes: > Looks like I'm falling behind on jargon. What's THEX? What's MAGNET-URI? THEX: Tree Hash EXchange format, a format for sharing full or partial hash trees which allow content subranges to be verified as belonging to the desired whole, before the entire content has arrived. Helpful for supporting parallel downloads from many sources, and resharing of partial content before the full file is available. See: http://www.open-content.net/specs/draft-jchapweske-thex-01.html MAGNET-URI: A URI scheme for handing off exact file identifiers to helper programs, for example from a web page to a P2P program. In one way, it could be thought of as a vendor- and project-neutral generalization of the "freenet:" and "ed2k:" URI-schemes used by the Freenet and EDonkey2000 peer-to-peer networks, respectively. See: http://magnet-uri.sf.net - Gojomo ____________________ Gordon Mohr Bitzi CTO . . . describe and discover files of every kind. _ http://bitzi.com _ . . . Bitzi knows bits -- because you teach it! From lgonze at panix.com Wed Sep 4 10:49:01 2002 From: lgonze at panix.com (Lucas Gonze) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] p2p for indymedia In-Reply-To: <005501c25430$f72bc740$640a000a@golden> Message-ID: Thanks, Gordon. - Lucas > THEX: Tree Hash EXchange format, a format for sharing full or > partial hash trees which allow content subranges to be verified > as belonging to the desired whole, before the entire content > has arrived. Helpful for supporting parallel downloads from many > sources, and resharing of partial content before the full file > is available. See: > > http://www.open-content.net/specs/draft-jchapweske-thex-01.html > > MAGNET-URI: A URI scheme for handing off exact file identifiers to > helper programs, for example from a web page to a P2P program. In > one way, it could be thought of as a vendor- and project-neutral > generalization of the "freenet:" and "ed2k:" URI-schemes used by > the Freenet and EDonkey2000 peer-to-peer networks, respectively. > See: > > http://magnet-uri.sf.net From bram at gawth.com Sat Sep 7 20:37:02 2002 From: bram at gawth.com (Bram Cohen) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] reminder: meeting tomorrow, the 8th Message-ID: Remember, there's a meeting tommorow, the 8th, 3pm, the metreon. -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From lgonze at panix.com Thu Sep 12 06:58:02 2002 From: lgonze at panix.com (Lucas Gonze) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] p2p for indymedia In-Reply-To: <9395B014-BF43-11D6-B4CE-0003935AE71A@at.org> Message-ID: Jim, This is an integration problem rather than a technology problem. There's no reason to stick to any one edge solution -- the opposite. You need to (1) defer download requests to edge servers wherever possible and (2) cause downloaded files to be made available via an edge server wherever possible. For example: *** A surfer visiting indymedia.org requests a media file like a news clip. Indymedia.org is maintaining connections over the gnutellanet; it does a search for the file; if the file is found, the indymedia.org web server does an HTTP redirect to the download location on the gnutellanet. *** That solves #1 (Murphy willing), but not #2. To get #2, you need to have the downloaded files inserted into the user's filesharer. Again using the gnutella example: *** After downloading the file from the HTTP redirect location, the user makes the file available via her own servent. If the user downloaded to a shared directory available via Gnutella, this will happen automatically. Otherwise the user will need to move the file to a shared directory. *** Given the above definition of the solution, it's easy to outline steps to implement it. For #1 you can run a scriptable Gnutella servent (Kevin Pritchard at Indymedia NYC is a good resource to talk to about this); have the servent maintain connections to the gnet; have a cgi script that looks for indymedia files via gnet and returns them to the user in a browser-friendly format. For #2 you can just put text on the download page that specifically asks users to download to a shared directory. That solution is suboptimal in plenty of ways, e.g. that gnutella is painful to use, but most open p2p networks should support something like it. The gist of it is that indymedia should piggyback on whatever p2p software people already have installed, and there will be some one-by-one process where you implement links to each p2p network in the laziest way possible. I'll bet that you can get help from each networks' developers, since most would be ecstatic to have a usecase that's so clearly tied to political speech. - Lucas From zooko at zooko.com Sun Sep 15 15:48:01 2002 From: zooko at zooko.com (Zooko) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] (no subject) Message-ID: The Mnet Project is pleased to announce v0.5.1-STABLE of Mnet. Mnet is a "universal file space" -- a global space in which you can store and retrieve files. The contents of the universal file space are independent of any particular server. It comes with a GUI file browser that looks a bit like a classical file-sharing tool such as Napster. Please visit the download page for precompiled packages for Linux, FreeBSD, Solaris, and Windows as well as instructions for building from source: http://mnet.sf.net/download.php More information about Mnet is available on the project web page: http://mnet.sf.net/ This release is the last planned released from the v0.5-STABLE branch of the Mnet source code. There are many known problems in this release which will be fixed in the next release, which will be named Mnet v0.6. Known problems in this release: * Downloading a file sometimes pauses for 15 minutes. * Downloading a file sometimes hangs indefinitely. * Files are not very persistent -- they vanish from the filespace. Please view the ChangeLog for more details: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/mnet/mnet/ChangeLog?rev=HEAD&content-type=text/vnd.viewcvs-markup Please try Mnet out and report bugs via e-mail to , or by using the SourceForge bug tracker: http://sourceforge.net/tracker/index.php?group_id=43482&atid=436453 The availability and persistence of files is influenced by how stable the servers are. If you run a stable Mnet server it will help. Regards, Zooko Lead Developer, Mnet Project From bradneuberg at yahoo.com Mon Sep 16 14:31:01 2002 From: bradneuberg at yahoo.com (Brad Neuberg) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Distributed Indexes + FEC = Message-ID: <20020916213057.90825.qmail@web14103.mail.yahoo.com> I'm interested in hearing people's opinions on how distributed indexes, such as Chord and Pastry, and using Forward Error Correction (FEC) can be combined. What is the most appropriate way to do this? Is it even appropriate? Has anyone done so and how is the performance and scalability of such a combination? Thanks, Brad Neuberg From z2258936 at student.unsw.edu.au Tue Sep 17 23:07:01 2002 From: z2258936 at student.unsw.edu.au (Iram Mahboob) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] [p2p] In-Reply-To: <20020917102842.651.96335.Mailman@capsicum.zgp.org> Message-ID: <5.1.0.14.0.20020918155823.00b799f8@pop3.student.unsw.edu.au> Hi all, The two p2p application that I know of are file sharing (Gnutella, Pastry etc. ) and multicast (YOID, end system multicast etc.). Are there any other applications. Could you please tell me their names. Iram At 03:28 17/09/2002 -0700, you wrote: >p2p-hackers@zgp.org From schmitz at aifb.uni-karlsruhe.de Wed Sep 18 03:01:02 2002 From: schmitz at aifb.uni-karlsruhe.de (Christoph Schmitz) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] [p2p] In-Reply-To: <5.1.0.14.0.20020918155823.00b799f8@pop3.student.unsw.edu.au> References: <20020917102842.651.96335.Mailman@capsicum.zgp.org> Message-ID: <3D88432B.19452.389E138@localhost> Hi, if you follow the definition of "using resources at the edges of the net", distributed computation a la Seti@Home would be P2P as well. See Andy Oram's Book for more examples. Best Regards, Christoph On 18 Sep 2002 at 16:06, Iram Mahboob wrote: > Hi all, > The two p2p application that I know of are file sharing (Gnutella, > Pastry etc. ) and multicast (YOID, end system multicast etc.). Are > there any other applications. Could you please tell me their names. > Iram > > At 03:28 17/09/2002 -0700, you wrote: > >p2p-hackers@zgp.org > > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > -- -- Christoph Schmitz -- AIFB Karlsruhe - Tel. 0721/608 6558 From rrrw at neofonie.de Wed Sep 18 04:22:01 2002 From: rrrw at neofonie.de (Ronald Wertlen) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] [p2p] In-Reply-To: <22160000.1032346603@[172.27.40.119]> References: <20020918102840.25250.85650.Mailman@capsicum.zgp.org> <22160000.1032346603@[172.27.40.119]> Message-ID: <25800000.1032348101@[172.27.40.119]> Hi, Just a note to Pastry: Pastry AFAIK is a routing algorithm with a prototype which is a file sharing app, one could also implement the Pastry method otherwise (JXTA Walkers?). Andy Oram's book is definitely an excellent intro to the topic. http://safari.oreilly.com/main.asp?bookname=peertopeer guess I should finish reading it myself... :-/ cheers, ron --On Wednesday, September 18, 2002 03:28:40 -0700 p2p-hackers-request@zgp.org wrote: > From: "Christoph Schmitz" > To: p2p-hackers@zgp.org > Date: Wed, 18 Sep 2002 09:11:07 +0200 > Subject: Re: [p2p-hackers] [p2p] > Reply-To: p2p-hackers@zgp.org > > Hi, > > if you follow the definition of "using resources at the edges of the > net", distributed computation a la Seti@Home would be P2P as well. > See Andy Oram's Book for more examples. > > Best Regards, > Christoph > > On 18 Sep 2002 at 16:06, Iram Mahboob wrote: > >> Hi all, >> The two p2p application that I know of are file sharing (Gnutella, >> Pastry etc. ) and multicast (YOID, end system multicast etc.). Are >> there any other applications. Could you please tell me their names. >> Iram >> >> At 03:28 17/09/2002 -0700, you wrote: >> > p2p-hackers@zgp.org >> >> >> _______________________________________________ >> p2p-hackers mailing list >> p2p-hackers@zgp.org >> http://zgp.org/mailman/listinfo/p2p-hackers >> > > > > > -- > -- Christoph Schmitz > -- AIFB Karlsruhe - Tel. 0721/608 6558 > From bdarla at KOM.tu-darmstadt.de Wed Sep 18 06:10:01 2002 From: bdarla at KOM.tu-darmstadt.de (Vasilios Darlagiannis) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] P2P CfP Message-ID: <3D887AF6.5060703@kom.tu-darmstadt.de> Dear friends, Please, feel free to redistribute the following CfP. Vasilis ------------ Special Issue on Peer-to-Peer Systems Call for Papers organized by the Journal PIK Peer-to-Peer has emerged as a promising new paradigm for distributed computing, with symmetric communication, as opposed to the Client-Server paradigm. Peer-to-Peer systems are characterized as being decentralized, self-organizing distributed systems, which received a special attention lately. The popularity of Peer-to-Peer networks, which can already be observed in IP-traffic measurements, stems certainly from Internet file sharing domain. However the Peer-to-Peer mindset can be employed beyond these content distribution applications, as in the area of bidirectional communication and collaboration, distributed computing and messaging. Thus the new perspective of Peer-to-Peer networking holds new challenges, e.g. concerning the traffic management in mobile and fixed networks hosted by Internet Service Providers, as well as in enterprise networks. On the other hand an increased flexibility and a higher fault tolerance of the overall system, and an increased network value are only a few possible advantages connected to the idea of Peer-to-Peer networking. The goal of the Special Issue is to present novel peer-to-peer technologies, protocols, applications and systems, and also to identify key research issues and challenges that lie ahead. Topics of interest include, but are not limited to: ? Novel Peer-to-Peer applications and systems ? Peer-to-Peer service development ? Peer-to-Peer infrastructure and overlay networks ? Protocols for resource management/discovery/reservation/scheduling ? Security and anti-censorship in Peer-to-Peer systems ? Performance and measurements issues of Peer-to-Peer systems ? Fault tolerance, scalability, availability, accessibility in Peer-to-Peer networks ? Quality of Service and billing/accounting issues in Peer-to-Peer networks Guidelines for Submission: Submissions are expected to consist of not more than 6 pages, with at most 6000 characters per page. The text can be written either in English or in German. Authors are invited to submit by email a single electronic version of their paper to the address p2p@lkn.ei.tum.de . Both Postscript and PDF formats are the only acceptable formats. Important Dates: Papers due: October 15, 2002 Notification to authors: November 15, 2002 Camera ready papers due: December 15, 2002 Guest Editors: J?rg Ebersp?cher, LKN, Technische Universit?t M?nchen Ralf Steinmetz, KOM, Technische Universit?t Darmstadt -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20020918/8a6ff8fb/attachment.htm From bram at gawth.com Sat Sep 21 21:51:02 2002 From: bram at gawth.com (Bram Cohen) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Next p2p-hackers meeting, october 13th Message-ID: The next p2p-hackers meeting, as per our regular calendar, will be October 13th, 3pm, the metreon, san francisco. It happens to be the day after my birthday. If you'd like to come to my birthday party, and think I'd like to invite you, then mail me and I'll send you an invite. -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From mfreed at cs.nyu.edu Mon Sep 23 10:04:02 2002 From: mfreed at cs.nyu.edu (Michael J. Freedman) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] New paper on peer-to-peer anonymizing IP layer Message-ID: Hi p2p hackers, Please find below our recent paper to be published at CCS 2002, entitled: "Tarzan: A Peer-to-Peer Anonymizing Network Layer." http://pdos.lcs.mit.edu/tarzan/docs/tarzan-ccs02.ps http://pdos.lcs.mit.edu/tarzan/docs/tarzan-ccs02.pdf One of the main advantages of a p2p model for anonymity is that it removes a network-edge to attack. The paper gives a fairly comprehensive discussion of the additional complexities and security risks of an open-admission system, as well as techniques with which to handle these risks. Two main points of interest: -- secure random selection protocol for choosing nodes -- cover traffic within restricted topology, but covering all users without n^2 complexity The former is especially applicable to many more peer-to-peer applications than merely those centered about anonymity. Thanks, --mike ABSTRACT: Tarzan is a peer-to-peer anonymous IP network overlay. Because it provides IP service, Tarzan is general-purpose and transparent to applications. Organized as a decentralized peer-to-peer overlay, Tarzan is fault-tolerant, highly scalable, and easy to manage. Tarzan achieves its anonymity with layered encryption and multi-hop routing, much like a Chaumian mix. A message initiator chooses a path of peers pseudo-randomly through a restricted topology in a way that adversaries cannot easily influence. Cover traffic prevents a global observer from using traffic analysis to identify an initiator. Protocols toward unbiased peer-selection offer new directions for distributing trust among untrusted entities. Tarzan provides anonymity to either clients or servers, without requiring that both participate. In both cases, Tarzan uses a network address translator (NAT) to bridge between Tarzan hosts and oblivious Internet hosts. Measurements show that Tarzan imposes minimal overhead over a corresponding non-anonymous overlay route. ----- "Not all those who wander are lost." www.michaelfreedman.org From Wolfgang.Mueller2 at uni-bayreuth.de Wed Sep 25 10:52:01 2002 From: Wolfgang.Mueller2 at uni-bayreuth.de (Wolfgang Mueller) Date: Sat Dec 9 22:12:03 2006 Subject: LaTeX Style? Re: [p2p-hackers] Call for participation: IPTPS'03 In-Reply-To: <3D6FFFEA.4F849A9A@cs.berkeley.edu> References: <3D6FFFEA.4F849A9A@cs.berkeley.edu> Message-ID: <200209251951.52452.wolfgang.mueller2@uni-bayreuth.de> Am Samstag, 31. August 2002 01:29 schrieb Ion Stoica: > Please accept my apology if you receive multiple copies of this message. > > Ion > > ---- Dear Ion, From the CfP it's not quite clear what paper format and what (LaTeX) style submissions to the workshop are supposed to have. This strongly influences the meaning of the phrase "5 pages or less". Is it LNCS style or the style that was used in the first workshop? Regards, Wolfgang M?ller From Wolfgang.Mueller2 at uni-bayreuth.de Wed Sep 25 10:56:01 2002 From: Wolfgang.Mueller2 at uni-bayreuth.de (Wolfgang Mueller) Date: Sat Dec 9 22:12:03 2006 Subject: LaTeX Style? Re: [p2p-hackers] Call for participation: IPTPS'03 In-Reply-To: <200209251951.52452.wolfgang.mueller2@uni-bayreuth.de> References: <3D6FFFEA.4F849A9A@cs.berkeley.edu> <200209251951.52452.wolfgang.mueller2@uni-bayreuth.de> Message-ID: <200209251955.03205.wolfgang.mueller2@uni-bayreuth.de> Am Mittwoch, 25. September 2002 19:51 schrieb Wolfgang Mueller: > Am Samstag, 31. August 2002 01:29 schrieb Ion Stoica: > > Please accept my apology if you receive multiple copies of this message. > > > > Ion > > > > ---- > > Dear Ion, > > From the CfP it's not quite clear what paper format and what (LaTeX) style > submissions to the workshop are supposed to have. This strongly influences > the meaning of the phrase "5 pages or less". Is it LNCS style or the style > that was used in the first workshop? > > Regards, > Wolfgang M?ller > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers No, this was not intended to go to the mailing list ;-) Cheers, Wolfgang From z2258936 at student.unsw.edu.au Thu Sep 26 04:05:01 2002 From: z2258936 at student.unsw.edu.au (Iram Mahboob) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] (no subject) Message-ID: <5.1.0.14.0.20020926164449.00ba4228@pop3.student.unsw.edu.au> Hi, Are the supernodes in Fast Track elected dynamically or are they fixed (like in Napster). Can someone please send me a pointer to a document that explains the protocol. I am trying to figure out how Fast Track works. Iram Mahboob From bram at gawth.com Fri Sep 27 05:59:02 2002 From: bram at gawth.com (Bram Cohen) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Reconciling databases Message-ID: This page gives papers which explain how to reconcile two databases which only contain small differences. The technique appears emminently useful in a wide variety of applications. http://sks.sourceforge.net/ -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From wesley at felter.org Fri Sep 27 15:06:01 2002 From: wesley at felter.org (Wes Felter) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] (no subject) In-Reply-To: <5.1.0.14.0.20020926164449.00ba4228@pop3.student.unsw.edu.au> References: <5.1.0.14.0.20020926164449.00ba4228@pop3.student.unsw.edu.au> Message-ID: <1033163712.1157.12.camel@arlx031.austin.ibm.com> On Thu, 2002-09-26 at 01:48, Iram Mahboob wrote: > Hi, > Are the supernodes in Fast Track elected dynamically or are they fixed > (like in Napster). Can someone please send me a pointer to a document that > explains the protocol. I am trying to figure out how Fast Track works. I think they're elected dynamically. You should find a copy of the source code for an old version of giFT, which supported the FastTrack protocol. -- Wes Felter - wesley@felter.org - http://felter.org/wesley/ From lisarein at finetuning.com Fri Sep 27 15:16:02 2002 From: lisarein at finetuning.com (Lisa Rein) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Help With My Research In-Reply-To: <1033163712.1157.12.camel@arlx031.austin.ibm.com> Message-ID: Hi gang, Note: please reply to me offlist at lisarein@finetuning.com unless I'm wrong and this is actually an interesting topic of discussion for you guys :-) I'm just asking for a little help with something I've been working on for Creative Commons: I've been conducting research in order to determine the different methods for embedding text within file formats (eg, comment fields, etc.). I'm interested in not only the different means available for doing so (which I can pretty much grab from the specs etc. -- although the means of accessing these fields also seems to be largely application-specific), but also I'm wondering (help me, oh help me people) if such techniques are actually being implemented by users for use within software NOW. For example: Sure you can put a URL in the comment field of an MP3 file (just as easily as someone else can remove it), but does anyone know of software actually looking for information in the comment fields of MP3 files? That kinda stuff. Thanks in advance for your inevitable insights! Lisa Rein Technical Architect Creative Commons lisarein@finetuning.com P.S. Give Peace A Chance: http://www.lisarein.com/peace/ http://www.onlisareinsradar.com http://www.lisarein.com http://www.finetuning.com From tboyle at rosehill.net Fri Sep 27 16:04:01 2002 From: tboyle at rosehill.net (Todd Boyle) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Help With My Research In-Reply-To: References: <1033163712.1157.12.camel@arlx031.austin.ibm.com> Message-ID: <5.1.0.14.0.20020927155635.027d78c0@popmail.cortland.com> Hi Lisa, Jeff Kandt of the Tipster project might have some comments, altho, the project ended without solving that problem http://tipster.weblogs.com/unresolved/ http://tipster.weblogs.com/discuss/msgReader$314?mode=day todd At 03:15 PM 9/27/2002, Lisa Rein wrote: >Hi gang, > >Note: please reply to me offlist at lisarein@finetuning.com unless I'm >wrong and this is actually an interesting topic of discussion for you guys >:-) > >I'm just asking for a little help with something I've been working on for >Creative Commons: > >I've been conducting research in order to determine the different methods >for embedding text within file formats (eg, comment fields, etc.). > >I'm interested in not only the different means available for doing so (which >I can pretty much grab from the specs etc. -- although the means of >accessing these fields also seems to be largely application-specific), >but also I'm wondering (help me, oh help me people) if such techniques are >actually being implemented by users for use within software NOW. > >For example: Sure you can put a URL in the comment field of an MP3 file >(just as easily as someone else can remove it), but does anyone know of >software actually looking for information in the comment fields of MP3 >files? That kinda stuff. > >Thanks in advance for your inevitable insights! > >Lisa Rein >Technical Architect >Creative Commons >lisarein@finetuning.com > > >P.S. Give Peace A Chance: http://www.lisarein.com/peace/ > > > > >http://www.onlisareinsradar.com >http://www.lisarein.com >http://www.finetuning.com > >_______________________________________________ >p2p-hackers mailing list >p2p-hackers@zgp.org >http://zgp.org/mailman/listinfo/p2p-hackers From KuldipSingh.Pabla at sun.com Sat Sep 28 10:31:01 2002 From: KuldipSingh.Pabla at sun.com (Kuldip Singh Pabla) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Re: p2p-hackers digest, Vol 1 #198 - 4 msgs References: <20020928102839.17878.390.Mailman@capsicum.zgp.org> Message-ID: <3D95E769.9E5778BA@sun.com> Hi Lisa, JavaDocs (and similar other tools) is a very good example of generating useful information out of inlined comments. It looks for specific kind of comments, beginning with "/**", and generates documentation, user guides, programmers guides etc. An example of a tutorial generated by this technique is at http://jxme.jxta.org YOu can get more information on JavaDocs at http://java.sun.com -Kuldip p2p-hackers-request@zgp.org wrote: > > Send p2p-hackers mailing list submissions to > p2p-hackers@zgp.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://zgp.org/mailman/listinfo/p2p-hackers > or, via email, send a message with subject or body 'help' to > p2p-hackers-request@zgp.org > > You can reach the person managing the list at > p2p-hackers-admin@zgp.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of p2p-hackers digest..." > > Today's Topics: > > 1. Reconciling databases (Bram Cohen) > 2. Re: (no subject) (Wes Felter) > 3. Help With My Research (Lisa Rein) > 4. Re: Help With My Research (Todd Boyle) > > --__--__-- > > Message: 1 > Date: Fri, 27 Sep 2002 05:58:35 -0700 (PDT) > From: Bram Cohen > To: p2p-hackers@zgp.org > Subject: [p2p-hackers] Reconciling databases > Reply-To: p2p-hackers@zgp.org > > This page gives papers which explain how to reconcile two databases which > only contain small differences. The technique appears emminently useful in > a wide variety of applications. > > http://sks.sourceforge.net/ > > -Bram Cohen > > "Markets can remain irrational longer than you can remain solvent" > -- John Maynard Keynes > > --__--__-- > > Message: 2 > Subject: Re: [p2p-hackers] (no subject) > From: Wes Felter > To: p2p-hackers@zgp.org > Date: 27 Sep 2002 16:55:12 -0500 > Reply-To: p2p-hackers@zgp.org > > On Thu, 2002-09-26 at 01:48, Iram Mahboob wrote: > > Hi, > > Are the supernodes in Fast Track elected dynamically or are they fixed > > (like in Napster). Can someone please send me a pointer to a document that > > explains the protocol. I am trying to figure out how Fast Track works. > > I think they're elected dynamically. You should find a copy of the > source code for an old version of giFT, which supported the FastTrack > protocol. > > -- > Wes Felter - wesley@felter.org - http://felter.org/wesley/ > > --__--__-- > > Message: 3 > Date: Fri, 27 Sep 2002 15:15:06 -0700 > From: Lisa Rein > To: > Subject: [p2p-hackers] Help With My Research > Reply-To: p2p-hackers@zgp.org > > Hi gang, > > Note: please reply to me offlist at lisarein@finetuning.com unless I'm > wrong and this is actually an interesting topic of discussion for you guys > :-) > > I'm just asking for a little help with something I've been working on for > Creative Commons: > > I've been conducting research in order to determine the different methods > for embedding text within file formats (eg, comment fields, etc.). > > I'm interested in not only the different means available for doing so (which > I can pretty much grab from the specs etc. -- although the means of > accessing these fields also seems to be largely application-specific), > but also I'm wondering (help me, oh help me people) if such techniques are > actually being implemented by users for use within software NOW. > > For example: Sure you can put a URL in the comment field of an MP3 file > (just as easily as someone else can remove it), but does anyone know of > software actually looking for information in the comment fields of MP3 > files? That kinda stuff. > > Thanks in advance for your inevitable insights! > > Lisa Rein > Technical Architect > Creative Commons > lisarein@finetuning.com > > P.S. Give Peace A Chance: http://www.lisarein.com/peace/ > > http://www.onlisareinsradar.com > http://www.lisarein.com > http://www.finetuning.com > > --__--__-- > > Message: 4 > Date: Fri, 27 Sep 2002 15:58:39 -0700 > To: p2p-hackers@zgp.org > From: Todd Boyle > Subject: Re: [p2p-hackers] Help With My Research > Cc: jeff@scrollbar.com > Reply-To: p2p-hackers@zgp.org > > Hi Lisa, > > Jeff Kandt of the Tipster project might have some comments, > altho, the project ended without solving that problem > http://tipster.weblogs.com/unresolved/ > http://tipster.weblogs.com/discuss/msgReader$314?mode=day > > todd > > At 03:15 PM 9/27/2002, Lisa Rein wrote: > >Hi gang, > > > >Note: please reply to me offlist at lisarein@finetuning.com unless I'm > >wrong and this is actually an interesting topic of discussion for you guys > >:-) > > > >I'm just asking for a little help with something I've been working on for > >Creative Commons: > > > >I've been conducting research in order to determine the different methods > >for embedding text within file formats (eg, comment fields, etc.). > > > >I'm interested in not only the different means available for doing so (which > >I can pretty much grab from the specs etc. -- although the means of > >accessing these fields also seems to be largely application-specific), > >but also I'm wondering (help me, oh help me people) if such techniques are > >actually being implemented by users for use within software NOW. > > > >For example: Sure you can put a URL in the comment field of an MP3 file > >(just as easily as someone else can remove it), but does anyone know of > >software actually looking for information in the comment fields of MP3 > >files? That kinda stuff. > > > >Thanks in advance for your inevitable insights! > > > >Lisa Rein > >Technical Architect > >Creative Commons > >lisarein@finetuning.com > > > > > >P.S. Give Peace A Chance: http://www.lisarein.com/peace/ > > > > > > > > > >http://www.onlisareinsradar.com > >http://www.lisarein.com > >http://www.finetuning.com > > > >_______________________________________________ > >p2p-hackers mailing list > >p2p-hackers@zgp.org > >http://zgp.org/mailman/listinfo/p2p-hackers > > --__--__-- > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > > End of p2p-hackers Digest -- Kuldip Singh Pabla Peer-2-Peer Networking From arachnid at notdot.net Sat Sep 28 23:15:01 2002 From: arachnid at notdot.net (Nick Johnson) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Restrictions on number of downloads and network efficiency Message-ID: <3D969A65.6020902@notdot.net> I was pondering the setup used by many (most?) current P2P networks, and the following occurred to me: Most current P2P clients work off the assumption that the number of uploads should be relatively small, so as to allow a reasonable bandwidth for each downloader, similar to the same tactic used by traffic-heavy HTTP and FTP servers. This means that a P2P servent often has to wait long times and multiple retries before it gets it's chance to download from a specific host. For many networks, getting to download is essentially random - who gets on straight after a slot becomes free? This problem is less for files that are very popular and have spread between many hosts, as there are more sources to download from. However, the popularity again ensures that getting a free slot will be difficult. Many networks support multiple & segmented downloading, and AFAIK many of these support a client providing uploads of a segment of a file even when it is not entirely downloaded. It seems to me that a better approach would be for servents to have a very high cap on the number of concurrent uploads, which should cause more clients to be able to get the file at a slow speed immeditately, instead of fewer being able to get it at a faster speed. While the two may seem the same, a greater number of clients beginning a download means that all these clients have segments of the file, however small, and are themselves potential servers of that file. Hence, a popular file can quickly spread in segments from one overloaded servent to many interested servents. Each servent is likely to have a different part of the file, so they can all start downloading the parts they do not have from other downloaders. Hopefully this should result in faster overall downloads and a more efficient network, spreading the load better. Combined with clients that start downloads at random positions (to prevent all servents having the beginning and few having the end), and a system that provides the addresses of other recent downloaders (and parent nodes that provided the file) to speed resource discovery, I belive this system could prove significantly more efficient than current limited-availablility systems. So, you made it to the end of all this. Does anyone have any comments on the practicality of this idea? It seems good to me, but n heads are better than 1 ;). Nick Johnson From gwachob at wachob.com Sat Sep 28 23:19:01 2002 From: gwachob at wachob.com (Gabe Wachob) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Restrictions on number of downloads and network efficiency In-Reply-To: <3D969A65.6020902@notdot.net> Message-ID: See bittorrent for an implementation of the idea you are talking about. Its not quite p2p in and of itself, but I think its what you are looking for: http://www.bitconjurer.org/BitTorrent -Gabe On Sun, 29 Sep 2002, Nick Johnson wrote: > I was pondering the setup used by many (most?) current P2P networks, and > the following occurred to me: > Most current P2P clients work off the assumption that the number of > uploads should be relatively small, so as to allow a reasonable > bandwidth for each downloader, similar to the same tactic used by > traffic-heavy HTTP and FTP servers. This means that a P2P servent often > has to wait long times and multiple retries before it gets it's chance > to download from a specific host. For many networks, getting to download > is essentially random - who gets on straight after a slot becomes free? > This problem is less for files that are very popular and have spread > between many hosts, as there are more sources to download from. However, > the popularity again ensures that getting a free slot will be difficult. > Many networks support multiple & segmented downloading, and AFAIK many > of these support a client providing uploads of a segment of a file even > when it is not entirely downloaded. > It seems to me that a better approach would be for servents to have a > very high cap on the number of concurrent uploads, which should cause > more clients to be able to get the file at a slow speed immeditately, > instead of fewer being able to get it at a faster speed. While the two > may seem the same, a greater number of clients beginning a download > means that all these clients have segments of the file, however small, > and are themselves potential servers of that file. Hence, a popular file > can quickly spread in segments from one overloaded servent to many > interested servents. Each servent is likely to have a different part of > the file, so they can all start downloading the parts they do not have > from other downloaders. Hopefully this should result in faster overall > downloads and a more efficient network, spreading the load better. > Combined with clients that start downloads at random positions (to > prevent all servents having the beginning and few having the end), and a > system that provides the addresses of other recent downloaders (and > parent nodes that provided the file) to speed resource discovery, I > belive this system could prove significantly more efficient than current > limited-availablility systems. > > So, you made it to the end of all this. Does anyone have any comments on > the practicality of this idea? It seems good to me, but n heads are > better than 1 ;). > > Nick Johnson > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > -- Gabe Wachob gwachob@wachob.com Personal http://www.wachob.com Founder, WiredObjects http://www.wiredobjects.com From unitethecows at msn.com Sat Sep 28 23:23:01 2002 From: unitethecows at msn.com (Russell Sailors) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Restrictions on number of downloads and network efficiency In-Reply-To: Message-ID: <001001c26780$8b6349d0$6901a8c0@unite> Please remove me from the mailer. Thank you... -----Original Message----- From: p2p-hackers-admin@zgp.org [mailto:p2p-hackers-admin@zgp.org] On Behalf Of Gabe Wachob Sent: Sunday, September 29, 2002 1:19 AM To: p2p-hackers@zgp.org Subject: Re: [p2p-hackers] Restrictions on number of downloads and network efficiency See bittorrent for an implementation of the idea you are talking about. Its not quite p2p in and of itself, but I think its what you are looking for: http://www.bitconjurer.org/BitTorrent -Gabe On Sun, 29 Sep 2002, Nick Johnson wrote: > I was pondering the setup used by many (most?) current P2P networks, and > the following occurred to me: > Most current P2P clients work off the assumption that the number of > uploads should be relatively small, so as to allow a reasonable > bandwidth for each downloader, similar to the same tactic used by > traffic-heavy HTTP and FTP servers. This means that a P2P servent often > has to wait long times and multiple retries before it gets it's chance > to download from a specific host. For many networks, getting to download > is essentially random - who gets on straight after a slot becomes free? > This problem is less for files that are very popular and have spread > between many hosts, as there are more sources to download from. However, > the popularity again ensures that getting a free slot will be difficult. > Many networks support multiple & segmented downloading, and AFAIK many > of these support a client providing uploads of a segment of a file even > when it is not entirely downloaded. > It seems to me that a better approach would be for servents to have a > very high cap on the number of concurrent uploads, which should cause > more clients to be able to get the file at a slow speed immeditately, > instead of fewer being able to get it at a faster speed. While the two > may seem the same, a greater number of clients beginning a download > means that all these clients have segments of the file, however small, > and are themselves potential servers of that file. Hence, a popular file > can quickly spread in segments from one overloaded servent to many > interested servents. Each servent is likely to have a different part of > the file, so they can all start downloading the parts they do not have > from other downloaders. Hopefully this should result in faster overall > downloads and a more efficient network, spreading the load better. > Combined with clients that start downloads at random positions (to > prevent all servents having the beginning and few having the end), and a > system that provides the addresses of other recent downloaders (and > parent nodes that provided the file) to speed resource discovery, I > belive this system could prove significantly more efficient than current > limited-availablility systems. > > So, you made it to the end of all this. Does anyone have any comments on > the practicality of this idea? It seems good to me, but n heads are > better than 1 ;). > > Nick Johnson > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > -- Gabe Wachob gwachob@wachob.com Personal http://www.wachob.com Founder, WiredObjects http://www.wiredobjects.com _______________________________________________ p2p-hackers mailing list p2p-hackers@zgp.org http://zgp.org/mailman/listinfo/p2p-hackers From arachnid at notdot.net Sun Sep 29 01:19:01 2002 From: arachnid at notdot.net (Nick Johnson) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Restrictions on number of downloads and network efficiency References: Message-ID: <3D96B782.70607@notdot.net> I've come across BitTorrent before, but don't recall seeing a protocol link previously ;) It's certainly an interesting protocol, and I can see its strengths and advantages. And as you pointed out, it does use a system similar to what I proposed. However, what I'm suggesting was more intended for fully P2P networks - instead of a single initial provider and multiple downloaders, a system where there are multiple providers of multiple different files. Basically, what I'm asking is - would changing the 'policy' of current P2P networks to allow more simultaneous downloads at a lower speed (assuming said network allows downloads from hosts with partial content) increase their efficiency and availability? I'd personally rather be downloading something at 8k/s from multiple 'slow' clients and uploading the segments I have at a similar speed, than waiting hours for a chance to get 8-16k/s from a single client. If existing networks wouldn't take much benifit from this, would a new network, designed around this and other principles do so? Incidentally, why the use of 'bencoding', as opposed to, say, XML? Nick Johnson Gabe Wachob wrote: > See bittorrent for an implementation of the idea you are talking about. > Its not quite p2p in and of itself, but I think its what you are looking > for: > > http://www.bitconjurer.org/BitTorrent > > -Gabe > > On Sun, 29 Sep 2002, Nick Johnson wrote: > > >>I was pondering the setup used by many (most?) current P2P networks, and >>the following occurred to me: >>Most current P2P clients work off the assumption that the number of >>uploads should be relatively small, so as to allow a reasonable >>bandwidth for each downloader, similar to the same tactic used by >>traffic-heavy HTTP and FTP servers. This means that a P2P servent often >>has to wait long times and multiple retries before it gets it's chance >>to download from a specific host. For many networks, getting to download >>is essentially random - who gets on straight after a slot becomes free? >>This problem is less for files that are very popular and have spread >>between many hosts, as there are more sources to download from. However, >>the popularity again ensures that getting a free slot will be difficult. >>Many networks support multiple & segmented downloading, and AFAIK many >>of these support a client providing uploads of a segment of a file even >>when it is not entirely downloaded. >>It seems to me that a better approach would be for servents to have a >>very high cap on the number of concurrent uploads, which should cause >>more clients to be able to get the file at a slow speed immeditately, >>instead of fewer being able to get it at a faster speed. While the two >>may seem the same, a greater number of clients beginning a download >>means that all these clients have segments of the file, however small, >>and are themselves potential servers of that file. Hence, a popular file >>can quickly spread in segments from one overloaded servent to many >>interested servents. Each servent is likely to have a different part of >>the file, so they can all start downloading the parts they do not have >>from other downloaders. Hopefully this should result in faster overall >>downloads and a more efficient network, spreading the load better. >>Combined with clients that start downloads at random positions (to >>prevent all servents having the beginning and few having the end), and a >>system that provides the addresses of other recent downloaders (and >>parent nodes that provided the file) to speed resource discovery, I >>belive this system could prove significantly more efficient than current >>limited-availablility systems. >> >>So, you made it to the end of all this. Does anyone have any comments on >>the practicality of this idea? It seems good to me, but n heads are >>better than 1 ;). >> >>Nick Johnson >> >>_______________________________________________ >>p2p-hackers mailing list >>p2p-hackers@zgp.org >>http://zgp.org/mailman/listinfo/p2p-hackers >> > > From justin at chapweske.com Sun Sep 29 08:06:01 2002 From: justin at chapweske.com (Justin Chapweske) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Help With My Research References: Message-ID: <3D97169E.2050806@chapweske.com> Well, there are a couple of things related to this that we've done with the Open Content Network (http://open-content.net/). The first is the use of HTTP header extensions to supply additional meta-data about the content being received. We use this technique for the Content-Addressable Web (http://open-content.net/specs/draft-jchapweske-caw-03.html). Similarly, something like WebDAV Properties might provide a more generic way to attach meta-data with HTTP objects. For THEX (http://open-content.net/specs/draft-jchapweske-thex-01.html), we went with a different approach and used DIME (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnglobspec/html/dimeindex.asp) to bundle a piece of XML meta-data together with its binary attachments. DIME is very simple to implement and I expect there to be a rapidly growing number of implementations available over the coming months. Thanks, -Justin Lisa Rein wrote: > Hi gang, > > Note: please reply to me offlist at lisarein@finetuning.com unless I'm > wrong and this is actually an interesting topic of discussion for you guys > :-) > > I'm just asking for a little help with something I've been working on for > Creative Commons: > > I've been conducting research in order to determine the different methods > for embedding text within file formats (eg, comment fields, etc.). > > I'm interested in not only the different means available for doing so (which > I can pretty much grab from the specs etc. -- although the means of > accessing these fields also seems to be largely application-specific), > but also I'm wondering (help me, oh help me people) if such techniques are > actually being implemented by users for use within software NOW. > > For example: Sure you can put a URL in the comment field of an MP3 file > (just as easily as someone else can remove it), but does anyone know of > software actually looking for information in the comment fields of MP3 > files? That kinda stuff. > > Thanks in advance for your inevitable insights! > > Lisa Rein > Technical Architect > Creative Commons > lisarein@finetuning.com > > > P.S. Give Peace A Chance: http://www.lisarein.com/peace/ > > > > > http://www.onlisareinsradar.com > http://www.lisarein.com > http://www.finetuning.com > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers -- Justin Chapweske, Onion Networks http://onionnetworks.com/ From bram at gawth.com Sun Sep 29 10:22:01 2002 From: bram at gawth.com (Bram Cohen) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Restrictions on number of downloads and network efficiency In-Reply-To: <3D96B782.70607@notdot.net> Message-ID: Nick Johnson wrote: > Basically, what I'm asking is - would changing the 'policy' of current > P2P networks to allow more simultaneous downloads at a lower speed > (assuming said network allows downloads from hosts with partial content) > increase their efficiency and availability? No. The big problem is bandwidth availability on a macro scale. If the total download capacity wanted is much greater than the total upload capacity offered, then downloads will slow, regardless of how that capacity is distributed among the downloaders. > I'd personally rather be downloading something at 8k/s from multiple > 'slow' clients and uploading the segments I have at a similar speed, > than waiting hours for a chance to get 8-16k/s from a single client. > If existing networks wouldn't take much benifit from this, would a new > network, designed around this and other principles do so? Many simultaneous transfers tend to do bad things to TCP's congestion control. > Incidentally, why the use of 'bencoding', as opposed to, say, XML? Because XML is bloated, a semantic mess, doesn't canonicalize, doesn't support verbatim string inclusions, and has poor extensibility support. In short, it sucks. -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From zooko at zooko.com Sun Sep 29 10:59:01 2002 From: zooko at zooko.com (Zooko) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Help With My Research In-Reply-To: Message from Lisa Rein of "Fri, 27 Sep 2002 15:15:06 PDT." References: Message-ID: Lisa Rein wrote: > > Hi gang, > > Note: please reply to me offlist at lisarein@finetuning.com unless I'm > wrong and this is actually an interesting topic of discussion for you guys > :-) I think it is very much on-topic, so I'm replying on-list. > For example: Sure you can put a URL in the comment field of an MP3 file > (just as easily as someone else can remove it), but does anyone know of > software actually looking for information in the comment fields of MP3 > files? That kinda stuff. I downloaded some mp3z from http://www.lisarein.com and appended id3 tags to them (using the cmd-line tool "id3"), and then uploaded them to Mnet. The Mnet client parsed the id3 tags while uploading, and used the contents thereof to seed the metadata fields, so that now when you search for "Performer: Lisa Rein" on Mnet, you find those tracks. I wouldn't have had to do that middle step if you had set the id3 fields yourself. There is an open task for Mnet to do the same for whatever embedded metadata the ogg format offers: [1] By the way, I enjoy your music (which I just discovered via this discussion). > Thanks in advance for your inevitable insights! Inevitable insights! What a phrase. Regards, Zooko, who occasionally has inevitable insights [1] http://sourceforge.net/pm/task.php?func=detailtask&project_task_id=62317&group_id=43482&group_project_id=21943 From bert at akamail.com Sun Sep 29 11:30:02 2002 From: bert at akamail.com (Bert) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Restrictions on number of downloads and network efficiency References: Message-ID: <3D974779.3060505@akamail.com> Bram Cohen wrote: >Nick Johnson wrote: > > > >>Basically, what I'm asking is - would changing the 'policy' of current >>P2P networks to allow more simultaneous downloads at a lower speed >>(assuming said network allows downloads from hosts with partial content) >>increase their efficiency and availability? >> >> > >No. The big problem is bandwidth availability on a macro scale. If the >total download capacity wanted is much greater than the total upload >capacity offered, then downloads will slow, regardless of how that >capacity is distributed among the downloaders. > Well that's a given. But let's put it more concretely: would more connections increase overall throughput? The answer is probably "it depends" :-) A simple queueing model assuming no churn would indicate it will help, since the system is seeded with data available for broadcast much more quickly. But who knows how that would translate to the real world with clients going on and offline all the time, etc... Churn is a problem because with slower connections the likelihood of any one node entirely completing a download drops.. more connections could make getting the first few bytes of a file more efficient (overall), but it's not clear if it will help you getting all the bytes, and could in fact hurt. So the answer would likely depend on churn, and possibly/probably other characteristics of the network. >>I'd personally rather be downloading something at 8k/s from multiple >>'slow' clients and uploading the segments I have at a similar speed, >>than waiting hours for a chance to get 8-16k/s from a single client. >>If existing networks wouldn't take much benifit from this, would a new >>network, designed around this and other principles do so? >> >> > >Many simultaneous transfers tend to do bad things to TCP's congestion >control. > But how many is many? Grid FTP for example intentionally opens up several connections because it has been shown in practice to reduce problems with TCP's congestion control compared to using only a single (or small number of) connections. I think they found throughput benefits up to 16 concurrent connections. But this is with a single point to point transfer, so whether or not it will generalize to multiple destinations may be an open question. From decoy at iki.fi Sun Sep 29 13:00:01 2002 From: decoy at iki.fi (Sampo Syreeni) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Restrictions on number of downloads and network efficiency In-Reply-To: Message-ID: On 2002-09-29, Bram Cohen uttered to p2p-hackers@zgp.org: >The big problem is bandwidth availability on a macro scale. If the total >download capacity wanted is much greater than the total upload capacity >offered, then downloads will slow, regardless of how that capacity is >distributed among the downloaders. Of course. But on the other hand, burstiness, adaptation to network conditions and maximal utilization of the total bandwidth available could probably be increased if the load was distributed over many connections. Also, I think the recent caching argument is valid -- distributing the blocks around the network as fast as possible should indeed make it possible to utilize shorter download paths in the average, and to increase the average degree of uplink loading. (I too acknowledge that the shorther paths will not help much if the nodes all reside at the bandwidth restricted edge of the Net. However, in the longer run a significant number can be expected to live in fast local networks where caching *will* have a significant effect. I seem to remember local service advertising protocols being discussed onlist, before...) In particular, I think an architecture like this might have some interesting long-term, market based consequences -- better adaptation might make it easier for new resource limited nodes to join the network, eventually bumping up the total uplink capacity. Any redundancy there is will also help, provided different data have different access frequencies (they do, or Britney wouldn't be an issue). Constant, well-behaved bidirectional loading may help mutate the network edge connectivity market towards a more symmetric configuration, too. >Many simultaneous transfers tend to do bad things to TCP's congestion >control. Indeed. The consequence is that we either use something besides TCP, or take over TCP's congestion control and manage it as a unit for all the connections simultaneously. First steps toward integrated congestion control have already been taken, in RFC 3124, but I believe the congestion manager described therein only handles multiple connections between two machines, not one-to-many. There's certainly room for further development, a couple of dissertations and a pile of RFC's, here. My first intuition would be to distribute average load metrics (adaptation will only work if we can guarantee smooth adaptation of incoming requests to available upload bandwidth), and to mimic some of the soft-state mechanisms used with IP multicast (some of the problems are quite similar due to one-to-many connectivity). >Because XML is bloated, a semantic mess, doesn't canonicalize, doesn't >support verbatim string inclusions, and has poor extensibility support. Being one of XML's many bitches, I thoroughly second that. XML is nicer than SGML, sure, but that fact alone hardly translates to elegance. -- Sampo Syreeni, aka decoy - mailto:decoy@iki.fi, tel:+358-50-5756111 student/math+cs/helsinki university, http://www.iki.fi/~decoy/front openpgp: 050985C2/025E D175 ABE5 027C 9494 EEB0 E090 8BA9 0509 85C2 From decoy at iki.fi Sun Sep 29 13:07:01 2002 From: decoy at iki.fi (Sampo Syreeni) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Restrictions on number of downloads and network efficiency In-Reply-To: <3D974779.3060505@akamail.com> Message-ID: On 2002-09-29, Bert uttered to p2p-hackers@zgp.org: >Grid FTP for example intentionally opens up several connections because >it has been shown in practice to reduce problems with TCP's congestion >control compared to using only a single (or small number of) >connections. Really? Even when you take into account competing TCP traffic? Somewhat I find that a bit unlikely. Do you have further statistics? -- Sampo Syreeni, aka decoy - mailto:decoy@iki.fi, tel:+358-50-5756111 student/math+cs/helsinki university, http://www.iki.fi/~decoy/front openpgp: 050985C2/025E D175 ABE5 027C 9494 EEB0 E090 8BA9 0509 85C2 From mfreed at cs.nyu.edu Sun Sep 29 13:23:01 2002 From: mfreed at cs.nyu.edu (Michael J. Freedman) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Restrictions on number of downloads and network efficiency In-Reply-To: Message-ID: On Sun, 29 Sep 2002, Sampo Syreeni wrote: > Date: Sun, 29 Sep 2002 22:59:52 +0300 (EEST) > From: Sampo Syreeni > Reply-To: p2p-hackers@zgp.org > To: p2p-hackers@zgp.org > Subject: Re: [p2p-hackers] Restrictions on number of downloads and > network efficiency > > On 2002-09-29, Bram Cohen uttered to p2p-hackers@zgp.org: > > >The big problem is bandwidth availability on a macro scale. If the total > >download capacity wanted is much greater than the total upload capacity > >offered, then downloads will slow, regardless of how that capacity is > >distributed among the downloaders. > > Of course. But on the other hand, burstiness, adaptation to network > conditions and maximal utilization of the total bandwidth available could > probably be increased if the load was distributed over many connections. > Also, I think the recent caching argument is valid -- distributing the > blocks around the network as fast as possible should indeed make it > possible to utilize shorter download paths in the average, and to increase > the average degree of uplink loading. (I too acknowledge that the shorther > paths will not help much if the nodes all reside at the bandwidth > restricted edge of the Net. However, in the longer run a significant > number can be expected to live in fast local networks where caching *will* > have a significant effect. I seem to remember local service advertising > protocols being discussed onlist, before...) > > In particular, I think an architecture like this might have some > interesting long-term, market based consequences -- better adaptation > might make it easier for new resource limited nodes to join the network, > eventually bumping up the total uplink capacity. Any redundancy there is > will also help, provided different data have different access frequencies > (they do, or Britney wouldn't be an issue). Constant, well-behaved > bidirectional loading may help mutate the network edge connectivity market > towards a more symmetric configuration, too. > > >Many simultaneous transfers tend to do bad things to TCP's congestion > >control. > > Indeed. The consequence is that we either use something besides TCP, or > take over TCP's congestion control and manage it as a unit for all the > connections simultaneously. First steps toward integrated congestion > control have already been taken, in RFC 3124, but I believe the congestion > manager described therein only handles multiple connections between two > machines, not one-to-many. There's certainly room for further development, > a couple of dissertations and a pile of RFC's, here. My first intuition > would be to distribute average load metrics (adaptation will only work if > we can guarantee smooth adaptation of incoming requests to available > upload bandwidth), and to mimic some of the soft-state mechanisms used > with IP multicast (some of the problems are quite similar due to > one-to-many connectivity). > > >Because XML is bloated, a semantic mess, doesn't canonicalize, doesn't > >support verbatim string inclusions, and has poor extensibility support. > > Being one of XML's many bitches, I thoroughly second that. XML is nicer > than SGML, sure, but that fact alone hardly translates to elegance. > -- > Sampo Syreeni, aka decoy - mailto:decoy@iki.fi, tel:+358-50-5756111 > student/math+cs/helsinki university, http://www.iki.fi/~decoy/front > openpgp: 050985C2/025E D175 ABE5 027C 9494 EEB0 E090 8BA9 0509 85C2 > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > ----- "Not all those who wander are lost." www.michaelfreedman.org From bert at akamail.com Sun Sep 29 13:25:01 2002 From: bert at akamail.com (Bert) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Restrictions on number of downloads and network efficiency References: Message-ID: <3D97622F.20408@akamail.com> Sampo Syreeni wrote: >On 2002-09-29, Bert uttered to p2p-hackers@zgp.org: > > > >>Grid FTP for example intentionally opens up several connections because >>it has been shown in practice to reduce problems with TCP's congestion >>control compared to using only a single (or small number of) >>connections. >> >> > >Really? Even when you take into account competing TCP traffic? Somewhat I >find that a bit unlikely. Do you have further statistics? > > There's lots of published material on this... check out the papers on the Globus website for example. A quick Google search also yielded this experimental results page which clearly shows increase in throughput with increased # of connections (to a point, of course): http://server11.infn.it/netgrid/task/task2/globusftp/ftp-summary.html From decoy at iki.fi Sun Sep 29 14:35:01 2002 From: decoy at iki.fi (Sampo Syreeni) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Restrictions on number of downloads and network efficiency In-Reply-To: <3D97622F.20408@akamail.com> Message-ID: On 2002-09-29, Bert uttered to p2p-hackers@zgp.org: >There's lots of published material on this... check out the papers on >the Globus website for example. A quick Google search also yielded this >experimental results page which clearly shows increase in throughput >with increased # of connections (to a point, of course): >http://server11.infn.it/netgrid/task/task2/globusftp/ftp-summary.html True. However, there's no data indicating that a number of GridFTP clients competing for the same bandwidth amongst themselves and ordinary FTP connections would achieve the same results, or that harm to plain FTP wouldn't be done. The point is, there are extremely simple ways in which to hog bandwidth besides starting multiple TCP connections. All you have to accomplish is the doing away of congestion control. But that isn't what we want. We want our applications to be well behaved net.citizens. What I worry about is that multiple parallel TCP connections might simply amount to hogging, due to the known shortcomings of TCP Bram already alluded to. -- Sampo Syreeni, aka decoy - mailto:decoy@iki.fi, tel:+358-50-5756111 student/math+cs/helsinki university, http://www.iki.fi/~decoy/front openpgp: 050985C2/025E D175 ABE5 027C 9494 EEB0 E090 8BA9 0509 85C2 From lisarein at finetuning.com Mon Sep 30 08:59:01 2002 From: lisarein at finetuning.com (Lisa Rein) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Help With My Research In-Reply-To: Message-ID: thanks everyone! This stuff is all exactly what I have been looking for. Am digesting information now...ummmm yummy yummy data...must duplicate results determined thus far... (I need to catch up with you guys and get all the formats and software mentioned up and running on my own machine.) Inevitable questions to follow...(they come with the inevitable insights :-) Thanks again, lisa > From: Zooko > Date: Sun, 29 Sep 2002 13:55:39 -0400 > To: p2p-hackers@zgp.org > Cc: lisarein@finetuning.com > Subject: Re: [p2p-hackers] Help With My Research > > > Lisa Rein wrote: >> >> Hi gang, >> >> Note: please reply to me offlist at lisarein@finetuning.com unless I'm >> wrong and this is actually an interesting topic of discussion for you guys >> :-) > > I think it is very much on-topic, so I'm replying on-list. > >> For example: Sure you can put a URL in the comment field of an MP3 file >> (just as easily as someone else can remove it), but does anyone know of >> software actually looking for information in the comment fields of MP3 >> files? That kinda stuff. > > I downloaded some mp3z from http://www.lisarein.com and appended id3 tags to > them (using the cmd-line tool "id3"), and then uploaded them to Mnet. The > Mnet client parsed the id3 tags while uploading, and used the contents thereof > to seed the metadata fields, so that now when you search for > "Performer: Lisa Rein" on Mnet, you find those tracks. I wouldn't have had to > do that middle step if you had set the id3 fields yourself. > > There is an open task for Mnet to do the same for whatever embedded metadata > the ogg format offers: [1] > > By the way, I enjoy your music (which I just discovered via this discussion). > >> Thanks in advance for your inevitable insights! > > Inevitable insights! What a phrase. > > Regards, > > Zooko, who occasionally has inevitable insights > > [1] > http://sourceforge.net/pm/task.php?func=detailtask&project_task_id=62317&group > _id=43482&group_project_id=21943 > From arachnid at notdot.net Mon Sep 30 22:07:01 2002 From: arachnid at notdot.net (Nick Johnson) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Restrictions on number of downloads and network efficiency References: <3D96B782.70607@notdot.net> Message-ID: <3D992D93.5080703@notdot.net> First of all, sorry about my lack of responses - my host stopped catching and storing emails for my domain (long story), and I didn't realise that until a couple of days had passed with no messages to p2p hackers. On 2002-09-29, Bram Cohen uttered to p2p-hackers@zgp.org: >>Many simultaneous transfers tend to do bad things to TCP's congestion >>control. > >Indeed. The consequence is that we either use something besides TCP, or >take over TCP's congestion control and manage it as a unit for all the >connections simultaneously. First steps toward integrated congestion >control have already been taken, in RFC 3124, but I believe the >congestion manager described therein only handles multiple connections >between two machines, not one-to-many. There's certainly room for >further development, a couple of dissertations and a pile of RFC's, >here. My first intuition would be to distribute average load metrics >(adaptation will only work if we can guarantee smooth adaptation of >incoming requests to available upload bandwidth), and to mimic some of >the soft-state mechanisms used with IP multicast (some of the problems >are quite similar due to one-to-many connectivity). I've wondered about this too - TCP seems to have multiple shortcomings in this area of endeavour (and others), and overhead of one sort or another tends to restrict some otherwise potentially useful approaches. Perhaps a good P2P organised UDP based protocol could be constructed to deal with these problems? Incidentally, my knowledge of TCP/IP and UDP/IP is merely moderate - I was unable to get into this years networking course at my University due to limited places and teacher shortages :/. I can see the point some people have made about needing to get blocks out quickly for best spread, so surely there must be a best-point tradeoff between number of concurrent connections and bandwidth spent on each connection? Has anyone attempted to establish this, or are connection limits on P2P servents arbitarially set at the moment? >>Because XML is bloated, a semantic mess, doesn't canonicalize, doesn't >>support verbatim string inclusions, and has poor extensibility >support. > >Being one of XML's many bitches, I thoroughly second that. XML is nicer >than SGML, sure, but that fact alone hardly translates to elegance. However, XMLs advantage lies in it's interoperability - once you have an XML parser, you can read any XML compliant document, and using XSLT you can probably write a stylesheet to transform it into a format you can read. Nick Johnson From justin at chapweske.com Mon Sep 30 22:19:01 2002 From: justin at chapweske.com (Justin Chapweske) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Restrictions on number of downloads and network efficiency References: <3D96B782.70607@notdot.net> <3D992D93.5080703@notdot.net> Message-ID: <3D993030.7050602@chapweske.com> If you are interested in solutions that approach the channel capacity of multipath networks while overcoming TCPs limitations, you may wish to check out Swarmcast (http://sf.net/projects/swarmcast/). Its use of FEC allows it to quickly maximize the utility of file segments in the network. Making most any random piece of data in the network useful for reconstructing the original file, even if you've already received most of the file. Also, if you are interested in getting a better grasp on how the LFN (Long Fat Network) problem effects TCP performance, you might want to check out this: (http://www.digitalfountain.com/solutions/calculator/index.htm?t=0). Now, Digital Fountain's solution also uses FEC, but its overkill for simply solving the LFN problem. A parallel HTTP or FTP library will solve LFN just fine. Nick Johnson wrote: > First of all, sorry about my lack of responses - my host stopped > catching and storing emails for my domain (long story), and I didn't > realise that until a couple of days had passed with no messages to p2p > hackers. > > On 2002-09-29, Bram Cohen uttered to p2p-hackers@zgp.org: > >>Many simultaneous transfers tend to do bad things to TCP's congestion > >>control. > > > >Indeed. The consequence is that we either use something besides TCP, or > >take over TCP's congestion control and manage it as a unit for all the > >connections simultaneously. First steps toward integrated congestion > >control have already been taken, in RFC 3124, but I believe the > >congestion manager described therein only handles multiple connections > >between two machines, not one-to-many. There's certainly room for > >further development, a couple of dissertations and a pile of RFC's, > >here. My first intuition would be to distribute average load metrics > >(adaptation will only work if we can guarantee smooth adaptation of > >incoming requests to available upload bandwidth), and to mimic some of > >the soft-state mechanisms used with IP multicast (some of the problems > >are quite similar due to one-to-many connectivity). > > I've wondered about this too - TCP seems to have multiple shortcomings > in this area of endeavour (and others), and overhead of one sort or > another tends to restrict some otherwise potentially useful approaches. > Perhaps a good P2P organised UDP based protocol could be constructed to > deal with these problems? > Incidentally, my knowledge of TCP/IP and UDP/IP is merely moderate - I > was unable to get into this years networking course at my University due > to limited places and teacher shortages :/. > > I can see the point some people have made about needing to get blocks > out quickly for best spread, so surely there must be a best-point > tradeoff between number of concurrent connections and bandwidth spent on > each connection? Has anyone attempted to establish this, or are > connection limits on P2P servents arbitarially set at the moment? > > >>Because XML is bloated, a semantic mess, doesn't canonicalize, doesn't > >>support verbatim string inclusions, and has poor extensibility >support. > > > >Being one of XML's many bitches, I thoroughly second that. XML is nicer > >than SGML, sure, but that fact alone hardly translates to elegance. > > However, XMLs advantage lies in it's interoperability - once you have an > XML parser, you can read any XML compliant document, and using XSLT you > can probably write a stylesheet to transform it into a format you can read. > > Nick Johnson > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers -- Justin Chapweske, Onion Networks http://onionnetworks.com/ From bram at gawth.com Mon Sep 30 22:50:01 2002 From: bram at gawth.com (Bram Cohen) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Restrictions on number of downloads and network efficiency In-Reply-To: <3D974779.3060505@akamail.com> Message-ID: Bert wrote: > But how many is many? Grid FTP for example intentionally opens up > several connections because it has been shown in practice to reduce > problems with TCP's congestion control compared to using only a single > (or small number of) connections. I think they found throughput > benefits up to 16 concurrent connections. But this is with a single > point to point transfer, so whether or not it will generalize to > multiple destinations may be an open question. Well, we have firm confirmation that opening up too many TCP connections will cause them to not back off properly, and that 16 is so ludicrously bad that it gets poor performance despite your bandwidth bullyness. -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From bram at gawth.com Mon Sep 30 22:57:01 2002 From: bram at gawth.com (Bram Cohen) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Restrictions on number of downloads and network efficiency In-Reply-To: Message-ID: Sampo Syreeni wrote: > burstiness, adaptation to network conditions and maximal utilization > of the total bandwidth available could probably be increased if the > load was distributed over many connections. Yes. BitTorrent uses four, that seems to be plenty. > Also, I think the recent caching argument is valid -- distributing the > blocks around the network as fast as possible should indeed make it > possible to utilize shorter download paths in the average That's dependant on the amount of time it takes to get a single block from one machine to another, which smaller pieces makes better, but more connections makes worse. BitTorrent uses megabyte pieces (configurable per file) which are hashed, and 32k sub-pieces in the communication layer to keep the time to deliver each individual one from being too much. Smaller pieces increase overhead a lot. I picked values for BitTorrent which keep it miniscule in all cases. > >Many simultaneous transfers tend to do bad things to TCP's congestion > >control. > > Indeed. The consequence is that we either use something besides TCP, or > take over TCP's congestion control and manage it as a unit for all the > connections simultaneously. That's completely unnecessary. BitTorrent uses TCP as a black box and it works great. -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From bram at gawth.com Mon Sep 30 23:02:01 2002 From: bram at gawth.com (Bram Cohen) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Restrictions on number of downloads and network efficiency In-Reply-To: <3D992D93.5080703@notdot.net> Message-ID: Nick Johnson wrote: > >>Because XML is bloated, a semantic mess, doesn't canonicalize, doesn't > >>support verbatim string inclusions, and has poor extensibility >support. > > > >Being one of XML's many bitches, I thoroughly second that. XML is nicer > >than SGML, sure, but that fact alone hardly translates to elegance. > > However, XMLs advantage lies in it's interoperability - once you have an > XML parser, you can read any XML compliant document, and using XSLT you > can probably write a stylesheet to transform it into a format you can read. Bencode parsers are vastly easier to write and a small fraction the size of XML parsers. The obsession with using XML for everything is just plain stupid. Any given protocol gains no interoperability whatsoever from using XML, since its semantics will be unique regardless of what data format it uses. It's like painting your house puke green because you already happen to have a few cans of paint of that color. -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From bram at gawth.com Mon Sep 30 23:10:01 2002 From: bram at gawth.com (Bram Cohen) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Restrictions on number of downloads and network efficiency In-Reply-To: <3D993030.7050602@chapweske.com> Message-ID: Justin Chapweske wrote: > If you are interested in solutions that approach the channel capacity of > multipath networks while overcoming TCPs limitations, you may wish to > check out Swarmcast (http://sf.net/projects/swarmcast/). Is anyone actually using swarmcast? Its functionality seems to be essentially the same as BitTorrent, and BitTorrent is getting quite a bit of usage - http://sourceforge.net/project/stats/?group_id=33044 > Its use of FEC allows it to quickly maximize the utility of file > segments in the network. Making most any random piece of data in the > network useful for reconstructing the original file, even if you've > already received most of the file. BitTorrent uses no FEC, and its doesn't have any piece diffusion problems. Downloading in random order works fine. > Also, if you are interested in getting a better grasp on how the LFN > (Long Fat Network) problem effects TCP performance, you might want to > check out this: > (http://www.digitalfountain.com/solutions/calculator/index.htm?t=0). > > Now, Digital Fountain's solution also uses FEC, but its overkill for > simply solving the LFN problem. A parallel HTTP or FTP library will > solve LFN just fine. Or you can just tweak the parameters of TCP a bit. Why do all this software development when parameter setting will work just fine? -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From justin at chapweske.com Mon Sep 30 23:46:01 2002 From: justin at chapweske.com (Justin Chapweske) Date: Sat Dec 9 22:12:03 2006 Subject: [p2p-hackers] Restrictions on number of downloads and network efficiency References: Message-ID: <3D994485.5010005@chapweske.com> > >>If you are interested in solutions that approach the channel capacity of >>multipath networks while overcoming TCPs limitations, you may wish to >>check out Swarmcast (http://sf.net/projects/swarmcast/). > > > Is anyone actually using swarmcast? Its functionality seems to be > essentially the same as BitTorrent, and BitTorrent is getting quite a bit > of usage - > > http://sourceforge.net/project/stats/?group_id=33044 > Swarmcast has been effectively abandoned, but is still very interesting in that it is the first and only system to approach the theoretical channel capacity of a multi-path network. While BitTorrent may be a practical solution, the original poster was eluding to something beyond basic swarming...something that could very quickly *globally* maximize the available bandwidth in the network. So, for the context of this conversation about the reaching the limits of performance, Swarmcast can acomplish this, while simple swarming can not. Although I guess I should qualify this by saying that Swarmcast is actually *simpler* swarming because it basically doesn't care which packets it is sending or receiving, its happy with whatever it receives. -- Justin Chapweske, Onion Networks http://onionnetworks.com/