META: Problems with collaborative filtering (was Re: Improving the list (was: Can we please dekookify the list?))

From: Bartley R. Troyan (btroyan@littlekoala.com)
Date: Tue Mar 02 1999 - 00:02:59 MST


Anders Sandberg wrote:
> While the kill thread and topic stuff might not be necessary, I like
> the idea of Kookify/Glorify! A more soft incentive to produce good
> posts.

I don't like the idea of Kookify/Glorify as I understand it. Let's say you
all decide I'm a Kook and virtually everyone heavily Kookifies me by clicking
the Kookify link at the bottom of every message I send to the list. I don't
care how you prevent multiple votes (it's not that hard), but it won't make a
difference in the end... If I post a lot of rubbish for a while, I will attain
an extremely high level of Kookiness after a short time.

Now, let's say perchance six months down the road the tradition develops that
people tend to filter out Kooks (based on an X-Kook-Level or X-Kook-Ratio
header or something included in all of the messages). They rarely, if ever,
read messages from someone as certifiably Kooky as me. And now that there
are, say 100 Million AOLers, there are also 10K subscribers to Extropians
mailing list and lots of other kooks posting all sorts of junk. So assume
that a filtered version of the list is created, that the AOL hordes (no
offence EvMick) can subscribe to on the web site. Extropians-No-Kooks provides
a list view in which the messages from Kooks at level x or greater are not
even sent out with normal list traffic (say, due to bandwidth and other
resource constraints. It's not cheap to send 300 pieces of email to 10K
different email addrs every single day. But that's the level of traffic one
must anticipate 6 months or a year down the road. It's a problem the entire
net will continue to face as it continues to grow rapidly).

Anyway, therefore I will never be de-Kookified, even if I suddenly experience
a Zen-like enlightenment and start posting like a networked super-brain better
than a merged consciousnesses of the best posters (you know who you are). I've
been rolled into a local minimum in Kook-space and now I can't get out. And
everybody loses out on my godlike wisdom and conspiracy-theory-debunking
insights.

My point is, maybe both the Glorifieds and the Kooks should be given an
automatic, periodic, proportional adjustment in the opposite direction of
their current tendency, just to make sure nobody gets stuck in any holes.

> I have been thinking of running some program through the list archives
> to do data mining, for example build sociograms (who responds to
> whom?), find threads where someone participates, who glorifies who and
> perhaps extract clusters of discussion. Maybe the way of improving
> quality of the list lies in making archives more powerful?

A lot of that stuff is interesting to me as well. I'll bet you'd find some of
cliques that respond primarily to each other, and ignore newbie/clueless posts
as well as posts from Already Certified Kooks. Then there are the debaters,
and the people with way too much time on their hands that respond and argue
everything. (I'm not saying that's bad, I'm just saying, well, I don't have
that much time. I can only afford to catch up with this list, all msgs dumped
into a separate folder, once or twice a day, and I am forced to use the delete
key without reading most messages).

Anyway, to implement the kind of data mining and analysis Anders describes, in
an efficient and easy-to-program manner, we would need a database, of course.
Anyone want to donate cycles and space on a mySQL server on a really fast box
with a gig or two of storage? Give me a login there, I'll give you everything
Anders described above and much more. We'd also need a way to easily modify
headers and footers of list messages on a per-message basis. It could start
small and grow to be an engaging, real-life experiment in collaborative
filtering. I wouldn't mind a share of that $10K if you all like it well
enough :-) And if list members mostly just ignore those funny new extra
footers on the bottom of all the list messages, so what. We'll take them
out. No harm done.

-Bart
(aspiring Extropian Kook)

> --
> -----------------------------------------------------------------------
> Anders Sandberg Towards Ascension!
> asa@nada.kth.se http://www.nada.kth.se/~asa/
> GCS/M/S/O d++ -p+ c++++ !l u+ e++ m++ s+/+ n--- h+/* f+ g+ w++ t+ r+ !y



This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:03:13 MST