Date: Tue, 07 Nov 2023 16:12:25 +0000
To: bitcoin-dev@lists.linuxfoundation.org
From: Andrew Chow <lists@achow101.com>
Message-ID: <a2435f58-9aff-4cfe-8d7a-8e7258e4f64e@achow101.com>
In-Reply-To: <CABaSBaz9OTSVa1KNk0GOrH3T-kRF_7OPVu0AtpuaFGVB=zhdwQ@mail.gmail.com>
References: <CABaSBaz9OTSVa1KNk0GOrH3T-kRF_7OPVu0AtpuaFGVB=zhdwQ@mail.gmail.com>
Feedback-ID: 53660394:user:proton
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Subject: Re: [bitcoin-dev] Future of the bitcoin-dev mailing list
Precedence: list

Thanks for writing this up.

I would prefer that we continue to have a mailing list where email is a=20
functional and first class user interface. So that would be to migrate=20
to groups.io or Google Groups. I think Google Groups is probably the=20
better choice of the two.

Although there are concerns that Google would shut down Google Groups or=20
specifically target a bitcoin-dev group, I think both are unlikely to=20
happen. Both Chromium and Android use Google Groups for their mailing=20
lists, so unless those go somewhere else, I doubt Google would=20
unceremoniously kill Google Groups. As for shutting down a bitcoin-dev=20
group specifically, given that there appears to be several thousand=20
groups whose sole purpose is to distribute spam, I don't think Google is=20
in the habit of shutting down groups.

Regardless of what we do, there's always the risk that someone will shut=20
down or stop maintaining the servers for any discussion medium. Even=20
self hosting requires someone to keep the servers up and do maintenance,=20
and that person (or people) could get bored of it, run out of money, get=20
hit by a bus, etc. We're experiencing that right now with the Linux=20
Foundation, and I don't think fear of that should prevent us from moving=20
to a different third party host.


Andrew Chow

On 11/07/2023 10:37 AM, Bryan Bishop via bitcoin-dev wrote:
> Hello,
>=20
> We would like to request community=C2=A0feedback and proposals on the fut=
ure=20
> of the mailing list.
>=20
> Our current mailing list host, Linux Foundation, has indicated for years=
=20
> that they have wanted to stop hosting mailing lists, which would mean=20
> the bitcoin-dev mailing list would need to move somewhere else. We=20
> temporarily avoided that, but recently LF has informed a moderator that=
=20
> they will cease hosting any mailing lists later this year.
>=20
> In this email, we will go over some of the history, options, and invite=
=20
> discussion ahead of the cutoff. We have some ideas but want to solicit=20
> feedback and proposals.
>=20
> Background
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=20
> The bitcoin-dev mailing list was originally hosted on Sourceforge.net.=20
> The bitcoin development mailing list has been a source of proposals,=20
> analysis, and developer discussion for many years in the bitcoin=20
> community, with many thousands of participants. Later, the mailing list=
=20
> was migrated to the Linux Foundation, and after that OSUOSL began to help=
.
>=20
> Linux Foundation first asked us to move the mailing list in 2017. They=20
> internally attempted to migrate all LF mailing lists from mailman2 to=20
> mailman3, but ultimately gave up. There were reports of scalability=20
> issues with mailman3 for large email communities. Ours definitely=20
> qualifies as.. large.
>=20
> 2019 migration plan: LF was to turn off mailman and all lists would=20
> migrate to the paid service provider groups.io <http://groups.io>. Back=
=20
> then we were given accounts to try the groups.io <http://groups.io>=20
> interface and administration features. Apparently we were not the only=20
> dev community who resisted change. To our surprise LF gave us several=20
> years of reprieve by instead handing the subdomain and server-side data=
=20
> to the non-profit OSUOSL lab who instead operated mailman2 for the past=
=20
> ~4 years.
>=20
> OSUOSL has for decades been well known for providing server=20
> infrastructure for Linux and Open Source development so they were a good=
=20
> fit. This however became an added maintenance burden for the small=20
> non-profit with limited resources. Several members of the Bitcoin dev=20
> community contributed funding to the lab in support of their Open Source=
=20
> development infrastructure goals. But throwing money at the problem=20
> isn=E2=80=99t going to fix the ongoing maintenance burden created by anti=
quated=20
> limitations of mailman2.
>=20
> Permalinks
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=20
> Linux Foundation has either offered or agreed to maintain archive=20
> permalinks so that content of historic importance is not lost.=20
> Fortunately for us while lists.linuxfoundation.org=20
> <http://lists.linuxfoundation.org> mailman will go down, they have=20
> agreed the read-only pipermail archives will remain online. So all old=20
> URLs will continue to remain valid. However, the moderators strongly=20
> advise that the community supplements with public-inbox instances to=20
> have canonical archive urls that are separate from any particular email=
=20
> software host.
>=20
> Public-Inbox
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=20
> https://public-inbox.org/README.html <https://public-inbox.org/README.htm=
l>
>=20
> =E2=80=9CPublic Inbox=E2=80=9D decentralized archiving - no matter what m=
ailing list=20
> server solution is used, anyone can use git to maintain their own=20
> mailing list archive and make it available to read on the web.
>=20
> Public Inbox is a tool that you can run yourself. You can transform your=
=20
> mbox file and it makes it browsable and viewable online. It commits=20
> every post to a git repository. It's kind of like a decentralized mail=20
> archiving tool. Anyone can publish the mail archive to any web server=20
> they wish.
>=20
> We should try to have one or more canonical archives that are served=20
> using public-inbox. But it doesn't matter if these are lost because=20
> anyone else can archive the mailing list in the same way and re-publish=
=20
> the archives.
>=20
> These git commits can also be stamped using opentimestamps, inserting=20
> their hashes into the bitcoin blockchain.
>=20
> LKML mailing list readers often use public-inbox's web interface, and=20
> they use the reply-to headers to populate their mail client and reply to=
=20
> threads of interest. This allows their reply to be properly threaded=20
> even if they were not a previous subscriber to that mailing list to=20
> receive the headers.
>=20
> public-inbox makes it so that it doesn't really matter where the list is=
=20
> hosted, as pertaining to reading the mailing list. There is still a=20
> disruption if the mailing list goes away, but the archives live on and=20
> then people can post elsewhere. The archive gets disconnected from the=20
> mailing list host in terms of posting. We could have a few canonical=20
> URLs for the hosts, separate from the mailing list server.
>=20
> mailman problems
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=20
> Over the years we have identified a number of problems with mailman2=20
> especially as it pertains to content moderation. There are presently a=20
> handful of different moderators, but mailman2 only has a single password=
=20
> for logging into the email management interface. There are no moderator=
=20
> audit logs to see which user (there is no concept of different users)=20
> acted on an email. There is no way to mark an email as being=20
> investigated by one or more of the moderators. Sometimes, while=20
> investigating the veracity of an email, another moderator would come in=
=20
> and approve a suspect email by accident.
>=20
> Anti spam has been an issue for the moderators. It's relentless. Without=
=20
> access to the underlying server, it has been difficult to fight spam.=20
> There is some support for filters in mailman2 but it's not great.
>=20
> 100% active moderation and approval of every email is unsustainable for=
=20
> volunteer moderators. A system that requires moderation is a heavy=20
> burden on the moderators and it slows down overall communication and=20
> productivity. There's lots of problems with this. Also, moderators can=20
> be blamed when they are merely slow while they are not actually censoring=
.
>=20
> Rejection emails can optionally be sent to=20
> bitcoin-dev-moderation@lists.ozlabs.org=20
> <mailto:bitcoin-dev-moderation@lists.ozlabs.org> but this is an option=20
> that a moderator has to remember to type in each time.
>=20
> Not to mention numerous bugs and vulnerabilities that have accumulated=20
> over the years for relatively unmaintained software. (Not disclosed here)
>=20
> Requirements and considerations
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D
>=20
> Looking towards the future, there are a number of properties that seem=20
> to be important for the bitcoin-dev mailing list community. First, it is=
=20
> important that backups of the entire archive should be easy for the=20
> public to copy or verify so that the system can be brought up elsewhere=
=20
> if necessary.
>=20
> Second, there seems to be demand for both an email threading interface=20
> (using mailing list software) as well as web-accessible interfaces (such=
=20
> as forum software). There seems to be very few options that cater to=20
> both email and web. Often, in forum software, email support is limited=20
> to email notifications and there is limited if any support for email=20
> user participation.
>=20
> Third, there should be better support for moderator tools and management=
=20
> of the mailing list. See above for complaints about problems with the=20
> mailman2 system.
>=20
> Burdens of running your own mailing list and email server
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D
>=20
> If you have never operated your own MTA you have no idea how difficult=20
> it is to keep secure and functional in the face of numerous challenges=20
> to deliverability. Anti-spam filtering is essential to prevent=20
> forwarding spam. The moment you forward even a single spam message you=20
> run the risk of the server IP address being added to blacklists.
>=20
> The problem of spam filtering is so bad that most IP addresses are=20
> presumed guilty even if they have no prior spam history, such as if=20
> their network or subnetwork had spam issues in the past.
>=20
> Even if you put unlimited time into managing your own email server,=20
> other people may not accept your email. Or you make one mistake, and=20
> then you get into permanent blacklists and it's hard to remove. The spam=
=20
> problem is so bad that most IPs are automatically on a=20
> guilty-until-proven-innocent blacklist.
>=20
> Often there is nothing you can do to get server IP addresses removed=20
> from spam blacklists or from "bad reputation" lists.
>=20
> Ironically, hashcash-style proof-of-work stamps to prevent spam are an=20
> appealing solution but not widely used in this community. Or anywhere.
>=20
> Infinite rejection or forwarding loops happen. They often need to be=20
> detected through vigilance and require manual sysadmin intervention to=20
> solve.
>=20
> Bitcoin's dev lists being hosted alongside other Open Source projects=20
> was previously protective. If that mailing list server became=20
> blacklisted there were a lot of other people who would notice and=20
> complain. If we run a Bitcoin-specific mail server we are on our own.=20
> 100% of the administrative burden falls upon our own people. There is=20
> also nothing we can do if some unknown admin decides they don't like us.
>=20
> Options
> =3D=3D=3D=3D=3D=3D=3D
>=20
> Web forums are an interesting option, but often don't have good email=20
> user integration. At most you can usually hope for email notifications=20
> and an ability to reply by email. It changes the model of the community=
=20
> from push (email) to pull (logging into a forum to read). RSS feeds can=
=20
> help a little bit.
>=20
> Many other projects have moved from mailing lists to forums (eg=20
> https://discuss.python.org/ <https://discuss.python.org/> =E2=80=93 see=
=20
> https://lwn.net/Articles/901744/ <https://lwn.net/Articles/901744/> ; or=
=20
> https://ethresear.ch/ <https://ethresear.ch/>), which seem easier to=20
> maintain and moderate, and can have lots of advanced features beyond=20
> plaintext, maybe-threading and maybe-HTML-markup.
>=20
> Who would host the forum? Would there be agreement around which forum=20
> software to use or which forum host? What about bitcointalk.org=20
> <http://bitcointalk.org> or delvingbitcoin.org=20
> <http://delvingbitcoin.org>? There are many options available. Maybe=20
> what we actually want isn=E2=80=99t so much a discussion forum, as an 'ar=
xiv of=20
> our own' where anons can post BIP drafts and the like?
>=20
> Given the problems with mailman2, and the decline of email communities=20
> in general, it seems that moving to mailman3 would not be a viable=20
> long-term option. This leaves us with Google Groups or groups.io=20
> <http://groups.io> as two remaining options.
>=20
> groups.io <http://groups.io> is an interesting option: they are a paid=20
> service that implements email communities along with online web forum=20
> support. However, their public changelog indicates it has been a few=20
> years since their last public change. They might be a smaller company=20
> and it is unclear how long they will be around or if this would be the=20
> right fit for hosting sometimes contentious bitcoin development=20
> discussions...
>=20
> Google Groups is another interesting option, and comes with different=20
> tradeoffs. It's the lowest effort to maintain option, and has both an=20
> email interface and web forum interface. Users can choose which mode=20
> they want to interact with.
>=20
> For the Google Groups web interface, you can use it with a non-gmail=20
> account, but you must create a Google Account which is free to do. it=20
> does not require any personal information to do so. This also allows you=
=20
> to add 2FA. Non-gmail non-google users are able to subscribe and post=20
> email from their non-gmail non-google email accounts. Tor seems to work=
=20
> for the web interface.
>=20
> Will Google shut it down, will they cut us off, will they shut down=20
> non-google users? The same problem exists with other third-party hosts.
>=20
> The moderation capabilities for Google Groups and groups.io=20
> <http://groups.io> seem to be comparable. It seems more likely that=20
> Google Groups will be able to handle email delivery issues far better=20
> than a small resource-constrained operation like groups.io=20
> <http://groups.io>. ((During feedback for this draft, luke-jr indicates=
=20
> that Google Workspaces has been known to use blacklisted IPs for=20
> business email!))
>=20
> On the other hand, groups.io <http://groups.io> is a paid service and=20
> you get what you pay for... hopefully?
>=20
> Finally, another option is to do literally nothing. It's less work=20
> overall. Users can switch to forums or other websites, or private=20
> one-on-one communication. It would remove a point of semi-centralization=
=20
> from the bitcoin ecosystem. It would hasten ossification, but on the=20
> other hand it would hasten ossification and this could be a negative=20
> too. Moderators would be less of a target.
>=20
> Unfortunately, by doing nothing, there would be no more widely used=20
> group email communication system between bitcoin developers. Developers=
=20
> become less coordinated, mayhem and chaos as people go to different=20
> communication platforms, a divided community is more vulnerable, etc.=20
> BIP1 and BIP2 would need to be revised for other venues.
>=20
> The main categories of what to move to are: web forums, mailing lists,=20
> and hybrids of those two options. Most everything is either self-hosted=
=20
> or you pay someone else to host it. It's kind of the same problem=20
> though. It largely depends on how good is the software and unfortunately=
=20
> running your own MTA that forwards mail is not a good option.
>=20
> Going forward
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=20
> We'd like to invite feedback and proposals from the community, and see=20
> what options are available. One potential option is a migration to=20
> Google Groups, but we're open to ideas at this point. We apologize for=20
> any inconvenience this disruption has caused.
>=20
>=20
> Bitcoin-dev mailing list moderation team
>=20
> Bryan Bishop
> Ruben Somsen
> Warren Togami
> various others.
>=20
> --=20
> - Bryan
> https://twitter.com/kanzure <https://twitter.com/kanzure>