@aral @cancel I've been running my own email server since about 2005. These days I don't have any problems at all with email not getting delivered. There was a time about a decade ago when it seemed like some services like hotmail had a default of not accepting email from small domains, or maybe I was a false positive on their spam detection. That no longer seems to be happening.
@cancel @aral Unfortunately this story has two sides:
The email protocol was and is abused to an enormous extend to send spam; phising; and most users don’t care why they get spam but rather that they have spam in their inbox. And because of that the big providers lock down their e-mail services.
I don’t like that either. Email hosting from a residential connection is impossible and even most of the common cloud providers are blacklisted; often because they do not react to abuse reports in a timely manner or even at all.
I still host a part of my emails by myself, but I do it from a datacenter connection form a reputable Ip-range. It works but I don’t use it for any kind of „mass-mailing“ because I don’t want to risk my IP.
For the other part of my personal mails I pay for a privacy friendly e2ee email provider.
For my websites and apps that send emails I use a email relay because it is a pain in the ass to send „mass-mails“ (that were requested and are not spam) so that they are actually delivered to my users and are not delivered to the spam folder or rejected.
So it’s probably right to say that email is no longer a open standard. Part of the problem is that most people rely on the few big mail providers like outlook/hotmail; gmail; yahoo and the national ones (like gmx and web.de for Germany)
Another observation; I used GMail from its start to about 3 years ago. Spam folder used to hover around 2500 mails, and Spam folder was limited to 30 days, so ~80 spam per day.
After I run my own server (again), that has dropped to ~10-20/day.
Somehow the spammers don't want to send to my server. I (seriously) wonder why?
I can only agree with the article. Also jumping through the hoops of BigMail, which is nothing more than BigSurveillance, but often find that emails are just disappearing. But since a fari amount of emails are delivered to (say) Gmail, I don't think it is blacklists.
I am starting to think it is AI algorithms that no one understands what they do, only see that a lot of the spam is reduced, without knowing how much false positives are filtered out as well.
@cancel
*14 years ago* i wrote an embeddable mail server, and promptly discovered that it was useless, because SPF authorization meant that mail coming from any local isp was going to blocked.
this wasn't because of Google & friends wanting to control things. it was because spammers were co-opting people's personal computers to send mail.
Yes, it's an oligopoly, but it's an oligopoly because the spammers made us take up defensive measures that forced us into a corner.
This is exactly what I would write, except that I haven't given up yet - but only because I'm more stubborn.
Been running my own email servers for 23 ... maybe 24 years? Didn't have a lot of problems until a few years ago. Like the article's author, my machines have never sent a single spam message, but deliverability to Gmail sucks, even with correct, strict SPF, DKIM, DMARC, correct DNS, and all the rest.
And yes, it's deliberate. They can't monetize public mailing lists.
@aral @cancel Once ran my email on my home broadband from a techie-friendly ISP (thanks @revk), & a Mac Mini with #Debian #Linux & an SMTP server (#Exim? #Postfix? after starting with the baptism of fire that is #Sendmail & the catechism of the Bat Book). But risks to deliverability and effort to keep up with new configurations discouraged me - especially if as this article suggests, the effort would be futile - so switched to one of the big players...
The problem is the oligopoly on email was inevitable due to spam. Basically a recipient can only trust a reasonable subset of possible senders in the push model. Plus spam filtering really does work better when you have lots of users.
The fediverse however is a "pull centric" model, which I think will hopefully act to limit spam and thus the necessary oligopoly needed to mitigate the spam therat.
@aral @cancel FWIW I have managed to get successful delivery again.
It took ensuring that reverse DNS of the IP-numbers of the MTA hos resolved to a name in the same DNS zone as the MX record, to get delivery.
It's possible that the same could have been accomplished with DNS SPF records, but I have never been able to set up SPF records that have seemed to have any effect.
@aral @cancel We have been self-hosting our email for even longer than that, and it still gets through (except when extended power outages take out our internet). Maybe the difference is that we have a business account with our ISP.
VPS's are blocked en-masse because it is just too easy for spammers to set them up and spew spam into the network from them, and keep making new ones faster than the old ones can be blocked.
The gatekeepers you mention aren't really the big companies like google, MS etc. The real gatekeepers are the dns blacklisters like SpamHaus and other minor players. They're the mail routing equivalent of totalitarian dictators. If your address is blacklisted, there's no chance and no appeal. Worse still, they're condoned or even supported by your ISP.
@aral @cancel I host an independent list server. My mails are delivered. It is not all that difficult, but there are lots and lots of hoops to jump through to get email working.
However, to run an email server är home is Impossible nowadays. The reason is spam and spam bots. It is very effective to block all domestic IPs from sending mail as almost none should be doing that anyway.
@aral @cancel we've been running a self-hosted (postfix) email server for at least 12 years.
Originally my son ran it in the cloud but didn't maintain it properly so it ended up being blacklisted when someone worked out how to use it as an open relay.
I took it over and moved it on a small server downstairs in a cupboard with a fixed IP line.
I set it up using an up to date guide on how to, use DKIM. etc. and haven't had any issues.
Note sure I want to run a Mastodon instance though.
I am not sure I completely follow this logic.
Yes, running an independent email server is haaaaard. But at the same time, there are tons of those mid/small-size email servers that will happily sell you a safe, not tracking mailbox. So it can be done. And it is being done.
What am I missing from this discussion?
@aral @cancel I don't know if I'm just lucky or something, but I've had my own email server (on a VPS) for the past five years maybe. I had some problems where my emails would end up in spam folders in the beginning, but , that was because of my own misconfiguration.
It is a bit of work to upgrade it now and then, but it's not a huge pain in the ass, and it works fine.
also gave up my own mail server about a year ago and went to an established provider. The oligopoly is not quite as concentrated as for social media - the culprit that wouldn't accept my mails anymore (as well as the provider that's my "solution" now) are mostly active on the German-speaking market.
So #BigTech may be stretching it, but it is a problem that an open protocol can de-facto only be used by subscribing to commercial services with a certain minimum market share.
@aral @cancel I self hosted circa 2000-2003, using a fanless Pentium MMX in a cardboard box running Slackware. Which was fun until it inevitably got hacked multiple times because I didn't know how to secure it, and learning would be boring. I can't imagine how much worse it is now to keep an email server remotely secure.
As a past mail platform sysadmin Boy Do I Have Thoughts...
Fundamentally the email problem is economic. There's a price point at which people are no longer willing to pay for email. Most people's price point is poverty or "nah that costs money".
Contrasted with that is the massive amounts of money needed to keep email systems even up and running in the face of the onslaught of spam. I worked at a company which purchased several anti-spam systems, had a full email platform team, had a separate anti-spam team which could barely keep up. We had an anti-spam system that, by volume, only rarely delivered actual legitimate email. It really spent most of its time handling spam.
There's a very thin space between "I expect this but am not paying $50/month for it how about $0/month" and the cost of the anti-spam measures. Out of that space comes all the profit the company expects to extract and also the parts used to actually store and deliver the mail. These incentives favour scale (try costing resilient, redundant storage for IMAP for a boutique server of a few hundred users, without going cloud).
It's of course fine that somebody can run an artisanal email server on their spare time as a hobby, it can be fun like woodworking or knitting, but it's a different proposition than running email industrially. In the same way as how hobby woodworking or knitting are different than running a furniture or clothing factory and selling into markets for those goods.
For my part, I'm not sure open, anybody-can-contact-me protocols work as well on the internet of 2023 as they did on the internet of 1983. Anybody includes a lot of people and software that you don't want email from.
I have no idea if email will stick around or end up eventually much like Usenet: existing but diminished. Time will definitely tell. The pressures on the whole system are vast and fairly intractable.
@aral @cancel there's some recognition and of a possible solution to this problem using #objectCapabilities #ocaps #ocappub
https://gitlab.com/spritely/ocappub/blob/master/README.org
@aral @cancel can you provide an example with similar "new tech entering public domain and then enshittified" where the oligopoly takeover didn't happen? Even electricity here In Europe is like that. There ARE alternative providers but they all piggyback on an infrastructure of two or three megacorps.
@aral @cancel @jon I guess I am one of the outliers. I have been running my own e-mail server for 10+ years, it Just Works(tm), I can send and receive from/to everyone, including Google, Microsoft, t-online. It runs email for all of the 20+ domains I own. I documented my current setup in a 5 part blog series at https://jan.wildeboer.net. So yes, it is absolutely possible to run your own email server, IMHO.
Yep, you got me ;)
When I wrote “like email” 2 hours ago, in fact I meant “like email in the old days” :)
https://digitalcourage.social/@sl007/109731764313382200
@aral @cancel When you say VPS, you mean hosted somewhere else? I'm trying to understand. I too was on Internet in 1987 although it wasn't until the late 1990s that I was dealing with sendmail and spamasassin and all kinds of recipes. It's been maybe 10 years since I ran a VPS (or dedicated for that matter) so maybe I'm out of touch.
@aral As someone who hosts his own email, and who works in email professionally, that rant bears little resemblance to the current state of email hosting. “hellbanning” sounds like a whiny 12yo, and if you know much about email the rest of it is just as wrong.
@aral Are you sure you, posting such stories, are not helping bigtech to kill email? There's plenty of examples online of people that still self-host email...
@djoerd You got me. It was me all along 🤫
@aral @cancel self hosted email still works fine, you just have to have it set up right with a domain and certs. If you can't run secure you're blocked, as you're more than likely just an open spam relay. People complaining about being blocked are just people who haven't learned how to set it up properly.
@aral @cancel Yes. This. I have had the same email on my own domain since 1993 or thereabouts, but for the past year or so I have gradually needed to migrate various accounts to use a gmail address. It is really annoying when the 'Verify your email account' and password reset emails from various sites simply never arrive. Some sent emails also seem to vanish without a trace
@aral @cancel I must say that personally, I cannot relate to that. I have a small dedicated server of my own, hosted with a big German hosting provider. It serves web and mail for 22 domains. I have implemented SPF, DKIM, Dmarc, TLS transport security, have valid forward and reverse DNS. Mail works. IP it not any blacklist. Am I just lucky?
@aral @cancel Honest to christ I read that when it came out and my reaction was and still is "what are you on about?" because I've been doing it for longer and still am and I don't take extraordinary measures. I have a DKIM record and proper rdns and my ducks generally in a row and the amount of time I spend managing this a month (above and beyond normal server maintenance) is very, very close to zero. Most months it is zero.
So every time somebody forwards that around again I'm like "what."
@aral @cancel @benjaminhollon I read this some time ago and having run my company’s mailserver for a while with some colleagues, this is spot on. I hope fediverse doesn’t fall into the same trap
@aral @cancel "tried all the silver bullets recommended by Hacker News, used kafkaesque request forms to prove legitimity, contacted the admins of some blacklists."
oh my gooddd. as someone who's had to do this in recent memory just because the domain name *extension* was flagged as "bad vibes" i feel this article in my bones.
yes, spam is an issue, but email deliverability has become a shitfest.
jeezaus
and this is a good heads-up. big tech will obviously try to throttle mastodon too =/
@aral @cancel Just don't comply with their demands. Eventually someone is going to have to contact you but can't. If enough independent servers are uncontactable, it will become a problem with the big companies. Eventually, companies will have to cave. @ryo has already made a blogpost about it. (Tor only though)
Something I know a little something about.
The forgoing post is only partly true.
Can you stand up an email server, open the ports, and hope to play nice? No, you can't, and it isn't because of evil gatekeepers, but because of the necessity of keeping spam traffic off the network.
I run email servers right now, and have off and on since the 1990s. My servers are hosted on private domains.
It is a straight pain in the ass, involving several layers of security, involved participation in a variety of DNS and SSL based security patterns which are somewhat effective today and may well not be effective at all tomorrow.
I'm playing by those rules, and will continue to do so as long as my emails continue to be delivered.
Anyone making this argument is either ignorant or simply doesn't want to deal with the trouble of running a modern email service.
Or maybe they're just being completely disingenuous; what I do know for certain is that my emails continue to be delivered.
@aral @cancel Question: why haven't people like you replaced email with a standardized web contact interface? I was thinking you'd provide a website link with some name for a soon-to-be-familiar-looking interface, where people can leave you messages or even files on your own server. The idea of SMTP, with personal messages running around through multiple unsecured servers, never really made that much sense anyway.
(But can you dodge the ID check for https now?)
@aral @cancel We can't let this happen with other self hosted services. It is currently much easier to run a fediverse instance, RSS server, and other services, but people using browsers and other tools provided by the big gatekeepers get scary messages about the site being unsafe. I tell people to switch to a different browser or email client that doesn't 'monitor them for their protection'. I don't know what other solutions we have to get around them and prevent this from expanding.
@aral @cancel This isn't simply about spam -- from
CALEA to #chatcontrol , the main purpose of online communications has been to generate records for prosecution. It is harder for companies to scan, AI-read, archive, and provide law enforcement services about your email ... if they don't have a stranglehold on the email! See also
@aral @cancel fucking connection providers blocking the ports for email. For the moment I solved by using a forward service. I don't manage a high volume of emails. Not the best for privacy but you can encrypt your email if this is an issue for you. Where I live if you don't own a static IP you're dead meat.
@aral @cancel The network is populated by hostile actors as well, solving spoofing and spam required giving the best connected nodes of the network a proportional responsibility for its authentication, to cooperate for the common good. That is how networks work. That's what decentralization actually looks like. Now the question is why did some nodes get so damn big?
@aral @cancel I have helped run an email server for a decade for a few thousand people. I think that to use the words "embrace" and "extend" misrepresents the problems, attributing malice where there is only a mismanagement of network effects and a lack of long-term vision. Much of the internet is rickety infrastructure maintained with the precarity of sailors treading water in a perennially sinking boat.
@aral @cancel I disagree. Running a secure email server with a good reputation is hard work, and not the kind of thing you can do casually any more, but I'm not convinced that the big providers have much of an advantage beyond having the resources to properly configure and manage their services (and IP blocks). Actually, even Microsoft has some low-rep IP blocks (which it manages to its advantage). If anyone has the time to manage their mail services properly, they absolutely can still do that.
@aral @cancel But why? The primary destroyer of e-mail was spam, not the silos. The silos triumphed because they were able to defend well enough against spam to keep e-mail useful. That’s my concern about the fediverse. I don’t see enough inherent hardening in the design against organized large scale abuse.
@aral this is breaking my heart, I shouldn’t have read this first thing in the morning 😭
I’m intimately familiar with this problem (been running my own mail since at least 1999) and they’ve almost worn me down too. When I consider this I can’t see any reason they can’t use the same trick to undermine any protocol, including ActivityPub.
The worst part is that my email server work just fine talking to other email servers, it’s just not compatible with AOL err.. Gmail.
Fuck
@aral @cancel all that being true, if email had opt-in for receiving email (aka 'follow') my inbox would be spam free. And the moving goalposts of DKIM & SPF records & mail server IP reputation etc that were added to try deal with spam/filters and allowed the monopolies to squash competitors, wouldn't be needed.
(Which isn't to say the fediverse doesn't have big centralisation vulnerabilities say around caching/media storage/notifications etc - but more that not all monopoly risks are the same)
Seeing a lot of parallels to my own experience here with self-hosted email.
I had a personal server setup for $5 a month, handling all the mail for my immediate family. Did all the same hoops mentioned in the post, and still saw emails going to spam or “hellbanned”.
Ended up moving it over to Fastmail, it was far too much hassle to keep running and fighting blocklists.
@aral @cancel many people here complacent about this but I've been given reason to be optimistic by the numbers who aren't, and also how some who were have heard my arguments and changed their minds. (Mainly on my tech account. I did a poll there recently and off the bat 30% favoured banning corporate instances and there were some great discussions in the replies.)
It's not a simple "my personal server" vs "big tech".
There are many smaller companies that provide email.
One of the advantages of using a specialist provider vs self-hosting is that it's their full-time job to manage spam filters and blacklists while ensuring their mail gets through.
You'll get better service than big tech.
I've worked for one before. Managing spam and malware, and following up blacklists is a lot of work.
@aral @cancel let's be very clear, the email oligarchy became a thing because everyone took email for granted. They failed to evolve the user experience. The email #ux literally stagnated for a long time. That's why Gmail was so huge, it came with fresh ideas and experiences. It made email pleasant again and forced the other big providers to do the same. Sadly local application based email wasn't as good and so everyone moved to the next available.
@aral @cancel I keep seeing this comparison and IMO it misses the point. Consolidation of non-technical users onto well-run servers isn’t an if but a when and at some point spam will likely force those servers to not default to federate. However the fediverse, like email, always allows for an escape hatch. Expecting to leave it default open for federation is not realistic. Similar to BGP, the era of the “assume positive intent” internet is long over.
@aral @cancel this got to me a bit and it's weird to think how much email has changed.
I've been round long enough to remember when everyone's email addresses were either their university, company or ISP. Even most universities now seem to just point their domain at Gmail or Microsoft. It's a real loss to the diversity of the net.
I can easily imagine a future where megacorp fediverse instances grow and then defederate from independent servers.
FWIW, just FTR, just my 2 cents etc...
I and many others are still fighting the good fight. See here:
https://stop.zona-m.net/tag/email/
the posts marked in the screenshot. If you like them, and those coming in the next reply about the other Standards That Must Not Die, thanks in advance for sharing them as much as you see fit.
@aral Exceptionally clear and well written. I also read his proposal for a solution. Could NGO's like @edri and @bitsoffreedom help by advocating? How should we go about this?
@aral @cancel the difference is that with email part of the deal is that you can send a message to anyone else with email. Nobody would join a group of 100 like-minded individuals and have their own email network in addition to the Big Email one.
The fediverse is more like a loose collection of forum sites which intercommunicate - so while yes, there could end up an oligopoly as far as the masses are concerned, that doesn't stop the open fediverse from existing alongside it.
I've always thought of email as a service like old telephony, electricity delivery system, or a reasonably well paved highway system to take long motorbike rides on.
The idea of hosting a mail server sounds interesting but coming from the days of teletype and BBS it's an anachronism akin to a "free" operator doing a directory lookup for a "land line" number.
@aral @cancel That's exactly what people get scared about when they hear that companies like Cloudflare get interested in the Fediverse.
I did not give it a deep look, but Cloudflare's Wildebeest runs exclusively on Cloudflare infrastructure. That's already a way towards a strong position for 'extend' and also 'extinguish' techniques like "spam filtering".
@aral @cancel this precisely. Cory Doctorow ( @pluralistic ) wrote about the same topic here: https://doctorow.medium.com/dead-letters-73924aa19f9d
@aral @cancel I don't think e-mail is example of embrace extend extinguish. I think it has two other problems:
1) people mostly did not self-hosted in past (you got your e-mail adress from employer, university, internet provider or big web portal)
2) SPAM
Also e-mail is ancient protocol and tooling is also ancient.
@aral@mastodon.ar.al @cancel@merveilles.town Sharing the fear. If mastodon.social defederates you, you're out. #fediverse
@cancel@merveilles.town @aral@mastodon.ar.al That's… honorable, but IMHO like refusing to talk to gmail as a SMTP. Most of the people are there.
@adnan@1210.nl @aral@mastodon.ar.al @cancel@merveilles.town you should have a look at https://microblog.pub/. 
@aral There is even a Wikipedia article about this lethal strategy: https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguish
“The #fediverse is like #email.”
Yes.
Now read this and understand it:
“I have been self-hosting my email since I got my first broadband connection at home in 1999 … But my emails are just not delivered anymore. I might as well not have an email server.
Email is now an oligopoly, a service gatekept by a few big companies which does not follow the principles of net neutrality … I lost. We lost. One cannot reliably deploy independent email servers.“
https://cfenollosa.com/blog/after-self-hosting-my-email-for-twenty-three-years-i-have-thrown-in-the-towel-the-oligopoly-has-won.html
Via @cancel