@aral@cancel I've been running my own email server since about 2005. These days I don't have any problems at all with email not getting delivered. There was a time about a decade ago when it seemed like some services like hotmail had a default of not accepting email from small domains, or maybe I was a false positive on their spam detection. That no longer seems to be happening.
@aral@cancel I've self hosted several mail servers for a while now (more than 5 years) and never had a real issue with deliverability. It is not that I send a lot of emails but still, something like once a day, to many different email providers.🤔
@cancel@aral Unfortunately this story has two sides: The email protocol was and is abused to an enormous extend to send spam; phising; and most users don’t care why they get spam but rather that they have spam in their inbox. And because of that the big providers lock down their e-mail services.
I don’t like that either. Email hosting from a residential connection is impossible and even most of the common cloud providers are blacklisted; often because they do not react to abuse reports in a timely manner or even at all.
I still host a part of my emails by myself, but I do it from a datacenter connection form a reputable Ip-range. It works but I don’t use it for any kind of „mass-mailing“ because I don’t want to risk my IP.
For the other part of my personal mails I pay for a privacy friendly e2ee email provider.
For my websites and apps that send emails I use a email relay because it is a pain in the ass to send „mass-mails“ (that were requested and are not spam) so that they are actually delivered to my users and are not delivered to the spam folder or rejected.
So it’s probably right to say that email is no longer a open standard. Part of the problem is that most people rely on the few big mail providers like outlook/hotmail; gmail; yahoo and the national ones (like gmx and web.de for Germany)
I can only agree with the article. Also jumping through the hoops of BigMail, which is nothing more than BigSurveillance, but often find that emails are just disappearing. But since a fari amount of emails are delivered to (say) Gmail, I don't think it is blacklists.
I am starting to think it is AI algorithms that no one understands what they do, only see that a lot of the spam is reduced, without knowing how much false positives are filtered out as well. @cancel
This is exactly what I would write, except that I haven't given up yet - but only because I'm more stubborn.
Been running my own email servers for 23 ... maybe 24 years? Didn't have a lot of problems until a few years ago. Like the article's author, my machines have never sent a single spam message, but deliverability to Gmail sucks, even with correct, strict SPF, DKIM, DMARC, correct DNS, and all the rest.
And yes, it's deliberate. They can't monetize public mailing lists.
@aral@cancel Once ran my email on my home broadband from a techie-friendly ISP (thanks @revk), & a Mac Mini with #Debian#Linux & an SMTP server (#Exim? #Postfix? after starting with the baptism of fire that is #Sendmail & the catechism of the Bat Book). But risks to deliverability and effort to keep up with new configurations discouraged me - especially if as this article suggests, the effort would be futile - so switched to one of the big players...
The problem is the oligopoly on email was inevitable due to spam. Basically a recipient can only trust a reasonable subset of possible senders in the push model. Plus spam filtering really does work better when you have lots of users.
The fediverse however is a "pull centric" model, which I think will hopefully act to limit spam and thus the necessary oligopoly needed to mitigate the spam therat.
@aral@cancel We have been self-hosting our email for even longer than that, and it still gets through (except when extended power outages take out our internet). Maybe the difference is that we have a business account with our ISP.
VPS's are blocked en-masse because it is just too easy for spammers to set them up and spew spam into the network from them, and keep making new ones faster than the old ones can be blocked.
@aral@cancel I've been using Fastmail since the mid-nineties. It's not free but as another person pointed out, small email platforms often provide spam control plus PRIVACY. Definitely worth it - better to be a customer than the product.
The gatekeepers you mention aren't really the big companies like google, MS etc. The real gatekeepers are the dns blacklisters like SpamHaus and other minor players. They're the mail routing equivalent of totalitarian dictators. If your address is blacklisted, there's no chance and no appeal. Worse still, they're condoned or even supported by your ISP.
@aral@cancel I host an independent list server. My mails are delivered. It is not all that difficult, but there are lots and lots of hoops to jump through to get email working.
However, to run an email server är home is Impossible nowadays. The reason is spam and spam bots. It is very effective to block all domestic IPs from sending mail as almost none should be doing that anyway.
@aral@cancel we've been running a self-hosted (postfix) email server for at least 12 years. Originally my son ran it in the cloud but didn't maintain it properly so it ended up being blacklisted when someone worked out how to use it as an open relay. I took it over and moved it on a small server downstairs in a cupboard with a fixed IP line. I set it up using an up to date guide on how to, use DKIM. etc. and haven't had any issues.
Note sure I want to run a Mastodon instance though.
Yes, running an independent email server is haaaaard. But at the same time, there are tons of those mid/small-size email servers that will happily sell you a safe, not tracking mailbox. So it can be done. And it is being done.
@aral@cancel I don't know if I'm just lucky or something, but I've had my own email server (on a VPS) for the past five years maybe. I had some problems where my emails would end up in spam folders in the beginning, but , that was because of my own misconfiguration.
It is a bit of work to upgrade it now and then, but it's not a huge pain in the ass, and it works fine.
also gave up my own mail server about a year ago and went to an established provider. The oligopoly is not quite as concentrated as for social media - the culprit that wouldn't accept my mails anymore (as well as the provider that's my "solution" now) are mostly active on the German-speaking market. So #BigTech may be stretching it, but it is a problem that an open protocol can de-facto only be used by subscribing to commercial services with a certain minimum market share.
@aral@cancel I self hosted circa 2000-2003, using a fanless Pentium MMX in a cardboard box running Slackware. Which was fun until it inevitably got hacked multiple times because I didn't know how to secure it, and learning would be boring. I can't imagine how much worse it is now to keep an email server remotely secure.
As a past mail platform sysadmin Boy Do I Have Thoughts...
Fundamentally the email problem is economic. There's a price point at which people are no longer willing to pay for email. Most people's price point is poverty or "nah that costs money".
Contrasted with that is the massive amounts of money needed to keep email systems even up and running in the face of the onslaught of spam. I worked at a company which purchased several anti-spam systems, had a full email platform team, had a separate anti-spam team which could barely keep up. We had an anti-spam system that, by volume, only rarely delivered actual legitimate email. It really spent most of its time handling spam.
There's a very thin space between "I expect this but am not paying $50/month for it how about $0/month" and the cost of the anti-spam measures. Out of that space comes all the profit the company expects to extract and also the parts used to actually store and deliver the mail. These incentives favour scale (try costing resilient, redundant storage for IMAP for a boutique server of a few hundred users, without going cloud).
It's of course fine that somebody can run an artisanal email server on their spare time as a hobby, it can be fun like woodworking or knitting, but it's a different proposition than running email industrially. In the same way as how hobby woodworking or knitting are different than running a furniture or clothing factory and selling into markets for those goods.
For my part, I'm not sure open, anybody-can-contact-me protocols work as well on the internet of 2023 as they did on the internet of 1983. Anybody includes a lot of people and software that you don't want email from.
I have no idea if email will stick around or end up eventually much like Usenet: existing but diminished. Time will definitely tell. The pressures on the whole system are vast and fairly intractable.
@aral@cancel sorry but it would take significantly smaller effort to find a reputable and trustworthy email provider and pay them a small fee, than moaning and complaining about the self-hosted email service gone wrong, which was, frankly, not the best idea in first place.
@aral@cancel can you provide an example with similar "new tech entering public domain and then enshittified" where the oligopoly takeover didn't happen? Even electricity here In Europe is like that. There ARE alternative providers but they all piggyback on an infrastructure of two or three megacorps.
@aral@cancel@jon I guess I am one of the outliers. I have been running my own e-mail server for 10+ years, it Just Works(tm), I can send and receive from/to everyone, including Google, Microsoft, t-online. It runs email for all of the 20+ domains I own. I documented my current setup in a 5 part blog series at https://jan.wildeboer.net. So yes, it is absolutely possible to run your own email server, IMHO.
@aral@cancel When you say VPS, you mean hosted somewhere else? I'm trying to understand. I too was on Internet in 1987 although it wasn't until the late 1990s that I was dealing with sendmail and spamasassin and all kinds of recipes. It's been maybe 10 years since I ran a VPS (or dedicated for that matter) so maybe I'm out of touch.
@aral As someone who hosts his own email, and who works in email professionally, that rant bears little resemblance to the current state of email hosting. “hellbanning” sounds like a whiny 12yo, and if you know much about email the rest of it is just as wrong.
@aral@cancel self hosted email still works fine, you just have to have it set up right with a domain and certs. If you can't run secure you're blocked, as you're more than likely just an open spam relay. People complaining about being blocked are just people who haven't learned how to set it up properly.
@aral@cancel Yes. This. I have had the same email on my own domain since 1993 or thereabouts, but for the past year or so I have gradually needed to migrate various accounts to use a gmail address. It is really annoying when the 'Verify your email account' and password reset emails from various sites simply never arrive. Some sent emails also seem to vanish without a trace
@aral@cancel I must say that personally, I cannot relate to that. I have a small dedicated server of my own, hosted with a big German hosting provider. It serves web and mail for 22 domains. I have implemented SPF, DKIM, Dmarc, TLS transport security, have valid forward and reverse DNS. Mail works. IP it not any blacklist. Am I just lucky?
@aral@cancel I remember reading that post last year. It's even worse when you consider that this enclosure of the email commons happened as part of the 'fight' against the scourge of junk email... which was caused primarily by the self-same big companies who've now enclosed email.
@aral@cancel Honest to christ I read that when it came out and my reaction was and still is "what are you on about?" because I've been doing it for longer and still am and I don't take extraordinary measures. I have a DKIM record and proper rdns and my ducks generally in a row and the amount of time I spend managing this a month (above and beyond normal server maintenance) is very, very close to zero. Most months it is zero.
So every time somebody forwards that around again I'm like "what."
@aral@cancel email’s usefulness appears over. Seemingly these days it’s been reduced to being an identifier, a method to rest passwords, a store of order confirmation emails, and most of all, a never ending barrage of marketing messages.
@aral@cancel hmmm... I have my own mail server in my basement and yes, it can be a pain to keep spam filtering up to date and sometimes Big Tech implement stuff which takes me a week to master, but I can still send and receive mail from and to everywhere.
@aral@cancel Just don't comply with their demands. Eventually someone is going to have to contact you but can't. If enough independent servers are uncontactable, it will become a problem with the big companies. Eventually, companies will have to cave. @ryo has already made a blogpost about it. (Tor only though)
Can you stand up an email server, open the ports, and hope to play nice? No, you can't, and it isn't because of evil gatekeepers, but because of the necessity of keeping spam traffic off the network.
I run email servers right now, and have off and on since the 1990s. My servers are hosted on private domains.
It is a straight pain in the ass, involving several layers of security, involved participation in a variety of DNS and SSL based security patterns which are somewhat effective today and may well not be effective at all tomorrow.
I'm playing by those rules, and will continue to do so as long as my emails continue to be delivered.
Anyone making this argument is either ignorant or simply doesn't want to deal with the trouble of running a modern email service.
Or maybe they're just being completely disingenuous; what I do know for certain is that my emails continue to be delivered.
@aral@cancel Question: why haven't people like you replaced email with a standardized web contact interface? I was thinking you'd provide a website link with some name for a soon-to-be-familiar-looking interface, where people can leave you messages or even files on your own server. The idea of SMTP, with personal messages running around through multiple unsecured servers, never really made that much sense anyway.
@aral@cancel We can't let this happen with other self hosted services. It is currently much easier to run a fediverse instance, RSS server, and other services, but people using browsers and other tools provided by the big gatekeepers get scary messages about the site being unsafe. I tell people to switch to a different browser or email client that doesn't 'monitor them for their protection'. I don't know what other solutions we have to get around them and prevent this from expanding.
@cancel@aral I’ve been running my own mail server on a VPS without issue for a few years now. Once all the SPF, rDNS, DKIM, certificates were sorted I made it past even the most strict servers like Google and Office 365.
@aral@cancel This isn't simply about spam -- from CALEA to #chatcontrol , the main purpose of online communications has been to generate records for prosecution. It is harder for companies to scan, AI-read, archive, and provide law enforcement services about your email ... if they don't have a stranglehold on the email! See also
@aral@cancel fucking connection providers blocking the ports for email. For the moment I solved by using a forward service. I don't manage a high volume of emails. Not the best for privacy but you can encrypt your email if this is an issue for you. Where I live if you don't own a static IP you're dead meat.
@aral@cancel The network is populated by hostile actors as well, solving spoofing and spam required giving the best connected nodes of the network a proportional responsibility for its authentication, to cooperate for the common good. That is how networks work. That's what decentralization actually looks like. Now the question is why did some nodes get so damn big?
@aral@cancel I have helped run an email server for a decade for a few thousand people. I think that to use the words "embrace" and "extend" misrepresents the problems, attributing malice where there is only a mismanagement of network effects and a lack of long-term vision. Much of the internet is rickety infrastructure maintained with the precarity of sailors treading water in a perennially sinking boat.
@aral@cancel I disagree. Running a secure email server with a good reputation is hard work, and not the kind of thing you can do casually any more, but I'm not convinced that the big providers have much of an advantage beyond having the resources to properly configure and manage their services (and IP blocks). Actually, even Microsoft has some low-rep IP blocks (which it manages to its advantage). If anyone has the time to manage their mail services properly, they absolutely can still do that.
@aral@cancel But why? The primary destroyer of e-mail was spam, not the silos. The silos triumphed because they were able to defend well enough against spam to keep e-mail useful. That’s my concern about the fediverse. I don’t see enough inherent hardening in the design against organized large scale abuse.
@aral@cancel while I do feel this I don't think its really that true. I self-host my email and only once every 30/60days do I have to fallback to a mail address from the big 5. 29/30 times it works fine.
@aral this is breaking my heart, I shouldn’t have read this first thing in the morning 😭
I’m intimately familiar with this problem (been running my own mail since at least 1999) and they’ve almost worn me down too. When I consider this I can’t see any reason they can’t use the same trick to undermine any protocol, including ActivityPub.
The worst part is that my email server work just fine talking to other email servers, it’s just not compatible with AOL err.. Gmail.
@aral@cancel all that being true, if email had opt-in for receiving email (aka 'follow') my inbox would be spam free. And the moving goalposts of DKIM & SPF records & mail server IP reputation etc that were added to try deal with spam/filters and allowed the monopolies to squash competitors, wouldn't be needed.
(Which isn't to say the fediverse doesn't have big centralisation vulnerabilities say around caching/media storage/notifications etc - but more that not all monopoly risks are the same)
@aral@cancel many people here complacent about this but I've been given reason to be optimistic by the numbers who aren't, and also how some who were have heard my arguments and changed their minds. (Mainly on my tech account. I did a poll there recently and off the bat 30% favoured banning corporate instances and there were some great discussions in the replies.)
That's why proactive blocking of entrenched interests coming in should be a strategy in fediverse communities. Most totalitarianism friendly instances get blocked I don't see why should Google, Apple, Microsoft, Medium, Valve, etc be allowed free reign.
@aral@cancel let's be very clear, the email oligarchy became a thing because everyone took email for granted. They failed to evolve the user experience. The email #ux literally stagnated for a long time. That's why Gmail was so huge, it came with fresh ideas and experiences. It made email pleasant again and forced the other big providers to do the same. Sadly local application based email wasn't as good and so everyone moved to the next available.
@aral@cancel I keep seeing this comparison and IMO it misses the point. Consolidation of non-technical users onto well-run servers isn’t an if but a when and at some point spam will likely force those servers to not default to federate. However the fediverse, like email, always allows for an escape hatch. Expecting to leave it default open for federation is not realistic. Similar to BGP, the era of the “assume positive intent” internet is long over.
@aral@cancel this got to me a bit and it's weird to think how much email has changed.
I've been round long enough to remember when everyone's email addresses were either their university, company or ISP. Even most universities now seem to just point their domain at Gmail or Microsoft. It's a real loss to the diversity of the net.
I can easily imagine a future where megacorp fediverse instances grow and then defederate from independent servers.
@aral@cancel I made an anonymous message box on my onion website that sends a message into a txt file on my computer. I think I can do a reply page setup. So people could check for my replies. That could go around all of the big tech bullshit.
@aral@cancel the difference is that with email part of the deal is that you can send a message to anyone else with email. Nobody would join a group of 100 like-minded individuals and have their own email network in addition to the Big Email one.
The fediverse is more like a loose collection of forum sites which intercommunicate - so while yes, there could end up an oligopoly as far as the masses are concerned, that doesn't stop the open fediverse from existing alongside it.
@aral@cancel That's exactly what people get scared about when they hear that companies like Cloudflare get interested in the Fediverse.
I did not give it a deep look, but Cloudflare's Wildebeest runs exclusively on Cloudflare infrastructure. That's already a way towards a strong position for 'extend' and also 'extinguish' techniques like "spam filtering".
@aral@cancel Correct. It is unlikely that individual fediverse servers will continue to be a thing and that would be a terrible shame because there is no fediverse without treating the smallest and/or the slowest (or not always online) servers the same as the biggest & the fastest ones.