On the lack of simple and reliable software to self-host a website

As far as I can tell, there is no computer program that makes it simple and easy to self-host a website for publishing content and hosting discussions.

By self-hosting, I don't mean necessarily on your own computer. A server that you rent out from some provider is acceptable too. As long as you can easily move to a different provider without loss.

Why this matters

This is important for the freedom of people on the internet.

The internet is becoming more and more centralized.

• You don't own your data

• BigTech can shut you down

The answer to this is to host your own website and be in control of the data on it.

But with the currently existing tools, it's not easy.

• Setup is difficult

• Maintenance is difficult

• Scaling is difficult

The current systems are all needlessly complex, and they often lack important capabilities.

When something is difficult to do, very few people will do it.

It doesn't matter if there are systems out there that let you do this if they are too complex and difficult. It doesn't even matter if they are open source. The fact that they are open source contributes almost nothing if they are also complex and near impossible to operate without extensive sysadmin knowledge.

In the following section I will outline what it takes for self-hosting system to be easy, reliable, capable and useful.

Outline

The system should have two components:

1. A management program that runs on the user's local machine for, well, management and administration,

2. A server program that actually runs the website and serves its content over http / https.

The management program manages the installation of the server program on a remote server (likely linux) given just the IP and root login/password of the server.

The installation time is basically just the file upload time. The actual installation is nothing but creating a specific user and home directory for that user and giving the server program the capability to listen on ports 80 and 443. It shouldn't take more than a fraction of a second.

Nothing on the server needs to be messed with or configured. No "packages" need to be installed. Ever.

The local manager program is an easy to use GUI and comes with a dashboard for monitoring the server and changing its settings.

There should be nothing to configure. There are only settings that tweak and tune the behavior of the server.

The difference between settings and configuration is that if you configure something wrong, the system might break and stop working. With settings, you can't break the server or cause it to stop working by accident.

All settings should be optional. The program should still be able to perform basic operations even if some information is not provided.

For example, if no domain name is configured, the server can still run and be accessible by IP:port combination.

The server should manage SSL certificate using services that follow the ACME protocol. It can use "LetsEncrypt" by default, and allow the user to switch to another provider via settings. If the certificate management subsystem fails for any reason, the server should be able to detect that and degrade gracefully to just serving over http.

The website should feature dynamic content and interaction with users. For example, at the very bare minimum, posting articles and having comments for discussing these articles with visitors. Much better would include ability to upload media (pictures, video, audio) and providing a full-featured forum and chatting functionality, along with email subscription and a way to sell digital (or physical) products without the high fees that other platforms charge.

Configuration of appearance would be nice to have but I don't think it's essential. The default appearance should be so good that no one would have a strong objection against it. The system could ship with a few builtin themes for people who like variety, but full-on customization is probably more trouble than is worth.

The local management program should enable the user (the website owner) to own all the data so they can change VPS providers, etc, without losing a single bit of data. For example, it could feature continuous backup to the local file system. It should also make it possible to backup data selectively. For example, backup articles but not forum posts or chat messages. Or maybe provide a date filter, for example, only backup chat messages 10 days and newer; discarding older ones.

The client side manager program should feature a dashboard that displays various analytics such as number of visitors over time, resource usage over time, data usage on disk, emails sent over time and their status .. etc.

The server should be able to handle reasonably high load on a single machine that is cheap to rent. For example, it should be able to effortlessly handle thousands of concurrent users that are actively using the site and interacting with it on $5/mo hosting. No one should be worried about "hug of death" when they post links on popular websites.

The server should also be able to scale horizontally by adding peer nodes. This would be considered an advanced use case that the vast majority of users wouldn't have to bother with. The way to do it will via the management program. Adding new nodes is not much different than the initial setup: just provide the IP and root login. The system will automatically manage all the nodes and make sure they shard the data and split the load in a reasonable manner.

The server should be able to send emails without the user having to signup for a third party emailing service. In other words, the program can act like a mail server and manage sending emails robustly.

Perhaps the user would have to configure some DNS records for this to work, so the program should be able to verify that the DNS records are set correctly, and in case of a problem, it should inform the user which setting needs to be edited and how, in very clear and easy to understand way. Ideally it should be able to manage the DNS records on behalf of the user, or provide the UI necessary for the user to edit DNS records in a more controlled way that does not lend itself to errors.

The program need not be "open source". That is a mere distraction and does not actually matter - despite what many may think. In fact, the program should be sold for money in order to be sustainable. The source code could be provided for additional fees, but the license does not need to abide by the FSF or OSI definition of open source. Namely, the right to "redistribute" the source code or the program that is built from it is a phony right that actively hinders the effort of making it sustainable for the makers to continue making and improving the program.

Current Situation

Almost every single tool for running a website or a specific type of web application requires sysadmin level knowledge to successfully get started with installing and running it, and even more knowledge is required to keep it running and recover from problems.

At minimum, they often require the installation and configuration of a database server, which is a separate program you install before you can install the website you want to run.

It is for this reason - I believe - that static site generators are gaining some popularity. They are easier to manage and they guarantee ownership of data. Since you write everything on your local machine, no one can take anything away from you.

The unfortunate thing about static websites is they lack interaction with visitors, which is the whole point of the web. If you want comments, you must use a third party hosted service, which defeats the whole point of owning your data and keeping the web decentralized.

For example, the open source blogging and newsletter platform "ghost" requires installing and configuring nginx, mysql, and nodejs. Wordpress requires php, apache and mysql, but the official ubuntu guide lists 14 packages that need to be installed. The guide includes cryptic configurations and unfamiliar system commands that even I as a technical person don't understand. Discourse requires Ruby, Regis, Postgres, Sidekiq, and nginx. They don't actually tell you how to configure these system. Instead, they only officially support installing with Docker. Initial configuration is done via CLI prompts, but later changes requires editing text files and running docker commands.

Why the situation persists

Programmers love complexity. In a way, programming is an activity where you wrangle with complexity and beat it down. It feels like an achievement when you take a complex and hard to use computer system and still manage to make it do something useful. So you have people who thrive on editing docker and nginx and haproxy config files to make a website work. Specially when they lack the skills necessary for low level programming. It's similar to why people play puzzle games. The complexity of configuring multiple unix-style programs towards performing a certain task is their puzzle game.

For companies, there's no economic incentive to make things easy and simple. If you make it hard to self-install, you can provide a hosted solution and charge users a subscription fee. Subscription fees are recurring, so they are a more attractive business model than one-off sales.

This is even more true in the case of open source projects.

The most common model of monetizing open-source used to be paid support (though it seems the trend is shifting towards providing managed hosting). In either case, if the product is easy to setup and maintain, there would be no reason for the users to pay for support or hosting, since they would be able to host for themselves and would not need support.

This mindset is infecting other kinds of applications. There are now people who want to turn what was previously self-sufficient stand-alone computer programs that don't really need an internet connection to work -- to turn them into web applications and provide them as "services" (aka SaaS) so they can charge recurring subscription fees.

There's almost no incentive for anyone to make a thing that is architecturally simple and easy to self-host.

A worse problem is that many recent programmers mentally reside very high on the abstraction ladder. They don't understand nor care what the computer is doing. They mostly think in terms of frameworks and libraries and programs and services.

This is why most web application are designed so that different programs handle different parts of the problem and then get glued together by some scripts and configuration files.

For example, one program acts as an http server, one program acts as data storage and querying server, one program acts as a message broker (whatever that means), one program acts as a memory cache. And then you have code written in a scripting language that handles responding to incoming requests. Then you write a bunch of configuration files for each of these programs so they can sort of know about each other and communicate with each other. Tell the http server where the scripting files and static files are located on the file system. Tell the scripts about the login information for the database server. etc.

This is actually a simplified version. In practice, many companies internal development process is so much more complicated. You can't just write code and then run it to see the effect of your changes. You have to run several programs (sometimes with cryptic or complicated command line arguments), edit some config files, and wait a few minutes for everything to startup before you can even do anything. After you make changes, you might sometimes have to restart several of these programs and wait for some time again before you can see the effect of your changes. It's crazy.

The best they can do to simplify the handling of this architectural mess is to "containerize" it. Which is very similar to having a virtual machine with many programs running inside it. It does work to some extent, but you must not underestimate how much developers love complexity. Once programmers get comfortable with containers, they start to design web applications that require multiple containers to be running at the same time. Of course this requires editing some config files for all the containers to know about each other. They call it "microservices". They consider it an advancement in software development. It's insane.

Collapsing the layers

While containers may solve /some/ of the problems of complicated infrastructure, they are ultimately just another layer on top of the stack. They suffer the same problem as everything else: they have a tendency to break for stupid reasons, and when they do, the problem is not tractable; at least not without extensive specific knowledge about the particular system being containerized.

The proper solution is to collapse all the layers and combine all the different services into the same program.

Serving http and managing data storage and querying don't need to be handled by different programs. They can be combined into the same program.

This requires a shift in perspective.

Instead of using scripting languages, you should use a language that has a compiler that produces executable binary files.

Instead of requiring MySQL or PostgreSQL to pre-exist on the system, you can use a library (such as SQLite) that can be embedded into the final program.

When everything is just code and you have a compiler, it becomes trivial to produce a self-contained program that is easy to ship.

You get the added benefit of not being run in a slow interpreter that struggles to serve concurrent requests. So you don't need to run 40 instances of the program and load-balance them. You just run one instance of the program, and if it encounters higher load, it just uses more system resources to deal with the load.

So much complexity can just completely disappear without loss of capability.

This is just low hanging fruit. And it's doable today; right now.

With more work and effort, it should be possible to achieve 10x performance on a single machine, and enable very easy horizontal scaling without resorting to a "cloud service"; just a few dedicated machines.