server.garden: the world's first Personal Datacenter TechnologyBy Forest Johnson On
The Internet means so much to almost everyone. We've come a long way since 1994, when anchors on The Today Show asked the set crew "what is internet anyway?" during a live broadcast.
These days, the internet has been imbued into almost everything, but it's structure has also changed a lot since the early days. Originally, the internet was built for full participation, that is, users on the network could publish content and run servers just as easily as they could download content from others.
Since then, internet businesses have poured billions of dollars into making it easier and easier for users to access their products. Using things like email and social media is so easy these days, primarily because of this monumental effort. These businesses invested so much money into making their products accessible because they knew that their product had to be easy to use in order to win over the majority of users, and once most folks were logged in to their systems, they would be able to leverage the data provided by you and I to gain power, influence and capital.
But in 2020, if you want that full participation on the network, for example, to own and manage your own email, rather than giving all of it to Google, it's still hard. Like, 1994 hard. No one has worked to make full participation easier for the average user, it's not as profitable.
If you google search "how to run your own email server" the top hit is a 20+ page arstechnica how-to guide from 2014 containing enough technical jargon, code, and obscure acronyms to behead the average user. The text of the article says, "This is hard, but you can do it, and here's how". But, I would argue that the subtext of the article is really saying, "This is too hard for you. Just give up, don't even try."
I want to challenge that narrative. I don't want to see the internet become rent or lease only for the average user, with a time and money barrier to entry for real ownership ranging from thousands to millions of dollars. Not just because of some abstract freedom or privacy ideology, but because I'm taking it personally. I believe in the internet, I grew up with it — in many ways, it raised me. I think it's incredibly important to the present and future.
What's a Personal Datacenter?
In the 70s and 80s, the "Personal Computer" was a new idea. Most people thought it was ridiculous, preposterous, or scary, and for good reason:
Early PCs were extremely difficult to use, requiring technical skills and a lot of patience.
Before the 1970s, computers took a team of specialists to operate, often composed of multiple pieces of expensive equipment that took up an entire room.
50 years later in 2020, PCs and smartphones have become ubiquitous, leaving a massive impact on life and culture. However, the expensive computing equipment that fills an entire room and requires a team of specialists to operate has not gone away. It has simply become faster and faster, more and more centralized, and evolved into the modern datacenter or "Cloud."
Just like early computers, datacenters are cryptic and scary to almost everyone. The idea that an average computer user could have a Personal Datacenter seems preposterous in 2020. But I would like to point out, the idea that the average person could use a computer at all seemed preposterous at one point too! When computers were almost exclusively terminal-based and required programming skill to use, of course practically no one could use them!
The entire field of computer usability had to be founded and developed by generations of brilliant designers and companies like Apple and Microsoft.
I believe that with enough imagination, smart interaction design and engineering, a Personal Datacenter could be just as approachable as a Personal Computer.
A Personal Datacenter would be to racks of Dell PowerEdge servers
what an Apple Macintosh was to an IBM Mainframe
- smaller & cheaper by orders of magnitude
- advanced technical skills not required to install or use
- works anywhere, no gotchas or special requirements
However, just as the original Mac was a capable computer, server.garden needs to be a capable datacenter. What does that mean?
What's the difference between a computer and a datacenter?
A datacenter is not just a rack of server computers. Datacenters have many important and distinguishing features:
- They accept incoming connection requests from anyone in the world
- In order to host web services, folks have to be able to connect to your server computer(s). The network has to be set up correctly for this to be possible.
- They are on all the time
- Datacenters are never fully turned off, even for maintenance or upgrades. Uptimes for datacenters range from 99.9% of the time to 99.999% of the time.
- They have at least two of everything
- If one component fails or one region has a power outage, the system as a whole has to keep working.
- They are fully automated
- Changing something in a datacenter should not require a human to physically flip a switch or log into a machine and issue commands. Things are generally managed by declarative code, configuration files, or a management user interface.
- Treated like livestock, not like pets
- Datacenter operators do not have personal relationships with thier servers, giving them cute names and special one-on-one attention. Instead, servers are given ID numbers and they are all subjected to the same automated process.
So, in order to succeed as a datacenter technology, server.garden has to be really good at these things.
What does it take to make a Personal Datacenter?
(I detailed a rough sketch of this in my Pragmatic Path Towards Non-technical Users Owning Their Own Data post from 2016)
1. At least two server computers that will be left on all the time
Luckily, these days, computers are really really cheap. You can get a Raspberry Pi computer with a hard drive and protective case for about $100, and it will only use about $5 worth of electricity per year. The new Raspberry Pi 4 with its upgraded CPU and integrated gigabit Disk/Network IO capacity is perfect as a low cost server platform.
2. At least one public IP address that can accept incoming connections from anywhere in the world
This is the tricky part, because hosting a server does not "just work" on most networks, especially home networks.
- Most users do not know how to set up their router for self-hosting.
- Many internet users don't own their local network hardware, so they couldn't set up self-hosting even if they knew how.
- For example, college students using dorm-wide WiFi or folks using a cellular WiFi hotspot in a rural area.
- Most residential Internet Service Providers (ISPs) do not explicitly allow or support self-hosting.
- Hosting an email server requires arcane network configurations that are only available from commercial ISPs and cloud service providers.
- If you called Comcast and asked them to set up a
reverse DNS PTR recordfor you, they wouldn't even know what you are asking for. If they did, they would probably say you have to upgrade to business class service for that.
- If you called Comcast and asked them to set up a
server.garden solves this by using what folks in the internet industry call "Hybrid Cloud Architechture", meaning that the system is built from both servers that you own & operate, and cloud servers that you pay someone else to operate.
Hybrid Cloud Architechture works well in this case because we get to chose where we draw the line between what we own and what the cloud provider owns. We let the cloud provider deal with all of the internet networking setup that is difficult for the average user and residential ISP, while retaining control of our datacenter and the internet traffic flowing in and out of it.
|Owned by Personal Datacenter User||Owned by Cloud Service Provider|
|Server Computers||Public IP Address|
|Transport Layer Security (TLS) Encryption Keys||Domain Name & DNS Records|
|TCP Packets (Plaintext Traffic)||TLS Packets (Encrypted Traffic)|
This is probably the best possible solution for many users, because:
- It will always work, no matter where you are, as long as you have internet access.
- It's really easy to set up.
- It doesn't significantly degrade the security or flexibility of your system.
Typically a cloud provider can offer this type of service for as little as $5 a month. As a matter of fact, for some use-cases, the amount of traffic would be so small that this could be provided as a free service.
Finally, if the user owns their router and they are technical enough to configure it on their own, this step could be skipped, or the user could start out with a cloud-based public IP and later migrate to using the IP addresses of the locations where they host servers.
3. A Familiar, Joyful User Experience
Last but certainly not least. The user experience is the heart and soul, the "special sauce" that makes
server.garden unique. Not just a new thing, but a new KIND of thing. The user will never be required to use a terminal, edit a text file, or manually manage cloud resources throughout the entire lifecycle of the Personal Datacenter.
The user's experience will start with video demos and tutorials that show other people performing the setup process. The setup process itself centers around a wizard built into a cross platform desktop application. The wizard will prompt the user to create a plan for how their datacenter will be put together. For example,
- How many nodes ?
- Raspberry Pi? ODROID? Old Laptop?
- How many hosting locations?
- For example, 2 locations,
- For example, 2 locations,
- What domain name to use?
- With which domain name registrar?
- Which cloud providers to use, if any.
- Object Storage
- Virtual Private Server
They will then be prompted to enter WiFi credentials for each Location, as well as API Token credentials for each Cloud Provider, accompanied by videos demonstrating how to sign up for an account with that provider and obtain an API token.
Finally, after all of the requirements for the user's datacenter plan are met, the wizard will prompt the user to insert the installation disk for each of the nodes in the datacenter. In the case of a Raspberry Pi, this would be the micro-SD card that the Pi boots from. The wizard will write a linux distribution to the installation disk, followed by the bootstrap application, configuration, and the credentials that the user entered earlier.
This way, as soon as the user inserts the disk and powers on the system, it will instantly configure itself and join the datacenter.
Once all the disks have been written, the wizard will prompt the user to insert the disks and turn on the nodes. As soon as a node connects, it will show up in the wizard within seconds. The setup application on each node will run a provisioning process to set up all of the cloud resources per the user's specification. This provisioning process will include things like creating a cloud instance to get an accessible public IP address, writing DNS records that point to the address, setting up a reverse tunnel to forward connections from the cloud instance to the nodes, and acquiring a valid TLS Key pair from Lets Encrypt.
From there, the nodes should already be reachable over HTTPS from anywhere in the world. The rest of setup will be completed via a management portal web application that the
server.garden instance hosts.
What can you do with a Personal Datacenter?
Once the datacenter is set up, the user can chose from a smorgasbord of applications to deploy onto it. The experience would be similar to an App Store. You can browse different kinds of apps, see comments and reviews, and click install. You can also monitor which apps you have installed, how much resources each app is consuming, how much it is being used, etc.
There are tons of great server applications and application stacks that already exist, and chances are most of them could be offered from such an app store relatively easily.
Here are some examples:
- The mailcow open email server solution, including:
- koushin, a simple and extensible webmail.
- neutron + ProtonMail/WebClient, ProtonMail's WebMail User Interface
- go-guerrilla The SMTP Daemon used by GuerillaMail.com
- NextCloud Groupware & Email
Cloud Storage & Collaboration Tools:
- draw.io Create rich diagrams
- Etherpad Collaborate on text documents in real time
- NextCloud Files Collaborate and share files
- owncloud Collaborate and share files
Real-time chat & Social
- Mattermost Rich real time chat
- Matrix / riot.im Rich real time chat
- Jitsi Meet Video Conferencing
- NextCloud Talk Video Conferencing
- Ghost The blogging platform that is serving this page
- WordPress The world's most popular CMS (Content Management System)
- PeerTube Video Publishing with P2P infrastructure
- webtorrent Video Publishing with P2P infrastructure
- Open Streaming Platform Real-time Video Streaming like Twitch.tv
- SequentialRead pico-publish easily host a single file or zipped directory on the internet
- openstreetmap Powerful mapping software
- framadate find an event date that works for everyone, like doodle
- framaforms Create forms, questionaires, polls, etc
- Go Git Service HTTP Git Server and collaboration platform like GitHub
- Gitea HTTP Git Server and collaboration platform like GitHub
- Gitlab HTTP Git Server and collaboration platform with continuous integration capabilities
- Zuul A continuous integration server
- Drone A continuous integration server
- Harbor Container Image Registry
- Portus Container Image Registry
Security & Privacy
- SequentialRead Password Manager simple end-to-end encrypted password manager
- pi-hole Network-level Ad Blocker
- bitcore Bitcoin Blockchain Server & API
- Bitcoin Server (bs) Bitcoin Full Node & Query Server
- ncdns Namecoin to DNS bridge daemon
- ipfs peer-to-peer hypermedia protocol server
- bitmessage Bitmessage Garlic Routing Daemon
Datacenter Operation Tools
- telegraf Plugin-driven Metrics Collection Agent
- influxdb High performance time-series metrics database
- Grafana Web front-end for graphing metrics
- flapjack Alert Management & SMS Alerts
- Grafana Loki Log Aggregation System
- Open Web Analytics
And I'm sure there are tons more I missed! Leave a comment with software you would like to run!