It's been a while, I don't have much to say, other than I recently updated some of my older projects:

💥PoW! Bot Deterrent

• About a year ago, I had hastily modified this to require all the static assets to be served from the same domain, in order to satisfy a Content Security Policy (CSP).
• Unfortunately, this broke 3rd party embedding use cases including my blog comments system.
• I also forgot to fully update the docs and example project, so it became a lot less usable. Sorry about that.
• I started getting bug reports about GrapheneOS users not being able to pass the bot deterrent.

As of now all those issues should be fixed.

I changed it so now there are two ways to use it, either with everything on the same domain, data-pow-bot-deterrent-static-assets-path, or with the static assets downloaded from the separate bot-deterrent domain, data-pow-bot-deterrent-static-assets-cross-origin-url.

I also updated all of my apps that use the 💥PoW! Bot Deterrent:

• sequentialread-comments (blog comments system)
• 📤📚 picopublish (minimalist upload-and-share tool)
• pow-bot-deterrent-rp (AI scraper bot blocker, similar to Anubis. Specifically intended for source code forges like Gitea, Forgejo, etc.)

I publish multi-architechture docker images for each of these to docker hub, so if you want to use the binaries that way, you can:

https://hub.docker.com/r/sequentialread/pow-bot-deterrent

https://hub.docker.com/r/sequentialread/comments

https://hub.docker.com/r/sequentialread/picopublish

https://hub.docker.com/r/sequentialread/pow-bot-deterrent-rp/tags

In fall 2024, I traveled to Japan and China to see my brother and his wife for their wedding. They met in Shanghai, where everyone seems to be impressed by his fluency in spoken and written Chinese, and he was impressed by the tree-climbing skills of a businesswoman he met 💑

I've long been curious about how the internet censorship in China works, what it feels like to interact with it, and was excited about the opportunity to experience it firsthand.

My brother has been working through it for a long time, almost 10 years now I think. He started off using the StreisandEffect/streisand project as his main solution. It included some shell scripts and Ansible playbooks to install various VPN servers on an AWS EC2 instance.

For client software back then, he said he used

shadowrocket on iOS and shadowsocksX-NG on macOS

I did a lot of research on how people get around the censorship, the history of anti-censorship technology in china, what matters most when using various VPN/proxy solutions, etc, and came up with a plan.

1. I would use V2Ray
   • Despite what people call it, v2ray is not really a VPN. Technically it's a hightly sophisticated "forward tunnel" system
   • This allows it to "blend into the background" more easily, and mask its presence with clever sleight of hand.
     • sleight of packets? sleight of protocols?
   • As far as I can tell, it's the most mature censorship-resistance solution avaliable
   • Client software is avaliable for iOS, Android, Mac, Windows, and Linux
2. I would configure the clients and servers specifically to leverage the sneaky capabilities that v2ray's various softwares offer.
   • Plausible deniability that "This is just normal internet traffic 😉"
   • ℹ️ NOTE: Configuring the client is of utmost importance, that's a large part of why v2ray can work as well as it does.
   • More information on that later.
3. I would make several VPN servers ahead of time.
   • Idea was that in case the first one got blocked, I would have a fighting chance to figure out what went wrong and try again with the next one.

Flawless Victory

I won't bury the lede. I had zero problems browsing the web and accessing my cloud accounts while I was in China. Gmail and google maps? No problem. Tank Man? No problem. Airplane tickets, Japanese hotel bookings, bank info, the list goes on.

I did witness some of my co-travelers getting blocked at various points, but even after that happened, onboarding them to my system solved these issues and did not precipitate any issues for them or for my services. I believe it was even used to stream Netflix at one point 🤣

I did have some trouble installing some Chinese apps, but I'm not sure if that was caused by the v2ray or not. Alipay worked fine, but Dianping (Similar to Yelp) didn't seem to like my phone.

Its true that this was a short duration trip ( about a week ), and I have a few unique advantages that made this easier for me to pull off, but I do believe it's not a complete fluke, and it's worth writing about.

Specifically, there were some unclear and confusing aspects of v2ray's documentation and configuration. I felt like it would be worthwhile to publish my solutions, I think they are cleaner and easier to understand than others I've seen around the web.

Server Configuration w/ Caddy

This is where I think I have novel stuff to contribute, on top of other guides that already exist.

I found the v2ray server configuration slightly confusing, and had some issues getting it to work at first, especially the HTTP path. So I decided to just leave it as default as possible and handle most of the HTTP stuff in Caddy, which I'm more familiar with.

First off, the

V2Ray configuration using Docker:

docker-compose.yml

services:
  v2ray:
    image: teddysun/v2ray:5.19.0
    ports:
      - 1310:1310
    container_name: v2ray
    environment:
      - TZ=Asia/Shanghai
    restart: always
    command: /usr/bin/v2ray run -config /v2ray/config.json
    volumes:
      - ./v2ray:/v2ray

v2ray/config.json

{
  "log": {
    "access": "/var/log/v2ray/access.log",
    "loglevel": "info"
  },
  "inbounds": [
    {
      "listen": "0.0.0.0",
      "port": 1310,
      "protocol": "vmess",
      "settings": {
        "clients": [
          {
            "id": "82d984d3-1cda-4bc5-bd49-f167e73c2719",
            "alterId": 0,
            "security": "auto"
          }
        ]
      },
      "streamSettings": {
        "network": "ws",
        "wsSettings": {
          "path": "/"
        }
      },
      "mux": {
        "enabled": true
      }
    }
  ],
  "outbounds": [
    {
      "protocol": "freedom",
      "tag": "freedom"
    }
  ],
  "dns": {
    "servers": [
      "8.8.8.8",
      "8.8.4.4"
    ]
  }
}

The inbounds[0].settings.clients[0].id value acts as the v2ray server password, which has to be entered into the client for it to be able to connect.

It must be a GUID, which can be generated with the uuidgen command:

forest@debian:~$ sudo apt-get install uuid-runtime
...
forest@debian:~$ uuidgen
82d984d3-1cda-4bc5-bd49-f167e73c2719
forest@debian:~$ 

The inbounds[0].streamSettings.network = "ws" value means WebSocket. After an HTTP connection is made, it will upgrade to a WebSocket, and start the VMESS protocol from there.

This may not be the ideal configuration, but it worked for me, and seemed to be the most immediately avaliable and reliable.

Next, we have the

Caddy configuration (Not using Docker):

I could have ran Caddy in Docker as well, but the server I was installing it on already had Caddy set up running on the host, with its config file at:

/etc/caddy/Caddyfile

mail.totally-legit-domain.com {
  root * /usr/share/caddy

  route /ws_io34857 {
    rewrite /ws_io34857* /
    reverse_proxy http://localhost:1310

  }

  route /login {
    respond 401 {
      body "401 Unauthorized"
      close
    }
  }

  route {
    file_server
  }
}

What all is going on in this Caddyfile?

Lets go thru it...

1. A separate subdomain is used for v2ray.
   • This encapsulates the v2ray config and allows it to coexist with other things on the same server.
2. The domain name looks like it might be a mailserver. I also used other common, legit sounding subdomains like social.xyz.com and webmail.xyz.com on other servers.
   • Domain names are 100% public. The domain name is transmitted in plaintext multiple times during each web request, once by the DNS protocol and once in the SNI (server name indication) field of the TLS ClientHello packet.
3. There are three handlers:
   • Requests to /ws_io34857 will have the /ws_io34857 path stripped and be reverse-proxied to port 1310, where our v2ray server is listening.
     • A non-guessable path is used to make it harder for a prober to analyse what this server is serving.
     • The path part of an HTTPS request is encrypted with TLS, so it won't leak like the domain name will.
   • Requests to /login will always return 401
   • All other requests will serve static files from /usr/share/caddy

Inside /usr/share/caddy, I put a slightly modified version of the login page from the alps webmail client as index.html:

So from the outside, this looks like a normal login page which matches the domain name. If you try to login, no matter what username and password you chose, you will always get a basic 401 Unauthorized response, which would look typical from any sort of basic web application written in Go, or really any web application that returns plain text responses for errors.

Since the v2ray requests to this domain are using TLS, the content of the requests is encrypted. And if I understand correctly, the VMESS protocol that v2ray uses specifically crafts the sizes and timings of the encrypted bytes flowing inside requests and responses to make the connection look like normal web traffic. (To some extent, it probably helps that real web traffic is being tunneled inside it!)

Client Configuration

I used v2rayNG as my primary client on Android (LineageOS), and QV2ray on Linux desktop.

I downloaded the v2rayNG APK directly from the main project's GitHub releases page:

The v2rayNG configuration that worked looked like this:

These client configurations can be shared between apps via text or QR code, however, not all apps / versions are compatible. Worst case, you can always enter it in manually.

This is not the most important part of the configuration though, this just describes how to contact the v2ray server WHEN the client has determined that it needs to.

But the magical part that allows this project to work is all about that WHEN. The client needs a ton of smart rules and lists to use to determine when it should try to connect via the tunnel vs when it should try to connect directly.

If I remember correctly, v2rayNG comes with all of this setup already, which was a major reason why I chose to use it. I had to patch together and set it up myself on the desktop app, which took me some time, especially since most of the documentation and community is in Chinese language.

There are a bunch of these rules already setup, but I also created my own based on a list of domains I'm always surfing: various community-hosted chats and resources. I'm not sure, but I believe this may have helped my v2ray servers to sneak by 100% undetected: They shared IP addresses with other active & recognizable services that were not blocked.

It might have also helped that lot of these IP addresses aren't on mainstrem American cloud providers: Its a mix of small scale colo and residential ISP addresses.

I made sure to add the same list and behavior on the desktop app as well.

At any rate, there's more to life than using the internet. So have some pictures of delicious foods I tried while I was in Shanghai.

Tools List

• Electronics Screwdriver Set
• Multimeter
• Soldering Iron
• Solder
• Solder Flux
• Wire

My friend who was a Barista gave us this nice Bonavita electric Kettle when he moved out:

And it worked for years and years, even while we had multiple people living in the house and using the kettle every day.

However, it started getting harder and harder to power it on; I would have to press the power button down for longer and longer to get it to start heating.

Eventually it simply wouldn't turn on anymore at all.

Personally, I've never tried to repair electronics like that. But after my initial experience with DC electronics (making a Lead Acid + solar battery for my Odroid XU4),

and also seeing multiple people pull off soldering projects at Layer Zero, I was inspired to try my hand.

Unfortunately for me, I was immediately confronted with anti-repair practices from Bonavita:

Why the hell do they have tri-lobe screws on the base of this thing? What the actual fuck???

This is especially bizzare once I got the bottom off (I used a flathead small enough to jam into the trilobe, lol).

Then I discovered phillips head screws holding the PCBs in place:

Observant readers may have already spotted something very suspicious on the bottom right: What's that little electrolytic capacitor shroud doing there!?!?!?

Turning over the Power Electronics board, I found where it came from:

Obviously I found the problem, so got to work replacing this capacitor. (Foreshadowing, lol).

Comments I got on the Cyberia Matrix chat about the project:

ivy (she/her)

pretty sure you can just pour some gatorade on that and tape it up

👍 3 😂 1

forest

its what plants crave

😂 3

carbide_flux

Naughty manufacturer putting an electrolytic next to a heating element. You should be fine to replace it. I suggest replacing the other one too.

Later that week, I traveled to Layer Zero where there's a great soldering station and a large pile of tech waste to pick from.

There, I found a capacitor with the exact same matching specs, 16v 470 microfarad, on an old ethernet switch I found in the tech waste bin.

forest

From Ethernet switch

I had de-soldered the replacement capicitor from its home on the ancient ethernet switch's board. Now I had to figure out how to attach it to the Power Electronics board of the kettle.

But first, I had to de-solder the exploded capacitor from the power electronics board.

De-soldering and pulling the broken capacitor off of the board

Music: Boards of Canada -- poppy seed / everything you do is a balloon

1 minute 0 seconds

Here are the two capacitors next to each-other, the exploded one and the replacement:

I tried a couple different ways to solder the new capacitor onto the kettle's Power Electronics board.

At first I thought i'd find some thick wire and solder short segments of it onto the holes on the circuitboard, then solder the wires onto the contacts of the capacitor.

However, the only thick wire I could find was braided from a gazzillion smaller strands, and trying to solder it to the PCB was extremely frustrating and taxing.

In the end, I gave up. I couldn't seem to get a clean solder joint with that wire. So next, I thought I'd try a breadboard jumper wire, similar to the one pictured:

I was able to get a nice secure joint when soldering the ends of one of those onto the board, but wasn't sure how to connect the capacitor.

At first, I tried stripping the middle wire segment. It was hard to strip, the wire was multiple tiny copper strands encased in colorful rubber. It was really hard to remove the rubber without cutting some of the strands.

To make it worse, I wasn't sure if I would have enough space to solder the capacitor onto those copper wires. The plastic end-housings were taking up a lot of real-estate on the tiny board.

So in the end, I bit the bullet and tried to clip off the plastic end-housings. To my delight, they split off cleanly, leaving just the thick contact wires and little sprigs of copper behind.

After that, I was easily able to bend them into shape, solidify the solder joint to the underlying circuitboard, and finally solder the capacitor onto them.

ℹ️NOTE: I wasn't sure how to tell which way to solder the capacitor -- I'm not sure if they have a "direction" or not like Diodes do. I guessed that they do. But becuase there was another capacitor on the same board, it gave me a clue. The capacitors all seem to have this directional arrow printed on the outside of the housing -- and the circuitboard underneath showed the outline of the capcitor with one side shaded and the other side clear. I assumed via intuition that the shaded side goes with the directional arrow on the outside of the capacitor, because that's what I observed on the one that had not exploded.

Even though this circuitboard has very large features and should be a lot easier to repair compared to a smaller-scale logic board, I still struggled with the solder joints and had to re-do them the first time.

I didn't use any flux at first and quickly regretted it.

Excited, I immediately put the kettle back together, confident it would turn on.

However...

It still seemed very sketchy. When I pressed the power button, nothing happened. I had to press it multiple times and hold it down for multiple seconds before the kettle finally sprung to life and started heating.

😰

                                                                                              
                                              ░░                                              
                                                                          ░░                  
              ░░                              ░░                          ░░                  
              ░░        ░░                    ░░                        ░░░░                  
              ░░    ░░  ░░                    ░░      ░░                ░░░░            ░░    
      ░░      ░░    ░░  ░░      ░░            ▒▒      ░░    ░░            ░░            ░░    
░░  ░░░░      ░░    ░░  ░░  ░░  ░░            ▓▓      ░░    ░░░░          ░░            ░░    
░░░░░░░░      ░░    ░░  ░░░░▒▒░░░░        ░░  ▓▓░░    ░░    ░░      ░░  ░░    ░░░░      ░░  ░░
▒▒░░  ░░      ░░  ░░░░  ▒▒░░▓▓░░░░░░      ▒▒  ▓▓▒▒    ░░  ░░░░  ░░  ░░  ░░  ░░░░░░    ░░░░  ░░
▒▒░░▒▒▒▒░░    ▒▒░░░░░░  ▒▒░░▓▓░░░░░░      ▒▒░░██▒▒░░░░░░  ░░▒▒░░░░░░▒▒  ▒▒░░░░░░░░    ░░░░  ▒▒
▒▒░░▒▒▒▒░░  ░░▒▒░░░░░░░░▒▒▒▒▒▒▒▒▒▒░░░░  ░░▒▒░░▓▓▒▒░░░░▒▒  ▒▒▒▒▓▓░░▓▓▒▒▒▒  ░░▒▒▒▒▒▒░░▒▒▒▒░░░░▒▒
▒▒▒▒▓▓▒▒░░░░░░▒▒▒▒░░▒▒░░▒▒▒▒▒▒▒▒▒▒░░▒▒░░▒▒▓▓░░▓▓▒▒▒▒░░▒▒░░▒▒▒▒██  ▓▓▒▒░░▒▒  ▒▒▓▓▒▒░░▒▒▒▒  ▒▒██
▒▒▒▒▓▓▒▒▒▒░░▒▒▒▒▒▒▒▒▓▓░░▒▒▒▒▒▒▓▓▒▒░░▓▓░░▒▒▒▒░░▓▓▒▒▒▒▒▒▓▓▒▒▓▓░░██░░▓▓▒▒▒▒▓▓░░▒▒▓▓▒▒▒▒▒▒▓▓▒▒▒▒██
▒▒▓▓▓▓▓▓▒▒▒▒▒▒▒▒░░▓▓▓▓▒▒▒▒▒▒▓▓██░░▒▒▒▒▒▒▓▓░░▒▒▒▒▒▒▒▒▓▓██░░██▒▒▓▓░░░░▓▓▒▒▓▓░░▒▒▓▓▓▓▒▒▒▒▓▓▒▒▒▒▓▓
▒▒▓▓▒▒▒▒▓▓██░░▓▓░░▒▒▓▓▒▒▒▒▓▓▓▓██░░▒▒▒▒▓▓▒▒░░▒▒░░▒▒░░▓▓▓▓░░▒▒░░▒▒▒▒░░▓▓▒▒▓▓  ░░▓▓▓▓▓▓▒▒▒▒▒▒░░░░
▒▒▓▓░░▒▒▒▒▒▒░░░░░░▒▒▒▒▒▒▒▒▒▒▒▒▓▓░░░░░░▓▓░░░░▒▒░░▒▒░░▓▓▒▒░░░░░░▒▒▒▒░░▓▓▒▒▓▓  ▒▒░░▒▒▒▒▒▒▒▒▓▓▒▒  
░░▒▒  ▒▒░░▒▒  ░░▒▒░░  ░░▒▒▓▓▒▒▓▓░░  ░░▓▓▒▒  ░░░░▓▓  ▒▒░░░░░░  ██▒▒  ▒▒░░▓▓░░▒▒░░▒▒░░▒▒▒▒▓▓▒▒  
░░░░  ▒▒▒▒░░░░  ▒▒  ▒▒  ▒▒░░░░▓▓  ░░░░▒▒░░  ░░░░▒▒  ▒▒░░    ░░░░▒▒  ▒▒░░▓▓▒▒▒▒  ░░  ▒▒░░▒▒▒▒  
      ░░░░░░░░░░    ░░░░▒▒▒▒  ░░      ▒▒░░  ░░░░░░  ▒▒░░      ░░▒▒░░░░░░░░▒▒░░      ▒▒░░░░▒▒  
░░    ░░  ░░        ░░  ▒▒░░  ░░            ░░░░░░  ▒▒░░        ░░  ░░░░░░░░░░      ░░░░░░░░  
      ░░░░  ░░  ░░      ░░    ░░        ░░    ░░    ▒▒░░        ░░  ░░  ░░░░        ▒▒░░░░░░  
      ░░░░                              ░░          ▒▒░░                                  ░░  
      ░░░░                              ░░          ░░                      ░░                
      ░░░░                              ░░      ░░                                            
                                                                             

After a few days off from the project, I came back with a fresh look.

Before I opened it, my intuition had been that the power button was dying.

forest

Does anyone have tri lobe screw driver(s)?
We have this nice electric kettle thingy
It stopped working, my suspicion is that the power button wore out

Upon re-inspection, I was able to tell that the voltage going to the control board seemed to be fine -- I saw a steady 5v DC on the header pins labeled VCC / GND

When I'd talked to my partner about it, they'd mentioned that they'd observed a difference betwen simply pressing the power button and dragging one's finger across it deliberately perpendicular to the switch's direction of travel -- and that sometimes dragging caused it to work better.

I had never noticed this, and it gave me a suspicion that perhaps my original idea, the power button, was right all along.

So I opened the stinker up again and started looking at the digital control board and all the switches on it.

ℹ️NOTE: I had a humbling experience with this... When I was a kid, my father gave me a TENMA 72-7720 multimeter as a birthday gift.

This thing definitely works,

however, it's slow to update its display, and requires manual adjustment to get to the right setting for probing whatever circuit you might be faced with.

At Layer Zero, someone donated a modern Fluke 117 multimeter. It makes a world of difference. It automatically detects the range of the measurement instead of requiring the user to select it, and it updates its display instantaneously. It also features a log-scale display of the measurement in the form of a segmented bar display-- this ended up being a lifesaver for this project.

I recognized the Fluke brand name as a quality brand from listening to the Marco Reps youtube channel, and this thing definitely did not dissapoint.

Using the Fluke at Layer Zero, connecting the probes across the power switch pins, I saw something very interesting which I had missed with the Tenma.

ℹ️ INFO: In part, this was due to my ignorance combined with the un-repair-ability of the Bonavita kettle. The kettle's digital control circuitboard's bottom side had been coated with some sort of insulating material -- I had to scrape and scrape and scrape at the solder joints before I could get any reading at all. To be fair, there's probably a good reason for this coating.. But it's annoying when I'm trying to fix the thing.

The view from the Tenma was just confusing -- I wasn't physically able to hold the probes steady in place for long enough to see a change within whatever narrow sensing range I had selected. Even after I started agressively scraping the coating off.

But once I had it under the Fluke at Layer Zero, I started seeing things. The real-time display (less than 10ms update latency) really illuminated the problem. Basically, the power button switch was toasted.

Instead of showing "infinity" ohms when it was open, it showed about 4 mega-ohms. When I scraped the coating off of other switches on the control board and tested them, they showed infinity ohms when open.

When the power switch was "closed" (power button pressed), the measurement would vary wildly between mega ohms, kilo ohms, etc, as I wiggled my finger around on the power button. The log scale display bar on the bottom of the Fluke's LCD made this obvious. This explains what my partner had experienced w/ directionality of the way they pressed the button influencing success rate!

So, I decided to replace the power button. Looking around at all the other buttons on the device, I decided I would sacrifice the Centigrade / Farenheit toggle button, and use it to replace the power button.

This went fairly easily, I had to melt and detach one 2-pin side of the 4-pin button at a time, but besides that, I didn't have any issues.

However.....

I put the thing back together again, and excitedly pushed the power button. To my delight, it instantly sprang to life on the first try, unlike before. I thought it was FIXED!!!!

....

I was wrong. After it turned on, instead of starting to heat, it clicked off and displayed 0F on the LCD.

....

Further investigation revealed that the switch I had removed was in fact an important part of the circuit for the temperature sensing feature -- It was the only switch on the board for whoom the horizontal sides of the switch bridged a PCB trace.

Pin Layout of Every Switch on the Board Except the Celcius / Fahrenhiet switch:

Pin Layout of the Celcius / Fahrenhiet switch:

It turns out that because of the type of switch the manufacturer had used, according to my housemate who's a technician / engineer,

Single pole double throw switch

By removing the Celcius / Fahrenhiet switch I had broken the device. The switch was conducting between those two wires by default until it was pressed, but I had broken that connection by removing it.

So, finally, I got a couple small segements of wire and soldered them long-ways across the place where the switch had originally been:

After this, the kettle was good as new! Here's to four more years of morning coffee.

I ain't buyin' shit!!!

Through a cruel coincidence, my mother has an iPhone, but her computer runs Windows.

She has struggled for years with this situation at every stage. Very uncool moves from both Apple and Microsoft have conspired to keep her phone and computer apart, and even to this day, she still couldn't figure out how to open the photos she takes with her phone...

There are tons of excellent peices of software out there for handling images -- Krita, ImageMagick, and others. However, as far as I can tell, none of them natively support HEIC images. Frankly, it's a very bad vibes situation.

So, after coming up emptyhanded, with nothing I could recommend to her that would directly open a HEIC file, I decided to take matters into my own hands and just make one. Little did I know, I was about to learn more than I ever really wanted to about why none of these exist.

EDIT: It turns out that the Windows Photos app actually was converting the HEIC photos to jpeg, she just wasn't able to find where the photos app was putting the jpeg files. So this might not be important for windows users, but I'm keeping this post up in case anyone needs to write, for example, a web application that can read HEIC images.

Code Repository

https://git.sequentialread.com/forest/heic-converter-gui

This application is written in Go, but the actual image conversion is handled by libde265 (C++ code) that was compiled to WebAssembly via Emscripten and runs inside an embedded WebAssembly runtime that was written in Go. (Similar to The Carcinization of Go Programs 🦀)

This is really good because it should work on any CPU architechture and any OS platform, there are no annoying platform-dependent C/C++ compiler issues to deal with. It runs about 10 times slower than the native option, but it works, and it's really easy to use, so I think thats what matters most.

🙇 Big thanks to the person who made that happen! 🙇

Download Links

• Windows 64 bit https://picopublish.sequentialread.com/files/heic-converter-f451bb.exe
• Linux 64 bit https://picopublish.sequentialread.com/files/heic-converter-8dd865

Usage Notes

If you want lossless output you can specify png 1 (the quality number is ignored for PNG).

Only use png if you want to edit the image and then compress it further with jpeg / avif. The png files will be massive.

This tool strips all EXIF data but it should preserve the rotation that is present in the EXIF Orientation attribute.

After over a year I'm trying to get back to posting here... A lot has been going on in my life and I haven't had much time or motivation to work on my own personal stuff.

I have heard of this concept called a "polyglot", a polyglot is a file that's valid in multiple different formats, or source code which compiles/runs in multiple different languages.

But what I'm after this time is not quite that... I want... A chunk of data which displays the same (or similar) string when represented in multiple different encodings (Base64 and ascii). I'm not sure what to call this.

I asked the Cyberia Chat and got a couple different interesting ideas:

forest (he/him)

OK so a polyglot is a program that runs OK as two different languages

What do you call something which functions the same regardless of whether it's encoded or decoded according to some scheme? (Base64 let's say)...??

And maybe "functions" defined quite loosely, like a "vanity" hash or something, it doesn't have to be exact, just legible

Idk if there is a name for this and now I wanna try to make one, LOL!

"Fixed Point" -> "Vanity Point"

SYMYƧ

I think eigenvalue is the closest since you are essentially saying you put a value through an encoding and the value remains unchanged , or only changed within certain limits

I think there's a more general term than eigenvalue though, but I'm blanking on it, but also I'm pretty sure most encodings can be phrased as linear algebra anyway, so 🤷‍♂️

"fixed point", that's the term

forest (he/him)

How bout a 'vanity point' of a given encoding

SYMYƧ

  forest (he/him)

How bout a 'vanity point' of a given encoding

Oh I like that!

Homomorphism

hemant (he/they)

yeah the term you're looking for is homomorphism. if f is homomorphic with respect to g then calling f and g is commutative

forest (he/him)

I'm sorry I would have to go over these math words again

hemant (he/they)

like you can do f first then g , or g first then f. and it would be the same

so if you imagine f is the computation you want to remain the same regardless of whether it's operating on a base64'd input or not, and g is the base64 function, and x is a not-base64'd input, then f(g(x)) = g(f(x))

forest (he/him)

  hemant (he/they)

so if you imagine f is the computation you want to remain the same regardless of whether it's operating on a base64'd input or not, and g is the base64 function, and x is a not-base64'd input, then f(g(x)) = g(f(x))

Oooooohhhhh yes so f = user reads the word 'example', g = base64

hemant (he/they)

yep

forest (he/him)

Makes sense thx

I'm not sure which name I like best,

• A vanity point on the Base64 function
• A value whose legibility is homomorphic over the Base64 function

But regardless, I had already gotten excited and decided I was going to do this, so now I had to figure out how, and where to start.

At first, was completely lost on how to do this. I knew would need to visualize the process a bit, so without knowing what my solution was going to end up being, I decided to simply write a program that displays the process of base64 encoding and decoding down to the individual bits involved.

My code would print out the same data from both the encoding perspective and the decoding perspective: as ascii characters, as decimal, and as binary.

During this process I learned a bit more about Base64, stuff that I already knew, but never really felt so viscerally before:

• An ascii character is 1 byte, 8 bits
• However, only 6 bits are represented by a single character in a Base64 string.
  • 2 ^ 6 = 64
• There is a 3/4 ratio here -- 3 bytes = 4 characters in Base64.
  • This means (in byte-aligned ascii terms) the bit-wise "meaning" of a given Base64 character will change depending on its position
  • Each subsequent character "shifts" the next one by two bits.

Eventually I was able to get my visualization to line up, and I came to a conclusion that seems obvious in retrospect:

The ascii string and Base64 string both represent the exact same binary string, the same bits.

Base64 can represent any bit string... While only some bit strings are valid ascii.

So I needed to try writing out words into a base64 encoded string in different ways, and continually check to make sure that the resulting bits had a valid ascii decoding.

I decided I would:

1. Represent the bits as a string of 1s and 0s, since it was more familiar to me.
2. Check each newly-written 8 bits against a list of valid 8-bit strings for ascii characters.
3. Use a recursive function (depth-first-search) to find solutions
4. Give my search function multiple different ways to write words into the Base64 encoded string (1337 speak!) so it's less likely to get stuck and never find a solution which has a valid ascii encoding.

var leetSp34k = map[string][]string{
	"a": {"4", "@"},
	"b": {"6", "8"},
	"e": {"3"},
	"i": {"1"},
	"j": {"7"},
	"l": {"1"},
	"o": {"0"},
	"q": {"9"},
	"s": {"$", "5"},
	"t": {"+", "7"},
	"u": {"v"},
	"v": {"u"},
	"z": {"2"},
}

I got promising results right away when trying to generate the word Example:

cFExam91
011100000101000100110001011010100110111101100101
cFExam9l
011100000101000100110001011010100110101001
cFExamp
011100000101000100110001011010100110101001001011
cFExampL
011100000101000100110001011010100110101001110101
cFExamp1
011100000101000100110001011010100110101001100101
cFExampl
011100000101101000110111010111
cFo3X
011100000101101000110111010111000000
cFo3XA
011100000101101000110111010111111000
cFo3X4
011100000101101000110111010111111000001100
cFo3X4M
011100000101101000110111010111111000100110

And with a bit of tuning, I was able to churn out much longer strings. Allowing the search function to repeat characters helped it find many more valid solutions and extend the length of strings it could represent:

$ echo 'UjBlOGExamp1elBLOGExamplelBLOGExamplelll' | base64 -d
R0e8a1jjuzPK8a1jjezPK8a1jjezYe

And if we suffix the decoded string with our chosen text, we get at least a basic version of the thing we were originally after: a value that "says the same thing" regardless of whether it's base64 encoded or not:

$ echo 'R0e8a1jjuzPK8a1jjezPK8a1jjezYeBlogExample1BlogExample1BlogExample' | base64 -w 0
UjBlOGExamp1elBLOGExamplelBLOGExamplelllQmxvZ0V4YW1wbGUxQmxvZ0V4YW1wbGUxQmxvZ0V4YW1wbGUK

This "half-and-half" approach is as good as I can do so far, I think... But I would be interested if anyone else has better ideas.

I did also notice some interesting things: some characters seem to work better than others depending on thier position in the string. I noticed t and r to be particularly troublesome: at least with my naive depth-first-search algorithm, I was not able to find a solution for BlogExampleString, but String worked fine.

At any rate, I'm sure there is more to explore, but for now, I'm ready to call it done.

The Code:

git.sequentialread.com/forest/base64-reverse-encoder

Matrix certainly has its problems, which folks are often quick to criticize:

• The protocol design makes moderation harder than it should be
• There's no "admin panel" for a matrix server; it's administered via CLI
• It has bugs and can be hard to maintain
• IRC users hate it because the matrix IRC bridge was made by matrix people not IRC people, so it ham-fistedly tries to fit matrix content into IRC instead of trying to limit a bridged room to content that IRC supports.

But I think Matrix's benefits outweigh its drawbacks, and I am happy to support the cyberia.club matrix server as its needed.

Disk space has been our single most troublesome problem so far.

For a long time, we kicked the can down the road by increasing the disk space on the VM we use to host our matrix server. But obviously, we can't do this forever.

Upon further investigation, we learned that the postgres database was the one using up all the disk space, and specifically, that one table in the database, called state_groups_state, was responsible for that.

Here is an excerpt from the "Origin Story" on the matrix-synapse-diskspace-janitor repository:

The problem at hand:

Matrix-synapse stores a lot of data that it has no way of cleaning up or deleting.

Specifically, there is a table it creates in the database called state_groups_state:

root@matrix:~# sudo -u postgres pg_dump synapse -t state_groups_state --schema-only
--
-- PostgreSQL database dump
--
...

CREATE TABLE public.state_groups_state (
    state_group bigint NOT NULL,
    room_id text NOT NULL,
    type text NOT NULL,
    state_key text NOT NULL,
    event_id text NOT NULL
);

I don't understand what this table is for, however, I can recognize fairly easily that it accounts for the grand majority of the disk space bloat of a matrix-synapse instance:

top 10 tables by disk space used, cyberia.club instance:

So, I think it's safe to say that if we can cut down the size of state_groups_state, then we can solve our disk space issues.

I know that there are other projects dedicated to this, like https://github.com/matrix-org/rust-synapse-compress-state

However, a cursory examination of the data in state_groups_state led me to believe maybe there is an easier and better way.

state_groups_state DOES have a room_id column on it. It's not indexed by room_id, but we can still count the # of rows for each room and rank them:

top 100 rooms by number of state_groups_state rows, cyberia.club instance:

In summary, it looks like

about 90% of the disk space used by matrix-synapse is in state_groups_state, and about 90% of the rows in state_groups_state come from just a handfull of rooms.

So from this information we have hatched a plan:

Just delete those rooms from our homeserver

However, unfortunately the matrix-synapse delete room API does not remove anything from state_groups_state.

This is similar to the way that the matrix-synapse message retention policies also do not remove anything from state_groups_state.

In fact, probably helps explain why state_groups_state gets hundreds of millions of rows and takes up so much disk space: Nothing ever deletes from it!!

We did come up with some shell scripts to handle this, but it was an annoying recurring manual maintenance burden. Due to the complicated, multi-step nature of the cleanup process, I ended up creating an application to handle it instead of continuing to try to script it. Yes, its probably overkill, but it was fun. And who knows, maybe it can be useful to someone else.

Anyways, this is what it looks like:

You log in with your matrix account:

And then you can see what's using disk space, and if there are any big rooms, you can delete them, including deleting the state_groups_state rows:

Long ago, I believed that Single Board Computers (SBCs) like the Raspberry Pi were going to become the ultimate hardware platform for home servers. The idea of taking a mass-produced cellphone System-on-a-Chip (SoC) and attaching an ethernet + SATA port seemed perfect. Everything was lining up for SBC dominance as the Raspberry Pi 3 and 4 finally fixed the IO and power issues that had plagued the earlier models & their compettitors. Gone are the days of sub-par software support for arm CPUs, as pretty much any major software can be installed on ARM these days and if not, it's usually a relatively painless process to run a quick build for your device.

SBCs had overcome cheap $5/month cloud instances in terms of their performance characteristics, with some models boasting 8 cores & quite impressive multi-core CPU benchmarks, especially when you consider their power consumption. The software support was looking excellent. New chips with another massive 4x leap in performance were just around the corner...

And then covid hit. War broke out in Europe, demand for chips soared while supply ran dry. Just In Time Manufacturing experienced its first major system shock, and industry is still recovering years later. Now you can't even buy a new Raspberry Pi if you wanted to, they're all permanently out of stock everywhere and have been for years.

There are still some arm SBCs for sale, including those 4x faster ones I mentioned, like the radxa Rock 5B on ameridroid. But there aren't any more $35 computers to be found; these days SBCs tend to run about $60-$200.

There aren't any more $35 computers to be found. Unless... 👀

Someone on the radxa discord had pointed out to me that maybe single board computers aren't the right choice for cheap homebrew server hardware. At first I was reluctant to give up on my SBC evangelism:

forest

Where I live, 1 watt of electricity costs about $1/year. ($0.12 per kWh)

20 * 24 * 365 * 0.12 * (1/1000) = $21 per year to run a 20w device

so if I was homo economicus, I should be willing to pay about $80 more for a device that uses 2-5 watts instead of 20 watts, assuming im gonna use it for 6+ years

What I didn't know, however, was that while the ARM chips of today are in extremely short supply, there appears to be a GLUT of used x86 hardware, so much so that some of these suckers are being listed for $30 with free shipping on ebay:

Shipping ain't cheap. Something being sold for $30 with free shipping is practically being given away for free from the seller's perspective.

By the way, if you want to replicate my ebay search, I believe it was:

PC Desktops & All-In-Ones between $15.00 and $50.00 with the following options:

• RAM:
  • ☑️ 4GB
  • ☑️ 8GB
• Selling Format:
  • ☑️ Buy It Now
• Condition:
  • ☑️ Used
  • ☑️ Refurbished
  • ☑️ Open Box

Moore's Legacy

I've been saying for a long time that computers are fast enough and we shouldn't need to buy new ones any more. Maybe that just means I'm getting old, but I think there's a lot of truth to it as well.

In the past, I imagined a future where computer hardware just kept getting cheaper and cheaper, better and better. To some extent that has been true, but not for NEW new hardware like the new arm SBCs. They have only ever gotten more expensive.

On the other hand, I had been ignoring the used hardware market; I wrote off the used PC & server options as:

• too physically large
• too loud
• too energy-hungry
• not as compatible

My first ever post on this blog, The "recycle" Computer ♻️ can speak to that a bit. That computer was definitely annoying. The fan and HDD made noise, it didn't support booting from USB, so re-installing the OS required a CD burner.

But these days, five years later, the computers that folks are willing to throw away or sell for $30 on ebay have changed dramatically! Now they have:

• Small form factor instead of Big Ole ATX
• Fanless/passive cooling heatsink designs
• Low-power x86 chips instead of 65 watt Pentiums
• 8GB of RAM instead of 2 or less
• Gigabit Ethernet / USB 3.0
• Modern BIOS which supports booting from USB
• eMMC modules or SSDs instead of 3.5" hard drives

I remember back when I started getting excited about ARM SBCs, these kind of computers existed, but they were new-ish, and almost always quite pricey. But now they're abundant like garage sale walmart bikes. Huh. Go figure.

Caveats

According to netizens from Free Geek, while almost all of the thin client boxes support SATA for storage, some of them only have room inside for the SATA SOM (System On Module) form factor. So they can't be used with hard drives, one would have to either purchase an SSD that comes in this form factor or else purchase a regular one that is known to contain a short stubby little board inside and remove the tin cover from the outside of it.

Upcycle That Dell

The dream of home-brewed redundancy and failover for less than $100 is still very much alive! I don't see any reason why these machines can't/shouldn't be picked up for a pittance and used as servers. I also don't see much of a reason to prefer SBCs or arm CPUs any more.

So while I'm figuring out where I want to go with my home-brew / community hosting software endeavors, I will be putting some of these ex-corporate thin client boxes on my holiday wishlist.

I can't think of a more compelling story for the hardware of the community-hosted internet revolution than that it was essentially dumpster-dived from behind a Cisco office park somewhere.

In my last post, I wrote a about how I'm changing direction away from my "trustless cloud-based gateway to make self-hosting easier" project called greenhouse. I also wrote little bit about where I want to go with my software projects in the future. In this post, I'm going to expand further on that. In fact, I'm writing this now so that I can help myself organize my thoughts and clarify my own ideas.

Last time I said that I wanted to produce a piece of software in which two server admins' servers can "federate" with each-other. Another 3rd friend who doesn't run thier own server can have an account on these two servers.

If the server on which the account was originally registered gets powered off or lost in a fire, the friend can still log in and use their account like normal on the one remaining server.

However, I think when I said "federates", it may have been a bit of a misnomer, perhaps it might be more accurate to say that the two servers form a cluster with each-other and that they replicate the data.

But wait a minute. Cluster? Really? What is this, a "cloud scale" enterprise solution? "Cluster" makes it sound like a supercomputer. I thought this was about something that folks run at home, ya know, livingroom servers?

But before we get into that, its time I define some of the terms I'm throwing around and give a brief refresher on what the heck these things actually are, how they work, why they exist, and most importantly, how they tend to fail.

Federation

Federation has become a bit of a buzzword among the FLOSS and self-hosting communities within the past few years. Federated systems like matrix are beginning to provide (in my opinion) the first real viable alternative to centralized social media platforms. Federated social media protocol standards like ActivityPub connect users not just on different server instances, but between different types of server software.

For example, if you post a photo on your PixelFed account, my GotoSocial account can "like" it and someone else's Mastodon account can leave a comment. Such is the nature of the emergent social network dubbed "The Fediverse".

📊 Legend

• Writable
• Read-only Replica

ℹ️ Note: Each account on this style of federated platform is intrinsically linked to its "homeserver", the server on which the account was created. Federation partners are free to store copies of accounts and activity records from other servers, but in general they're not allowed to create or update anything that "belongs" to another server.

The homeserver concept is deeply embedded in the design of the ActivityPub protocol; there's no way around it. Every single ActivityPub message contains an id property which must be a URL identifying the dial-able address of the server who is considered the authority for that information.

{
 "@context":"https://www.w3.org/ns/activitystreams",
 "id":"https://friend.camp/users/darius/outbox",
 "type":"OrderedCollection",
 ...

These new "grassroots" distributed platforms are succeeding because they have managed to generate/harness network effects: They're good enough at forming and maintaining connections between servers that even though there may be hundreds of different people and organizations operating servers, having one account on one server means you can theoretically "reach" the entire network.

At the same time, federation is opt-in and fine-grained enough that new servers joining in aren't instantly swamped by traffic from the entire network. If I operate a federated social media server and invite 100 users to join, they may eventually "follow" thousands of other accounts on hundreds of other servers.

That might sound like a lot for my poor server to process! But it's nothing compared to the millions and millions of fediverse accounts that exist. Computers are really fast, and for them, hundreds or thousands of things is a piece of cake.

💡 Key Idea: Each fediverse server is operated by a different administrator, and may play by different rules. Servers can ban eachother, limit eachother's federation based on arbitrary criteria, etc. They can have software bugs and be incompatible with each-other, the sky is the limit. Each server admin has complete control over what happens on thier server.

Clustering

"Cluster" usually refers to software that was designed to run on multiple computers in order to harness thier aggregate computational power and data bandwidth. This practice is often called "scaling horizontally", and it is desirable from a business perspective because, in simple terms, it's a lot cheaper to buy & maintain a bunch of small computers than it is to buy & maintain one huge computer. Plus, if we play our cards right the army of smaller computers can be much more reliable, since it can avoid single points of failure.

📊 Legend

• Writable
• Read-only Replica

As you can see, this one is a bit of a doozy.

ℹ️ Note: In this example, there is only one administrator. Every single cluster system I have ever encountered is like this: Intended to be deployed entirely by one organization, entirely under one umbrella, always within a single datacenter LAN.

In this case, there is no self-determination angle, clusters as we know them were borne primarily from economic factors that only influence large scale software and web service companies.

• There are three servers, A, B, and C
  • Three is often the minimum number of servers for a cluster to reach its full potential.
  • Large clusters may include hundreds of servers.
• The data is split into three "shards", 1, 2, and 3.
  
  • 🧐 Like the number of servers, the number of shards will vary... + Show
    

    • For example, in Apache Kafka, the number of Partitions (another word for shards) determines the maximum number of clients (Kafka consumers) who can all collaborate on the same task at the same time.
    • Since one node will often be designated as the "primary" or single writer for a shard, the number of shards will also limit the number of cluster nodes who can work together on writing data to the sharded index.
    • Too many shards may increase the amount of metadata handling and shuffling that the cluster has to do, in some cases dramatically slowing down it's operation.
• Note that each shard is replicated to two nodes.
  • This guarantees that if one node goes down, the system continues operating.
  • As long as the replication is working, even if the node who is currently the writer for a given shard goes down, its replication partner node can be instantly promoted to be the new writer.
• The cluster client library may be able to predict which shards it needs for a given query and only contact nodes owning those shards.
  • If it can't, that's fine too. Whichever node is contacted by the client can forward the request to the appropriate server(s).
  • For example in the above diagram, if the client asked Server B to write to shard 1, Server B would simply proxy the request to Server A, because Server A holds the "primary" of shard 1, that is, Server A is the designated single writer for shard 1.

🤔 Remember how the ActivityPub federation protocol put a single id property on everything, and that id is a URL which specifies the dial-able location of the authoritative server for this info?

{
 "@context":"https://www.w3.org/ns/activitystreams",
 "id":"https://friend.camp/users/darius/outbox",
 "type":"OrderedCollection",
 ...

Lets try to find a comparable example for Apache Kafka. How does kafka handle this?

Apologies for the incoming wall of information; please bear with me through it. Or if you're not feelin it, just skip this part, most of it is details that don't matter for the overall story anyways.

A Kafka cluster is composed of one or more servers called Brokers. A Kafka cluster also has one or more Topics. You can think of a Topic as following the spirit of a Table in a relational SQL database. Topics are broken up into multiple partitions, where each partition represents some arbitrary slice of the data inside the topic. Not as a slice of time, but as a slice of the entire history of the topic.

In the following screenshot from the cute kafka explainer https://www.gentlydownthe.stream, a kafka topic is represented as a river:

Each partition is just a sequence of messages, and each message in a partition is identified by its offset within the sequence. So the first message in the partition would be at offset 0 and the 100th message would be at offset 99. The offset can only increase, never decrease, as the partition is an append-only log.

This part of kafka utilizes a separate consensus cluster software called ZooKeeper. Yes, that disturbingly proportioned little cartoon man holding a poop shovel really is the official logo 😬

So at the end of the day, the full "address" of a message in kafka might look something like <topic>/<partition>/<offset>. (Although, you would never it see written out as a path like that.) Anyways, when it comes time to grab a single message like that, how would Kafka do it? There's no directly dial-able address here like there is in ActivityPub.

Why, it uses the Broker Index and Topic Index, of course 🤪!

/brokers/ids/[0...N] --> { "host":...,"port":... }

/brokers/topics/[topic]/partitions/[0...N]/state --> {
  "leader":...,
  "isr":[...]
}

So first we look up the partition's current state. The leader will be the broker ID of the broker who has been assigned as the single writer for that topic partition.

The isr stands for "In Sync Replicas" (No, unfortunately its not an nsync cover band 😥)

The "In Sync Replicas" is a list of broker IDs representing all brokers who are currently replicating the partition from the leader in real time.

From there, we simply look up the hostname and port of the broker in the broker index by id. In this way, we can connect to any of the brokers we found and issue a fetch request to grab messages starting from a specific offset. Or, if we want to issue a publish request to append a new message to the end of the partition, we have to specifically use the leader broker.

💡 Key Idea: The topics, partitions, and messages don't know or care which broker they are on, how many brokers there are, etc. This is a separation of the logical data organization scheme from the underlying hardware configuration.

The cluster's internal protocols handle the real-time tracking of where everything is, and that internal metadata is kept separate from the data that the system is storing/processing.

This kind of separation exists in every piece of cluster software that I've seen. The replication we see in the logical layout provides "high avaliability", while the separation between the logical layout and the physical reality of the dial-able hardware helps with "fault tolerance". It's much easier to recover from a node failure if it doesn't require millions of URLs embedded in the data to be updated.

Put together, these two properties give clusters superpowers that normal single-computer programs can only dream of.

1. A computer can catch on fire and explode without causing a noticeable impact to the system.
2. A destroyed node can be replaced while the system is running, again, no noticable impact from a client's perspective.
3. The system can be upgraded to a new version, including breaking changes, while it's running. You don't have to turn it off to upgrade it.
4. The software's performance isn't limited as much by the hardware it runs on because its design spreads load out evenly among multiple computers.
   • Often times if you need the cluster to do more work or do the work faster, you can simply add another node.

🧐 Fun Fact: why clusters can provide faster storage if you're on a LAN + Show

Clustered databases and message brokers can also employ unique tricks to accelerate important storage tasks.

Engineers noticed that network and CPU hardware has improved beyond the limits of storage hardware like hard drives and SSDs. Specifically, the latency (round trip time) between two servers in a datacenter is hundreds of times faster than the time it takes to write data to an SSD.

Clusters can use thier strength in numbers to leverage thier fast CPUs and fast network for a massive performance gain over a traditional database performing the same task. When an application wants to write data and have confidence the data wont be lost, the cluster only has to replicate it to another node over the network before it can report the write as "suceeded". It doesn't have to wait for the write to be flushed to disk.

The chance of both of those nodes crashing at exactly the same time before either of them can flush the data to disk is so small that most clustered systems simply assume it will never happen. So a write can be considered "durable" or "fully persisted" before it even touches the disk.

Of course, I can't talk about clusters without mentioning the Big Important One that everyone is excited about: ☸️ Kubernetes.

Mostly up to this point, I have been talking about storage-related cluster software. ElasticSearch, Kafka, Cassandra. But Kubernetes is different, it's all about computation, running lots of programs instead of storing data.

I don't dislike Kubernetes, in fact on the surface level it seems like it could be an option for any project that aims to configure & run multiple programs spread across multiple computers.

But about three or four years ago I made an explicit decision not to try to use it for home-server related projects. I think it's simply not the appropriate technology.

Why?

Ultimately it came down to the network. As far as I know, from design, development, testing, and into production, the core of the Kubernetes cluster system, the "etcd" / "control plane" components were always intended to live on the same LAN. Whether the control plane nodes could talk to each-other directly wasn't in question, it was simply assumed that they always could. Not only that, it was also assumed they could talk to eachother with super low latency.

But in the home server environment, it just ain't so. Sure, you could run multiple servers in the same house. That would work fine, but it isn't going to increase the reliability of your services very much. Most outages will be caused by things like ISP outages, power outages, the cat knocks over the router and unplugs it, you lose your house in a natural disaster or move the server to a different house in a different city, etc.

Really, if we're trying to make a home server "cluster", we're gonna need multiple people's homes as well as multiple servers. But therein lies the problem: unlike the cloud and datacenter machines, these home servers are living in what an aquaintance of mine calls "NAT Jail." They can't be directly contacted from outside unless the self-hoster performs some port forwarding configuration on thier router first.

And even then, those home IP address are liable to change somewhat regularly. Many self-hosters may not even be able to configure port forwarding if they wanted to ☹️. It's all situational. So if I wanted to use Kubernetes in this context, it would probably require a VPN to be set up across all the nodes ahead of time. That's already enough of a red flag for me to call it quits; I want my multi-server solution to help me network my servers together from behind NATs and firewalls, not to demand that I have already done that part perfectly before I can turn it on.

I could definitely use Kubernetes as a single-node cluster, as many people do. In fact, someone is already marketing a self-hosting focused product around it. But if I was going to do that, why not just use docker-compose? Or use single-node docker swarm like co-op cloud does?

To be fair, I haven't kept up with the developement of various different lightweight Kubernetes distributions like k3s, and who knows, maybe in the year 2022 there is now a kubernetes distribution designed specifically for people who run servers in the mountains with solar panels and handmade radio antennas 😛

EDIT: after sharing a draft of this post on the cyberia.club matrix chat, my friend Winterhuman from the very cool IPNS-Link project pointed me to exactly the thing I was missing:

Winterhuman

  forest (he/him) Just read [your post] and thought you'd be interested in https://github.com/c3os-io/c3os, it's a Kubernetes distro, but, it uses Libp2p to escape the "NAT Jail" you talked about

forest

heh, I knew it had to exist

forest

very cool, although I think I would prefer something that has an installer instead of being a linux image build script. Installer is more portable, the ARM hardware requires different images for every single different board 🤮
and I still have reservations about kubernetes and to a lesser extent VPNs
but the parts it is composed of / overall approach is super interesting.

Winterhuman

You could do something similar, minus the config and the Libp2p stuff being built-in, by using https://github.com/hyprspace/hyprspace to create a VPN first (it can adapt to IP changes) and then layer stuff on top of it

forest

I still have to figure out how I wanna structure it
I've never used a VPN with docker before so that would be my 1st step.

At least for now, I stand by my decision to skip the Kubernetes for self-hosting. At the end of the day this c3os thing is still just an easier way to set up a VPN as a pre-req, and Kubernetes still has a lot of issues with complexity and usability in my opinion.

Honorable Mention: P2P Networks

But Forest,

You ask,

Distributed Computing? Decentralization? Self Determination? My friend, you are describing a peer-to-peer network. You should just use ipfs and all of your problems will be solved.

Well, dear reader, if I caught you thinking this, don't worry. I get it. I love P2P as much as the next guy; I've dreamed up a hairbrained scheme to rebuild twitch.tv as a peer to peer network that runs in the web browser, hacked on WebTorrent on my livestream, and I even wrote a passionate plea to breathe life into Namecoin.

Ok, wow Forest, if you love P2P so much why don't you marry it?

The TL;DR is that the web browsers everyone uses still do not support IPFS in 2022 and I'm unsure if they ever will. I'm not holding my breath.

I do believe in transformative P2P applications that are worth downloading a client app for. I've seen two so far in my lifetime:

• BitTorrent
• cryptocurrency

😇 Anti-cryptocurrency folks, please don't write me off right away! + Show

... please don't write me off right away! I probably agree with most of your position. Measured in dollars or work hours, the majority of activity in cryptocurrency does probably fall under the "scam" category.

The future doesn't look bright for Proof of Work cryptocurrencies like Bitcoin. As they begin to chomp at a slice of the macroeconomic pie, they start to incentivize really counterproductive dead-end behavior like

"Hey, lets buy every single GPU on the market and mine Etherium with them!"

I know this is a divisive topic, but I can't see it in black and white.

If you look at the history of Cryptocurrency and see what went down with Bitcoin, Yes, it is primarily a story of misguided dreams, greed, human folly, capitalism, scams, stolen pirate treasure hoards, etc. But I also see a bit of a David vs Golaith story where Bitcoin, the software, protocol, and community holds its own against stormy seas and the will of the incumbent financial system.

According to Blockstream, everyone was supposed to be using the Lightning Network for cryptocurrency payments by now. But that didn't happen. Folks rejected it because that wasn't what they wanted. Why participate in turning cryptocurrency into a bank-style network where everybody owes everybody else and none of the debts can ever be paid when the original idealistic Bitcoin from the 2009 whitepaper seems to work fine? Especially when the people pushing Lightning network, people on a large bank's payroll, are also ferociously defending a nonsensical technical decision which permanently limits the bandwidth of "original style" bitcoin transactions network-wide. Instead, everyone started adopting altcoins for thier small payments. Coins whose developers refused to be bought out, like Litecoin, Monero and Bitcoin Cash.

Cryptocurrency has not yet been assimilated or destroyed by banks, despite some definite effort on thier part. It may be destroying itself instead. Being voluntarily dismanted those who are tired of trying to find the diamond in the rough and see no long-term path forward. Since we've started to see systemic issues with Proof of Work mining, I can't be sad about that.

May the part that is worth keeping survive, if any. I would still be hyped for a workable long-term solution to break Zooko's triangle like how Bitcoin and Namecoin do in "production" today.

But I think anything to do with media or publishing is still going to have to at least support the web browser, either with oldskool HTTP or with P2P support directly in the browser. Not a special browser like Tor Browser, but at least Firefox, and probably Chrome too.

IMO, things like ZeroNet, Beaker Browser and GNUnet are interesting and compelling, but with no mainstream web-browser adoption anywhere on the horizon, they're nothing more than toys or prototypes with no possibility of generating enough network effect to get going.

It's like I wrote 6 years ago:

If your app doesn't have a URL, who's going to use it?

WebTorrent is a perfect example of P2P support in the browser. If you haven't heard of it before, I highly recommend checking out the demo on thier front page: https://webtorrent.io/

In fact, it's so cool, I'll include a screenshot of it here:

In case you aren't sure what's going on here, it really is torrenting a video file in real time in your web-browser. Those yellow dots on the left are the "torrent swarm", the other people accessing the demo at the same time as you. They are all sharing chunks of the video file to each-other to accelerate the download. In fact, unlike a traditional HTTP file server, the more people who attempt to torrent this file at once, the FASTER the download will go for everyone.

It's amazing to me that software like this is already mature and has been doing this in the web browser for years, and it's now powering potentially useful apps like PeerTube. I think there's a lot more untapped potential for these kind of peer-to-peer enabled web apps.

However, we can never lose ourselves in the hype. Remember that this is only ever going to work on your mom or dad's computer because someone is running a traditional HTTP server to provide them with the "backwards compatible" web application and JavaScript to make this happen.

If you want to harness this kind of power yourself, if you want to send someone a link to your own version of this, you're probably going to have to set up that HTTP server too.

I think we're going to be using HTTP and TLS for many more years; in my opinion it makes sense to start from there.

I'm still searching for a way for folks to set up thier own publicly dialable HTTP servers hosted at home or within thier own community. Ideally, without nearly tearing thier hair out in frustration in the process.

Bro you said "brief refresher," wtf

Ok, that exposition may have gotten a little out of hand. These are complex topics:

• Federated Social Media Protocols/Platforms
• Storage and Compute Cluster Applications
• Also, Peer-to-Peer networks apparently?

And they're all so close to my heart, filled with personal stories, feelings, funny anecdotes, etc.

Ok ok, I promised I would cover:

how they work, why they exist, and most importantly, how they tend to fail.

I believe I have covered the first two fairly well. So now it's time for...

most importantly, how they tend to fail

I used to work on distributed systems a lot at a my old job. I did a lot of work upgrading Kafka and Cassandra clusters while they were running & spent a lot of time monitoring the health of the clusters, learning about what can go wrong with them and debugging issues as they came up.

Cluster applications are much more complex, so they're harder to interact with. They almost always require special client software or client libraries to interact with. Sometimes these clients feel like they're missing some important features, but doing anything with the cluster "by hand" without the client can be scary or prohibitively difficult.

They're also very hard to debug. When you get off the beaten path, thier user-interfaces, especially when it comes to error handling, range from "very poor" to "non-existant". It was so bad. I remember having to SIGKILL (run kill -9) on stuck kafka broker processes quite often. Not a comfortable feeling on a production system. We never lost any data tho!

🎶🎤 I guess I’ll have to shut you down for good this time... 🎶 + Show

I guess I’ll have to shut you down for good this time
Already tried a SIGQUIT, so now it’s KILL DASH 9
You gotta learn when it’s time for your thread to yield;
It shoulda slept; instead you stepped and now your fate is sealed
I’ll take your process off the run queue without even asking
Cause my flow is like reentrant and preemptive multitasking
Your sad rhymes are spinnin’ like you’re in a deadlock
You’re like a synchronous sock that don’t know when to block;

All in all, I think that clusters are easily misunderstood and almost always difficult to work with. There's just so much going on with them, especially complex clusters like Kubernetes, it can be overwhelming.

Part of this is because of the complexity, but I think it's also because contemporary demands for clusters were almost exclusively about more storage, more bandwidth, more stuff. These clusters were only ever intended to be operated by highly trained mega-nerds in a professional setting; they could get away with little to no interface for the operators. Instead, all the development effort went into optimizations and new features.

The environments where these clusters run and the jobs they fulfil are some of the most demanding ever known to computing. It's not surprising to me that they're difficult to use.

💢 Clusters fail before they get started. People like me refuse to use them because they are simply too big and difficult, the requirements are too high and inflexible and there's relatively little payoff.

In the homebrew realm, clusters fail specifically because they've all been designed to run on a shared LAN. In thier current implementations, the individual cluster nodes can't/shouldn't be spread to multiple different localities.

Whoever has power and internet at home that's reliable enough to justify deploying a whole cluster on... Well, they must live in a datacenter, so they're irrelevant in this case 😆

Federated social media services, on the otherhand ARE often designed to be easy to self-host, sometimes even including how-to guides for hosting at home. They're fairly easy to get started with, but sometimes struggle to achieve longevity.

Many people have commented on this phenomenon at length. Personally, I'm a fan of Darius Kazemi's discussion of some of the issues at play on his how-to guide https://runyourown.social/.

Some highlights from my memory of that guide:

• Administering one of these sites is a significant & continuous burden.
  • It's a big responsibilty: if you stop maintaining it, everyone who has been depending on you will at best have to go through a migration process, and at worst lose their account and everything that was attached to it.
  • You'll probably want to maintain a federation block list to keep various kinds of 4chan-esque "nasties" off of your server.
• You probably want to limit the number of users; servers that have grown out of control invariably run into problems.
  • Often these problems are more social than technical: There will be disputes. There will be situations where you have to make moderation decisions which will alienate some of your users. You can't please everybody.
• Allowing anyone to register an account at any time is probably a recipie for disaster for similar reasons.

The way I see it, the average Fedi instance is somewhere in between:

1. a personal experiment that can always be shut down

and

2. a service that many people rely on & can't be allowed to fail

It's an awkward in-between for sure, and I think this is the crux of the issue with the federation concept as it's implemented in the Fediverse.

In order to preserve the autonomy of each fedi admin, all of the user data under thier care is intrinsically linked to the domain name of the server they operate. It may be possible for one admin to pass the torch to another person, but I suspect this rarely happens in practice.

Usually when an admin loses interest in maintaining something like this, it'll happen without warning, and by the time it becomes apparent, it might already be too late to try to transfer ownership. Life happens. Sometimes circumstances change and folks won't have the spare time for this kinda thing. That has to be ok. Technology is supposed to work for US, not the other way around.

💢 Instances of Federated network software fail because they place a constant burden of maintenance on (usually) a single administrator.

These instances are often designed to be easy to set up, but as soon as folks start using them, they can become entrenched and both difficult to maintain as well as difficult to transfer to someone else. For example, it's not enough to transfer a backup of the server's data or the physical server itself, one would also have to transfer ownership of the domain name.

It can turn into a quagmire where the users stand to lose a lot and the admin doesn't have any easy way out.

Cool story. But what is this all about?

I created greenhouse because I was frustrated with how difficult the network configuration part of self-hosting can be. Specifically, I was frustrated by the lack of solutions that always work no matter what.

I thought I saw a need: All of these self-hosted software setup guides are super easy and simple until you get to the part about NATs, routers, port forwarding, DNS, dynamic DNS, etc. Then everything goes off the rails into seemingly endless tangents and what-ifs.

I've written on this before: Shells are well standardized. Linux is fairly standardized. The web protocols like TCP, TLS, and HTTP are incredibly well standardized. But every home network is different, there's no single way to configure the network for a server.

Greenhouse was an attempt to allow the self-hoster to flip the table and refuse to configure the home network at all. It was Infrastructure as a Service that outsources the DNS configuration and TCP listener into an easy-to-use public-cloud-based service while keeping the TLS (encryption stuff) on the self-hosted server.

I believe that greenhouse failed because on its own, it failed to address a widespread need. As I explained in my last post, greenhouse catered to self-hosting fundamentalists, but at the same time, it IS a 3rd party service, the bane of every self-hosting fundamentalist's existence.

So I ended up trying to rethink the whole thing, rethink my whole approach.

I was inspired by the way that the fediverse had seemed to address two extremely common needs at once.

1. Convenience. Just sign up for an account; don't have to run your own server
2. A sense of data custody and belonging within a tight-knit community

Even better, unlike greenhouse, this time the venn diagram has a huge overlap, it's practically a circle.

In fact, I think the same venn diagram can be generalized to talk about huge swaths of the internet today. It's funny to me that everyone wants a server but no-one wants to run one. But I think it's all too true.

The list of internet usecases which could conceivably fall into this category goes on and on.

It reminds me of the children's story The Little Red Hen.

Except this time she's baking digital bread with near-zero marginal costs, so even if the hen and her chicks eat first, there should still be enough left at the end for the the Pig, the Duck, the Cat, and everyone else in the pasture.

But there's just one problem with this little "online era" adaptation of The Little Red Hen. If there's one thing I learned in my career as a digital bread factory technician (and my subsequent mini-retirement as a trappist digital bread monk), it's that when it comes to the best of breads that bring the boys to the yard, nobody can bake em alone.

That is, popular web technology, especially when we consider it as the entire stack all the way from the server's power supply to the router, the operating system, processes, protocols, html, images, text, usability, design appeal, accessibility, client compatibility and everything else, its simply way too much for one person to carry.

I have been trying to carry all of it myself in my work on greenhouse and my other personal projects that never saw the light of day, and it's been hard. I haven't seen any success yet.

But I'm trying to stay positive. I love the internet because no matter how uniquely impossible your personal struggles may seem, there's probably a niche community out there somewhere where you can commiserate or find some kind of solace. No matter how obscure your problem, you can always find someone else who has the same problem. The internet is amazing for forming communities and meeting new people who you have something in common with.

And in those communities, through collaborating with my friends, I have seen some successes. The graph of greenhouse users and activity went down and to the right. But another project that I developed with my friend j3s, capsul.org, is going up and to the right, and it has been for over two years now!

As happy as I am to see capsul succeed, I'm still a bit of a self-hosting fundamentalist myself, and I want to put the majority of my work into projects that're friendly to self-determination and small energy efficient servers at home.

But I'm tired of trying to "manifest" a reality where anyone can do this themselves. It's hard enough for me to do what I do, and I've been working my ass off for my entire life to get to where I am. Self-hosting is such a shitty term because it sorta implies that you do it all by yourself. Not only is that unrealistic, it's also a lot less fun.

I'm still trying to figure out what I want to build, let alone what to call it. I want to build slow-cluster software that's all about NAT traversal and good user interface instead of about scalability. I want to build something like the fediverse, software that can form circles of friends. Not a social media of URLs, but a homebrew web server infrastructure cleanly abstracted away from the hardware, abstracted away from the individual, and free to flow like water.

(Albiet slowly when it has to flow through the internet connection in grandma's basement 👵)

Maybe I want to make a homebrew "cloud" IaaSS (Infrastructure as a SelfService 😛) that server admins can operate with a couple friends. Whatever it is, I think it should combine the best aspects of both Federation and Clusters, while attempting to avoid the worst pitfalls of both.

So in the end, It's not Federation vs. Clustering...

It's

Federation 💘 Clustering

👶?

Tune in next time 😉

Previous post in this series: 🥳 Greenhouse Enters Alpha Test Phase!! 🎉

I did the thing! I created greenhouse almost exactly like how I imagined it six years ago in 2016.

I recieved a lot of feedback on the project when I shared it with the world. While most people agreed it was pretty cool, I recieved a lot of criticism as well:

👹 Self-hosting is dangerous and you are inviting people to get hacked! The lawyers will come after you! The world is scary and everyone should just keep hiding in thier webcage that the big-tech panopticons provide!

🤮 Your website sucks! It looks bad!

Some criticism came in the form of valid questions, questions to which my answers had to be disappointing. Because greenhouse is so different & doesn't really fit into a well-known category, it's architechture, feature set, and user experience confused and surprised many potential users:

🧐 How do I self-host the tunnnel gateway server part of greenhouse?

You can, but it's really hard. You aren't supposed to, you don't need to, that's the whole point. You just run the tunnel client daemon on your server computer and configure it with the CLI or GUI app. It's like this because I wanted it to be as easily as possible to get started with.

🤨 Does it support caching? Like on the "edge" server where caching is most effective?

No, it can't support caching like that because the greenhouse tunnel server can't read your traffic; the HTTP traffic stays encrypted via HTTPS. IMO this is a good thing.

🥱 Why does my page load so slowly? How can I make it faster?

It's because you live in Europe and right now there is only one greenhouse tunnel server, and it's in NA. In the future I'll have multiple servers in different regions but for now it's just the one.

Also, the tunnel introduces multiple network roundtrips right now. In the future I can at least cut those extra trips in half by replacing yamux with the QUIC protocol, but the traffic will always have to go through the tunnel during the first page load no matter what, so it will always be higher latency than normal.

The harshest criticism, however, came in the form of usage statistics. Most people agreed it was a neat idea, and some people did try it out, but over time, it got used less and less. It did not grow organically.

So where does this leave me? Did I fail? What does it all mean?

It's no secret that greenhouse has always been an ideologically motivated project. It was practically fundamentalist in terms of how it viewed self-hosting: The greenhouse service was supposed to make it easier for someone to run thier own server, leave it on 24/7, without them having to relinquish any privacy, agency, ownership, and control. If you didn't want to run a server, greenhouse wouldn't be very useful to you.

But at the same time, in the interest of ease of use, it was designed from the ground up as a public-cloud-style Infrastructure As A Service (IaaS) provider. You pay the service some miniscule amount of money per unit of bandwidth that you consume; at the end of the month it would bill your credit card automatically. At least that was the plan.

Even though it was designed to follow the principle of least authority and defer all security-relevant decisions and processes to the self-hoster and thier server, at the end of the day greenhouse is a 3rd party service provider.

Through the combination of these two factors, I think it was always doomed to fail. On one had, it'll probably only appeal to self-hosting fundamentalists. But on the other hand, its a 3rd party service, and self-hosting fundamentalists tend to hate 3rd party services.

In terms of the technology and how it worked, greenhouse was fine. Great, even. But in terms of how it made people feel, it was disapointing. And to make matters worse, because it required a server computer to be running 24/7, its usefulness to the average person was severely limited.

I spent a lot of time and effort creating a GUI for greenhouse and porting it to the Mac and Windows platforms because I wanted to include those platforms and include everyone who uses them. But I think in the end what I produced was nothing more than a curiosity. No one wants to leave thier Mac or PC laptop on all the time so the server will stay up.

How to fix it? How my thinking has changed

I mentioned that the idea for greenhouse is over 6 years old at this point. A lot has changed in 6 years, both in terms of the cutting edge of self-hosted services as well as my own personal experiences with them and feelings about them.

First of all, I saw the rise of "the fediverse" (ActivityPub-compatible social-media and microblogging servers starting with mastodon) and similar networks like matrix.

I also joined / helped build and maintain infrastructure for Cyberia Computer Club, including the loosely associated VPS provider capsul.org. At first I was reluctant to spend my time on capsul because I saw it as only marginally better than the large-scale commercial offerings like DigitalOcean. Same shit, different day kind of thing. My ideology was telling me that instead of custodial services, I should be focusing on projects that allow people to have ownership (physical custody) of their data and processes.

But over time, my outlook started to change. I had a lot of fun with Cyberia as it continued to grow and develop, including renting a commercial unit & founding a new hackerspace dubbed Layer Zero.

I also read some very passionate takes on the Fediverse architechture and how it can change the way we view software and networks, like Darius Kazemi's runyourown.social and Roel Roscam Abbing (rra)'s Seven Theses On The Fediverse And The Becoming Of Floss. These ideas made a big impact on me. I think the short, pithy description on homebrewserver.club might summarize it best:

Take the ‘home’ in homebrewserver.club literally and the ‘self’ in self-hosting figuratively
That means we try to host from our homes rather than from data centres - a.k.a. ‘the cloud’ - and we try to host for and with our communities rather than just for ourselves.

That "for and with our communities" bit was the part I was missing. It's not "same shit, different day" like I had originally feared; with fediverse-style networks and similar community projects, yes, the person with the server can still surveil, censor, and falsify everything of "yours" that they host for you... But that power dynamic is different when it exists outside of commerce and capital, when it's a part of a local gift economy or federated network of "indie" & self-determined servers.

So I think I ultimately want to redesign and re-brand greenhouse/server.garden as software that folks mostly run on thier own computers, and as something that can be connected to form ad-hoc federated networks.

Not just tech, but also trust

Right now greenhouse is purely a "trustless" networking service, but redesigning it this way could open up new possibilities because it would open up a new dimension of trust: The trust the fedi-users have for thier server admins, and the trust that fedi-admins have for the fellow admins that they federate with.

"Trustless" can be cool, but having a server to trust is incredibly practical. I can imagine:

• Github Pages / Backblaze / neocities style static content hosting and object storage
• Amazon RDS style "managed" replicated relational databases
• Fly.io style distributed linux container platform
  • Specifically, an easy-ish way to do redundancy and failover for arbitrary server apps.

I have a lot of ideas for this, as usual I've been thinking about it a lot... In my imagination, two or more friends can set up thier own home servers and install this software on it, then trust each-other's servers so they'll federate with eachother. Then our two-or-more admins can create accounts for anyone who wants to use the servers. The admins' friends can use them for static content publishing, as a greenhouse-style network gateway for thier own server, and potentially even as a "cloud-ish" compute provider. The best part: if one of those servers goes down, the other one can pick up the slack.

Its a lot, but I think this is the direction I wanna go in. I'll probably start with the static content hosting / object storage feature, eventually integrate the threshold / greenhouse daemon tunnel gateway feature, and see how things progress from there.

I have all kinds of ideas for this; with both a tunnel gateway and static content capability, the platform could offer a kind of hybrid hosting where you run a server app on your computer or phone which is "live" while your computer is turned on, greenhouse style, but as soon as you turn it off, it falls back to a cached version that is hosted by the federated platform.

For example, folks could run thier own owncast video livestreams this way without having to set up a server for it. When they turn off their computer the stream goes down, but the website stays up. I think this could be useful for all kinds of apps, it would add a new dimension of flexibility where folks can casually self-host applications on the internet, even interactive applications, from computers which aren't servers — laptops/phones/gaming rigs which aren't on all the time.

By the way, I'm still trying to figure out what to call it. I need to come up with a name and a way to "market" it so that it makes people feel cool or "sexy" when they use it. (Thanks to my friend j3s for these insights 😛)

If you have any suggestions, leave a comment!

This post is part of a new experiment; I'm toying with the idea of writing a concise and approachable "folk textbook" covering the conceptual basics of what I do; using Linux and other operating systems like MacOS and Windows, home routers, networking software, servers, virtual machines, linux containers, etc. I'm starting out by writing a few drafts or tidbits in this style as blog articles.

If you're new to running server applications or even if you're an experienced application developer who's never studied computer networking (TCP/IP specifically), there's a good chance you use listening addresses all the time without fully understanding how they work or what they're doing.

I definitely spent many years as a professional software developer before I started to learn all the details of how listening addresses work, and I wish I had found a resource like this article earlier.

To make matters worse, it seems like every application has slightly different behaviour when it comes to listening addresses and may parse or represent them differently. Too often these applications are not designed to be easy to use and they don't explain anything; they simply assume that you already know what you are doing.

Why Listening Addresses Matter

When starting to run a new server application such as a webserver or other network service, often we will start it up, try to connect to it, and... No dice. Why isn't it working? Why can't we connect? There are many possible reasons why, and this article aims to explain what could go wrong and how to figure out what's happening.

Our first step: Figure out whether or not the application is listening at all, and if so, what listening address(es) it's using.

Identifying a Listening Address

A listening address is typically writen like a normal network address, that is, it's written as <host>:<port>.

<host> here should almost always be an IP address, and <port> should almost always be a single port number. However, in terms of how they are represented in configuration files and logs, there's no strict standard, and developers ocasionally implement some, shall we say, "creative" representations.

Here are some examples of what I would call "standard" listening addresses that use IPv4 addresses:

• 127.0.0.1:8080
• 0.0.0.0:22
• 123.45.67.8:35871

And here are some that use IPv6 addresses:

• [::1]:8080
• [::]:22
• [2607:fb90:1788:efbd:d3f9:555:4ddb:c8ed]:35871

And here are some that use various shorthand notations or alternate representations:

• localhost:8080
• ::1:8080
• :22
• *:22
• :::22
• example.com:35871
• 127.0.0.1:3000-4000

Any given server application should write the address it decided to listen on to its log immediately after it starts up. We should be able to connect to the application by connecting to that address. But how would one connect to 0.0.0.0:22 or :22? What do those addresses even mean?

About "Special" IPv4 and IPv6 Addresses

You may already be aware of some "special" IP addresses, for example

127.0.0.1 which is called the loopback address, the address of the localhost domain.

OR

192.168.0.1 which might be the address of your home router.

If you want to learn more about these, I recommend the Wikepedia article on IPv4.

There's another special IPv4 address which (as far as I know) is only used by listening addresses:

0.0.0.0 technically means something like "null" or "no address", but when used as a listening address, it's interpreted as "Listen on ALL Addresses".

Don't ask me why, but IPv6 addresses are written with colons : in between the numbers instead of periods . (in my opinion this was a huge mistake 😫)

So this special address for IPv6 would be written as 0:0:0:0:0:0:0:0, but you never actually see it written that way. IPv6 also introduced a special shorthand where repeated 0:s are abbreviated as ::. So, confusingly, the IPv6 "Listen on ALL Addresses" IP address is written as ::.

To make matters worse, when an IPv6 address is represented as a part of a network address, we have to be able to tell the difference between the IPv6 colons and the <host>:<port> colon. So those network addresses are written like [::]:22 or [::1]:8080.

In case you were wondering, ::1 (expanded, it would be 0:0:0:0:0:0:0:1) is the IPv6 "loopback address" like IPv4's 127.0.0.1, the address that loops back to the same computer who is dailing.

Rant: A Server Application is Giving Us The Silent Treatment...

Here's a concrete example of just how terrible the user interface and usability of server software tends to be. The following is based on a true story...

The venerable web server nginx (pronounced "engine-x" 🤦) doesn't log anything at all when we run it:

$ sudo /usr/sbin/nginx
$ 

In fact, it exits immediately (gives me my command prompt back) as if it refused to run or perhaps crashed. So what gives? Lets check which process exit code it outputted. (For process exit codes, 0 (Zero) means success, and any positive number means failure).

$ sudo /usr/sbin/nginx
$ echo $?
0

It returned 0, so it says that it worked, what gives? We get a clue if we try running it again:

$ sudo /usr/sbin/nginx
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)

This time it's complaining that it can't listen on port 80 because some other process is already listening on port 80... But it's not referring to it as a "port", it's referring to it as an "address". We've spotted our first listening addresses, 0.0.0.0:80 and [::]:80, in the wild! In this case, it looks like nginx is trying and failing to listen on port 80 on all IPv4 and IPv6 addresses.

It turns out that by default, nginx runs in "daemon" mode. And by default, it doesn't log anything while it does this. That means when I ran sudo /usr/sbin/nginx and it appeared to have cancelled or failed, what actually happened was the nginx server spawned in a separate process. I never saw any output because my shell was not attached to that other process. That's also why it complained about Address already in use the second time I ran it: it was trying to start up a second copy of the server while the first one was still running.

After a bit of looking things up online, I returned with the following command line options for nginx:

$ sudo /usr/sbin/nginx -g 'daemon off; error_log stderr info;'
2022/01/30 21:54:21 [notice] 431278#431278: using the "epoll" event method
2022/01/30 21:54:21 [notice] 431278#431278: nginx/1.18.0 (Ubuntu)
2022/01/30 21:54:21 [notice] 431278#431278: OS: Linux 5.4.0-96-generic
2022/01/30 21:54:21 [notice] 431278#431278: getrlimit(RLIMIT_NOFILE): 1024:1048576
2022/01/30 21:54:21 [notice] 431278#431278: start worker processes
2022/01/30 21:54:21 [notice] 431278#431278: start worker process 431279
2022/01/30 21:54:21 [notice] 431278#431278: start worker process 431280
2022/01/30 21:54:21 [notice] 431278#431278: start worker process 431281
...

The -g flag stands for "-global directives", it allows me to add a couple extra configurations on top of the existing configuration file.

The daemon off; part tells it to run like a normal process instead of in "daemon mode" (That is, run in my shell, output its log to the shell so we can see what it's doing).

Finally, error_log stderr info; tells it to output its error log to the process' stderr stream, which will be displayed in the shell.

However, it still doesn't log the port(s) it's listening on 😒

IMO, this should be embarrassing for the developers/maintainers of nginx; that someone trying to use thier software couldn't even tell if it was running or not. Thier software did not report anything at all when it was executed. Even after traversing the first usability gap, the user still doesn't know whether or not its listening on some port, and if so, which port it's listening on.

But they probably aren't embarrassed at all; they probably view this whole kerfluffle as "the user's fault" for being ignorant; for not already knowing how nginx works or how to answer thier questions on their own. In my opinion, that's a toxic take. The whole point is that we are trying to learn about nginx and server software in general. It is nginx that is at fault here for making this such a painful experience; it doesn't have to be this way IMO. There is plenty of other server oriented software which manages to keep its command line interface approachable.

It turns out that nginx DOES in fact log the ports / addresses it's listening on, however, it logs them at the debug log level, so they are hidden by default unless we specify that we want to see the log at the debug log level instead of the info log level. error_log stderr debug instead of error_log stderr info:

forest@thingpad:~$ sudo /usr/sbin/nginx -g 'daemon off; error_log stderr debug;'
2022/07/26 12:08:21 [debug] 19486#19486: bind() 0.0.0.0:80 #6 
2022/07/26 12:08:21 [debug] 19486#19486: bind() [::]:80 #7 
2022/07/26 12:08:21 [notice] 19486#19486: using the "epoll" event method
2022/07/26 12:08:21 [debug] 19486#19486: counter: 00007F4CAA115080, 1
2022/07/26 12:08:21 [notice] 19486#19486: nginx/1.18.0 (Ubuntu)
2022/07/26 12:08:21 [notice] 19486#19486: OS: Linux 5.4.0-122-generic
2022/07/26 12:08:21 [notice] 19486#19486: getrlimit(RLIMIT_NOFILE): 1024:1048576
2022/07/26 12:08:21 [debug] 19486#19486: write: 8, 00007FFFEC8AE470, 6, 0
2022/07/26 12:08:21 [debug] 19486#19486: setproctitle: "nginx: master process /usr/sbin/nginx -g daemon off; error_log stderr debug;"
2022/07/26 12:08:21 [notice] 19486#19486: start worker processes

There are those two listening addresses, 0.0.0.0:80 and [::]:80, again! Last time we saw them in an error message, but this time we see them in a debug log. Finally, nginx is running properly and it's also at least sort of letting us know what it's doing.

In my opinion, an application with a proper user interface wouldn't run in daemon mode by default, and it would log something like this when it starts up.

nginx is starting up!
...
I am now listening publicly on 0.0.0.0:80 and [::]:80
You can connect to me at http://localhost/

Or if it was going to run in daemon mode by default, it would log something like this when run:

I am now spawning the nginx daemon process. 
...
Startup succeeded, nginx daemon is now running!
If you would like to see the output of that process, please check the log file /var/log/nginx.log

Flipping the Table (╯°□°)╯︵ ┻━┻

Luckily, we don't have to depend on applications like nginx which may be "unreliable narrators" at times.

We can always just ask the operating system what network addresses are listening and which application process those "listening sockets" are associated with.

Use:

• Linux sudo ss -plunt or sudo netstat -plunt
• MacOS lsof -i -P -n | grep LISTEN
• Windows netstat -aon on Windows.

ℹ️ NOTE: Those flags stand for -tcp, -udp, -listening, inlcude -program name, and display port number as a -number instead of listing the associated protocol. sudo is included because the -program name flag only works when netstat/ss are running as root)

On my computer, the netstat and ss commands outputted too much text to copy and paste here, so I've edited the output down a bit:

Netid  State    Recv-Q  Send-Q  Local Address:Port   Peer Address:Port  Process
...
tcp    LISTEN   0       4096        127.0.0.1:35625       0.0.0.0:*     containerd
tcp    LISTEN   0       50            0.0.0.0:139         0.0.0.0:*     smbd
tcp    LISTEN   0       4096          0.0.0.0:111         0.0.0.0:*     rpcbind
tcp    LISTEN   0       511           0.0.0.0:80          0.0.0.0:*     nginx
tcp    LISTEN   0       32      192.168.122.1:53          0.0.0.0:*     dnsmasq
...
tcp    LISTEN   0       250              [::]:3142           [::]:*     apt-cacher-ng
tcp    LISTEN   0       50               [::]:139            [::]:*     smbd
tcp    LISTEN   0       4096             [::]:111            [::]:*     rpcbind
tcp    LISTEN   0       511              [::]:80             [::]:*     nginx
tcp    LISTEN   0       5               [::1]:631            [::]:*     cupsd
...  

Here we can obtain the same information: There are two listening sockets owned by the nginx process, and they have the Local Address:Ports 0.0.0.0:80 and [::]:80. In otherwords, nginx is listening on port 80 on all IPv4 and IPv6 addresses.

How Listening Addresses Work

The 0.0.0.0:80 and [::]:80 example above demonstrates the simplest case for a listening address.

With those two listening sockets, if anyone dials that computer from anywhere, literally anywhere, they will be able to connect on that port (:80). Of course, there are caveats. The listening computer may not be routable from everywhere. Or a firewall might be blocking the traffic. The firewall could be anywhere; on the dialing computer (client), the listening computer (server) or some equipment in-between.

At any rate, when they connect, the listening computer's operating system will notify the process attached to the listening socket, thus triggering the connection handler code to run inside the server application process that created the listening socket.

A listening address like 127.0.0.1:80 or [::1]:80 is more complicated. It's a real address, so when a process asks the operating system to listen on this address, the OS doesn't mess around: it listens exactly only on that port on that address. This can quickly become a problem, in fact I would say this is probably the most common source of pain that humans encounter when it comes to listening addresses.

The address can only be dialed via the network that it is a part of. (See the List of IPv4 Networks that are part of the IPv4 specification).

127.0.0.1 is in the 127.0.0.0/8 loopback network range, so in order to be able to dial it, you have to be on the loopback network. And by definition, the loopback network only has one computer on it: The computer who dialed! This network is used exclusively for a computer to dial itself.

💀 If you are morbidly curious what the heck the /8 in 127.0.0.0/8 is, + Show

It's called a "netmask". Overall, the whole 127.0.0.0/8 format is called CIDR Block notation.

The netmask controls how many bits of the address are used to represent the address of the network itself, and how many bits are used to represent the different addresses inside the network. Decreasing the CIDR netmask number by one doubles the number of addresses inside the network, and increasing it by one cuts the number of addresses inside the network in half. There can only be one /0 network, 0.0.0.0/0 which contains every possible IPv4 address. And there are as many /32 networks as there are possible addresses, with one address per network.

Might seem like a whole heck of a lot of addresses to allocate for the sole purpose of a computer dialing itself, but hey, at least if we ever see an address starting with 127, we know its for sure a local address.

I don't know how many 127.0.0.0/8 addresses I've seen besides 127.0.0.1. Maybe one? Two? If you use these addresses for something and you've been able to make use of the massive number of them that are avaliable, leave a comment. I would love to hear about it.

So if your application is listening on 127.0.0.1:80 and [::1]:80, then it will only accept connections coming from the same computer.

The same effect can be achieved for any other network by listening on whatever address your computer has on that network. For example, my home LAN's network is 192.168.0.0/24 and my server's address on that network is 192.168.0.24:

root@odroidxu4:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
4: enx001e0636dda6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:1e:06:36:dd:a6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.24/24 brd 192.168.0.255 scope global dynamic noprefixroute enx001e0636dda6
       valid_lft 3117sec preferred_lft 3117sec
    inet6 fe80::21a3:f80e:be6a:e049/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

So if I told an application running on that server to listen on 192.168.0.24:80, it would only be dial-able by computers on my home LAN.

Typically this behaviour is only utilized by choosing to listen on either 0.0.0.0 and :: (all addresses) or listen on 127.0.0.1 and ::1 (listen for local connections only).

But it's important to choose the right one. You may wish to listen for local connections only for security reasons, or you may wish to listen on all addresses because you have something you want to share with the world.

So if your server software isn't responding, you can use ss or netstat to figure out if it's listening at all and if so, what listening addresses it's using. It's possible that it may be listening on the wrong address:

• Listening only on the loopback address while someone from outside is trying to connect
  • This is a common problem with docker containers. Processes running inside docker containers shouldn't listen on the loopback address; if they do they will only be accessible from inside that container.
• Listening only on IPv6 while someone is trying to connect via IPv4
• Listening only on IPv4 while someone is trying to connect via IPv6

Dual-Stack Sockets

I just said that sometimes the application might be

Listening only on IPv6 while someone is trying to connect via IPv4

It's worth noting that this is supposed to never happen.

Linux specifically uses something called "dual stack sockets" by default, controlled by the net.ipv6.bindv6only sysctl. This means that if you create a listener for IPv6 connections on all addresses, it should "automagically" listen for IPv4 connections on all addresses as well. This behaviour appears to be fairly well-standardized across operating systems.

In my previous nginx example, nginx is actually going out of its way to avoid using a dual-stack socket; in its default configuration

# Default server configuration
#
server {
	listen 80 default_server;
	listen [::]:80 default_server;

it very clearly specifies to the OS that it wants to create one socket for IPv4 and one socket for IPv6.

However, if I write my own program in Golang or Node.js to create a listening server:

📄 main.go

package main

import (
	"net/http"
)

func main() {
	http.ListenAndServe(":8080", nil)
}

📄 index.js

"use strict";

const express = require("express");

const app = express();

app.listen(8080);

Then I'll only see one listening socket, even though I should be able to dial the app via both IPv4 and IPv6:

forest@thingpad:~$ sudo ss -tlpn 
State     Recv-Q  Send-Q  Local Address:Port  Peer Address:Port  Process  
...
LISTEN    0       511     *:8080              *:*                node
...

forest@thingpad:~$ curl 192.168.0.46:8080
<!DOCTYPE html>
<html lang="en">...blahblahhtmlblah...

forest@thingpad:~$ curl '[::1]:8080'
<!DOCTYPE html>
<html lang="en">...blahblahhtmlblah..

It's worth noting that netstat on linux won't provide any indication that this is happening, it will display the dual-stack listening socket as if it was a normal IPv6 listening socket. However, the newer program ss will differentiate between the two. Also, ss uses the unambiguous IPv6 address format [::]:8080 over the messier :::8080. So ss is definitely preferred.

Compare the netstat output:

forest@thingpad:~$ sudo netstat -tlpn 
Active Internet connections (only servers)
Proto Recv-Q Send-Q  Local Address  Foreign Address  State    PID/Program name    
...            
tcp        0      0  0.0.0.0:80     0.0.0.0:*        LISTEN   1567/nginx               
tcp6       0      0  :::80          :::*             LISTEN   1567/nginx      
...
tcp6       0      0  :::8080        :::*             LISTEN   35378/node    
...

Versus the ss output:

forest@thingpad:~$ sudo ss -tlpn 
State     Recv-Q  Send-Q  Local Address:Port  Peer Address:Port  Process                                                                                                                                               
...
LISTEN    0       511     0.0.0.0:80          0.0.0.0:*          nginx       
LISTEN    0       511     [::]:80             [::]:*             nginx
...
LISTEN    0       511     *:8080              *:*                node
...

It's rare, but sometimes something will go wrong with a dual-stack socket and we'll wish that we could split it up like how nginx does. I've usually encountered this when using uncommon software or in a funky operating system environment. Some examples:

• I was unable to get gpsd to listen on both IPv4 and IPv6 at the same time on my Odroid single board computer.
  • To solve this, I ended up configuring it to listen on IPv4 only, and I wrote a simple UDP proxy server to listen on IPv6 and forward packets to it.
• Once upon a time I encountered an issue where Caddy Server appeared to have dual-stack-related listening issues on an alpine linux capsul.
• A fellow cyberian reported issues with dual-stack listening sockets on an Ubuntu WSL virtual machine. The issue only occurred when dialing the host.docker.internal from inside a docker container.
  • To solve this, the cyberian switched the server application from listening on [::]:1337 to listening on 0.0.0.0:1337.

Theory, Meet Practicality

When running a server application that opens a listening port, you might not even be able to specify which address it listens on. Or even if you could specify the address, you may not be able to specify multiple addresses or specify the details of how the application asks the operating system to listen. (For example, what value it assigns to a socket option flag like IPV6_V6ONLY)

In fact, operating systems or environments can be slightly different in how they handle these listen syscalls. Programing language standard libraries / runtimes differ in how they make the syscalls to request listening sockets as well. So even as an application developer, you may not always have the luxury of specifying exactly how your application listens for connections.

However, despite any limitations that may exist, understanding some of these details should help your server software efforts bear fruit and may help you avoid frustration / disapointment. By understanding the inner workings of any limitation you may run up against, you open up the possibility of side-stepping it with a quick hack or even directly addressing/eliminating it. Happy hosting!

No, I'm not changing this into a food blog, but I have been taking a break from working on my software projects for a couple months now while I'm trying to take care of my health.

I had fun with this little side project and it's easy for me to use this blog as a format for sharing it.

When I first tried coffee and started drinking it regularly in my early 20s, I loved the drug but hated the taste. Back then I bought a bottle of caffeine powder online and started simply eating the powder instead of making coffee. It still tasted bad, but "in a pinch" (quite literally) it'd get the job done. If you've never tasted pure caffeine, its very bitter, kind of similar to the taste of the white part on the inside of a grapefruit rind.

While I was in the dormitory crudely shoveling white powder into my mouth, I was thinking about how I could make a better energy drink. Back then I couldn't afford energy drinks, but even if I could, I wouldn't have wanted to drink em all the time, they're usually syrupy sweet, even worse than a typical soda. I was never liked artificial sweeteners either. I wanted something other than a 40 gram mound of sugar to mask the bitterness.

I know that salt can help cut down bitter flavors in food, and I also figured that if I went for a flavor that's "supposed" to be bitter, it would jive better with the caffeine. Back then I guess all of this was firmly in daydream territory, I had other stuff going on and no time for caffeine adventures in the kitchen.

Fast forward 8 years: Since then, a lot has changed:

• I got over my aversion to coffee and now I drink it regularly
• Some poor kid decided to shovel a whole spoonful of caffeine powder into his mouth and ended up with a heart attack, so the United States Food and Drug Administration banned the retail sale of anhydrous caffeine powder
• I was given homebrewing equipment as a gift from a stranger, and
• Before I ever got around to using the homebrewing equipment, I realized that I needed to quit drinking alcohol 😬
• I guess La Croix somehow became cool, everyone started drinking White Claw, and the craze even expanded to energy drinks

So what was I to do?

I never forgot about my dream of a tasty but unsweetened or semi-sweet energy drink. Like most of my daydreamed creations, someone else beat me to it. This little number on the right is produced by everyone's favorite energy drink monopolist Monster Beverage. I drank a couple of these recently and I can say I think my theory about the bitter flavor is correct. I tried thier cucumber lime flavor as well as the grapefruit, and you can definitely taste the caffeine in the cucumber one while its almost unnoticeable in the grapefruit.

That said, energy drinks are ridiculously expensive for seemingly no reason ( I mentioned they're ruled by a monopolist, right? ) and this one's no exception. Oh, and it's a bit weak for my taste at 160mg. Over the course of a day, I want the full caffeine experience, and for me that weighs in at 300mg or more. (About 16oz of coffee)

This was the straw that broke the camel's back; I had to finally try to concoct my own, and fate had granted me the tools to do so:

The Caffeine Question

When I was discussing this at first, the question came up,

will caffeine kill or inhibit the yeast?

I did some research on this, and came across a paper on this exact subject:

Investigating the caffeine effects in the yeast Saccharomyces cerevisiae...

A small excerpt:

...the IC50 was taken as the value that causes a 50% reduction of the maximal growth rate...
[the IC50 was] 8 mM (1.55 mg ml−1) for caffeine

I was a little bit confused by these units, 8 mM here means 8 millimolar concentration in other words, 0.008 moles of caffeine per liter. 1.55 mg ml−1 appears to be just a fancy way of saying 1.55mg per ml or 1.5 grams per liter. If you do the math, it checks out: 0.008 moles of caffeine weighs about 1.5 grams.

So in other words, in order to slow the yeast by about 50%, the caffeine concentration has to be 1.5 grams per liter. But coffee is only about 0.6 grams per liter, and that's the concentration I was planning on targeting, so I figured it should be fine.

Deciding on a Process

At first, I set out to learn a little bit about homebrewing. I knew that it's possible to brew a beer with a negligible alcohol content, and I knew that one can brew low-alchohol sweet and fizzy ginger beer at home without any fancy equipment, but I didn't know how to approach the project.

I wanted to come up with a process which would yield a relatively shelf-stable product that's carbonated and ready to drink without having to be stored in the fridge.

The wild yeast "probiotic active cultures" ginger-beer-type process seemed like a non-starter, because it requires regular care and feeding and has to stay in the fridge.

I also knew that if you bottle an actively fermenting liquid like beer, and it has a lot of sugar in it, the yeast will create so much pressure inside the bottle that it'll explode on the shelf. Or even worse, as soon as you try to pick it up and open it!

At first, I thought my goal would be to figure out how to calibrate my mixture somehow so that the yeast would stop growing at the right time in order to prevent the bottle from exploding. But it turns out that's simply not a thing; it can't be done. Yeast love to eat, and there's no trivial way to stop them as long as more sugar is available in the bottle:

• Non-Fermentable Sugars exist, but these are those same weird sugars and artificial sweeteners I was trying to avoid in the first place.
• Pasteurization is an option, but it takes a lot of work and involves risk:
  • Heating the bottles to 180°F after they have already been carbonated and pressurized by the yeast may cause them to explode if you aren't careful
  • If you don't manage to kill all the yeasts they will definitely explode the bottle later
  • Heating the bottles will reduce the carbonation overall since all the CO2 will come out of solution and hang out in the air pocket at the top of the bottle until it cools down

In the end, I decided to eschew both of these options and choose simplicity: I would only include enough sugar to carbonate the drink, no more, no less. The yeast should eat all the sugar, and in the end, the drink should be essentially zero-sugar. From various eyewitness accounts and "priming sugar" calculators online, I estimated this amount of sugar at approximately 160 grams for a 5 gallon batch.

Essentially the process would be exactly the same as the process for making beer, except my "wort" (fermentable liquid) has much less sugar in it, with caffeine taking its place, and I'm skipping the bulk fermentation process and going directly to bottling shortly after "pitching" the yeast (adding the yeast to the mixture after it cools down).

Coming up with a Recipe

aka "The Full Grapefruit Experience"

I created the following diagram to explain my overall thesis here. Each ingredient is located around the area of the grapefruit that it's supposed to "taste like" or "represent" when it's in the finished drink.

I'm no food scientist or chemist, and to be honest I didn't try very hard in this part. The core idea was to take a grapefruit apart, swap the pith (white part of the rind) out for caffeine, and put it back together as a drink.

• The oil from the rind is important because that's what's going to make it smell like grapefruit. Smell is a huge part of taste.
• It's going to be bitter like the white pith no matter what we do because of the caffeine, so might as well embrace that
• I can't make the drink sweet like the inside of the grapefruit, but I can at least try to make it sour and tart.
• I think some gatorade-style salts (electrolytes) might help balance the flavor.

At the same time, I tried to include ingredients that have health benefits while also complimenting what I'm trying to do with the flavor.

In the end, my recipe for twelve 750ml bottles looked like this:

• 9 liters of water
• 1 and a half cups of grapefruit juice
  • contains about 37 grams of sugar for the carbonation process
• 50 grams of grapefruit "oleo saccharum" (sugar syrup citrus rind oil extract)
  • extracted from the rinds of two large grapefruits
  • contains about 45 grams of sugar for the carbonation process
• total 82 grams of sugars
  • It's important to have no more than about 8.5 grams of sugar per liter of liquid (about 160 grams per 5 gallons), otherwise the bottles may overcarbonate and explode
• Half a packet of champagne yeast
• 6 grams of caffeine
  • 500mg per bottle, 250mg per serving
  • extracted from the contents of 30 200mg caffeine gel caps
• 15 grams of calcium ascorbate
  • 1000mg of vitamin c per bottle, 500mg per serving, similar to the "Emergen-C" drink mix
  • About 25% daily value of calcium per bottle
• 38 grams of potassium citrate
  • 1000mg of potassium per bottle (~30% daily value per bottle)
  • potassium citrate has an interesting tart taste
  • My partner had a potassium deficiency recently
• I was going to add some plain citric acid or lemon juice for a more sour/acidic flavor, but I forgot.
• I wanted to add zinc and magnesium salts for a more complete mix of electrolytes but I got lazy and left them out.
  • I think a single $2 lemon flavor magnesium-based laxative drink from walgreens would have been perfect here.

These pills are cut with rice flour, so to separate the caffeine from the rice flour, I emptied them into a bowl and then poured hot water over it. The caffeine will dissolve and the rice flour will sink to the bottom. Most of the liquid can be decanted off the top, and the rest can be grabbed by pouring the slurry thru a coffee filter.

In order to try to extract the most out of the grapefruit peel, I threw the left over syrupy peel into a blender with some hot/warm water, then poured the results through a coffee filter. (Plastic blender users: Be careful! Too hot of water can crack your blender carafe!)

The following video shows my brew after adding all the dry ingredients and pouring in the blended-grapefruit-peel-water. Some solids had formed overnight in the water based peel extract, but they dissolved later.

For the grapefruit juice, I simply took the red grapefruit flesh out and chucked it in the blender:

After adding the grapefruit juice and the grapefruit peel extract syrup, I heated the mixture to 180°F to kill off any bacteria or yeasts in there.

Once I felt confident it was sterilized, I covered the pot and moved the it to the sink so it could chill faster in a water bath. I wanted it to cool down below 80°F before I added the yeast.

Finally, after the yeast is thoroughly mixed in, it's time to bottle!

These bottles are currently fermenting and will need some time before they are ready ⁠— about 3-7 days, assuming everything goes to plan.

...

...

...

72 hours later

After 72 hours I transferred one of the bottles to the fridge, then the next day I cracked it open to try a glass:

I have to say, it tasted pretty bad on its own at first. It doesn't taste like it spoiled or something went wrong with the fermentation, it's just... The flavor is way off, especially when compared to a commercial product like the True North drinks.

The fermentation/carbonation seemed to work, although I think it's not quite done yet after 3 days. The pressure was pretty low when I opened the bottle, so I expect it will continue fermenting and becoming more and more bubbly over the coming days.

I will say, however, I succeeded in one aspect: It's potent, but it doesn't taste like caffeine. The caffeine made it through the process intact; I can feel its effects after drinking a 10oz glass, however the grapefruit flavors and citrate salts masked the bitterness of the caffeine perfectly.

If I was to do this again, I might try:

• less salts and more acid
• filtering the liquid after cooling but before bottling
  • EDIT: for the 2nd batch I ended up filtering the ingredients with a coffee filter instead of trying to filter the entire batch. See the comment below for the results!!
• I could consider pasteurizing it after sufficient carbonation has been achieved, this would allow me to add more sugar and make it semi-sweet
• I might do some research on the citrus peel oil and if its possible to make it stay in an emulsion somehow EDIT: Nope, the floaters were caused by solids that should have been filtered out, not by oils.

epilogue

I ended up making a simple syrup out of a cup of sugar and a cup and a half of water + the left-over "oleo-saccharum" grapefruit peel extract. Adding a spoonful or two of this to each serving improves the flavor dramatically.

Having the fresh citrus peel syrup in it makes it "smell right" and taste a lot better. Most of my friends that tried the final version of the drink with the syrup included seemed to like it, describing it as "refreshing", "interesting", and "strong".

EDIT: I ended up making 2nd and 3rd batches, see my comments below! Second batch ended up looking a lot better, with no floaters in it, and I think it tastes better too. For the third I experimented with a new flavor, cherry-lime.

It gets fully carbonated after about a week, honestly I was very satisfied with how the carbonation turned out. As the brew aged in the bottles, it also started to get more of a sort of "dry" taste that I remembered from extra-dry wine or cider, which used to be my favorite, so obviously I was pretty happy about that. I don't know how long it will stay good in the bottles as I've always drank it all before two months elapsed 😇 but I think it tastes best after at least 2 weeks at room temp.

This post is part of a new experiment; I'm toying with the idea of writing a concise and approachable "folk textbook" covering the conceptual basics of what I do; using Linux and other operating systems like MacOS and Windows, home routers, networking software, servers, virtual machines, linux containers, etc. I'm starting out by writing a few drafts or tidbits in this style as blog articles.

I actually started out on this path when I was writing on a different subject, but I got stuck halfway through: I wanted to mention processes and process exit codes, but I couldn't come up with any high-quality primary sources to cite. As far as I can tell, neither wikipedia nor the linux man pages (linux manual) contain a concise definition of what a process is including it's common features, i.e. things that every process has / things that every process can do.

I know I've mentioned linux a lot here, but I'd like to emphasize that this article is about processes in all major desktop and mobile phone operating systems, including Windows, and the similarities that they have.

Folks on the cyberia.club matrix server suggested an operating systems textbook...

forest

what would you consider a primary source for this information ?

what is a process exit code / what's the anatomy of a process, what things does a process have / what does it do?

I can't find anything on for example the man pages 😦

Argh. this is hilarious to me that this crucial conceptual information is not even written down in a public online place outside of anecdotes

Starless

A comp sci textbook, perhaps?

forest

its easy to find textbook pdfs online but they are kinda crap for what I want because

• the information is presented in book form and IMO no one wants to read like that any more
• you can't make a hyperlink to a section in a PDF afaik
• they aren't good for linking to anywasy because they aren't permanent because of copyright claims

... but what I really wanted was something I could link to. So...

here weeee gooooo! (If you notice any factual innacuracies in this article, please let me know or leave a comment below!)

Table of Contents

• Overview of Processes
• The Typical "Create Process" Options
  • Which Program to Run
  • Program Arguments
  • Standard Input Stream
  • Standard Output Stream
  • Standard Error Stream
  • File Descriptors (stdin, stdout, stderr continued)
  • Environment Variables
  • Working Directory
  • Run as User
• Typical Features of a Running Process
  • Process ID (PID)
  • User ID (UID)
  • Group ID (GID)
  • Currently Open Files (File Descriptors / File Handles)
  • Currently Open Sockets (Network Connections or Listening Servers)
  • Signals (SIGINT, SIGTERM, etc)
• After a Process Exits
  • Exit Code

Overview of Processes

All typical operating systems (MacOS, Windows, Linux, BSD, and others) have a notion of a "Process", and there are far more similarities than differences between the ways that these different OSes implement processes.

When you launch a program (For example, by clicking on that program in the Start Menu (Windows), or Dock (MacOS / Ubuntu Linux), you are instructing the operating system on your computer to launch a new process in order to run the program you clicked on.

There are two other common ways to launch a process on a computer:

1. In an interactive shell or "command line" — cmd.exe (Windows) or bash/zsh (Linux/MacOS) — every command that you run creates a process.
2. If you are writing your own program, it's almost certain that the programming language you are using includes some process.Start() functionality in its standard library. Such a function will spawn a new process in the operating system, typically a subprocess of the process calling the function.

Programs running as processes on your computer may perform arbitrary computations, but in order to make themselves useful they'll also need to interact with your computer hardware. For example, to respond to mouse and keyboard input, display pixels on the screen, read and write files, and connect to other computers on the internet.

In order to increase stability, security, and performance, all modern operating systems separate processes from the hardware and from each-other.

Process A is not allowed to read or write data used by Process B.

Any given process is not allowed to directly touch the screen, disk, or network. Instead, it must ask the operating system to do those things on its behalf. This is called a "system call" or "syscall".

Once started, a process will run either until the loaded program calls the exit() syscall, or until the process is forcefully killed by the operating system.

When it comes to creation of new processes, the syscalls involved, fork() and exec(), are much more complex. Typically, programmers interact with these syscalls through a more simplified process API in their language of choice.

The Typical "Create Process" Options

When we wish to start a new process, whether we do it by clicking on an application icon, running a command, or in our code, there are many settings being applied to that new process. Depending on how the process is launched, it may or may not be possible for you to control all of these settings, but it's important to understand that they always can be specified, you just have to figure out how.

Which Program to Run

First, we need to tell the operating system which executable file (program) we want it to run.

In many cases, like when running a command inside a shell, we can simply provide the name of a program, and the full file path of that program's executable file will be automatically resolved for us via the PATH environment variable.

🤔 Example: In my Linux bash shell, when I type apt moo, the shell

1. looks into the PATH variable
   • The value of PATH looks something like this: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2. searches each folder mentioned in the PATH variable
   • It's a list of folders separated by colons (:)
     • /usr/local/sbin
     • /usr/local/bin
     • /usr/sbin
     • /usr/bin
     • /bin
3. resolves to the first file called apt inside one of those folders
   • you can see this in action with the which command on MacOS/Linux or where command on Windows:
     • which apt → /usr/bin/apt
     • where wmic → C:\Windows\System32\wbem\WMIC.exe
4. Spawns a new process running the /usr/bin/apt executable binary file with moo as the first and only program argument, invoking the "Super Cow Powers" apt easter egg:

forest@thingpad:~$ apt moo
                 (__) 
                 (oo) 
           /------\/ 
          / |    ||   
         *  /\---/\ 
            ~~   ~~   
..."Have you mooed today?"...

Program Arguments

Arguments are meant to tell the program what to do or how to run, the name "argument" comes from math terminology for functions, for example, in f(x, y), x and y are called the arguments of the function f.

In this case, f is our program, and x is the first argument, y is the second argument.

ℹ️ NOTE: when writing program code or scripts, the "Which Program to Run" and "Program Arguments" information is often combined into a single list or array called arguments, args, or argv. So args[0] will be the program, and args[1] will be the 1st argument.

(argv is short for Argument Vector, it's called this because in C programming, arrays are commonly called "vectors")

Arguments are strings, that means they must contain text, and any section of text is a valid argument.

In shells like bash and cmd.exe on Windows, space is a syntax character; arguments are automatically separated by spaces. So if you want to pass an argument that CONTAINS spaces, you must wrap it in quotation marks to denote that it is a single string.

Consider this bash script file named myscript.sh:

#!/bin/bash

echo ""
echo "the currently running program is: $0"
echo "the first argument is: $1"
echo "the second argument is: $2"

Here are two examples where I ran this script in order to demonstrate
arguments:

$ ./myscript.sh i like eggs

the currently running program is: ./myscript.sh
the first argument is: i
the second argument is: like

$ ./myscript.sh "i like eggs"

the currently running program is: ./myscript.sh
the first argument is: i like eggs
the second argument is: 

Standard Input Stream

Standard Input, often called stdin, is a stream of data that the process can read from. No encoding is specified, so it's a stream of arbitrary bytes. It could be text, or it could be something else.

stdin is usually used by multi-purpose programs like grep and sed which can process another program's output. This is colloquially called "piping".

Standard Output Stream

Standard Output, often called stdout, is a stream of data that the process can write to. stdout is raw "anything goes" bytes just like stdin.

stdout is commonly used for program output. By default, when running a program in an interactive shell, that program's standard output will be displayed in the shell.

The output does not have to be text and it does not have to be displayed to the screen; it could be raw binary data and it could be piped to another program or written to a file. For example on linux, the gzip program will output the compressed bytes directly.

For programs which do not have any direct output, stdout is commonly used for logging. Log messages may be written to either stdout, stderr, or both.

⚠️ NOTE: Many command line programs will behave differently when they detect that they are running inside an interactive shell, so they may format their output differently.

As an example, the ls (list directory) command on linux will display files separated with spaces in interactive mode, and separated with newlines otherwise:

~/example$ ls
file1  file2  file3
~/example$ echo "$(ls)"
file1
file2
file3

Standard Error Stream

Standard Error, often called stderr, is a second stream of data that the process can write to. It's raw "anything goes" bytes just like stdout.

Unlike stdout, stderr should always output text and should be used exclusively for logging. By default, it is always displayed in the shell and never piped into another program's input.

File Descriptors (stdin, stdout, stderr continued)

Sometimes we might want to change the shell's default "always display stderr, use stdout for piping" behavior. For example when we want to write the standard error stream to a log file, or we want to process it further before it's displayed.

In the bourne shell sh and its derivatives like bash and zsh, the stream-redirection operator > can be used to modify this behavior by specifying both the source and the destination as file descriptors.

Each process has its own numbered list of file descriptors, and by convention, the first three are always the standard io streams:

file descriptor 0: stdin
file descriptor 1: stdout
file descriptor 2: stderr

As a concrete example, some non-standard programs write thier help (usage instructions) to stderr, but we want to search for lines containing a specific string inside that output:

nmcli device help | grep wifi

This won't work because grep will only see the (empty) stdout from nmcli (Network Manager CLI). But if we redirect nmcli's stderr (file descriptor 2) into its stdout (file descriptor 1), then grep can filter the output for display:

nmcli device help 2>&1 | grep wifi

The & shell syntax is specifying that &1 is a destination file descriptor.

As another example, say we wanted to write the stdout and stderr streams to two separate files, for example, we have a web server that logs its status information and errors to stderr, while it writes its access log to stdout:

caddy run 2>/var/log/caddy-status.log >/var/log/caddy-access.log

Environment Variables

Environment variables are similar to program arguments, with two major differences:

1. Instead of being a simple list of strings, environment variables are named; each variable has a name and a value, and you can't have two variables with the same name
2. Environment variables are always inherited from the parent process unless otherwise specified

Some environment variable names like PATH and HOME are fairly standard across all operating systems, while others are specific to a single program.

⚠️ NOTE: If you run a program and then change an environment variable, your change will not affect any currently-running processes! Each process runs in its own environment, and that environment is copied from the parent when the process starts.

On Linux, you can run the env command to display all currently specified environment variables. On windows (PowerShell), you would run dir env:.

Here's a trimmed-down example from my computer:

$ env
SHELL=/bin/bash
DESKTOP_SESSION=ubuntu
HOME=/home/forest
USERNAME=forest
LANG=en_US.UTF-8
USER=forest
PATH=/home/forest/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

ℹ️ NOTE: you can set or override an evironment variable for a process you launch in a shell by including the environment variable before the command.

MacOS/Linux: MY_VAR=example mycommand
Windows: cmd /C "set MY_VAR=example && mycommand"

Working Directory

Every process has a "Working Directory", or a folder that it's running inside. If that process tries to read or write a file with a relative path, the path will be relative to the working directory. For example if the file path specified was myfile.txt, and the working directory was /Users/example/, then the file read or written would be /Users/example/myfile.txt.

The working directory is typically inherited from the parent process which spawned it, however the working directory can be changed if desired.

Run as User

By default, any new process will run as same user who was running the parent process. If you want to run a process as a different user, you will have to use a special tool that allows this.

• MacOS/Linux:
  • sudo (super-user do) & su (substitute user)
• BSD/Linux:
  • doas (do as)
• Windows
  • Right Click → Run As Administrator...
  • runas

If you are using a background process manager like systemd or OpenRC on Linux, launchd on MacOS, or Windows Services, you'll be able to specify the user to run the process as inside your configuration file.

Typical Features of a Running Process

Process ID (PID)

The process ID is a unique integer ID number that the operating system assigns to each process.

You can access the list of processes along with thier IDs and other information with:

MacOS/Linux: ps (Processes) command
Windows: Task Manager / tasklist command

User ID (UID)

The user ID is a unique integer that the operating system uses to keep track of which user is running this process. There's also the EUID ("effective user id") for binaries like sudo which are configured to be able to change the user they are running as.

Group ID (GID)

Similar to User ID, but for group membership.

Currently Open Files (File Descriptors / File Handles)

The operating system keeps track of which files have been opened, and by which process. On Windows, you may not be able to delete a file or unmount a disk if a process is still accessing it.

ℹ️ NOTE: to list currently open files on your computer:

MacOS/Linux: lsof (List Open Files) command
Windows: Sysinternals Process Explorer application or the openfiles command

Currently Open Sockets (Network Connections or Listening Servers)

Similar to files, the operating system also keeps track of network connections (sockets) to websites or services (local or on the internet) as well as listening sockets, aka servers, that each process has created.

Sockets may use either TCP (Transmission Control Protocol) or UDP (User Datagram Protocol). TCP guarantees that the connection is reliable and data is transmitted and recieved in proper order. When data cannot be transmitted or recieved, TCP will return a read error or write error. UDP offers higher efficiency and lower latency, but it doesn't guarantee proper ordering of transmitted data, and it can't provide feedback when transmission fails.

Besides TCP/UDP, there are two main types of sockets,

1. "Connected/Waiting" sockets aka "client" sockets

• Applications like web browsers create client sockets to connect to web servers and other types of servers on the internet (or to servers running on your computer or local network)
• "Client" sockets have two port numbers:
  • source port
    • a randomly generated large number like "41386"
  • destination port
    • the port number that the client connected to, for example port 80 for http or port 443 for https

2. "Listening" sockets aka "server" sockets

• Web servers themselves create listening sockets in order to "open a port" for incoming client connections
• "Server" sockets only have one port number, the listening port

ℹ️ NOTE: to list sockets on your computer:

Linux: netstat (Network Statistics) command / ss (Socket Statistics) command
MacOS: lsof -i -P -n | grep LISTEN (list open files, then search for LISTEN)
Windows: Sysinternals TCPView or the netstat command

Signals (SIGINT, SIGTERM, etc)

Processes can send signals to each-other. Typically this is used to shut other processes down when they are no longer wanted.

For example, say you ran a process in your shell, it's taking forever to complete, and you don't want it to run any more. You might press Ctrl + c on your keyboard, which will instruct your terminal to send a terminate signal or SIGTERM to the currently executing process.

If you've ever done this, you might already know that the Ctrl + c isn't guaranteed to end a process, some processes may ignore your request; perhaps they've crashed, they're in turn waiting for something else before they think they can exit(), etc.

In this case, you may escalate to kill -SIGKILL <PID> which instructs the operating system to remove the process immediately with no questions asked.

After a Process Exits

Exit Code

The exit code is an integer is commonly used to denote the reason why the program exited.

An exit code of 0 should always mean that the program succeeded or ran normally, while an exit code of 1 means that the program failed or crashed. Other numbers may be used and given specific meanings on a program-by-program basis.

If you ever see an exit code of -1, that may mean that the program never actually exited, perhaps it was SIGKILLed by the operating system instead.

ℹ️ NOTE: You can access the exit code of the previous command in the shell. It's $? in bash (MacOS/Linux) and %ErrorLevel% in cmd.exe (Windows)

If you notice any factual innacuracies in this article, please let me know or leave a comment below!

This is a cross-post with the cyberia computer club blog.

Previous post in this series: Capsul Rollin' Onwards with a Web Application

What is this?

If you're a wondering "what is capsul?", see:

https://capsul.org

Here's a quick summary of what's in this post:

• cryptocurrency payments are back

• we visited the server in person for maintenance

• most capsuls disks should have trim/discard support now, so you can run the fstrim command to optimize your capsul's disk. (please do this, it will save us a lot of disk space!!)

• we updated most of our operating system images and added a new rocky linux image!

• potential ideas for future development on capsul

• exciting news about a new server and a new capsul fork being developed by co-op cloud / servers.coop

What happened to the cryptocurrency payment option?

Life happens. Cyberia Computer Club has been hustling and bustling to build out our new in-person space in Minneapolis, MN:

https://wiki.cyberia.club/hypha/cyberia_hq/faq

Hackerspace, lab, clubhouse, we aren't sure what to call it yet, but we're extremely excited to finish with the renovations and move in!

In the meantime, something went wrong with the physical machine hosting our BTCPay server and we didn't have anywhere convenient to move it, nor time to replace it, so we simply disabled cryptocurrency payments temporarily in September 2021.

Many of yall have emailed us asking "what gives??", and I'm glad to finally be able to announce that

"the situation has been dealt with",

we have a brand new server and the blockchain syncing process is complete, cryptocurrency payments in bitcoin, litecoin, and monero are back online now!

     -->   https://capsul.org/payment/btcpay   <--    

Also, fun fact, the BTCPay server is currently located at my house, but I didn't have a good way to expose it to the internet, so I simply pointed the btcpay.cyberia.club domain at my greenhouse subdomain

root@beet:~# nslookup btcpay.cyberia.club
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
btcpay.cyberia.club	canonical name = forest.greenhouseusers.com.
Name:	forest.greenhouseusers.com
Address: 68.183.194.212

Then published the server to the internet thru greenhouse!

root@beet:~# greenhouse tunnel https://btcpay.cyberia.club to tcp://localhost:3000
Now applying new tunnel configuration...
  - waiting for underlying services to start
  - creating threshold tunnels
  - testing threshold tunnels
  - configuring caddy
  - waiting for caddy to obtain https certificates from Let's Encrypt
  - final testing
Your tunnel was configured successfully!

That one time capsul was almost fsync()'d to death

Guess what? Yall loved capsul so much, you wore our disks out. Well, almost.

We use redundant solid state disks + the ZFS file system for your capsul's block storage needs, and it turns out that some of our users like to write files. A lot.

Over time, SSDs will wear out, mostly dependent on how many writes hit the disk. Baikal, the server behind capsul.org, is a bit different from a typical desktop computer, as it hosts about 100 virtual machines, each with thier own list of application processes, for over 50 individual capsul users, each of whom may be providing services to many other individuals in turn.

The disk-wear-out situation was exacerbated by our geographical separation from the server; we live in Minneapolis, MN, but the server is in Georgia. We wanted to install NVME drives to expand our storage capacity ahead of growing demand, but when we would mail PCI-e to NVME adapters to CyberWurx, our datacenter colocation provider, they kept telling us the adapter didn't fit inside the 1U chassis of the server.

At one point, we were forced to take a risk and undo the redundancy of the disks in order to expand our storage capacity and prevent "out of disk space" errors from crashing your capsuls. It was a calculated risk, trading certain doom now for the potential possibility of doom later.

Well, time passed while we were busy with other projects, and those non-redundant disks started wearing out. According to the "smartmon" monitoring indicator, they reached about 25% lifespan remaining. Once the disk theoretically hit 0%, it would become read-only in order to protect itself from total data loss. So we had to replace them before that happened.

We were so scared of what could happen if we slept on this that we booked a flight to Atlanta for maintenance. We wanted to replace the disks in person, and ensure we could restore the ZFS disk mirroring feature.

We even custom 3d-printed a bracket for the tiny PCI-e NVME drive that we needed in order to restore redundancy for the disks, just to make 100% sure that the maintenance we were doing would succeed & maintain stability for everyone who has placed thier trust in us and voted with thier shells, investing thier time and money on virtual machines that we maintain on a volunteer basis.

Unfortunately, "100% sure" was still not good enough, the new NVME drive didn't work as a ZFS mirroring partner at first ⁠— the existing NVME drive was 951GB, and the one we had purchased was 931GB. It was too small and ZFS would not accept that. f0x suggested:

[you could] start a new pool on the new disk, zfs send all the old data over, then have an equally sized partition on the old disk then add that to the mirror

But we had no idea how to do that exactly or how long it would take & we didn't want to change the plan at the last second, so instead we ended up taking the train from the datacenter to Best Buy to buy a new disk instead.

The actual formatted sizes of these drives are typically never printed on the packaging or even mentioned on PDF datasheets online. When I could find an actual number for a model, it was always the lower 931GB. So, we ended up buying a "2TB" drive as it was the only one BestBuy had which we could guarantee would work.

So, lesson learned the hard way. If you want to use ZFS mirroring and maybe replace a drive later, make sure to choose a fixed partition size which is slightly smaller than the typical avaliable space on the size of drive you're using, in case the replacement drive was manufactured with slightly less avaliable formatted space!!!

Once mirroring was restored, we made sure to test it in practice by carefully removing a disk from the server while it's running:

While we could have theoretically done this maintenance remotely with the folks at CyberWurx performing the physical parts replacement per a ticket we open with them, we wanted to be sure we could meet the timeline that the disks had set for US. That's no knock on CyberWurx, moreso a knock on us for yolo-ing this server into "production" with tape and no test environment :D

The reality is we are vounteer supported. Right now the payments that the club receives from capusl users don't add up to enough to compensate (make ends meet for) your average professional software developer or sysadmin, at least if local tech labor market stats are to be believed.

We are all also working on other things, we can't devote all of our time to capsul. But we do care about capsul, we want our service to live, mostly because we use it ourselves, but also because the club benefits from it.

We want it to be easy and fun to use, while also staying easy and fun to maintain. A system that's agressively maintained will be a lot more likely to remain maintained when it's no one's job to come in every weekday for that.

That's why we also decided to upgrade to the latest stable Debian major version on baikal while we were there. We encountered no issues during the upgrade besides a couple of initial omissions in our package source lists. The installer also notified us of several configuration files we had modified, presenting us with a git-merge-ish interface that displayed diffs and allowed us to decide to keep our changes, replace our file with the new version, or merge the two manually.

I can't speak more accurately about it than that, as j3s did this part and I just watched :)

Looking to the future

We wanted to upgrade to this new Debian version because it had a new major version of QEMU, supporting virtio-blk storage devices that can pass-through file system discard commands to the host operating system.

We didn't see any benefits right away, as the vms stayed defined in libvirt as their original machine types, either pc-i440fx-3.1 or a type from the pc-q35 family.

After returning home, we noticed that when we created a new capsul, it would come up as the pc-i440fx-5.2 machine type and the main disk on the guest would display discard support in the form of a non-zero DISC-MAX size displayed by the lsblk -D command:

localhost:~# sudo lsblk -D
NAME DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
sr0         0        0B       0B         0
vda       512      512B       2G         0

Most of our capsuls were pc-i440fx ones, and we upgraded them to pc-i440fx-5.2, which finally got discards working for the grand majority of capsuls.

If you see discard settings like that on your capsul, you should also be able to run fstrim -v / on your capsul which saves us disk space on baikal:

welcome, cyberian ^(;,;)^
your machine awaits

localhost:~# sudo lsblk -D
NAME DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
sr0         0        0B       0B         0
vda       512      512B       2G         0

localhost:~# sudo fstrim -v /
/: 15.1 GiB (16185487360 bytes) trimmed

^ Please do this if you are able to!

You might also be able to enable an fstrim service or timer which will run fstrim to clean up and optimize your disk periodically.

However, some of the older vms were the pc-q35 family of QEMU machine type, and while I was able to get one of ours to upgrade to pc-i440fx-5.2, discard support still did not show up in the guest OS. We're not sure what's happening there yet.

We also improved capsul's monitoring features; we began work on proper infrastructure-as-code-style diffing functionality, so we get notified if any key aspects of your capsuls are out of whack. In the past this had been an issue, with DHCP leases expiring during maintenance downtimes and capsuls stealing each-others assigned IP addresses when we turn everything back on.

capsul-flask now also includes an admin panel with 1-click-fix actions built in, leveraging this data:

https://git.cyberia.club/cyberia/capsul-flask/src/commit/b013f9c9758f2cc062f1ecefc4d7deef3aa484f2/capsulflask/admin.py#L36-L202

I acknowledge that this is a bit of a silly system, but it's an artifact of how we do what we do. Capsul is always changing and evolving, and the web app was built on the idea of simply "providing a button for" any manual action that would have to be taken, either by a user or by an admin.

At one point, back when capsul was called "cvm", everything was done by hand over email and the commandline, so of course anything that reduced the amount of manual administration work was welcome, and we are still working on that today.

When we build new UIs and prototype features, we learn more about how our system works, we expand what's possible for capsul, and we come up with new ways to organize data and intelligently direct the venerable virtualization software our service is built on.

I think that's what the "agile development" buzzword from professional software development circles was supposed to be about: freedom to experiment means better designs because we get the opportunity to experience some of the consequences before we fully commit to any specific design. A touch of humility and flexibility goes a long way in my opinion.

We do have a lot of ideas about how to continue making capsul easier for everyone involved, things like:

1. Metered billing w/ stripe, so you get a monthly bill with auto-pay to your credit card, and you only pay for the resources you use, similar to what service providers like Backblaze do. (Note: of course we would also allow you to pre-pay with cryptocurrency if you wish)

2. Looking into rewrite options for some parts of the system: perhaps driving QEMU from capsul-flask directly instead of going through libvirt, and perhaps rewriting the web application in golang instead of sticking with flask.

3. JSON API designed to make it easier to manage capsuls in code, scripts, or with an infrastructure-as-code tool like Terraform.

4. IO throttling your vms: As I mentioned before, the vms wear out the disks fast. We had hoped that enabling discards would help with this, but it appears that it hasn't done much to decrease the growth rate of the smartmon wearout indicator metric. So, most likely we will have to enforce some form of limit on the amount of disk writes your capsul can perform while it's running day in and day out. 80-90% of capsul users will never see this limit, but our heaviest writers will be required to either change thier software so it writes less, or pay more money for service. In any case, we'll send you a warning email long before we throttle your capsul's disk.

And last but not least, Cybera Computer Club Congress voted to use a couple thousand of the capsulbux we've recieved in payment to purchase a new server, allowing us to expand the service ahead of demand and improve our processes all the way from hardware up.

(No tape this time!)

Shown: Dell PowerEdge R640 1U server with two 10-core xeon silver 4114 processors and 256GB of RAM. (Upgradable to 768GB!!)

Can I help?

Yes! We are not the only ones working on capsul these days. For example, another group, https://coopcloud.tech has forked capsul-flask and set up thier own instance at

https://yolo.servers.coop

Thier source code repository is here (not sure this is the right one):

https://git.autonomic.zone/3wordchant/capsul-flask

Having more people setting up instances of capsul-flask really helps us, whether folks are simply testing or aiming to run it in production like we do.

Unfortunately we don't have a direct incentive to work on making capsul-flask easier to set up until folks ask us how to do it. Autonomic helped us a lot as they made thier way through our terrible documentation and asked for better organization / clarification along the way, leading to much more expansive and organized README files.

They also gave a great shove in the right direction when they decided to contribute most of a basic automated testing implementation and the beginnings of a JSON API at the same time. They are building a command line tool called abra that can create capsuls upon the users request, as well as many other things like installing applications. I think it's very neat :)

Also, just donating or using the service helps support cyberia.club, both in terms of maintaing capsul.org and reaching out and supporting our local community.

We accept donations via either a credit card (stripe) or in Bitcoin, Litecoin, or Monero via our BTCPay server:

https://cyberia.club/donate

For the capsul source code, navigate to:

https://git.cyberia.club/cyberia/capsul-flask

As always, you may contact us at:

support@cyberia.club

Or on matrix:

#services:cyberia.club

For information on what matrix chat is and how to use it, see https://cyberia.club/matrix

Previous post in this series: Capsul Rollin' Onwards with a Web Application

That computer can browse the web, but "the web" (or, web users around the globe) typically can't browse files or web applications running on that computer. Publishing a server on the internet takes work, costs money, and requires maintenance and upkeep. It can be especially difficult if you want to use your computer, not rent someone else's. If you live in a dormitory or if you get internet access via wifi from your neighbor or via tethering to your phone, you may not be able to perform the necessary modifications to your router's configuration even if you knew how.

Greenhouse provides an easy way to make one or more computers into servers that anyone in the world can connect to securely & reliably, regardless of where or how the servers are connected to the internet.

Greenhouse is being designed from the ground up as a trustless service, that is, you don't have to trust me or whoever's running the service to keep your data secure — it's designed so it can't access your data in the first place.

For more information about why I am building this, see greenhouse.server.garden and The "Pragmatic Path" 4-Year Update: Introducing Greenhouse!

You may also check out the source code at git.sequentialread.com/forest/greenhouse

Previous post in this series: Greenhouse Development Update 4 - September

Last time I mentioned that my todo list for releasing the alpha version had grown from 8 items to 24. Well, by now it's grown to 38! I've conducted preliminary usability tests, a pre-alpha release, adopted a mascot, and implemented telemetry.

It's time to, as the kids say,   ship it

Right now the service is free for early-adopters. Not free so I can hook you on it & extract capital from you later, free because I haven't gotten around to implementing payments yet 😛

When I do, I'm planning on billing $0.01 per gigabyte of bandwidth, which is approximately what you would pay if you went with a provider like DigitalOcean and used about half of the bandwidth they allocated to your VPS (Virtual Private Server). However, unlike the other providers, Greenhouse will have no minimum payment, so you only pay for the bandwidth you use. I estimated that this will make it up to a hundred times cheaper than a traditional VPS, practically free, for many use cases.

When compared to a similar cloud service, PageKite:

Of course, these use cases and greenhouse prices are estimates. The actual bandwidth used may depend a lot on your server configuration, how you built your site/service, and of course how popular it is. As an anecdotal example, during the last month, my server used about 80GB of bandwidth.

Greenhouse also offers security and data ownership by default, plus a much easier setup process which works on Mac, Windows, and Linux, affording both a point and click interface and a CLI (Command Line Interface).

Support for hosting email servers hasn't landed in Greenhouse yet, but it is planned for the future. Unfortunately it will have to cost more, as it requires a dedicated IPV4 address. On the bright side, it'll be bundled with quite a lot of monthly bandwidth.

Besides that, I don't feel like writing any more description or introduction to what you are about to witness. You can check out greenhouse.server.garden and some of my previous blog posts if you want that. Without further ado, enjoy the demo video, and please give me feedback if you decide to try out the service!

"As a user, I want my server to be online 😀"

7 minutes 42 seconds

Help me make self-hosting ✨ radically easier ✨!!

⚗️🧪 greenhouse-alpha.server.garden

That computer can browse the web, but "the web" (or, web users around the globe) typically can't browse files or web applications running on that computer. Publishing a server on the internet takes work, costs money, and requires maintenance and upkeep. It can be especially difficult if you want to use your computer, not rent someone else's. If you live in a dormitory or if you get internet access via wifi from your neighbor or via tethering to your phone, you may not be able to perform the necessary modifications to your router's configuration even if you knew how.

Greenhouse provides an easy way to make one or more computers into servers that anyone in the world can connect to securely & reliably, regardless of where or how the servers are connected to the internet.

Greenhouse is being designed from the ground up as a trustless service, that is, you don't have to trust me or whoever's running the service to keep your data secure — it's designed so it can't access your data in the first place.

For more information about why I am building this, see greenhouse.server.garden and The "Pragmatic Path" 4-Year Update: Introducing Greenhouse!

You may also check out the source code at git.sequentialread.com/forest/greenhouse

Previous post in this series: Greenhouse Development Update 3 - August

Next post in this series: 🥳 Greenhouse Enters Alpha Test Phase!! 🎉

September is ending. Wake up!!!11!!!!!!!!

No, seriously, as I write this, it's about 22 minutes until midnight, that time when the calendar rolls over to October 1st.

I spose I've delayed writing a Greenhouse update post for September until now because I was secretly hoping I could get the alpha version released this month. Well, it's definitely not happening at this point. However, at least I can say it's not for a lack of trying.

This month I tried to make a short list of tasks that I'd have to complete before releasing an alpha version of greenhouse. To me, alpha means that it's relatively unstable, it probably doesn't have very many users, and it's free; payments have not been implemented yet.

I believe I started out with a list of about 8 items. By the time I had completed half of them, the list had grown to 17 items 😛

Here is my todo list today:

- [.] alpha test
  - [X] view logs from CLI
  - [X] view logs from desktop app
  - [X] delete tunnels from desktop app
  - [X] greenhouse webapp external domain support (greenhouse-alpha.server.garden)
  - [X] greenhouse webapp scheduled tasks: external domain validation, rebalance
  - [X] greenhouse webapp auto-start cloud-side and daemon
  - [X] greenhouse webapp general cleanup
  - [X] greenhouse internal how-to pages (using-your-own-domain-name-with-greenhouse)
  - [ ] greenhouse webapp alpha test invite/auth system
  - [X] desktop app listening port chooser
  - [X] desktop app file chooser
  - [ ] greenhouse daemon / cli / desktop app development environment quickstart docs
  - [X] Desktop app build / packaging Linux
  - [.] Desktop app build / packaging Windows
  - [.] Desktop app build / packaging MacOs
  - [X] curl | sh installation method for daemon/CLI on Linux
  - [ ] Desktop app installs background service on first run on MacOS
  - [X] Some sort of installation method for daemon/cli on windows?
  - [ ] Telemetry/monitoring: greenhouse
  - [ ] Telemetry/monitoring: greenhouse-desktop
  - [ ] Telemetry/monitoring: greenhouse-cli
  - [ ] Telemetry/monitoring: greenhouse-daemon
  - [ ] Telemetry/monitoring: threshold

By now it looks like it's grown to 24 items, but on the bright side, only 8 of them aren't started yet.

I wasn't sure just how quick-and-dirty I wanted the alpha version to be. For example, does "alpha test version" mean it only works on Linux? Does it mean the Windows and Mac implementations are there, but super hacky?

I spent a lot of my time this month grappling with these decisions. Often times I would investigate what work would actually be required & attempt to decide based on the technical reality I was facing, rather than a theoretical estimate.

I think I've been erring on the side of "do it right to begin with" to some extent, at least where the amount of work associated with that decision is minimal.

Here's a brief list of decisions I've made so far:

• The linux installer for the alpha version will just be a shell script, aka the world-famous curl ... | sudo sh
• The linux shell script installer will only support systemd-based linux distributions, however there will be reasonable DIY instructions for folks who don't use systemd.
• There are two different installers for linux:
  • The shell script which installs the background service and command line tool
  • A source repo or debian package for the desktop application.
• Windows and Mac are fundamentally different from linux in terms of use case: On these operating systems, the desktop application is king, it is not optional. Everything else is based on the desktop application installation.
• For the desktop application, I will leverage the fman build system (fbs) Python/QT framework and its installer support. I decided on this framework early on, because I wanted an alternative to electron, something which could produce a more lightweight application experience while at the same time minimizing the pain associated with cross-platform development.
  • On linux, it generates .deb and .rpm packages, potentially others as well. I haven't looked into "PPA"-style ways of hosting these yet, for now they will be direct downloads.
  • On Windows, it creates an executable Nullsoft Scriptable Installation System NSIS installer. This is actually quite nice, as far as I can tell this installer gives a great user experience when implemented correctly, but it also doesn't limit you in terms of what you can do.
  • On MacOS, it creates a .dmg disk image containing a .app package. The good news is, mac users love this and they know exactly what to do with it. The bad news is it's not an installer, so we don't get an opportunity to run a script while it's being installed. I will probably end up building the daemon and CLI installer into the desktop app on MacOS; it won't install until the first time you launch the app.
• You will be able to use your own domain name with the alpha version, although this feature is going to be sort of hacky for now. Most of the advanced DNS-related features will have to come later.

Here is a screenshot of the work-in-progress alpha version of the web application:

I think I got a lot done this month, for one thing, I finished the CLI (command line interface). Here's a breif overview of what it looks like:

$ greenhouse

Usage: greenhouse COMMAND [...]

Commands: status, register, tunnel, ls, rm, logs

To read the instructions for a specific command, you can run for example:

greenhouse help register
greenhouse tunnel -h
greenhouse ls --help

Advanced users may wish to configure this cli with environment variables. 
For instructions on how to do that, run 'greenhouse help config'

$ greenhouse status

Greenhouse Daemon:
    Server Name: thingpad
    Configured Tunnels: 1

    Caddy Server Status:
        Enabled: true
        Running: true
        PID: 411094
        Up for 3m29s
        Health Check: not implemented yet

    Threshold Tunnel Client Status:
        Enabled: true
        Running: true
        PID: 411088
        Up for 3m29s
        Health Check: not implemented yet
 
Greenhouse Account:
    Email Address: forest.n.johnson@gmail.com
    Allocated Port Range: 10000-10019

    Authorized Domains: 
        - greenhouse-alpha.server.garden
        - forest.greenhouseusers.com

    Client States:
        greenhouse_internal_node:
             Current: ClientConnected
            Previous: ClientClosed
        thingpad:
             Current: ClientConnected
            Previous: ClientClosed

$ greenhouse tunnel

Error: expected at least 3 arguments for tunnel command, but only saw 0:

Usage: greenhouse tunnel LISTEN_URL to LOCAL_URL
   OR: greenhouse tunnel --json '{...}'

  Open a new tunnel. 
  LISTEN_URL supports the protocol schemes https://, http://, tls://, and tcp://.
  LOCAL_URL supports the protocol schemes http://, tcp://, and file://
	For details on the format that --json accepts, run 'greenhouse help json'

  e.g:
  greenhouse tunnel https://www.cli-user.greenhouseusers.com to http://localhost:80
  greenhouse tunnel https://files.cli-user.greenhouseusers.com to file:///home/cli-user/Public
  greenhouse tunnel tcp://:10014 to tcp://localhost:22
  greenhouse tunnel --json '{"domain": "cli-user.greenhouseusers.com", "public_port": 443, ...}'
		
$ greenhouse tunnel https://nginx.forest.greenhouseusers.com to http://localhost:80

Now applying new tunnel configuration...

  - waiting for underlying services to start
  - creating threshold tunnels
  - testing threshold tunnels
  - configuring caddy
  - waiting for caddy to obtain https certificates from Let's Encrypt
  - final testing

Your tunnel was configured successfully!

I also made serious headway on getting the software built and deployed for all platforms. At this point, windows is most of the way there. ¡Bienvenido a IIS!

I managed to get the greenhouse-daemon (called greenhouse-background-service on windows 😇) installing as a windows service via the NSIS installer, and it even creates a special service user for that as well. I mainly wanted to do that to isolate greenhouse for its own protection; make it a bit harder for malware running on the windows machine to swipe the encryption keys that are used for TLS, for example. I'm still fairly green on file permissions / security on Windows, so if any of yall are Windows experts, I'd love to hear your thoughts in the comments!

There are still quite a few bugs and issues on Windows, but just the fact that I was able to get everything going via a single .exe installer is exciting!

I also got the desktop application running on MacOS, in the picture below I am using the Sosumi snap to virtualize MacOS on my linux workstation.

Here's the roadmap as it stands today. The main changes since last time; the cross platform installer feature was moved from the beta phase to the alpha phase, the CLI was finished, and a lot of other features went from "MVP implemented" to "ship it".

See the previous post in this series: Greenhouse Development Update 3 - August

Next post in this series: 🥳 Greenhouse Enters Alpha Test Phase!! 🎉

For more information about why I am building this, see greenhouse.server.garden and The "Pragmatic Path" 4-Year Update: Introducing Greenhouse!

You may also check out the source code at git.sequentialread.com/forest/greenhouse