Through the rabbit hole I found a blank screen

With my home lab, which I affectionately named SINGULARITY, I mostly interact with it via ssh over my home network. However, I've been tweaking BIOS settings a lot so I hooked the PC up to my wifes' keyboard/video/mouse and was doing a lot of reboots.

But then I stopped tweaking and went back to using the machine as a node on my network. But as I sat there doing my work I started feeling anxious and I didn't know why. I felt as though someone were watching me. Try as I may, I couldn't shake this anxious feeling. It turns out that the login prompt was showing and the monitor wasn't turning off! Mercy me!

I run the machine headless, which means its running at a lower runlevel, so I had no GUI tools or screensaver to set.

Incidentally, I enabled the low runlevel by issuing this simple command:

sudo systemctl set-default

If I wanted the GUI back, I'd just do:

sudo systemctl enable --force
sudo systemctl set-default

But I digress.

I poked around and didn't find a direct solution. It did seem like the command I wanted to run was setterm -blank 1, which would blank the screen and put the monitor into power-save mode. This is exactly what i wanted but the command vomits if you set it in a remote terminal, like Windows Terminal:

$ setterm -blank 1
setterm: terminal xterm-256color does not support --blank

After a little experimenting, I found that when I'm logged directly into the terminal, $TERM is set to linux. So with this little bit in my .bashrc, I was able to run setterm -blank 1 only when I'm logged in locally:

if [ "$TERM" == "linux" ]; then
setterm -blank 1
elif [ -e /usr/share/terminfo/x/xterm-256color ]; then
export TERM='xterm-256color'
export TERM='xterm-color'

Great! Now when I login, a minute of inactivity blanks the monitor!

But the problem here was on reboot, I wouldn't be logged in. So I dived into login details more and found there's a file called /lib/systemd/system/getty@.service that specifies the command used for the login prompt:

ExecStart=-/sbin/agetty -o '-p -- \u' --noclear %I $TERM

Still, I didn't see a way to inject setterm -blank 1 in there. But agetty does take a -l parameter allowing you to specify a login program. So I created /bin/login2:

setterm -blank 1
/bin/login $*

That's so ugly I love it! I then invoked this piece of art from agetty:

ExecStart=-/sbin/agetty -l /bin/login2 -o '-p -- \u' --noclear %I $TERM

After a reboot, I found this did not work at all.

Why? because agetty blocks on awaiting login name. It queries for a login name and then passes it to /bin/login. So setterm -blank 1 within login2 does not get invoked until you enter the login name. Damn.

But wait, there's more! agetty also takes a -a argument, autologin name! So I descend further into the rabbit hole and create this beauty:

ExecStart=-/sbin/agetty -a nic -l /bin/login2 -o '-p -- \u' --noclear %I $TERM

This bypassed the username prompt, executed setterm -blank 1, but this left me with an awkward prompt for the password, and this filled me with even more anxiety than the original non-blanking screen!

I saw that /bin/login can take a -f argument to force a login. Documentation says "Do not perform authentication, user is preauthenticated." Hahahah, so of course I went ahead and added it. Fuck it, it's a home lab and this is only a security vulnerability for local access.

Now when I reboot, I basically get logged in automatically. But then, at this point, my .bashrc does the job of calling setterm -blank 1 so why the hell do I need /bin/login2 for?

Sooooo... I started thinking about if I can pass -f from the original agetty invocation and found that yes, this is possible. That's what the -o option is for. Now I can get rid of /bin/login2 and just have this:

ExecStart=-/sbin/agetty -a nic -o '-f -p -- \u' --noclear %I $TERM



Home Kubernetes Lab

Over the weekend, I created a home lab. I had an aging PC that had windows on it and I decided to wipe it away and install Ubuntu.

I created a home lab late last year on my main PC, but I didn't like having all of that infrastructure on a PC where I did general purpose programming and gaming. Right off the bat, Windows hypervisor and the VM's I'd need for a proper lab took up precious RAM. And there was also the matter of having the VM's up and running as nodes in my Kubernetes cluster, having Rancher running etc. It all seemed like extra baggage for my PC which I wanted to be as lean as possible and ready to handle large tasks (like running games) using all of the resources in the machine.

So this PC has 16GB of RAM and an i5 4-core processor.  It's unfortunate that I have 32GB of DIMMs installed on the box. My mother board says it supports 32GB of RAM but for some reason I can't get the BIOS to recognize it. Turns out to a CLI-only Ubuntu, this is all of the power I need. Anyway, figuring out how to use the other 16GB of ram is an Area of Research! The box has an old SSD I pulled from my old Apple Mac Pro and performance of everything is just lightning.

When it came to a hypervisor to use, I chose kvm. It seemed like the right choice for my needs and did't take a lot of know-how to get working. This link helped:

When it came to VM's to create, I used this Minimal CD image of Ubuntu: During the installation of each VM, I chose OpenSSH server only since that's all I needed. VM installation was still manual and a little time-consuming. At some point I hope to automate it, but since I only have 16GB of RAM, I'm only creating 203 VM's as Kubernetes worker nodes and that will be that.

A few useful installs:

  • sudo apt install libguestfs-tools
  • sudo apt install virtinst
  • sudo apt install libosinfo-bin
  • sudo apt install cpu-checker

I decided to use Rancher as the k8s cluster management layer since we use that at work and I figure it would be a good idea to get more familiar with it. Obviously the new server needed docker and this article was helpful in describing how to get docker to install properly on Ubuntu: It turns out just saying "apt install" is not the best choice. You need a few extra steps.

Rancher was up and running without much effort. Rancher runs in a container directly on the host, on a server I call SINGULARITY. I have two VM's called star01 and star02 which run as work nodes. star01 also runs etcd and the control plane. I'll probably create a new VM star03 soon so I can experiment with deploying various software in clusters which require 3+ nodes, but I can prob get by with 2-nodes for a while.

An open item is having star01/02 DNS available on my LAN without using a bridged network. I tried implementing the technique described here, but it didn't work on my first try. I'll have to try again. It would probably help if I understood NetworkManager and DNS better. More areas of research!

A few things I'm interested in learning about in the short term:

  • How to create a docker image that manipulates mounts on the host for external storage for configuration. Rancher does this and it's kind of magic to me right now.
  • Figure out how to have all of my kvm guest hostnames recognized on the host, if not my entire home LAN.
  • How to backup my Rancher and cluster configuration so I can restore it if I want to blow away my lab and start from scratch. This blog post is good documentation if I want to start over, but obviously I'd rather automate away the toil.

Some useful commands:

  • Make sure your kvm guests restart after host reboot: virsh autostart vmName
  • List kvm guests and their IP addresses: virsh net-dhcp-leases default
  • Dump network details: sudo virsh net-dumpxml default
  • Edit network details: sudo virsh net-edit default

Ciao for now.