Updating the Firmware on my Turing Pi 2 Boards

Wherein I update my Turing Pi 2 boards to a new firmware.

During the migration of my Homelab to a fleet of Raspberry Pi 4, I bought two Turing Pi 2 boards and put eight Raspberry Pi CM4 8GB into them. You can read more about my setup here.

The board has a nice Board Management Controller (BMC). It is an Allwinner SoC with 128 MB of RAM and 128 MB of flash for the OS. It’s running an embedded Linux distribution. This BMC implements a few interesting features:

• Turning power on/off for each individual node
• Connecting to the serial console of each of the nodes
• Flashing each of the nodes if it has internal storage, like Pi CM4 with eMMC

There is also an internal Ethernet switch chip, which connects the four nodes and two external 1GbE ports for external connectivity. Sadly, that chip is not really controllable yet, even with the newest firmware. So it’s still only usable as a dumb switch.

As originally delivered, the firmware was perfectly workable, I’ve been running my boards with it for two years. But it was also a bit problematic, as it provided a web UI without any authentication at all. And you could flash nodes via it and turn them on and off.

The v2 firmware I’m installing in this blog post has proper authentication, and adds support for a CLI tool which can be used to control the board remotely, in addition to the web UI.

The new version also adds support for the Turing Pi RK1. That’s an SBC based on the RK3588 SoC, developed by the same people who designed the Turing Pi 2 board. And I’ve got one of those laying around. But more on that later.

Firmware update

I was still on the original v1 firmware and needed to update to the most recent v2.1. One note: For some reason, their docs point to this part of their website from the docs, but the newer firmware versions are only available in their GitHub repo.

For the update, I followed these docs. While the new version of the firmware supports flashing via the web UI, this update from v1 to v2 needs to be done via the SD card.

But before I could get started, I had to solve the Homelab uptime problem. As I’m running a very serious operation here, I could of course not tolerate any downtime for any services. But at the same time, this update couldn’t be done online, so I would have to take down four of my CM4. Meaning 16 cores and 32 GB of RAM would be temporarily unavailable. And while my Homelab is intentionally designed with some slack, I didn’t have quite that much slack left. So I went to my Ceph hosts, specifically the largest, which has 32 GB of RAM. Which it currently does definitely not need. Here is what my three Ceph hosts look like under normal operation:

ceph1:
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests       Limits
  --------           --------       ------
  cpu                3 (75%)        0 (0%)
  memory             11412Mi (72%)  14624Mi (92%)
ceph2:
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests       Limits
  --------           --------       ------
  cpu                3400m (85%)    0 (0%)
  memory             11824Mi (77%)  15648Mi (102%)
ceph3:
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests       Limits
  --------           --------       ------
  cpu                3150m (39%)    0 (0%)
  memory             10276Mi (32%)  14012Mi (44%)

So at least going by the limits, two of them were almost full already. But ceph3 still had enough unused resources to hold a few Pods during the downtime.

So I removed the NoSchedule taint from it like this:

kubectl taint nodes ceph3 homelab/taint.role=ceph:NoSchedule-

With that, it would be allowed to run non-Ceph Pods.

I then drained the four CM4 on the first board:

kubectl drain --delete-emptydir-data --force --ignore-daemonsets worker1 worker2 worker3 worker4

And finally shut them all down:

ansible "worker1:worker2:worker3:worker4" -a "systemctl poweroff"

I will describe the effects and show some metrics about the shutdown’s impact on my cluster later in this post.

To prepare the update, I needed to put the firmware on an SD card. Here are the commands to do that:

wget https://github.com/turing-machines/BMC-Firmware/releases/download/v2.1.0/tp2-firmware-sdcard-v2.1.0.img
dd if=tp2-firmware-sdcard-v2.1.0.img of=/non-existant-path bs=1M status=progress

The of parameter should then point to the SD card. Point it to the device, e.g. /dev/sde, not /dev/sde1.

With that done, I opened the board’s schematics to hunt for the SD card slot:

The slot is entirely inaccessible while the board is mounted in a case.

So I had to do the one thing about Homelabbing I actually don’t enjoy that much: Touching hardware. I’m still thinking that at some point, I should click myself something roughly equivalent to my Homelab at a large cloud provider and see how expensive it really is.

Here is the board still in the case:

I was a bit apprehensive about taking the case out of the rack, to be honest. This particular case was mounted before I figured out how rack rails work. And I was pretty brutal with the ones for this case. I didn’t see any problem with taking it out. But I was a bit worried whether I would be able to put it back in again. Luckily, it all worked out at the end.

With the board removed from the case, I was finally able to access the SD card slot:

I’ve finally found the place to put the SD card. This was already a lot more work than I had initially thought. But from hereon out, everything went quite smoothly. I connected the board’s UART to my trusty USB-to-Serial adapter and launched minicom:

minicom -b 115200 -D /dev/ttyUSB0

After starting the board, I was greeted with this text on said console:

 _____ _   _ ____  ___ _   _  ____
|_   _| | | |  _ \|_ _| \ | |/ ___|
  | | | | | | |_) || ||  \| | |  _
  | | | |_| |  _ < | || |\  | |_| |
  |_|  \___/|_| \_\___|_| \_|\____|

This utility will perform a fresh installation of the Turing Pi 2 BMC firmware.

Note that this will ERASE ALL USER DATA stored on the Turing Pi 2 BMC, thus
restoring back to factory defaults. Do NOT proceed unless you have first backed
up any files that you care about!

If you wish to confirm the operation and proceed, either:
1) Type 'CONFIRM' at the below prompt
2) Press one of the front panel buttons (POWER or RESET), or the KEY1 button on
   the Turing Pi 2 board itself, three times in a row

If you are here in error, please remove the microSD card from the Turing Pi 2
board and reset the BMC.

Type "CONFIRM" to continue:

I typed CONFIRM here as instructed, but the flashing could also be started by pressing a button on the board itself 3x in quick succession. So a serial connection is not strictly necessary. The flashing went pretty quickly, in under a minute, with relatively little output:

INFO: Legacy Allwinner boot code has been found and erased
INFO: Legacy Allwinner boot code has been found and erased
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: Legacy Allwinner boot code has been found and erased
INFO: Legacy Allwinner boot code has been found and erased
[+] DONE: Please remove the microSD card and reset the BMC.

As instructed, I removed the SD card and rebooted the board. I was immediately greeted by some Linux boot output and then a login prompt. Everything had worked nicely.

I needed to do a few little adaptions to the configuration though. First, the MAC had changed, so I needed to update it in my DHCP static leases. At least the MAC is now static by default, instead of being generated anew every time the board boots, as with the previous firmware. Then I set the hostname in /etc/hosts and /etc/hostname and updated the root password. Finally, I created a non-root user:

• mkdir -p /home/myuser
• adduser -s /bin/bash -D myuser
• Then I went into /etc/shadow and replaced the ! with a * in the password field. No idea why I had run with the -D option initially.

Then I could switch to the user and enter a few SSH keys. Finally, I also had to change the SSHD configuration. Because by default, it allows password logins and logins of root. I added/changed the following configs:

• PubkeyAuthentication yes
• PermitRootLogin no
• PasswordAuthentication no
• AllowUsers myuser

The restart of SSHD was a bit tricky. There’s a script they tell you to use:

/etc/init.d/S50sshd restart

The problem is that that’s a shell script which first stops SSHD and then starts it again. As you might imagine, that doesn’t go too well when executed via SSH. While the stop action is still run, the terminal is disconnected after that, because SSH is gone. So the start command is never run.

But that’s it already. After those steps I was able to put the board back into the case and start with the next one.

With that done too, the only thing left was to add back the taint to my Ceph node so it’s no longer used for non-Ceph stuff:

kubectl taint nodes ceph3 homelab/taint.role=ceph:NoSchedule

Same command as removing the taint, just without the - at the end.

Results

Well, first of all: Most things still worked. Emphasis on most. The one thing which no longer seems to work is connecting to the node’s serial consoles. It just outputs garbage. And it seems that’s because the new BCMd opens the serial console devices as well. So you now can’t open them anymore when SSH’d into the board. Which really isn’t ideal.

But onto the good things. First, there’s been a rework of the UI. Here’s an example of the old UI:

In addition to this update of the UI, there’s now an HTTP API and a CLI tool for using it, the tpi tool. It can do all of the things the web UI can, and then some. For example, this command will show the serial console output of node 2:

tpi uart --host my.turingpi --user root --node 2 get

This functionality is the reason that the serial consoles are useless when you’re actually logged into the BMC, see this issue.

Finally: The new firmware has support for flashing and using the Turing RK1. An RK3588-based SBC that can be plugged into the Turing Pi 2 boards. I should be excited about it. I bought one of them a while ago, with 32 GB of RAM. It would be really cool to use these as an upgrade path for my Homelab. The RK3588 is a good chip and makes a nice Homelab host with 8 cores. But there’s exactly zero broad support for the chip in Linux. There is an Ubuntu that’s supposed to work, ubuntu-rockchip. But the maintainer stepped back from it about a year ago. So what am I supposed to do? Run a chip with an outdated kernel, running services accessible from the Internet? I don’t think so.

So very likely, I will keep it as a high-tech paperweight. What do you mean I sound miffed with my past self for dropping money on a piece of hardware with shaky software support? Naaah, not at all. 😒

How the Homelab reacted

Before ending this post, I’d like to do a quick section about what the Homelab looked like during the periods where I took one of the Turing Pi boards out of the cluster. Those Pi CM4 8GB modules in the two Turing Pi 2 boards are my main worker nodes. I’ve also got one more node, an older x86 SBC I keep around for apps which don’t support aarch64. So with taking out one of the boards, I lose 16 cores and 32 GB of RAM. And my Homelab didn’t really bat an eye at that. I’ve already got a bit of slack in the resources. The idea being that I can take out a few nodes for maintenance, e.g. reboots during the regular host updates, without having to take down the entire Homelab.

So there wasn’t much of a change in the k8s cluster and my apps. Temporarily allowing Pods onto my largest Ceph node took care of that. But there was a reduction in the overall power consumption of the Homelab.

This plot shows that one full board with 16 cores and 32 GB worth of Pi 4 consume somewhere around 20 - 25 W under normal conditions. But there was one effect I will have to think about a bit more:

In the plot, I switched off the four Pi CM4 on the board around 21:04, resulting in the initial drop from 156 W to about 146 W. Then, at about 21:16, I pulled the power plug from the PSU, resulting in another drop to 138 W. Which honestly looks a bit weird? By that time, the only thing powered on were two 120mm fans on pretty low revolutions and the BMC itself, which is a piddly Allwinner SoC and really shouldn’t eat almost 10 W. The only thing I can come up with is: Could this be the fact that the power supply is a 550 W unit? And it’s running at very low power draw and is terribly inefficient at that point?

Might warrant some more investigations.

And just in case I don’t find time for another post before it: Happy Christmas to all of you who celebrate. I hope you find at least a modicum of peace and quiet.