(FYI: I’m working on a pre-built SMP kernel and patched Gentoo tarball for the Wii U, so stay tuned for that.)

I was originally going to write a long post about how we got here, but then I decided that I don’t actually want to write that, so instead, I’m just going to list some of the things we (me and Quarky) learned while working on this.

1: The EXI Bootstub is Not Used in Wii U Mode.

The primary source for all of our knowledge on the Wii U’s CPU comes from one article written by Marcan of fail0verflow. It is the reason we know how to start up and take control of the primary CPU core, despite all of Nintendo’s safeguards, how to initialize several important CPU registers, and how to start up the secondary CPU cores. Because Marcan is a veteran console hacker, and because he was right about everything else, everyone (including us) took the information in this article as gospel; we assumed everything in it was true.

In order to start up the secondary CPU cores correctly, we need to know where they start executing code from, and according to Marcan, this location was a piece of hardware leftover from the Wii called the EXI bootstub.

Cores 1 and 2 boot with MSR[IP]=1, thus at the high vectors. The old Wii PPC EXI bootstub hardware is still present and controls the instructions read from there, so just perform the normal EXI bootstub configuration sequence from the Starbuck.

-Marcan, fail0verflow.

So, as per Marcan’s instructions, we wrote our initialization code to the EXI bootstub and started up the secondary cores, but they never did what we expected. Then, one day, as I was staring at a dump of the decrypted kernel ancast image in Ghidra, I realized that, when mapped to the known location of the ancast image in Cafe OS (the Wii U’s built-in OS), my dump extended past the high boot vector, where the EXI bootstub was supposed to be, so I scrolled down and lo and behold: there was a bootstub there! After discovering this, I tried writing our bootstub overtop that part of the ancast image and it worked! The secondary cores were finally behaving as expected!

Conclusion: In Wii U mode at least, the high boot vector (the one used by cores 1 & 2) is not mapped to the EXI bootstub: it’s mapped to MEM0 as part of the ancast image at address 0x08100100

PS: it turns out that developers behind emulators like Cemu and WUFE already knew that 0xFFE00000 (the location of the kernel ancast image) is mapped to MEM0. Marcan must have known this too, as the Cafe OS kernel immediately jumps to a location above 0xFFE00000 (using an absolute jump), and then starts up the secondary cores soon afterwards, and there is clearly no code present to initialize the EXI bootstub. I assume, therefore, his statement quoted above was either a mistake, or in reference to vWii mode (which in fairness was the focus of his article.)

2: CAR and/or BCR Control Cache Coherency

With the ability to run whatever code I wanted on the secondary cores, I was now able to send them into the Linux CPU initialization code. However, there was a problem: despite having proof that they were running, my debugging code was saying the secondary cores were not coming up at all! Specifically, I had a variable in RAM which the cores would write their ID* number to upon starting up. I also wrote some code to print out the value of that variable after starting each core. My code told me that the value of the variable wasn’t changing, but when I looked in /dev/mem afterwards, I saw that its value was exactly what I expected it to be if the cores were starting up. I eventually figured out this was due to cache coherency problems; cores 1 and 2 were coming online and writing to that value exactly as they were supposed to, but core 0 wasn’t seeing the change, either because the new value was stuck in the write cache of the secondary core, or because the read cache of the primary core wasn’t being flushed. Adding some dcbst instructions to the CPU init code and a cache flush to the debugging code seemed to fix the issue, but similar problems just kept cropping up.

Out of ideas and desperate for a solution, I once again turned to the fail0verflow article written by Marcan, and I noticed something: there were 2 unknown CPU registers mentioned we weren’t using: “CAR” and “BCR”. The article gave no indication of what they did, but I decided it couldn’t hurt to try initializing them, and so I wrote some code to do so using the same values Cafe OS did, as per the article. Once again, it worked! Magically, all of my cache coherency problems disappeared, and, after dealing** with some driver bugs, I was able to get the cores fully started and (somewhat)** working in Linux.

Conclusion: CAR and BCR, in some way, control the new cross-core cache coherency mechanisms in the Wii U CPU.

*By ID number I mean the core number, so core 1 would write “1” and core 2 would write “2”. I also had another debugging variable that told me how far into the process of starting up each core was getting, and this variable told me that the secondary cores were getting quite far in despite the kernel claiming they’d never started.

**What I’m referring to here was a bug in our IPI driver that caused the wrong interrupt to be triggered. Fixing that allowed the cores to come up successfully, but the IPIs still weren’t working and Quarky ended up having to completely redo the IPI implementation in order to make the system usable.

3: The Bit Order of the ICI Bits of SCR is Backwards.

This one was discovered by Quarky. As I said above, in order to get the system stable enough to use, she ended up having to completely redo the IPI implementation. While doing so, she learned about a much easier method to implement IPIs than the one we were previously been using: SCR already contained bits for sending interrupts between cores!

pain

WiiUBrew calls these “ICIs” instead of “IPIs”, but they’re the same thing*. By flipping one of these bits from off to on, you could send an interrupt to the corresponding core, which was exactly what we needed. However, while picking through Cafe OS to figure out exactly how these IPI bits worked, she discovered that the order of the bits listed on WiiUBrew was backwards. Bit 18 controls the IPI to core 2, not core 0, and bit 20 controls the IPI to core 0, not core 2.

Conclusion: The order of the ICI bits of SCR (bits 18-20) is backwards.

*ICI = Inter-core Interrupt. IPI = Inter-processor Interrupt. Linux always calls them IPIs.

Hopefully you found this information useful or interesting.

Up until recently, wiiulinux.net has been a collection of manually written HTML & CSS files, good old Web 1.0 style.

What the site used to look like. CC0.

However, a few weeks ago (at time of writing), I decided it would be a good idea to get with the times and migrate to WordPress, so it would be easier to update the site, add new information, and actually allow this site live up to its tagline. The Home of Linux on the Wii U. A few days ago, I finally committed to actually making it happen.

Problem number 1: I didn’t have MariaDB or PHP installed. But that’s an easy enough problem to fix; just install them, right? No. That’s because I’m running Gentoo. Unlike, say, Debian or Arch, which distribute ready to use binary packages, on Gentoo every package is compiled locally. In my case, that means compiling them on the Wii U’s 1.24 GHz G3-derived CPU. Making matters worse, Linux on the Wii U [at time of writing] is limited to using one core! Needless to say, compiling MariaDB and PHP on a single core 1.24 GHz G3 takes a while.

Problem number 2: NGINX. Because the Wii U is so comically under-powered, I settled on NGINX as my web server, because it has a reputation of being faster than Apache. The problem here is that running WordPress on NGINX is harder than doing so on Apache, though it is possible. One of the things you’ll need to make this work is to build of PHP with FPM support. However, Gentoo does not build PHP with FPM by default, and I had no idea I needed to change that because I didn’t read any “Setting up WordPress using NGINX” tutorials until after I installed PHP! So I had to tell Gentoo “Please enable FPM support,” and wait another 2 hours for PHP to build again.

By this point, a day and a half in, I was only just now ready to install WordPress. Well, almost ready. I first had to set up MariaDB and tweak my NGINX config, but that was all pretty straightforward. So, I downloaded WordPress, unzipped it into my new NGINX web root, and was just about to start setting it up when suddenly, I remembered, in the access log, I had seen people trying to access /wp-admin/setup-config.php?step=1 and I realized “Hey, maybe I oughta take the server off the internet while doing this.” After avoiding my Wii U becoming compromised, I proceeded to finish the WordPress setup uneventfully. And within 3 minutes of installing WordPress, I had broken it so badly I could no longer log in. Here’s what happened.

Problem number 3: SSL. I noticed all the links on the site pointed to the local IP address of the Wii U, which is a problem for a public website, so I poked around in the WordPress settings, and changed the page URL to https://www.wiiulinux.net. Upon doing so, I became unable to log in.

So, a bit of context: wiiulinux.net does not handle requests from the internet directly, but instead runs behind a reverse proxy. Because of this, I had never bothered to set up SSL on the Wii U side, as the reverse proxy handled it. This meant requests the Wii U always received HTTP requests. But I just told WordPress to expect HTTPS. What was happening was, WordPress saw my HTTP request, and redirected me to https://www.wiiulinux.net. The reverse proxy forwarded my request as HTTP, resulting in an endless loop that prevented me from logging in. However, even once my Dad figured out what was going on, it took us another hour to actually get SSL working on NGINX and get the problem solved. Unfortunately, this was not the end of our problems.

Problem number 4: NGINX again. Now that I had WordPress working again, I wanted to get to work making the site look nice. However, the page editor refused to load. I discovered upon closer inspection that this was caused by the server failing to find a bunch of important files! So, my Dad and I and set spent the next 3 hours trying to figure out why this was happening and what how to fix it. He was methodically thinking through the problem, I just threw ideas at the wall to see if that would fix it. Eventually, my Dad figured out that all we had to do was slightly rewrite 1 rule in the NGINX config.

#try_files $uri $uri/ /index.php?$args =404;

location / {
    try_files $uri $uri/ /index.php?$args;
}

Yep. That’s it. After making that change, the editor worked perfectly. By this point, it was near midnight, so I decided to go to bed and work on designing the site tomorrow. But WordPress had one last, headache inducing surprise in store for me.

Problem number 6: Designing a page with WordPress. For the header, I wanted to have 2 inwards facing Tux penguins, like I did before. That requires flipping one of the images, but I couldn’t just add transform: scaleX(-1); to the CSS one of the images because, as far as I can tell, there is no way to add custom CSS to a specific instance of a block in WordPress. You can only add it to every occurrence of that block on the whole site. Luckily, my Dad saved the day once again when he had the idea to just make a flipped version of Tux and have two separate, independent image blocks. Although this is not the last time WordPress’s design has caused me pain, for the sake of brevity I’ll spare you the details. Suffice to say, it’s been frustrating.

For all it’s many, many faults though, I do still believe the migration to WordPress is a positive step for wiiulinux.net. WordPress can provide a really nice experience, if you’re using it as intended; ie: for writing blog posts. Writing this post using WordPress has been way easier than it would have been had I written it manually. Thank you so much to my Dad, who was absolutely essential in helping me through this transition. Thank you for all your hard work and your kind, patient support while helping me. Here’s to a more active future for wiiulinux.net. Maybe it’ll finally live up to it’s hyperbolic tagline.

That’s all for now. Bye!