/diːˈlaːn(.)ˌʕaːzəbaːniː/

Web engine CI on a shoestring budget

2025-12-18T10:00:00+00:00

Servo is a greenfield web browser engine that supports many platforms. Automated testing for the project requires building Servo for all of those platforms, plus several additional configurations, and running nearly two million tests including the entire Web Platform Tests. How do we do all of that in under half an hour, without a hyperscaler budget for compute and an entire team to keep it all running smoothly, and securely enough to run untrusted code from contributors?

We’ve answered these questions by building a CI runner orchestration system for GitHub Actions that we can run on our own servers, using ephemeral virtual machines for security and reproducibility. We also discuss how we implemented graceful fallback from self-hosted runners to GitHub-hosted runners, the lessons we learned in automating image rebuilds, and how we could port the system to other CI platforms like Forgejo Actions.

This is a transcript post for a talk I gave internally at Igalia.

Let's talk about how Servo can have fast CI for not a lot of money.

Servo is a greenfield web browser engine. And being a web browser engine, it has some pretty demanding requirements for its CI setup.

We have to build and run Servo for many platforms, including three desktop platforms and two mobile platforms.

We have to run many, many tests, the main bulk of which is the entire Web Platform Tests suite, which is almost 2 million subtests. We also have several smaller test suites as well, like the WebGPU tests and the DevTools tests and so on.

We have to build Servo in many different configurations for special needs. So we might build Servo with the oldest version of Rust that we still support, just to make sure that still works. We might build Servo as a library in the same way that it would be consumed by embedders. We have to lint the codebase. We have to build with optimizations for nightly and monthly releases. We have to build with other optimizations for benchmarking work, and so on.

And as you'll see throughout this talk, we do this on GitHub and GitHub Actions, but GitHub Actions is a very frustrating CI service, and it has many baffling limitations. And as time goes on, I think Servo being on GitHub and GitHub Actions will be more for the network effects we had early on than for any particular merits of these platforms.

On GitHub Actions, GitHub provides their own first-party runners.

And these runners are very useful for small workloads, as well as the logic that coordinates workloads. So this would include things like taking a tryjob request for "linux", and turning that into a run that just builds Servo for Linux. Or you might get a try job request for "full", and we'll turn that into a run that builds Servo for all of the platforms and runs all of the tests.

But for a project of our scale, these runners really fall apart once you get into any workload beyond that.

They have runners that are free of charge. They are very tiny, resource constrained, and it seems GitHub tries to cram as many of these runners onto each server as they possibly can.

They have runners that you can pay for, but they're very very expensive. And I believe this is because you not only pay a premium for like hyperscaler cloud compute rates, but you also pay a premium on top of that for the convenience of having these runners where you can essentially just flip a switch and get faster builds. So they really make you pay for this.

Not only that, but using GitHub hosted runners you can't really customize the image that runs on the runners besides being able to pull in containers, which is also kind of slow and not useful all the time. If there are tools and dependencies that you need that aren't in those images, you need to install them every single time you run a job or a workflow, which is such a huge waste of time, energy, and money, no matter whose money it is.

There are also some caching features on GitHub Actions, but they're honestly kind of a joke. The caching performs really poorly, and there's not a lot you can cache with them. So in general, they're not really suitable for doing things like incremental builds.

So we have all these slow builds, and they're getting slower, and we want to make them faster. So we considered several alternatives.

The first that comes to mind are these third-party runner providers. These are things like Namespace, Warpbuild, Buildjet. There's so many of them. The caveat with these is that they're almost always... almost as expensive per hour as GitHub's first-party runners. And I think this is because they try to focus on providing features like better caching, that allow you to accumulate less hours on their runners. And in doing so, they don't really have any incentive to also compete on the hourly rate.

There is one exception: there's a system called RunsOn. It's sort of an off-the-shelf thing that you can grab this software and operate it yourself, but you do have to use AWS. So it's very tightly coupled to AWS. And both of these alternatives, they often lack support for certain platforms on their runners. Many of them are missing Windows or missing macOS or missing both. And RunsOn is missing macOS support, and probably won't get macOS support for the foreseeable future.

We considered offloading some of the work that these free GitHub hosted runners do onto our own servers with, let's call them like, "proxy jobs". The idea would be that you'd still use free GitHub hosted runners, but you'd do the work remotely on another server. The trouble with this is that then you're still using these free GitHub hosted runners, which take up your quota of your maximum concurrent free runners.

And it's also tricky to do this without losing access to the broader ecosystem of prebuilt Actions. So these Actions are like steps that you can pull in that will let you do useful things like managing artifacts and installing dependencies and so on. But it's one thing to say, okay, my workload is this shell script, and I'm going to run it remotely now. That's easy enough to do, but it's a lot harder to say, okay, well, I've got this workflow that has a bunch of scripts and a bunch of external Actions made by other people, and I'm going to run all of this remotely. I'm going to make sure that all of these Actions are also compatible with being run remotely. That's a lot trickier. That said, you should probably avoid relying too heavily on this ecosystem anyway. It's a great way to get locked into the platform, and working with YAML honestly really sucks.

So we could set up an entire CI service. There are some CI services like Jenkins, Bamboo... honestly, most CI services nowadays have support for built-in container orchestration. So they can spin up containers for each runner, for each job, autonomously. And while they can do containers, none of them have really solved the problem of virtual machine orchestration out of the box. And this is a problem for us, because we want to use virtual machines for that security and peace of mind, which I'll explain in more detail a little bit later. And seeing as we lack the dedicated personnel to operate an entire CI service and have that be on the critical path — we didn't want to have someone on call for outages — this was probably not the best option for us at the moment.

What we decided to do was set up some self-hosted runners.

These solve most of our problems, primarily because we can throw as much hardware and resources at these runners as we want. We can define the contention ratios.

And better yet, we can also customize the images that the runners use. By being able to customize the runner images, we can move a lot of steps and a lot of work that used to be done on every single workflow run, and do it only once, or once per build. We only rebuild the runner images, say, once a week. So that significantly saves... it cuts down on the amount of work that we have to do.

This is not just installing tools and dependencies, but it's also enabling incremental builds quite powerfully, which we'll see a little bit later on.

How much faster is this system with the self-hosted runners? Well, it turns out, quite a lot faster.

We've got this typical workflow that you use when you're testing your commits, when you're making a pull request. And if you kick off a tryjob like this, several of the jobs that we've now offloaded onto self-hosted runners are now taking 70, 80, even 90% less time than they did on GitHub hosted runners, which is excellent.

And even for the web platform tests, we found more modest time savings with the capacity that we've allocated to them. But as a result, even though the savings are a bit more modest, what's worth highlighting here is that we were able to go from twenty runners — we had to parallelise this test suite across twenty runners before — and now we only need three.

Some things that make our system unique and that we're pretty proud of — and these things apply to all versions of our system, which we've been developing over the last 12 to 18 months.

The first is that we build on top of the native CI service of the Git forge. So in this case, it's GitHub and GitHub Actions. It could be Forgejo and Forgejo Actions in the future, and we're working on that already.

We also want to give users more or less a transparent user experience, the idea being that users should not notice any changes in their day-to-day work, besides the fact that their builds have gotten faster. And I think for the most part, we have achieved that.

Our system supports all of the platforms that GitHub Actions supports for their runners, including Linux, Windows, and macOS, and we could even support some of the other platforms that Forgejo Actions supports in the future, including BSD.

We have the ability to set up a job so that it can try to use self-hosted runners if they're available, but fall back to GitHub hosted runners if there's none available for whatever reason, like maybe they're all busy for the foreseeable future, or the servers are down or something like that. We have the ability to fall back to GitHub hosted runners, and this is something that was quite complicated to build, actually. And we have a whole section of this talk explaining how that works, because this is something that's not actually possible with the feature set that GitHub provides in their CI system.

It's secure enough for a large public project like Servo, where we don't necessarily know all of our contributors all that well personally. And this is in large part because we use virtual machines, instead of just containers, for each run and for each job. My understanding is that it is possible to build a system like this securely with containers, but in practice it's a lot harder to get that right as a security boundary than if you had the benefit of a hypervisor.

And of course it is all completely self-hosted, which makes it about as cheap as it gets, because your operating expenses are really just the costs of bare metal compute.

Now, how cheap is that? Well, in our deployment in Servo, we spend about 300 EUR per month on servers that do general-purpose runners, and these handle most of our workload.

If we were to compare that to if we were running on GitHub-hosted runners, there'd be almost an order of magnitude increase in costs, somewhere like 1400 EUR to over 2000 EUR per month if we were doing the same work on GitHub-hosted runners.

There are also significant increases if we went with third-party runner providers as well, although not quite as much.

But in truth, this is actually kind of an unfair comparison, because it assumes that we would need the same amount of hours between if we were running on self-hosted runners or if we were running on GitHub hosted runners. And something that you'll see throughout this talk is that this is very much not the case. We spend so many fewer hours running our jobs, because we have to do so much less work on them. So in reality, the gap between our expenses with these two approaches would actually be a lot wider.

There are three ways to use our CI runner system.

The first one is the way that GitHub designed self-hosted runners to be used. The way they intend you to use self-hosted runners is that when you define a job, you use this runs-on setting to declare what kind of runners your job should run on. And you can use labels for GitHub hosted runners, or you can define arbitrary labels for your own runners and select those instead. But you have to choose one or the other in advance. And if you do that, you can't have any kind of fallback, which was a bit of a drawback for us, especially early on. One benefit of this, though, is that you get the natural ability to queue up jobs. Because if, at the time of queuing up a new job, if there's no self-hosted runners available yet, the job will stay in a queued state until a self-hosted runner becomes available. And it's nice that you essentially get that for free. But you have no ability to have this kind of graceful fallback.

So we built some features to allow you to do graceful fallback. And how this works is that each of the servers that operates these runners has a web API as well. And you can hit that web API to check if there are runners available and reserve them. Reserving them is something you have to do if you're doing graceful fallback. But I'll explain that in more detail a bit later on. And in doing so, because you have a job that can check if there are runners available, you can now parameterize the runs-on setting and decide "I want to use a self-hosted runner, or a GitHub hosted runner". It's unclear if this is going to be possible on Forgejo Actions yet, so we may have to add that feature, but it's certainly possible on GitHub Actions.

Now, the downside of this is that you do lose the ability, that natural ability, to queue up jobs, and I'll explain why that is a bit later. But in short, we have a queue API that mitigates this problem, because you can hit the queue API, and it can have a full view of the runner capacity, and either tell you to wait, or forward your request once capacity becomes available.

Some things that you can do with our system, that you can't do with GitHub hosted runners. One of them is check out the repo significantly faster.

Something about GitHub Actions and how it works is that, if you run a job, you run a workflow in a repo, you don't actually get a checkout of the repo. You don't get a clone of the repo out of the box, unless you explicitly add a step that does a checkout. And this is fine for the most part, it works well enough for most users and most repos.

But the Servo repo has over 130,000 files across 6,000 directories, and that's just the tracked files and directories. And as a result, even if we use things like shallow clones, there's just kind of no getting around the fact that cloning this repo and checking it out is just slow. It's unavoidably slow.

And it's especially slow on Windows and macOS, where the filesystem performance is honestly often pretty poor compared to Linux. So we want to make our checkouts faster.

Well, what we can do is, we can actually move the process of cloning and checking out the repo from the build process, and move that into the image build process. So we only do it once, when we're building the runner images.

Then what we can do is go into the jobs that run on self-hosted runners, and switch out the stock checkout action with our own action. And our own action will just use the existing clone of the repo, it will fetch the commit that we actually need to build on, then switch to it, and check it out.

And as a result, we can check out the Servo repo pretty reliably in a couple of seconds, instead of having to check out the entire repo from scratch.

Something that flows on from that, though, is that if we are baking a copy of the Servo repo into our runner images, well, what if we just bake a copy of the built artifacts as well? Like, why don't we just build Servo, and bake that into the image?

And this will allow us to do incremental builds, because Servo is a Rust project, and we use Cargo, and Cargo supports incremental builds. As a result, by doing this, when you run a job on our CI system, most of the time we only have to rebuild a handful of crates that have changed, and not have to rebuild all of Servo from scratch.

Now, this is not perfect. Sometimes we'll have some compilation units that get falsely rebuilt, but this works well enough, for the most part, that it's actually a significant time savings.

I also probably wouldn't trust this for building artifacts that you actually want to release in a finished product, just because of the kinds of bugs that we've seen in Cargo's incremental build support.

But for everything else, just your typical builds where you do like commit checks and such, this is very very helpful.

Servo has this system deployed, and it's had it deployed for at least the last year or so.

Nowadays we have three servers which are modestly large, and we use these for the vast majority of our workload. We also have two smaller servers that we use for specific benchmarking tasks.

The stack on these servers, if you could call it that, is largely things that I personally was very familiar with, because I built this. So we've got NixOS for config management, the hypervisor is libvirt and Linux KVM, and the storage is backed by ZFS. The process of actually building the images, and managing the lifecycle of the virtual machines though, is done with some custom orchestration tooling that we've written.

This tooling consists of two services. The monitor service, which runs on every server that operates self-hosted runners, and the queue service, which is single and global.

So the monitor service does the vast majority of the work. It rebuilds virtual machine images, these templates. It clones the templates into virtual machines for each runner, for each job. It registers the runners with the CI service using its API, so that it can receive jobs. And it can also label the runners to tie them to specific jobs when asked to reserve them. This is optional, but it is a required step if we're doing graceful fallback.

Now, with graceful fallback, you do lose the ability to naturally queue up jobs. So what we've put on top of that is a single queue service that sits in front of the cluster, and it essentially acts as a reverse proxy. It's quite thin and simple, and there's a single one of them, not one per server, for the same reason as the general principle of, like, when you go to a supermarket, it's more efficient to have a single large queue, a single long queue of customers that gets dispatched to a bunch of shopkeepers. That's more efficient than having a sort of per-shopkeeper queue, especially when it comes to, like, moving jobs dynamically in response to the availability.

So we've got a whole section here about how graceful fallback works. I might cut this from shorter versions of the talk, but let's jump into it.

On GitHub Actions, every job has this runs-on setting, that defines what kind of runner it needs to get assigned to. It has to define this in advance, before the job runs.

And annoyingly, once you choose, "I want to run on a GitHub hosted runner" or "I want to run on a self-hosted runner", once you decide that, there's no turning back. Now, if you've decided that my job needs to run on a self-hosted runner, it can only run on a self-hosted runner. And now the outcome of your job, and the outcome of your workflow, now depends on that job actually getting a self-hosted runner with the matching criteria and succeeding. There's no way for that to become optional anymore.

And you might say, okay, well, maybe I can work around this by spinning up my self-hosted runners, but just giving them the same labels that GitHub assigns to their runners, maybe it'll do something smart? Like maybe... it will run my self-hosted runners if they're available and fall back. But no, the system has no sense of priority and you can't even define any sense of priority of like, I want to try this kind of runner and fall back to another kind of runner. This is simply not possible with the GitHub Action system.

So it may seem like it's impossible to do graceful fallback, but we found a way.

[self-hosted label] | no => [GitHub-hosted label] 2. $ echo "label=${label}" | tee -a $GITHUB_OUTPUT - Use the step output in runs-on runs-on: ${{ needs.decision.outputs.label }} - But two decisions must not be made concurrently">

What we can do is add a decision job, for each workload job that may need to run on self-hosted runners. And we prepend this job, and its job is to choose a runner label.

So how it essentially works is: it checks if a runner is available, and based on that, either chooses a self-hosted label or a GitHub hosted label. And it chooses it and sets it in an output. And this output can get pulled in... in our workload job, the job that actually does our work, because now you can parameterize the runs-on setting, so that it takes the value from this previous decision job.

Unfortunately, it seems like this may not be possible in Forgejo Actions just yet, so we might have to develop support for that, but it's certainly possible in GitHub Actions today, and it has been for quite a while.

The one caveat with this approach is that you need to be careful to only do this decision process one at a time. You should not do two instances of this process concurrently and interleave them.

- runs-on: ${{ needs.decision.outputs.label }} runs-on: 6826776b-5c18-4ef5-8129-4644a698ae59 - Initially do this directly in the decision job (servo#33081)">

The reason for this is something you'll see if you think about a lot of the answers that you get on Stack Overflow and the GitHub forums. If you look up, "how do I solve this problem? How do I define a job where I can fall back from self-hosted runners to GitHub hosted runners?"

And most of the answers there, they'll have a problem where they check if there are any runners that are available, and then, they will make the decision, committing to either a self-hosted runner or a GitHub hosted runner. The trouble is that in between, if another decision job comes in and tries to make the same kind of decision, they can end up "earmarking" the same runner for two jobs. But each runner is only meant to run one job, and it can only run one job, so one of the jobs will get left without a runner.

So we can start to fix this by actually reserving the runners when we're doing graceful fallback. And how we've done it so far, is that we've used the GitHub Actions API to label the runner when we want to reserve it, and we label it with a unique ID. Then the workload job can put that unique ID, that label, in its runs-on setting. Instead of a general runner label, it can tie itself to this specific, uniquely identified runner label.

And we did it this way initially, because it allowed us to do it inside the decision job, at first. I think in the future, we will have to move away from this, because on Forgejo Actions, the way runner labels work is quite different. They're not something that you can sort of update after the fact. In fact, they're actually kind of defined by the runner process. So this approach for reserving runners won't work on Forgejo Actions. We'll probably have to do that internally on the runner servers. But in the meantime, we use labeling. Yeah, so at first we did this inside the decision job.

There are some problems with this. One of them is that now the decision job has to have a GitHub token that has permissions to manage runners and update their labels. And this is something that we'd like to avoid if possible, because we'd like our jobs to have least privilege that they need to do their work.

And something I didn't mention is that reserving runners in this way doesn't actually solve the problem on its own, because you've now transformed the problem to being, okay, we're going to check if the runner is not yet reserved. We're going to check if there's an unreserved runner, and then, we're going to label the runner. But in between, there's a moment where another process doing the same thing could make the same decision. And as a result, if we just did this, we could end up with a situation where one runner gets assigned two unique labels, but it can only fulfill one of them. So we have that same problem again.

So you might say, okay, well, it looks like GitHub Actions has this neat job concurrency feature. I mean, they say you can use this to define a job in a way where only one of them will run at a time, and you can't run them concurrently, so let's try using that to solve this problem.

What you'll find, though, is that if you try to solve the problem with job concurrency, as soon as there's even the slightest bit of contention, you'll just have your runs starting to fail spuriously, and you'll have to keep rerunning your jobs, and it'll be so much more annoying.

And the reason for this is that, if you look more closely at the docs, job concurrency essentially has a queue limited to one job. So at a maximum, you can have one job, one instance of the job that's running, one instance of the job that's queued, and then if another instance of the job comes in while you have one running and one queued, then those extra jobs just get cancelled, and the build fails. So unfortunately, job concurrency does not solve this problem.

So to solve these problems, what we did is we moved that decision and runner reservation process out of the decision jobs, and into the servers that operate the runners themselves. And we do this with an API that runs on the servers.

One smaller problem you might notice though, is that there's still a possibility that you could reserve a runner, but then after you've reserved the runner, it might fail to actually run. And this has become a lot less likely in our system, in our experience nowadays, now that we've ironed out most of the bugs, but it can still happen from time to time, usually due to infrastructure or network connectivity failures.

And we wanted to solve this by setting a time limit on how long a job can be queued for, because if it can't actually get a runner in practice, it will get stuck in that queued state indefinitely. But unfortunately, while we can set a time limit on how long a job can run for once it's been assigned, we can't actually set a time limit on how long a job can be queued for.

So we have to rely on the default built-in limit for all jobs, where I think there's like a limit of a day or two? So like, 24 or 48 hours or so, for how long a job can be queued? And this is just a really long time. So as a result, whenever this happens, you essentially have to cancel the job manually, or if you don't have permission to do that, you have to go ask someone who has permission, to go and cancel your job for you, which is really annoying.

So we solved that using timeout jobs. A timeout job is a sibling to your workload job, and it acts like a watchdog, and it ensures that it actually got a runner when you expect to get a self-hosted runner.

And how that works is, we wait a short amount of time, just like a minute or two, which should be long enough for the runner to actually start running the job, and then we query the API of the CI service to check if the workload job actually started running, or if it's still in that queued state. If it's still queued after two minutes, we cancel the run.

Unfortunately, we can't just cancel the job run. We do have to cancel the whole workflow run, which is quite annoying. But, you know, it's GitHub, nothing is surprising anymore.

Thankfully, we only have to run this when we actually get a self-hosted runner, so we can make it conditional. But over time, in Servo's deployment, we have actually stopped using these, to free up some of those GitHub-hosted runner resources.

One challenge that comes in making these timeout jobs is identifying the workload jobs uniquely, so we can look up whether it's still queued or whether it's started running. There are unique IDs for each job run. These are just like an incrementing number, and you'd think we'd be able to use this number to look up the workload job uniquely and robustly.

Unfortunately, you can't know the run ID of the job [correction 2025-12-24] ~~until it starts, and it may not ever start... or at least you may not know it~~ until the workflow runs, and there can be many instances of the job in the workflow because of workflow calls. Workflow calls are a feature that essentially allows you to inline the contents of a workflow in another as many times as you like. And as a result, you can have multiple copies, multiple instances of a job that run independently within one workflow run. So we definitely need a way to uniquely look up our instance of the workload job.

The trouble is that the only job relationship you can do in GitHub Actions is a needs relationship, and that's inappropriate for our situation here, because we can say that the workload job needs the decision job, we can say the timeout job needs the decision job — and in fact we do both of these, we "need" to do both of these — but we can't say that the timeout job needs the workload job, because of how needs works.

How needs works is that if job A needs job B, then job B has to actually get assigned a runner, and run, and complete its run — it has to finish — before job A can even start. And in this situation, we're making a timeout job to catch situations where the workload job never ends up running, so if we expressed a needs relationship between them, then the timeout job would never run, in these cases at least.

And even if we could express a needs relationship between jobs, like maybe we could walk the job tree, and go from the timeout job, through the decision job via the needs relationship, and then walk back down to the workload job using the same kind of needs relationship... unfortunately, none of these needs relationships are actually exposed in the API for a running workflow. So like, they're used for scheduling, but when you actually go and query the API, you can't tell what job needs what other job. They're just three jobs, and they're unrelated to one another.

in the name: name: Linux [${{ needs.decision.outputs.unique-id }}] name: Linux [6826776b-5c18-4ef5-8129-4644a698ae59] - Query the CI service API for all jobs in the workflow run - Check the status of the job whose name contains [${{ needs.decision.outputs.unique-id }}] - Yes, really, we have to string-match the name :)))">

So how we had to end up solving this is, we had to tie these jobs together by generating a unique ID, a UUID, and putting it in the friendly name, like the display name of the job, like this.

And to query the CI service to find out if that job is still queued, we need to query it for the whole workflow run, and just look at all of the jobs, and then find the job whose name contains that unique ID.

Then we can check the status, and see if it's still queued. This is really, really silly. Yes, we really do have to string match the name, which is bananas! But this is GitHub Actions, so this is what we have to do.

One thing I didn't mention is that being able to reserve runners needs to be kind of a privileged operation, because we don't just want an arbitrary client on the internet to be able to erroneously or maliciously reserve runners. Even if they may not be able to do a whole lot with those runners, they can still deny service.

So to use the monitor API to do graceful fallback and to request and reserve a runner, normally we would require knowledge of some kind of shared secret, like an API token, and that's what we've done for most of the life of this system.

The trouble with this is that there are many kinds of workflow runs that don't have access to the secrets defined in the repo. A big one is pull_request runs. Most of the time, pull_request runs don't have access to secrets defined in the repo. And there is another kind of run called a pull_request_target run, and those do have access to secrets, but they also have some pretty gnarly security implications that mean that in general, you wanna avoid using these for pull requests anyway.

So if you're stuck with pull_request runs for your pull requests, does that mean that you can't use self-hosted runners? How do we allow pull_request runs to request and reserve runners in a way that it can prove that its request is genuine and authorized?

What we do is we use artifacts. We upload a small artifact that encodes the details of the request and publish the artifact against the run. So in the artifact, we'd say, "I want two Ubuntu runners" or "one Windows runner" or something like that.

And then we would hit the monitor API, we hit a different endpoint that just says, go to this repo, go to this run ID, and then check the artifacts! You'll see my request there! And this does not require an API token, it's not a privileged operation. What's privileged is publishing the artifact, and that's unforgeable; the only entity who can publish the artifact is the workflow itself.

All we have to do then, all we have to be careful to do, is to delete the artifact after we reserve the runner, so that it can't be replayed by a malicious client. And in the event that deleting the artifact fails, we can also set the minimum auto-delete period of, I think at the moment it's 24 hours, just in case that fails.

Graceful fallback normally means that you lose the ability to naturally queue jobs for self-hosted runners. And this happens because when you hit the API, requesting, you know, are there any available runners, please reserve one for me. If there aren't any available at the time of the request, we will immediately fall back to running on a GitHub hosted runner.

But the thing is, is that our self-hosted runners are generally so much faster! A GitHub hosted run might take five times as long as the equivalent self-hosted run, and if that was the case, it would actually be beneficial to wait up to 80% of that time, and we'd still probably save time.

So to increase the utilization of our self-hosted runners, what we can do is run a small queue service that sits in front of the monitors, and it essentially acts as a reverse proxy. It will take the same kind of requests for reserving runners as before, but it will have a global view of the availability and the runner capacity at any given moment.

And based on that, it will either respond to the client saying, go away and please try again later, or it will forward the request onto one of the monitors based on the available capacity.

We also have some lessons here about how to automate building virtual machine images, essentially. And these are lessons that we've learned over the last year or two.

GitHub Actions uses Packer for their first-party runner images, so they use Packer to build those images. And our monitor service also automates building the images, but we don't use Packer.

Initially, we had a handful of scripts that we just kicked off manually whenever we needed to update our images, but we've now fully automated the process. And we've done this using some modules in our monitor service, so there's some high-level Rust code that drives these image rebuilds, and it even uses reflink copies to take advantage of copy-on-write time savings and space savings with ZFS.

Now, one of the complications we ran into when building images, over the past year or so, is that we used to pull a sort of clever trick where, we would do as much of the process of configuring the virtual machine image on the host, actually, as possible, rather than doing configuration inside the guest, and having to spin up the guest so that we can configure it. And we were able to do this by essentially mounting the root file system of the guest image, on the host, and then injecting files, like injecting tools and scripts and other things that are needed by the image and needed to configure the image.

But we stopped being able to do this eventually because, well, essentially because of macOS. macOS has no file system that's usable for building Servo that can also be mounted on Linux with write support. Because like, just think about them, right? We've got HFS+, which Linux can write to but only if there's no journaling and you can't install macOS on HFS+ without journaling. There's APFS, which Linux has no support for. There's exFAT, which has no support for symlinks, so a lot of tools like uv will break. There's NTFS, which we thought would be our savior, but when we tried to use it, we ran into all sorts of weird build failures, which we believe are due to some kind of filesystem syncing race condition or something like that, so NTFS was unusable as well.

In the end, what we had to do is, if a guest image needed any tools or dependencies, it had to download them from a web server on the host.

We eventually settled on a consistent approach to automating the process of installing the OSes and configuring them for each runner template.

And the approach that we used was to install the OS using whatever method is easiest, and to bootstrap the config using any native config management system, if there is one included with the operating system.

But once we've kicked off that process, we then use the native config management as little as possible. And we do this because a lot of the config management tools that are built into these operating systems are quite quirky and annoying, and they are built for needs that we don't have, the primary need being that they can manage the configuration of a system over time, keeping it up to date with any changes. The thing about these runner images, though, is that each runner image only needs to get built once, it only needs to get configured once, and then after after that it'll be cloned for each runner, and then it will only run once, and this kind of sort of "one shot" use case is something that's kind of overkill with config management systems.

We can just do it with the usual scripting and automation facilities of the operating system. How does that look like in practice?

Well for Linux, we install the OS using pre-built Ubuntu cloud images, that we just download from the mirrors.

We bootstrap the config using cloud-init, which is such a painful... it's so painful to use cloud-init. We use it because it's included with the operating system, so that means it's the fastest possible way to get started.

We use it as little as possible: we just configure the logs to go to a TTY, we configure the network, so we can connect to the network, and then once we're on the network, we just curl and run a bash script which does the rest.

The same goes for Windows.

We install the operating system using an automated answers file called autounattend.xml. There's a nice little generator here which you can use if you don't want to have to set up a whole Windows server to set up unattended installations. You generate that XML file, and you can also use that XML file to bootstrap the config.

Again we use it as little as possible, because writing automations as XML elements is kind of a pain. So we essentially just set up a scheduled task to run the next stage of the config, we install the network driver, we import some registry settings, and we reboot. That's it. The rest of it is done with a PowerShell script.

The same goes for macOS.

Now, unfortunately, installing the OS and bootstrapping the config does have to be done by hand. And this is because if you want to automate a macOS installation, your two options, more or less, are enterprise device management solutions, which cost a lot of money and mean that you have to have a macOS server around to control and orchestrate the servers. But if you don't want to use one of those enterprise solutions, what most open systems that are faced with this problem end up doing is to throw OpenCV at the problem. I've seen several projects use OpenCV to OCR the setup wizard, which is... it's certainly a bold strategy. It's not really for me.

What I decided to do instead is just install the OS by hand, and pipe curl into sh to kick off the config management process. And this is something that we only really have to do once, because we do it once, and then we take a snapshot of it, and then we never have to do it again, at least until the next version of macOS comes out.

So this bootstrap script just does a handful of minimal things: it enables automatic login, it sets up a LaunchAgent to ensure that we can run our own code on each boot, and then it does a handful of other things which it honestly doesn't really have to do in this script. We could probably do these things in the zsh script which we then curl and run. And that's where the remainder of the work is done.

So looking forward, some things that we'd like to do with this system.

The first is to decouple it from Servo. So we built this CI system quite organically over the past 12 to 18 months, and we built it around Servo's needs specifically, but we think this system could be more broadly useful for other projects. We'll just have to abstract away some of the Servo specific bits, so that it can be more easily used on other projects, and that's something we're looking into now.

Something else that we'll have to do sooner or later is add support for macOS runners on Apple Silicon, that is ARM64, and the reason we have to do this is that macOS 26, which is the most recent version of macOS that came out in September, that's just a couple months ago, that is the last version of macOS that will support x86 CPUs. And at the moment, our macOS runners run on x86 CPUs, on the host and in the guest.

This is a little bit complicated because at the moment, our macOS runners actually run on Linux hosts, using a Linux-KVM-based sort of Hackintosh-y kind of solution. And there is no counterpart for this for arm64 hosts and guests, and I'm not sure there ever will be one. So we're going to have to port the system so that it can run on macOS hosts, so we can use actual Mac hardware for this, which is easy enough to do, and that's in progress.

But we're also going to have to port the system so it can run with other hypervisors. And this is because, although libvirt supports macOS hosts, the support for the macOS Hypervisor framework and Virtualization framework is not mature enough to actually run macOS guests in libvirt. And I'm not sure how long that will take to develop, so in the meantime, we've been looking at porting the system so that when you're on a Mac, you can run with UTM instead, and that's been working fairly well so far.

We're also looking at porting the system so that it can run with Forgejo Actions and not just GitHub Actions. So Forgejo Actions is an open alternative to GitHub Actions that tries to be loosely compatible with GitHub Actions, and in our experience, from experimentation so far, we found that it mostly is loosely compatible. We think we'll only have to make some fairly minor changes to our system to make it work on both CI systems.

That said, this CI system could potentially be more broadly applicable to other CI services as well, because virtual machine orchestration is something that we haven't really seen any CI services have a great built-in solution for. So if this interests you and you want to use it on your project on some other CI service, then we'd appreciate knowing about that, because that could be something we would explore next.

The remaining ideas are things that I think we could look into to make our runners more efficient.

The big one is autoscaling. So at the moment when you set up a server to operate some self-hosted runners, you essentially have to statically pre-configure how many runners of each kind of runner you want to be kept operating. And this has worked well enough for us for the most part, but it does mean that there's some kind of wasted resources sometimes, when the moment-to-moment needs of the jobs that are queued up aren't well fitted to the composition of your runner configuration. So if we had the ability to dynamically respond to demand, or some kind of autoscaling, I think we could improve our runner utilization rates a little bit, and sort of get more out of the same amount of runner capacity, the same amount of server capacity.

There's a couple ideas here, also, about reducing boot times for the runners, which can be quite helpful if you have a big backlog of jobs queued up for these servers, and this is because time spent booting up each runner, each virtual machine, is time that cannot be spent doing real work.

So two ways we can think of to reduce these boot times are, to have hot spares ready to go, the idea being that, if you can spin up more runners than you actually intend to run concurrently, and just have them sitting around, then you can kind of amortize the boot times, and sort of get the boot process process out of the way. And the way you do this is by spinning up a whole bunch of runners, say maybe you spin up like twenty runners, even though you only intend to run four of them concurrently.

And what you do is you give these runners a token amount of RAM to start with. You give them like one gig of RAM instead of 16 or 32 gigs of RAM. And then when a job comes in, and you actually want to assign the runner out so that it can do the work, then you dynamically increase the RAM from one gig, or that token amount, to the actual amount, like 16 gigs or 32 gigs. And this should be fairly easy to do in practice. This is actually supported in libvirt using a feature known as memory ballooning. But there are some minor caveats, like you do lose the ability to do certain optimizations, like you can't do huge pages backing on the memory anymore. But for the most part, this should be fairly technically simple to implement.

Something that could be more interesting in the longer term is microVMs, things like Firecracker, which as I understand it, these microVMs can sort of take the concept of paravirtualization to its logical extreme. And what it means is that on kernels that support being run as microVMs, you can boot them in like one or two seconds, instead of 20 or 30 or 60 seconds. And this could save a great deal of time, at least for jobs that run on Linux and BSD. I don't know if I said Linux and macOS, but I meant Linux and BSD.

So yea, we now have a system that we use to speed up our builds in Servo's CI, and it works fairly well for us. And we think that it's potentially useful for other projects as well.

So if you're interested to find out more, or you're interested to find out how you can use the system in your own projects, go to our GitHub repo at servo/ci-runners, or you can go here for a link to the slides. Thanks!

Generative AI in Servo

2025-04-11T12:00:00+00:00

Servo has shown that we can build a browser with a modern, parallel layout engine in a fraction of the cost of the big incumbents, thanks to our powerful tooling, our strong community, and our thorough documentation. But we can, and should, build Servo without generative AI tools like GitHub Copilot.

I’m the lead author of our monthly updates and the Servo book, a member of the Technical Steering Committee, and a coauthor of our current AI policy (permalink). That policy was inspired by Gentoo’s AI policy, and has in turn inspired the AI policies of Loupe and Amaranth.

Recently the TSC voted in favour of two proposals that relax our ban on AI contributions. This was a mistake, and it was also a mistake to wait until after we had made our decision to seek community feedback (see § On governance). § Your feedback made it clear that those proposals are the wrong way forward for Servo.

I call on the TSC to explicitly reaffirm that generative AI tools like Copilot are not welcome in Servo, and make it clear that we intend to keep it that way indefinitely, in both our policy and the community, so we can start rebuilding trust. It’s not enough to say oops, sorry, we will not be moving forward with these proposals.

Like any logic written by humans, this policy does have some unintended consequences. Our intent was to ban AI tools that generate bullshit [a] in inscrutable ways, including GitHub Copilot and ChatGPT. But there are other tools that use similar underlying technology in more useful and less problematic ways (see § Potential exceptions). Reviewing these tools for use in Servo should be a community-driven process.

We should not punish contributors for honest mistakes, but we should make our policy easier to follow. Some ways to do this include documenting the tools that are known to be allowed and not allowed, documenting how to turn off features that are not allowed, and giving contributors a way to declare that they’ve read and followed the policy.

The declaration would be a good place to provide a dated link to the policy, giving contributors the best chance to understand the policy and knowingly follow it (or violate it). This is not perfect, and it won’t always be easy to enforce, but it should give contributors and maintainers a foundation of trust.

Potential exceptions

Proposals for exceptions should start in the community, and should focus on a specific tool used for a specific purpose. If the proposal is for a specific kind of tool, it must come with concrete examples of which tools are to be allowed. Much of the harm being caused by generative AI in the world around us comes from people using open-ended tools that are not fit for any purpose, or even treating them like they are AGI.

The goal of these discussions would be to understand:

the underlying challenges faced by contributors
how effective the tool is for the purpose
how well the tool and purpose mitigate the issues in the policy
whether there are any existing or alternative solutions
whether those solutions have problems that need to be addressed

Sometimes the purpose may need to be constrained to mitigate the issues in the policy. Let’s look at a couple of examples.

For some tasks like speech recognition [b] and machine translation [c] [d], tools with large language models and transformers are the state of the art (other than humans). This means those tools may be probabilistic tools, and strictly speaking, they may be generative AI tools, because the models they use are generative models. Generative AI does not necessarily mean “AI that generates bullshit in inscrutable ways”.

Speech recognition can be used in a variety of ways. If plumbed into ChatGPT, it will have all of the same problems as ChatGPT. If used for automatic captions, it can make videos and calls accessible to people that can’t hear well (myself included), but it can also infantilise us by censoring profanities and make serious errors that cause real harm. If deployed for that purpose by an online video platform, it can undermine the labour of human transcribers and lower the overall quality of captions.

If used as an input method, it would be a clear win for accessibility. My understanding of speech input tools is that they have a clear (if configurable) mapping from the things you say to the text they generate or the edits they make, so they may be a good fit.

In that case, maintainer burden and correctness and security would not be an issue, because the author is in complete control of what they write. Copyright issues seem less of a concern to me, since these tools operate on such a small scale (words and symbols) that they are unlikely to reproduce a copyrightable amount of text verbatim, but I am not a lawyer. As for ethical issues, these tools are generally trained once then run on the author’s device. When used as an input method, they are not being used to undermine labour or justify layoffs. I’m not sure about the process of training their models.

Machine translation can also be used in a variety of ways. If deployed by a language learning app, it can ruin the quality of your core product, but hey, then you can lay off those pesky human translators. If used to localise your product, your users will finally be able to compress to postcode file. If used to localise your docs, it can make your docs worse than useless unless you take very careful precautions. What if we allowed contributors to use machine translation to communicate with each other, but not in code commits, documentation, or any other work products?

Deployed carelessly, they will waste the reader’s time, and undermine the labour of actual human translators who would otherwise be happy to contribute to Servo. If constrained to collaboration, it would still be far from perfect, but it may be worthwhile.

Maintainer burden should be mitigated, because this won’t change the amount or kind of text that needs to be reviewed. Correctness and security too, because this won’t change the text that can be committed to Servo. I can’t comment on the copyright issues, because I am not a lawyer. The ethical issues may be significantly reduced, because this use case wasn’t a market for human translators in the first place.

Your feedback

I appreciate the feedback you gave on the Fediverse, on Bluesky, and on Reddit. I also appreciate the comments on GitHub from several people who were more on the favouring side of the proposal, even though we reached different conclusions in most cases. One comment argued that it’s possible to use AI autocomplete safely by accepting the completions one word at a time.

That said, the overall consensus in our community was overwhelmingly clear, including among many of those who were in favour of the proposals. None of the benefits of generative AI tools are worth the cost in community goodwill [e].

Much of the dissent on GitHub was already covered by our existing policy, but there were quite a few arguments worth highlighting.

Speech-to-text input is ok [f] [g].

Machine translation is generally not useful or effective for technical writing [h] [i] [j]. It can be, if some precautions are taken [k]. It may be less ethically encumbered than generative AI tools [l]. Client-side machine translation is ok [m]. Machine translation for collaboration is ok [n] [o].

The proposals. Proposal 1 is ill-defined [p]. Proposal 2 has an ill-defined distinction between autocompletes and “full” code generation [q] [r] [s].

Documentation is just as technical as code [u]. Wrong documentation is worse than no documentation [v] [w] [x]. Good documentation requires human context [y] [z].

GitHub Copilot is not a good tool for answering questions [ab]. It isn’t even that good of a programming tool [ac]. Using it may be incompatible with the DCO [ad]. Using it could make us depend on Microsoft to protect us against legal liability [ae].

Correctness. Generative AI code is wrong at an alarming rate [af]. Generative AI tools will lie to us with complete confidence [ag]. Generative AI tools (and users of those tools) cannot explain their reasoning [ah] [ai]. Humans as supervisors are ill-equipped to deal with the subtle errors that generative AI tools make [aj] [ak] [al] [am]. Even experts can easily be misled by these tools [an]. Typing is not the hard part of programming [ao], as even some of those in favour have said:

If I could offload that part of the work to copilot, I would be left with more energy for the challenging part.

Project health. Partially lifting the ban will create uncertainty that increases maintainer burden for all contributions [ap] [aq]. Becoming dependent on tools with non-free models is risky [ar]. Generative AI tools may not be fair use [as] → [at]. Outside of Servo, people have spent so much time cleaning up after LLM-generated mess [au].

Material. Servo contributor refuses to spend time cleaning up after LLM-generated mess [av]. Others will stop donating [aw] [ax] [ay] [az] [ba] [bb] [bc] [bd] [be] [bf] [bg], will stop contributing [bh], will not start donating [bi], will not start contributing [bj] [bk], or will not start promoting [bl] the project.

Broader context. Allowing AI contributions is a bad signal for the project’s relationship with the broader AI movement [bm] [bn] [bo]. The modern AI movement is backed by overwhelming capital interests, and must be opposed equally strongly [bp]. People often “need” GitHub or Firefox, but no one “needs” Servo, so we can and should be held to a higher standard [bq]. Rejection of AI is only credible if the project rejects AI contributions [br]. We can attract funding from AI-adjacent parties without getting into AI ourselves [bs], though that may be easier said than done [bt].

On governance

Several people have raised concerns about how Servo’s governance could have led to this decision, and some have even suspected foul play. But like most discussions in the TSC, most of the discussion around AI contributions happened async on Zulip, and we didn’t save anything special for the synchronous monthly public calls. As a result, whenever the discussion overflowed the sync meeting, we just continued it internally, so the public minutes were missing the vast majority of the discussion (and the decisions). These decisions should probably have happened in public.

Our decisions followed the TSC’s usual process, with a strong preference for resolving disagreements by consensus rather than by voting, but we didn’t have any consistent structure for moving from one to the other. This may have made the decision process prone to being blocked and dominated by the most persistent participants.

Contrast this with decision making within Igalia, where we also prefer consensus before voting, but the consensus process is always used to inform proposals that are drafted by more than one person and then always voted on. Most polls are “yes” or “no” by majority, and only a few polls for the most critical matters allow vetoing. This ensures that proposals have meaningful support before being considered, and if only one person is strongly against something, they are heard but they generally can’t single-handedly block the decision with debate.

We also didn’t have any consistent structure for community consultation, so it wasn’t clear how or when we should seek feedback. A public RFC process may have helped with this, and would also help us collaborate on and document other decisions.

More personally, I did not participate in the extensive discussion in January and February that helped move consensus in the TSC towards allowing the non-code and Copilot exceptions until fairly late. Some of that was because I was on leave, including for the vote on the initial Copilot “experiments”, but most of it was that I didn’t have the bandwidth. Doing politics is hard, exhausting work, and there’s only so much of it you can do, even when you’re not wearing three other hats.

Meet the CSS highlight pseudos

2022-09-01T15:00:00+00:00

A year and a half ago, I was asked to help upstream a Chromium patch allowing authors to recolor spelling and grammar errors in CSS. At the time, I didn’t realise that this was part of a far more ambitious effort to reimagine spelling errors, grammar errors, text selections, and more as a coherent system that didn’t yet exist as such in any browser. That system is known as the highlight pseudos, and this post will focus on the design of said system and its consequences for authors.

This is the third part of a series (part one, part two) about Igalia’s work towards making the CSS highlight pseudos a reality.

What are they?
Can I use them?
How do I use them?
How do they work?
Gotchas

What are they?

CSS has four highlight pseudos and an open set of author-defined custom highlight pseudos. They have their roots in ::selection, which was a rudimentary and non-standard, but widely supported, way of styling text and images selected by the user.

The built-in highlights are ::selection for user-selected content, ::target-text for linking to text fragments, ::spelling-error for misspelled words, and ::grammar-error for text with grammar errors, while the custom highlights are known as ::highlight(x) where x is the author-defined highlight name.

Can I use them?

::selection has long been supported by all of the major browsers, and ::target-text shipped in Chromium 89. But for most of that time, no browser had yet implemented the more robust highlight pseudo system in the CSS pseudo spec.

::highlight() and the custom highlight API shipped in Chromium 105, thanks to the work by members¹ of the Microsoft Edge team. They are also available in Safari 14.1 (including iOS 14.5) as an experimental feature (Highlight API). You can enable that feature in the Develop menu, or for iOS, under Settings > Safari > Advanced.

Chromium 105 also implements the vast majority of the new highlight pseudo system. This includes highlight overlay painting, which was enabled for all highlight pseudos, and highlight inheritance, which was enabled for ::highlight() only.

Chromium 108 includes ::spelling-error and ::grammar-error as an experimental feature, together with the new ‘text-decoration-line’ values ‘spelling-error’ and ‘grammar-error’. Chromium 111 enables highlight inheritance for ::selection and ::target-text as an experimental feature, in addition to ::highlight() and the spelling and grammar pseudos (which always use highlight inheritance). You can enable these features at

chrome://flags/#enable-experimental-web-platform-features

Click the table below to see if your browser supports these features.

act: 
sel: 
cha:

	yours	Chromium	Safari	Firefox
Custom highlights	noyes	105	14.1*	?
• ::highlight()	noyes	105	14.1*	?
• CSSOM API	noyes	105	14.1* (ab)	?
::spelling-error	noyes	108*	?	?
Highlight overlay painting	noyes	105	?	?
Highlight inheritance (::selection)	noyes	111*	?	?
Highlight inheritance (::highlight)	noyes	105	?	?

How do I use them?

While you can write rules for highlight pseudos that target all elements, as was commonly done for pre-standard ::selection, selecting specific elements can be more powerful, allowing descendants to cleanly override highlight styles.

the fox jumps over the dog

(the ^quick fox, mind you)

<style>
    :root::selection {
        color: white;
        background-color: black;
    }
    aside::selection {
        background-color: darkred;
    }
</style>
<body>
    <p>the fox jumps over the dog
    <aside>
        (the <sup>quick</sup> fox, mind you)
    </aside>
</body>

Previously the same code would yield…

the fox jumps over the dog

(the ^quick fox, mind you)

(in older browsers)

Notice how none of the text is white on black, because there are always other elements (body, p, aside, sup) between the root and the text.

…unless you also selected the descendants of :root and aside:

:root::selection,
:root *::selection
/* (or just ::selection) */ {
    color: white;
    background-color: black;
}
aside::selection,
aside *::selection {
    background-color: green;
}

Note that a bare ::selection rule still means *::selection, and like any universal rule, it can interfere with inheritance when mixed with non-universal highlight rules.

the fox jumps over the dog

(the ^quick fox, mind you)

<style>
    ::selection {
        color: white;
        background-color: black;
    }
    aside::selection {
        background-color: darkred;
    }
</style>
<body>
    <p>the fox jumps over the dog
    <aside>
        (the <sup>quick</sup> fox, mind you)
    </aside>
</body>

sup::selection would have inherited ‘darkred’ from aside::selection, but the universal ::selection rule matches it directly, so it becomes black.

::selection is primarily controlled by user input, though pages can both read and write the active ranges via the Selection API with getSelection().

::target-text is activated by navigating to a URL ending in a fragment directive, which has its own syntax embedded in the #fragment. For example:

#foo:~:text=bar targets #foo and highlights the first occurrence of “bar”
#:~:text=the,dog highlights the first range of text from “the” to “dog”

::spelling-error and ::grammar-error are controlled by the user’s spell checker, which is only used where the user can input text, such as with textarea or contenteditable, subject to the spellcheck attribute (which also affects grammar checking). For privacy reasons, pages can’t read the active ranges of these highlights, despite being visible to the user.

::highlight() is controlled via the Highlight API with CSS.highlights. CSS.highlights is a maplike object, which means the interface is the same as a Map of strings (highlight names) to Highlight objects. Highlight objects, in turn, are setlike objects, which you can use like a Set of Range or StaticRange objects.

Hello, world!

<style>
    ::highlight(foo) { background: yellow; }
</style>
<script>
    const foo = new Highlight;
    CSS.highlights.set("foo", foo); // maplike

    const range = new Range;
    range.setStart(document.body.firstChild, 0);
    range.setEnd(document.body.firstChild, 5);
    foo.add(range); // setlike
</script>
<body>Hello, world!</body>

You can use getComputedStyle() to query resolved highlight styles under a particular element. Regardless of which parts (if any) are highlighted, the styles returned are as if the given highlight is active and all other highlights are inactive.

<style>
    ::selection { background: #00FF00; }
    ::highlight(foo) { background: #FF00FF; }
</style>
<script>
    getSelection().removeAllRanges();
    getSelection().selectAllChildren(document.body);

    const style = getComputedStyle(document.body, "::highlight(foo)");
    console.log(style.backgroundColor);
</script>
<body>Hello, world!</body>

This code always prints “rgb(255, 0, 255)”, even though only ::selection is active.

How do they work?

Highlight pseudos are defined as pseudo-elements, but they actually have very little in common with other pseudo-elements like ::before and ::first-line.

Unlike other pseudos, they generate highlight overlays, not boxes, and these overlays are like layers over the original content. Where text is highlighted, a highlight overlay can add backgrounds and text shadows, while the text proper and any other decorations are “lifted” to the very top.

quikc brown fox originating element

quikc brown fox ::spelling-error

quikc brown fox ::target-text

quikc brown fox

quikc brown fox ::selection

quikc brown fox

You can think of highlight pseudos as innermost pseudo-elements that always exist at the bottom of any tree of elements and other pseudos, but unlike other pseudos, they don’t inherit their styles from that element tree.

Instead each highlight pseudo forms its own inheritance tree, parallel to the element tree. This means body::selection inherits from html::selection, not from ‘body’ itself.

At this point, you can probably see that the highlight pseudos are quite different from the rest of CSS, but there are also several special cases and rules needed to make them a coherent system.

For the typical appearance of spelling and grammar errors, highlight pseudos need to be able to add their own decorations, and they need to be able to leave the underlying foreground color unchanged. Highlight inheritance happens separately from the element tree, so we need some way to refer to the underlying foreground color.

That escape hatch is to set ‘color’ itself to ‘currentColor’, which is the default if nothing in the highlight tree sets ‘color’.

quick → quikc
quick → quikc

:root::spelling-error {
    /* color: currentColor; */
    text-decoration: red wavy underline;
}

To make highlight inheritance actually useful for ‘text-decoration’ and ‘background-color’, all properties are inherited in highlight styles, even those that are not usually inherited.

^quick fox

<style>
    aside::selection {
        background-color: yellow;
    }
</style>
<aside>
    <sup>quick</sup> fox
</aside>

Only a handful of properties are settable in highlight styles, for performance and privacy reasons. In general, properties that you can’t set in highlight styles come from the originating element, which is to say from the non-highlight styles.

A A

<style>
    :root::selection {
        text-shadow: 0.25em 0.25lh lightblue;
        background-color: transparent;
    }
    big {
        font-size: 2em;
        line-height: 1;
    }
</style>
<aside>
    <sup>quick</sup> fox
</aside>

When selected, the <big> is still 2em/1, and the selection shadow takes that into account, even though you are not allowed to set ‘font-size’ or ‘line-height’ in :root::selection.

This would conflict with the usual rules³ for decorating boxes, because descendants would get two decorations, one propagated and one inherited. We resolved this by making decorations added by highlights not propagate to any descendants.

quick fox

^quick fox

^quikc fxo

<style>
    .blue {
        text-decoration: blue underline;
    }
    :root::spelling-error {
        text-decoration: red wavy underline;
    }
</style>
<div class="blue">
    <sup>quick</sup> fox
</div>
<div contenteditable spellcheck lang="en">
    <sup>quikc</sup> fxo
</div>

The blue decoration propagates to the sup element from the decorating box, so there should be a single line at the normal baseline. On the other hand, the spelling decoration is inherited by sup::spelling-error, so there should be separate lines for “quikc” and “fxo” at their respective baselines.

Unstyled highlight pseudos generally don’t change the appearance of the original content, so the default ‘color’ and ‘background-color’ in highlights are ‘currentColor’ and ‘transparent’ respectively, the latter being the property’s initial value. But two highlight pseudos, ::selection and ::target-text, have UA default foreground and background colors.

For compatibility with ::selection in older browsers, the UA default ‘color’ and ‘background-color’ (e.g. white on blue) is only used if neither were set by the author. This rule is known as paired cascade, and for consistency it also applies to ::target-text.

	default on default plus more text
+	`p { color: rebeccapurple; } ::selection { background: yellow; }`
=	currentColor on yellow plus more text

It’s common for selected text to almost invert the original text colors, turning black on white into white on blue, for example. To guarantee that the original decorations remain as legible as the text when highlighted, which is especially important for decorations with semantic meaning (e.g. line-through), originating decorations are recolored to the highlight ‘color’. This doesn’t apply to decorations added by highlights though, because that would break the typical appearance of spelling and grammar errors.

do not buy bread

<style>
    del {
        text-decoration: darkred line-through;
    }
    ::selection {
        color: white;
        background: darkblue;
    }
</style>
<div>
    do <del>not</del> buy bread
</div>

This line-through decoration becomes white like the rest of the text when selected, even though it was explicitly set to ‘darkred’ in the original content.

The default style rules for highlight pseudos might look something like this. Notice the new ‘spelling-error’ and ‘grammar-error’ decorations, which authors can use to imitate native spelling and grammar errors.

:root::selection { background-color: Highlight; color: HighlightText; }
:root::target-text { background-color: Mark; color: MarkText; }
:root::spelling-error { text-decoration: spelling-error; }
:root::grammar-error { text-decoration: grammar-error; }

This doesn’t completely describe ::selection and ::target-text, due to paired cascade.

The way the highlight pseudos have been designed naturally leads to some limitations.

Gotchas

Removing decorations and shadows

Older browsers with ::selection tend to treat it purely as a way to change the original content’s styles, including text shadows and other decorations. Some tutorial content has even been written to that effect:

One of the most helpful uses for ::selection is turning off a text-shadow during selection. A text-shadow can clash with the selection’s background color and make the text difficult to read. Set text-shadow: none; to make text clear and easy to read during selection.

Under the spec, highlight pseudos can no longer remove or really change the original content’s decorations and shadows. Setting these properties in highlight pseudos to values other than ‘none’ adds decorations and shadows to the overlays when they are active.

del {
    text-decoration: line-through;
    text-shadow: 2px 2px red;
}
::highlight(undelete) {
    text-decoration: none;
    text-shadow: none;
}

This code means that ::highlight(undelete) adds no decorations or shadows, not that it removes the line-through and red shadow when del is highlighted.

While the new :has() selector might appear to offer a solution to this problem, pseudo-element selectors are not allowed in :has(), at least not yet.

del:has(::highlight(undelete)) {
    text-decoration: none;
    text-shadow: none;
}

This code does not work.

Removing shadows that might clash with highlight backgrounds (as suggested in the tutorial above) will no longer be as necessary anyway, since highlight backgrounds now paint on top of the original text shadows.

Faultlore

→

Faultlore

If you still want to ensure those shadows don’t clash with highlights in older browsers, you can set ‘text-shadow’ to ‘none’, which is harmless in newer browsers.

::selection { text-shadow: none; }

This rule might be helpful for older browsers, but note that like any universal rule, it can interfere with inheritance of ‘text-shadow’ when combined with more specific rules.

As for line decorations, if you’re really determined, you can work around this limitation by using ‘-webkit-text-fill-color’, a standard property (believe it or not) that controls the foreground fill color of text⁴.

::highlight(undelete) {
    color: transparent;
    -webkit-text-fill-color: CanvasText;
}

This hack hides any original decorations (in visual media), because those decorations are recolored to the highlight ‘color’, but it might change the text color too.

Fun fact: because of ‘-webkit-text-fill-color’ and its stroke-related siblings, it isn’t always possible for highlight pseudos to avoid changing the foreground colors of text, at least not without out-of-band knowledge of what those colors are.

the quick fox

↓

the quikc fox

p { color: blue; }
em {
    -webkit-text-fill-color: yellow;
    -webkit-text-stroke: 1px green;
}
:root::spelling-error {
    /* default styles */
    color: currentColor;
    -webkit-text-fill-color: currentColor;
    -webkit-text-stroke-color: 0 currentColor;
    text-decoration: spelling-error;
}
em::spelling-error {
    /* styles needed to preserve text colors */
    -webkit-text-fill-color: yellow;
    -webkit-text-stroke: 1px green;
}

When a word in em is misspelled, it will become blue like the rest of p, unless the fill and stroke properties are set in ::spelling-error accordingly.

Accessing global constants

Update (2024-04-29): this section is no longer true (see § Custom properties), but you can click here to read what I wrote originally.

Highlight pseudos also don’t automatically have access to custom properties set in the element tree, which can make things tricky if you have a design system that exposes a color palette via custom properties on :root.

:root {
    --primary: #420420;
    --secondary: #C0FFEE;
    --accent: #663399;
}
::selection {
    background: var(--accent);
    color: var(--secondary);
}

This code does not work.

You can work around this by adding selectors for the necessary highlight pseudos to the rule defining the constants, or if the necessary highlight pseudos are unknown, by rewriting each constant as a custom @property rule.

:root, :root::selection {
    --primary: #420420;
    --secondary: #C0FFEE;
    --accent: #663399;
}

@property --primary {
    initial-value: #420420;
    syntax: "*"; inherits: false;
}
@property --secondary {
    initial-value: #C0FFEE;
    syntax: "*"; inherits: false;
}
@property --accent {
    initial-value: #663399;
    syntax: "*"; inherits: false;
}

Custom properties

You can use custom properties in highlight styles, but you will not be able to set or override them there. Custom property values come from the nearest originating element.

This is unfortunate, but allowing you to set custom properties in highlight styles broke a lot of existing content on the web (and existing advice on Stack Overflow). For more details, see these posts by my colleague Stephen Chenney:

The CSS Highlight Inheritance Model (January 2024)
CSS Custom Properties in Highlight Pseudos (April 2024)
(and the CSSWG issues, #6641 and #9909)

the fox jumps over the dog

(the ^quick fox, mind you)

<style>
    ::selection /* = *::selection (universal) */ {
        /* using --selection-color in ::selection is ok... */
        background-color: var(--selection-color); /* 🙆‍♀️ */

        /* ...but you will no longer be allowed to set it! */
        --selection-color: red; /* 🙅‍♀️ */
    }
    body {
        --selection-color: lightgreen;
    }
    aside {
        --selection-color: yellow;
    }
</style>
<body>
    <p>the fox jumps over the dog
    <aside>
        (the <sup>quick</sup> fox, mind you)
    </aside>
</body>

Spec issues

While the design of the highlight pseudos has mostly settled, there are still some unresolved issues to watch out for.

how to use spelling and grammar decorations with the UA default colors (#7522)
values of non-applicable properties, e.g. ‘text-shadow’ with em units (#7591)
the meaning of underline- and emphasis-related properties in highlights (#7101)
whether ‘-webkit-text-fill-color’ and friends are allowed in highlights (#7580)
some browsers “tweak” the colors or alphas set in highlight styles (#6853)
how the highlight pseudos are supposed to interact with SVG (svgwg#894)

What now?

The highlight pseudos are a radical departure from older browsers with ::selection, and have some significant differences with CSS as we know it. Now that we have some experimental support, we want your help to play around with these features and help us make them as useful and ergonomic as possible before they’re set in stone.

Special thanks to Rego, Brian, Eric (Igalia), Florian, fantasai (CSSWG), Emilio (Mozilla), and Dan for their work in shaping the highlight pseudos (and this post). We would also like to thank Bloomberg for sponsoring this work.

Dan, Fernando, Sanket, Luis, Bo, and anyone else I missed. ↩
See this demo for more details. ↩ ↩²
CSSWG discussion also found that decorating box semantics are undesirable for decorations added by highlights anyway. ↩
This is actually the case everywhere the WHATWG compat spec applies, at all times. If you think about it, the only reason why setting ‘color’ to ‘red’ makes your text red is because ‘-webkit-text-fill-color’ defaults to ‘currentColor’. ↩

Chromium spelling and grammar, part 2

2021-12-16T12:30:00+00:00

Modern web browsers can help users with their word processing needs by drawing squiggly lines under possible spelling or grammar errors in their input. CSS will give authors more control over when and how they appear, with the new ::spelling- and ::grammar-error pseudo-elements, and spelling- and grammar-error text decorations. Since part 1 in May, we’ve done a fair bit of work in both Chromium and the CSSWG towards making this possible.

The client funding this work had an internal patch that allowed you to change the colors of those squiggly lines, and our job was to upstream it. The patch itself was pretty simple, but turning that into an upstream feature is a much bigger can of worms. So far, we’ve landed over 30 patches, including dozens of new web platform tests, opened 8 spec issues, and run into some gnarly bugs going back to at least 2009.

Check out our project index for a complete list of demos, tests, patches, and issues. For more details about the CSS highlight pseudos in particular, check out my BlinkOn 15 talk, including the highlight painting visualiser.

(slides)

Implementation status
Charlie’s lawyerings
Squiggly lines
- Platform “conventions”
- Precise wavy decorations
Phase-locked decorations
- Bézier bounding box
- Cover me!
Highlight inheritance
What now?

Implementation status

Chromium 96 includes a rudimentary version of highlight inheritance, with support for ::highlight in Chromium 98 (Fernando Fiori). This is currently behind a Blink feature:

--enable-blink-features=HighlightInheritance

Adding to our initial support for ::{spelling,grammar}-error, we’ve since made progress on the new {spelling,grammar}-error decorations. While they are accepted but ignored in Chromium 96, you’ll be able to see them in Chromium 98, with our early paint support.

Chromium 96 also makes it possible to change the color of native squiggly lines by setting ‘text-decoration-color’ on either of the new pseudo-elements. This feature, and the features above, are behind another flag:

--enable-blink-features=CSSSpellingGrammarErrors

Charlie’s bird spec lawyerings

I’ve learned a lot of things while working on this project. One interesting lesson was that no matter how clearly a feature is specified, and how much discussion goes into spec details, half the questions won’t become apparent until someone starts building it.

What happens when both highlight and originating content define text shadows? What if multiple highlights do the same? What order do we paint these shadows in? (#3932)
What happens to the originating content’s decorations when highlighted? What happens when highlights define their own decorations? Which decorations get recolored to the foreground color for clarity? What’s the painting order? Does it even mean anything for a highlight to set ‘text-decoration-color’ only? (#6022)
Some browsers invert the author’s ::selection background based on contrast with the foreground color. Should this be allowed, or does it do more harm than good? (#6150)
- What about other “tweaks”? What if a browser needs to force translucency to make its selection highlighting work? (#6853)
- How do we even write reftests if they are allowed? (no issue)
- While we’re talking about testing, how do we even test ::{spelling,grammar}-error without a way to guarantee that some text is treated as an error? (wpt#30863)
How does paired cascade work? Does “use” mean used value? Which properties are “highlight colors”? Do we really mean ::selection only, and color and background-color only? What does it mean for a highlight color to have been “specified by the author”? Does the user origin stylesheet count as “specified”? Do unset and revert count as “specified”? Does unset mean inherit even when the property is not normally inherited? (#6386)
Should custom properties be allowed? What about variable references? Do we force non-inherited custom properties to become inherited like we do for non-custom properties? Should we provide a better way to set custom properties in a way that affects highlight pseudos? (#6264, #6641)
What if existing content relies on implicitly inheriting a highlight foreground color when setting background-color explicitly, or vice versa? Do we need to accommodate this for compat? (#6774)
The spec effectively recommends that ::{spelling,grammar}-error (and requires that ::highlight) force the text color to black by default. Surely we want to not change the color by default? (#6779)
Does color:currentColor point to the next active highlight overlay below, or are inactive highlights included too? What happens when the author tries to getComputedStyle with ::selection? (#6818)
Do decorations “propagate” to descendants in highlights like they would normally? How do we reconcile that with highlight inheritance? How do we ensure that “decorating box” semantics aren’t broken? (#6829)

Squiggly lines

Since landing ‘text-decoration-color’ support for the new pseudos, my colleague Rego has taken the lead on the rest of the core spelling and grammar features, starting with the new ‘text-decoration-line’ values.

Currently, when setting ‘text-decoration-color’ on the pseudos, we change the color, but ‘text-decoration-line’ is still ‘none’, which doesn’t really make sense. This might sound like it required gross hacks, but the style system just gives us a blob of properties, where ‘color’ and ‘line’ are independent. All of the logic that uses them is in paint and layout.

CL:3162169

We started by adding the new values to the stylesheet parser. While highlight painting still needs a lot more work before we can do so, the idea is that eventually the pseudos and decorations will meet in the default stylesheet.

::spelling-error { text-decoration-line: spelling-error; }
::grammar-error { text-decoration-line: grammar-error; }

CL:3194336

Something that’s often neglected in CSS tests is dynamic testing, which checks that the rendering updates correctly when styles are changed by JavaScript, since the easiest and most common way to write a rendering test involves no scripting at all.

In this case, only ::selection had dynamic tests, and only ::selection actually worked correctly, so we then fixed the other pseudos.

CL:3177663

Platform “conventions”

Blink’s squiggly lines look quite different to anything CSS can achieve with wavy or dotted decorations, and they’re painted on unrelated codepaths (more details). We want to unify these codepaths, to make them easier to maintain and help us integrate them with CSS, but this creates a few complications.

The CSS codepath naïvely paints as many bézier curves as needed to span the necessary width, but the squiggly codepath has always painted a single rectangle with a cached texture, which is probably more efficient. This texture used to be a hardcoded bitmap, but even when we made the decorations scale with the user’s dpi, we still kept the same technique, so the approach we use for CSS decorations might be too slow.

Another question is the actual appearance of spelling and grammar decorations. We don’t necessarily want to make them identical to the default wavy or dotted decorations, because it might be nice to tell when, say, a wavy-decorated word is misspelled.

We also want to conform to platform conventions where possible, and you would think there’s at least a consistent convention for macOS… but not exactly. One thing that’s clear is that gradients are no longer conventional.

macOS (compare demo₀)

Safari	Notes	TextEdit	Keynote

CL:3139819

But anyway, if we’re adding new decoration values that mimic the native ones, which codepath do we paint them with? We decided to go down the CSS route — leaving native squiggly lines untouched for now — and take this time to refactor and extend those decoration painters for the needs of spelling and grammar errors.

Precise wavy decorations

To that end, one of the biggest improvements we’ve landed is making wavy decorations start and stop exactly where needed, rather than falling short. This includes the new spelling and grammar decoration values, other than on macOS.

Wavy decorations under ‘letter-spacing’, top version 96, bottom version 97 (demo₀).

CL:3237072

CL:3264203

You may have noticed that the decorations in that last example sometimes extend to the right of “h”. This is working as expected: ‘letter-spacing’ adds a space after letters, not between them, even though it Really Should Not. I tried wrapping the last letter of each word in a span, but then the letter appears to have its own decoration, out of phase with the rest of the word. This is because Blink lacks phase-locked decorations.

Phase-locked decorations

Blink uses an inheritance hack to propagate decorations from parents to children, rather than properly implementing the concept of decorating box. In other words, we paint two independent decorations, whereas we should paint one decoration that spans the entire word. This has been the cause of a lot of bugs, and is widely regarded as a bad move.

Note that we don’t actually have to paint the decoration in a single pass, we only have to render as if that was the case. For example, when testing the same change in Firefox, the decoration appears to jitter near the last letter, which suggests that the decoration is probably being painted separately for that element.

Gecko goes above and beyond with this, even synchronising separate decorations introduced under the same block, which allows authors to make it look like their decorations change color partway through.

A related problem in the highlight painting space is that the spec calls for “recoloring” originating decorations to the highlight foreground color. By making these decorations “lose their color”, we avoid situations where a decoration becomes illegible when highlighted, despite being legible in its original context.

I’ve partially implemented this for ::selection in Chromium 95, by adding a special case that splits originating decorations into two clipped paints with different colors — though not yet the correct colors — while carefully keeping them in phase.

highlight-painting-004 and -ref3, version 97. In this test, the originating element has a red underline, while ::selection introduces a purple line-through. The underline needs to become blue in the highlighted part, to match the ::selection ‘color’, but for now, we match its ‘text-decoration-color’.

To paint the highlighted part of the decoration, we clip the canvas to a rectangle as wide as the background, and paint the decoration in full. To paint the rest, we clip “out” the canvas to the same rectangle, which means we don’t touch anything inside the rectangle.

But how tall should that rectangle be? Short answer: infinity.

Bézier bounding box

Long answer: Skia doesn’t let us clip to an infinitely tall rectangle, so it depends on several things, including ‘text-decoration-thickness’, ‘text-underline-offset’, and in the case of wavy decorations, the amplitude of the bézier curves.

In the code, there was a pretty diagram that illustrated the four relevant points to each “wave” repeated in the decoration. Clearly, it suggested that the pattern in that example was bounded by the control points, but I had no idea whether this was true for all cubic béziers, my terrible search engine skills failed me again, and I don’t like assuming.

/*                   controlPoint1
 *                         +
 *
 *
 *                  . .
 *                .     .
 *              .         .
 * (x1, y1) p1 +           .            + p2 (x2, y2)
 *                          .         .
 *                            .     .
 *                              . .
 *
 *
 *                         +
 *                   controlPoint2
 */

To avoid getting stuck on those questions for too long, and because I genuinely didn’t know how to determine the amplitude of a bézier curve, I went with three times the background height. This should be Good Enough™ for most content, but you can easily break it with, say, a very large ‘text-underline-offset’.

Weeks later, I stumbled upon a video by Freya Holmér answering that very question.

So, how do we get [the bounding] box?

The naïve solution is to simply use the control points of the bézier curve. This can be good enough, but what we really want is the “tight bounding box”; in some cases, the difference between the two is huge.

For now, the code still clips to a fixed three times the background height, but at least we now have some ideas for how to properly measure these decorations:

use the minimum and maximum y values of the control points (naïve)
find better min and max y values by evaluating the derivative at its zeros
use a dedicated function for this purpose like SkDCubic::convexHull?

Cover me!

Writing the reference pages for that test was also a fun challenge. When written the obvious way, Blink would actually fail, because in general we make no attempt to keep any decoration paints in phase.

highlight-painting-004-ref1 and -ref3, version 96.

The ref that Blink ended up matching has five layers. Each layer contains the word “quick” in full with any decorations spanning the whole word, but only part of the layer is shown. This is achieved by an elaborate system of positioned “covers” and “hiders”: the former clips a layer from the right with a white rectangle, while the latter clips a layer from the left by way of right:0 wrapped in overflow:hidden.

Wanna know the best part though? All three refs are identical in Firefox. Someday, hopefully, this will also be true for Blink.

Highlight inheritance

Presto (Opera), uniquely, supported inheritance for ::selection before it was cool, by mapping those styles to synthesised (internal) ‘selection-color’ and ‘selection-background’ properties that were marked as inherited.

Blink also has internal properties for things like :visited links and forced colors, where we need to keep track of both “original” and “new” colors. This works well enough, but internal properties add a great deal of complexity to the code that applies and consumes styles. Now that there are multiple highlight pseudos, supporting a lot more than just ‘color’ and ‘background-color’, this complexity is hard to justify.

To understand the approach we went with, let’s look at how CSS works in Chromium.

Blink style 101

CSS is managed by Blink’s style system, which at its highest level consists of the engine, the resolver, and the ComputedStyle data structure. The engine maintains all of the style-related state for a document, including all of its stylesheet rules and the information needed to recalculate styles efficiently when the document changes. The resolver’s job is to calculate styles for some element, writing the results to a new ComputedStyle object.

ComputedStyle itself is also interesting. Blink recognises over 600 properties, including internal properties, shorthands (like ‘margin’), and aliases (like ‘-webkit-transform’), so most of the fields and methods are actually generated (ComputedStyleBase) with the help of some Python scripts.

These fields are “sharded” into field groups, so we can efficiently reuse style data from ancestors and previous resolver outputs. Some of these field groups are human-defined, like “surround” for all of the margin/border/padding properties, but there are also several raredata groups generated from property popularity stats.

When resolving styles, we usually clone an “empty” ComputedStyle, then we copy over the inherited properties from the parent to this fresh new object. Many of these live in the “inherited” field group, so all we need to do for them is copy a single pointer. At this point, we have the parent’s inherited properties, and everything else as initial values, so if the element doesn’t have any rules of its own, we’re more or less done.

Otherwise, we search for matching rules, sort all of their declarations by things like specificity, then apply the winning declarations by overwriting various ComputedStyle fields. If the field we’re overwriting is in a field group, we need to clone the field group too, to avoid clobbering someone else’s styles.

Blink style 102: pseudo-elements

For ordinary elements, as well as pseudo-elements with a clear place in the DOM tree (e.g. ::before, ::marker), we resolve styles as part of style’s regular tree traversal. We start by updating :root’s styles, then any children affected by the update, and so on. But for other pseudos we usually use a “lazy” approach, where we don’t bother resolving styles unless they are needed by a later phase of the rendering process, like layout or paint.

Let’s say we’re resolving styles for some ordinary element. When we’re searching for matching rules, if we find one that actually matches our ::selection, we make a note in our pseudo bits saying we’ve seen rules for that pseudo, but otherwise ignore the rule.

Once we’re in paint, if the user has selected some text, then we need to know our ::selection styles, so we check our pseudo bits. If the ::selection bit was set, we call our resolver with a special request for pseudo styles, then cache the result into a vector inside the originating element’s ComputedStyle.

This is how ::selection used to work, and at first I tried to keep it that way.

Status quo

My initial solution was to make paint pass in a custom inheritance parent with its style request. Normally pseudo styles inherit from the originating element, but here they would inherit from the parent’s highlight styles, which we would obtain recursively. Then in the resolver, if we’re dealing with a highlight, we copy non-inherited properties too.

On the surface, this worked, but to make it correct, we had to work around an optimisation where the resolver would bail out early if there were no matching rules. Worse still, we had to bypass the pseudo cache entirely. While we already had to do so under :window-inactive, the performance penalty there was at least pretty contained.

PS7

If we copy over the parent’s inherited properties as usual, and for highlights, copy the non-inherited properties too, that more or less means we’re copying all the fields, so why not do away with that and just clone the parent’s ComputedStyle?

PS10

The pseudo cache is only designed for pseudos whose styles won’t need to change between the originating element’s style updates. For most pseudos, this is true anyway, as long as we bypass the cache under pseudo-classes like :window-inactive.

These caches are rarely actually cleared, but when the next update happens, the whole ComputedStyle — including the cache — gets discarded. Caching results with custom inheritance parents is usually frowned upon, because changing the parent you inherit your styles from can yield different styles. But for highlights, we will always have the same parent throughout an update cycle, so surely we can use the cache here?

PS13

…well, yes and no.

Given an element that inherits a bunch of highlight styles, the initial styles are correct. But when those inherited values change in some ancestor, our highlight styles fail to update! This is a classic cache invalidation bug. Our invalidation system wasn’t even the problem — it’s just unaware of lazily resolved styles in pseudo caches. This is usually fine, because most pseudos inherit from the originating element, but not here.

Storing highlight styles

With the pseudo cache being unsuitable for highlight styles, we needed some other way of storing them. Only a handful of properties are allowed in highlight styles, so why not make a dedicated type with only those fields?

The declarations and basic methods for CSS properties are entirely generated, so let’s write some new templates…

{% macro declare_highlight_class(name, fields, field_templates): -%}
class {{name}} : public RefCounted<{{name}}> {
 public:
  static scoped_refptr<{{name}}> Create() { /* ... */ }
  scoped_refptr<{{name}}> Copy() const { /* ... */ }
  bool operator==(const {{name}}& other) const { /* ... */ }
  bool operator!=(const {{name}}& other) const { /* ... */ }
  {% for field in fields %}
  {{declare_storage(field)}}
  {% endfor %}
  {% for field in fields %}
  {{field_templates[field.field_template]
      .decl_public_methods(field.without_group())
    |indent(2)}}
  {% endfor %}
 private:
  {{name}}();
  CORE_EXPORT {{name}}(const {{name}}&);
};
{%- endmacro %}

…then use them in the ComputedStyleBase template.

{{declare_highlight_class(
    'StyleHighlightData',
    computed_style.all_fields
        |sort(attribute='name')
        |selectattr('valid_for_highlight')
        |list,
    field_templates)
  |indent(2)}}

PS16

Trouble is, all of the methods that apply and serialise property values — and there are hundreds of them — take a ComputedStyle, not some other type.

const blink::Color Color::ColorIncludingFallback(
    bool visited_link,
    const ComputedStyle& style) const { /* ... */ }

const CSSValue* Color::CSSValueFromComputedStyleInternal(
    const ComputedStyle& style,
    const LayoutObject*,
    bool allow_visited_style) const { /* ... */ }

Combined with the fact that our copy-on-write field groups mitigate a lot of the wasted memory (well hopefully anyway), we quickly abandoned this dedicated type.

PS25

We then optimised the top-level struct a bit, saving a few pointer widths by moving the four highlight style pointers into a separate type, but this was still less than ideal. We were widening ComputedStyle by one pointer, but the vast majority of web content doesn’t use highlight pseudos at all, and ComputedStyle and ComputedStyleBase are very sensitive to size changes. To give you an idea of how much it matters, Blink even throws a compile-time error if the size inadvertently changes!

struct SameSizeAsComputedStyleBase {
  SameSizeAsComputedStyleBase() { Alias(&pointers); Alias(&bitfields); }
 private:
  void* pointers[9];
  unsigned bitfields[5];
};

struct SameSizeAsComputedStyle : public SameSizeAsComputedStyleBase,
                                 public RefCounted<SameSizeAsComputedStyle> {
  SameSizeAsComputedStyle() { Alias(&own_pointers); }
 private:
  void* own_pointers[1];
};

ASSERT_SIZE(ComputedStyle, SameSizeAsComputedStyle);

To move highlights out of the top-level and into a raredata group, we had to get rid of all the fancy generated code and Just write a plain struct, which has the added benefit of making the code easier to read. Luckily, we were only using that code to loop through the four highlight pseudos at this point, not dozens or hundreds of properties.

Then all we needed was a bit of JSON to tell the code generator to add an “extra” field, and find an appropriate field group for us ("*"). Because this field is not for a popular CSS property, or a property at all really, it automatically goes in a raredata group.

[{
  name: "HighlightData",
  inherited: true,
  field_template: "external",
  type_name: "StyleHighlightData",
  include_paths: ["third_party/blink/renderer/core/style/style_highlight_data.h"],
  default_value: "",
  wrapper_pointer_name: "DataRef",
  field_group: "*",
  computed_style_custom_functions: ["initial", "getter", "setter", "resetter"],
}]

PS35

Single-pass resolution

With our new storage ready, we now needed to actually write to it. We want to resolve highlight styles as part of the regular style update cycle, so that they can eventually benefit from style invalidation.

Looking at the resolver, I thought wow, there does seem to be a lot of redundant work being done when resolving highlight styles in a separate request, so why not weave highlight resolution into the resolver while we’re at it?

PS36

@@ third_party/blink/renderer/core/css/css_selector.h @@
   enum RelationType {
+    kHighlights,
@@ third_party/blink/renderer/core/css/css_selector.cc @@
       case kShadowSlot:
+      case kHighlights:
@@ third_party/blink/renderer/core/css/element_rule_collector.h @@
   MatchedRule(const RuleData* rule_data,
               unsigned style_sheet_index,
               const CSSStyleSheet* parent_style_sheet,
+              absl::optional<PseudoId> highlight)
@@ third_party/blink/renderer/core/css/resolver/match_result.h @@
   void AddMatchedProperties(
       const CSSPropertyValueSet* properties,
       unsigned link_match_type = CSSSelector::kMatchAll,
       ValidPropertyFilter = ValidPropertyFilter::kNoFilter,
+      absl::optional<PseudoId> highlight = absl::nullopt);
@@ ... @@
   const MatchedPropertiesVector& GetMatchedProperties(
+      absl::optional<PseudoId> highlight) const {
+    DCHECK(!highlight || highlight_matched_properties_.Contains(*highlight));
+    return highlight ? *highlight_matched_properties_.at(*highlight)
                      : matched_properties_;
@@ ... @@
   MatchedPropertiesVector matched_properties_;
+  HeapHashMap<PseudoId, Member<MatchedPropertiesVector>>
+      highlight_matched_properties_;
@@ third_party/blink/renderer/core/css/resolver/style_cascade.h @@
   void Apply(CascadeFilter = CascadeFilter());
+  void ApplyHighlight(PseudoId);
@@ third_party/blink/renderer/core/css/resolver/style_cascade.cc @@
 const CSSValue* ValueAt(const MatchResult& result,
+                        absl::optional<PseudoId> highlight,
@@ ... @@
 const TreeScope& TreeScopeAt(const MatchResult& result,
+                             absl::optional<PseudoId> highlight,
                              uint32_t position) {

In general we must find a less intrusive way to implement this. We can not have |highlight| params on everything.

— andruud, Blink style owner

You know what? Fair enough.

⭯ PS35

Multi-pass resolution

Element::Recalc{,Own}Style are pretty big friends of the style system. They drive the style update cycle by determining how the tree has changed, making a resolver request for the element, and determining which descendants also need to be updated.

This makes them the perfect place to update highlight styles. All we need to do is make an additional resolver request for each highlight pseudo, store it in the highlight data, and bob’s your uncle.

StyleRecalcChange Element::RecalcOwnStyle(
    const StyleRecalcChange change,
    const StyleRecalcContext& style_recalc_context) {
  // ...
  if (new_style) {
    StyleHighlightData* highlights = new_style->MutableHighlightData();
    if (new_style->HasPseudoElementStyle(kPseudoIdSelection)) {
      ComputedStyle* parent = ParentComputedStyle()->HighlightData()->Selection();
      StyleRequest request{kPseudoIdSelection, parent};
      highlights->SetSelection(StyleForPseudoElement(style_recalc_context, parent));
    }
    // kPseudoIdTargetText
    // kPseudoIdSpellingError
    // kPseudoIdGrammarError
    // ...
  }
  // SetComputedStyle(new_style);
  // ...
}

PS43

Pathology in legacy

So far, I had been writing this patch as a replacement for the old inheritance logic, but since we decided to defer highlight inheritance for ::highlight to a later patch, we had to undelete the old behaviour and switch between them with a Blink feature.

Another reason for the feature gate was performance. Of the pages in the wild already using highlight pseudos, most of them probably use universal ::selection rules, if only because of how useless the old model was for more complex use cases.

::selection { color: lime; background: green; }

But ::selection isn’t magic — it literally means *::selection, which makes the rule match everywhere in the ::selection tree. When highlight inheritance is enabled, that means we end up cloning highlight styles for each descendant, only to apply the same property values, which wastes time and memory.

The reality is a bit more complicated than this, because ‘color’ and ‘background-color’ are actually in field groups that would also need to be cloned.

Under the old model, where lack of inheritance made this necessary, *::selection rules suffered from roughly the same problem, but the lazy style resolution meant that time and memory was only wasted on the elements directly containing selected content.

As a result, this will need to be fixed before we can enable the feature for everyone.

PS51

Paired cascade

Next we tried to reimplement paired cascade. For compatibility reasons, ::selection has special logic for the browser’s default ‘color’ and ‘background-color’ (e.g. white on blue), where we only use those colors if neither of them were set by the author. Otherwise, they default to initial values, usually black on transparent.

	default on default
+	`::selection { background: yellow; }`
=	initial on yellow

The spec says so in a mere 22 words:

The UA must use its own highlight colors for ::selection only when neither color nor background-color has been specified by the author.

Brevity is a good thing, and this seemed clear enough to me in the past. But once I actually had to implement it, I had questions about almost every word (#6386). While they aren’t entirely resolved, we’ve been getting pretty close over the last few weeks.

PS52

Who’s got green?

Much of the remaining work was to fix test failures and other bugs. These included crashes under legacy layout, since we only implemented this for LayoutNG, and functional changes leaking out of the feature gate. One of the reftest failures was also interesting to deal with. Let’s minimise it and take a look.

<!doctype html><meta charset="utf-8">
<title>active selection and background-color (basic)</title>
<style>
    main { color: fuchsia; background: red; }
    main::selection { background: green; }
</style>
<p>Pass if text is fuchsia on green, not fuchsia on red.
<main>Selected Text</main>
<script>/* selectNodeContents(main); */</script>

In the past, the “Selected Text” would render as fuchsia on green, and the test passes. But under highlight inheritance it fails, rendering as initial (black) on green, because we now inherit styles in a tree for each pseudo, not from the originating element.

Selected Text → Selected Text

So if the test is wrong, then how do we fix it? Well… it depends on the intent of the test, at least if we want to Do The Right Thing and preserve that. Clearly the primary intent of the test is ‘background-color’, given the <title>, but tests can also have secondary, less explicit intents. In this case, the flavour text¹ even mentions fuchsia!

It might have helped if the test had a <meta name=assert>, an optional field dedicated to conveying intent, but probably not. Most of the assert tags I’ve seen are poorly written anyway, being a more or less verbose adaptation of the title or flavour text, and there’s a good chance that the intent for fuchsia (if any) was simply to inherit it from the originating element, so we would still need to invent a new intent.

We could change the reference to initial (black) on green, which would serve as a secondary test that we don’t inherit from the originating element, or remove the existing ‘color’, which would serve as a secondary test for paired cascade. But I didn’t think it through that far at the time, so I gave ::selection a new ‘color’, achieving neither.

 main::selection {
+ color: aqua;
  background: green; }
 </style>
 <p>Pass if text is
- fuchsia
+ aqua
 on green, not fuchsia on red.

Because the selected and unselected text colors were now different, I created another test failure, though only under legacy layout. The reference for this test was straightforward: aqua on green, no mention of fuchsia. This makes sense on the surface, given that all of the text under test was selected.

In this case, the tip of the “t” was crossing the right edge of the selection as ink overflow, and were carefully painting the overflow in the unselected color. The test would have failed under LayoutNG too, if not for an optimisation that skips this technique when everything is selected. Let me illustrate with an exaggerated example:

This behaviour is generally considered desirable, at least when there are unselected characters, so Blink isn’t exactly wrong here. It’s definitely possible to make the active-selection tests account for this — and the tools to do so already exist in the Web Platform Tests — but I don’t have the time to pursue this right now.

PS76

What now?

After the holidays, we plan to:

Resolve the remaining spec issues. These issues are critical for finishing highlight inheritance and allowing highlights to add their own decorations.
Port ::selection’s painting logic to the other highlights. We might even use this as an opportunity to roll ::selection into the marker system.

Other work needed before we can ship the spelling and grammar features:

Ship highlight inheritance. This includes addressing any spec resolutions, fixing the performance issues, and adding devtools support.
Integrate spelling and grammar errors with decoration painting (bug 1257553).
Make automated testing possible for spelling and grammar errors (wpt#30863).

Special thanks to Rego, Frédéric (Igalia), Rune, andruud (Google), Florian, fantasai (CSSWG), Emilio (Mozilla), and Fernando (Microsoft). We would also like to thank Bloomberg for sponsoring this work.

This is an automated reftest, so the instructions in <p> have no effect on the outcome. We require them anyway, because they add a bit of redundancy that helps humans understand and verify the test’s assertions. ↩

Group sharing on a FreeBSD home server

2021-09-24T15:30:00+00:00

My partner and I share a home server for our storage needs, running on FreeBSD 12 with ZFS. We have our own users, delan and aria, and a group (delanria) that in theory we can use for common areas, like our software collection. Services like torrent clients and media libraries have their own users too, and all of these users need to write to things others have created. This was easier said than done.

Unix permissions 101
- Ownership and creation
Shared storage
Login classes
Fixing sudo(8)
Fixing services
Fixing samba(8)
Funny execute bits
What did we learn?

Unix permissions 101

Files are owned by a user (the owner) and a group (the group). Users belong to groups, and each group they join grants them access to any files owned by that group. Processes run as a user (euid) and a group (egid), more or less.

Traditional permissions for files consist of twelve bits: nine for whether the owning user (u), owning group (g), and others (o) can read (r), write (w), or execute (x); three for controlling execution behaviour. Of the latter three, setuid (u+s) makes the file run with its owning user as euid, setgid (g+s) does that for owning group and egid, and sticky (t) was mostly only used historically.

Directories repurpose the execute bits for “search”, which essentially means whether you can blindly access children (subject to the children’s permissions). This is distinct from the read bits, which control whether you can list children, and write bits, for creating and deleting children. They also repurpose the sticky bit for “restricted deletion”, where children can only be deleted by their owners, rather than anyone who can write to the directory (useful for /tmp).

This permission model frankly sucks. It’s clear that these twelve bits are a messy and leaky abstraction over filesystem access rights. But I’ve never used ACL:s outside of Windows, and I don’t intend to change that any time soon. I know that ls(1) indicates ACL:s with a plus (drwxrwxrwx+), that there are apparently “POSIX” and “NFS” flavours of ACL:s, and… that’s pretty much it.

Ownership and creation

When you create a file, it’s owned by you (euid), but the owning group depends. Unfortunately, sometimes it’s your “current” group (egid), which is controlled by newgrp(1) and defaults to your “login” group, which is usually something like “users” or “staff” or a group with the same name as your user.

FreeBSD makes group ownership of directories easy, because new files always inherit the owning group, so that “sometimes” is never! But on Linux, they’re only inherited when the parent is setgid, and otherwise take their owning group from you (egid).

Unix requires the nine main permission bits upfront when creating a file or directory, and convention is for programs that don’t know or care about these bits to give 666₈ (u+rw, g+rw, o+rw) or 777₈ (u+rwx, g+rwx, o+rwx) respectively.

// create a directory (u+rwx, g+rwx, o+rwx)
mkdir("path/to/foo", 0777);
// create an executable file (u+rwx, g+rwx, o+rwx)
creat("path/to/bar", 0777);
// create a non-executable file (u+rw, g+rw, o+rw)
creat("path/to/baz", 0666);

The bits are then filtered by the umask, an environmental setting that almost always defaults to 022₈ (g-w, o-w). This is where the common permissions of 644₈ (u+rw, g+r, o+r) for files and 755₈ (u+rwx, g+rx, o+rx) for directories comes from.

$ ls -l path/to
drwxr-xr-x  [...]  foo
drwxr-xr-x  [...]  bar
drw-r--r--  [...]  baz

Shared storage

Let’s say we have some groups.

$ rg delan,aria /etc/group
wheel:*:0:root,delan,aria
delanria:*:1003:delan,aria
sonarr:*:351:delan,aria
radarr:*:352:delan,aria
_sabnzbd:*:350:delan,aria,sonarr,radarr
qbittorrent:*:850:delan,aria,sonarr,radarr

Let’s also say I have a notes directory and a software directory, both of which are shared with my partner including write permissions (g+w).

$ ls -la /ocean/notes
drwxrwxr-x  [...]  delan  delanria  [...]  .

$ ls -la /ocean/software
drwxrwxr-x  [...]  delan  delanria  [...]  .
drwxrwxr-x  [...]  delan  delanria  [...]  accounting
drwxrwxr-x  [...]  delan  delanria  [...]  benchmarks
drwxrwxr-x  [...]  delan  delanria  [...]  drivers
                                           ...

When adding a new note, or adding a category of software, the immediate problem we run into is that a umask of 022₈ (g-w, o-w) makes the new directory group-read-only (g+rx).

delan@storage$ echo bread > /ocean/notes/buy
delan@storage$ mkdir /ocean/software/fonts
aria@storage$ echo soul > /ocean/notes/sell
aria@storage$ mkdir /ocean/software/games

$ ls -ld /ocean/notes/{,buy,sell}
drwxrwxr-x  [...]  delan  delanria  [...]  .
-rw-r--r--  [...]  delan  delanria  [...]  buy
-rw-r--r--  [...]  aria   delanria  [...]  sell

$ ls -ld /ocean/software/{,fonts,games}
drwxrwxr-x  [...]  delan  delanria  [...]  .
drwxr-xr-x  [...]  delan  delanria  [...]  fonts
drwxr-xr-x  [...]  aria   delanria  [...]  games

This prevents the other partner from changing notes or adding software, unless we periodically “fix up” the permissions in common areas.

delan@storage$ echo out >> /ocean/notes/sell
zsh: permission denied: /ocean/notes/sell

aria@storage$ cd /ocean/software/fonts
aria@storage$ curl -sSO https://fonts.example.com/iosevka.otf
curl: (23) Failure writing output to destination

$ chmod -R g+w /ocean/{notes,software}

We can avoid these problems by setting our umask to 002₈ (o-w), which is controlled by each user’s login class. In my login.conf(5), many settings are defined only in the “default” login class, including the umask, which are directly inherited by the other classes.

$ cat /etc/login.conf
default:\
	:passwd_format=sha512:\
	:...:\
	:umask=022:
standard:\
	:tc=default:
daemon:\
	:...:\
	:tc=default:
...

Put a pin in the line with “passwd_format”, we’ll need that later.

If we change the umask setting and rebuild, logging in yields the expected umask, fixing the scenarios above… except when switching users with sudo(8). Switching users with su(1) via sudo(8) works as expected, so what gives?

# vim /etc/login.conf
# cap_mkdb /etc/login.conf
# su -l delan
delan@storage$ umask
002

delan@storage$ sudo -iu aria umask
022

delan@storage$ sudo su -l aria -c 'umask'
002

Fixing sudo(8)

At first, the only way we can get the expected umask when switching users with sudo(8) is to explicitly ask for a login class, such as the user’s default class:

delan@storage$ sudo -iu aria -c - umask
002

This is because of defaults for three sudoers(5) settings: use_loginclass, umask_override, and umask. These settings mean that sudo(8) forms the new umask as follows.

use_loginclass is off, so we start by taking our umask from the environment in which sudo was invoked, in this case, 002₈. umask_override is off, so our next step will do a bitwise OR, rather than replacing our umask entirely. The umask setting is 022₈, so our final umask is 002₈ OR 022₈, which is… well… 022₈.

To fix this, we can turn use_loginclass on, which usually¹ takes our initial umask from the user’s default login class, or turn the umask setting off, which tells sudo(8) not to modify that initial umask.

Defaults use_loginclass  # option 1: use_loginclass on
Defaults !umask          # option 2: umask off

# option 3: umask off, but in a way that makes no sense
# (seriously, sudo authors, why did you add this case?
#  this just makes it impossible to actually set 0777!)
Defaults umask=0777

Fixing services

According to the rc.subr(8) manual, services ostensibly use the “daemon” login class by default. But despite setting the umask in all of our login classes to 002₈ (o-w), services like qbittorrent and sabnzbd continue to create things group-read-only. Once again, what gives?

As the manual says, the ${name}_login_class is used with ${name}_limits. The former points to a login class containing our initial set of resource limits, and the latter overrides those limits, by way of limits(1). Indeed, if we look under the hood, the login class is only ever used in the arguments passed to limits(1).

$ rg _login_class /etc/rc.subr
#   ${name}_login_class n   Login class to use, else "daemon".
       _prepend=\$${name}_prepend  _login_class=\${${name}_login_class:-daemon} \
                  _doit="$_cd limits -C $_login_class $_limits $_doit"

But resource limits aren’t the only settings defined by login classes, which as we saw earlier, also says things like “passwd_format is sha512”! So this begs the question: is umask considered a resource limit for the purposes of limits(1)?

root@storage$ umask 000; ulimit 69
root@storage$ limits sh -c 'umask; ulimit'
0000
69

root@storage$ limits -C daemon sh -c 'umask; ulimit'
0000
unlimited

No. In fact, not only is the umask of the “daemon” login class not consulted when running a service, but rc(8) and init(8) themselves don’t even run in a login class. You can see the former for yourself by adding a couple of lines to /etc/rc and rebooting.

$ head -2 /etc/rc
#!/bin/sh
umask=$(umask)          # add this

$ tail -2 /etc/rc
echo "umask is $umask"  # add this
exit 0

As for the latter, you can write a script that spews out the umask, then reboot and tell loader(8) to tell init(8) to immediately exec that script.

$ sudo chmod +x /root/umask
$ cat /root/umask
#!/bin/sh
while :; do umask; done

loader> set init_exec=/root/umask
loader> boot

In both cases, the umask is 022₈ (g-w, o-w). This is because login classes aren’t magic, nor are they omnipotent! The only processes subject to them are those spawned by login(1), or things like login(1)², such as su(1), sudo(8), or sshd(8).

All other processes ultimately inherit their umask from the “kernel” process (pid 0), whose umask is hardcoded to, you guessed it, 022₈ (g-w, o-w).

// sys/kern/init_main.c
static void
proc0_init(void *dummy __unused)
{
	struct proc *p;
	// [...]
	p->p_pd = pdinit(NULL, false);
	// [...]
}

// sys/kern/kern_descrip.c
struct pwddesc *
pdinit(struct pwddesc *pdp, bool keeplock)
{
	// [...]
	newpdp->pd_cmask = CMASK;
	// [...]
}

// sys/sys/param.h
#define	CMASK	022		/* default file mask: S_IWGRP|S_IWOTH */

So getting back on task, how do we run our services with another umask? One way might be to add a line setting the umask to the beginning of /etc/rc, but this is rather drastic, and the security of this… smells questionable.

The solution I’ve settled on is to sneak a umask command into the per-service rc.conf(5) for specific services. This works because in rc.subr(5), load_rc_config executes /etc/rc.conf “if it has not yet been read in” (whatever that means), then executes /etc/rc.conf.d/foo if it exists. Most of the time, these files contain variables only, but they’re just shell scripts. It’s shell scripts all the way down.

$ ls -l /etc/rc.conf.d
-rw-r--r--  [...]  _umask
lrwxr-xr-x  [...]  qbittorrent -> _umask
lrwxr-xr-x  [...]  radarr -> _umask
lrwxr-xr-x  [...]  sabnzbd -> _umask
lrwxr-xr-x  [...]  sonarr -> _umask

$ cat /etc/rc.conf.d/_umask
umask 002

Fixing samba(8)

That rc.conf(5) hack doesn’t work for samba(8), where the umask that applies to things created by clients is controlled by internal configuration, just like sudo(8). In this case, “create mode” and “directory mode” are the settings to change.

[global]
create mode = 0775     # like umask 002
directory mode = 0775  # like umask 002

While we’re at it, if you want to set execute bits on all new files, you can use “force create mode”. Aria likes this, but I’m not so sure.

[global]
# force create mode = 0111  # like ugo+x

Funny execute bits

At this point, it looked like we were done, but something caught my eye when Aria created some files for testing. The files had owning user execute (u+x), but not the other two execute bits. For the last bloody time, what gives?

$ ls -l
-rwxrw-r--  [...]  aria  delanria  [...]  foo.txt

Turns out there’s an old samba(8) feature that repurposes the three execute bits for the three legacy DOS attributes respectively: archive, system, hidden. The setting for the first execute bit (“map archive”) is on by default, and Windows had created the file with the archive bit on, hence owning user execute (u+x)!

Nowadays extended attributes are a better way to store those attributes, which is on by default as “store dos attributes”. But I was worried that I would need to do something messy like vfs_streams_xattr(8), remembering that FreeBSD and ZFS don’t support xattrs, at least not until FreeBSD 13. After all, ZFS says that they’re not supported!

root@storage$ zfs set xattr=on ocean
property 'xattr' not supported on FreeBSD: permission denied

Turns out that error is misleading, and xattrs more or less work fine.

root@storage$ lsextattr user foo.txt
foo.txt DOSATTRIB

(archive bit only)
root@storage$ getextattr -x user DOSATTRIB foo.txt
foo.txt 00 00 04 00 04 00 00 00 51 00 00 00 20
> 00 00 00 44 a8 2c de 2d b1 d7 01 44 a8 2c de
> 2d b1 d7 01

(archive + system + hidden)
root@storage$ getextattr -x user DOSATTRIB foo.txt
foo.txt 00 00 04 00 04 00 00 00 51 00 00 00 23
> 00 00 00 44 a8 2c de 2d b1 d7 01 44 a8 2c de
> 2d b1 d7 01

(none of those attributes)
root@storage$ getextattr -x user DOSATTRIB foo.txt
foo.txt 00 00 04 00 04 00 00 00 51 00 00 00 00
> 00 00 00 44 a8 2c de 2d b1 d7 01 44 a8 2c de
> 2d b1 d7 01

To top it all off, the smb.conf(5) manual says that “store dos attributes” should have automatically disabled the execute-bit-based attribute mapping, but they actually don’t. I guess the manual was wrong.

[global]
map archive = no
map system = no
map hidden = no

What did we learn?

FreeBSD gives us owning group inheritance for free, but inheriting the group-writable bit requires changing the umask. This can be done for human use by way of the login classes in login.conf(5), with special tweaks needed for sudo(8), but services only use login classes for resource limits, not the umask, which has a hardcoded default of 022₈ (g-w, o-w).

Most services can have the desired umask set imperatively in /etc/rc.conf.d, but samba(8) needs to be configured with its own “create mask” and “directory mask” settings.

When in doubt, read the source code.

If the umask setting is explicitly set (other than to turn it off), then the initial umask is always taken from the invoking environment, not login classes as you might expect from turning use_loginclass on. This includes explicitly setting it to 022₈, and yes, that means your 022₈ and the default 022₈ are different. sudo(8) is complicated. ↩
More precisely, things that call setusercontext(8) or setclasscontext(8) with flags containing LOGIN_SETUMASK. ↩

Chromium spelling and grammar features

2021-05-17T10:30:00+00:00

Back in September, I wrote about my wonderful internship with Igalia’s web platform team. I’m thrilled to have since joined Igalia full-time, starting in the very last week of last year. My first project has been implementing the new CSS spelling and grammar features in Chromium. Life has been pretty hectic since Aria and I moved back to Perth, but more on that in another post. For now, let’s step back and review our progress.

The squiggly lines that indicate possible spelling or grammar errors have been a staple of word processing on computers for decades. But on the web, these indicators are powered by the browser, which doesn’t always have the information needed to place and render them most appropriately. For example, authors might want to provide their own grammar checker (placement), or tweak colors to improve contrast (rendering).

To address this, the CSS pseudo and text decoration specs have defined new pseudo-elements ::spelling-error and ::grammar-error, allowing authors to style those indicators, and new text-decoration-line values spelling-error and grammar-error, allowing authors to mark up their text with the same kind of decorations as native indicators.

Current status
CSS unification
Fifteen years in the making
Highlight painting
Processing model
Stay tuned!

Current status

I’ve sent an Intent to Prototype, as well as requests for positions from Mozilla and Apple.

I’ve landed a patch that paves the way for ::spelling-error + ::grammar-error support internally, and I’m hopefully(!) around halfway done with implementing both the new painting rules and the new processing model.

The spec updates, led by Florian Rivoal, were largely done by the end of 2017. As the first impl of both the features themselves and much of the underlying highlight specs, there were always going to be questions and rough edges to be clarified.

Two issues were raised before we even started, I’ve since sent in another two, and I’ll need to raise at least two more by the time we’re done. I’ve also landed three WPT patches, including three new tests and fixes for countless more.

highlight-painting-003.html

In the course of my work on these features, I’ve already fixed at least two other bugs that weren’t of my own creation, and reported four more:

1171741	Selecting text causes emphasis marks to be painted twice
1172177	Erroneous viewport-size-dependent clipping of some text shadows
1176649	text-shadow paints with incorrect offset for vertical scripts in vertical writing modes
1180068	text-shadow erroneously paints over text proper in mixed upright/sideways fragments

CJK CSS unification

My colleague Rego noticed that the squiggly lines for spelling and grammar errors look slightly different to a naïve red or green wavy underline. How, why, and should we unify squiggly and wavy lines? Some further investigation revealed that the two kinds of decorations are drawn very differently with completely separate code paths.

non-macOS (demo₀)

100%	200%

Left (bolder text): nearest wavy decorations.
Right (lighter text): native squiggly lines.

The case for unifying squiggly and wavy lines became a lot more complicated too. For example, our squiggly lines are actually dots on macOS. More specifically, they are round dots with an alpha gradient, matching the platform’s native controls. These details are beyond what can be expressed in terms of a dotted underline, so if we were to unify by making squiggly lines equivalent to such a decoration, we would lose that benefit.

macOS (demo₀)

100%	200%

Left (bolder text): nearest dotted decorations.
Right (lighter text): native squiggly lines.

The spec doesn’t require that spelling-error and grammar-error lines be expressible in terms of other decoration lines, so unification won’t block shipping. I decided it would be best to revisit this once I landed some patches and familiarised myself with the code.

Fifteen years in the making

::spelling-error and ::grammar-error are defined as highlight pseudo-elements, together with ::selection and ::target-text. The spec’s processing model and rendering rules are both very different to how ::selection (or ::target-text) has been implemented in any browser so far. Now that we’re implementing more than just the first couple of pseudos, we really ought to comply with the new spec, which complicates our job somewhat.

I’ll talk about ::selection a fair bit below, because most of the spec discussion I found happened before the others were defined, going back as far as 2006. Highlight pseudos like ::selection are tricky because they aren’t tree-abiding: the selected parts of the document aren’t generally a child of any one element.

But even then, how hard could it be?

What is ::selection? How does it interact with other pseudo-elements? Is it a singleton, or does each element have a ::selection pseudo-element? How do we reconcile the ::selection “tree”, if any, with the element tree?
Can child ::selection styles override parent ::selection styles? What about the child’s “real element” styles? How exactly do parent ::selection styles propagate to child ::selection styles? Do we use a tweaked cascade or tweaked inheritance?
What happens when authors specify ::selection styles that affect layout? What about styles that rely on how ::selection relates to the element tree, like outline or translucent background-color?
What happens when child ::selection styles specify only color or only background-color but not both? Does the other inherit as usual? If we want a special case tying these two properties together, how does it interact with other properties?
Does the ::selection background-color paint over text, or under it? What about “replaced” content like images? If we paint over text, do we need to make the author’s color translucent, and if so, how?
Is text in the ::selection color painted in addition to, or instead of, the same text in its original color? What about background-color?
Can the default UA stylesheet describe the platform’s ::selection style? How?
How naughty were browsers that implemented ::selection without a -vendor-prefix before it was standardised? Are vendor prefixes even a good idea?
Most importantly, how do we introduce a new processing model and rendering rules without breaking existing content?

For answers to most of these questions, check out my notes₅.

By the time I started to understand the problem space, two weeks had passed.

Pretty intense for my very first foray into www-style!

Highlight painting

The current spec isolates each highlight pseudo into an “overlay”, and allows each of them to have independent backgrounds, shadows, and other decorations.

Like other browsers, Chromium implemented an older model, where matching ::selection rules are only used to change things like the text color and shadows (except for background-color, which has always been independent).

But the closer I looked, the deeper the problems ran.

Shadows and backgrounds

everyone’s shadow code is complete made-up horseshit but mostly i blame the fact that someone decided to add ‘shadow’ to the (very small!) special list of styles ::selection could modify

— Gankra, 2021

I whipped up a quick demo₃ with some backgrounds and shadows, and the result was… not good. “So the originating text shadow (yellow) paints over the ::selection background (grey), except when it paints under, and sometimes it even paints over the text (black)? Why is the ::selection shadow clipped to the ::selection background? What?”

highlight-painting-001.html (based on demo₃)

Some of these were easier to fix than others. To fix backgrounds, we essentially push the code that paints the background waaaaay down NGTextFragmentPainter, so that it’s before painting the selected text but after pretty much everything else. We then fix shadows similarly, reordering the text paints from “before with shadows, after with shadows, selected with shadows” to an order that keeps shadows behind text.

These initial fixes are now live in Chromium 90, but we still need to deal with the ::selection shadow clipping. What’s up with that?

Shadow clipping

The weird shadow clipping was a side effect of how we ensured that the ::selection text color changes exactly where the ::selection background starts:

we clip out and paint the selected text in original color, then
we clip (in) and paint the selected text in ::selection color.

This is useful for both subtle reasons, like ink overflow…

demo₆

…and not so subtle reasons, like allowing the user to clearly and precisely select graphemes in ligature-heavy languages like Sorani. In this example, یلا is three letters (îla), but only two glyphs. This isn’t explicitly required by any spec, but it’s definitely intentional.

demo₆

If you use Chromium, you may notice that the ref for that demo appears to select more text. What we’re really doing with ::selection painting is pretending that ligatures are divisible into horizontal parts and guessing how wide each part is. Current font technology just doesn’t provide the metadata to do this more “correctly”.

Firefox always allows splitting ligature styles, including with real elements, and there are at least two good arguments in favour of this approach. Chromium has (reasonably) decided that while the technique is ok for ::selection, perhaps even desirable, it’s not the way to go for ordinary markup.

demo₄

But anyway, back to the point at hand. text-shadow means “paint the text again, under the text proper, with these colors and offsets”. We want to clip the ::selection shadow for the same reasons we clip the text proper in ::selection color, but the coordinates need to be offset for each shadow. That we don’t is the bug here.

demo₆

When painting the ::selection shadow (blue), we need to clip the canvas to the dotted line, but we were actually clipping to the solid line.

Consensus seems to be that not doing so is undesirable, and in theory, fixing this would be straightforward, but in practice… 😵‍💫

The first confounding factor was that NGTextFragmentPainter and NGTextPainter were… a tangled mess. Even the owners weren’t sure this was the most helpful architecture:

// TODO(layout-dev): Does this distinction make sense?
class CORE_EXPORT NGTextPainter : public TextPainterBase { /* ... */ }

Years of typographical features have been duct-taped on without a systemic approach to managing complexity, including decorations, shadows, ellipses, background clipping, RTL text, vertical text, ruby text, emphasis marks, print rendering, drag-and-drop rendering, selections, highlights, “markers”, and SVG features like stroke and fill.

A third of the logic was in TextPainterBase, so good luck not breaking legacy. Shadows were painted with a now-deprecated Skia feature called a DrawLooper, which allows you to repeat a procedure a bunch of times with different tweaks, such as canvas transformations and color changes. It’s almost specifically designed for shadows, but it’s technically possible to repeat procedures that have nothing to do with drawing text.

// SkCanvas* canvas;
// SkPaint paint;
// SkScalar x, y;
// sk_sp<SkTextBlob> blob;
// sk_sp<SkDrawLooper> looper;
looper->apply(canvas, paint, [&blob, x, y](SkCanvas* c, const SkPaint& p) {
    // procedure to be looped
    c->drawTextBlob(blob, x, y, p);
});

My solution was based on the observation that loopers draw offset shadows by “moving” the canvas with a transform before each iteration, but transforming the canvas only affects subsequent operations. We were clipping the canvas once, before running the looper, but if we could somehow reclip the canvas after each transform, the clip region would “move” together with each shadow, and we wouldn’t even need to change the coordinates!

I prototyped a fix that seemed to handle everything I threw at it, and informed by the challenges that involved, I also refactored out the code for selections, highlights, and markers. Stephen and I decided that adding clipping as a fixed function to DrawLooper made more sense than adding it to the procedure. At the time, this was true.

demo₆

The prototype made my most complex test case (at the time) pass, with the exception of ink overflow color, which was a limitation of my ref (both renderings are acceptable).

I then took a couple weeks off to move to Perth.

Vertical vertigo

“Wait… isn’t the original purpose of vertical writing modes, you know, vertical scripts? I wonder if those work as well as horizontal scripts being rotated sideways…”

“…what? Let’s see what they look like without my patch…”

“…what?”

demo₆

Left: vertical script in vertical-rl, with patch.
Right: same test case, without patch.

Notice how the shadows are offset in the wrong direction. They should be painted southeast of the text proper, but were being painted northeast.

When painting a text fragment with a vertical writing-mode, we rotate the canvas by 90° cw (or ccw for sideways-lr). This is good for horizontal scripts like Latin or Sorani, because they usually need to be painted sideways.

But for vertical scripts like Han, we usually need to keep the canvas unrotated. A single text fragment can contain text in multiple scripts, so we actually achieve this by rotating the canvas back for the parts in vertical scripts.

So far so good right?

This is what we were doing when painting text with vertical scripts and shadows (example limited to a single script and single shadow for simplicity):

Let space be our original “physical” coordinate space
Let offset be the shadow’s offset in space
Let selection be the selection rect coordinates in space
Vertical writing mode, so rotate canvas by 90°, yielding space′
Let offset′ be the result of mapping offset into space′
Let selection′ be the result of mapping selection into space′
Old: clip the canvas to selection′
Configure a DrawLooper that will:
- move the canvas by offset′
- New: clip the canvas to selection′
- draw the text for the shadow
Vertical script, so rotate canvas back by 90°, yielding space″
Run the DrawLooper, which carries out the steps above

The looper is told to move and clip the canvas to offset′ and selection′, which are coordinates in space′, but when it eventually tries to do that, the canvas is in space″.

offset′ being in the wrong space is why shadows have always been painted in the wrong place for vertical scripts. By reordering the clip to selection′ so it happens after the rotation to space″, we were now clipping the canvas to the wrong coordinates, which in turn made the text invisible in our demo₆!

Cursed

Fixing this again proved harder than it seemed on the surface, because text painting in Chromium involves the coordination of four components: paint, shaping, cc, and Skia.

In paint, the text painters are given a “fragment” of text to be painted in a given style. They know the writing mode, because that’s part of the style, but they know very little about the text itself. The first rotation (for the vertical writing mode) happens here, and we configure the DrawLooper here (except for its procedure, which we pass in shaping).

In shaping, we find the best glyphs for each character, and determine what scripts the text fragment is made of, then split the text into “blobs”. The second rotation (for the vertical script) happens here, and we throw in a skew transform too if the text we’re painting is oblique (or fake italic, which is again known only to shaping).

In cc, we expose a Skia-like API that can either dispatch to Skia immediately or collect operations into a queue for later. DrawLooper is in the process of being moved here, because the Skia maintainers don’t want it.

Skia provides a stateful canvas, which more or less creates visible output.

With each canvas transform, existing coordinates need to be remapped into the new space before they can be used again, and we were doing them imperatively in two different components. Worse still, while layout (ng) — the phase that happens before paint — uses the type system to enforce correct handling of coordinates (e.g. PhysicalOffset, LogicalRect), the same is not true for paint onwards.

Everything is in PhysicalRect and friends, often erroneously, or in “untyped” coordinates like FloatRect or SkRect. In one case, a PhysicalOffset is used in both physical and non-physical (rotated for writing-mode) spaces, to refer to two different points at different corners of the text. Here… let me illustrate.

When painting horizontal text in vertical-rl, we rotate the canvas 90° cw around A so that the text’s left descent corner lands on B. The left ascent corner moves from B to C.

That single variable was used to intentionally refer to both B and C at different times in a function, because the coordinates for B in space happen to be numerically the same as those for C in space′. aaaaaaaAAAAAAAAAAAAA-

-AAAAAAAAAAAAA

To be fair, each of these flaws has a reasonable explanation.

Layout is a confusing place where we constantly need to deal with different coordinate spaces, so ideally we would iron everything out so that paint can work purely in physical space. Half the point of types like LogicalRect is to provide getters and setters for concepts like “inline start” and “block end”.

For most of the things we paint, this is ok, even desirable. Rects like ::selection backgrounds must be painted in physical space, so we can round the coordinates to integers for crisp edges. Text is the only exception: the history of computer typography means that vertical text is, to some extent, seen internally as rotated horizontal text.

DrawLooper is handy for painting shadows, and it might^{[citation needed]} even reduce serialisation overhead in cc. But the way we currently configure them, baking coordinates into them before shaping, makes it even harder to handle vertical text correctly.

Last but not least, Chromium’s pre-standard text painting order was “all rects for highlights and markers first, then all texts”. This made the imperative canvas rotations almost acceptable, if you ignore the shadow bugs, because we didn’t need to rotate the canvas back and forth nearly as many times.

Once I moved to Perth, I spent over three weeks trying to find a systemic solution to these problems, but I just wasn’t getting anywhere meaningful. In the interests of working a bit more breadth-first and avoiding burnout, I’ve shelved highlight painting for now.

Processing model

Let’s return to how computed styles for highlight selectors should work.

The consensus was that parent ::selection styles should somehow propagate to the ::selection styles of their children, so authors can use their existing CSS skills to define both general ::selection styles and more specific styles under certain elements. This was unlike all existing implementations, where the only selector that worked the way you would expect was ::selection, that is to say, *::selection.

At first, that “somehow” was by tweaking the cascade to take parent ::selection rules into account. Emilio raised performance concerns with this, so the spec was changed, instead tweaking inheritance to make ::selection styles inherit from parent ::selection styles (and never from originating or “real” elements).

This is what I’m working on now. I’ve got a patch that gets most of the way, first by fixing inherit, then by fixing unset, then with a couple more fixes for styles where the cascade doesn’t yield any value, but there are still a few kinks ahead:

impl work has raised at least three questions that need CSSWG clarification;
we need to optimise it, maybe more than before, to avoid perf regressions;
we still need to check if style invalidation works correctly; and
we probably want new devtools features to visualise highlight inheritance.

Stay tuned!

Beyond my colleagues at Igalia, special thanks go to Stephen, Rune, Koji (Google), and Emilio (Mozilla) for putting up with all of my questions, not to mention Florian and fantasai from the CSSWG, plus Gankra (Mozilla) for her writing about text rendering, which has proved both inspiring and reassuring.

nonymous + bore(1): DNS toys for Rust

2021-01-03T21:00:00+00:00

I’ve been writing a DNS implementation in Rust. This project started out as a vehicle for learning Rust, but the more I learned, the more challenging goals I was able to set, to the point where I can see its potential to become useful in its own right. Here’s a post about what I’ve learned so far while writing nonymous, an embedded-friendly DNS library with #![no_std] and no-alloc support, and bore(1), a CLI tool for sending DNS queries.

most people don’t realise the opposite of anonymous is nonymous

— the hacker known as “Alex”, 2020

Rust already has a mature DNS implementation that I’ve heard wonderful things about, and there’s a long way to go before nonymous approaches anything resembling feature-complete or production-ready. But bore(1) is useful enough that I actually reach for it in 90% of the situations I would have previously used dig(1)…

$ bore daz.cat any
;; 172.19.128.1:53 (172.19.128.39:55722)
;; NoError #49916 Query 1 11 0 3 flags qr rd ra
;; EDNS(0) UDP 4096 flags

;; question section
; daz.cat. IN ANY

;; answer section
daz.cat. 3600 IN SOA daria.daz.cat. delan.azabani.com. 2020082100 600 60 1814400 60
daz.cat. 3600 IN NS daria.daz.cat.
daz.cat. 3600 IN NS ns2.he.net.
daz.cat. 3600 IN NS ns2.afraid.org.
daz.cat. 3600 IN CAA 128 issue "letsencrypt.org"
daz.cat. 3600 IN CAA 128 issuewild ";"
daz.cat. 3600 IN CAA 0 iodef "mailto:delan@azabani.com"
daz.cat. 3600 IN A 107.191.57.160
daz.cat. 3600 IN AAAA 2001:19f0:5800:8a45:ec4:7aff:fe15:d8a2

;; authority section

;; additional section
daria.daz.cat. 3600 IN AAAA 2403:5800:7300:6300:cccc:ffff:feee:8001
daria.daz.cat. 3600 IN A 180.150.30.255
; EDNS OPT RR was here

…and some situations that the incumbent struggles with, like dumping, replaying, and debugging messages.

$ bore bore.test ns --encode | tee query.dns | xd
40 C1 01 00  00 01 00 00  00 00 00 01  04 62 6F 72  @┴☺��☺�����☺♦bor  0
65 04 74 65  73 74 00 00  02 00 01 00  00 29 10 00  e♦test��☻�☺��)��  10
00 00 00 00  00 00                                  ������            20

$ < query.dns bore --load --dump | tee response.dns | xd
40 C1 85 80  00 01 00 02  00 00 00 01  04 62 6F 72  @┴àÇ�☺�☻���☺♦bor  0
65 04 74 65  73 74 00 00  02 00 01 C0  0C 00 02 00  e♦test��☻�☺└��☻�  10
01 00 00 0E  10 00 02 00  00 C0 0C 00  02 00 01 00  ☺�����☻��└��☻�☺�  20
00 0E 10 00  00 00 00 29  10 00 00 00  00 00 00 00  �������)��������  30

$ bore --decode < examples/badrdata.dns
;; NoError #0 Query 1 3 0 0 flags qr rd ra

;; question section
; . IN NS

;; answer section
. 13 IN NS a.root-servers.net.
. 13 IN NS \# 0
. 13 IN NS \# 2 C0 45

;; authority section

;; additional section

What is DNS anyway?
Decoder designs
- Naïve decoders (hg-v0)
- Zero-copy views (hg-v1, hg-v2, hg-v3, hg-v4)
- RDATA views
Encoder designs
- Early builders
- State machines
Open questions
Next steps

What is DNS anyway?

DNS is a distributed database that stores information in a hierarchy of names. The most familiar example of these is IP addresses (the information) and hostnames (the names). This is how your browser knows to contact 107.191.57.160 when you go to opacus.daz.cat.

When we build a DNS implementation, that might mean:

a protocol decoder, to understand incoming DNS messages
a protocol encoder, to serialise outgoing DNS messages
a query tool like dig(1), to send queries and print responses
a resolver, which contacts authorities to answer queries
an authority, which maintains information about a domain

Let’s explore the challenges behind the first two.

Naïve decoders

My initial approach was based around a trait that would describe a type that we can instantiate from something we can Read. After all, the network is just like a stream that you pipe into your program… right?

pub trait Decode<T: Read, E>: 'static + Sized {
    fn decode(source: &mut T) -> Result<Self, E>;
}

So if we defined a Message type that represents a message, we could then define how to parse one out of an octet stream.

/// ```rust
/// let mut source = &b"\x13\x13\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"[..];
/// let message = Message::decode(&mut source)?;
/// ```
pub struct Message {
    header: Header,
    // ...
}

pub struct Header {
    id: u16,
    qr: bool,
    opcode: u8,
    aa: bool,
    // ...
}

impl<T: Read> Decode<T, MessageError> for Message {
    fn decode(source: &mut T) -> Result<Self, MessageError> {
        let header = Header::decode(source)?;
        // ...

        Ok(Self { header, /* ... */ })
    }
}

This approach has a few problems. The most obvious one is that Read isn’t available in #![no_std], but this wouldn’t be too hard to work around with a shim trait.

A deeper problem is that parsing DNS messages in one pass without random access is incompatible with message compression, which allows names in some places to “point” to labels somewhere else in the message. For example, this message represents a.root-servers.net. in full, then reuses part of that with b. followed by “go to 1Eh for the rest”:

The solution I reached for here was, in retrospect, very inelegant: a pair of Read adapters that allow the caller to read behind or ahead (respectively) of the current position in the underlying stream.

pub struct Rewind<I: Read> {
    inner: I,
    memory: Vec<u8>,
}

pub struct Peek<I: Read> {
    inner: I,
    future: Vec<u8>,
}

impl<I: Read> Read for Rewind<I> { /* ... */ }
impl<I: Read> Read for Peek<I> { /* ... */ }

impl<I: Read> Rewind<I> {
    pub fn rewind(&self, position: usize) -> Option<Peek<Cursor<[u8]>>> { /* ... */ }
}

impl<I: Read> Peek<I> {
    pub fn peek(&self, position: usize) -> std::io::Result<&[u8]> { /* ... */ }
}

Note that many of the names of types and other symbols have been changed to make this post more clear and consistent. For example, Rewind was actually called Elephant(?!), and I actually flip-flopped between View and Consume.

hg-v0

This approach only made sense under the premise that we should be able to stream DNS messages from a Read into the decoder, a premise that I clung to because I thought we might not know how long a message is without decoding it.

As it turns out, this isn’t actually a problem for DNS as used with its two most common transports. For UDP, each datagram contains exactly one message, and datagrams are inherently of fixed length. For TCP, streams can convey many messages, but the sender has to prefix each message with its length.

With that cleared up, I decided that this kind of “streaming” decoder wasn’t worth the effort, and I went back to the drawing board.

Zero-copy views

While I was at the drawing board, I also started developing some ideas that would pave the way for zero-copy decoding.

Looking back at the old Header design below, notice how we painstakingly unpack everything from each field into neat little Rust fields? Each thing we unpack involves some copying that adds precious instructions to the critical path.

pub struct Header {
    id: u16,
    qr: bool,
    opcode: u8,
    aa: bool,
    // ...
}

What if we could walk through a DNS message as quickly as possible, doing only the work that’s absolutely necessary to reach the end of the message? This turns out to be an interesting problem to solve, because most of the message is of unknown length. A protocol element of unknown length means that its length can only be known by descending into, and walking through, that protocol element.

This is distinct from other elements of variable length, where the length can be determined from surrounding information, but don’t worry about this just yet. Let’s consider this overview of DNS protocol elements. Walking through the header is easy — skip 12 octets — but the rest of the message is of unknown length.

This is because each section is of unknown length. Even if questions and records were of known but variable length, there’s a variable number of them in each section.

To make matters worse, questions and records themselves are of unknown length anyway. Notice that rdata is a good example of an element of known but variable length.

At the end of the day, the root cause is that names themselves are of unknown length. While labels are of known but variable length, there’s a variable number of them in each name. The length of a label depends on a couple of different things, and this has surprisingly interesting implications for extensibility¹.

The crux of my approach to zero-copy decoding is that walking to the end of a message in this way is, on some level, proof that the message is structurally sound. When that proof succeeds, we want to return some type that represents the proof. This is what I call a view, and it allows the caller to interrogate the message efficiently, because many of their “questions” can be made infallible², and confidently, because we’ve proven that those infallible “questions” are truly infallible (panic-free).

A view under this definition can be a unit type (no fields), but in practice, we should also include any information that the caller can use to answer their “questions” even more efficiently. To keep our design embedded-friendly, let’s avoid the need for a separate allocation by limiting ourselves to constant space.

For records, that’s easy enough: one slice over the whole message (for compressed names), plus where the record starts in the message, and where the fixed part starts, or equivalent.

pub struct Record<'s> {
    start: &'s [u8],
    name: Name<'s>,
    rest: &'s [u8],
}

pub struct Name<'s> {
    start: &'s [u8],
    slice: &'s [u8],
}

As for messages, I think the most useful information we can return in constant space is a slice over the whole message, plus slices indicating where each section starts, to give question and record iterators what they need to know to start immediately.

pub struct Message<'s> {
    start: &'s [u8],
    header: Header<'s>,
    qd: &'s [u8],
    an: &'s [u8],
    ns: &'s [u8],
    ar: &'s [u8],
}

pub struct Header<'s> {
    start: &'s [u8],
    slice: &'s [u8],
}

If we require the caller to provide the whole message upfront, we can dispense with all of that Read goop and ask for two slices (&[u8]): one with the part of the message that this decoder should focus on, and one over the whole message for compressed names.

// Ok((the view, slice over the remaining input))
pub type ViewResult<'s, T> = Result<(T, &'s [u8]), ()>;

pub trait View<'s>: Sized {
    fn view(start: &'s [u8], source: &'s [u8]) -> ViewResult<Self>;
}

To speed up our decoding of compressed names, let’s cache the set of pointer destinations that are known to be good.

pub struct Context<'s> {
    start: &'s [u8],
    cache: Seen,
}

pub trait View<'s>: Sized {
    fn view(context: Context<'s>, source: &'s [u8]) -> ViewResult<Self>;
}

Note that this version actually used a type called Slice, but unlike the one in hg-v2, it was just an alias for &[u8].

hg-v1

Now let’s add some error handling, and while we’re at it, replace start and source with a single type that represents a subslice that maintains a reference to the whole slice.

pub struct Slice<'s> {
    // rust-lang/rust#27186
    start: usize,
    stop: usize,
    whole: &'s [u8],
}

impl<'s> Slice<'s> {
    // TODO replace once slice_index_methods is stable
    fn slice<R: RangeBounds<usize>>(self, range: R) -> Self { /* ... */ }

    // ([p..q], len ≤ q-p) -> ([p..p+len], [p+len..q])
    pub fn assert(self, len: usize) -> Result<(Self, Self), SliceError> { /* ... */ }

    // ([p..q], offset) -> [offset..]
    pub fn jump(self, offset: usize) -> Result<Self, SliceError> { /* ... */ }
}

pub type ViewResult<'s, T> = Result<(T, Slice<'s>), <T as View<'s>>::Error>;

pub trait View<'s>: Sized {
    type Error;
    fn view(source: Slice<'s>) -> ViewResult<'s, Self>;
}

pub struct Message<'s> {
    slice: Slice<'s>,
    qd: Slice<'s>,
    an: Slice<'s>,
    ns: Slice<'s>,
    ar: Slice<'s>,
    opt: Option<Extension<'s>>,
}

pub struct Extension<'s> {
    slice: Slice<'s>,
    content: Slice<'s>,
}

pub struct Record<'s> {
    slice: Slice<'s>,
    name: Name<'s>,
    fixed: Slice<'s>,
    rdata: Slice<'s>,
}

pub struct Name<'s> {
    slice: Slice<'s>,
}

hg-v2

I was unsatisfied by the size of my view types, both before and after introducing Slice. Record, for example, went from being 8 pointers long (four &[u8] fields, each having *const u8 and usize) to 16 pointers long (four Slice fields, each having &[u8] and two usize). Message was already pretty heavy at 14 pointers long, but now it was 28 pointers long.

The compiler doesn’t (or can’t) always avoid copying these values when we pass them around, so I would often see memmoves of 64 or even 224 bytes (amd64), and they were becoming difficult to ignore in my benchmarks. This was getting out of hand.

The good news was that most of these details were either redundant, or could be inferred from other details in constant time. 12 out of 28 pointer widths in Message were redundant copies of the *const u8 and usize that defined the extent of the whole message. Of the remaining 16 pointer widths, at least four can be inferred in constant time, by exploiting the fact that the header and four sections “touch” each other in memory:

.qd.start = .slice.start + 12
.an.start = .qd.stop
.ns.start = .an.stop
.ar.start = .ns.stop

Let’s redefine this in terms of a starting offset and four section lengths:

let offset = .slice.start
let qd_len = .qd.stop − .qd.start
let an_len = .an.stop − .an.start
let ns_len = .ns.stop − .ns.start
let ar_len = .ar.stop − .ar.start

We can still determine where all the sections are in constant time:

qd section starts at offset + 12
an section starts at offset + 12 + qd_len
ns section starts at offset + 12 + qd_len + an_len
ar section starts at offset + 12 + qd_len + an_len + ns_len

I took this to its logical extreme, cutting Message down to 8 pointers long (6–20 shorter) and Record to just 3 (5–13 shorter). Two pointer widths in each case were saved by eliminating the slice reference over the whole message. The caller already has a copy of this reference, because that’s where it came from in the first place!

pub type ViewResult<'s, T> = Result<(T, Range<usize>), <T as View>::Error>;

pub trait View: Sized {
    type Error;
    fn view(
        source: &[u8],
        range: Range<usize>,
        cache: &mut Option<Seen>,
    ) -> ViewResult<Self>;
}

pub struct Message {
    offset: usize,
    qd_len: usize,
    an_len: usize,
    ns_len: usize,
    ar_len: usize,
    opt: Option<Extension>,
}

pub struct Extension {
    inner: Record,
}

pub struct Record {
    offset: usize,
    name_len: usize,
    rdata_len: usize,
}

pub struct Name {
    offset: usize,
    len: usize,
}

hg-v3

Removing references to the message buffer was exciting at first — look ma, no lifetimes! But it was a step too far, because as a result, the caller had to pass that reference to every method that interrogates their messages. Not only was this awkward, but now Rust can no longer ensure that our “infallible” methods are actually interrogating the correct buffer! Pass in the wrong buffer and we’ll panic, or worse, blindly return nonsense.

Let’s put that reference back into each view. We can limit the cost to just one &[u8] (two pointer widths) in each view by avoiding composition, creating views for inner protocol elements on the fly.

pub struct Record<'s> {
    source: &'s [u8],
    offset: usize,
    name: Name<'s>,
    rdata_len: usize,
}

pub struct Name<'s> {
    source: &'s [u8],
    offset: usize,
    len: usize,
}

→

pub struct Record<'s> {
    source: &'s [u8],
    offset: usize,
    name_len: usize,
    rdata_len: usize,
}

impl Record<'_> {
    pub fn name(&self) -> Name {
        Name {
            source: self.source,
            offset: self.offset,
            len: self.name_len,
        }
    }
}

hg-v4

RDATA views

Each view has a Display implementation that prints the protocol element in a zone-or-dig(1)-like format, from labels all the way up to entire messages. This is how bore(1) does most of its formatting work.

There are views for each record type’s rdata too, and these views walk through protocol elements and prove structural soundness in the same way. The key difference here is that these views only run when needed, like when the Display formatter for a Record calls the method below. Because records have an rdlength field, we can treat rdata as an opaque blob while we’re on the Message view’s critical path.

The main limitation of this method is that it returns a Box, which requires alloc. I’m sure there’s a way to rework this for no-alloc support, but so far I’ve only really thought about this piece of the puzzle for long enough to get bore(1) working.

pub fn rdata(&self) -> Result<Box<dyn Rdata + '_>, RdataError> {
    let start = self.rdata_offset();
    let stop = start + self.rdata_len;
    Ok(match (self.class().value(), self.r#type().value()) {
        (_, 6) => Box::new(Soa::view(self.source, start..stop)?.0),
        (_, 15) => Box::new(Mx::view(self.source, start..stop)?.0),
        (_, 16) => Box::new(Txt::view(self.source, start..stop)?.0),
        (_, 257) => Box::new(Caa::view(self.source, start..stop)?.0),
        // RFC 3597 § 4
        (_, 2) | (_, 5) | (_, 12) => Box::new(CompressibleName::view(
            self.source, start..stop)?.0),
        (1, 1) => Box::new(InAddress::view(self.source, start..stop)?.0),
        (1, 28) => Box::new(InAaaa::view(self.source, start..stop)?.0),
        _ => Box::new(Unknown::view(self.source, start..stop)?.0),
    })
}

Confession: the logic for incompressible names isn’t actually implemented yet, so Name is effectively CompressibleName. This is why I’ve been able to erroneously use Name in the Question and Record views.

e74ceb3cba72c

8548d9fdd3e64

Early builders

My earliest DNS encoder actually boiled down to bore(1) — née scoop(1) — throwing together a message from hardcoded parts, if that even counts. From there I built my encoders incrementally, using the query tool as a guide for what to work on next.

// unbound(8) requires RD (no cache snooping) by default
let header = b"\x01\x00\x00\x01\x00\x00\x00\x00\x00\x01";
let question = b"\0\x00\x02\x00\x01";
let opt = b"\0\x00\x29\x10\x00\x00\x00\x00\x00\x00\x00";

let query: Vec<_> = id
    .iter()
    .copied()
    .chain(header.iter().copied())
    .chain(question.iter().copied())
    .chain(opt.iter().copied())
    .collect();

30ebc50b6f112

One by one, I replaced hardcoded parts with actual encoders, until there were none left.

let (qname, qtype) = match (qname, qtype) {
    (None, None) => (".", "NS"),
    (None, Some(_)) => unreachable!(),
    (Some(qname), None) => (qname, "A"),
    (Some(qname), Some(qtype)) => (qname, qtype),
};

// unbound(8) requires RD (no cache snooping) by default
let header = Header::query(random()).rd(true)?.qdcount(1)?.arcount(1)?;

let query = emit::Message::new(header)
    .qd(Question::new(
        qname.parse()?,
        qtype.parse()?,
        qclass.parse()?,
    ))?
    .opt(Extension::default())?
    .emit_to_vec()?;

b75227c5b20e3

463335b20a7c5

Each of these encoders was backed by the same kind of “naïve” type as in my early decoders. They exposed a thin builder API that didn’t enforce any kind of structural soundness, like the header’s qdcount reflecting the number of questions in the message.

pub struct Message {
    header: Header,
    qd: Vec<Question>,
    an: Vec<Record>,
    ns: Vec<Record>,
    ar: Vec<Record>,
    opt: Option<Extension>,
}

pub struct Header {
    // ...
    qdcount: u16,
    ancount: u16,
    nscount: u16,
    arcount: u16,
}

impl Message {
    pub fn qd(mut self, value: Question) -> Self {
        self.qd.push(value);

        self
    }
}

impl Header {
    pub fn qdcount(mut self, value: u16) -> Self {
        self.qdcount = value;

        self
    }
}

0.0.0

ddc6231c4ed02

b00b8b30028e7

0.0.1

State machines

Clearly that wasn’t the kind of quality and safety we would expect of a library. The encoder types also heavily relied on multiple layers of allocations, which made them awkward to adapt to no-alloc targets.

Thus began an incredible journey³ that took me ten months (albeit in my spare time):

I had two key ideas around where I wanted to go with the new encoders.

We can avoid intermediate copying and separate allocations by writing protocol elements directly into the buffer that will actually get sent. The deeply variable-length nature of DNS complicates this, but we can avoid opening that can of worms by writing them in order, so let’s see how long we can survive with this requirement.

We can use session types — essentially a state machine in the type system — to enforce as much structural soundness as possible at compile time. Perhaps we could even use them to force library consumers to write protocol elements in order. I owe the inspiration behind this to Pascal Hertleif (Elegant Library APIs in Rust), and especially Ana Hobden (Pretty State Machine Patterns in Rust).

The design I worked with for most of the way was an adaptation of Ana’s Generically Sophistication pattern. I defined a generic container that wraps our state types, providing our shared “context” fields like output buffer (sink) and maximum payload size (limit).

pub struct Emit<'s, S: Sink, I> {
    sink: &'s mut S,
    limit: Option<NonZeroU16>,
    inner: I,
}

I then defined a state for each protocol element, as well as types for any substates it might have, using generics in the same way.

pub struct Message<P, Q> {
    parent: P,
    section: Q,
}

pub struct QdSection;
pub struct AnSection;
pub struct NsSection;
pub struct ArSection;

State-dependent methods, including state transitions, are implemented on the Emit<...> types of the appropriate states.

macro_rules! transitions {
    {$($machine:ident.$field:ident {$($from:ident -> $to:ident;)*})*} => {$($(
        impl<'s, S: Sink, P> From<Emit<'s, S, $machine<P, $from>>> for Emit<'s, S, $machine<P, $to>> {
            fn from(machine: Emit<'s, S, $machine<P, $from>>) -> Self {
                machine.map(|x| $machine { parent: x.parent, $field: $to })
            }
        }
    )*)*};
}

transitions! {
    Message.section {
        QdSection -> AnSection;
        QdSection -> NsSection;
        QdSection -> ArSection;
        AnSection -> NsSection;
        AnSection -> ArSection;
        NsSection -> ArSection;
    }
}

When we transition from encoding a protocol element to encoding one of its parts, we wrap the old state in the new state using generics, forming a stack that remembers the state to return to. If you think these type signatures are barely comprehensible (and I certainly do), you probably wouldn’t want to see the helper functions behind which I tucked the heavy lifting of these “downward” state transitions.

pub struct Record<P, Q> {
    parent: P,
    step: Q,
}

pub struct RecordName;
pub struct RecordData;

impl<'s, S: Sink, P> Emit<'s, S, Message<P, AnSection>> {
    /// Start building a new [`Record`] on the end of the message’s answer section.
    pub fn record(
        self,
    ) -> Result<Emit<'s, S, Record<Message<P, AnSection>, RecordName>>, MessageError> {
        Ok(self.an_increment()?.child::<Record<_, _>>())
    }
}

42ec3b432289c

7b30adb4e2561

At this point, we had a working prototype that could make some useful guarantees, and as complicated as the internals were, the bore(1) code consuming it almost looked like an ordinary builder chain!

// FIXME EDNS OPT RR
// unbound(8) requires RD (no cache snooping) by default
let mut query = Vec::default();
Emit::new(&mut query, None)
    .child::<Message<_, _>>()
    .id(random())
    .rd(true)
    .question()?
    .qname()
    .labels(qname)?
    .finish()
    .finish(qtype.parse()?, qclass.parse()?);

d55670718f597

7370009af1b7b

41a8da0a04f29

I then reworked all of my type signatures to own a sink, rather than mutably borrow one, and relinquish it back to the caller once encoding is finished (successfully or otherwise). I feel like I decided to do this for more important reasons than look-ma-no-lifetimes, but I genuinely don’t remember.

75e5599e0fb6a

The main consequence of my design that I was unhappy about was that the whole encoder API would get rendered on the same rustdoc page, because all of our state-dependent methods were implemented on various flavours of Emit<...>.

To find out how to build a Message, you can’t go to Message, because all of the docs are in Emit. Does this look like a clear and inviting API to you?

I spent a very long time trying to find an elegant way around this. I never ended up asking for advice on the Discord servers, because where would I even start? I felt like I would have to explain so much to justify my predicament that it was beyond overwhelming. Regardless of whether my fear was proportional or exaggerated (or scars from Stack Overflow⁴), the outcome was the same.

I think I came to the conclusion that there wouldn’t be a good solution until Rust lands a new feature like arbitrary_self_types, which would allow me to move my state-dependent methods to an impl on the element type:

impl<S: Sink, P, Q> Message<P, Q> {
    /// Write the given value to the qr field in the message header.
    pub fn qr(mut self: Emit<S, Self>, value: bool) {}
}

I eventually worked around this by duplicating some code, splitting Emit into a separate type for each protocol element. Emit became FooBuilder, Foo became FooBuildee, and so on, and I renamed most of the other traits to evoke one of these two terms.

51175861ea760

22e62e3c3cb32

cd538edc15ba3

0.0.2

Open questions

I’m in two minds about the designs I arrived at. On the one hand, I’m pleased that I’ve finally managed to come up with something that works, satisfies all of my self-imposed requirements, and lands somewhere in the vicinity of user-friendly. At the same time, I’m unsatisfied with the complexity of the encoder side, and worried that someday, when I learn more about DNS, my whole approach will be rendered unusable.

I’ve only written the state machines for barely more than the core protocol elements in RFC 1035, and it’s already testing the limits of what I can maintain. I’ve bought some time by extracting some boilerplate into macros by example, like the code that defines simple From transitions, or the code that juggles Builder types and Buildee types, but I’m not sure how much further I can go with that.

I wonder if this is the kind of metaprogramming situation where I ought to write some custom derive macros? I’ve managed to procrastinate learning about proc macros until today (and counting), so this is not a rhetorical question.

As for my fear that this will all be rendered unusable someday? That stems from the fact that I’ve been navigating all of these design questions as a first-time DNS implementer. While I think I understand DNS reasonably well as a sysadmin, that doesn’t really answer the more intricate questions, like how do my interface decisions affect your ability to implement DNS algorithms or endpoints well?

Perhaps I could have avoided this by writing more DNS components, like a resolver or an authority, before I imposed these challenging requirements on myself. But given that I would then be definitely reimplementing the wheel, that isn’t nearly as exciting, is it?

Next steps

If nothing else, bore(1) seems like one promising fruit of my endeavour, so I’m keen to implement a bunch more of the features I rely on day-to-day:

conversion of IP addresses to PTR queries under in-addr.arpa. + ip6.arpa.
trace mode (simulating an iterative resolver with non-rd queries)
Windows system resolver support (already implemented in hg-v2!)

I’m not sure where to go with nonymous though, and I think that’s because I have no idea how promising it is as a project that other people might find useful someday. Depending on the answer, my next steps might be anything between “full speed ahead, higher layers, rigorous conformance, the whole shebang” and “retire the project and focus on bore(1)”. But as a first-time DNS implementer without any actual embedded experience, the odds are stacked towards the latter end of the spectrum.

In any case, hacking on nonymous has been loads of fun, and it was definitely worthwhile for me personally. It’s probably where I tried out like half of my new Rust knowledge — especially all the cool things I had read about — for the first time:

releasing a library and communicating changes
designing a good API (API guidelines, elegant API design)
session types and compile-time state machines
lots and lots of macros by example (TLBORM)
FFI and Windows system programming (PlatformResolvers)
unsafe, MaybeUninit, #![forbid(unsafe_code)] (Seen)
profiling (cpuprofiler, perf + flamegraph) and benchmarking
figuring out CI and test coverage for a complex Rust project
and probably a whole bunch more!

There are a few different label types, and the encoded length of a label depends on this — normal labels are effectively Pascal strings, compressed labels are two octets long, and so on. Only half of the label type “namespace” is currently in use, so in theory the other half is open for future innovation.

What’s interesting is that it ended up being effectively impossible to invent new label types, because we didn’t think to define a way for old software to know how long new label types are (e.g. TLV), so they can at least decode the rest of the message. As a result, new label types can’t be deployed until everyone that might encounter them updates their software, which is a massive chicken-and-egg situation.

Extensibility of internet protocols is a surprisingly hard problem (at least if Postel is all you’ve heard of), and we’ve had to learn how to do it the hard way. Check out Cryptographic Agility by Adam Langley for the lessons SSL taught us. ↩
An “infallible” function is one that can’t fail, always returning successfully, rather than returning a Result or panicking. ↩
But not an Incredible Journey™. ↩
I was a very active user around a decade ago, but I eventually grew bored of the site, to the point where I don’t even go there when I have a question. I think this is because the site’s culture and governance made it devolve into a place that rewards unhealthy behaviours like racing to be the first answer, closing questions as duplicate/off-topic, and wielding “XY problem” like an aggressive Maslow’s hammer. ↩

Introducing xd(1)

2020-11-15T19:00:00+00:00

xd(1) is a tool that dumps binary input in a more human-readable format. There are countless tools that fit this description, including od(1), hexdump(1), xxd(1), and a bunch of Rust crates in the same vein, but this one has a new trick up its sleeve.

A column with printable ASCII bytes shown as exactly that, and everything else as a dot, is such a strong convention that it basically defines the visual identity of hex-dumping tools. It’s not a bad convention too — the unobtrusive dots for C0 controls and high bytes make ASCII text stand out, as if you had installed strings(1) on your pattern-matching neurons. But imagine what kinds of patterns you could spot in binary data, if only there was a unique glyph for each of the other 161 possible bytes?

The features released so far are pretty minimal beyond this very idea, but my hope is for xd(1) to eventually join the likes of rg(1) and fd(1) as another product of Rust’s renaissance of best-in-class solutions to CLI problems.

Modern character encodings like UTF-8 won’t work for a bunch of reasons, including the fact that they’re variable-length, which would break the one-to-one relationship between bytes and glyphs that we’re going for. Many common single-byte character sets like ISO-8859-1 or Windows-1252 won’t either, because they tend to leave 00h–1Fh and 80h–9Fh undefined to accommodate the C0 and C1 controls, which aren’t the simple printable characters we’re after.

I’ll let Raymond take it from here:

Is there a code page that matches ASCII for the first 128 values and can round trip arbitrary bytes through Unicode?

You may find yourself looking for such a code page when you have a chunk of binary data with embedded ASCII text. You want to be able to dig out and even manipulate the ASCII text, and treat the non-ASCII parts as mysterious characters that have no meaning, but you need to be able to convert them back into the original bytes.

[…]

Okay, I’ll cut to the chase. The code page I use for this sort of thing is code page 437. Every bytes is defined and maps to a unique Unicode code point, and it agrees with ASCII for the first 128 values.

— Raymond Chen’s The Old New Thing post

In particular, we use a mapping of code page 437 to Unicode that includes the dingbats and other symbols that lived in the ROM at 01h–1Fh, plus a tweak that replaces U+0000 NULL with U+2400 SYMBOL FOR NULL at 00h. The end result is a reversible text representation that’s fairly visually distinct¹.

Learning how to ship

[2016-01-04 00:36:31] <aedomsan>
what are you working on delan?

[2016-01-04 00:39:47] <delan>
aedomsan: a tool like od/hexdump/xxd where every byte has a glyph instead of dots everywhere

[five months later]

[2016-06-18 19:34:48] <Jaci> 
delan you still got that hexview program?

[2016-06-19 18:33:35] <delan>
yeah Jaci, I’m still working on it

This is actually my third attempt at executing this idea in this form².

I first wrote a version of this tool in C, before I even got into Rust, and it was pretty elaborate. There were custom format strings powered by a nearly-incomprehensible hand-rolled parser, a cute little --help system that would essentially popen(3) a pager like less(1) and pipe documentation text from inside the program, and the project even taught me how to use fuzzing to fix bugs!

A couple of years later, I ported it to Rust as my first foray into the language. I kept many of the more… useful features (the popen(3)-to-pager trick was more cute than useful), learned to use parser combinators with the new custom format parser, and wrote enough functionality that I could start emulating the competition.

The problem was that I hadn’t really released many of my projects before, and I was more than happy to perpetually avoid dealing with that can of worms in favour of using the project as a training ground for learning how to build new things. Rust certainly helped — unlike C, it has a fantastic package manager and a place to put your packages — and that’s where almost everything else I’ve managed to release (or help release) has gone.

But the lesson that really made this project see the light of day? To build something that has any broader impact, you can’t just experiment forever, rewriting your project over and over (but better this time) and trying to perfect a feature set that will blow everyone out of the water on version zero.

At the end of the day, you’ve got to actually turn it into a thing that’s useful to someone (even just you or your friends), be it a package they can install, a service they can use, a video and writeup they can enjoy, or something. And from there, you can iterate.

Though not perfect, for example, if you’re looking at the various spaces (20h FFh), middle dots (07h F9 FAh), squares (DCh FEh), box drawing characters, or that U+207F (FCh) that inexplicably isn’t superscript (ⁿ) in my terminal font. ↩
It’s my fourth in general, if you count the time I convinced my colleague at $dayjob to use this technique to display untrusted paths in customer backups (and wrote a tool to help with that). ↩

Output version control for static sites

2020-10-03T13:00:00+00:00

I use Jekyll, a static site generator that’s most notable for powering GitHub Pages. Two years ago, I noticed that the permalinks for several of my blog posts were broken by a Jekyll bug after I upgraded to Ruby 2.4. I’ve since written a Git-based system to stop this kind of breakage from ever reappearing, which should be useful to anyone who uses a static site generator and wants to avoid corruption or inadvertent changes.

The bug

You can read more about what happened in the commit where I first wrote the system, but in short, Jekyll used to suffer from a bug where the day a post gets filed under can vary depending on Ruby version… because time zones.

I was using an affected version, and when I upgraded to Ruby 2.4 with a new OS release, ten posts got “moved” to the next day. It took me five months to notice this, and search engines had already adjusted to the damage.

I migrated my site from Jekyll 3.0.1 to 3.8.3, which wasn’t too hard thanks to the project’s stability philosophy, but I also created some static redirects to keep the broken permalinks working, in case any backlinks were made during that time.

_layouts/legacy.html

<!doctype html><meta charset="utf-8">
<meta http-equiv="Refresh" content="0;
    URL={{ site.url }}{{ site.baseurl }}{{ content | strip_newlines }}">

2014/01/03/forcing-single-timezone-jekyll.html

---
layout: legacy
---
{% post_url 2014-01-02-forcing-single-timezone-jekyll %}

The system

But anyway, back to the system. The idea behind output version control is to track the actual files that get served, so when we want to generate and deploy a new version of our site, we can see how the site would actually change.

The whole system is powered by a makefile, which sets up a special Git repository at _staging that tracks the site generator’s output. This repo is unrelated to any Git repo we might be using for the source of the site, so it will never get confused by branches or refactoring.

_staging lets you examine the changes that would be made by deploying your site. If you’re happy with the changes, we clone the repo to _production, so to use this system, reconfigure your web server to serve your site from _production/_site, rather than _site or wherever your site generator writes to.

Let’s read it together. I like to put .POSIX: on the first line out of habit, though strictly speaking our use of .PHONY: makes this not a portable makefile anyway. BUNDLE lets us override how we run Bundler, like make BUNDLE=bundle24 on OpenBSD for example.

.POSIX:

BUNDLE = bundle

When we make dry or make with no targets, generate the site into _site, copy _site to the output repo’s working tree, check if the index is dirty, and copy the working tree to the index. Our use of the index is interesting here. By checking and bailing out if the index is dirty, we’re forced to either deploy or reject any previously staged changes, which might prevent accidental data loss.

dry: _staging
	$(BUNDLE) exec jekyll build
	rsync -a --delete _site _staging
	git -C _staging diff --cached --quiet
	git -C _staging add _site

When we make examine, display the staged changes as a diff between HEAD and the index, so we can decide whether we’re happy with them.

examine: _staging
	git -C _staging diff --cached

When we make deploy, commit the changes as the new HEAD, then update the clone at _production to reflect them.

deploy: _staging _production
	git -C _staging commit --allow-empty --allow-empty-message -m ''
	git -C _production pull

When we make reject, throw away the staged changes by resetting the index to HEAD. In this situation, as well as when make dry bails out, the working tree could still be dirty. While we could clean it with --hard and git clean -dffx, we don’t need to, because we only ever copy the working tree to the index after completely rewriting it with rsync(1).

reject: _staging
	git -C _staging reset

To set up _staging for the first time, create the directory and make it a Git repo. To set up _production for the first time, clone _staging as a Git repo.

_staging:
	mkdir -p -- '$@'
	cd -- '$@' && git init

_production:
	git clone _staging _production

Never skip the commands under dry + examine + deploy + reject based on modified times, even if there happens to be a file with one of those names in the site’s sources. .PHONY: isn’t yet in the POSIX spec, but it has widespread support, including the GNU and OpenBSD versions.

.PHONY: dry examine deploy reject

That’s all there is to it! Is this the perfect system? No, but it suffices for my needs.

Limitations

If your site’s output has non-deterministic parts, those parts can make the output of make examine noisy. The one example I’ve encountered is jekyll-feed’s <updated> tags, when I added Atom feeds a few months ago. You could probably fix this with textconv or an external diff command.

We can, of course, access older versions of our site’s output with git -C _production, but the system as described doesn’t store the inputs that they were generated from (or even hints as to where we can find them, like commit hashes). Feel free to implement this, but note that your site’s sources aren’t necessarily the only relevant input! For example, the inputs that broke my permalinks were my Ruby and Jekyll versions.

_production and _staging will store an unbounded amount of history, so if you need to reclaim space, you’ll want to use something like git rebase -i --root to squash old commits followed by a git gc.

There’s still a lot of unnecessary overhead. The initial clone to _production uses hard links but subsequent pulls don’t, so we should probably use rsync(1) rather than cloning _staging. When the output advances to _staging and then to _production, we can also delete the redundant copies in _site and then in _staging.

There’s no easy way for make examine to accept additional arguments, such as --word-diff, that are then passed to git diff --cached. If we used a shell script instead of a makefile, this wouldn’t be a problem:

action="$1"; shift

case "$action" in
    (examine) git -C _staging diff --cached "$@" ;;
    # ...
esac

B-side: incremental builds

I’ve got a thing for using secondary Git repositories in project tooling. When I worked at Atlassian, I wrote an incremental build system for Maven using a similar technique. Maven doesn’t support incremental builds out of the box, but mvn -pl x,y,... tells the reactor to only try to build a subset of our project’s modules. If we could figure out which modules need to be rebuilt, we can implement coarse-grained incremental builds!

In this case, the secondary Git repo, which I call a “shadow repo”, tracks the inputs rather than the outputs. But Delan, you ask, doesn’t the primary repo already do that? Kind of, but the user (i.e. one of my colleagues) is constantly messing around with it, and it’s difficult to implement robust incremental builds atop a repo that we don’t control, because we have no way of knowing what the inputs were during the last successful build. The system I replaced tried anyway, and it got confused by everything from dirty indexes and working trees to the user switching branches.

My new system roughly worked as follows. We create the shadow repo, which is --bare to avoid the redundant working tree. To interact with it, we point GIT_DIR to the shadow repo and GIT_WORK_TREE to the real repo’s working tree.

#!/bin/sh
set -eu

top="$(git rev-parse --show-toplevel)" || exit 66
git init --bare -- "$top/.shadow/repository"

shadow() {
    GIT_DIR="$top/.shadow/repository" GIT_WORK_TREE="$top" git "$@"
}

Before each build attempt, we copy everything except build outputs to the index, thanks to the project’s own gitignore(5), and get a list of paths that have changed since the last successful build (HEAD).

shadow add -- "$top"
shadow status -z

We then convert the list of paths to a set of Maven modules, by finding their nearest ancestors containing a pom.xml, then pass that set to mvn -am -pl. I no longer have any of the original source code, so I’ll leave this as an exercise for the reader.

If and only if the build succeeds, we commit the new build inputs. We get those from the index, which also prevents false negatives in the next build attempt, were the user to have edited any files during the build process.

commit() {
    shadow commit -m '' --allow-empty --allow-empty-message "$@"
}

commit --amend || commit

Closing thoughts

I hope you found that interesting! Perhaps you’ll even see Git in a new light, finding ways to use the stupid content tracker™ other than version control for source code.

I’ve been afraid to check if anyone has done this before, because I felt like if I was aware of any prior art, I would lose my enthusiasm for solving this problem and writing about it. But now that I’m done, that no longer matters. Let me know if great minds think alike!

My internship with Igalia

2020-09-27T09:00:00+00:00

I was looking for a job late last year when I saw a tweet about a place called Igalia. The more I learned about them, the more interested I became, and before long I applied to join their Web Platform team. I didn’t have enough experience for a permanent position, but they did offer me a place in their Coding Experience program, which as far as I can tell is basically an internship, and I thoroughly enjoyed it. Here’s an overview of what I did and what I learned.

Why Igalia?
ſtylesheet
- Fixing the bug
- Writing some tests
MathML tasks
Acknowledgements

Why Igalia?

There’s a wide range of work I can do as a computer programmer, but the vast majority of it seems to be in closed-source web applications, as an employee with a limited voice in the decisions that affect my work.

At the time, all of my work since I graduated had been exactly that, or in builds and releases for said applications. That was interesting enough for a while, but I wanted to make a bigger impact, work on something I actually cared about of my own volition, and ideally move towards getting paid to do systems programming.

Igalia appeals to me, with their focus on open-source projects, systems programming, and standards work. Even better, as a field, the web platform has been my one true love, and building things on it is how I got into programming over 15 years ago. But what cements their place as my “dream job” is how they work: as a distributed worker’s cooperative.

What I mean by “distributed” is that members can work from anywhere in the world, paid in a way that fairly adjusts for location, and in whatever setting they thrive in (such as home). This alone was huge, as someone who can’t sustainably work in an office five days a week, had to move 4000 km away from home to do so, and had just left an employer that was actively hostile to remote work.

Andy Wingo (author of that tweet) offers some insight into the “worker’s cooperative” part in these three posts. Igalia’s rough goal here, as far as I can tell, is that everyone gets a voice in deciding what the collective works on and how (to the extent that those decisions affect them), equal ownership of the business, and equivalent pay modulo effort and cost of living. This appeals to me as an anarchist, but also as a worker that has often been on the receiving end of unethical work, poor working conditions, and lack of autonomy.

ſtylesheet

One goal of my internship was to help the Web Platform team with their MathML work, but I was also there to familiarise myself with working on the web platform, and my first task was purely for the latter.

Many parts of the web platform have case-insensitive keywords that control an API or language feature, like link@rel (the <link rel="..."> attribute), but thanks to Unicode, there’s more than one level of case-insensitivity. Unicode case-insensitivity won’t break backwards compatibility of web content over time, but to improve interoperability and simplify implementations, things like the HTML spec tend to explicitly call for ASCII case-insensitivity, at least for keywords that are nominally ASCII.

That makes Blink’s widespread use of Unicode case-insensitivity in these situations a bug, and my job was to fix that bug, which sounds simple enough, until you realise that doing so is technically a breaking change. You see, there are already a couple of non-ASCII characters that can introduce esoteric ways to write many of those keywords.

More importantly, the web platform is almost¹ unique in that breaking existing content is, in general, not allowed. But this time a breaking change was unavoidable, like any time where an implementation is fixed to align with the standard, or some behaviour is standardised after incompatible implementations appear. There might be content out there that relies on something like <link rel="ſtylesheet"> because it worked on Chromium.

There are a few ways to minimise the impact of these breaking changes, like adding analytics to browsers to count how many pages would be affected, or searching archives of web content, but in this case we decided the risk was low enough that I could simply fix the bug and write some tests.

Fixing the bug

It’s hard to get a usable LSP setup going for a project as big as a browser. I switched between ccls and clangd a bunch of times, but I never quite got either working too well. My main machine is also getting pretty long in the tooth, which made indexing take forever and updating my branches expensive.

I considered writing an LSP client that would allow me to kick off an index on one of Igalia’s 128-thread build boxes without an editor, but I eventually settled on using Chromium Code Search to jump around and investigate things. Firefox similarly has Searchfox², but WebKit doesn’t yet have a public counterpart³.

I was looking for callers of three deprecated functions, but not all of them were relevant to the bug, and not all of those needed tests, and so on. To help me analyse and categorise all of the potential call sites, I wrote some pretty intricate regular expressions for Sublime Text 2. This one finds all callers of DeprecatedEqualIgnoringCase, with two arguments, where one of them is an ASCII literal that wouldn’t need new tests (skSK):

(?x-i)
(?<escape>\\['"?\\abfnrtv]){0}
(?<literal>"(?:(?=[ -~])[^"skSK]|(?&escape))*"){0}
(?<any>(?:[^(),]|(\((?:[^()]*|(?-1))\)))*+){0}
DeprecatedEqualIgnoringCase
(\s*\(\s*+(?:
    (?&literal)\s*,\s*+(?&any)
    |(?&any)\s*,\s*+(?&literal)
)\s*\))

After my first patch, which I wrote by hand, I also used those to do the actual replacing, maintaining a huge analysis of all the cases that remained after my second patch.

Writing some tests

Each of the major engines has its own web content tests, and automated tests are strongly preferred over manual tests if at all possible. All of the tests I wrote were automated, and most were Web Platform Tests, which are especially cool because they’re a shared suite of web content tests that can be run on any browser. Chromium and Firefox even automatically upstream changes to their vendored WPT trees!

Many of my tests were for values of HTML attributes whose invalid value default was a different state to the keyword’s state. In these cases, I didn’t even need to assert anything about the attribute’s actual behaviour! All I had to do was write a tag, read the attribute in JavaScript, and check if the value we get back corresponds to the intended feature (bad) or the invalid value default (good).

Some legacy HTML attributes are now specified in terms of CSS “presentational hints”, so I checked the results of getComputedStyle for those, but the coolest tests I learned to write were reftests. Very few web platform features guarantee that every user agent on every platform will render them identically down to the pixel, and over time, unrelated platform changes can affect a test’s expected rendering. Both of these things are ok, but they make it impractical for tests to compare web content against screenshots. Reftests consist of a test page that uses the feature being tested, and a reference page that should look the same without using the feature. The reference page is like a screenshot, but it’s subject to all of the same variables as the test page, such as font rendering.

Ever heard of the Acid Tests? Acid2 is more or less a reftest, because it has a reference page that only uses a screenshot for the platform-independent parts. Acid1 uses a screenshot of the whole test, hence “except font rasterization and form widgets”.

I had a lot of fun writing my two form-related tests, because I actually had to submit forms to observe those features’ behaviour. WPT has server-side testing infrastructure that can help with this, and for such tests, I would need to spin up the provided web server or run the finished product with wpt.live⁴.

In both cases, I avoided the need for that with a <form method="GET"> that targets an iframe, plus a helper page that sends its query string back to the test page.

MathML tasks

MathML was meant to be the native language for mathematics on the web, and that’s still true today, but two decades later, browser support still has a long way to go. There are several reasons for this, notably including the largely volunteer-driven development of MathML and its implementations, but over the last few years, Igalia has helped change that on three fronts: writing a Chromium implementation, improving the Firefox and WebKit implementations, and improving the specs themselves.

MathML 3 was made a Recommendation in 2014, and like any spec, it has shortcomings that only subsequent experience could identify. Proposals by the MathML Refresh CG like MathML Core are trying to address them in a bunch of ways, like simplifying the spec, setting clearer expectations around rendering, and redefining features in terms of better-supported CSS constructs. My remaining tasks touched on some of these.

mo@maxsize

Moving onto WebKit, my next task was to remove some dead code. Past versions of MathML specify a very complex <mstyle> with its own inheritance system that’s incompatible with CSS, as well as several attributes that were rarely if ever used by authors, both of which are a burden on implementors.

One of those attributes was mstyle@maxsize, which would serve as the default mo@maxsize instead of infinity. With the former removed from the spec, there was no longer a need for an explicit infinity value, so I removed the code for that.

It turns out WebKit never got around to implementing mstyle@maxsize anyway, so there was no functional change.

STIXGeneral

There’s a lot of MathML content that gets rendered like any other text, but stretchy and large operators are a bit more involved than just drawing a single glyph at a single size. A well-known example of a stretchy operator is square root notation, which consists of a radical (the squiggly part) and a vinculum (the overline part) that stretches to cover the expression being rooted.

\sqrt{x y} = \sqrt{x} \sqrt{y}

Traditionally this was achieved by knowing where the glyphs for the separate parts lived in each font, so we could stretch and draw them independently. Unicode assignments for stretchy operator parts helped, but that wasn’t enough to yield ideal rendering, because many fonts use Private Use Area characters for some operators, and ordinary fonts don’t give applications the necessary tools to control mathematical layout precisely.

OpenType MATH tables eventually solved this problem, but that meant Firefox essentially had three code paths: one for OpenType MATH fonts, one with font-specific operator data, and one generic Unicode path for all other fonts. That second one adds a lot of complexity, and there was only one font left with its own operator data: STIXGeneral.

The goal was ultimately to remove that code path, dropping support for the font. That sounded easy enough until we realised that STIXGeneral remains preinstalled on macOS, as the only stock mathematics font, to this day.

My task here was to add a feature flag that disables the code path on nightly builds, and gather data around how many pages would be affected. The patch was straightforward, with one change to allow Document::WarnOnceAbout to work with parameterised l10n messages, and I wrote a cute little data URL test page for the warning messages.

data:text/html;base64,PCFkb2N0eXBlIGh0bWw+CjxtZXRhIGNoYXJzZXQ9InV0Zi04Ij4KPHN0eWxlPgogIG1hdGg6Zmlyc3Qtb2YtdHlwZSB7CiAgICBmb250LWZhbWlseTogTGF0aW4gTW9kZXJuIE1hdGg7CiAgfQogIG1hdGggewogICAgZm9udC1mYW1pbHk6IFNUSVhHZW5lcmFsLCBMYXRpbiBNb2Rlcm4gTWF0aDsKICB9Cjwvc3R5bGU+CjxtYXRoIGRpc3BsYXk9ImJsb2NrIiBtYXRoc2l6ZT0iN2VtIj4KICA8bW8+4oiRPC9tbz48bW8gZGlzcGxheXN0eWxlPSJmYWxzZSI+4oiRPC9tbz4KPC9tYXRoPgo8YnV0dG9uIHR5cGU9ImJ1dHRvbiI+U1RJWEdlbmVyYWw8L2J1dHRvbj4KPHNjcmlwdD4KICBkb2N1bWVudC5xdWVyeVNlbGVjdG9yKCJidXR0b24iKS5hZGRFdmVudExpc3RlbmVyKCJjbGljayIsICh7IHRhcmdldCB9KSA9PiB7CiAgICBjb25zdCBzb3VyY2UgPSBkb2N1bWVudC5xdWVyeVNlbGVjdG9yKCJtYXRoIik7CiAgICB0YXJnZXQuYWZ0ZXIoc291cmNlLmNsb25lTm9kZSh0cnVlKSk7CiAgfSk7Cjwvc2NyaXB0Pgo=

Turning the feature flag on broke a test though, and I couldn’t for the life of me reproduce it locally. Fred and I tried every possible strategy we could imagine short of interactively debugging CI, on and off for six weeks, but it looked like the flaky behaviour involved some sort of race against @font-face loading. Eventually we gave up and disabled the feature flag just for that test, and I landed my patch.

padding + border + margin

Another way to improve the relationship between MathML and CSS has been defining how existing CSS constructs from the HTML world, including the box model properties, apply to MathML content. In this case, the consensus was that these properties would “inflate” the content box as necessary, making the element occupy more space.

Existing implementations in WebKit and Firefox didn’t really handle them at all because it wasn’t in the spec, so the last task I had time for was to change that.

A modern browser starts by parsing documents into an element tree, which is also exposed to authors as the DOM, but when it comes to rendering, that tree is converted to a layout tree, which represents the boxes to be drawn in a hierarchy of position/size influence. The layout tree consists of layout nodes (Chromium), renderer nodes (WebKit), or frame nodes (Firefox), but these all refer to the same concept.

I started with Firefox and <mspace> because that was the only element that could not contain children. <mspace> represents, well, a space. It has attributes for width, height (height above the baseline), and depth (height below the baseline), each of which can be negative to bring surrounding elements closer together.

I found the element’s frame node and noticed this method:

void nsMathMLmspaceFrame::Reflow(nsPresContext* aPresContext,
                                 ReflowOutput& aDesiredSize,
                                 const ReflowInput& aReflowInput,
                                 nsReflowStatus& aStatus) {
  // [...]

  mBoundingMetrics = nsBoundingMetrics();
  mBoundingMetrics.width = mWidth;
  mBoundingMetrics.ascent = mHeight;
  mBoundingMetrics.descent = mDepth;
  mBoundingMetrics.leftBearing = 0;
  mBoundingMetrics.rightBearing = mBoundingMetrics.width;

  aDesiredSize.SetBlockStartAscent(mHeight);
  aDesiredSize.Width() = std::max(0, mBoundingMetrics.width);
  aDesiredSize.Height() = aDesiredSize.BlockStartAscent() + mDepth;

  // [...]
}

Reflow is the process of traversing the layout tree and figuring out the positions and sizes of all of its nodes, and in Firefox that involves a depth-first tree of nsIFrame::Reflow calls, starting from the initial containing block. An <mspace> frame never has children, so our reflow logic was more or less to take the three attributes, then return a ReflowOutput that tells the parent we need that much space.

To handle padding and border, we add that to our desired size. “Physical” here means the nsMargin in terms of absolute directions like left and right, as opposed to the LogicalMargin in terms of flow-relative directions, which are aware of direction (LTR + RTL) and writing-mode (horizontal + vertical + sideways). We want to use LogicalMargin in most situations, but MathML Core is currently strictly horizontal-tb and sums of left and right are inherently direction-safe, so nsMargin was the way to go here.

auto borderPadding = aReflowInput.ComputedPhysicalBorderPadding();
aDesiredSize.Width() = std::max(0, mBoundingMetrics.width) + borderPadding.LeftRight();
aDesiredSize.Height() = aDesiredSize.BlockStartAscent() + mDepth + borderPadding.TopBottom();

That was enough to pass the <mspace> cases in the Web Platform Tests, but the test page I had put together to play around with my patch yielded both good news and bad news. Let’s look at the reference, which uses <div> elements and flexbox rather than MathML.

The good news was that Firefox already drew borders, or at least border colours, even though the layout of them was all wrong.

The bad news was that while my patch made each element look Bigger Than Before, the baselines were misaligned. More importantly, the <mspace> elements and even the whole <math> elements still overlapped each other… almost as if… their parents were unaware of how much space they needed when positioning them!

I fixed the first two problems by adding the padding and border to the nsBoundingMetrics as well, because that controls the sizes and positions of MathML content. That left the overlapping of the <math> elements, because while they contain MathML content, they themselves are HTML content as far as their ancestors are concerned.

auto borderPadding = aReflowInput.ComputedPhysicalBorderPadding();
mBoundingMetrics.width = mWidth + borderPadding.LeftRight();
mBoundingMetrics.ascent = mHeight + borderPadding.Side(eSideTop);
mBoundingMetrics.descent = mDepth + borderPadding.Side(eSideBottom);

It turns out that in Firefox, MathML frames also need to report their width to their parent via nsMathMLContainerFrame::MeasureForWidth. With the <mspace> counterpart updated, plus the WPT expectations files updated to mark the <mspace> test cases as passing, my patch was ready to land.

/* virtual */
nsresult nsMathMLmspaceFrame::MeasureForWidth(DrawTarget* aDrawTarget,
                                              ReflowOutput& aDesiredSize) {
  // [...]

  auto offsets = IntrinsicISizeOffsets();
  mBoundingMetrics.width = mWidth + offsets.padding + offsets.border;

  // [...]
}

I also put together a test page (reference) for the interaction between negative mspace@width and padding, which more or less rendered as expected, but it potentially revealed a bug in the layout of <math> elements that are flex items. My guess is that flex items use a code path that clamps negative sizes to zero at some point, like we have to do in ReflowOutput, resulting in excess space for the item.

Reftest for padding with negative mspace@width: reference page, without patch, with patch.

Margins were trickier to implement because, with Firefox and MathML content at least, the positions of elements are the parent’s responsibility to calculate. I spent a very long time reading nsMathMLContainerFrame, which is the base implementation for most MathML parents, and eventually figured out where and how to handle margins. With a patch that updates RowChildFrameIterator and Place, and yet another test page (reference) that passed with my patch, we were close to having a template for the remaining MathML elements!

Reftest for margin: reference page, without patch, with patch.

You can see my approach over at D87594, but the patch needed reworking and I ran out of time before I could land it.

Acknowledgements

This internship was incredibly valuable. While I was only able to finish the first trimester for mental health reasons, over the last nine months I’ve learned C++, learned how the web platform and browser engines work, gained ample experience reading specs, worked with countless people in the open-source community, and contributed to three major engines plus the Web Platform Tests.

Were I able to continue, I would also look forward to (more) experience contributing to specs, and probably helping Igalia with their MathML in Chromium project. In any case, my time with the collective has only strengthened my desire to someday join full-time.

Thanks to Caitlin for her advice and support, Eva and Javier and Pablo for getting me settled in so quickly, Manuel and Fred and Rob from the Web Platform team, and Yoav and Emilio for their help on the Chromium and Firefox parts of my work.

Windows is the other major platform that does this. Check out The Old New Thing by Raymond Chen to learn more. ↩
Searchfox more or less supersedes MXR and DXR. ↩
Igalia has a Searchfox-based WebKit code browser, and I found it useful, but it’s not yet ready for public consumption. ↩
See also wpt.fyi, which tracks results of each test case across major browsers. ↩

Farewell, car

2020-08-15T20:00:00+00:00

It’s been two years since I lost my car to a kangaroo. My partner and I went on a road trip from Sydney to Canberra on a whim, but we didn’t reach our destination. We were fine, but the kangaroo sadly wasn’t. Let our story be a lesson to you: don’t drive out of town at night without a roo bar.

Left: the night of the crash (includes GPS EXIF tags). Right: a few months later.

Remarkably the car still kinda worked after we had it towed, enough for it to limp around on some quiet streets near my house at the time, but it was far from roadworthy. I kept it out of the rain at first, but I soon learned that I couldn’t really afford to fix all of the damage, barring some sort of miraculous wrecker dive (for which I had no way to get to, no way to carry the spoils home, and no tools to install). We moved to Strathfield, but left the car parked on the same street in Arncliffe. The next wet season came around, but this time I neglected the matter, sealing its fate.

This car wasn’t my first, but it might as well have been, because I had my first for less than eight weeks. I guess I’ve got a knack for owning and wrecking cars based on the Ford C1. The BK family has a special place in my heart, on multiple levels. At its peak, three of my friends and my best friend’s brother owned one too.

This vessel carried me through my P plates, as far as Perth, Adelaide, Sydney, Canberra, Melbourne, and the Gold Coast. It took me and both of the partners I’ve had on road trips far and wide, to countless ComSSA LAN and RFLAN and avaLANche parties, and it hauled me and all of my stuff from Perth to Sydney when I graduated.

Left: SA/Vic border on National Highway A8. Right: eastern end of the Nullarbor plain.

I poured a lot of love into this car, including custom plates, good tyres, and 18″ RX-8 rims from someone in Canberra. I really liked the stock aesthetic, and in general I think aftermarket rims are tacky.

My friend Nathaniel took me wrecker diving for a centre console from a facelift and installed it for me, giving me an aux jack. The best part was that my stock pre-facelift head unit — fancy internal 6-stack CD player that I never used and all — supported it natively! You see, the tape deck, in models that had one, was connected to the head unit with a glorified aux cord. We plugged it in, held down two buttons to reconfigure the thing (if I remember correctly), and it worked like a tape deck.

I forgive you for losing air conditioning just two nights before I crossed the Nullarbor, for the very first time, in the middle of summer. I forgive you for your flaky dealer-fitted cruise control (drive-by-wire was introduced in the facelift) that barely worked the first time, dying completely before my round trip back home the next year.

But most of all, I’ll miss you.

November 2015 to July 2018.

Bad Apple!! for taskmgr

2020-06-29T19:00:00+00:00

Bad Apple!! is a Touhou song whose music video has been described as “the doom of music videos”, because it’s been played on everything from string lights to oscilloscopes. This post is about that video, writing an improved “video player” for Task Manager’s CPU graphs, and using virtual machines to push my feeble hardware to the limit.

I definitely cherry-picked that thumbnail, not gonna lie.

Prior art

The idea started in March, with a video that appeared to play it on an AMD 3990X, but that one turned out to be “fake”, at least in the sense that it relied heavily on video editing. More recently, @kbeckmann and @winocm played it over 64 and 128 processors, which at first inspired me to replicate what they did, and pretty soon I wanted to bring something new to the table.

The biggest obstacle for me was hardware. The “shaded tiles” style of CPU graphs only kick in when you have 64 or more processors, but my main machine only has eightish, and my laptop only has twelvish. There’s a whole spectrum of approaches I could choose from:

Using video editing to render an imitation (already done)
Modifying taskmgr’s memory to feed it fake activity numbers
Using a virtual machine to Download More Cores
Just recording it on real hardware (already done)

I’m really into virtualisation, so I decided to solve this with libvirt and KVM. While I could probably ask Igalia to let me run a guest on one of their 128-thread build boxes, I was interested in a challenge, driven by the same urge as when I built a home router out of an old Pentium III¹. That turned out to be challenging and interesting, and I’m surprised I was able to mitigate enough of the problems that the result was worth sharing.

Getting started

My main Windows install is a libvirt guest with GPU passthrough, so at first I tried the path of least resistance. I cranked up the CPU topology, rebooted, and… it struggled, taking several minutes to reach the login screen.

I figured that such an extreme overcommitment might be fine after boot, and the only problem was booting that way, so I tried CPU hotplug, but nothing happened. While Windows Server apparently got an update that fixed hotplug, and Windows 10 also received that update, that didn’t mean that Windows 10 supports hotplug.

What I would have seen if there was any justice in this world.

I put together a Windows Server 2019 guest, and at last I could use CPU hotplug! But the guest also booted fine with the overcommitment in place (or at least it does now, if I’ve got my timeline wrong). Perhaps the sheer lack of crap it had to load, by virtue of being a fresh install and Windows Server, did the trick.

I turned off the malware scanner for good measure.

The black screen phase

My video player has its roots in @kbeckmann’s bad_cpu.py, so let’s start there. The latter spawns a process for each “pixel” of the CPU graph, each of which sets its affinity to one CPU, then spins and sleeps as a kind of pulse-width modulation over CPU activity, just like a microwave or (some types of) dimmer switch.

Sadly bad_cpu.py doesn’t scale beyond 64 processors without modifications, because of how the Windows API communicates affinity. Since the dawn of time, affinity has been represented as a register-width bit mask. Windows is big on backwards compatibility, so when support for more than 64 (or 32) processors was introduced, the developers introduced the concept of processor groups.

The library we were using didn’t provide a way to change processor group affinity, Python’s bindings for the Windows API didn’t include SetThreadGroupAffinity, and even if it did, what if the interpreter creates other threads than the one GetCurrentThread would give us a handle for?

That’s when I decided to Rewrite It In Rust. As far as Windows was concerned, this initially involved three things:

GetLogicalProcessorInformationEx to find out what the processor groups are
InitializeProcThreadAttributeList and UpdateProcThreadAttribute to wrap each processor group affinity value in some fancy paper and a bow
CreateRemoteThreadEx to create each pixel thread with its affinity value

I struggled with steps one and two for most of Friday.

GetLogicalProcessorInformationEx is named after GetLogicalProcessorInformation, a similar function that doesn’t support processor groups. The latter had an example, from which I learned that it writes an array of fixed-size structures, so I assumed that this was also true for the former, but when I tried calling the function, I got nonsense. It wasn’t until I found this usage in the wild that I learned the newer function writes an array of variable-size structures!

While it was clear that our affinity values had to outlive the attribute lists that wrapped them, it was unclear whether the attribute lists had to outlive the threads they were used to create. I eventually tried to do both, but I don’t think my approach guaranteed that they wouldn’t move. I couldn’t figure out how to Pin them, nor was I even sure whether that was the appropriate tool.

c8cba094f63d3

7d01d8501f5fb

By the end of Friday, I got some test patterns going, but I was disappointed to see that the percentages were distorted downwards, making pixels too light, when too many pixels needed to be dark. Migrating the virtual machine to my laptop helped, but not enough.

12/240 versus 120/240 pixels at 50%.

From there, implementing external video input was easy, probably the easiest part of the project. Slurp up the file, throw in a couple of loops with Rust’s wonderful ChunksExact, and dole out each pixel to the appropriate vector.

55dade1bcfcac

Performance tuning

Having seen disappointing results from both bad_cpu.py and my new player, at first I looked beyond the code for solutions. The ideas I came up with fell into three buckets:

Messing around with schedulers, be it on the guest (α β γ δ) or the host (ε). I didn’t know enough to come up with informed ideas, so I didn’t get very far.
Optimising the virtual machine, based on the same advice that I heed from the VFIO community and their resources. libvirt’s defaults are generally top notch, but it still leaves room for tweaks like switching to 1 GiB static huge pages and removing the memory balloon. That said, nothing had an effect to write home about.
Pinning the guest threads, which I’ve pulled into its own dot point because it did have a notable effect. Massive and negative. This is a common tweak for normal guests that aren’t oversubscribed, but here? The guest performance tanked like it did when I tried to overcommit my main install, and CPU hotplug just delayed the tanking. Even without nohz_full (which I later learned was a silly idea when overcommitting pins). Even without rcu_nocbs. Even without isolcpus or cset shield.

I also ran into a bunch of limits while figuring out how far I could push my hardware.

32 (KAFFINITY on i686-pc-windows-gnu, rustup’s reasonable default)
64 (KAFFINITY on x86_64-pc-windows-msvc, which I switched to)
255 (pc-q35-4.2 without iommu@eim)
288 (pc-q35-4.2 with iommu@eim)
64 sockets (Windows limitation?)

To go beyond 255 logical processors and reach the 288 limit, we needed iommu@eim, which also needed iommu@intremap and the qemu ioapic.

<domain>
  <features>
    <ioapic driver="qemu"/>
  </features>
  <devices>
    <iommu model="intel">
      <driver intremap="on" eim="on"/>
    </iommu>
  </devices>
</domain>

For what it’s worth, by the time I recorded the video, I ended up with this configuration.

High Frame Rate™

Task Manager has a bunch of update speeds, but the fastest is only twice a second. The good news is that we can also press F5 to update as often as we want. Once I added a thread that calls SendInput after every frame, we could play the video at 4 fps.

20e17d39c0c98

Well… kind of. SendInput can only press F5 in the foreground window, so I had to Alt+Tab every time I ran the program. SendMessage would probably be better, but I later worked around that with FindWindowW and SetForegroundWindow.

096347fca5582

7c6e17d450517

From here on, I turn off automatic updates, because they split our carefully timed refresh cycles into two corrupted updates.

Making the player press F5 between frames also ensures that Task Manager’s cycles are in phase with the frames (albeit with a small error between F5 and refresh). This might not seem like a problem at first, much like the number of days from X January to X February is always the number of days in January, but when we’re too far out of phase, the dark pixel in a dark-to-light transition won’t be dark enough (and vice versa).

When the refresh is 20% late, the 80% pixel becomes 70%.

There are limits to this technique. The higher the frame rate, the more of our CPU time gets consumed by Task Manager, until the tiles become wildly inaccurate.

Timing is everything

By this point, I was noticing more and more severe timing problems. The simple approach taken by bad_cpu.py was perfect for real hardware, but on an overcommitted virtual machine, that didn’t cut it. There was excessive visual noise, and when I started playing the original video side-by-side on another machine, I noticed that our playback finished around 30 seconds late.

My first thought was that Instant, despite being monotonic, is “not guaranteed to be steady”, such that “some seconds may be longer than others”. But on Windows, it’s based on QueryPerformanceCounter, which provides much stricter guarantees. QPC uses the best available counter on a given machine (e.g. RDTSC, RDTSCP, HPET, ACPI PM timer) plus its own magic² to create timestamps unaffected by system time, dynamic frequency scaling, multiple processors, and hypervisor conditions (assuming a reasonable hypervisor). These timestamps are even meaningful across threads (± 1 tick)!

I then worried that our F5 thread was getting delayed by all of the busier pixel threads. I didn’t measure this one, and I doubt it was a huge problem unless we’re on a frame that’s almost completely filled, but I used SetThreadPriority to keep the pixel threads out of the way. I would have preferred to set the threads’ priorities at creation time with something like a ProcThreadAttributeList, but that didn’t seem to be possible. THREAD_MODE_BACKGROUND_BEGIN looked attractive, but upon closer inspection, that’s an orthogonal setting for “resource” (I/O) scheduling, which we never do during playback, so I chose THREAD_PRIORITY_IDLE.

9883355e6e864

One of the biggest problems turned out to be keeping the pixel threads in sync. Each pixel thread started its playback as soon as it could execute. I used an AtomicUsize with fetch_add(1, Relaxed) to measure the time from right before the first CreateRemoteThreadEx to the start of the last pixel thread to gain execution, and it was a massive 150–250 milliseconds! I struggled to imagine which of Windows’ synchronisation primitives would be an efficient solution here, but Rust came to the rescue with Barrier, which blocks waiters until a given number of waiters are waiting. That number is one for every pixel thread plus one for the F5 thread, and I added waits to the start of both functions’ main loops.

With all of the threads in sync, I eliminated the sleeps from the pixel threads, elevating the importance of the F5 thread to what I’ve since called the “clock” thread. The pixel threads’ main loop was wait → spin → sleep, but the clock thread was wait → sleep → F5, so skipping directly to the wait was effectively a sleep.

9992c03d6068f

This improved the picture quality, but playback still consistently finished too late. Given the simple algorithm “spin for darkness / fps, sleep for (1 − darkness) / fps”, this kind of problem is always present (if negligible) even on real hardware. Calculating how long to sleep for is very fast but not instant, and sleeping for some amount of time doesn’t guarantee that the scheduler will return us to execution in that time.

The first thing I tried was to adjust the sleep time to catch up with (or if necessary, wait for) the nominal time we should start the next frame, based on when playback started. I called the difference between what a naïve sleep would give us and the nominal time lateness. This helped at very low resolutions where I disabled most of the pixels on the canvas, but what I otherwise found was that our adjusted sleep was always zero.

There was so much lateness that we couldn’t keep up.

This was actually bad news and good news at the same time. The bad news was that we would need to do more to fix our timing problems, but the good news? Even when we effectively removed sleeps, the picture was still recognisable, and if anything, the change helped us with our “washed out” distortion. I was worried that this might not be the case, because if you look at each pixel thread in isolation and ignore the Barrier, wouldn’t all spin and no sleep yield 100% activity?

I now believe — but correct me if I’m wrong — the percentages are based on how much of each wall second (or other period) a given processor is busy for, and when a guest processor is preempted against its will, that time doesn’t count, like how you don’t experience the time you’re asleep³. This would explain the distortion that occurs when a guest processor can’t run because the host processors are contended!

One thing that I was surprised to see made zero difference was switching from a debug build to a release build. It ended up being so irrelevant to this workload that while I put --release in the README, I forgot to use it when recording the video!

7a9150a4e5e9c

But the technique that made the biggest difference to our timing problems was dropping frames. When we’re running more than 1 / fps late, not even a 100% empty frame gives us enough room for sleep adjustment that we’ll catch up completely by the end of the frame. Figuring out how the clock thread could safely communicate this to the pixel threads was difficult, but I decided to publish the next frame index with a RwLock, which I would describe as Rust’s borrowing rule plus the ability to wait.

Strictly speaking that wasn’t enough, because while the Barrier syncs up all of the threads at the start of each frame, we have no way of knowing when the write acquisition (clock thread) happens relative to the read acquisitions (pixel threads). Did it happen before all of them, after all of them, or somewhere in between? Either of the first two would probably be fine if only we could make it consistent, but somewhere in between is very bad. If that happens, some pixels will draw their part of the old frame and others the new frame, which sounds like the relatively minor problem known as horizontal tearing… until you realise that the read acquisitions don’t happen in scanline order.

To prevent that, I added a second wait on the Barrier to both functions’ main loops, dividing each frame into two phases based on the next frame index: read-only and write-only. In the read-only phase, all threads read that variable (and do the rest of their work), then in the write-only phase, the clock thread writes the new index. I’m sure there are better tools for the job though — it relies on me writing code that obeys my own rule, without any compile-time or runtime enforcement.

156c45ea6c6e3

5065d9566d7b6

8a2d3d2288aa8

The present tense

I wrote a video player for taskmgr, and it now works well enough to play Bad Apple!! on an 11x6 canvas with only a laptop and the power of virtualisation. Picture fidelity and frame rates are far better during the black-on-white shots, but at least some of the white-on-black shots managed to yield more than a complete blur.

Playback at 4 fps or even 10 fps is now possible, which is an improvement that even the Real Hardware folks can enjoy, but if I could drive 66 pixels with only 12 hardware threads, imagine what we could do with 64 or even 128!

I still use that router at home today, almost five years later, and I’m only planning to decommission it because I recently upgraded to gigabit internet, which the poor thing just can’t keep up with. ↩
This includes workarounds for lying firmware, handling of unsynchronised TSC values across processors, and much more. I would seriously recommend reading Microsoft’s documentation about this, it’s very interesting! ↩
I don’t think this is actually true, but I’m sticking with the analogy. ↩

An open letter to Bitbucket about Mercurial

2020-06-07T09:00:00+00:00

In August 2019, the Bitbucket team announced that they would “sunset” Mercurial on 1 June 2020 (now 1 July 2020) by removing all Mercurial repositories and associated content. I urge them to reconsider that approach with this open letter.

Hi. I’ve enjoyed using Bitbucket for a long time, though I’m pretty sure I joined after you added Git support, and I’m not afraid to make other sites enjoy you too.

The merits of discontinuing active Mercurial support are their own can of worms. I don’t really have a strong opinion there one way or another, and I’m sure you’ve heard more than enough about that in your forum thread, which received so many replies that I couldn’t read them all.

I have a passing interest in Mercurial (thanks to you), but that’s nothing compared to how much I care about digital preservation, that is to say, the long-term health of everything we create with our computers.

On this basis, I believe your approach is irresponsible. I don’t have all the answers here, but there’s a lot of middle ground between “full Mercurial support forever” and “delete fucking everything”. You can do better.

Let’s explore some of that middle ground, but first, an appeal.

If nothing else…

Archive Team will start building a best-effort archive soon, with or without your help. Please be nice to them, don’t block them, and have someone on your team take point on any questions or problems that come up. You can reach them in #kickthebucket on hackint, or hit me up and I can pass on some contact details.

If you’re not going to change your mind, it would be best to reach out to the people who live and breathe online digital preservation, above or at the Internet Archive, and find out how you can make their work easier. I would imagine that this could mean anything from passing on dumps of the underlying data to setting up redirects to an archive, but the opinions in this letter are my own.

Why you should care

Leaving it up to third-party archivists to figure out how to scrape everything from a production service is far from ideal. When this happens, it might not be easy (or even possible) for users to remove or supersede their archived content. There are things that can’t be scraped of course, including private repos and some metadata like environment variables, but with some collaboration, there might be ways to preserve those safely too.

Placing the burden of migration on authors, while convenient and flexible, will effectively destroy the works of authors who aren’t around to defend them. No author stays an active user forever, every author dies someday, and not all works will find a new maintainer before one of those things happen.

All of the recommended migration options destroy issues, pull requests, downloads, and Pipelines results, or at least all of the links to them. The raw commits are no substitute for the knowledge and history in that content. Said archivists are working on some more complete tools, but even if they’re complete tomorrow, only you can prevent link rot.

Build with heart and balance

There’s a difference between obligation and responsibility. In the world as we know it, your only strict obligations are to those who have paid you to do something.

When you build a medium for people to collaborate and share their works with the world, you have responsibilities to those works. You can think of this as the price of inviting free users to build your platform and establish your place as a major competitor to that other service with the five-limbed feline mascot.

Our cultural output over the last century is in unprecedented danger of being lost forever, and yes, that includes code. This isn’t a foregone conclusion though, and you have the power to shut down your service in a way that avoids destruction.

More middle ground

You could step down to read-only access, potentially by converting the content to a static HTML format. No one says read-only has to mean seamless integration with your evolving frontend, and if anything, everyone wins if you do it in a way that relieves you of having to think of compatibility with each change.

You could convert the repos to Git in place. If you do nothing, authors are going to have to convert and/or relocate their repos anyway, but no matter what they choose, they lose all of their issues and other non-code content. You are in a unique position to keep this content alive, as well as keep commit links working by, for example, redirecting Mercurial hashes to Git hashes.

Google Code operated in the same space from 2006 to 2015, and even mentioned you in their farewell post. When that service shut down, every project was archived, including code, issues, wikis, and downloads. Links were kept alive, like this project link. Authors were given a way (albeit not a very convenient one) to remove archived projects, or set up redirects, after the archive date.

None of these options are easy, but I trust you to do the right thing and try.

Ad vigendum per aspera

2018-10-10T05:13:13+00:00

The last four years have fucked with me in ways out of which I’ve only really started clawing myself over the last year or so. I went from feeling like I could do anything and improve the lives of the people around me, to feeling like an unreliable piece of shit whose only skills were botching my responsibilities and disappointing the people I knew. I gave up on learning to play the hand I was dealt, neglected the people I love, and grew to cope with all of my problems the only way I knew: by ignoring them. But the scales are finally starting to tip in my favour, and it’s time for me to end my exile.

2016

I quit my internship, like the last two, because I refused to continue taking money when I couldn’t think clearly enough to do even half of the work I should’ve been doing. Three internships — 2014, 2015, 2016 — and three early resignations. This was a huge blow: my teaching job was the only one of the four jobs I had ever been hired for that I could hold down for more than a few months, but I immediately dismissed it as an outlier and started believing this was the just deserts for my intrinsic incompetence.

I learned that I have ADHD, a condition whose symptoms would later be described as comparable to early dementia (Callahan et al. via @FioraAeterna), and started treating it with stimulants, which went a long way towards helping me through my final year.

I crossed the finish line without failing a single unit (though a couple were close). I still lived with my parents, so unbeknownst to me, my life was about to get way harder, and this gave me a false sense of confidence in the completeness of stimulant therapy alone.

2017

I fucked off to Sydney to start my new job, leaving way too many loose ends untied. I had underdelivered on the handover that I promised the new ComSSA committee, and to add insult to injury, went unreachable for months at a time, and would end up waiting a year before making a serious attempt to deliver the rest.

A couple of months in, work sent me to my alma mater to shill our graduate program, together with the two colleagues that I graduated with. I procrastinated writing my slide deck until my taxi ride there, but unsatisfied with the mere punishment of presenting bad slides, my utterly broken time management delivered me to the venue so late that my part should’ve started and ended by the time I arrived, forcing my colleague to improvise it.

Without the skills I needed to exercise agency over my cursed brain, my life quickly deteriorated and I floundered at work. I spent what felt like every waking hour either at work or struggling to get ready for work or ensuring that I could make it to work on time, but for all my effort I received a mediocre performance review and no promotion.

Mum suggested that we plan my time in Perth when I went home for the holidays. This was pretty foreign to me — planning wasn’t something I had ever done of my own volition — but so was the result, which marked the start of my belated learning of the truth behind the “drugs and therapy” advice that every Mental Illness 101 includes.

2018

One of my closest friends invited me to his wedding. I prepared my outfit and gift and I was happy with the fruits of my preparation, but I left some of it to the morning of the day. I rushed to show up on time but misread my invitation, mistaking the reception place for the wedding place, and I missed the wedding.

I’ve started getting the hang of my limitations and adapting to them with appropriate tools. I remind myself to do things with timers, alarms, or calendar events, depending on how long I’m deferring them by. I’m learning how to use org-mode and Trello as structured extensions of my memory, rather than write-only black holes like Sublime Text buffers. These are just a couple of examples, and ADHD management is definitely a subject worth exploring further, but that’s a job for another day.

I’ve turned away when I knew I made a mistake
Instead of dealing with it

I’ve wasted good chances I’ve had in this life
That other people won’t even get
I put off setting things right with Andy
And now he’s dead

— Run From What’s Comfortable by Pat The Bunny

Reading this essay about ADHD by @gravislizard was a crucial moment that brought me to tears, and I think it’s just as useful an introduction to what I grapple with every day. Pretty much all of the experiences he wrote about resonate with me, but there’s one that I’ll highlight shortly, because it helps explain something I wrote a couple of years ago:

To all who have sent me a thing and are still waiting for a response (be it for a day or for ten months), I’m sorry. I don’t mean to be rude.

The short version of my excuse goes like this. Every time I reply to a message, I feel like I’m faced with two more to which I haven’t, and I feel overwhelmed, so I cope with that feeling by ignoring the cause. As the problem grows, I feel more anxious about the thought of saying “sorry for the late response” with a straight face, when all this time, I knew how long you’ve been waiting, and you know how long I’ve kept you waiting, so I ignore it. I haven’t learned how to break a task into manageable pieces, and if I can’t solve a problem thoroughly in one sitting or a few, I ignore it.

What might not make sense is why I would see replying to a message as a difficult task (though I’ve conflated difficult with big there), but as he explains, it can be really hard for me to follow through and finish my replies between getting lost in the woods and straight up forgetting, but I needed to realise how neglectful it was for me to concede defeat:

Suppose I’m at work and I think, “I need to send an email to <people> about <issue>.” Before I can do anything, I’m already thinking about the issue.

I try to write the email, but as I’m writing it, questions fill my head. I try to push them aside and just concentrate on the initial task, send a simple email, but the questions keep hounding me. I have to stop and get answers to them or I can’t think about the email. And if I can’t get answers, I’m stuck. I just can’t proceed. I end up staring at the empty compose window for ten minutes, writing the first sentence over and over, because my mind is so far out of the game I can’t even do basic grammar.

ADHD’s also a motherfucker. get texted at 8AM “wanna hang out tonight”, read it, forget to reply until 3PM.

— @gravislizard on 31 December 2017

I’m not saying you owe people a timely reply to their texts, right? hey check this out: i absolutely am. like hey it’s your friendships but when you don’t reply to people it actually does hurt them. source: me spending the entire day feeling like shit on both sides of this

— @gravislizard on 31 December 2017

If you reach out to me, you should hear back from me by the end of the day (or the next at worst). I’ve made commitments like this with a few of the people in my life, and those have been much easier said than done, so how will this be any different?

I’ll try to reply in chunks that mirror my thought processes, rather than set out to write one perfect message that’s completely devoid of uncertainty and unknowns. The latter encourages the very kind of “depth-first search” that will leave me lost in the woods, and it’s not really how conversations are supposed to work.

I’ll get comfortable with admitting that from time to time I’m too cooked to respond properly. Even if it means you might need to suspend your disbelief, because you’ve only sent me a funny video and how hard can a response to that be? Accepting that some of my 503 errors might make me look silly is better than working myself up to an extended panic attack that I’ll cope with by ignoring you for three months.

I’ll remind myself that — like any goal I want to convert to a habit — some number of failures will be inevitable, and they’re a signal that I should look for ways to tweak my approach and make it more reliable, not that my whole effort was pointless and I should put it in the bin. As obvious as this sounds, it wasn’t obvious to me until a year ago, before which I believed I couldn’t make or break a habit for longer than three days.

One thing that hasn’t helped my state of mind over the years was the knowledge that I’m trans while feeling unable to do anything about it, but I’m elated to say I’ve finally started my transition (they/them for now, but don’t panic if you drop a he/him here and there). Conveniently enough دیلان (dîlan) isn’t exclusively masculine, and one source even describes it as a woman’s name, so I’m pretty comfortable with my name.

Shout out to Adam, Alex, Alex, Aria, Brock, Carmel, Cyra, Emily, Jaci, James, Jason, Josh, Kieran, Krystal, Luke, Mike, Mike, Millie, Sandro, Scott, my parents, and anyone who hasn’t given up on me by now. I wouldn’t be here without your support.

I can see the light at the end of the tunnel. Will you meet me there?

Choosing a software licence

2015-12-21T14:00:00+00:00

The licence that one releases their software under is often a topic that’s given less thought then it perhaps deserves. I’ve been releasing my code under the MIT (Expat) licence for as long as I can remember, but I thought it might be prudent to take a closer look at what’s out there in the wild world of software licensing.

Teal deer

Unless otherwise specified, neither this page, nor any of the pages that I’m about to link to, necessarily constitute Proper Legal Advice™. If your project matters enough that the choice of its licence may have a significant effect, please hire an actual lawyer. This means that you should definitely steer clear of Lionel Hutz.

If you want to choose a licence, and you want to make the most reasonable choice you can in the least possible time, use GitHub’s Choose a License. For most scenarios, the choice basically boils down to Expat, Apache 2.0, or the GPL.

If you’re looking for a comprehensive reference on the pros and cons of the licence you’re considering, check out the Free Software Foundation’s Various Licenses and Comments about Them, which is only occasionally didactic about copyleft licensing. You may also find TLDRLegal and OSS Watch’s Licence differentiator useful.

Licence compatibility

Whether or not two licences are “compatible” is important for when programs of those licences are combined, derived, or linked together, but the details depend on who you ask. For some, compatibility is achieved when works of the given licences may be combined to form a larger software project, while for others, compatibility also requires that one of the constituent works’ licences be able to “dominate”, licensing the entire derived work. Compatibility is generally not a symmetric relation.

The public domain

Placing your work in the public domain is simple, in theory, as doing so should relinquish your ownership and all of your rights over said work. Very few jursdictions, however, provide a practical way to do this, and in some jurisdictions, you may not be allowed to do this at all! For people in those places, your work will effectively continue to be “all rights reserved”, rendering it unusable for them.

Enter CC0. It’s essentially a licence that attempts to waive all rights over a given work in a given jurisdiction, and failing that, places the work under a licence with the most permissive terms allowed in that jurisdiction.

CC0 is usually referred to as a “tool” instead of a “licence”, because where a work may be placed in the public domain, CC0 no longer bears the role of a licence. Unlike most Creative Commons licences, CC0 is suitable for use with any kind of work, including software. The FSF prefers CC0 for releasing works into the public domain.

The novelty licences

Other attempts to solve the issue of placing works in the public domain, include Banlu Kemiyatorn and Sam Hocevar’s Do What the Fuck You Want to Public License, or WTFPL for short. While on the surface, writing a licence to informally relinquish all rights seems like a reasonable solution, the Open Source Initiative has noted that without a clear and explicit grant of rights, the WTFPL is no more universal than any other lone attempt to place a work in the public domain.

By lacking any explicit legal terms, the WTFPL also fails to disclaim authors’ warranties and liabilities for their software, protections that are provided by virtually all serious free and open licences. As such, while “novelty” licences like the WTFPL are certainly amusing, they’re not a wise choice for any significant projects.

The copyleft licences

In an effort to ensure that free and open works remain so, even after they’re distributed or modified to create derivative works, “copyleft” licences like the GNU General Public License were created. Richard Stallman described the concept succinctly in his GNU Manifesto: “no distributor [of the GNU operating system] will be allowed to restrict its further redistribution”.

When distributing software released under a copyleft licence, whether or not any modifications have been made, the source must be included or otherwise made readily available along with any binaries or other forms of the software.

Some examples of copyleft licences include the GNU GPL family, the Mozilla Public License family, as well as Sun Microsystems’ derivatives of the MPL, the Sun Public License and the Common Development and Distribution License.

While version 2.0 of the MPL has been updated to explicitly introduce compatibility with some of the GPL licences, the SPL and the CDDL are based on version 1.0 and 1.1 of the MPL, and this is — at least in part — why they’re incompatible with the GPL.

The permissive licences

Free and open licences which are not copyleft are usually referred to as “permissive” licences. Like all free and open licences, permissive licences adhere to the Free Software Definition, and programs licensed under a permissive licence grant anyone the freedom to:

Run the program in any manner, and for any purpose;
Study how the program works, and change it to suit one’s needs;
Redistribute copies (of the original program) to others; and
Distribute copies of one’s modified versions to others.

Permissive licences almost always specify a common set of additional terms:

The authors disclaim all warranties and liabilities for their software;
Copies and derivative works must credit the original authors; and
Copies and derivatives must retain the original copyright notices.

There are loads of permissive licences out there, including but not limited to:

The MIT licences (Expat, X11, and XFree86);
The BSD licences (“prior”, “original”, “revised”, “simplified”, and FreeBSD);
The University of Illinois/NCSA licence (UIUC or NCSA for short);
The ISC licences (with or without the FSF’s clarification); and
The Apache licences (versions 1.0, 1.1, and 2.0).

They’re all worth mentioning here, because they share a handful of subtle differences.

Advertising and endorsement clauses

Being permissive licences, the MIT, BSD, NCSA, ISC, and Apache families of licences are nearly identical in their spirit and terms.

Chronologically speaking, there are five licences that have been used in the BSD and its derivatives, although only the last three are really relevant for modern usage:

The “prior” BSD licence (1988), as used by 4.3BSD-Tahoe;
The “original” BSD licence (1990), with four clauses;
The “revised” BSD licence (1999), with three clauses;
The “simplified” BSD licence, with two clauses; and
The FreeBSD licence, which also has two clauses.

The earliest two versions of the licence are roughly equivalent, and they have clauses that are known as the “advertising clauses”, which require that any advertising for products or services that contain licensed code must include an acknowledgement of its authors.

You might correctly imagine that this requirement can prove very unwieldy, very quickly, and as such, they aren’t approved by the OSI, nor are they compatible with many popular licences like the GPL. The “revised” version removes the advertising clause, making it far less troublesome and incompatible with other free and open licences.

All three of these versions, however, include another clause that requires permission prior to using the names of a work’s authors and contributors to “endorse or promote” derivative works. Contrary to my initial assumption, this doesn’t preclude the “revised” version from compatibility with the GPL, nor from being considered free and open.

The “simplified” BSD licence removes this “endorsement” clause too, leaving behind a fairly concise permissive licence, and the FreeBSD licence is based on this version, with the addition of a paragraph stating that the “views and conclusions” of the individual authors don’t represent the FreeBSD Project as a whole.

As for the MIT family of licences:

The Expat licence is similar to the “simplified” BSD licence;
The X11 licence is similar to the “revised” BSD licence; and
The XFree86 licence is similar to the “original” BSD licence.

In other words, the X11 licence is like the Expat licence plus an endorsement clause, and the XFree86 licence is like the X11 licence plus an advertising clause.

The NCSA licence, put simply, is based on the “revised” BSD licence, with some minor improvements in wording drawn from the X11 licence to improve precision.

Version 1.0 of the Apache licence is similar to the “original” BSD licence by virtue of having an advertising clause, while version 1.1 removes the clause, making it vaguely like the “revised” BSD licence. These versions of the Apache licence weren’t really designed for general use by other projects however, and their strict and specific prohibitions on the use of Apache-related names, along with their incompatibility with the GPL because of these, make them rather unattractive options.

Finally and most elegantly, the ISC licences are “functionally equivalent” to the Expat and “simplified” BSD licences, but thanks to the Berne convention, they’re over 30% less wordy than Expat, and over 40% less wordy than the BSD two-clause!

Patent grants and retaliation clauses

Most of the permissive licences we’ve discussed so far were written before software patents were widespread. With these licences, anyone who uses a program that’s covered by one or more patents may risk being the target of litigation from patent holders.

There are two kinds of clauses that a software licence can use to help mitigate this risk: grant clauses grant users a licence to use any necessary patents, and retaliation clauses revoke these patent licences from anyone who initiates litigation.

Enter version 2.0 of the Apache licence. This licence contains both a grant clause and a retaliation clause, making it widely recommended as the most robust licence with respect to its protections against patent litigation. The retaliation clause alone subsequently made its way into the GPL 3.0 and the MPL 2.0.

The Creative Commons licences

Mainly geared towards artistic and other creative works, the Creative Commons family of licences give authors a variety of choices about how their works may be used. Regardless of the choices that a given author makes, anyone may — at the very least — distribute any work that’s in the Creative Commons worldwide, but:

The work must not be modified, regardless of its distribution;
Attribution for the authors of the work must be preserved; and
The distribution must not take place for commercial purposes.

From here, authors may tweak the terms of the licence by:

Allowing the commercial distribution of their work; and/or
Allowing derivative works:
- Provided that they use the same licence; or
- Regardless of the resultant licence.

It’s not a good idea to use any of the Creative Commons licences, except for CC0, for software projects, because they lack specific terms that are desirable for software licensing, and they’re incompatible with most free and open software licences, among other reasons.

When readable source isn’t open source

That one is allowed to read the source code of a work alone doesn’t imply that the FSF will consider its licence “free”, nor will the OSI necessarily call its licence “open”. In the case of the OSI, the availability and legibility of source code merely forms a part of one of the ten criteria that the Open Source Definition comprises.

One example of such a licence is the TrueCrypt licence, which not only has excessive restrictions on how licensed works may be derived from or distributed, but also forbids anyone who doesn’t understand the licence from using a licensed work, which is a clear violation of freedom 0 of the Free Software Definition.

Another licence that fits the bill here is the JSON licence, which Douglas Crockford uses for many of his popular programs such as JSLint and JSMin. It’s identical to the Expat licence, with the exception of an additional sentence: “The Software shall be used for Good, not Evil.”

While many people may agree that this additional “restriction” is clearly tongue-in-cheek, as it’d be ineffective at preventing any actual evildoers from using a licensed work to do their evils, it makes the JSON licence a nightmare to use for any meaningful project.

The subjective nature of evil could potentially put users of a licensed work at risk of being a target of litigation without their knowledge, and this restriction is also a clear violation of freedom 0, making it incompatible with essentially all free and open licences.

Colloquial ambiguities

Just as the Creative Commons, GPL, MPL, and Apache licences must be disambiguated with a version and possibly some licensing settings, the “MIT licence” and “BSD licence” need to be qualified with which one of the three or five variants they encompass is being discussed respectively.

Referring to the NCSA licence isn’t at all ambiguous, and the only ambiguity when discussing the ISC licence is the clarifying replacement of “and distribute” with “and/or distribute” to allow the licence to be approved by the FSF, because some projects like OpenBSD continue to use the old wording.

Vanity and aesthetics

It’s also interesting to see that these free and open licences are usually named after:

A software project: GNU GPL, MPL, Apache, BSD, Expat, X11, XFree86;
A university: MIT, UIUC/NCSA;
Another organisation: Creative Commons, ISC; or
Its contents: WTFPL.

While you could chalk it up to me being a little bit weird, I have a slight but irrational aversion to preferring licences that fall into the first two categories.

Relicensing my work

At the exceedingly rare risk of these being foreshadowing words, I don’t see myself being the target of patent litigation any time soon, and I don’t feel like a copyleft licence provides a meaningful benefit for my code over a permissive one.

I think I’ll relicense my projects under the ISC licence over the next week or two — not that it matters, as virtually nobody actually uses the code I’ve written, but because it’s cute as fuck and it shares its semantics with the Expat licence that I’m currently using.

The modern OpenBSD home router

2015-08-06T17:00:00+00:00

It’s no secret that most consumer routers ship with software that’s flaky at best, and prohibitively insecure at worst. While I’ve had good experiences with OpenWrt and pfSense, I wanted to build a router from the ground up, both to understand the stack and to have something to tinker with. I found many solid tutorials out there, but few of them covered the intricacies of both PPP and IPv6. Here’s what I’ve learned.

Choosing the hardware

The hardware itself is not very modern at all. In the interest of a challenge, I chose the oldest computer that could reasonably replace the Billion 7800N that I was using. Meet the IBM Aptiva, model 2194, whose tiny 95 watt power supply drives some hardware that’s nearly old as myself:

cpu0: Intel Pentium III ("GenuineIntel" 686-class) 602 MHz
bios0: vendor IBM version "PTKT09AUS" date 05/15/2000
bios0: IBM 219443A
ral0 at pci1 dev 3 function 0 "Ralink RT2561S" rev 0x00: irq 11, address 00:21:29:e2:c6:03
em0 at pci1 dev 4 function 0 "Intel 82541GI" rev 0x05: irq 5, address 00:1b:21:56:16:9c
em1 at pci1 dev 5 function 0 "Intel 82541GI" rev 0x05: irq 3, address 00:1b:21:56:1b:86
spdmem0 at iic0 addr 0x50: 128MB SDRAM non-parity PC133CL2
spdmem1 at iic0 addr 0x51: 128MB SDRAM non-parity PC133CL2

After upgrading the memory’s speed and capacity, choosing Ethernet cards was easy. Almost all of mine were based on a Realtek RTL81xx or Intel 82451PI controller, both of which have excellent support with OpenBSD. The Realtek-based cards I owned had an empty socket for a PXE ROM, or no socket at all, so I opted for a pair of Intel PRO/1000 GT Desktop cards instead. A quick test with nc(1) shows that I can only push about 27 MiB/s through the router before the CPU becomes a bottleneck.

Finding a wireless card was more difficult, even though the spare cards I had only supported 802.11g. Of those, only one has a driver that supports hostap mode:

SMC SMCWPCI-G (Qualcomm Atheros AR5005G(S) 168C:001A): no driver
Netgear WPN311 V1H2 Rev. A3 (Atheros AR5005G(S) 168C:001A): no driver
Netgear WG311v3 Rev. A1 (Marvell Libertas 88W8335 11AB:1FAA): malo(4) ✗
Cisco Linksys WMP54G ver. 4.1 (Ralink RT2561S 1814:0301): ral(4) ✓

Conveniently, unlike Marvell, Ralink has also allowed OpenBSD to freely distribute the required firmware without necessitating the use of fw_update(1).

I’m still connected to the outside world through an ADSL service with Internode. Because internal modems for anything newer than G.992.1 are hard to come by, I simply dug out an old Netgear DG834Gv2 and put it into bridge mode.

To put the DG834Gv2 into bridge mode, head to /setup.cgi?next_file=mode.htm on the device, then ensure that the ADSL parameters are correct and wireless is disabled. The 7800N would not only have been a waste of fancy hardware, but doing the same on that made it unreachable over IPv4, whereas the DG834Gv2 remained reachable. On the other hand, the 7800N can lie to your DSLAM or MSAN about the SNR of an ADSL connection.

Having configured the hardware and installed OpenBSD 5.7 without any problems, connecting to the Internet soon became my next task.

Interface layout

Behind every fancy router is a bridge, and a bridge is what I made:

% cat /etc/hostname.bridge0
add vether0
add em0
add em1
add ral0
up

em0 goes to the DG834Gv2, em1 goes to a D-Link DGS-1008D, a dumb switch, ral0 hosts the wireless network, and vether0 serves two purposes: not only does it decouple whether or not IPv4 and IPv6 will work from whether or not a particular physical interface is up, but it also yields a stable interface identifier that’s independent of any physical interface.

% cat /etc/hostname.em0
up

% cat /etc/hostname.em1
up

% cat /etc/hostname.ral0
media autoselect mode 11g mediaopt hostap
nwid deLAN
wpakey hunter13
wpaprotos wpa2
wpaakms psk
wpaciphers ccmp
wpagroupcipher ccmp
up

% cat /etc/hostname.vether0
inet 172.19.1.1 255.255.0.0
inet6 eui64

% sh /etc/netstart em0 em1 ral0 vether0 bridge0

Reaching the outside world

As far as I know, Internode is the only residential ISP in Australia that provides native IPv6 without the use of any transition mechanisms. While some consumer routers like the 7800N Just Work™ in this regard, others don’t, like the Netgear WNDR3700v2, for which the best connectivity reachable with stock firmware is 6to4, because it otherwise erroneously tries to establish two separate PPP sessions — one for each protocol.

Looking downwards, the stack of protocols sent over an Internode ADSL connection, sometimes known as PPPoEoA for short, is rather complex:

Transport layer or application layer payload
IPv4 packet header or IPv6 packet header [RFC 791, 2460]
PPP packet header (2 octets, type 0021₁₆ or 0057₁₆) [RFC 1661, 1332, 5072]
PPPoE header (6 octets) [RFC 2516 § 4]
Ethernet II frame (18 octets, EtherType 8863₁₆ or 8864₁₆) [IEEE 802.3-2012]

From here below, the ADSL side of the DG834Gv2 continues:

LLC Encapsulation for Bridged Protocols (10 octets) [RFC 2684 § 5.2]:
- IEEE 802 SNAP header (5 octets, OUI 00:80:C2₁₆, PID 0001₁₆) [IEEE 802]
- IEEE 802.2 LLC header (3 octets, AA:AA:03₁₆) [IEEE 802.2]
AAL5 CPCS-PDU trailer (8 octets) [RFC 2684 § 4, ITU-T Rec. I.363.5]
ATM cell header (5 octets)
ADSL2+ physical layer [ITU-T Rec. G.992.5]

Alternatively, the Ethernet side of the DG834Gv2 continues:

Ethernet packet (8 octets) [IEEE 802.3-2012]
Ethernet interpacket gap (12 octets) [IEEE 802.3-2012]
100BASE-TX physical layer [IEEE 802.3-2012]

IPv4 addresses are obtained with the PPP Internet Protocol Control Protocol [RFC 1332]. This includes dynamic and static addresses, but does not include routed subnets. IPv6 is where the situation becomes more confusing. The IPv6 Control Protocol [RFC 5072] is only used to negotiate a unique interface identifier for the client, because PPP is only concerned with link-local communication between a pair of peers. Even if no globally routable prefixes are assigned, once this negotiation is complete, IPv6 traffic to the ISP peer is technically possible, although not very useful of course.

After some hours hacking away because the OpenBSD 5.7 manual page for pppoe(4) was incorrect, I reached this configuration:

% cat /etc/hostname.pppoe0
!/sbin/ifconfig em0 up
inet 0.0.0.0 255.255.255.255 NONE \
	pppoedev em0 \
	authproto chap \
	authname 'azabani@internode.on.net' \
	authkey hunter2
dest 0.0.0.1
inet6 eui64
!/sbin/route add 0.0.0.0/0 -ifp pppoe0 0.0.0.1
!/sbin/route add ::/0 -ifp pppoe0 fe80::

% sh /etc/netstart pppoe0

The two errors were subtle but fatal. Placing inet6 eui64 before the PPPoE parameters inadvertently brings the interface up, after which the parameters can’t be changed. Viable solutions include placing down just after inet6 eui64, or simply moving inet6 eui64 after the PPPoE parameters. Because of a peculiarity with route(8), add default fe80:: doesn’t work either unless the -inet6 option is specified, because 0.0.0.0/0 is assumed, and incompatible address families ensue. I’ve since sent a patch to tech@openbsd.org.

From here, either or both of two paths can be taken to obtain a globally routable IPv6 prefix: NDP Router Solicitation [RFC 4861] for the dynamic /64, or DHCPv6 IA_PD [RFC 3633] for the static /56.

% ifconfig pppoe0 inet6 autoconf

One command is required to start sending Router Solicitation messages, and ifconfig(8) handles it now that rtsold(8) has been removed. Because [RFC 4861] doesn’t specify whether or not routers are allowed to send Router Solicitation messages, OpenBSD errs on the side of caution, and will not send them if net.inet6.ip6.forwarding is enabled.

The wide-dhcpv6 package provides dhcp6c(8) for DHCPv6 IA_PD, or prefix delegation. Note that even though the /56 prefix is static, the prefix delegation process must still occur so that Internode can update its routing tables, something that I spent half a day scratching my head at when I tried to cut corners.

% pkg_add wide-dhcpv6

% cat /etc/rc.d/dhcp6c
#!/bin/sh

daemon="/usr/local/sbin/dhcp6c"

. /etc/rc.d/rc.subr

rc_reload=NO

rc_cmd $1

% cat /etc/dhcp6c.conf
interface pppoe0 {
	send ia-pd 0;
	send domain-name-servers;
	send rapid-commit;
};

id-assoc pd {
	prefix-interface vether0 {
		sla-id 0;
		sla-len 8;
	};
};

% echo 'dhcp6c_flags=pppoe0' | tee -a /etc/rc.conf.local
dhcp6c_flags=pppoe0

% echo '!/etc/rc.d/dhcp6c restart' | tee -a /etc/hostname.pppoe0
!/etc/rc.d/dhcp6c restart

% /etc/rc.d/dhcp6c restart
dhcp6c(ok)

Turning a client into a router

Most of the work involved in configuring routing involves pf(4) and pf.conf(5), but I left that until last. Before that I set the DNS search domain and resolvers to sane defaults, and enabled packet forwarding for both IPv4 and IPv6.

% cat /etc/resolv.conf
search home.daz.cat
nameserver 192.231.203.132
nameserver 192.231.203.3

% cat /etc/sysctl.conf
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

% xargs sysctl < /etc/sysctl.conf
net.inet.ip.forwarding: 0 -> 1
net.inet6.ip6.forwarding: 0 -> 1

From there I moved to DHCP, something that most people are familiar with.

% cat /etc/dhcpd.conf
subnet 172.19.0.0 netmask 255.255.0.0 {
	range 172.19.2.1 172.19.2.254;
	default-lease-time 3600;
	max-lease-time 604800;
	option routers 172.19.1.1;
	option domain-name-servers 192.231.203.132, 192.231.203.3;
	option domain-name "home.daz.cat";
}

% echo 'dhcpd_flags=vether0' | tee -a /etc/rc.conf.local
dhcpd_flags=vether0

% /etc/rc.d/dhcpd restart
dhcpd(ok)

As for IPv6, rtadvd(8) goes a long way by sending Router Advertisement messages, propagating the globally routable IPv6 prefix where it’s available. I’m leaving clients to generate their own interface identifiers using SLAAC [RFC 4862, 4941] instead of running a stateful DHCPv6 [RFC 3315] server. In its stateless form, DHCPv6 is only necessary to advertise IPv6 DNS servers, which I’ll get around to doing eventually.

% echo 'rtadvd_flags=vether0' | tee -a /etc/rc.conf.local
rtadvd_flags=vether0

% /etc/rc.d/rtadvd restart
rtadvd(ok)

pf.conf(5) is where I spent a few full days of my time. I paid careful attention to RFCs when I was deciding which address blocks to drop traffic from [RFC 6890] and which ICMPv6 message types and codes to allow through the router [RFC 4890].

### ~~~ Interface layout ~~~ ###

# em0: 802.3ab to ADSL modem (172.19.0.1/16)
# em1: 802.3ab to internal switch
# ral0: 802.11g in hostap mode
# vether0: persists addresses 172.19.1.1/16 and 2001:44b8:6116:1c00::eui64/64
# bridge0: Ethernet bridge over all of the above
# pflog0: target interface for blocked packets
# pppoe0: PPPoE session over em0

### ~~~ Constants and variables ~~~ ###

# All addresses associated with this host
self = "{ (egress), (vether0) }"

# RFC 6890: Special-Purpose IP Address Registries:
# https://www.iana.org/assignments/iana-ipv4-special-registry/
# https://www.iana.org/assignments/iana-ipv6-special-registry/

# Included below are all address blocks with either Forwardable = False,
# Global = False, or both, but excluding 2001::/23 because it is often
# superseded by more specific allocations, as of 2015-08-05.

table <martians> const { \
	0.0.0.0/8, \
	10.0.0.0/8, \
	100.64.0.0/10, \
	127.0.0.0/8, \
	169.254.0.0/16, \
	172.16.0.0/12, \
	192.0.0.0/24, \
	192.0.2.0/24, \
	192.168.0.0/16, \
	198.18.0.0/15, \
	198.51.100.0/24, \
	203.0.113.0/24, \
	240.0.0.0/4, \
	255.255.255.255/32, \
	::1/128, \
	::/128, \
	::ffff:0:0/96, \
	100::/64, \
	2001::/32, \
	2001:2::/48, \
	2001:db8::/32, \
	fc00::/7, \
	fe80::/10 \
}

### ~~~ Default rules ~~~ ###

# Never touch loopback interfaces
set skip on lo

# Normalise packets, especially IPv4 DF and Identification
match in all scrub (no-df random-id)

# Limit the MSS on PPPoE to 1440 octets
match on pppoe0 scrub (max-mss 1440)

# Block all packets by default, logging them to pflog0
block log

### ~~~ Link-scoped services ~~~ ###

# DHCPv6 client: make IA_PD requests and receive responses to them
pass out quick on egress inet6 proto udp from (egress) to ff02::1:2 port dhcpv6-server
pass in quick on egress inet6 proto udp to (egress) port dhcpv6-client

### ~~~ Bulk pass rules ~~~ ###

# Pass all traffic on internal interfaces
# vether0 is necessary here, but bridge0 is not
pass quick on { vether0 em0 em1 ral0 }

# Pass all outbound IPv6 traffic
pass out quick on egress inet6 from { egress, (vether0:network) } modulate state

# Pass all outbound IPv4 traffic from this host
pass out quick on egress inet from egress modulate state

# NAT all outbound IPv4 traffic from the rest of our network
pass out quick on egress inet from (vether0:network) nat-to (egress) modulate state

### ~~~ Block undesirable traffic ~~~ ###

# These rules must not precede the DHCPv6 client or NAT rules above
block log quick on egress from { no-route, urpf-failed, <martians> }
block log quick on egress to { no-route, <martians> }

### ~~~ Pass some ICMP and ICMPv6 traffic ~~~ ####

# Pass all inbound ICMP echo requests
pass in quick on egress inet proto icmp icmp-type echoreq

# RFC 4890: Recommendations for Filtering ICMPv6 Messages in Firewalls
pass quick on egress inet6 proto icmp6 icmp6-type { 1, 2, 128, 129 }
pass quick on egress inet6 proto icmp6 icmp6-type 3 code 0
pass quick on egress inet6 proto icmp6 icmp6-type 3 code 1
pass quick on egress inet6 proto icmp6 icmp6-type 4 code 0
pass quick on egress inet6 proto icmp6 icmp6-type 4 code 1
pass quick on egress inet6 proto icmp6 icmp6-type 4 code 2

### ~~~ Open services on this router ~~~ ###

# OpenSSH server
pass in on egress proto { tcp, udp } to $self port ssh

Watch this space!

Future tasks which I’m yet to complete include:

Configuring ntpd(8)
Advertising IPv6 DNS servers with dhcp6s(8)
Hosting an authoritative DNS zone for home.daz.cat with nsd(8)
Caching DNS queries with unbound(8)

You can find the root overlay for my router on Bitbucket.

On gender identity

2015-04-06T11:00:00+00:00

Human gender and sexuality are immensely complex concepts due to our complexity as a species. Despite this, it’s quite simple to handle gender identity in an everyday setting. You can initially guess someone’s gender, because you’ll probably be correct most of the time. Otherwise, accept any corrections or simply ask them how they wish to be referred, and respect their identity. If you deliberately misgender a person because they don’t pass your personal threshold of physical characteristics to be worthy of their gender, you’re not just being rude, but you’re also espousing an excessively simplified and internally inconsistent conceptual understanding of gender.

Perhaps the most pernicious idea which comes from this is that where a transgender person is not genuinely a member of the gender with which they identify until they undergo sex reassignment surgery. It’s petty because there are virtually no valid reasons to care about someone’s genitalia outside of a sexual, reproductive, or medical context. The procedure or lack thereof should make no difference to how a person will interact with most other people. Some transgender people may not be able to undergo this surgery due to medical, financial, or other constraints, while others may not even want to do so at all. This may be for any reason, but the decision is ultimately a personal one.

This reasoning extends to other elements of gender transitioning, such as hormone replacement therapy, changes to physical or social expressions, adherence to any number of gender roles, sexual orientation or behaviour, and the use of pronouns. Each of these is an optional personal choice for the individual involved, and any or all of them may be subject to financial, social, legal, or other limitations. Using a checklist of these elements to determine whether or not a person deserves to openly identify as their gender is as absurd as using their physical characteristics to determine their gender, a social construct which is purely psychological in nature.

Let’s take a look, by example, at why refusing to refer to someone using the gender with which they identify is indicative of a flawed understanding of gender. What of a biological female who identifies as a woman, but expresses many masculine characteristics due to choice, heredity, or otherwise? What of her inverse, where a man who is born male is simply very androgynous or even feminine? What of her converse or her contrapositive, where a transgender person undergoes few or no elements of transitioning, resulting in a gender identity which remains different to their outward expressions of gender? What of an intersex, agender, or genderqueer person? How about a person who identifies with a gender which may be prevalent in their culture, but does not fall within the typical gender binary or other framework of yours?

The intersection of biological sex with our set of diverse and intricate cultures makes any perceived ability to compartmentalise people into neat little boxes impossible. Given our position as a sapient, introspective, and in some ways advanced species, a personal ideal of mine would be to see more humans and their cultures transcend the existence of the concept of gender altogether, while simultaneously relegating biological sex to cease its influence beyond its own inherent purview, that is to say, the sexual and reproductive.

Until then we’re mostly stuck with the gender categories that are prevalent in our local cultures, but nobody should have to do anything or look a certain way to be qualified to identify with their gender of choice. You may not have an interest in gender identity, and that’s fine, but if you respect someone, let alone care about them, be sure to refer to them in alignment with their wishes — it’s truly that simple.

The Curtin Courses Handbook

2014-11-14T11:00:00+00:00

Almost every university has a handbook which should serve as an authoritative source regarding which units a student needs to complete to be able to graduate. If you are a computing student at Curtin University, this is not the case.

Background

Be forewarned that this is a huge tangential rant.

2013 arrived with a significant refactoring of courses at Curtin University. Many (but not all) science, commerce and engineering courses were combined into courses representing each faculty ¹. Each of the old courses became majors in the new system, and what were previously majors are now known as ‘streams’.

Because none of the content had actually changed, only three ends were achieved by this process. Students would find it more difficult to search for a relevant degree with TISC and the structure of courses was made significantly more confusing, but at least Curtin’s perceived student counts for each ‘course’ would increase markedly.

Students enrolling in 2015 will be greeted with a mass renaming of all of their courses and units, purely because StudentOne, the software used to manage enrolments, was sold to a university in the eastern states of Australia. To reduce their software maintenance costs, the software’s new owners have mandated their own unit and course naming schema.

The three defining characteristics of Curtin’s implementation of this new system are more prominent unique ‘unit codes’ like COMP3001, the lack of numbers after unit names, and unit names which are unnecessarily and independently unique. For instance, ‘Design and Analysis of Algorithms 300’ becomes COMP3001 ‘Design and Analysis of Algorithms’.

These rules are followed religiously except when they’re not followed at all. The unimaginative bureaucrats at Curtin couldn’t reconcile the new schema with pairs of units that run across a whole year, like ‘Software Engineering Project 320/330/401/402’, which became ISAD3000/3001/4000/4001 ‘Software Engineering Project 1/2/A/B’.

On the other hand, if you’re a student learning the Chinese language, what were logically named ‘Chinese 111/112/211/212/311/312/321/322’ are now the following pointless behemoth of slightly varied unit names:

CHIN1000 (v.1) Beginning Chinese
CHIN1001 (v.1) Beginning Chinese Developing
CHIN2001 (v.1) Pre-Intermediate Chinese
CHIN2002 (v.1) Pre-Intermediate Chinese Developing
CHIN3000 (v.1) Intermediate Chinese
CHIN3001 (v.1) Intermediate Chinese Developing
CHIN3002 (v.1) Advanced Chinese
CHIN3003 (v.1) Advanced Chinese Developing

I think I’ve made my point. The powers that be often do very silly things.

What’s wrong with the handbook?

This is a whole new kettle of fish.

Luke is a student ² who wants to start studying computing in 2015. Luke needs to enrol in four distinct but overlapping entities:

Thank the Department of Computing for fighting to ensure that degrees are awarded ‘Bachelor of Science (Computing)’ because this is Kafkaesque. While the pre-major is surprisingly correct, the major and stream, which constitute the remainder of the course, are where it all goes downhill. Let’s wear the shoes of Luke’s frustration.

Luke is scheduled to enrol in COMP2003 during year 2 semester 1.

What Luke doesn’t realise is that COMP2003 only ever runs in semester 2:

For Luke’s third year, the handbook requires enrolment in COMP3004 and COMP3005:

Doing so would be impossible as the unit doesn’t exist except in Sri Lanka:

What does exist are ISAD3000 and ISAD3001, which are incorrectly listed as being specific to the Software Engineering Stream, even though everyone in the Computing Major must complete them to graduate:

If Luke were to follow the handbook and StudentOne when enrolling in units every semester, I daresay that he won’t even be cleared to graduate after three years of hard work, especially given the misinformation that he must choose between COMP3001 and a final year project, when both are actually mandatory.

Who do I trust?

In a hypothetical world where Department of Computing staff don’t work tirelessly to mitigate the enrolment problems of egregiously misled students, it’d be a wonder that students manage to even graduate with a computing degree from Curtin University.

If you want an authoritative, correct indication of your course structure, pick up a leaflet from the table outside David Cooper’s office at 314.343, just opposite the reception window for the School of Electrical Engineering and Computing.

Pedantry

The hierarchy of learning areas is faculty, then school, then department, but faculty is not quite the correct term here, because there is a Faculty of Science and Engineering and the Curtin Business School (a faculty) which awards degrees other than commerce. ‘Faculty’ is too big, while ‘school’ is too small. ↩
Luke is hypothetical of course, but I know at least five people with this name who have done computing at Curtin, so Luke is probably a good choice. ↩

First experiences with OpenBSD

2014-08-09T08:00:00+00:00

Tired of the sluggishness of Windows on my laptop and interested in experimenting with a Unix-like that I haven't tried before, I gave OpenBSD a second shot after a brief stint on my netbook a couple of years back. Coming from exclusively using Gentoo, this seemed like a natural choice due to its minimalism and the shared roots between ports and portage. Reading Matthew D. Fuller's comparison of BSD and Linux inspired me to venture beyond the familiar walls of the Linux world.

It would be prudent to start with a hint of background. I first discovered Linux via a cover disc on the APC magazine which included a live image for Mandriva Linux 2006.0. Until then, I had been using Windows XP, Windows 98 and Windows 95 in reverse order. While I dabbled with it, the concept of actually installing something that wasn't Windows on my computer never crossed my mind.

In 2009, I installed Ubuntu 9.04 cold turkey and continued using it for nearly two years, at which point I distro-hopped to Debian. For the entirety of 2012, I used Gentoo and I even had a small distcc cluster, but I eventually became annoyed at the poor quality of Linux userland software. From then until now, I had returned to Windows 7 with a sometimes painful drop of Cygwin where I needed a Unixish environment.

Creating an installation medium

I'm currently using an Asus U31SD which has no optical disc drive, calling for the use of a USB flash drive. I wanted to partition my large flash drive such that I could use it simultaneously as installation media and a backup destination. This was not trivial to configure in a way compatible with Windows.

Windows makes an arbitrary, silly distinction whereby it tries very hard to avoid partitioning 'removable disks', a category which includes USB flash drives, but not USB portable hard drives. Using diskmgmt.msc, diskpart or any stock GUI tool will create a partition table if the disk has no recognisable volume or partition table, but will only allow one partition to be created. Behold the idiocy:

DISKPART> clean

DiskPart succeeded in cleaning the disk.

DISKPART> convert mbr

DiskPart successfully converted the selected disk to MBR format.

DISKPART> create partition primary size=4096

DiskPart succeeded in creating the specified partition.

DISKPART> create partition primary size=4096

No usable free extent could be found. It may be that there is insufficient
free space to create a partition at the specified size and offset. Specify
different size and offset values or don't specify either to create the
maximum sized partition. It may be that the disk is partitioned using the MBR disk
partitioning format and the disk contains either 4 primary partitions, (no
more partitions may be created), or 3 primary partitions and one extended
partition, (only logical drives may be created).

DISKPART> clean

DiskPart succeeded in cleaning the disk.

DISKPART> convert gpt

DiskPart successfully converted the selected disk to GPT format.

DISKPART> create partition primary size=4096

DiskPart succeeded in creating the specified partition.

DISKPART> create partition primary size=4096

Virtual Disk Service error:
The operation is not supported on a non-empty removable disk.

If the disk has a direct volume with no partition table, Windows will roll with that and pretend that there is one partition. If multiple partitions are created on a disk elsewhere, its behaviour on Windows appears to be unpredictable, but it looks like a drive letter can be assigned to the first partition entry in the MBR, depending on the phase of the moon.

There is allegedly a 'removable media bit' in some drive controllers which can be flipped by a tool for Lexar drives called BootIt. Sadly it appeared to have no effect on my 64 GB Corsair drive. I tried creating many partition configurations using OpenBSD's own fdisk in a virtual machine, as well as GNU fdisk and parted on Debian. None of them played nicely with Windows. All I wanted was an NTFS or FAT32 partition plus a BSD disklabel.

What did work was dumping the contents of install55.fs or miniroot55.fs onto the entire drive, then adding another partition on entry 0 from there. These files contain not only a BSD disklabel with one root FFS, but ostensibly also an MBR surrounding that.

# ftp ftp://ftp.ii.net/pub/OpenBSD/5.5/amd64/install55.fs
# dd if=install55.fs of=/dev/rsd0c bs=65536
# echo 'e 0\n7\nn\n491520\n123043840\nw\np\nq' | fdisk -e sd0 | sed ':a;$q;N;6,$D;ba'
 0: 07     30 151  58 -   7689 183  46 [      491520:   123043840 ] NTFS
 1: 00      0   0   0 -      0   0   0 [           0:           0 ] unused
 2: 00      0   0   0 -      0   0   0 [           0:           0 ] unused
 3: A6      0   1   2 -     30 151  57 [          64:      491456 ] OpenBSD
fdisk: 1> #
# echo 'SCSI\nA\nw\np\nq' | disklabel -E sd0 | sed ':a;$q;N;5,$D;ba'
  a:           491456               64  4.2BSD   2048 16384    1 # /
  c:        123535360                0  unused
  i:        123043840           491520    NTFS
> No label changes.

It's perhaps here that I should mention that I found OpenBSD's distinction between block devices (for disks, partitions and slices) and their corresponding raw character device files quite bizarre. I suppose I just need to remember to prepend r to any administrative task pertaining to partitioning or filesystem creation.

The installation process proper

This was actually relatively straightforward, save for the wireless NIC iwn0 not working until firmware files are automatically downloaded after installation. Using the wired alc0 gigabit Ethernet interface temporarily did the trick, but networking was not even necessary at this stage, given that I had used the full install55.fs image.

iwn0: error, 2, could not read firmware iwn-6030
iwn0: could not read firmware
iwn0: no link ............. sleeping

First boot was problematic, as the kernel hung on something ACPI:

...
cpu3: smt 1, core 1, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
acpiec0 at acpi0
acpihpet0 at acpi0: 14318179 Hz
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-63

A quick and dirty solution I'm using right now is to disable it:

boot> -c
UKC> disable acpi
352 acpi0 disabled
UKC> quit

At this point I had to start over because I foolishly elected to start xdm(1) on boot, leaving me unable to use the keyboard or mouse. Choosing the right option, I am now greeted with an odd monospaced serif console. I then disabled ACPI permanently:

# cp /bsd{,.old}
# echo quit | config -efu /bsd

I missed the automatic firmware installation on first boot, so I had to invoke the process manually:

# echo dhcp > /etc/hostname.alc0
# . /etc/netstart alc0
# fw_update

Configuring a connection to my wireless network revealed the elegant nature of hostname.if(5), which strongly reminded me of the file format used with iptables-{save,restore}(8). It looks like connecting to Curtin University's network might need wpa_supplicant(8), however.

# echo 'nwid deLAN\nwpakey REDACTED\ndhcp\nrtsol' > /etc/hostname.iwn0
# . /etc/netstart iwn0

Some sysctl values prevent global IPv6 address configuration from working automatically, but after my surprise upon seeing the default values, they were quickly fixed:

# ifconfig iwn0 | grep inet6
        inet6 fe80::b6b6:76ff:fe1e:7bfc%iwn0 prefixlen 64 scopeid 0x1
# rtsol iwn0
rtsol: kernel is configured not to accept RAs
rtsol: kernel is configured not to accept redirects
# sed -E 's/^#(net.inet6.(icmp6.rediraccept|ip6.accept_rtadv)=)/\1/' /etc/sysctl.conf > /etc/sysctl.conf.
# mv /etc/sysctl.conf{.,}
# sysctl net.inet6.icmp6.rediraccept=1
# sysctl net.inet6.ip6.accept_rtadv=1
# rtsol iwn0
# ifconfig iwn0 | grep inet6
        inet6 fe80::b6b6:76ff:fe1e:7bfc%iwn0 prefixlen 64 scopeid 0x1
        inet6 2001:44b8:6116:1c00:b6b6:76ff:fe1e:7bfc prefixlen 64 autoconf pltime 3593 vltime 7194
        inet6 2001:44b8:6116:1c00:4487:8545:2941:c33c prefixlen 64 autoconf autoconfprivacy pltime 3593
vltime 7194

Getting X11 working

Upon trying to run Xorg -configure, it turns out that the problems I had been having with X11 seem to stem from the pms driver which is used for the Synaptics touchpad. For now, I'll simply disable the driver and use an external USB mouse as a pointing device.

boot> -c
UKC> disable pms
259 pms* disabled
UKC> quit
# echo quit | config -efu /bsd

Finishing user configuration

All that's left is to download my dotfiles and install some packages.

# sed -E 's/^# (%wheel.+\) SETENV.+)/\1/' /etc/sudoers > /etc/sudoers.
# mv /etc/sudoers{.,}
# chmod 440 /etc/sudoers
$ export PKG_PATH=ftp://ftp.ii.net/pub/OpenBSD/5.5/packages/amd64/
$ sudo pkg_add -v zsh vim curl git irssi gnupg ...
$ git clone --bare https://github.com/delan/config .git
$ sed -E 's/(bare = )true/\1false/' .git/config > .git/config.
$ mv .git/config{.,}
$ git checkout HEAD
$ chsh -s zsh
$ exec zsh
neptune%

Miscellaneous thoughts

I very much appreciated the clean separation between the base system, the packages and the ports tree, and grew to rely on having an eloquent manual page for every command, driver and configuration file in the base system.

I don't know how I have managed to achieve this, but I can use vim key bindings in GUI applications, including those that use GTK+. Perhaps it is influenced by running bindkey -v in my .zshrc.

OpenPorts.se was very useful for when I couldn't recall the exact name of a package I was looking for. In the long run however, it was much better to fetch and configure the ports tree locally as recommended. This also buys one the ability to cleanly configure and compile software from source, which is especially useful for dwm, for example.

Using pkg_info -t is a useful way of finding packages that one has explicitly installed so far:

neptune% pkg_info -t         
ImageMagick-6.7.7.7p6 image processing tools
chromium-32.0.1700.102-proprietary Chromium browser
dwm-6.0             dynamic window manager
git-1.8.5.3         GIT - Tree History Storage Tool
gnupg-2.0.22p0      gnu privacy guard - a free PGP replacement
irssi-0.8.15p5      modular IRC client with many features (ipv6,socks,proxy)
iwn-firmware-5.10p0 firmware binary images for iwn(4) driver
p7zip-9.20.1p0      file archiver with high compression ratio
povray-3.6.1p5      3D image rendering package
scrot-0.8p3         commandline screen capture util
texlive_texmf-full-2013 texlive texmf for extra macros
uvideo-firmware-1.2p1 firmware binary images for uvideo(4) driver
vim-7.4.135p0-no_x11 vi clone, many additional features
zip-3.0             create/update ZIP files compatible with PKZip(tm)
zsh-5.0.2           Z shell, Bourne shell-compatible

In broad strokes, the tasks I have yet to complete are:

Find out why ACPI causes the kernel to hang, and fix that;
Find out why pms(4) with my touchpad breaks X11;
Create ports for software I miss such as the Proggy fonts; and
Create ports for software that I've written, after cleaning up the code.

Writing an IRC sedbot

2014-02-08T14:30:00+00:00

The ComSSA IRC has an official utility bot, KhlavKalash, which currently does some trivia like URL title fetching and server uptime. It uses Twisted for IRC and has a good plugin system with Yapsy. Feeling a bit bored, I decided to try my hand at implementing what I call a "sedbot", which interprets messages that are sed replacement expressions, executing the replacement on the last normal message.

As I'm writing this, KhlavKalash allows you to create plugins by simply dropping Python files into a subdirectory with your own classes, each of which represents essentially a simple bot. The current interface is limited to callbacks on normal messages matching supplied regular expressions, and allows you to return zero or one response message to be sent back. This could change to become more flexible in the future.

Initially I planned to make replacements operate on the requesting user's last message, as opposed to the last message from anyone in the channel. This would currently not be possible with the KhlavKalash API because requesting nicks are not passed to the plugins. Thankfully, I discovered that most sedbots operate on the last message in general anyway, as it allows users to "correct" messages from other people, often for snarky effect.

The most difficult, or rather, verbose, part of the implementation was actually parsing the sed expressions. Not the regular expressions themselves, but simply splitting s/a/b/f into a, b and f — you can't simply split the strong on / because patterns and replacements can have slashes in them via backslash escapes. I wrote a simple finite state machine to parse the components of sed expressions, and about 100 lines later, I was in business.

There were a couple of things I couldn't implement in an obvious way. The regex module is much better than re but uses the same API. You can use the count argument to implement the global replacement flag, and the regex.IGNORECASE bit in flags to implement /i, but there isn't an easy way to choose which occurrence to start replacing from, which is provided by sed via a numeric flag, e.g. s/a/b/42.

sed allows you to specify multiple expressions by separating them with semicolons. This is unambiguous, because the last component of a sed expression are the flags, and a semicolon is not a valid flag. Some simple additions to my parser's two flags states and support for this was implemented. Unfortunately, this comes at the expense of the ability to allow the omission of the final slash, something I hadn't yet done but was requested by a few users, because a semicolon in the replacement section would be ambiguous.

I tackled Unicode support by treating incoming byte strings as UTF-8, and returning unicode strings back to the core module. An interesting thing to note is that the IRC protocol has no concept of text encoding at all; it only sees bytes, leaving it up to clients to decide how to interpret them. Ideally all clients would use UTF-8, but this is not always the case. Regardless, the bot assumes UTF-8 everywhere, in the hope that anyone using non-ASCII characters is using UTF-8, but there really is little other choice.

Formal grammars are apparently covered in Theoretical Foundations of Computer Science 300, which I'm not due to study until some time next year. Having written a couple of simple finite state machine parsers in the past as well as today, I was keen to see how it fit into the grand scheme of things.

I started looking for more information after writing the bot, and I found out that the very fact that I used a state machine indicates that the language of sed expressions (if you are not parsing the regular expressions inside) is a regular language, which is a subset of the context-free languages that can be parsed by, you guessed it, regular expressions.

Accepting the implicit challenge that the realisation yielded, I managed to create a regular expression that can be used to split a sed replacement expression into its three parts. I decided against switching to it, as the old parser is probably more readable and extensible, but it's worth sharing. Imagine these two lines as one:

^(?:s/((?:[^\\/]|\\.)*)/((?:[^\\/]|\\.)*)/((?:g|i|\d+)*))(?:;s/((?:[^\\/]|\\.)*
)/((?:[^\\/]|\\.)*)/((?:g|i|\d+)*))*$

You can use the regular expression to parse sed replacement expressions, even a semicolon delimited sequence of expressions, like the FSM parser. The regex will yield three captured groups per sed expression, one each for the needle, replacement and flags components respectively. All you need to do is collapse backslash escapes, and split flags as necessary, both of which are easy tasks. With verbose mode I can document the behemoth:

^			# start of the message
(?:			# BEGIN first sed expression
  s/			#   sed replacement expression delimiter
  (			#   BEGIN needle component
    (?:			#     BEGIN single needle character
      [^\\/]		#       anything that isn't a slash or backslash...
      |\\.		#       ...or any backslash escape
    )*			#     END single needle character, zero or more
  )			#   END needle component
  /			#   slash between needle and replacement
  (			#   BEGIN replacement component
    (?:			#     BEGIN single replacement character
      [^\\/]|\\.	#       escape or non-slash-backslash, as above
    )*			#     END single replacement character, zero or more
  )			#   END replacement component
  /			#   slash between replacement and flags
  (			#   BEGIN flags component
    (?:			#     BEGIN single flag
      g|i|\d+		#       "g", "i" or a sequence of digits
    )*			#     END single flag, zero or more
  )			#   END flags component
)			# END first sed expression
(?:			# BEGIN optional subsequent sed expressions
  ;			#   semicolon between sed expressions
  s/			#   sed replacement expression delimiter, as above
  ((?:[^\\/]|\\.)*)	#   needle component, as above
  /			#   slash between needle and replacement, as above
  ((?:[^\\/]|\\.)*)	#   replacement component, as above
  /			#   slash between replacement and flags, as above
  ((?:g|i|\d+)*)	#   flags component, as above
)*			# END optional subsequent sed expressions, zero or more
$			# end of the message

Sidestepping NXDOMAIN hijacking

2014-02-04T06:00:00+00:00

Until May last year, I was a customer of name.com for a domain name and DNS services. Having just grown out of a comfortable walled garden of all-in-one shared hosting and DNS perhaps about a year prior, I knew little other than that I should avoid GoDaddy. I since discovered that name.com was pulling a scummy trick that an increasing number of name service providers engaged in.

NXDOMAIN hijacking is not news now. Many ISPs and even OpenDNS have started doing it, and while it not only breaks the DNS specifications and poses a security risk, name.com's flippant responses and actions in holding customers liable for name.com's automatically generated content are clearly unacceptable. Specifically, the actions of name.com have been covered quite extensively.

That's not what this post is about though. You can easily avoid all of this by switching to a vendor that isn't bad, such as Namecheap, Hover or Gandi. If you stay with a provider like name.com, consider using something like Hurricane Electric's free DNS service, which supports IPv6 and includes the ability to run primary or secondary nameservers.

Time to explore some commonly suggested technical workarounds provided to users of the broken name services, for some fun and no profit. To begin, take this output of nslookup which exhibits the issue:

C:\Users\delan>nslookup notexistent.azabani.com.
Server:  resolv.on.ii.net
Address:  2001:44b8:1::1

Non-authoritative answer:
Name:    notexistent.azabani.com
Address:  174.37.172.71

A simplistic solution is to create wildcard A and AAAA records to IP addresses guaranteed to never route nor reach a compliant networked device. RFC 3513 § 2.5.2 stipulates that the address :: "must never be assigned to any node". However, as you may expect, there are still addresses returned by this method; any failure is in connection, not resolution.

*	A	0.0.0.0
*	AAAA	::

C:\Users\delan>nslookup notexistent.azabani.com.
Server:  resolv.on.ii.net
Address:  2001:44b8:1::1

Non-authoritative answer:
Name:    notexistent.azabani.com
Addresses:  ::
          0.0.0.0

Interestingly, the presence of any type of record will disable address hijacking, but the response will not actually be NXDOMAIN, but rather a NOERROR with no answer records. You can see that by creating a TXT wildcard record.

*	TXT	unused

C:\Users\delan>nslookup notexistent.azabani.com.
Server:  resolv.on.ii.net
Address:  2001:44b8:1::1

*** No internal type for both IPv4 and IPv6 Addresses (A+AAAA) records
available for notexistent

What does appear to do the trick is a wildcard CNAME to a target domain name that will never resolve. RFC 2606 § 2 and the more recent RFC 6761 § 6.4 define .invalid as a reserved TLD which is guaranteed to neither have any public domain registrations, nor ever be resolved by a compliant server.

*	CNAME	working.around.dns.hijacking.invalid

C:\Users\delan>nslookup notexistent.azabani.com.
Server:  resolv.on.ii.net
Address:  2001:44b8:1::1

*** resolv.on.ii.net can't find notexistent: Non-existent domain

Network installing Windows XP

2014-02-02T04:00:00+00:00

This was originally going to be a quick guide to Windows Deployment Services with Windows Server 2012. Using it to install Windows Vista and newer is trivial though, with the introduction of the WIM format. However, upon finding a laptop that needed Windows XP and could only externally boot via PXE, things changed a little.

It's obligatory for me to remind you that as this is published, Windows XP will die in just over two months' time, so you shouldn't use it unless absolutely necessary. Nevertheless it's not completely terrible; it even has decent IPv6 support up its sleeve, while striking the careful balance of good support for programs from the 16-bit to modern eras.

The trouble started when I acquired a Toshiba Tecra M2 with a faulty optical drive. To make things worse, it's too old to support USB boot. To really rub in the salt, it supports SD card boot, but only via floppy emulation of a file you place in the FAT filesystem root named $TOSFD00.VFD. I have no idea what they were thinking.

While from Vista onwards it's simply a case of having WDS and dropping in the WIM file, installing Windows XP is full of pitfalls that took me a great deal of poring over the Microsoft Knowledge Base and a variety of forums to decipher. Hopefully this will help someone avoid the same adventure, to put it nicely.

Essentially, the process involves creating a model installation of Windows XP, which is then captured into a WIM file by booting the capture image, then deployed in a similar way to how Windows Vista or later would be deployed.

Configuration notes

The simplest configuration is to have Windows Server 2012 on a physical machine while the model installation is generated in a virtual machine running on the server. This is the layout that will be used for this guide, but others may be useful if for instance, you don't have access to a spare computer or network.

One such alternative configuration is having both the server and the model installation as virtual machines on one host; this may require the use of a virtual network facility in your virtualisation software, so that both virtual machines are connected to one another in one isolated network.

This guide also assumes the use of VMware Workstation 10; Microsoft Virtual PC and Oracle VM VirtualBox probably work as well, but to use the latter, make sure the Extension Pack is installed or PXE booting WDS images will fail. Also, Windows Deployment Services has been available since Windows Server 2003 with SP2, so you could probably use an older version of Windows Server, although I haven't tested it.

Prerequisites

Windows Server 2012 as the WDS and DHCP/TFTP/PXE server
VMware Workstation to generate an installation of Windows XP
Windows 7 x86 media for its boot.wim
Windows XP Professional media

The boot.wim serves the dual role of being the installer for the completed XP image as well as being used to capture the model XP installation. Beware of the Windows XP media; unlike newer version of Windows, there are different media for OEM, Retail and Volume Licensing keys, and each are mutually incompatible by only accepting their specific class of keys.

Preparing the server

Estimated time: 20 minutes

Choose a wired network interface for use with the target computer, and assign it a static IP address, for example 192.168.123.1/24. This is the address that will be used for the rest of this guide. Add the DHCP Server and Windows Deployment Services roles.

Ensure that the DHCP server is running and add two scopes, one listening for the target on 192.168.123.0/24, and one listening for the VM on the subnet chosen by VMware in its default host-only network interface. Of course, leave out the first address from both ranges, as they're already taken. Leave all other settings alone, and activate the scopes.

Start the Windows Deployment Services Configuration Wizard and select Standalone server. The wizard should detect that the DHCP server has been configured and started, and should automatically check both the Do not listen and Configure DHCP checkboxes. If these are not both checked, then the DHCP server is either not configured or is not running; if so, cancel the wizard and fix that first.

Later in the same wizard, select Respond to all clients, and finish the wizard. There is a chance the wizard will say that the service didn't respond in a timely fashion. If this occurs, right click the server and start the service manually.

Finally, create an image group in the install images section, ready to later receive the captured installation image.

Adding boot images for capture and setup

Estimated time: 10 minutes

Windows XP is unlike later versions in that the installation media has no monolithic .wim file that can be extracted during setup. While RIS, essentially the predecessor to WDS, supports deploying Windows XP using the original media, WDS only supports .wim files. For Windows XP, that means creating a model installation and then capturing it back to the server as a deployable image.

Two boot images must be added: the capture image, which the VM will boot into after Windows XP is installed, and the setup image, which the target will use to install Windows XP.

Adding the setup image is simple: mount the Windows 7 media and add \sources\boot.wim. Once this is done, generate the capture image from the setup image by right clicking it and clicking Create Capture Image.

When prompted for a location, save the image anywhere you want, even %TEMP%, as you'll have an opportunity to use the same wizard to import the image once it is completed. If you choose to cleverly save the image in the actual repository location, and then tell the wizard to import the image, you will end up with two copies of the same capture image in WDS.

Creating a model installation in VMware

Estimated time: 60 minutes

Create a virtual machine with typical settings, but be sure to choose "install OS later", or VMware will try to do an "easy install" that automates the installation of tools and drivers. While this isn't fatal, it is simply unnecessary as the target computer isn't a VM.

Open the virtual machine's settings after creation and increase the RAM to 1024 MB; without doing so, booting the WDS capture image may fail. Change the network adapter type to Host-only and attach the Windows XP media to the virtual optical drive. Disconnect the virtual network adapter "at power on" to save a significant amount of time during the "Installing Network" stage of installation.

When indicating that the guest is Windows XP, VMware will default to virtualising an AMD 10/100 NIC, which is not supported by the capture image. You must change it to an Intel PRO/1000 by editing the VM's .vmx file and adding this line:

ethernet0.virtualDev = "e1000"

Launch the VM, enter the BIOS setup and change the boot order to:

LAN > CD > Removable Devices > HDD

Now install Windows XP. Press F5 when the F6 prompt appears (that is not a mistake), and choose the computer type that is compatible with your target. The VMware virtual machine defaults to ACPI Uniprocessor PC, which works for many computers, but you may need to choose an older type, such as ACPI PC for the Tecra, or even Standard PC, if your target is that old.

Although the Microsoft documentation states that the Standard PC HAL is compatible with all six computer types, you cannot easily take advantage of this to create a "universal" image because WDS does very picky filtering of the available images on the target based on the HAL, only allowing an exact match, but not other compatible matches.

Leave all of the regional and time settings at their defaults as they will be discarded anyway later. If you are deploying an OEM installation to the target, you can safely skip entering the product key, as you'll have to enter it again further in the process regardless.

During the OOBE wizard, set your username to SYSTEM. This will ensure that no user account is created, and you can create one for yourself after deployment. Unfortunately, this trick doesn't make the OOBE wizard rerun after deployment. What will happen is that you will be logged in as Administrator because SYSTEM isn't for interactive use. You can't, however, enter Administrator itself in the wizard, as that is explicitly rejected.

It might appear that you can skip the OOBE and have it come back after deployment by cancelling the screen resolution change dialog, but whlie that will skip the OOBE (and leave you with a paltry 640x480 resolution) the wizard will never return. In essence, this is thus equivalent to the SYSTEM trick, but without the resolution boost.

Preparing the completed installation for capture

Estimated time: 40 minutes

Inside the virtual machine now that Windows XP is installed, extract the contents of DEPLOY.CAB from the Windows XP media to C:\sysprep. You must extract the files to this precise location. If you extract elsewhere, Sysprep will ignore the changes you make to sysprep.inf, generate a default configuration in the correct location, and the target will throw a STOP 0x0000007B during deployment.

Note that if you are capturing and deploying to the same physical target, and you are not using a virtual machine for the model installation, you should skip to the part where you run sysprep.exe.

Copy the entire \i386\ folder (including the folder itself, not just the contents) from the Windows XP media to C:\sysprep. This ensures that all drivers and additional files are available to the target as necessary during deployment.

Run setupmgr.exe and create a sysprep.inf with all settings left at their defaults. Now add the following lines to the [Unattended] section of sysprep.inf:

    DriverSigningPolicy=Ignore
    UpdateInstalledDrivers=Yes

Add the following to the end of sysprep.inf:

[Sysprep]
    BuildMassStorageSection=Yes

[SysprepMassStorage]

Ensure that you leave at least one line break after [SysprepMassStorage].

Now run sysprep.exe, selecting Use Mini-Setup and Detect non-PnP hardware. Click Reseal then OK. This process may take about half an hour, so grab a coffee while you wait.

Capturing the model installation

Estimated time: 30 minutes

Your virtual machine should be switched off now, as Sysprep shuts down after it has finished working. Configure the virtual network adapter to connect at power on, and boot the VM over the network.

Press F12 to launch the WDS boot menu when prompted, and select the capture image. Follow the wizard's instructions, which should be fairly straightforward. When the capture completes, the image should be available on the WDS repertoire, but you will have to close and reopen the WDS management console if you wish to see it there.

The capturing process can take quite a while, so another coffee break opportunity is available.

Deploying the Windows XP image

Estimated time: 20 minutes

Connect the target computer to the interface you chose earlier. Some old computers will not be able to negotiate a crossover connection using a normal cable; for them you will have to use an actual crossover cable.

Boot the target over the network, pressing F12 when necessary and this time choosing Microsoft Windows Setup (x86) from the boot image menu. The first part of the deployment is much like a typical Windows 7 installation. When asked to authenticate to the WDS server, use the server's computer name as the domain.

If you do not see the captured installation image in the list, then the computer type selected at the beginning of the VM installation does not match the one for the target. In this case you will need to start again from Creating a model installation in VMware, choosing a different computer type.

If the target enters a reboot loop with the 0x0000007B BSOD, then the mass storage drivers were not properly included in the Sysprep process, either because you didn't modify sysprep.inf correctly, or because you extracted Sysprep to some location other than C:\sysprep.

After installation

Once Windows XP is deployed and running on the target, you will be logged in as the Administrator by default, with no password unless you set one during the setup. You should probably set a password and then create yourself a regular administrator account, for a marginal increase in security.

After deployment, you are probably going to install some drivers for the target computer. If you want, you can even sysprep again and capture the installation from the target, just like you did from the virtual machine. This will result in an image including drivers for the target computer.

Forwarding Curtin email

2014-01-23T02:00:00+00:00

In an effort to avoid using any email UI other than the familiar embrace of Gmail via Google Apps, one of the first things I did when I became a Curtin student was to forward my email. I couldn't quite filter all of the incoming emails properly though, until last night I had a sudden realisation.

Most emails to my Curtin account look something like this:

Delivered-To: delan@azabani.com
Return-Path: <delan.azabani@student.curtin.edu.au>
Resent-From: <delan.azabani@student.curtin.edu.au>
To: <delan.azabani@student.curtin.edu.au>

The obvious approach was to configure the Outlook Web App to forward email to delan@azabani.com, and that seemed to work reasonably. On the Gmail side, I simply created a filter that took all emails matching to:delan.azabani@student.curtin.edu.au and added the curtin label to them.

Except that didn't work for the significant minority of messages where all of the recipients were blind carbon copied, myself included. Why not filter on the Return-Path or Resent-From headers? Gmail doesn't allow users to search arbitrary headers, only those that are whitelisted. This includes From, To, Subject, Cc, Bcc and Message-ID.

Gmail also allows you to search the Delivered-To headers of messages, and this seemed useless to me because the value would always be delan@azabani.com. There's where the realisation hit me: combined with Gmail's automatic "plus" aliasing, the Delivered-To header is in fact the perfect solution.

Now that I forward Curtin messages to delan+curtin@azabani.com and filter on deliveredto:delan+curtin@azabani.com, no more emails need to be manually labelled. In retrospect, that it took me over a year as a Curtin student to forward my mail properly is somewhat disappointing.

Network protocol checking tools

2014-01-09T12:00:00+00:00

I've seen several quick, web-based configuration testing tools for clients and servers of various protocols, so it could potentially be helpful to someone if I compiled a list of them here. These tools test not only that your servers are working, but that they are configured in accordance with best practices, and I use them all heavily. Let's start with TLS, DNS and SMTP, though I'll try to find more in future and append them here.

DNS — authority

DNS — resolver

SMTP

TLS

NeverSSL by Colm MacCárthaigh
~~https~~://badssl.com by April King and Lucas Garron

TLS — server

SSL Server Test by Qualys
SSL Check by GlobalSign

TLS — client

How's My SSL? by Jeff Hodges
SSL Client Test by Qualys

XMPP

IM Observatory by Thijs Alkemade

CAA

CAA Record Helper by Opsmate

Forcing a single timezone in Jekyll

2014-01-02T23:00:00+00:00

Alternatively, How I Learned to Stop Worrying and Love UTC. It was a little more subtle than I had expected to ensure that Jekyll uses an arbitrary timezone for dates in posts and their generated URLs, but is that even a Good Thing™ to do?

My posts were timestamped in their YAML front matter like so:

date: 2014-01-03 07:00:00 +0800

Then I tried adding this to _config.yml:

timezone: Australia/Perth

It appeared that this had no effect at all. URLs were still being generated based on the date in UTC, and displayed post dates were still:

2014-01-02 23:00 UTC

Technically the same date, but not quite what I wanted. After emailing GitHub and receiving very quick and friendly responses, it turns out that both the timezone offset and the seconds must be omitted:

date: 2014-01-03 07:00

However, in the time that passed, I realised that doing this would actually be short-sighted; it would lock this website into one timezone. Not permanently, perhaps, but one would have to use the same timezone for all posts.

What if, one day, I decide that Perth no longer has the same homely charm, and I grow tired of living in a city where absolutely nothing happens? Heresy, I know, but it's possible.

As an aside, when doing the above, the displayed post dates would be like:

2014-01-03 07:00 Aus

Perhaps using %Z in Jekyll simply takes the first three characters of the tzdata name.

Just stick with UTC.

Hacking the Casio fx-82MS

2014-01-02T14:00:00+00:00

The fx-82MS is a cheap two-line scientific calculator that I fondly remember using for countless hours since the beginning of high school. It's so reliable that after using the same one for over six years, the worst that's happened is a bit of corrosion on the rear screws. Here are a couple of useful tricks I've found after digging through forums and videos.

Upgrade your calculator to an fx-570MS

Casio's two-line scientific calculators all appear to use the same circuit board and firmware. A combination of several soldered contacts determine which features of the firmware to enable by default. A pencil can be used to permanently change the effective model on older hardware revisions, but this has since been prevented with a hard black obstacle injected onto that area of the board.

You can also 'upgrade' your calculator to an fx-570MS, which is one of the most feature-complete and expensive Casio scientific calculators by exploiting an overflow bug, which lasts until the next reboot.

Enter standard deviation mode, by pressing [MODE] [2].
Press [0], then press [M+] until the words "Data full" appear on the screen.
Press [M+] [2] [UP].
Press [1] [3] repeatedly until no more can be entered.
Press [=]. You will see the words "Data full".
Press [=] [0] [1] [AC].

With a few caveats, the available modes should be expanded to:

COMP (general computation)
CMPLX (complex numbers)
SD (standard deviation)
REG (regressions)
BASE (base conversions)
EQN (solving equations)
MAT (matrices)
VCT (vectors)

Of particular note is an an image of the fx-570MS, which will become very useful for you when you need to find a key, because although the key layout will be the same as on your calculator, the labels will not be.

Diagnostic routine

This is a capability built into the firmware, presumably to allow easy, initial in-factory testing of the screen and keypad. At any time during the routine, pressing [ON] will reboot the calculator, ending the test.

While holding [SHIFT] and [7] down, press [ON]. The display will turn on all of its elements.
Press [SHIFT]. The display will clear completely.
Press [SHIFT]. The display will turn on half of its elements in a pattern.
Press [SHIFT]. The display will turn on the other half of its elements.
Press [SHIFT]. On the first line of the display, you will see "25" on the left, and "4" on the right. On the second line of the display, you will see all zeroes, including in the exponent.
Press [SHIFT] nine times. The display will cycle its second line through all ones, then all twos, and so on, until the second line consists of all nines.
Press [SHIFT]. The display will clear and the second line will be set to "0". This commences the keypad testing routine.
Press the keys of the keypad in the order,
[SHIFT] [ALPHA] [MODE] [LEFT] [UP] [RIGHT] [x^-1] [nCr] [Pol(] [x³] [DOWN]
then the rest of the keys, as you would read them left to right, until [=]. On the first line of the display, you will see 25 on the left, and OK on the right. On the second line of the display, you will see 4.
Press [ON] to reboot the calculator and exit the test.

Web workers in a single script file

2014-01-02T12:00:00+00:00

Web workers are an immensely useful tool, enabling concurrency and perhaps parallelism with JavaScript. Before their introduction, any non-trivial computation would completely lock up the user interface of the current document, or even the entire browser.

One could work around this by breaking up a large job into very small parts and chain them with timers, but this is highly inefficient and choosing an ideal amount of work for each part across all clients is almost impossible.

However, the interface for creating workers is tricky; the Worker constructor takes an external script's filename to execute. Suppose you are developing a JavaScript library that relies upon workers heavily. Are you forced to have two separate script files? No!

Interestingly, it is possible to have a single JavaScript file act as both a parent and a worker, but the implementation is not completely intuitive.

First of all, the single-file library must be aware of whether it is the parent or a worker. This can be achieved by detecting the presence of the DOM document:

var is_worker = !this.document;

Of course, if you need to execute the above in a context where this isn't the global object, then you can reliably obtain the global with something like

var global = (function(){ return this; })();

Surely, spawning a worker from the parent is now as simple as this, right?

var worker = new Worker('mylibrary.js');

Not quite. Worker paths are not resolved relative to the parent script file's path, but instead, relative to the path of the parent page. If the script isn't in the same directory as the page, the above will fail. Also, if the user has renamed the library file, workers will break.

The path of the script relative to the page must be used instead. This can be obtained by appending a dummy element with document.write and getting the previous element's src. The previous element, of course, is the script tag of the parent script, as conveniently for this situation, scripts block the building of the DOM tree while running.

var script_path = is_worker ? null : (function() {
	var id = +new Date + Math.random();
	document.write('<script id="dummy' + id + '"><\/script>');
	return document.getElementById('dummy' + id).
		previousSibling.src;
})();

The test for is_worker is present because DOM manipulation can only be done when the script isn't running as a worker. Thankfully, only the parent script needs to know the path to start workers, unless you want to start subworkers, but the method to do that already resolves paths relative to the worker's location.

A <script> tag is used as the dummy element because it's guaranteed to be a valid child element, be it <head> or <body>, unlike context-sensitive elements such as <div>. It's also not an element that might have side-effects on page display depending on CSS.

Now, workers can be spawned in the script with

var worker = new Worker(script_path);

The library in a single file might look something like this:

(function(global) {
	var is_worker = !this.document;
	var script_path = is_worker ? null : (function() {
		var id = +new Date + Math.random();
		document.write('<script id="dummy' + id + '"><\/script>');
		return document.getElementById('dummy' + id).
			previousSibling.src;
	})();
	function msg_from_parent(e) {
		// event handler for parent -> worker messages
	}
	function msg_from_worker(e) {
		// event handler for worker -> parent messages
	}
	function new_worker() {
		var w = new Worker(script_path);
		w.addEventListener('message', msg_from_worker, false);
		return w;
	}
	if (is_worker)
		global.addEventListener('message', msg_from_parent, false);
	// the rest of the library goes here
	// to spawn a worker, use new_worker()
})(this);

/diːˈlaːn(.)ˌʕaːzəbaːniː/

Web engine CI on a shoestring budget

Generative AI in Servo

Potential exceptions

Your feedback

On governance

Meet the CSS highlight pseudos

Contents

What are they?

Can I use them?

How do I use them?

How do they work?

Gotchas

Removing decorations and shadows

Accessing global constants

Custom properties

Spec issues

What now?

Chromium spelling and grammar, part 2

Contents

Implementation status

Charlie’s bird spec lawyerings

Squiggly lines

Platform “conventions”

Precise wavy decorations

Phase-locked decorations

Bézier bounding box

Cover me!

Highlight inheritance

Blink style 101

Blink style 102: pseudo-elements

Status quo

Storing highlight styles

Single-pass resolution

Multi-pass resolution

Pathology in legacy

Paired cascade

Who’s got green?

What now?

Group sharing on a FreeBSD home server

Unix permissions 101

Ownership and creation

Shared storage

Login classes

Fixing sudo(8)

Fixing services

Fixing samba(8)

Funny execute bits

What did we learn?

Chromium spelling and grammar features

Contents

Current status

CJK CSS unification

Fifteen years in the making

Highlight painting

Shadows and backgrounds

Shadow clipping

Vertical vertigo

Cursed

-AAAAAAAAAAAAA

Processing model

Stay tuned!

nonymous + bore(1): DNS toys for Rust

Contents

What is DNS anyway?

Naïve decoders

Zero-copy views

RDATA views

Early builders

State machines

Open questions

Next steps

Introducing xd(1)

Learning how to ship

Output version control for static sites

The bug

The system

Limitations

B-side: incremental builds

Closing thoughts