Chameth.com - posts like apple-google-aligned-incentives, travel-toolkit, understanding-docker-volume-mounts, why-you-should-be-using-https

The Case of the Unchanging Config

2026-04-09T00:00:00Z

Last week I was attempting to make it so I could share pictures on IRC directly from my client. This sounds simple, but it involves a bouncer¹ that proxies the request to a standalone image hosting service that I had to modify to be compatible. At one point my testing loop was:

Make a change to the hosting service
Commit it
Tag a new release
Wait for it to build
Update the version I’m running on my server
Reconfigure the bouncer to pass the new parameter or change the URL or whatever
Try uploading a photo from my phone
Realise I’ve overlooked something and go back to step one

I could have set up a local copy of everything and tested it sensibly. I should have set up a local copy of everything and tested it sensibly. But it seemed like such a trivial change, and setting up the whole environment seemed like such a pain. After the third or so iteration of failure I was pretty annoyed with myself, computers, and basically everything.

My biggest annoyance was that my bouncer would not pick up the new URL from the config when I changed it. It’s meant to reload the config when it receives a SIGHUP, and it claimed to in the logs, but I could clearly see it was still hitting the old URL. Restarting the bouncer to update the config is a pain, as it disconnects me from all the IRC networks, and has to reconnect to them all, reauthenticate, etc. It also mildly spams everyone who shares a channel with me².

When I finally got everything working I had a look at the bouncer source, and thought I’d spotted the issue. I raised a bug report, ending in this remark:

It looks like the config is reloaded properly, but the handler for uploads is created once at startup and has its own copy of the uploader, so effectively snapshots the config to whatever it is at startup:

fileUploadHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    cfg := srv.Config()
    h := fileupload.Handler{
        Uploader:    cfg.FileUploader,
        DB:          db,
        Auth:        cfg.Auth,
        HTTPOrigins: cfg.HTTPOrigins,
    }
    h.ServeHTTP(w, r)
})

I’m not quite sure what I thought I saw there, but that’s not right. I want to call it a hallucination, but that feels like a very overloaded word these days. I guess I was just seeing what I wanted to see, and that was a good excuse to stop investigating. Still, even if I misidentified the cause, the bug was still present, right? … Right?

The Science of Deduction

When the developer got back and said he couldn’t reproduce my issue, I went back to my install and immediately reproduced it. With a calmer head, I figured it was probably something with my particular set up. First thing to check: am I actually running the version I think I am?

Unfortunately the bouncer doesn’t actually seem to expose the version anywhere that I can see. It’s not in the logs, it doesn’t have a -version flag, and none of the IRC-based status commands seem to include it. But I know it’s a Go app, and I know Go embeds the version information. go version -m <binary> will dump it all out, but the binary is inside a Docker image, and the Docker image is one of my nice, minimal, artisanal ones so doesn’t ship a go binary. No problem, docker compose cp bouncer:/bnc ./bnc yoinks the binary out of the container, and then dumping the version shows that, yes, I am running the version I thought I was. Hmm.

The next tool I reached for in my toolbox was strace. Maybe it’s not actually reading the file for some reason? I immediately executed strace -p <pid> -e trace=openat,open,read,pread64 -f without having to look any part of that up. Yep. Definitely. Then I edited the config, HUP’d the bouncer, and saw that it was… reading the config file. As it’s meant to. By default strace truncates strings to 32 bytes, so I couldn’t actually see the line I’d changed. Some more definitely-not-RTFMing later, and rerunning it with an extra -s 65536 let me see the full config. Surprise! The config hadn’t changed!

To confirm my findings, I used docker compose cp again, this time yoinking the config file from inside the container. The inside config file was definitely different to the outside config file. What? My hypothesis at this point was “something something Docker nonsense”. I mount the config as read-only, and was wondering if that meant that Docker was doing something weird instead of just bind mounting it. A quick trip to /proc/<pid>/mounts showed that it was, in fact, not doing anything weird, and was just bind mounting it.

If the file is bind mounted, then surely it’s the same file? I ran stat on the file on the host, noted the inode number, then pondered how to actually run stat on the file inside the container, given the aforementioned awkwardly minimal image. The solution was easy: access it via /proc/<pid>/root/. I could’ve saved myself a bunch of docker compose cp if I’d thought about that earlier. Oh well. The inode of that file was different. What?

The answer was DNS. It’s always DNS. Oh, sorry, force of habit. I meant the answer was vim. I was editing the config in vim, and when it saves files, by default, it writes the new content to a temporary file and does an atomic rename. That’s normally a good thing: it prevents corruption if the write fails midway through³. In this case, though, that means the new file has a new inode. Bind mounting a file binds to the inode, so the container just keeps clutching onto the original config from when it was started, blissfully unaware that the party has relocated down the street.

This problem is likely to happen whenever you bind mount a file into a container. When I mentioned this to a friend, he immediately responded “oh yeah, never do that”, and went on to describe the horrible hacks he’s had to add to Ansible to sidestep the issue. The nicer solution is to just bind mount an entire directory if you can, as then it doesn’t matter what happens to the files within it. I really like having the config files sat alongside the Docker compose files, though; having to create a directory just to work around some bind mount weirdness upsets me.

Now I knew what the problem was, I found there was an issue raised against Docker twelve years ago⁴. The first response was “that’s expected”, and I fully agree with the author: “respectfully, that might be expected by you, but it was not expected by me”!⁵ As a result of the issue they added a nice note to the docs:

Note: Many tools used to edit files including vi and sed --in-place may result in an inode change. Since Docker v1.1.0, this will produce an error such as “sed: cannot rename ./sedKdJ9Dy: Device or resource busy”. In the case where you want to edit the mounted file, it is often easiest to instead mount the parent directory.

But in the intervening twelve years, both the note and the functionality described have gone missing. Ho-hum.

I found it interesting how I’ve only just hit this problem, given how long I’ve used Docker. But I realised that almost everything I run I’m happy to just restart. Cattle, not pets, and so on. My IRC bouncer is one of the few exceptions to that. The only other thing I regularly hot reloaded was Centauri, my reverse proxy, but that had a whole config directory mounted because it was shared between containers, so nicely sidestepped the foot-gun.

So lesson learnt: check for weird bind mount issues before raising issues about config hot reloading. It’ll join the esteemed company of “maybe the drive is full and causing completely unrelated problems?”, “perhaps everything is dog slow because the kernel ran out of entropy?”, and “did systemd sneakily take over that functionality while you weren’t looking?” in the troubleshooting checklist.

An IRC bouncer is basically an always-on proxy. It connects to the IRC networks for you, then your clients connect to your bouncer. The bouncer can then send incoming messages to all your different clients, cache them when you’re offline and replay them later, and lots of other nice things people take for granted in their chat apps these days. ↩︎
Join/part/quit spam is part of IRC, and clients have ways of handling it, but I still feel bad about doing it excessively. ↩︎
You can disable this in vim by setting backupcopy=yes. Good luck redoing all the debugging if you ever accidentally remove that from your vimrc, though! ↩︎
Which is weird, because Docker can’t possibly be that old. That would make me much older than I’m prepared to accept. ↩︎
If you read the thread they didn’t actually mean to sound so dismissive, but it’s still pretty funny. “Yes, it’s expected that the foot-gun causes your foot to hurt. Duh. What did you expect?” ↩︎

How tech companies failed to build the Star Trek computer

2025-07-16T00:00:00Z

A computer core room on the Enterprise-D

In most Star Trek series, the ship or station computer is ever-present in the background, waiting to be called on by the main characters¹. It nearly always does exactly the right thing, and there’s little limit to the functions it can perform. Take this mundane example from DS9:

KIRA: Computer, establish link with the Bajoran Medical Index for the Northwestern District.
COMPUTER: Link established.
KIRA: Access all information on Doctor Surmak Ren.
COMPUTER: There are no records matching that name.
KIRA: Try the Northeastern District, same search.
COMPUTER: Doctor Surmak Ren, currently serving as Chief Administrator of the Ilvian Medical Complex.
KIRA: Computer, open a channel to the Ilvian Medical Complex. Administrator’s office.

The computer is doing some kind of networking to a database only identified by name. It does a search and summarises the lack of results. It then repeats the process with another database, and succinctly announces the results. Finally, it opens a communication channel to a specific room in a facility, based only on its name.

This whole interaction is remarkably boring². Kira doesn’t have to know any URLs or API endpoints, or what protocol she wants to use. She doesn’t have to open a specific app and then login and then try the query again. She just says what she wants and the computer does it.

It seems like this should be one of the most easily obtainable bits of sci-fi wizardry with our current technology. We have multiple massive companies throwing lots of money at digital assistants, LLMs that are improving at an insane rate, but we’re somehow not even close to the usability or usefulness of the Trek computers. What gives?

Boring is, well, boring.

Larry Page once said something that might help explain it:

The Star Trek computer doesn’t seem that interesting. They ask it random questions, it thinks for a while. I think we can do better than that.

This is the same Larry Page that founded Google, whose mission statement is “to organize the world’s information and make it universally accessible and useful”. Of all people, surely he should find an omnipresent computer that can answer ‘random questions’ interesting?! It seems like it should be the epitome of Google’s mission!

Google’s “better than that” seems to have been to stuff LLMs into every product they can, even when you don’t want them there. Even when they’re worse than the normal content they displace. These things look exciting when they’re part of a scripted demo at Google I/O, but they fall flat and just get in the way when they’re exposed to the reality of day-to-day use.

The Star Trek computer is the opposite: it isn’t snazzy, but it is genuinely useful. That means it’s not an attractive target for the company execs who want marketing opportunities, and it’s not appealing for engineers who need to demonstrate “impact”. But even if Google did try to make the Trek computer, there are other problems…

Assistants need to be free

A significant amount of tech companies’ business models currently revolves around trapping users in walled gardens. They want you using their ecosystem; that way they get more data from you, and you’re more likely to spend more money on their other offerings that work together. There’s barely any incentive to allow any kind of interoperability with other platforms outside carefully contracted integrations.

I remember trying to help a family member move their photos from iCloud to Google Photos. At one point they turned around and said, exasperated, “why is this so hard? Aren’t they both in the cloud?!”. It’s easy to dismiss that as someone who hasn’t quite grasped the fundamental idea that “the cloud” is just someone else’s computers, but that’s not the whole story. There’s no reason why there shouldn’t be a quick and easy transfer: both services already allow uploading and downloading, there’s just no incentive for the companies involved to make it so³.

These kinds of misaligned incentives and walled garden business models cause even more problems when it comes to digital assistants. Siri is basically never going to be able to interact with, say, your Google Drive; ~~Bard~~ Gemini is never going to be able to send a message via iMessage. Even when there are appropriately blessed interactions, they’re so clunky. Can you imagine Captain Picard saying “Computer, ask the turbolift skill to take me to deck 5”?

Someone else’s computer

Software issues aside, there’s still a key difference between the Star Trek computers and our current batch of digital assistants: where they run. The Trek computers are all housed within the ship or station they serve; they can connect elsewhere to gather information, but they run entirely independently. If they go wrong, a local engineer can go in and fix things. While some of our assistants may have physical hardware in your home, they don’t work without a vast cloud apparatus behind them. If your Internet connection fails, they become paperweights. If the company running them decide to remove some functionality you depend on, you have no recourse.

That kind of helplessness isn’t limited to assistants, either. There’s a rapidly growing trend of being unable to modify or repair hardware you fully own and control. Part of this is just that they’re becoming more complex: it’s a lot harder to replace a microchip than a gear, but companies are also going out of their way to make it more difficult for users through draconian DRM regimes⁴ and aggressive intellectual property enforcement. If the US Navy can’t repair their own equipment because a corporation says so, what hope do consumers have?

We’re approaching a point where you don’t actually own anything. Software is cloud and subscription based, hardware is unrepairable. Even cars can be remotely updated and have features added or removed. The Federation wouldn’t allow a third party control over their ships⁵, so why are we so happy to put up with it in everything we consume?

A small ray of hope?

The most promising way of tackling all of these problems is through legislation. The EU’s Digital Market Act is an attempt to force ‘gatekeepers’ like Google, Apple and Meta, to allow third-party access to their services. It seems like a pretty reasonable approach, but the tech companies are unsurprisingly resisting it. Apple in particular have gone out of their way to refuse to comply, and when forced to do so have limited the functionality to people in Europe. Still, the DMA is a promising start, and if similar legislation is introduced (and robustly enforced) elsewhere it might start forcing companies to behave a bit better.

There are also smaller companies that actually do the right thing. Framework make laptops that are user-serviceable; Fairphone do the same for mobile phones. Smaller software companies provide useful, open APIs. The average person on the street will probably have never heard of these, unfortunately, but they do still exist. Maybe as the bigger tech companies tighten the screws more, people will turn to alternatives like this? Or maybe we’ll just keep accepting that our computers work for everyone but us?

Unless, of course, the computer is playing the role of the episode’s MacGuffin and has contracted space-computer-COVID or something, then it’s a lot less in-the-background. ↩︎
It’s almost like it only exists to move the plot along. ↩︎
You can generally export your data, thanks to a combination of legislation and efforts like Google’s “Data Liberation Front”, but I’ve never seen an export format that could then just be imported into an equivalent commercial product. ↩︎
Oh, you’ve changed the screen on your iPhone? Better hope it can do the secret handshake with the Apple hardware. ↩︎
I think there might actually have been an episode where that did in fact happen. We’ll just ignore that as a plot contrivance. ↩︎

Building a travel toolkit

2025-01-02T00:00:00Z

The container for my toolkit, a UGREEN “travel accessories” bag

For a while now, whenever I’ve traveled anywhere I’ve dumped a selection of tools into my backpack just in case I need them. Originally this was mostly focused around being able to open and fix my laptop if anything went wrong, but over time has slowly expanded to include other useful tools.

Repeatedly packing these tools and then returning them to where they belonged was tedious, though. I couldn’t just leave them in the backpack as I often needed them for non-travelling purposes, and I didn’t want to just buy duplicate tools that spent most of their life a few meters away from each other, so I ignored the problem and just dealt with the tedium.

Eventually I acquired enough tools that — while not quite duplicates — I wouldn’t miss often if they were kept in the backpack. Building on these I’ve created a little travel toolkit that contains everything I might want (and more) while away.

The base

The biggest challenge I had before was finding a screwdriver set. I have an iFixit Mako set that did the in-and-out-of-the-bag routine for several years. Nothing else covered the range of drivers I wanted for generic electronic tinkering and more chunky work. I finally found one in the form of a Wera Tool-Check PLUS TR-Edition. “TR” stands for “Tool Rebel” and is Wera’s “fan club” brand. Its signature features over the normal Tool-Check PLUS are a fancy fabric case and a 152mm bit extender; but it also comes with a much different set of bits, which is why I picked it up.

Here’s a comparison

Type	Standard version	Tool Rebel edition
Phillips	2x PH1 3x PH2 1x PH3	1x PH00 1x PH0 1x PH1 1x PH2 1x PH3
Pozidrive	None	1x PZ1 3x PZ2
Slotted	1x 5.5mm	1x 4mm 1x 4.5mm 2x 5.5mm 1x 6.5mm
Torx	1x T10 1x T15 1x T20 1x T25 1x T30 1x T40	1x T10 1x T15 1x T20 1x T25 1x T30 1x T40
Security Torx	1x T10 1x T15 1x T20 1x T25 1x T30	1x T25
Hex	1x H3 1x H4 1x H5 1x H6 1x H8	1x H1.5 1x H2 1x H2.5 1x H3 1x H4 1x H6 1x H8

There are only five bits present in the standard version that are missing in the TR edition: 5mm hex (pretty uncommon) and four sizes of security torx (rare). In exchange for those and the removal of a bunch of duplicates, you get two extra phillips sizes (0 and 00, both very useful for electronics and spectacles), some pozidrive bits (extremely common on random hardware/furniture/etc), smaller hex bits (for electronics, 3D printers, and so on), and a few other bits and pieces.

Overall it’s a vastly superior selection for my needs. And while it doesn’t have the full range of bits provided by the iFixit kit, it does cover almost everything I’m ever actually likely to need while travelling, and it’s probably around a third of the size.

The other core part of the toolkit is the bag. I had a UGREEN “travel accessory” bag hanging around unused, and noticed it was almost the exact right size for the Tool-Check. Now, how many other tools could I stuff in there?

All the tools

The array of tools that make up the travel toolkit

The kit can be broken down into a few sections:

Screwdriving

The aforementioned Wera Tool-Check
Its 152mm bit extender

I’ve talked enough about this already, I think, so I’ll spare you any additional commentary on screwdrivers and bits.

Splitting, gripping and cutting

Engineer tweezers
Engineer mini snips
Engineer mini screw removal pliers
Knipex mini waterpump pliers
Two iFixit spudgers (one metal, one plastic)
iFixit jimmy

I have bigger or slightly different versions of these that stay in my normal toolkit. The tweezers and snips are useful in a variety of situations, and the spudgers are handy for getting inside laptops and other devices. The pliers are less useful, but they’re one of those things where when you do need them there aren’t any good alternatives.

Sewing

Two needles
Some black thread
Set of mini clips

You never know when you’re going to rip a seam in something. I find the clips a lot less fiddly than using pins for keeping material in place when I’m trying to sew.

Computering

Retractable ethernet cable
RJ45 coupler
USB-C to RJ45 adapter
Dual USB-A and USB-C thumb drive, with an Arch ISO on it

Sometimes Wi-Fi just doesn’t cut it, or you need to fiddle with a router that doesn’t expose its admin UI over Wi-Fi. The USB-C adapter is for use with my phone, as it’s often easier to move that get the laptop where it needs to be.

I can boot from the thumb drive into an Arch environment, and from there reinstall my laptop from scratch if needed. There’s enough space left over so it can be used to transfer files as well.

Miscellaneousing

Fabric tape measure
Spirit level
Sharpie
Superglue
Spare batteries: 2 CR2032s, 2 AA, 2 AAA

I’m almost certain I won’t use the spirit level, but it’s tiny and I had a spare so it went in. Everything else is just in the “generally handy to have” category.

Putting it all together

The nicely-organised side of the toolkit

The Tool-Check PLUS fits almost perfectly into one side of the UGREEN bag, leaving a little wiggle room along the side for the longer parts like spudgers. The middle pockets can’t fit much in, but keep the batteries and sewing bits out of the way. Everything else is in a big zip-up compartment in the other side, which is slightly messy but still easy enough to dig through to find whatever I need.

The complete kit weighs in at around 1.2KG, so it’s not exactly lightweight, but it goes in a bag with a heavy laptop and all of its peripherals so it’s not as noticeable as it might otherwise be.

It remains to be seen how much use I’ll actually get out of any of this. Maybe I’ll put a bit of paper in so that I can track how often I use each tool…

Apple, Google and aligned incentives

2020-10-17T00:00:00Z

The HTC Dream, the first phone released running Android.

For the past decade I’ve exclusively used Android phones. I got the HTC Dream (aka the T-Mobile G1) shortly after it came out, and dutifully upgraded every 1-2 years. In that timespan I used Android as the basis for my Master’s Thesis, took a job on the Android team at Google, and eventually became a contractor specialising in Android app development. So when I switched to using an iPhone earlier this year a few people were surprised¹.

The good old days

When Android was announced in 2007 – alongside the formation of the Open Handset Alliance – it was positioned as a bastion of openness: it would be built on open standards and the operating system would be open source. At the time iPhones were strongly coupled to iTunes and Apple was exercising strict control over what app developers could do.

When the HTC Dream was released it lived up to expectations. You could write apps for it without shelling out for a Mac! You could get root access, and it was running Linux under the hood! A whole ecosystem of custom firmwares and bootloaders started to appear, thanks to the open source nature of the OS. It shipped with some Google apps, but they were just normal apps that served as examples of what could be done.

After the Dream came a line of Nexus devices. These were Android’s flagship devices, designed to show off what a good Android phone should look like. Both the hardware and software releases tended to feature interesting, useful upgrades. Some of the original open source apps were replaced with closed source, Google proprietary ones, but that was OK - the open source versions lived on in the open source project as examples of what you could do. The devices allowed flashing custom firmware, and the OS source was always released… eventually.

The downfall

The Pixel and Pixel XL

Over time, Android has got less and less free. The Nexus line of phones gave way to the Pixel range, which got rid of the clearly demarcated border where Android ended and Google began. The Android 11 highlights list is dotted with sections that say “On Pixel devices…”, and every single one is a software feature that could be implemented on any device, but Google have decided to keep it proprietary instead of releasing it as part of the open source platform.

At the same time more and more functionality has been added to Google Play Services. This was originally a shared location for Google specific services - in 2012 it merely handled some Google+ functionality and dealing with OAuth for Google accounts. These days it contains a huge swathe of Google services, as well as platform functionality such as push notifications, barcode scanning, geolocation, and so on. Google Play Services isn’t part of the Android Open Source Project, and is only available under license from Google. You can’t really have a device without these functions, so as a manufacturer you have the choice between agreeing to whatever terms Google requires² or spending an awful lot of development time creating an alternative.

While these are fairly abstract arguments about what a free platform should look like, during the same period there has been a marked restriction in how you can actually use Android devices – both as an end-user and as a developer. iOS has always had very stark restrictions on what apps can do in the background; Android started its life allowing pretty much anything, like any good general purpose computing device. This inevitably lead to lots of apps doing lots of things that ranged from stupid to mildly suboptimal, creating a tragedy of the commons amongst apps. The victim was the phone’s battery life, and Google’s solution was a progressive series of restrictions on what apps can do and when. This includes limits on when push notifications are delivered to a device, how frequently apps can wake up, and so forth. Some of these can be bypassed by the end-user, but not all, and the process is fairly cumbersome.

Aligned incentives

From my point of view, Android and iOS are now pretty much in a similar place. They’re not general purpose computers, but app consoles: much like games consoles they consist of hardware and software that the user lacks control over but accepts in order to access the library of apps/games.

Given there’s no clear winner between them in terms of hardware and software, my decision came down to a more holistic question: how aligned are their incentives to my own? Apple is a product company: they make their money by producing shiny things that people want to purchase; Google is an advertising company: they make their money by using my personal information to show me targeted adverts.

For Google, Android was originally a strategic move to ensure that Apple couldn’t dominate the mobile web, and by extension the revenue from ads. As a company, there’s nothing to really push them forward in any particular direction other than one that facilitates advertising. Of course, Google employs tonnes of good people who want to do good things which counterbalances this, but I’d still prefer to deal with a company that has an intrinsic motivation to do things I want them to do, rather than one forced to by regulation, custom, or the good intent of their employees.

So the decision became obvious: if I have no strong opinions about the software and hardware, Apple is the clear winner because their incentives are a lot better aligned to mine.

Image credits

Photo	Creator	Licence	Source
HTC Dream	Akela NDE	CC BY-SA 3.0	Wikimedia
Pixel and Pixel XL	Maurizio Pesce from Milan, Italia	CC BY 2.0	Wikimedia

Or, at least, politely feigned surprise. ↩︎
And Google’s terms were particularly onerous: as well as requiring Chrome and Google Search to be preinstalled, they prevented manufacturers from selling any devices powered by alternative versions of Android - e.g., Samsung wouldn’t be allowed to sell a refrigerator that ran Amazon’s FireOS. The European Union handed Google a $5,000,000,000 fine for this anti-competitive behaviour, which Google are still in the process of contesting. ↩︎

Understanding Docker volume mounts

2019-04-01T00:00:00Z

The Docker project logo

One thing that always confuses me with Docker is how exactly mounting volumes behaves. At a basic level it’s fairly straight forward: you declare a volume in a Dockerfile, and then either explicitly mount something there or docker automatically creates an anonymous volume for you. Done. But it turns out there’s quite a few edge cases…

Changing ownership of the folder

Perhaps the most common operation done on a Docker volume other than simply mounting it is trying to change the ownership of the directory. If your Docker process runs as a certain user you probably want the directory to be writable by that user.

At first we might try something like:

FROM alpine
RUN adduser -D -u 1113 test123
USER test123
VOLUME /testing

But changing the user doesn’t seem to have any effect on the volume. Why? Checking the docs for the USER instruction shows that only affects certain future operations — namely RUN, CMD, and ENTRYPOINT. It doesn’t affect the VOLUME instruction; if it did, you’d probably just get a permission denied error unless the user you switch to had privileges to create mount points.

OK, so instead we might try using the good old chown command:

FROM alpine
RUN adduser -D -u 1113 test123
VOLUME /testing
RUN chown test123 /testing

But again, the directory is just owned by root at runtime. Back to the docs, this time for the VOLUME instruction and towards the bottom is this little tidbit:

Changing the volume from within the Dockerfile: If any build steps change the data within the volume after it has been declared, those changes will be discarded.

As soon as Docker hits the VOLUME instruction the directory becomes a mount point, and anything we do to the temporary volume mounted there is discarded during the build process. So we have to change the ownership before the instruction, which may seem a little counter-intuitive:

FROM alpine
RUN adduser -D -u 1113 test123
RUN mkdir /testing && chown test123 /testing
VOLUME /testing

Now when the container runs, the /testing directory is owned by the test123 user. It’s not quite over, yet, though. This works if we let Docker create a volume automatically for us, or if we create a named volume and mount that; if we try and mount a host directory, though, it falls flat:

$ docker run --rm -it -v "$PWD/testing:/testing" testing ls -al /testing
total 8
drwxr-xr-x    2 1000     1000          4096 Apr  1 19:39 .
drwxr-xr-x    1 root     root          4096 Apr  1 20:44 ..

Docker handles mounting host directories differently to mounting volumes, even though the syntax is basically the same. Host directories are bind mounted directly into the container, so the permissions and ownership are the same as the directory on your host. The only way to fix them are to either change the permissions on the host, or have the container change them at runtime (assuming it has sufficient privileges).

One final wrinkle in all this happens when you use the same volume in multiple containers. Here we have two images built from the Dockerfile above, one with userid 1113 and one with userid 1114:

$ docker volume create testing
testing

$ docker run --rm -it -v testing:/testing testing1113 ls -nal /testing
total 8
drwxr-xr-x    2 1113     0             4096 Apr  1 19:49 .
drwxr-xr-x    1 0        0             4096 Apr  1 20:51 ..

$ docker run --rm -it -v testing:/testing testing1114 ls -nal /testing
total 8
drwxr-xr-x    2 1114     0             4096 Apr  1 20:47 .
drwxr-xr-x    1 0        0             4096 Apr  1 20:52 ..

$ docker run --rm -it -v testing:/testing testing1114 touch /testing/Hello

$ docker run --rm -it -v testing:/testing testing1113 ls -nal /testing
total 8
drwxr-xr-x    2 1114     0             4096 Apr  1 20:52 .
drwxr-xr-x    1 0        0             4096 Apr  1 20:53 ..
-rw-r--r--    1 0        0                0 Apr  1 20:52 Hello

$ docker run --rm -it -v testing:/testing testing1114 ls -nal /testing
total 8
drwxr-xr-x    2 1114     0             4096 Apr  1 20:52 .
drwxr-xr-x    1 0        0             4096 Apr  1 20:52 ..
-rw-r--r--    1 0        0                0 Apr  1 20:52 Hello

Can you see what’s going on? When the volume is empty, the ownership changes based on the mount point in the container. Once it has something in it, the ownership is fixed.

So Docker behaves differently with regard to permissions:

when the folder is mounted from the host vs a volume
when the volume is empty vs having content

Pre-populating mounts with files from the image

One of the more esoteric features of the way Docker handles volume mounts is that in some cases files from the image are copied over into the container. For example:

$ docker volume create testing
testing

$ docker run --rm -it -v testing:/etc testing sleep 1

$ docker run --rm -it -v testing:/tmp testing ls -al /tmp
total 184
drwxr-xr-x   15 root     root          4096 Apr  1 20:58 .
drwxr-xr-x    1 root     root          4096 Apr  1 20:59 ..
-rw-r--r--    1 root     root             4 Jun  7  2018 TZ
-rw-r--r--    1 root     root             6 Dec 20 21:31 alpine-release
...

The first container we run mounts the newly created testing volume at /etc. Docker copies all the existing files and folders into the volume; when we then run the second container with the volume mounted at /tmp, we can see all the files that were in the first container’s /etc.

As with permissions, this behaviour is anything but consistent. Say we switch from a volume to a host directory:

$ mkdir testing
$ docker run --rm -it -v "$PWD/testing:/usr/bin" testing sleep 1
$ ls -al testing
total 8
drwxr-xr-x 2 root  root  4096 Apr  1 22:05 .
drwxr-xr-x 3 chris chris 4096 Apr  1 22:05 ..

Nothing is copied in, and inside the container the folder will be empty. Based on our discoveries with permissions, it’s reasonable to assume the same will happen with a non-empty volume too:

$ docker volume create testing
testing

$ docker run --rm -it -v testing:/testing testing touch /testing/Hello

$ docker run --rm -it -v testing:/usr/bin testing sleep 1

$ docker run --rm -it -v testing:/tmp testing ls -al /tmp
total 8
drwxr-xr-x    2 root     root          4096 Apr  1 21:09 .
drwxr-xr-x    1 root     root          4096 Apr  1 21:09 ..
-rw-r--r--    1 root     root             0 Apr  1 21:08 Hello

So at least that’s consistent. If you’re very observant, though, you might notice I switched from /etc/ to /usr/bin in the examples. That’s because within the container /etc/ has some files bind-mounted into it, such as /etc/resolv.conf, and these do always result in files being created in the mounted volumes or folders:

$ mkdir testing
$ docker run --rm -it -v "$PWD/testing:/etc" testing sleep 1
$ ls -al testing
total 8
drwxr-xr-x 2 chris chris 4096 Apr  1 22:12 .
drwxr-xr-x 3 chris chris 4096 Apr  1 22:12 ..
-rwxr-xr-x 1 root  root     0 Apr  1 22:12 hostname
-rwxr-xr-x 1 root  root     0 Apr  1 22:12 hosts
-rwxr-xr-x 1 root  root     0 Apr  1 22:12 resolv.conf

Summary

Docker treats mounting host folders and mounting volumes differently. Don’t just assume that you can swap one for another and get the exact same behaviour.
Empty volumes will inherit permissions and files from the image they are mounted in; non-empty volumes and host folders will not.
Relying on Docker copying files into volumes is a very bad idea, as if you change those files in a future version of your image they will not be copied unless the volume is deleted and recreated.

I can’t find anywhere that these points are documented properly; if you know of anywhere, please drop me a message!