Chameth.com - posts like how-tech-companies-failed-to-build-the-star-trek-computer but not debugging-beyond-the-debugger, migrating-from-github-to-forgejo, why-you-should-be-using-https

How tech companies failed to build the Star Trek computer

2025-07-16T00:00:00Z

A computer core room on the Enterprise-D

In most Star Trek series, the ship or station computer is ever-present in the background, waiting to be called on by the main characters¹. It nearly always does exactly the right thing, and there’s little limit to the functions it can perform. Take this mundane example from DS9:

KIRA: Computer, establish link with the Bajoran Medical Index for the Northwestern District.
COMPUTER: Link established.
KIRA: Access all information on Doctor Surmak Ren.
COMPUTER: There are no records matching that name.
KIRA: Try the Northeastern District, same search.
COMPUTER: Doctor Surmak Ren, currently serving as Chief Administrator of the Ilvian Medical Complex.
KIRA: Computer, open a channel to the Ilvian Medical Complex. Administrator’s office.

The computer is doing some kind of networking to a database only identified by name. It does a search and summarises the lack of results. It then repeats the process with another database, and succinctly announces the results. Finally, it opens a communication channel to a specific room in a facility, based only on its name.

This whole interaction is remarkably boring². Kira doesn’t have to know any URLs or API endpoints, or what protocol she wants to use. She doesn’t have to open a specific app and then login and then try the query again. She just says what she wants and the computer does it.

It seems like this should be one of the most easily obtainable bits of sci-fi wizardry with our current technology. We have multiple massive companies throwing lots of money at digital assistants, LLMs that are improving at an insane rate, but we’re somehow not even close to the usability or usefulness of the Trek computers. What gives?

Boring is, well, boring.

Larry Page once said something that might help explain it:

The Star Trek computer doesn’t seem that interesting. They ask it random questions, it thinks for a while. I think we can do better than that.

This is the same Larry Page that founded Google, whose mission statement is “to organize the world’s information and make it universally accessible and useful”. Of all people, surely he should find an omnipresent computer that can answer ‘random questions’ interesting?! It seems like it should be the epitome of Google’s mission!

Google’s “better than that” seems to have been to stuff LLMs into every product they can, even when you don’t want them there. Even when they’re worse than the normal content they displace. These things look exciting when they’re part of a scripted demo at Google I/O, but they fall flat and just get in the way when they’re exposed to the reality of day-to-day use.

The Star Trek computer is the opposite: it isn’t snazzy, but it is genuinely useful. That means it’s not an attractive target for the company execs who want marketing opportunities, and it’s not appealing for engineers who need to demonstrate “impact”. But even if Google did try to make the Trek computer, there are other problems…

Assistants need to be free

A significant amount of tech companies’ business models currently revolves around trapping users in walled gardens. They want you using their ecosystem; that way they get more data from you, and you’re more likely to spend more money on their other offerings that work together. There’s barely any incentive to allow any kind of interoperability with other platforms outside carefully contracted integrations.

I remember trying to help a family member move their photos from iCloud to Google Photos. At one point they turned around and said, exasperated, “why is this so hard? Aren’t they both in the cloud?!”. It’s easy to dismiss that as someone who hasn’t quite grasped the fundamental idea that “the cloud” is just someone else’s computers, but that’s not the whole story. There’s no reason why there shouldn’t be a quick and easy transfer: both services already allow uploading and downloading, there’s just no incentive for the companies involved to make it so³.

These kinds of misaligned incentives and walled garden business models cause even more problems when it comes to digital assistants. Siri is basically never going to be able to interact with, say, your Google Drive; ~~Bard~~ Gemini is never going to be able to send a message via iMessage. Even when there are appropriately blessed interactions, they’re so clunky. Can you imagine Captain Picard saying “Computer, ask the turbolift skill to take me to deck 5”?

Someone else’s computer

Software issues aside, there’s still a key difference between the Star Trek computers and our current batch of digital assistants: where they run. The Trek computers are all housed within the ship or station they serve; they can connect elsewhere to gather information, but they run entirely independently. If they go wrong, a local engineer can go in and fix things. While some of our assistants may have physical hardware in your home, they don’t work without a vast cloud apparatus behind them. If your Internet connection fails, they become paperweights. If the company running them decide to remove some functionality you depend on, you have no recourse.

That kind of helplessness isn’t limited to assistants, either. There’s a rapidly growing trend of being unable to modify or repair hardware you fully own and control. Part of this is just that they’re becoming more complex: it’s a lot harder to replace a microchip than a gear, but companies are also going out of their way to make it more difficult for users through draconian DRM regimes⁴ and aggressive intellectual property enforcement. If the US Navy can’t repair their own equipment because a corporation says so, what hope do consumers have?

We’re approaching a point where you don’t actually own anything. Software is cloud and subscription based, hardware is unrepairable. Even cars can be remotely updated and have features added or removed. The Federation wouldn’t allow a third party control over their ships⁵, so why are we so happy to put up with it in everything we consume?

A small ray of hope?

The most promising way of tackling all of these problems is through legislation. The EU’s Digital Market Act is an attempt to force ‘gatekeepers’ like Google, Apple and Meta, to allow third-party access to their services. It seems like a pretty reasonable approach, but the tech companies are unsurprisingly resisting it. Apple in particular have gone out of their way to refuse to comply, and when forced to do so have limited the functionality to people in Europe. Still, the DMA is a promising start, and if similar legislation is introduced (and robustly enforced) elsewhere it might start forcing companies to behave a bit better.

There are also smaller companies that actually do the right thing. Framework make laptops that are user-serviceable; Fairphone do the same for mobile phones. Smaller software companies provide useful, open APIs. The average person on the street will probably have never heard of these, unfortunately, but they do still exist. Maybe as the bigger tech companies tighten the screws more, people will turn to alternatives like this? Or maybe we’ll just keep accepting that our computers work for everyone but us?

Unless, of course, the computer is playing the role of the episode’s MacGuffin and has contracted space-computer-COVID or something, then it’s a lot less in-the-background. ↩︎
It’s almost like it only exists to move the plot along. ↩︎
You can generally export your data, thanks to a combination of legislation and efforts like Google’s “Data Liberation Front”, but I’ve never seen an export format that could then just be imported into an equivalent commercial product. ↩︎
Oh, you’ve changed the screen on your iPhone? Better hope it can do the secret handshake with the Apple hardware. ↩︎
I think there might actually have been an episode where that did in fact happen. We’ll just ignore that as a plot contrivance. ↩︎

The Ethics of LLMs

2025-06-22T00:00:00Z

I’ve written about LLMs a few times recently, carefully dodging the issue of ethics each time. I didn’t want to bog down the other posts with it, and I wanted some time to think over the issues. Now I’ve had time to think, it’s time to remove my head from the sand. There are a lot of different angles to consider, and a lot of it is more nuanced than is often presented. It’s not all doom and gloom, and it’s also not the most amazing thing since sliced bread. Who would have thought?

It’s worth noting that I’m just setting out my position here. I’m not trying to convince you to change your mind. To set the scene a bit, I mainly use Claude Code as a programming tool. I use the Claude chat interface sometimes to proofread things, do random one-off data analysis, or help organise things. More rarely I’ll try to use it to brainstorm things, or recommend things, or do more “creative” things, but I don’t trust it enough in those domains to do it often. I don’t use it for research or as a Google replacement¹, which I recognise probably makes me a weird half-in-the-water, half-out-of-it class of user.

Copyright & Corporate Control

One of the key issues, and something that is being prosecuted in several court cases right now, is how LLMs interact with the copyright system. And by “interact with” I mean “run roughshod all over”. It seems pretty obvious from my lay perspective that if having 10 seconds of pop music in the background of a YouTube video is copyright infringement, then Meta pirating books via BitTorrent must also be.

That said, the entire copyright system as it exists now is fundamentally broken. I don’t mind the idea of individual creators having protection, but I absolutely hate how copyright is wielded as a blunt instrument by massive corporations to intimidate individuals and to try to earn more money. If a law enables a huge conglomerate to sue the author of an open source project for a theoretical 75 trillion dollars then that is a bad law. Copyright today has been twisted well beyond what it was meant to be originally, thanks to lobbying by people and businesses with lots of money that want to keep it.

So if copyright law is not fit for purpose, are LLMs fine? Well, not quite. The argument that LLM training is a bit like a child learning to read holds some merit, but it falls flat when LLMs can regurgitate the input material verbatim². Personally I don’t care if they steal all the content of the New York Times and all the Disney characters that are still in copyright. It’s the individual harm that bothers me: the struggling author whose words are slurped up by the LLMs and regurgitated, potentially costing them readers; the open source dev who chooses a copyleft licence on their work, only for an LLM to dump complete copies of functions in someone else’s codebase with no attribution or licence, stopping any future improvements from being released for the public good.

I can mostly mitigate those concerns in my personal use. When coding I generally direct the agent closely enough that the opportunity to drop in massive chunks of someone else’s code is negligible. I don’t use LLMs to generate big chunks of written content. Overall I’m reasonably happy that I’m not making anyone worse off.

One related area that doesn’t affect me much because of my limited use, but does concern me a lot, is corporate control. Most of the frontier LLMs are built by big tech companies. As more and more people are using LLMs, they become an avenue for control over information. It’s clear the people in charge should not be trusted with that responsibility, as Grok’s whole Boer War thing demonstrated. I’m not sure what the answer is, though. Would a government built LLM be any better? Probably not. The requirements to actually do the training are so high that the only people who can do it are the ones you wouldn’t want to.

So I’m left in the uncomfortable position of using tools by companies I don’t trust, trained on data they shouldn’t have been trained on. Still, there can’t be many more ethical issues, right?

Forced Features & Flattery

Before we get onto the more obvious ethical topics, let’s go on a brief detour so I can complain about how LLMs are being crowbarred into seemingly EVERY SINGLE SERVICE. I don’t want an LLM in my text editor. Or when shopping. Or taking screenshots of everything my computer does every few seconds. I don’t want LLMs anywhere outside a clearly-defined LLM box. With these forced integrations, you’re even more in the dark about what their prompts are, what information they have access to, what underlying model they’re using, and so on.

There’s no good reason for 90% of these implementations; there’s just some weird Silicon Valley hype train that everyone is scared to miss out on. And as a result we get more and more screen real estate taken up with things with animated gradients that can’t be turned off and are prompted to talk in an insufferable manner.

The problem isn’t just that they’re everywhere, though, it’s how they behave when they’re there. The overly sycophantic behaviour that most LLMs adopt can hook users in the same way a social media app can fiddle with its algorithms to encourage people to keep scrolling. It’s a lot harder to dismiss an LLM when it’s blowing smoke up your ass. Now I don’t believe this behaviour is actually intentionally nefarious, but more of an artifact of the models overfitting on user approval instead of something useful like accuracy.

The combination of going too far to please the user and hallucinating things is incredibly problematic. How are ordinary, non-tech savvy, people meant to deal with these models being forced in front of them? If you ask a question and get back a flattering answer that’s roughly the shape of what you were expecting, would you question it? Would a student, who was doing their homework in Google Docs when an LLM inserted itself into their lives question it?

This isn’t just a problem for naive users, either: I occasionally ask Claude Code “Can we do X?” and discover much later that the answer should either be an emphatic “NO”, or at least have a page of caveats attached. But the LLM wouldn’t want to displease me, so it tries to do what’s asked of it, with all kinds of weird hacks, and ultimately gets itself tied up in knots. This is not how a tool should behave. If I wanted someone to say “Yes, and” repeatedly I’d join an improv troupe.

Environmental Expenses

The biggest ethical issue with LLMs is their environmental impact. Are we adding more fuel to the dumpster fire that is our planet, just so my phone can inaccurately summarise news headlines for me? The answer is a resounding “maybe”. The companies making the LLM models are cagey about how much energy goes into making them and then answering queries. There are estimates for the latter, but there are several orders of magnitude between the most optimistic and most pessimistic. LLMs are definitely not good for the environment. Basically no computer activity is³. But you don’t see people campaigning to shutter YouTube (for example) because of its environmental impact. I struggle to see the difference.

I’m not saying considering the environmental impact isn’t important. We are killing the planet. We are not doing enough about it. More attention to the climate crisis in general is a good thing. But even with the most pessimistic estimates on power usage, my personal LLM usage is a drop in the ocean. I can have far more impact by reducing the amount I travel, avoiding red meat, and so on.

One issue I do have, though, is the forced integrations I mentioned. I’m happy with my personal usage. I’m not happy that a whole slew of LLMs are being operated on my behalf. How much power is wasted by Google generating LLM results in every search results page, versus how much benefit it provides? How do people know if a search box will just use a few Watts doing a database search, or burn kilowatts⁴ spinning up an LLM and filling its context window?

Then there’s the issue with training. Who knows what resources go into training these models? Whether the amortised cost of that across all its users is worth it? I think the answer to the first question is “only the tech companies making the models”, and their utter silence on the second question says volumes.

But this kind of industrial power isn’t unique to LLMs. There are many industries that do far worse. And that’s not a justification, but I think it makes sense to look at them more holistically. In an ideal world, where we have functioning governments that listen to scientists⁵, we could address all of this with legislation on emissions and clean power. Then all of these industries will either manage to function in a manner compatible with keeping the planet alive, or will have to stop. I appreciate that actually legislating that is about as likely as waking up one day to find Claude has become an AGI, though.

Slop & Survival

We’re nearly done, I promise. You can see why I didn’t want to try to tackle these issues in an earlier post! The last big issue I want to talk about is slop. It feels like it’s everywhere, and still getting worse.

The Internet has slowly morphed from a wonderful place full of individual, quirky sites, into a desert of bland corporate silos, and now into a wasteland of AI slop. It’s becoming more and more difficult to find original information. There’s this conspiracy theory called “Dead Internet Theory” that basically says nearly all traffic on the Internet is generated by bots as part of a co-ordinated effort to manipulate us. Obviously it’s unhinged, but at the same time some elements of it are not that far off the mark. How long will it be before the Internet is so drowned in slop that it is, effectively, dead?

The other issue is what happens to the human jobs that have been replaced with slop? It’s way too easy for a money-pinching publication to get rid of their human staff in favour of pushing a few buttons and publishing the output. You’d hope that market forces would balance things out, but there doesn’t seem to be much sign of that happening. There are small movements of people putting badges and other marks on their work to show it was made by a human, perhaps that will take off?

On a more personal level: what happens to software developers? As it stands, code agents are nowhere near good enough to replace a human. At least not if you want a non-trivial output that works, can be changed in the future, and doesn’t have massive issues. Even if they get substantially better, I feel like you’d still need a knowledgeable human in the loop to keep a rein on everything. Does that mean the world needs far fewer software engineers? Probably. But it doesn’t mean we’re all going to be out of jobs overnight as some people seem to be predicting. The only way I can see that ever happening is if there’s a radical shift in how software is made: a new language of some kind that’s uniquely suited to LLMs, or new ways of composing software together from smaller parts⁶.

So… Yeah. Where does that leave us? I’ve written several thousand words about how LLMs are an ethical mess, but I’m going to carry on using them. That’s uncomfortable. But it’s not unlike using Amazon: they’re not a great company, I’d rather give the money to a local shop or supplier, but I’m also not going to massively inconvenience myself by being quixotic about it. Life is about compromises like this, and it’d be silly to think otherwise. I’ll stand on morals on some things⁷, but not to the extent of becoming a reclusive old crank shouting at the clouds.

That’s what Kagi is for. It’s great. You should try it. ↩︎
Or, maybe even worse, with added hallucinations. ↩︎
Hell, basically no human activity is good for the environment. ↩︎
±several kilowatts, because who knows? ↩︎
and didn’t elect people who withdraw from climate agreements, re-open coal plants, and so on… ↩︎
I don’t think I’m just saying this to reassure myself… ↩︎
I’ll grudgingly give Jeff Bezos money, but I sure as hell am not going to use any service from Elon Musk. ↩︎

Apple, Google and aligned incentives

2020-10-17T00:00:00Z

The HTC Dream, the first phone released running Android.

For the past decade I’ve exclusively used Android phones. I got the HTC Dream (aka the T-Mobile G1) shortly after it came out, and dutifully upgraded every 1-2 years. In that timespan I used Android as the basis for my Master’s Thesis, took a job on the Android team at Google, and eventually became a contractor specialising in Android app development. So when I switched to using an iPhone earlier this year a few people were surprised¹.

The good old days

When Android was announced in 2007 – alongside the formation of the Open Handset Alliance – it was positioned as a bastion of openness: it would be built on open standards and the operating system would be open source. At the time iPhones were strongly coupled to iTunes and Apple was exercising strict control over what app developers could do.

When the HTC Dream was released it lived up to expectations. You could write apps for it without shelling out for a Mac! You could get root access, and it was running Linux under the hood! A whole ecosystem of custom firmwares and bootloaders started to appear, thanks to the open source nature of the OS. It shipped with some Google apps, but they were just normal apps that served as examples of what could be done.

After the Dream came a line of Nexus devices. These were Android’s flagship devices, designed to show off what a good Android phone should look like. Both the hardware and software releases tended to feature interesting, useful upgrades. Some of the original open source apps were replaced with closed source, Google proprietary ones, but that was OK - the open source versions lived on in the open source project as examples of what you could do. The devices allowed flashing custom firmware, and the OS source was always released… eventually.

The downfall

The Pixel and Pixel XL

Over time, Android has got less and less free. The Nexus line of phones gave way to the Pixel range, which got rid of the clearly demarcated border where Android ended and Google began. The Android 11 highlights list is dotted with sections that say “On Pixel devices…”, and every single one is a software feature that could be implemented on any device, but Google have decided to keep it proprietary instead of releasing it as part of the open source platform.

At the same time more and more functionality has been added to Google Play Services. This was originally a shared location for Google specific services - in 2012 it merely handled some Google+ functionality and dealing with OAuth for Google accounts. These days it contains a huge swathe of Google services, as well as platform functionality such as push notifications, barcode scanning, geolocation, and so on. Google Play Services isn’t part of the Android Open Source Project, and is only available under license from Google. You can’t really have a device without these functions, so as a manufacturer you have the choice between agreeing to whatever terms Google requires² or spending an awful lot of development time creating an alternative.

While these are fairly abstract arguments about what a free platform should look like, during the same period there has been a marked restriction in how you can actually use Android devices – both as an end-user and as a developer. iOS has always had very stark restrictions on what apps can do in the background; Android started its life allowing pretty much anything, like any good general purpose computing device. This inevitably lead to lots of apps doing lots of things that ranged from stupid to mildly suboptimal, creating a tragedy of the commons amongst apps. The victim was the phone’s battery life, and Google’s solution was a progressive series of restrictions on what apps can do and when. This includes limits on when push notifications are delivered to a device, how frequently apps can wake up, and so forth. Some of these can be bypassed by the end-user, but not all, and the process is fairly cumbersome.

Aligned incentives

From my point of view, Android and iOS are now pretty much in a similar place. They’re not general purpose computers, but app consoles: much like games consoles they consist of hardware and software that the user lacks control over but accepts in order to access the library of apps/games.

Given there’s no clear winner between them in terms of hardware and software, my decision came down to a more holistic question: how aligned are their incentives to my own? Apple is a product company: they make their money by producing shiny things that people want to purchase; Google is an advertising company: they make their money by using my personal information to show me targeted adverts.

For Google, Android was originally a strategic move to ensure that Apple couldn’t dominate the mobile web, and by extension the revenue from ads. As a company, there’s nothing to really push them forward in any particular direction other than one that facilitates advertising. Of course, Google employs tonnes of good people who want to do good things which counterbalances this, but I’d still prefer to deal with a company that has an intrinsic motivation to do things I want them to do, rather than one forced to by regulation, custom, or the good intent of their employees.

So the decision became obvious: if I have no strong opinions about the software and hardware, Apple is the clear winner because their incentives are a lot better aligned to mine.

Image credits

Photo	Creator	Licence	Source
HTC Dream	Akela NDE	CC BY-SA 3.0	Wikimedia
Pixel and Pixel XL	Maurizio Pesce from Milan, Italia	CC BY 2.0	Wikimedia

Or, at least, politely feigned surprise. ↩︎
And Google’s terms were particularly onerous: as well as requiring Chrome and Google Search to be preinstalled, they prevented manufacturers from selling any devices powered by alternative versions of Android - e.g., Samsung wouldn’t be allowed to sell a refrigerator that ran Amazon’s FireOS. The European Union handed Google a $5,000,000,000 fine for this anti-competitive behaviour, which Google are still in the process of contesting. ↩︎