The Inhuman State of Publishing

Over the past few weeks, we've seen multiple announcements that juried awards for short fiction, and magazines and anthologies that buy short fiction, have unwittingly recognised or accepted stories produced (or widely suspected of being produced) using generative AI. This is a particular issue in the science fiction and fantasy fields, since for various historical reasons those are the only ones in the English-language marketplace where published short story writers consistently make money from their work, which makes the genre an appealing target for scammers. We've been hearing for a couple of years about SFF markets being bombarded with AI output, ever since Clarkesworld magazine, which is almost always open for submissions, had to briefly close in 2023 due to being overwhelmed by the volume of LLM-produced manuscripts it was receiving.

Like the vast majority of people in SFF, I hate generative AI slop. I hate that half the pages returned by any search engine query are full of padded, bland, unreliable text. My eyes glaze over when I try to read it. I don't like to look at slopped-out images on book covers, on the walls of local businesses, or in political memes shared on Facebook. I ditched Google as a search engine and switched to the subscription service Kagi so I wouldn't have to look at a half-accurate LLM search summary every time I looked something up online.

The news of university students cheating their way through courses without learning anything is alarming and depressing. The idea that LLMs, which can aggregate existing human knowledge, are working to prevent anyone developing the human expertise to think anything new, makes me want to crawl in a hole. I also don't want to live near a massive AI data centre or share a municipal power grid with one, and I can't imagine many people do.

What's going on in tech

But at the same time that the tech industry is adding pointless and aggravating AI functionality to every device and application I use, the one place LLMs as a technology seem to actually be paying off is in software development. People I know who work as developers seem to find code generation and analysis tools quite useful, and they are being used by, from what I can see, an overwhelming majority of people in the IT industry. And unlike creative writers and nonfiction writers and artists, coders seem to generally be quite happy about this.

There aren't systematic and reliable surveys on this, but there are some partial and unreliable ones, and we have our own social circles and professional networks as a kind of smell test of those. I know a lot of people who work as writers or in publishing more generally, and they are largely pretty miserable about the current state of affairs. The ones who are pleased by the possibility of generating their work using LLMs tend to be people already caught up in the self-publishing treadmill, where authors need to turn out a book every few weeks, spend tens of thousands on ads and other expenses, and if they're lucky, sell enough to turn a small profit for doing what amounts to a full-time job. It's a volume game, and if there's one thing we can say about LLMs it's that they produce in volume.

Among developers I know, and according to things like a recent survey among users of Clojure, the world's most satisfying programming language, the pattern is reversed. People in tech are often quite down on the tech industry in general, the horror of having to be in LinkedIn if you're looking for work, a host of other issues. But there's a much more positive view of LLM tools for writing code, documenting existing code, finding and fixing bugs, etc. And when I think about this and [Darth Vader voice] search my feelings I find that I don't have the same gut-level revulsion at the idea of workers using LLMs to write software that I do to their use in writing journalism or memoirs or fiction.

What's going on here? You could say it's some variation on the Gell-Mann effect: "Generative AI is terrible for the industry I work in, but I guess maybe it's okay for other industries." I don't think that is what's happening, because in my case I did write code professionally for a number of years. I haven't been paid to code this decade, but I still do some sysadmin work and some hobby code projects, so I feel like I have a foot in both worlds.

I haven't used LLM code tools personally – I don't like interacting with chatbots, why would I do it for a hobby project? – but recently, when I reported a minor bug in a big open source software package, the developers replied within a couple of hours with a Claude Code transcript that confirmed the bug existed, identified where in the codebase it was located, and suggested a simple fix. The bug was patched and merged into the release version of the software by the following day, which almost certainly would not have been the case for a fairly low-priority issue like this five years ago.

So I'm willing to concede there might be some uses for LLMs in software development, even if the negatives might still outweigh the positives. Ultimately it isn't going to matter much what I personally think about this: if developers find these tools useful they're going to use them no matter how much disapproval they get about it from science fiction writers. But this is a chewy philosophical dilemma, so I do want to try thinking about it and sorting out my own position on the issue. And although a lot of chatbots right now are running on hardware in pollution-generating data centres, the technology has advanced enough that even if the AI bubble popped tomorrow and all those data centres were converted to Laser Tag arenas, there are enough sets of model weights available for download to allow coders to switch to LLM coding on their own machines without much interruption. It may not continue at exactly the same rate, but it will continue.

The current state of the art

When I say the technology has advanced, I think it's easy for people discussing this topic to talk past one another. People who use these tools tell me, and I believe it, that they have improved fast and that the state of the art now is markedly different from what it was six months ago, and in another world from what it was 18 months ago. So if a critic confidently proclaims, based on experiences from last winter or even last spring, that LLMs produce buggy code that humans have to tediously inspect and fix, they may be saying something that was correct a little while ago but no longer is. I believe there are specific commercial niches where the results are still unimpressive, but in general if someone argues that the technology simply doesn't work it's worth mentally annotating with a little [citation needed].

One question we might want to ask is: are LLMs also improving at writing fiction? The recent short story announcements definitely tell us that they're fooling more people, but whether their fiction is "better" isn't such an easy question to answer. Something that has made it possible to create LLMs that produce better code is that we have some objective standards to apply. For any individual code block you can ask "does it compile?" or "does it add unnecessary steps?" or "is it incredibly slow and also a resource hog?" We don't have objective standards for what makes a paragraph good: everything up to and maybe even including basic comprehensibility is a value judgement.

So it's difficult to say if the recent story accepted by Bona Books is an improvement on what was being generated two years ago. If we look at the editors' announcement, which boils down to "we were excited to publish this story, but then learned it might be AI, and on second inspection the story is bad" we can get a little insight into what might be going on. First of all, they were excited to receive a story that so closely fit their call for submissions. But of course, if you wanted a set of instructions ready-made for prompting an LLM, a themed call for submissions would be a great place to start, so it's not too surprising that the output they got matched what they were looking for.

Secondly, if editors are looking to encourage marginalised voices, as the Bona Books editors were, they may be inclined to give authors the benefit of the doubt. But it's an editor's job to take a critical approach to a story and in the age of LLMs, "benefit of the doubt" leaves them exposed to being tricked by AI grifters.

For my part, I would say it isn't possible for an LLM to get better at generating fiction, because when I'm reading fiction I want to be receiving communication from another human mind. To attempt an analogy: if someone in one of my groupchats started using an LLM to generate their posts it's impossible for those posts to be good. The purpose of the groupchat is to spend time talking to people I like, and if some of its members decided that was too much effort and an algorithm could do the talking for them that's inalterably pathetic loser shit that violates the social contract. If a passage of prose contains a counter-intuitive turn of phrase, maybe it was produced by a human mind and it's worth spending some effort figuring out why the author made that choice, or maybe it was generated by an LLM, in which case it is just a byproduct of applying statistics to a vast corpus of text and there is no "why" beyond that.

When I'm reading journalism I want it to be produced by a journalist. It's possible for a journalist to be bad: they can choose to fabricate quotes or events, or to turn in sloppy and inaccurate work. It isn't possible for an LLM to make that choice, since LLMs produce sloppy work and fabrications as part of business as usual. If you judge a journalist's work by their track record, which you should, the track record of any LLM is that it's incapable of producing reliable journalism, and that's an inherent feature of the technology. If a journalist starts by using an LLM to produce a story, and then manually corrects and modifies it so it's fit to be published under their byline, what they're doing is starting with something that they should assume contains fabrications and theoretically working on it until it's accurate. But we know that using LLMs for this tends to make people lazy so that fabrications and other unethical material leak through – we've seen plenty of examples of that already.

There's a similar risk for fiction writers who still write their own work but use the LLM as a kind of ancillary tool, generating ideas meant to spur their own creativity. Aside from the fact that this sounds like a Yakov Smirnoff routine — in Soviet Russia, LLM prompt you! — it feels like it's slightly too tempting to start off that way but let the LLM do more over time, laundering down your individuality and squandering your gifts in the process. Boosters tend to promote this kind of technology as a kind of force multiplier: AI x human operator = superintelligence. But judging by the artistic achievements of AI to date it seems a more likely outcome is LLM x zoned-out human = slop.

I'm aware that this part of my argument is somewhat idiosyncratic and not everyone will agree with it, but a piece of LLM text meant to be resemble communication between humans is, in my view, incapable of being the thing it is supposed to be. I'm not sure I would say the same about a piece of software whose code was generated by an LLM. I don't use Microsoft Excel, for example, to receive communication from the human mind(s) that built it; it's a tool. A tool can be well-suited or poorly-suited to a task. A rock you find on the ground can be a good tool for some task; it's the intention of the user rather than the intention of the maker that counts.

The two cultures

One way code is different from human communication is that there's a vast tradition of code that automatically generates other code, whether that's a high-level programming language that automatically generates machine code, or something specialised like a web framework that provides a shorthand for producing a bunch of boilerplate code to create and serve webpages. I could boil this down to: you can write a program that generates code, but you can't write a book that automatically writes another book.

Also, a huge amount of code is boilerplate. When I was living in the UK developing bioinformatics software I spent maybe a quarter of my time working on the interesting stuff finding ways to manipulate nucleotide ambiguity symbols, and the other three quarters of my time working on reading from and writing to the database, or handling file uploads, or fiddling with Bootstrap templates so the information I retrieved from the database displayed properly on the screen. Things that should basically be solved problems, in other words. My specific situation was just different enough that I kept having to find slightly new ways to solve them. And in my experience a huge amount of software development is like this. So LLMs, which can produce any kind of boilerplate text on command with as many modifications as you like, are quite useful in this situation and the fact that they're derivative isn't really a huge strike against them.

Prose writers who have spoken in favour of LLM-generated text seem (I'm generalising) to be the type who think an idea is the most important thing to communicate in a piece of writing, and assembling the words is just drudgework. But the actual embodied text is the writing. Any paragraph has a thousand conscious or unconscious choices that go into it: formal register, syntax, vocabulary, rhythm, construction. And things that don't necessarily register for every reader: subtext, allusion, ambiguity. The thing that makes my eyes glaze over when reading LLM-generated text is its lack of information density, because a lot of these choices are replaced by rote pastiche. You can't satisfyingly turn a short prompt into a long piece of writing because you can't write a paragraph in the abstract.

By contrast, you can write code in the abstract, as anyone who has ever been asked to write pseudocode during a job interview can testify. You can determine all the things you want a section of code to do before you sit down to work on it. One of my favourite things about the days when I was working in SQL and Lisp was that when the power went out in my company's office I was able to switch to figuring out my problem with little pen and paper diagrams, then transcribe my solution into code when the workstation booted up again. Code is just notation for communicating logic to a computer and as long as your logic is efficient, the stylistic choices you make within it don't make the software run any differently. People have developed all sorts of conventions around specific styles of code formatting, but those are to ensure the code is readable by humans. The machine doesn't care.

One of the reasons I don't want anything to do with LLMs currently is that I've heard second-hand accounts of cases of chatbot psychosis, and they are terrifying. I don't think we know the full risks of things like this, but it feels like using the technology to solve an engineering problem is less scary territory than treating it as if it's an artistic collaborator.

Last, I want to highlight a cultural difference between the ways authors and coders think about what they write. There's a very famous site for programmers called StackOverflow where people can post about problems they're having and the site's users can suggest solutions. For example, at one point I didn't know how to safely write a specific function in a web app, so I opened StackOverflow, described what my issue was, and within 20 minutes someone had posted a solution to my problem.

I've never used a site like this for writing fiction or articles, and if I was stuck on a paragraph I wouldn't really welcome some stranger writing one for me. Pride of authorship would hold me back from just using it wholesale, I'd worry about some sort of copyright issue and in any case, writing styles are unique enough that one paragraph by another author in the middle of a longer piece would probably be quite jarring. But in the case of this code problem, I looked at what the other user suggested, understood that it did the thing I wanted, and then copy-pasted it directly into my app. As the other user intended! There are real differences between the way authorship is viewed in the code and prose communities, and in the way coders more often share their work for others to use without compensation (because having a good StackOverflow score and a lot of open source commits on GitHub was often a good way to find well-paying employment in IT).

I'm not going to get deeply into copyright issues in this piece, which is already long enough, but under current US law LLM output can't be copyrighted. In writing, copyright is a load-bearing pillar for enforcing what's sometimes called the "author's monopoly" that allows them to profit from their work. In software the author's monopoly is less important since there are so many examples of people who make money from writing free and open source software. So even though the same copyright laws apply to both fields, the actual role of copyright is different. This is a much bigger issue than I have the space or expertise to cover, but I thought it was worth mentioning.

Apparently StackOverflow has become a bit of a ghost town in the LLM code era, which is a real shame, since it was an incredible learning resource in its day. As I start to wrap up this essay I want to spend a little time thinking about some second- and third-order effects of LLM coding.

What does it all mean

One question I still have is "What is AI code generation really worth?" By which I mean, even if reliable code can be generated quickly, will that make developers hugely more productive, will it make software better, are things possible now that would have been impossible without it. These questions basically come down to: is actually writing the code a major limiting factor in software development?

For single-developer projects like ones I've worked on where I was basically chief cook and bottle washer and spent all my time coding, the answer would have been yes. For projects where I was part of a small team (4-5 developers sitting across from one another), also probably yes. With an agentic LLM we could have accomplished tasks in minutes not hours, hours not days, days not weeks.

For a big software package though, production software with thousands of interacting components, used by millions of users, I start to feel less sure. In a lot of corporate developer jobs it feels like writing code is the part of job you do when you're not on Zoom calls discussing what you want the code to accomplish.

And you still need expertise. When I filed my open source bug report and the developers found a solution using Claude, they still needed enough programming experience and familiarity with the codebase to know if the proposed solution was going to cause huge problems elsewhere. Individual blocks of code can be assessed using objective criteria, but the design of a software package still does involve a lot of value judgements that can't be automated away. Skilled humans are still needed to make decisions and oversee the LLM, and anyone who thinks otherwise is going to end up with massive problems for the forseeable future. This is a point of unanimity among developers I know.

There's also the issue that this technology makes bad developers able to write exponentially more code, potentially causing more problems for everyone else. One irritation I have about the way people have been predicting the societal impacts of AI is that you get the impression a lot of boosters only know how to extrapolate using the multiplication table. If a developer can do a month's work of coding in a day with Claude Code, then that means in a year they could do [frantically tapping at a calculator]... 20 years worth of work! But that's not how technological advancements actually function in society. Ask any science fiction writer. Or economist. Or historian. Unfortunately the future supply of people in these disciplines is currently under threat from generative AI technology.

A basic question I have is this: let's say I had a lot of money and that I hated Microsoft Word. At least one of those is true. And let's say that, like various people I know, I've tried using the free/open source equivalents and found that they didn't do all the things I need. If I wanted to hire a bunch of people to build me a fully-functional word processor from scratch that was MS Word compatible but better, then in the LLM code generation era, how many people would I need and how long would it take them? If these tools worked the way their biggest enthusiasts seem to claim, the answer would be "not very many people" and "a matter of weeks." I suspect the actual answer is that it would take slightly fewer people and a slightly shorter amount of time than before LLMs, and you'd run up a huge token bill in the process. But watch this space, I guess.

There are still a bunch of unanswered questions, like: if you need senior developers in order to code safely with LLMs, and LLMs eliminate junior developer positions, where are you going to get the next generation of senior developers from? But it's also not clear that companies actually are eliminating junior developer positions.

And if bad employees equipped with LLMs can do more damage in the future, will that make hiring managers even more gunshy than they are now? We as a society still haven't figured out a reliable way of sorting the job applicants who would be good at a job from the ones who wouldn't, even though the number of interviews candidates have to attend to get a permanent position is increasing faster than the number of blades in a shaving razor. By this time next year, maybe a 50-interview hiring process will be a thing. More work for everyone! Not productive work but, you know, work.

But IF your team has no bad programmers on it and IF everyone uses the tools responsibly and IF senior management doesn't start thinking experience is unnecessary and IF hackers having access to these tools doesn't lead to an ever-escalating infosec arms race and IF about a dozen other things go right then, sure, the technology might lead to a sustained increase in productivity over what we had before.

None of the above is meant as any kind of conclusion, it's just a set of thoughts on a developing issue that I decided to try having in public. If you stuck with me this far, thank you for reading and I hope you found it... generative.