Fine-Tuned an AI ~100x Smaller Than GPT, And It Won

I wanted to answer two fairly simple questions:

1. How practical is it to fine-tune an LLM yourself?

2. When does fine-tuning actually make more sense than just writing a better prompt?

So I picked a narrow problem: detecting which sentences in a document were written by a human and which were written by AI.

Then I fine-tuned Qwen3-4B-Instruct, a 4-billion-parameter model, on a single 16GB consumer GPU.

On my frozen 1,750-document benchmark, the tuned model ended up beating both the original Qwen model and GPT-5.6 Sol.

Not by a tiny amount either.

Before this turns into "4B models are better than frontier models": no.

This was a specific model trained for one narrow task, tested on a benchmark built for that task.

And that is exactly why the result is interesting.

The first problem wasn't training. It was data.

I started with four public datasets:

• RAID — human and AI-generated documents across different domains and generators.

• RoFT — documents where the transition from human to AI writing is known.

• M4GT — more mixed-generation examples across different domains and models.

• CoAuthor — real collaborative writing between humans and GPT, including documents that switch between human and AI multiple times.

Unfortunately, none of these datasets stored the problem in exactly the same format.

One might effectively say:

AI starts from sentence 4.

Another might contain labels at a different level.

Another might have several transitions.

So before training anything, I normalized everything into one representation:

sentences[] + labels[]

Every sentence had a corresponding Human/AI label.

This ended up being a much bigger part of the work than downloading a model and calling train().

The dataset had two problems

The first was class imbalance.

Initially, around 70% of all sentences were AI-labeled. One of the sources was more than 90% AI.

A model trained blindly on that distribution could get surprisingly decent accuracy by developing the sophisticated strategy of:

"Just say AI."

Very advanced.

Since I had enough data, I removed some examples and brought the AI ratio down to around 60%.

The second problem was more interesting.

Real documents containing several authorship switches, something like:

Human → Human → AI → Human → AI → AI → Human

made up only about 3.5% of the dataset.

And almost all of those examples came from one source.

That isn't great if mixed authorship is one of the things you actually care about.

So I created additional multi-switch documents by recombining existing real labeled sentences from the datasets.

Importantly, I wasn't asking an LLM to generate fake training text. The sentences were still real dataset samples. I was changing how they were combined.

That brought multi-switch coverage to roughly 20%.

Finally, the data was divided into:

• 81% training

• 9% validation

• 10% test

I also kept the poetry domain completely outside training to give the model at least one writing style it had never learned from directly.

Accuracy alone wasn't useful

Because around 60% of the dataset was AI, a model predicting AI constantly could report roughly 60% accuracy while being completely useless.

So my main metric became Macro F1.

It evaluates performance on both Human and AI classes independently and then gives them equal importance.

I also tracked a much harsher metric: exact-document accuracy.

For a document to count as correct, the model has to classify every sentence correctly.

Nine correct sentences out of ten?

Wrong document.

It is brutal, but very easy to understand.

First, establish a baseline

Before fine-tuning anything, I ran a random classifier.

It produced:

• Sentence accuracy: 0.501

• Exact-document accuracy: 0.036

• Macro F1: 0.490

Pretty much what we would expect.

Then I gave the untouched Qwen3-4B model a strong prompt with clear instructions and few-shot examples.

It reached:

• Sentence accuracy: 0.553

• Exact-document accuracy: 0.372

• Macro F1: 0.541

The full benchmark took around 6 hours locally.

Then came the frontier models.

GPT-5.6 Sol:

• Sentence accuracy: 0.800

• Exact-document accuracy: 0.641

• Macro F1: 0.795

• Around $7 for the benchmark using batching

GPT-5.6 Luna:

• Sentence accuracy: 0.755

• Exact-document accuracy: 0.606

• Macro F1: 0.751

• Around $0.75

So the target was clear.

A useful fine-tune didn't just need to improve Qwen.

It had to close a very large gap.

Fine-tuning a 4B model on 16GB VRAM

I used LoRA, with Qwen3-4B-Instruct running in BF16 without quantization.

The base model itself is easy enough to estimate:

4 billion parameters × 16 bits
= 64 billion bits
= roughly 8GB of weights

But full fine-tuning needs much more than the weights.

You also need memory for gradients, optimizer states, activations and other training overhead.

For a model around this size, the rough comparison looks like this:

Instead of changing the original weight matrices, LoRA learns much smaller adapter matrices.

For example, rather than training an entire 4096 × 4096 matrix, I used low-rank matrices with rank 16.

That reduced the number of trainable parameters enough to make the experiment practical on my RTX 5060 Ti 16GB.

Training took roughly 10 hours, with checkpoints so I could stop and continue rather than treating the GPU like it owed me money.

Three bugs nearly ruined the experiment

The model training was probably the least surprising part.

The infrastructure around it caused more problems.

1. The silent zero-loss bug

The training system needs to know which tokens belong to the assistant response so it knows what output to optimize.

The chat template I initially used did not provide the expected generation mask.

The scary part wasn't that it crashed.

It didn't.

Training could happily continue while learning effectively nothing.

A very productive-looking way to waste ten hours.

2. The evaluation prompt mismatch

At one point AI-F1 dropped to 0.0, while Human-F1 stayed around 0.69.

That looked less like a bad model and more like a broken experiment.

It turned out training and evaluation were silently using different system prompts.

Fixing the prompt fixed the result.

This is also why I don't trust a single metric printed at the end of a training script without checking what produced it.

3. A parser edge case

One validation document contained reference-list text that looked like my sentence markers.

The parser interpreted those as additional sentences and broke scoring.

The fix was simple: validate inputs and skip-and-log malformed cases instead of crashing the entire evaluation run.

Not an exciting machine-learning problem.

Still capable of invalidating the machine-learning experiment.

Then I ran the same benchmark again

The tuned 4B model scored:

• 0.913 sentence accuracy

• 0.823 exact-document accuracy

• 0.915 Macro F1

Against GPT-5.6 Sol:

• 0.800 sentence accuracy

• 0.641 exact-document accuracy

• 0.795 Macro F1

The strong-prompt version of the exact same Qwen model had only reached 0.541 Macro F1.

That difference is the part I care about most.

The base model already contained plenty of general language capability.

What it lacked was specialization for this particular decision boundary and output behavior.

Prompting helped a little.

Fine-tuning changed the task performance completely.

So when should you fine-tune?

I think the answer is much simpler than people make it.

Fine-tuning starts making sense when you have a task that is:

narrow, clearly measurable and repeated many times.

You pay the setup cost once, specialize a smaller model, and then potentially get:

• better task-specific accuracy,

• lower inference cost,

• local/private deployment,

• predictable output behavior,

• and no API charge every time the task runs.

But if the task is broad, changes constantly, or you're only going to run it occasionally, prompting a strong general model is usually the better engineering decision.

I wouldn't fine-tune a model just because I can.

The useful question is whether the task is stable enough, valuable enough and repeated enough to justify specialization.

In this experiment, it was.

And a 4B model running on hardware sitting next to my desk was enough.