Good morning and welcome... Today is Sunday, October the eleventh, 2026, and you are listening to Not a Llama. It is paper day, so instead of the news, I explain one new research paper. Today it is called LoGRA, a method from NVIDIA — that scales reinforcement learning on large language models by compressing gradients. The authors are Shaokun Zhang and colleagues. It is worth a listen if you have ever hit a memory wall trying to train a bigger model.

So let's start with the problem this paper is trying to solve. Reinforcement learning has become one of the best ways to improve large language models. You feed the model feedback on its own answers, and it learns to do better. But there is a catch — this kind of training eats up enormous amounts of memory, and that memory limit decides who gets to use it at all.

The trouble goes well beyond simply storing the model itself. When a computer trains, it also has to hold the gradients — the numbers that point the way to fixing mistakes — plus the optimizer states that help the algorithm adapt. Together these can take several times more space than the weights alone. To put a number on it, a seven billion parameter model needs about fourteen gigabytes just for its weights, while the optimizer adds roughly five times that. So a machine powerful enough to run a model may still be far too small to improve it. That is the wall the researchers are trying to break through.

So how do they actually get past that wall? The team behind the new paper, called LoGRA, had a fairly simple idea — instead of saving the full picture of every gradient, they keep just a compressed sketch of it.

Think of it like shipping a large painting. You could send the whole thing, carefully rolled in a wide tube, taking up the entire trunk. Or you could take a quick photo of it, send that tiny file, and let the person on the other side reconstruct the gist. The sketch is not a perfect copy, but it carries the important directions.

The intuition here is that learning from rewards simply does not need the whole detailed picture. Each answer usually gets just one pass or fail signal, so a compact representation should be enough to learn from. And they use these small sketches for two jobs at once — they update the model, and they also copy the current version of the model across many computers, which trims both memory and the traffic between machines.

The trick is that they keep the sketch deliberately small. They pick a low rank, which is just a fancy way of saying they keep only a handful of the most important directions out of thousands. For a large matrix, that shrinks the storage needed from about sixty-four megabytes down to a single megabyte — a 64-to-1 reduction.

Then, to keep training from going haywire, they add a safety mechanism. Before applying each update, they estimate how much it would change the model's behavior, and they scale it to fit a preset budget. That way the model keeps improving without jumping so far that it destabilizes everything.

So let's look under the hood... and see how this actually works.

The core move is to replace full gradients with something much smaller. The researchers call these low-rank sketches. Rank is just how many independent directions the compression keeps. A lower rank means more squeezing. They pick a small number, something like sixty-four directions out of thousands, and project the gradient into that smaller space using a fixed random matrix. The effect is dramatic. For a large matrix, storage drops from about sixty-four megabytes down to a single megabyte.

Once compressed, the update gets merged straight into the model's weights. That is the whole loop in its simplest form.

But a small update can be a reckless one. If the model changes too much in one step, it can wreck everything it already learned. So the team adds a safety valve. Before applying each update, they estimate how much it would shift the model's behavior. They measure this using something called KL divergence, which is just a way of saying how differently two probability distributions behave. Here it tells you how differently the model would answer.

They take that estimate and scale the update to fit a preset budget, a limit on how much the model is allowed to change per step. This follows an old idea from reinforcement learning called the trust region. The point is simple. If you keep each step small, the local guesses about improvement actually hold up, instead of blowing up mid-training.

So the whole picture comes together in three pieces. First, compress the gradient into a tiny sketch. Second, fold that sketch into the weights. Third, check the size of the step against a budget before you commit to it.

That is the mechanism in a nutshell. It is elegant because the same small sketch does double duty. It updates the model, and it also carries the update across many machines when the team needs to sync up. That cuts memory and traffic at the same time.

The result is that a method that once needed hardware too large for most people can now run where it mattered most. The researchers were able to train a model far bigger than the standard approach allowed, and do it without it collapsing. But before we get to those numbers, there is one important caveat worth holding in mind.

Well, let's get to those numbers, and there are some good ones here. The team tested their method across reasoning tasks and compared it against the standard approach, the dense Adam method. At the smaller model, the one with a billion and a half parameters, they trimmed average training memory by about a fifth, dropping from around nine gigabytes down to just over seven. At the seven billion parameter model, the savings were far bigger. They cut memory by nearly half, from about thirty-two gigabytes down to seventeen.

But the real story is not just about saving space — it is about what that freed up made possible. On the smaller model, accuracy actually improved, with the peak score climbing from around sixty-four percent to nearly sixty-eight. On the bigger model, performance held steady, sitting at about seventy-two percent either way.

And then came the moment that really matters. The researchers were able to train a model with twenty-seven billion parameters for over a thousand steps on a single machine — the standard method simply could not do this. It ran out of memory before it could complete even a single update. With the new approach, the model's score climbed from about forty percent to a peak of sixty-five percent, and it held there without collapsing.

So the method turns what was once impossible on a single machine into something that actually works. But before you get too excited, there is one important caveat. This is not a magic bullet. The paper itself admits that some of their estimates are rough — and the comparison against a different compression technique did not show any clear edge. So take these numbers with a grain of salt.

So where does that leave us? Let's talk about what this actually means for the people who build with models or run them day to day.

The headline is real. LoGRA offers a practical way to lower the wall that has kept reinforcement learning out of reach — it shrinks the memory needed to train, which lets bigger models run on hardware that was previously too small. And it trims the traffic between machines when you sync up. That is a genuine step forward.

But there are honest limits to hold in mind. The safety mechanism relies on an estimate of how much a step will change the model. That estimate is local, so the actual change can drift from what was predicted. The team also admits they could not pin down exactly why some of their choices help — a bit of a letdown. Their test against a different compression method showed no clear edge in accuracy or speed.

And the experiments only cover mathematical reasoning. We do not yet know if this works as well for other kinds of work.

So who benefits most? Anyone whose training keeps stalling because of memory. For them, this turns the impossible into something feasible on a single machine. If memory is not your bottleneck, the rough edges may not bother you, but neither will the savings. It is a solid tool for a specific job — not a magic fix. Take the numbers with a little salt, and weigh the approximations against what you actually need to run.

So LoGRA shows that compressing gradients into low-rank sketches can trim training memory and let bigger models learn through reinforcement learning on the same hardware... Next up, watch whether the results hold up beyond math puzzles. Thank you for listening, and the news returns tomorrow morning.