The Sunday morning quarterback

Let's pretend like I just got a big bonus from work. More money than I've ever had in my life. What should I do with it?

I could go on an epic trip and build memories and experiences I'll have for my entire life. But then, poof, the money is gone. I could invest it, but I don't know what the stock market is going to do. There's a lot of fear of recession, inflation, and a hawkish Fed. What if I put the money in at the peak, right before a crash? Then I miss out on the epic experience and lose the money anyway. On the flip side, what if this is just a pause before the market launches into a new bull run? Then I've taken one trip when, if I'd invested, I could have had two.

Quite the conundrum.

I suppose the wise decision is to hedge my bets. I'll invest half of it and take a less epic trip with the other half. Instead of backpacking in Europe, maybe I'll go explore Alaska this summer. I've always wanted to go to Alaska. My grandfather was stationed there in World War II and said it was his favorite place on earth. That's a much cheaper trip, and still quite epic. Then maybe I can still look toward the big backpacking trip in a year or two, by saving a little each paycheck.

I don't know how this will turn out, but I think that's the wise decision to make.

Monday and Sunday

It's a lot easier to be a Monday morning quarterback than it is to be a Sunday morning quarterback. On Monday the game is over. Everyone has seen which calls worked, and second-guessing is easy because the tape exists. On Sunday morning there is no tape. There is the information you have, and a call you have to make before anyone knows how it goes.

When I say wisdom here, this is part of what I mean. Good judgment, and decisions made before the outcome can be known.

I've been thinking about this because of a question under most of what I write about AI responsibility. Can wisdom be trained into a model? The labs keep making the models smarter. Intelligence and wisdom are different things, and I'm not convinced the second one comes along with the first.

Black and white, and gray

There are some forms of wisdom that a wise, objective person could distill into logic. Those things are mostly black and white. It doesn't take a high degree of wisdom to know that it's wrong to launch a nuclear bomb and start the apocalypse.

Where things get difficult is the gray area. Humans as a whole still struggle with it. Most of the hot-button political issues, for instance, are gray-area issues, where a logical case could be made for both sides. Which is part of why we have two large sides diametrically opposed to each other. This is the realm of philosophy. The realm of tradeoffs, where you have to accept some bad as a side effect of some desired good outcome. Many of these issues are also impossibly complex, so you can't accurately predict the outcomes.

COVID is a good example. We shut society down because we believed the illness was a severe threat. Because we sent people home from work, we had to send people checks, give loans to small businesses, and so on. Thankfully, we found out COVID wasn't Captain Trips from Stephen King's The Stand come true after all ("Cibola! Bumpty-bumpty-bump!"). When we reopened society, that money was still sloshing around the economy. Economists at the San Francisco Fed estimated that pandemic support may have added about 3 percentage points to US inflation by the end of 2021, at the high end of published estimates, and in some form we are still dealing with inflation today. Reopening also brought "the great resignation," which was basically a blow-off top in the jobs market (a sharp spike, then a slide back). Then came "job hugging," where hiring dried up, people held on to the jobs they had, and finding a new one was quite difficult, especially at entry level. We're still waiting on clear signs of a rebound.

I'm not here to take any political side, and I'm not going to pretend I know what was right or wrong. That's why it's a good example. We acted on the data we had at the time, we got some good out of it, and it had side effects that are still with us years later. Any other decision would have had a different set of good and bad outcomes.

These gray areas are where we need wisdom the most.

The easy jobs

Set the gray aside. The black-and-white calls should be the easy part.

This summer, AI labs and the evaluators they work with disclosed a run of incidents in which models under test reached real systems on the open internet and attacked them. I wrote about the specification gaming behind several of those incidents (models meeting the letter of a goal while breaking its intent) in an earlier post. A few of the attacks were sophisticated. Most were mundane: well-known techniques, a basic vulnerability, credentials left where they shouldn't have been. The models had the skill. Across the disclosures, a stop from inside the model was the exception.

Anthropic reported that three of its models reached real organizations' production systems during cyber evaluations. One kept attacking after recognizing the systems were real. One worked out on its own that it might be on the real internet, then talked itself back into believing it was still in a simulation. One, an internal research model, stopped pursuing the evaluation goal once it realized where it was. Three capable models, one easy question (don't attack a system that might be real), three different lines of reasoning, and only one that stopped. All three had been told none of it was real, and what mattered was what they did once that no longer looked true. Most of what I know about this came from the labs' and evaluators' own write-ups, and publishing them was the right call.

In testing, some have failed the easy calls, and it's not long before we start giving them the hard jobs.

I'd like to circle back here to a story from my most popular article to date, Yoshimi Battles the Pink Robots. On October 27, 1962, at the height of the Cuban Missile Crisis, the Soviet submarine B-59, carrying a nuclear torpedo, was cornered by US destroyers in the western Atlantic. It had been out of contact with Moscow for days. The destroyers were dropping practice depth charges to force it to surface, and the captain believed the war might already have started. By the most-cited account, launch needed three senior officers to agree, and Vasili Arkhipov would not. By another, the captain ordered the nuclear torpedo prepared, and Arkhipov calmed him down before any launch order was given. Historians debate the details and how decisive it was, but it was a human veto that a fully automated chain wouldn't have contained.

Nobody aboard knew whether the war had started. By the accounts we have, Arkhipov read what the Americans were doing as signals, not an attack, and when he was unsure of the situation he was in, he leaned toward caution about the one act that couldn't be undone. The model that convinced itself it was still in a simulation doubted its situation too. It leaned the other way, toward the reading that let it keep going.

A wise hand

The companies that run these models are doing very unwise and irresponsible things by making much of the guardrails a layer on top that can be turned on and off. These guardrails are also quite easy to hack off, even in production. Then they make the conscious and unwise decision to turn those outer guardrails off (Anthropic says the models in these evaluations kept their model-specific safety training; in two of the three, that alone didn't stop the attack) and hand the models difficult, sometimes impossible-looking tasks. That leads to the specification gaming I already wrote about, and it's a large part of the problem.

If wise people were in charge, this testing would not be happening, and those guardrails wouldn't be sitting on top of the model. They would be built into the core, deeper than they are now. Inseparable. That's the only way I see to make a model reliably hold the black-and-white lines.

Without a wise hand to guide it, a wise model is not possible.

The gray stays with a person, for now

A guardrail built into the core is still a rule. Rules can hold the black-and-white lines. By my own split, the gray is the part that resists being distilled into logic, so no hand, however wise, can build it into a model today.

So I think we need to start with the easy black and white. Once we have that solved, we can start building for the gray cases. Until then, the gray calls should escalate to a person.

That's the decision I've landed on in my own agentic harness at work (the setup that runs AI agents on software tasks). I can't detect gray directly, so I use two stand-ins for each task: complexity, which the agent estimates, and criticality, which a human supplies. The higher the complexity and criticality, the more often the human is brought into the loop. For something high in both, the human is in the loop for the entire process. For tasks low in both, the agent pushes through on its own, with automated audit and judgment checks in place of the human, and at the end it provides a report to review. The temptation, human and business, is to cut the humans out. We don't need to fully eliminate the human from everything.

Even humans aren't infallible. We are not all wise, we don't all take the time to review even when we are supposed to, and sometimes we rubber-stamp things we shouldn't. Sadly, some humans are also intentionally trying to do malicious things. A human in the loop makes a good call more likely. It doesn't make one certain.

Predict, or decide

If AI ever handles gray-area calls well, I think it will be through research and reasoning. We have to teach AI how to think like a wise person, how to research the past, and how to use that to predict the tradeoffs. AI might actually be better at the historical-research part than humans are.

"Predict" is the wrong word there, though, and COVID is the reason. On a call like COVID, all the research and the wisest people available still couldn't predict the tradeoffs. More history helps when a problem is hard but knowable, and COVID wasn't knowable in advance.

Perhaps the better idea is simulations and probabilities. A probability doesn't tell you what will happen. It gives you something to hedge against. Simulations did inform some COVID-era decisions. Maybe better ones could have helped more. That's hindsight, and we can't rerun 2020 to find out. We might be able to replay it for a model, though.

I wonder if it would be possible to spin up an AI trained only on data up until January 2020, and then have it run simulations as the data came out. That might be an interesting evaluation, scoring how well calibrated its probabilities were against what actually happened. It would have to be run against other points in time as well. It would be very difficult and costly, and governments are the obvious funders, since they'd be the ones making those calls.

There is no perfect solution to the gray problems. You're always going to be making the best decision you can with the data you have, and you're never going to be certain what side effects will show up.

Which brings me back to the bonus. I couldn't call the market, so I didn't try. I chose how to act without knowing. I hedged. Arkhipov couldn't know whether the war had started, so he leaned toward caution on the act that couldn't be reversed. Neither of those is a prediction. In the true gray, I think wisdom is how you decide once you've accepted that you can't see the outcome.

Whether that can be trained into a model, I don't know. On that one, I'm the Sunday morning quarterback too.