I don’t choose which model runs which task anymore. My agent does it, on a schedule, and my job is to read the recommendation and say yes or no.
The agent is Hermes, from Nous Research, running on my own machine. It’s what I hand things to: research, code, the multi-step work that would otherwise sit in an open tab for a week.
The problem was never picking wrong
It was not revisiting. You choose a model once, write it into a config, and then nothing ever prompts you to look again. Meanwhile something ships every few weeks, prices move, and the thing that was the obvious pick for a subagent quietly stops being it.
Nobody sends you a notification about that. The config just gets a little more wrong every month, and you find out when something feels slow or a bill looks odd.
What runs on Saturday
Every Saturday at 9am, a self-evaluation. Four steps, and none of them are clever:
- Research which models people are actually running right now. Release notes, forums, the places where practitioners complain. Not benchmark tables.
- Pull the current numbers: pricing per million tokens, context window, rate limits.
- Compare that against what my config is running today, for the main agent and for subagents separately.
- Write a short recommendation, with the cost-benefit spelled out.
Splitting the main agent from the subagents is the part I didn’t think about when I set this up. They want different things. The main loop is where judgment lives, so it gets the expensive model and I stop arguing about it. Subagents doing mechanical fan-out, twenty of them at once, want cheap and fast, and the quality bar is “did it come back with the right file.”
Same task, two answers. Before this ran on a schedule I was using one model for both, which is the kind of thing you only notice when something forces you to look.
It recommends, I decide
Hermes doesn’t edit the config. It writes a report and stops.
That’s deliberate. It’s the same rule I use with agent PRs at work: whoever merges owns every line, and “the agent chose it” is not an answer. The report takes me a few minutes on Saturday morning. Most weeks I skim it and change nothing.
It won’t find the best model
Community signal is noisy and it lags. Half of what shows up is someone’s first impression from a weekend of use. A model that reads great in a thread can be wrong for how I actually work.
That’s fine, because optimal was never the bar. The bar is not being six months stale, and a rough answer every week clears it easily. I’d guess parts of my setup are still a bit off. They’re less off than when I only checked after something annoyed me.
The general shape is worth stealing even if you never touch Hermes: take the decision you keep meaning to revisit, put it on a recurring job, and let the reminder be a report instead of a nagging feeling.