What Seven Years of AutoML Taught Us About Going Code-First
Nick Ledford, Chief Risk Officer, VALID Systems
We ran our fraud and returned item models on a commercial AutoML platform for seven years. Last year we moved that work to Snowflake and rebuilt it in code. The first weeks felt overwhelming.
After a lot of hard work and many development sessions, our new models are outperforming the ones they replaced. The reason is not that we found a better algorithm. It is that we now control the parts of the modeling process the platform used to decide for us.
Here is what we gained, what it cost, and what we are still working out.
The appeal of AutoML is straightforward. It removes friction. You bring the data. The platform handles feature engineering, model selection, and tuning. You get a deployable model without writing the code underneath it.
That tradeoff served us for a long time. Cost was the first reason we looked elsewhere. At that point we had not yet mapped out everything a switch would let us do.
We tested before we committed. In the year before the cutover we ran a limited proof of concept with a narrow goal: confirm that a model we built ourselves in Snowflake could match what the AutoML platform produced. Once we had that parity, we committed to the full migration.
Snowflake is code-first. Every step from data preparation through model tuning has to be written. Nothing is generated for you.
The flexibility has been worth the added complexity, but the full picture only came into focus after we had spent real time building there. The switch gave us a level of control over model optimization that the AutoML platform never exposed.
That control matters most in a domain like ours. Fraud data is heavily imbalanced. A wrong model is expensive in both directions. A missed loss costs money. So does a good customer negatively impacted by a bad decision.
Losing a point-and-click platform did not mean losing an interface.
We use Snowflake Streamlit apps as the UI layer for our modeling work. The app starts new modeling projects. It is also the hub where we review parameters and performance across every run in a project.
Streamlit gave back the workflow convenience we lost. The modeling logic stays in our own code instead of inside a vendor platform.
Only a small share of check deposits are returned unpaid or fraudulent. In some portfolios the fraud rate is a fraction of a percent of all deposits. That imbalance shapes everything about how these models get built.
Once the platform constraints came off, we built out three techniques we had wanted for years. All three are judged against a custom metric rather than standard evaluation metrics that only tell part of the story.
Class balance testing. Non-fraud transactions vastly outnumber fraud transactions, and much of that majority class is redundant. A model does not need every example of “not fraud” to perform well. Class balance testing reduces the majority class and tests different ratios to find the one that produces the best model. We could run this kind of test before, but each test meant pulling a separate dataset and creating multiple modeling projects. Now it is part of the workflow.
Hyperparameter optimization. Hyperparameters are the settings that govern how a model builds its decision structure, such as how deep a tree grows and how splits are evaluated. On the AutoML platform this was effectively closed. We could see the parameters it landed on. We could not control how many tests it ran or what sample size it used. We went back and re-tuned older models with our own process, and we found better configurations than the platform had ever surfaced. That told us we had been working inside its tuning limits longer than we realized.
Feature reduction testing. This step finds features that are unimportant or redundant with other features, then tests whether removing them improves the model. Simpler models are easier to explain, easier to validate, and easier to support. At times they perform better as well.
A metric in dollars, not line items. The change with the least visibility has had the most impact, and it is what we optimize against. Standard classification metrics count items: transactions flagged correctly or incorrectly. We optimize against a custom metric that measures performance in dollars of risk instead. Every technique above is tuned against that metric. A model that gets the highest-dollar cases right is worth more to us than one that maximizes the count of correct calls.
Every new model is outperforming the version it replaced. On some models we are seeing up to a 15% improvement in returned item loss detected.
That is an early number, not a final one. We have more optimization ideas in the pipeline.
We built the new development process with model risk management in mind from the start.
There is no separate tool doing this. The notebooks are designed to do it. Each model version scores its out-of-time test population and writes those scores to a Snowflake table stamped with that exact model version that is registered within Snowflake. The validation performance snapshots are generated from those score files.
The practical effect is that documentation is a byproduct of modeling rather than a separate exercise after the fact. Of everything this migration changed, that is the one our team notices most.
More flexibility means more to manage, and we have started adjusting the team in response.
We recently moved one team member off day-to-day modeling and monitoring into a dedicated role covering two things: the health of the models and servers we run in production, and the pipeline of ongoing model improvements. We chose someone who picks up new tools without hesitating.
Under the old platform our work settled into a pattern. Once a client relationship matured and a model had enough data behind it, we rarely retrained. Retraining happened in response to a real shift in fraud trends, or to an acquisition. Most of the job was monitoring.
That is changing. As attrition modeling and other machine learning products come online, the expanding scope is what drives future hiring, not the fraud modeling workload on its own.
One further idea grew out of this migration.
The rules-based layer of our fraud decisioning sits alongside the machine learning model. It runs through a legacy interface that predates our current stack and that we cannot easily change on our own. Now that our modeling lives in Snowflake, we have started asking why that interface and the data behind it are not there too.
The idea on the table is to rebuild it in Streamlit, as a separate application from the modeling hub. That would let us change rules and workflows on our own schedule. It is still an idea. The discussion continues once the model migration is complete.
AI tools are now a daily part of how we build and maintain these models. We would not have reached our current architecture without them.
We use Claude for framework and architecture decisions, with Snowflake's built-in AI covering certain cases. Our use has matured. Early on it skewed toward basic tasks. Now we use it to diagnose model and notebook errors, and to work out how a one-off fix should fold back into the shared codebase so it holds for everyone rather than for the one case that surfaced it.
Code-first is not a universal recommendation, and I would push back on anyone who tells you it is. Here is roughly how I would frame the decision for another team.
Choose code-first if:
Choose AutoML if:
Our own model validation work is what forced us into the depth that code-first assumes going in. Skipping that foundation and jumping straight to code is an easy way for this transition to go wrong.
Cataloging what the process generates. A code-first process produces a large volume of data and metadata about every run, and that volume is getting harder to catalog and reuse. We want to build a self-improvement loop that mines it for patterns: what separates our best models from the rest, and whether that can become a better starting blueprint for the next one. The work has started. It is not built out.
Scoping the rules interface rebuild. We know we want it. We have not written detailed requirements, scoped the timeline, the owner, or the budget.
Trading seven years on AutoML for a code-first environment meant giving up a faster path to a working model. What we got back is control over the decisions that actually determine model quality: how classes are balanced, how hyperparameters are tuned, which features earn their place, and what the model is optimized to get right.
Measured in our own terms, in dollars of risk rather than item counts, that control is already paying off.
For a team earlier in this work, the takeaway is not that code-first is always the right call. It is that the right platform depends on how much foundation you have already built.
A Question for You
What decisions is your modeling platform making for you—and which of those decisions would you want to control?