Most fraud models are built to answer one question: will this transaction go bad? Ours must answer a harder one first: will this customer even take the instant funds offer?
Instant Funds Availability only makes money when a customer adopts it, basically electing to receive their deposited funds early. That single fact reshapes almost every modeling decision we make downstream: what data we use, what our target variable represents, how we tune a model, and most critically, how we validate that swapping in a new model won’t quietly erode revenue even while every accuracy metric looks fine. Here’s how we approach that challenge end to end, from data intake through live model replacement.
A standard fraud model asks whether a transaction is bad. We have to ask that question not only for the whole population, but more importantly for the population that actually generates our revenue and risk: adopters. Every modeling decision below traces back to that distinction.
For a new client, the starting point is simple in principle and hard in practice: get as much history as possible, over as long a timeframe as possible, to capture seasonality. In practice, a full year of transaction data from a mid-size or large client can be too large to train on directly without a prohibitively long training cycle.
Our answer is a down-sampling strategy rather than a blunt data cut: keep as much data as possible for seasonality, but sample only a fraction of the “good” (non-return) transactions, while keeping all the target-relevant events in the sample. We’ve tested a wide range of down-sample levels, from very lean samples up through roughly half the data, to find the point where model quality stops improving relative to training cost.
There’s no universal winning ratio, and I don’t think there ever will be. It really comes down to the size of a client’s data, how much history we have to work with, and their ratio of positive to negative transactions. Every client has a different makeup that drives different performances, so we tune this fresh each time and confirm it against an out-of-sample holdout.
For brand-new clients without enough history to support that approach, we test it a couple of ways: first we’ll train on the full dataset the client can give us, with no down-sampling, to see how predictive it is on its own. In parallel, we have the ability to run a consortium model built from data across similar clients and compare the two to see which performs better.
Because IFA only monetizes adopters, our modeling target isn’t simply “will this item return.” A non-adopter’s return is a different risk problem entirely. Our practical target must prioritize returns from adopters specifically, otherwise the model is optimizing against noise the product will never actually be exposed to. This is one of the more actionable takeaways here for any team building models around adoption-gated or opt-in financial products generally: scope the target to the population that generates revenue and risk, not always the full population.
Once the target is defined, we can test differential weighting and a multitude of model types and hyperparameter adjustments depending on what a given client’s loss profile looks like. If a client’s dominant exposure is fraud-driven return types rather than standard NSF (non-sufficient funds) returns, we can weight the model to emphasize fraud signal, effectively telling it, “get this category right even if it costs precision elsewhere,” while leaving NSF returns at a baseline weighting. Fraud versus NSF is just one example; in practice there’s a long list of weights we can consider or test utilizing any of our client’s features. These are typically a manual judgment call, aimed at whatever pain point is most prevalent for a given client and rarely a one-size-fits-all setting.
Hyperparameter adjustments are also a critical part of maximizing our model performance. With built-in optimizers, we can test a large variation of these settings, effectively learning along the way to narrow down to the optimal parameters for that specific client.
Not every feature a client provides is useful to a given model. In some cases, a meaningful share of available features contribute no lift at all. So, model development includes systematic feature-set testing: we start by feeding in as many features and interactions as possible, then retrain with reduced sets to identify which combination a specific client will benefit from.
Different clients also show materially unique loss patterns, driven by differences in the fraud typologies and bad-actor behavior each institution’s customer base attracts. Rather than dismissing it as random variance, we build engineered feature interactions, ratios and composites of existing statistics that can often outperform any single raw feature on its own. By utilizing additional Python packages, we can test hundreds of mathematical variations to help find those important interactions that may not have been discovered yet or just weren’t as valuable in other modeling scenarios.
Our prior workflow, built on an off-the-shelf AutoML platform, was clean but relatively rigid: feed it data, get back a fixed number of trained candidate models and a recommendation, with limited ability to inject custom logic mid-training. Testing a different down-sampling strategy meant starting a new project from scratch every time.
Moving our core training workflows onto a code-first structure has let us embed those iterative steps directly into an automated training pipeline instead of manually restarting for each variant. A good example is a newly built automated resampling process: we can feed the model a complete dataset and let it cycle through different sampling balances on its own to find where the most accurate models land — work that used to mean manually testing each sampled dataset as a separate project. The real benefit isn’t raw compute speed so much as workflow flexibility: more experimentation per model, less manual re-running, and outcomes logged for comparison instead of re-derived by hand.
Perhaps the least visible but most consequential piece of our process is a secondary model that never goes to production: an internal-only adoption model, built specifically to estimate the likelihood that a given customer will accept the IFA offer.
This model exists to solve one specific validation problem. When a new candidate model is about to replace one we already have live, a champion/challenger swap, we don’t just compare aggregate accuracy metrics. We also isolate the population where the old and new models actually disagree: items the old model would have declined but the new one would pass, and vice versa. We then apply the adoption model specifically to that disagreement population to estimate whether the swap is actually going to net positive revenue and acceptable fraud exposure before deployment. It avoids discovering after the fact that a model with strong aggregate metrics is quietly swapping in a disproportionate number of non-adopters or higher-fraud customers.
The adoption model itself is trained only on transactions where a customer was actually given the option to adopt, which makes up the large majority of the transactions our IFA models decision and gives us a diverse sample to work with. For a brand-new client without that history yet, we lean on consortium data compiled from trends across other clients to guide those early predictions. The final test of the adoption model’s own accuracy waits until after a new model is live. At that point we can effectively reverse engineer our original swap analysis, apply it to the population that was actually decisioned by the new model, and check the adoption model’s scores against what really happened, validating both the IFA model and the adoption model together.
This two-model validation step is, in our view, one of the more differentiating parts of our process relative to a simpler “pick the model with the best accuracy” approach. Without these critical steps, the model can look strong in aggregate and still be a net-negative swap once adoption behavior is accounted for.
If you asked me to boil down how we decide whether a new model is safe to deploy, it comes down to this sequence:
This varies a lot depending on where a client is in their lifecycle. Newer clients can fall into a monthly or quarterly refresh cadence until their stats season and the model stabilizes. Established client models get refreshed on a case-by-case basis. We regularly review them for performance, accuracy, new volume trends, and loss trends that warrant a full retrain and test cycle. More often than you might expect, even after a full retrain, we find the live model is still outperforming any challenger. A lot of that comes down to our ability to layer controls strategically within our fraud control tool, essentially fine-tuning areas of concern by combining the rules-based layer, the ML model, and statistical features without needing to ship a brand-new model at all.
Production decisioning isn’t the ML model alone. A rules-based layer runs ahead of the machine learning model, and results from both, combined with statistical features, feed into a fraud control tool that lets us tune enforcement granularity: broad, portfolio-wide adjustments when fraud is diffuse, or narrow, geographically or institution-specific tuning when fraud is concentrated in a specific region or account segment.
We consider this layered architecture one of the more meaningful points of differentiation in our approach. It’s the combination of these layers, more than any single piece, that lets us respond to fraud patterns with precision and speed rather than blunt, portfolio-wide changes.
Generative AI tooling is currently less a modeling technique in itself for us and more an accelerant for the engineering work around models: scaffolding automation code, maximizing optimization, and helping troubleshoot failures in long-running training jobs, like memory-related errors that are otherwise time consuming to trace through complex pipeline code. It’s become a standard part of my debugging workflow at this point. We often run the same issue through more than one AI tool to compare suggestions and land on the fastest, most accurate fix. We’d still call this early-stage for us: meaningful time savings already, with more application areas still being explored.
The throughline across data strategy, target design, model tuning, feature engineering, and swap validation is the same: an adoption-gated product requires validating not just whether this is a good fraud model, but whether it’s a good fraud model for the population that will generate revenue. That distinction, and the extra validation step it necessitates before any production swap, is arguably the most transferable insight here for other teams building models around opt-in financial products.