Inside Tink: getting top model performance for Income Check
We get into the backstage of Tink tech to learn more about how our teams are tackling challenges and continuously upgrading our solutions – and ways of thinking. Here, Data Scientist Ida Janér explores why income verification is all about top-performing models – and how we improved accuracy by embracing confusion.
Tink’s Income Check allows businesses to instantly verify a person's income by analysing their transaction data.
For it to work well, having a high-performing classification model (with a high accuracy in machine learning terms) was fundamental.
Tinker Ida Janér explains how data scientists in the Risk team worked to improve the model performance, and the key learnings they took from the process.
By Ida Janér
Earlier this year we launched Income Check, which allows businesses to instantly verify a person's income with secure, real-time data directly from their bank account.
With the help of pattern recognition, incoming transactions to a user's bank account can be grouped and filtered by recurrence and other criteria to extract only the ones that constitute income streams. Each stream is classified and labelled with the type of either salary, pension, benefits, cash deposit or other.
When we launched Income Check we knew from the outset how important it would be to build a high-performing model, with a high accuracy level in machine learning terms.
But even more important would be to put in place a sustainable and scalable way to reach that level, and stay there. So the data scientists in the Risk team got to work, and now Income Check is at 99% accuracy in Sweden. (Meaning in 99% of cases the model understands whether a transaction should be viewed as an income or not.)
So what did it take to get up to that 99% accuracy level?
Building the model
The first step is to observe incoming transactions from real transaction statements and extract keywords from transaction descriptions.
To help us understand which among the transactions actually are an income (and should be considered as such in the context of affordability checks for example), we first analyse the transactions for parameters such as date, frequency, and stability.
Based on this initial analysis we train a model to predict whether a given transaction should be viewed as income or not. From then on it’s all about measuring and improving the performance of the model – and in particular its accuracy – when it’s really put into action.
Improving the model
If the last 18 months have taught us anything it’s that circumstances can change quickly. The world is constantly evolving, and so any data model that relies on real world data must be constantly retrained and updated.
Any retraining of the model based on changes to the data and current climate (such as changes in the payout pattern from a pension fund company, for instance) then needs to be validated to ensure they deliver an increase in accuracy, so we always provide the best possible proven model to our customers.
And naturally the changes, and the models, become increasingly tailored to a particular market behaviour as our local knowledge grows.
To reach top model levels of success, embrace confusion
A confusion matrix is a summary of prediction results on a classification problem. The number of correct and incorrect predictions are summarised with count values and broken down by each class. This is the key to the confusion matrix.
The confusion matrix shows the ways in which your classification model is confused when it makes predictions. It gives you insight not only into the errors being made by your classifier but more importantly, the types of errors that are being made.
True positives (TP): transactions that the model correctly predicts as income
True negatives (TN): transactions correctly predicted as not being income
False positives (FP): transactions incorrectly predicted as income
False negatives (FN): transactions incorrectly predicted as not income
Accuracy is the share of correct predictions.
Precision tells us how certain we are that the transactions predicted as income actually are income.
Recall tells us how good the model is at identifying all transactions belonging to income.
Optimising accuracy – a case study
The team took two different markets (Finland and Sweden) over a six month period and reviewed over 25,000 cases where a transaction history had been analysed and income streams were identified and provided by our Income Check product.
Here are some examples of what our model re-training phase consists of.
Reviewing new data
The new data is reviewed based on some predetermined definitions of income. Part of the data is used for model training, and part is used for model performance evaluation. Splitting the data in this way is important to get an unbiased review of the actual model performance
Revisiting model parameters
We revisit parameters such as keywords and thresholds (setting minimum and maximum cutoffs for different transaction metrics), against which we measure numerical values belonging to the transactions (such as amount, number of occurrences and periodicity).
Analysing specific market behaviour
Since each market is unique, we also need to tailor the model to capture market-specific behaviours. This goes beyond just language differences – different markets also come with specific patterns in payouts for different types of income. This sort of market-specific tuning is done in collaboration with local representatives who can provide a deep local domain knowledge.
Taking this approach and applying a review based on the confusion matrix helped us understand how good our model was in each country in terms of accuracy, precision and recall.
Just going by ‘did we identify an income?’ was no use if it turned out that what we thought was an income actually wasn’t, or if there were income streams we did not spot at all. Not everyone’s income is a stable monthly salary.
Here’s a look at how performance improved in that period.
In both markets we saw that analysing performance and retraining accordingly for each class in the confusion matrix saw equivalent gains for both precision and recall.
This means that the model not only classified more transactions as income, but it was also certain that those transactions actually were income. This then is reflected in the significant increase in accuracy in Finland (due to diminishing returns, it was harder to improve from the already high accuracy levels in Sweden).
We improved the salary and benefit classes, as well as being able to look at inconsistencies in periodicity and amounts in Finland, while in Sweden improvements were reached mainly due to us improving pension and benefit intelligence.
But the key learning was how different the approach needed to be in each country in order to achieve the improvements to the model.
What to improve next?
We have talked about why constantly reviewing and retraining a model is necessary in an ever-changing world, especially when they are informing potentially life-changing affordability decisions. For us, success isn’t just about returning a value – but making sure you remove all confusion so you can feel 100% confident with the results.
Since beginning this quality improvement process for Income Check, we won’t stop at Sweden and Finland. Each market and every customer can expect the same approach and diligent focus on improving accuracy – because we simply wouldn’t accept anything else.
More in Product
2024-09-18
14 min read
Connecting the dots: how UX optimisations are driving success rates
We’ve previously explored small tweaks that get big results in open banking conversion rates. This deep dive drills further into how to reduce friction – and improve success rates through a fresh round of incremental changes in our UX.
Read more
2024-08-05
5 min read
Feature update: Savings Goals elevates your money management experience
Reaching financial goals can be daunting – so we’ve updated Savings Goals, a feature of Tink Money Manager designed to help banks empower customers to proactively save and achieve financial wellness.
Read more
2024-07-17
2 min read
Banking on engagement
This Tink white paper introduces new consumer and retail banking executive research from key European markets, setting the scene for banks to take the next step with Personal Finance Management (PFM).
Read more
Get started with Tink
Contact our team to learn more about what we can help you build – or create an account to get started right away.
