ML monitoring should not be centered around data drift alone! There are two important problems with univariate data drift detection: 1. Lack of context: univariate data drift monitoring does not take into account the relationship between the variable being monitored and other variables in the system. The relationship between these variables might be complex. 2. Sensitivity to outliers: univariate data drift monitoring may be sensitive to outliers This leads to univariate data drift detection causing way too many false alarms, which is terrible. Imagine getting called to your ML solution every time saying something is wrong, when in fact nothing is wrong at all. What happens? Yes, alarm fatigue. You start paying less attention, or stop paying attention at all. Until, at one point, your model starts explicitly hurting the business. You then wake up, and it's too late. NannyML invented a sweet alternative called Direct Loss Estimation. It predicts the possible bandwidth of your predictions, in production. When your predictions go out of bounds, you get notified. And this is where we get back to good old univariate data drift detection! We can now use it as one of the tools for root cause analysis. Finding out why your model has degraded. Learn more about this in the blog by Santiago Viquez down here ⬇ #datascience #machinelearning #mlops

Great thanks to #nannyml for their amazing algorithms! It turns out they have algorithms that can accurately evaluate performance of both classification (M-CBPE) and regression (DLE) models on new data, even before the target values are available. I'll add links with detailed descriptions in the comments. While here I'd like to discuss why is this so great. The importance of having such algorithms for monitoring models in production has been brilliantly explained by #nannyml themselves. But this is only the beginning of the list of their possible successful use cases. Let me share my proposals on a few more of them. In one of my previous posts, I touched upon the problem of feature selection when there is a long time gap between the freshest data for which a target is available and the data coming in for inference. Traditional feature selection methods based on the impact of adding/dropping a feature on validation metrics are not very useful if the validation data is significantly different from the inference data. That time my suggestion was to use the Population Stability Index (PSI) to detect features with the most significant distribution shift and to take a closer look at them. But how exactly do this? Imagine that you couldn't find a transformation to stabilise the distribution of a problematic feature without noticeable worsening its metrics on validation. The answer to the question of whether to drop it is hard to formalise, thus it requires experience, a good understanding of prediction patterns, and experimentation. Of course, the probability to make a wrong decision remains high. With M-CBPE and DLE, such a decision can be made with maximum justification, practically eliminating the probability of errors: based on the change in selected performance metrics values expected on inference. In addition, these algorithms allow us to evaluate model quality metrics in situations where the difference between training and prediction data is expected from the outset. For example, if representativeness was lost during dirty data cleaning, or when creating the training dataset by matching. #Kudos to #nannyml and thank you for your contribution to #datascience! UPD: please, keep in mind that both methods cannot cope with the situation when the concept shift takes noticeable place. So, we are waiting for new brilliant solutions from #nannyml #machinelearning #modelmonitoring #featureselection #modelevaluation

🚀 Exciting News for Data Practitioners! 🚀 Are you tired of the hassle and expense of annotating production data to monitor your machine learning models? Well, Allow me to introduce you to NannyML, the open-source post-deployment model monitoring framework in Python that's about to make your life a whole lot easier. NannyML comes packed with some seriously clever features that are a game-changer for those of us who work with data. What's the best part? It doesn't rely on labeled data! Yes, you heard that right – all the magic happens with the features you're already capturing while your model is in production. Let's dive into the essence of NannyML: 🔍 Feature Drift Detection: NannyML helps you identify univariate drift by comparing feature distributions across different chunks of data. Think of these 'chunks' as snapshots of your data – they can be based on time, size, or number. 📊 Advanced Analysis: For more complex scenarios involving multivariate drift, NannyML checks the Principal Component Analysis (PCA) data reconstruction error across chunks. In simple terms, it ensures the stability of key components over time, so you can spot any deviations. But wait, there's more: 📈 Model Performance Estimation: NannyML doesn't stop at drift detection; it also estimates your model's performance, helping you maintain top-notch results. 💰 Business Value Estimation: It even assists you in assessing the business value of your models, ensuring they continue to deliver the desired outcomes. 🔍 Data Quality Monitoring: Keeping an eye on data quality? NannyML has your back, ensuring your data remains reliable and consistent. While NannyML offers a plethora of functionalities, let's focus on feature drift detection, which doesn't require labeled data. Unfortunately, it won't help with concept drift, but that's a small trade-off for the convenience it provides. 📊 Univariate Drift: Check if feature distributions have shifted between reference and analysis periods. This can be a game-changer for maintaining model accuracy. 📉 Output Drift: You can also track shifts in the distribution of predicted classes over time, ensuring your model's predictions remain on point. 📊 Multivariate Drift: NannyML goes a step further by looking at the overall shift in feature distributions, validating the univariate results. In conclusion, NannyML is a powerful tool that simplifies drift detection for production models. With its intuitive interface and open-source nature, there's no excuse not to use it. Don't let your production models lose their business value due to neglect. Embrace NannyML and ensure your models stay on the right track! 🚀

91% is an incredible percentage. Just goes to demonstrate how crucial it is to keep an eye on the performance of the models during production. Furthermore, there may be additional factors at play in addition to data drift that causes degradation. I am very happy to learn that NannyML an open-source python module exists to assist us in starting a performance monitoring project. Have you utilized NannyML or a different model performance monitoring tool?  #datascience #ai #machinelearning …see more

Another cool thing for the day! 💡 Thanks, Wojtek Kuberski and NannyML! #python #machinelearning #datascience

Does the pattern below look familiar to you? If not, you can consider yourself incredibly lucky! For everyone else, check out NannyML: NannyML is an open-source python library that allows you to estimate post-deployment model performance (without access to targets), detect data drift, and intelligently link data drift alerts back to changes in model performance. Built for data scientists, NannyML has an easy-to-use interface, interactive visualizations, is completely model-agnostic and currently supports all tabular binary classification use cases. (Source: lnkd.in/e2RjQsQg) #DataScience #MachineLearning #AI #DeepLearning

Thank you Hakim Elakhrass, Niels Nuyttens and NannyML team for sending this #contributions #swag all the way from Belgium. Really appreciate the note, Hakim. Looking forward to more #opensource #contributions to the wonderful NannyML #modelmonitoring tool. …see more

Great resources and libraries I have come across recently: NannyML: An open source tool for monitoring models post deployment. I find the ability to estimate performance without labels especially helpful, as we have a one year lag on our labels. It is also possible to detect data/classifier drift. 𝐎𝐩𝐭𝐁𝐢𝐧𝐧𝐢𝐧𝐠: The go to data binning library. Super useful for discovering interesting intervals in variables, and for binning and preprocessing before applying a logistic regression on top. 𝐓𝐚𝐛𝐃𝐃𝐏𝐌: A new method for modelling tabular data. While I have only briefly browsed the paper, it seems promising for generating tabular data, outperforming both Smote and GANs in most of the tested cases! The paper was published only a week ago, but I'm probably already late to the party talking about this one. What are your favorite new tools? Links in the comments👇 ------------------------------------------------------------------ Talking data science, credit risk, cryptocurrency, software development - Connect & start the conversation! #machinelearning #python #datascience #opensource