Nowadays the required conditions to obtain value from machine learning, and solve complex business problems are at our reach. We have now the capacity to create and manage large datasets, there are several accelerators on several cloud platforms and with affordable costs that support viable business cases. On top of this the research and development on the field enabled us to have access to experts in the field that are capacitated to bring ML based solutions to our companies and societies in general. But operationalizing ML models comes with challenges that created the need for MLOps.
We at Link Redglue have been investing in creating capacity to address complex business challenges with Machine Learning, both through services and product offers. Supported on our strong data engineering skills we’ve been solving complex use cases and providing solutions for different industries, with the certain that ML, in some use cases, is a true game changer that can change the rules of some games.
Given the current skills and data architectures is reasonably straight forward having a capable data scientist creating a new model, training that model with relevant data is usually more of a challenge, and deploying and operating it in productive environments is even more challenging. Have no doubts that operationalizing ML models is hard, and it’s hard because they have to be trained with real data and respond to change what forces re-training and re-deployment, and also new customizations can force a start from scratch. This on top of new tools and environments that IT departments are still learning how to manage and support.
A problem of a similar nature as been solved before in software development, that achieved continuous integration and delivery with DevOps practices, but ML challenge is different, in part because it requires data, and I mean real live data, earlier on the process. To solve this challenge MLOps arrived and changed the approach, and enabled agilization of the process, allowing a better time to production for new models that address the needs of business areas, leveraging data-based value creation in organizations, enabling concurrent models training and automated pipelines. Always making sure the lifecycle allows reproducibility, traceability and verifiability because we’re working with real customers data.
That are several tools and technologies that enable the creation of MLOps setups, recently Link Redglue did a setup in a financial institution, where we used, aside other technologies, databricks and MLFlow to create a full MLOps setup that enables leveraging the value the current data scientists can deliver to the organization and most of all to the customers.
