Dynamic Orchestration of Data Pipelines via Agentic AI: Adaptive Resource Allocation and Workflow Optimization in Cloud-Native Analytics Platforms

Abstract

The move towards more dynamic machine learning (ML) is present in all sorts of areas within the field, creating a tantalizing possibility to use such methods to not only create models that are more in tune with the constantly changing world we live in and the data which is being generated from it but also a world where the automated pipeline processes behind the ML modeling are severely underutilized. One area that has received little attention in this burgeoning area is most of the work being done under the umbrella of agentic AI, or methods aimed at creating agents that are semi-autonomously able to do complex tasks with little human interaction. This paper aims to summarize some of the work, both in terms of completed tasks as well as the technologies that will allow us to create a new paradigm for an automated data ML pipeline, both in terms of how data is consumed for ML tasks as well as how tasks are set up, monitored, and completed. Using an agent-centric approach we seek to allow agents to both have different expertise levels with regards to different types of ML tasks, be able to learn over time by themselves which are the most competent agents for certain tasks, as well as having users not need to worry about piping together a complex set of steps for a complicated task. Rather the user will simply provide a high-level description of what they need accomplished, as well as any constraints, and the agent system will find the best pathway for accomplishing the task and either execute the pathway itself or coordinate with other agents to complete the task. Such a paradigm will allow for the automated orchestration of very complicated tasks as well as the execution and monitoring of larger, often cumbersome and fragile task pathways, and allow domain specialists for any field to be able to utilize ML pipelines to accomplish their goals while the technical details are handled by specialized ML agents.