AI-300T00: Operationalize Machine Learning and Generative AI Solutions

AI-300T00: Operationalize Machine Learning and Generative AI Solutions is an instructor-led course for data scientists, machine learning engineers, AI engineers, and DevOps professionals who want to design, implement, automate, monitor, and optimize production-ready AI systems on Microsoft Azure.

Students learn how to operationalize traditional machine learning models with Azure Machine Learning and operationalize generative AI applications and agents with Microsoft Foundry. The course emphasizes MLOps, GenAIOps, automation, CI/CD, GitHub Actions, infrastructure as code, model lifecycle management, prompt versioning, evaluation, observability, monitoring, fine-tuning, and production optimization.

Certification: Microsoft Certified: Machine Learning Operations Engineer Associate
Exam: AI-300: Operationalizing Machine Learning and Generative AI Solutions

Why choose Dynamics Edge for AI-300 training?

Dynamics Edge delivers AI-300 training with practical Azure Machine Learning and Microsoft Foundry examples, hands-on labs, certification review, and implementation-focused discussion. The course helps teams move from experimental AI models and agents into reliable, governed, monitored, and continuously improved production AI systems.

• Learn how to design and implement MLOps infrastructure using Azure Machine Learning, GitHub Actions, Azure CLI, and Bicep.
• Practice model experimentation, MLflow tracking, automated machine learning, hyperparameter tuning, pipelines, registration, deployment, and monitoring.
• Operationalize GenAIOps with Microsoft Foundry projects, agent specifications, prompt versioning, Git workflows, cloud evaluators, tracing, and fine-tuning.
• Implement responsible AI, quality assurance, evaluation rubrics, observability, Application Insights, and production troubleshooting.
• Prepare for the Machine Learning Operations Engineer Associate certification with structured review and hands-on lab reinforcement.

What will you learn in AI-300 training?

Students learn how to build secure, scalable, automated AI operations solutions across the full lifecycle of machine learning and generative AI systems.

• Create and manage Azure Machine Learning workspaces, compute, datastores, environments, components, pipelines, registries, and endpoints.
• Track experiments and model training with MLflow, notebooks, scripts, command jobs, AutoML, and hyperparameter sweep jobs.
• Deploy machine learning models to batch and online endpoints, then monitor drift, performance, retraining, rollout, rollback, and model lifecycle state.
• Design GenAIOps infrastructure using Microsoft Foundry, Azure Developer CLI, Bicep, source control, prompts, agents, test datasets, and automated evaluations.
• Monitor, trace, evaluate, optimize, and fine-tune generative AI agents using Application Insights, distributed tracing, cloud evaluators, prompt variants, and quality metrics.

Operationalize Machine Learning and Generative AI Solutions AI-300 Course Outline

Module 1: Plan Azure AI operations

Students learn how AI operations combines MLOps and GenAIOps to make AI systems reliable, repeatable, secure, and production-ready. The module introduces Azure Machine Learning, Microsoft Foundry, GitHub Actions, Azure CLI, Bicep, source control, automation, and observability.

Topics include:

• Describe MLOps, GenAIOps, and AIOps responsibilities.
• Identify production AI lifecycle requirements.
• Plan collaboration across data science, DevOps, and business teams.
• Select Azure Machine Learning and Microsoft Foundry operational tools.
• Apply automation, monitoring, and governance principles.

Module 2: Create and manage Azure Machine Learning resources

Students learn how to create and manage the foundational Azure Machine Learning resources required for MLOps. The module covers workspaces, compute targets, datastores, environments, components, registries, identities, and access controls.

Topics include:

• Create and manage Azure Machine Learning workspaces.
• Create and manage compute targets and datastores.
• Create and manage environments and reusable components.
• Share assets across workspaces by using registries.
• Configure identity, RBAC, and secure access.

Module 3: Find the best model with automated machine learning

Students learn how automated machine learning helps explore algorithms, preprocessing options, featurization, and metrics. The module explains how to choose tasks, configure AutoML experiments, set limits, restrict algorithms, evaluate models, and retrieve the best run.

Topics include:

• Choose the correct AutoML task.
• Configure featurization, scaling, normalization, and preprocessing.
• Run automated machine learning experiments.
• Restrict algorithm selection and set experiment limits.
• Evaluate and compare candidate models.

Module 4: Track model training with notebooks and MLflow

Students learn how MLflow supports experiment tracking in notebooks. The module covers tracking parameters, metrics, artifacts, custom logs, autologging, MLflow tracking URIs, and comparing experiment results in Azure Machine Learning.

Topics include:

• Configure MLflow for Azure Machine Learning tracking.
• Track metrics, parameters, and artifacts.
• Enable autologging for common machine learning libraries.
• Train and track models in notebooks.
• Retrieve and compare MLflow metrics.

Module 5: Create Responsible AI dashboards

Students learn how Responsible AI dashboards help identify fairness, reliability, safety, interpretability, and model performance issues. The module introduces data exploration, error analysis, feature importance, fairness assessment, and mitigation concepts.

Topics include:

• Understand Responsible AI principles.
• Analyze data and model behavior.
• Assess errors across data cohorts.
• Interpret feature importance.
• Evaluate fairness and disparity.

Module 6: Optimize model training with command jobs

Students learn how to move from notebook experimentation to repeatable training scripts. The module covers converting notebooks to scripts, creating command jobs, using parameters, logging metrics, and tracking jobs with MLflow.

Topics include:

• Convert notebooks to reusable scripts.
• Configure command jobs.
• Use parameters for repeatable training.
• Track job metrics with MLflow.
• Use custom logging and autologging.

Module 7: Perform hyperparameter tuning

Students learn how to use sweep jobs to tune machine learning models. The module covers search spaces, sampling methods, early termination, bandit policy, median stopping policy, and truncation selection policy.

Topics include:

• Define hyperparameter search spaces.
• Configure sweep jobs.
• Select sampling methods.
• Configure early termination.
• Compare tuning results.

Module 8: Run pipelines in Azure Machine Learning

Students learn how machine learning pipelines automate reusable workflows. The module covers components, component metadata, YAML files, component registration, pipeline construction, pipeline jobs, and scheduling.

Topics include:

• Create reusable pipeline components.
• Register components in Azure Machine Learning.
• Build pipeline workflows.
• Run pipeline jobs.
• Schedule recurring pipelines.

Module 9: Design MLOps architecture with GitHub Actions

Students learn how to automate machine learning lifecycle operations using source control and CI/CD. The module covers GitHub integration, trunk-based development, GitHub Actions, secure access, training automation, registries, and production promotion.

Topics include:

• Design an MLOps architecture.
• Integrate GitHub with Azure Machine Learning.
• Trigger Azure Machine Learning jobs with GitHub Actions.
• Register MLflow models in Azure Machine Learning.
• Promote models through development and production workflows.

Module 10: Deploy models to production endpoints

Students learn how to deploy machine learning models to batch and managed online endpoints. The module covers endpoint creation, deployment options, MLflow models, custom models, testing, troubleshooting, progressive rollout, rollback, and archiving.

Topics include:

• Deploy models to batch endpoints.
• Deploy models to managed online endpoints.
• Test and troubleshoot model endpoints.
• Implement progressive rollout and safe rollback.
• Archive bad or outdated model versions.

Module 11: Monitor and maintain machine learning models

Students learn how to monitor production models and maintain model quality. The module covers data drift, model performance, retraining triggers, alert thresholds, endpoint monitoring, and continuous improvement.

Topics include:

• Monitor model performance metrics.
• Detect and analyze data drift.
• Configure retraining triggers.
• Monitor production endpoint behavior.
• Improve models based on operational feedback.

Module 12: Plan and prepare GenAIOps infrastructure

Students learn how to prepare infrastructure for generative AI applications and agents. The module covers agent specifications, business use cases, model selection, tool selection, knowledge sources, Azure Developer CLI, Microsoft Foundry hubs and projects, and monitoring resources.

Topics include:

• Define agent specifications and user intent.
• Select models for production scenarios.
• Identify tools and knowledge sources.
• Provision Microsoft Foundry resources with Azure Developer CLI.
• Prepare Application Insights for monitoring.

Module 13: Manage prompts and agents with GitHub

Students learn how prompt versioning and source control support safe GenAIOps. The module covers prompt repositories, prompt files, configuration files, development and production environments, branch workflows, and controlled deployment.

Topics include:

• Version prompts using GitHub.
• Structure repositories for prompts and agent configuration.
• Create prompt and agent versions.
• Test prompt behavior safely before production.
• Use safe deployment workflows.

Module 14: Evaluate and optimize AI agents through experiments

Students learn how to evaluate AI agents using structured experiments. The module covers test prompts, prompt variants, evaluation rubrics, calibrated scoring, Git-based experiment branches, quality thresholds, and evidence-based deployment decisions.

Topics include:

• Design agent evaluation experiments.
• Create branches for prompt and model variants.
• Use rubrics to score responses consistently.
• Compare experiment results.
• Select prompt optimizations for production.

Module 15: Automate AI evaluations with Microsoft Foundry

Students learn how automated evaluation supports scalable quality assurance. The module covers cloud evaluators, built-in metrics, human evaluation, shadow rating, automated gates, low-score review, and continuous evaluation workflows.

Topics include:

• Select built-in evaluators.
• Create test datasets and data mappings.
• Run cloud-based AI evaluations.
• Compare automated and human evaluation results.
• Configure automated quality gates.

Module 16: Monitor and trace generative AI agents

Students learn how observability helps operate generative AI agents in production. The module covers monitoring, tracing, Application Insights, latency, throughput, response time, cost metrics, token usage, spans, trace journeys, and debugging.

Topics include:

• Configure monitoring for generative AI agents.
• Implement distributed tracing.
• Track latency, throughput, response times, and cost.
• Compare prompt versions using telemetry.
• Troubleshoot runtime behavior with trace data.

Module 17: Optimize and fine-tune AI agents

Students learn how to improve generative AI agent performance through fine-tuning and optimization. The module covers supervised fine-tuning, reinforcement fine-tuning, direct preference optimization, dataset preparation, hyperparameters, quality problems, and production validation.

Topics include:

• Compare SFT, RFT, and DPO approaches.
• Prepare and validate fine-tuning datasets.
• Configure fine-tuning hyperparameters.
• Analyze real agent quality problems.
• Validate optimized agent behavior.

Module 18: Optimize RAG performance and accuracy

Students learn how retrieval-augmented generation performance can be improved through retrieval tuning. The module covers chunking, similarity thresholds, embedding selection, hybrid search, relevance metrics, A/B testing, and performance measurement.

Topics include:

• Tune chunk sizes and retrieval strategies.
• Adjust similarity thresholds.
• Select and evaluate embedding models.
• Implement hybrid semantic and keyword search.
• Use relevance metrics and A/B testing.

Module 19: Secure and govern production AI operations

Students learn how security and governance apply across MLOps and GenAIOps. The module covers managed identities, RBAC, private networking, secure GitHub access, infrastructure as code, production controls, restricted network access, and compliance considerations.

Topics include:

• Configure managed identities and RBAC.
• Restrict network access to AI resources.
• Secure GitHub integration and automation.
• Deploy resources using Bicep and Azure CLI.
• Apply governance controls to production AI systems.

Module 20: Operate production AI systems continuously

Students learn how to operate AI systems as ongoing products rather than one-time deployments. The module covers operational metrics, cost tracking, quality review, retraining, prompt updates, rollout strategy, rollback, incident response, and continuous improvement.

Topics include:

• Track operational health and cost.
• Manage rollout and rollback strategies.
• Review model and prompt quality over time.
• Respond to production incidents.
• Improve AI systems through feedback loops.

Hands-on labs

The AI-300 labs support hands-on practice for learners operationalizing machine learning and generative AI solutions. This single consolidated lab list is based on the most important exercise and speaker-note topics found in the uploaded AI-300 PowerPoint deck.

• Lab 1: Find the best classification model with Azure Machine Learning by using automated machine learning and model comparison.
• Lab 2: Train a classification model in an interactive notebook and track the experiment with MLflow.
• Lab 3: Create and explore a Responsible AI dashboard to assess reliability, fairness, errors, and model interpretability.
• Lab 4: Optimize model training in Azure Machine Learning by converting notebooks to scripts and running command jobs.
• Lab 5: Track model training with MLflow using custom logging and autologging.
• Lab 6: Perform hyperparameter tuning with a sweep job, search spaces, sampling methods, and early termination.
• Lab 7: Run pipelines in Azure Machine Learning by creating components, building a pipeline, running the pipeline, and scheduling it.
• Lab 8: Design an MLOps architecture with GitHub Actions, source control, Azure Machine Learning jobs, and secure automation.
• Lab 9: Register an MLflow model in Azure Machine Learning and manage model lifecycle versions.
• Lab 10: Deploy a model to a batch endpoint and troubleshoot batch scoring jobs.
• Lab 11: Deploy a model to a managed online endpoint, test the endpoint, and review deployment logs.
• Lab 12: Deploy and monitor a model using GitHub Actions environments, shared Azure Machine Learning registries, production endpoints, drift monitoring, retraining, promotion, rollback, and model archiving.
• Lab 13: Plan and prepare a GenAIOps solution by provisioning Microsoft Foundry hub, project, and monitoring resources with Azure Developer CLI.
• Lab 14: Develop prompt and agent versions by deploying multiple Trail Guide Agent versions to Microsoft Foundry and testing their behavior.
• Lab 15: Design and optimize prompts using Git-based experimentation, baseline scoring, experiment branches, quality rubrics, and evidence-based prompt promotion.
• Lab 16: Automate AI evaluations with Microsoft Foundry cloud evaluators, test datasets, automated metrics, and quality gates.
• Lab 17: Monitor and trace a generative AI agent using Application Insights and distributed tracing.
• Lab 18: Compare prompt versions using latency, response quality, trace data, and production telemetry.
• Lab 19: Optimize AI agents with fine-tuning by comparing supervised fine-tuning, reinforcement fine-tuning, and direct preference optimization.
• Lab 20: Build a production AI operations workflow that combines evaluation, observability, prompt versioning, model versioning, rollout, rollback, retraining, and continuous improvement.

Certification alignment

This course supports preparation for Exam AI-300: Operationalizing Machine Learning and Generative AI Solutions and the Microsoft Certified: Machine Learning Operations Engineer Associate certification. The exam validates the ability to set up MLOps and GenAIOps infrastructure, manage machine learning model lifecycle operations, implement AI quality assurance, monitor production systems, and optimize generative AI systems.

AI-300 skills measured

• Design and implement an MLOps infrastructure.
• Implement machine learning model lifecycle and operations.
• Design and implement a GenAIOps infrastructure.
• Implement generative AI quality assurance and observability.
• Optimize generative AI systems and model performance.

Course review

Students should leave the course able to operationalize machine learning and generative AI systems on Azure. The course review should reinforce Azure Machine Learning workspaces, compute, datastores, environments, components, registries, AutoML, MLflow, command jobs, sweep jobs, Responsible AI dashboards, pipelines, GitHub Actions, model registration, batch endpoints, online endpoints, model monitoring, Microsoft Foundry, GenAIOps, prompt versioning, agent versions, cloud evaluators, tracing, Application Insights, fine-tuning, RAG optimization, rollout, rollback, and continuous improvement.

Certification exam review

Exam review should focus on production AI operations and scenario-based implementation decisions. Priority review areas should include Azure Machine Learning resources, compute targets, datastores, environments, components, registries, Bicep, Azure CLI, GitHub Actions, secure Git integration, AutoML, MLflow tracking, notebooks, command jobs, hyperparameter tuning, training pipelines, model registration, batch endpoints, online endpoints, drift detection, retraining, model versioning, Microsoft Foundry projects, managed identities, RBAC, private networking, prompt versioning, prompt variants, cloud evaluators, groundedness, relevance, coherence, fluency, harmful content evaluation, Application Insights, tracing, token consumption, latency, cost metrics, RAG tuning, embedding models, hybrid search, fine-tuning, synthetic data, rollout strategy, rollback strategy, and production monitoring.