The increasing availability of generative AI models and tools means businesses can now utilize similar foundation models (FMs) as their rivals. Real competitive advantage stems from developing AI specifically tailored to a business’s unique needs, a customization not easily replicated. While current FMs possess significant intelligence and reasoning, this intelligence lacks specific context. A model might understand how to process information, but it doesn’t inherently grasp a business’s specific thought processes, terminology, data patterns, or industry limitations.
Developing models that truly comprehend a business relies on their ability to learn from specific data and preferences. This learning process often mirrors human learning: models first gain general knowledge through pre-training, then acquire specialized understanding via supervised fine-tuning, and finally align with particular preferences using methods like direct preference optimization (DPO). During inference, models apply their learned knowledge to real-world tasks and can continue adapting through parameter-efficient methods like Low-Rank Adaptation (LoRA) without needing to retrain the entire base model.
This comprehensive learning path, from pre-training large FMs to customizing them for specific applications, is now supported by Amazon SageMaker AI.
During AWS re:Invent 2025, Amazon SageMaker AI unveiled major advancements designed to revolutionize model customization and large-scale training. These new features aim to resolve two ongoing issues: the difficulty and time involved in tailoring FMs for particular uses, and expensive infrastructure failures that can halt weeks of training progress.
Since its launch in 2017, Amazon SageMaker AI has focused on making AI development accessible to various skill levels. With over 450 capabilities introduced, SageMaker AI consistently removes obstacles to innovation. This article examines how new serverless model customization, elastic training, checkpointless training, and serverless MLflow collectively speed up AI development, potentially reducing timelines from months to days.
Serverless AI model customization with advanced reinforcement learning
The latest serverless model customization feature within Amazon SageMaker AI significantly shortens a process that typically took months, now completing it in days. For AI developers seeking maximum abstraction, an AI agent-guided workflow (currently in preview) is being introduced, enabling advanced model customization through natural language.
Instead of needing extensive knowledge of reinforcement learning, users can now articulate business objectives in plain language. The AI agent engages in a multi-turn conversation to grasp the use case, then produces a detailed specification. This includes dataset guidelines, evaluation criteria, relevant metrics, and a suggested model, all of which a team can implement without requiring specialized expertise.
This AI agentic workflow supports supervised fine-tuning (SFT), direct preference optimization (DPO), reinforcement learning from AI feedback (RLAIF), and Reinforcement Learning from Verifiable Rewards (RLVR). These reinforcement learning capabilities allow models to learn from human preferences and verifiable outcomes, resulting in AI that better aligns with business objectives. Users can also generate synthetic data when real-world data is scarce, analyze data quality, and manage training and evaluation for accuracy and responsible AI controls. This serverless approach eliminates infrastructure complexity.
For AI developers desiring more control over customization, SageMaker AI provides a user-friendly interface incorporating best practices. Via SageMaker Studio, users can select from popular models such as Amazon Nova, Meta’s Llama, Qwen, DeepSeek, and GPT-OSS, then choose their preferred customization method.
The self-guided workflow offers flexibility throughout the process. Users can upload their own datasets or choose from existing ones, configure hyperparameters like batch size and learning rate with suggested defaults, and opt for either parameter-efficient fine-tuning with LoRA or full fine-tuning. The interface integrates with the new MLflow capability for automated experiment tracking, providing insight into training progress and model performance from a single view.
Similar to the AI agentic method, self-guided customization is entirely serverless. SageMaker AI automatically manages compute provisioning, scaling, and optimization, allowing users to concentrate on model development rather than infrastructure management. Pay-per-token pricing eliminates the need to select instance types or manage clusters.
Collinear AI reduced its experimentation cycles from weeks to days by utilizing SageMaker AI’s serverless model customization. Soumyadeep Bakshi, Co-founder of Collinear AI, stated: “At Collinear, curated datasets and simulation environments are built for frontier AI labs and Fortune 500 enterprises to enhance their models. Fine-tuning AI models is crucial for creating high-fidelity simulations, a process that previously involved integrating various systems for training, evaluation, and deployment. The new Amazon SageMaker AI serverless model customization capability now provides a unified approach, enabling experimentation cycles to be shortened from weeks to days. This comprehensive serverless tooling allows a focus on developing superior training data and simulations for customers, rather than on infrastructure maintenance or juggling disparate platforms.”
Bridging model customization and pre-training
While serverless model customization speeds up development for specific applications using fine-tuning and reinforcement learning, organizations are also increasingly adopting generative AI across various business functions. Applications demanding deep domain expertise or particular business context require models that genuinely comprehend proprietary knowledge, workflows, and unique needs. Methods like prompt engineering and Retrieval Augmented Generation (RAG) are effective for many scenarios, but they have inherent limitations when it comes to embedding specialized knowledge into a model’s fundamental understanding. When organizations try more profound customization through continued pre-training (CPT) using only their proprietary data, they often face catastrophic forgetting, where models lose their core capabilities as they acquire new information.
Amazon SageMaker AI supports the full range of model development, from serverless customization with advanced reinforcement learning to constructing frontier models from early checkpoints. For organizations possessing proprietary data that require models with deep domain expertise beyond what customization alone can offer, a new capability was recently introduced. This addresses the limitations of traditional methods while maintaining foundational model capabilities.
Recently, Amazon Nova Forge was introduced. Available on Amazon SageMaker AI, this new service allows AI developers to create their own frontier models using Amazon Nova. Nova Forge enables model development to begin from early checkpoints across pre-training, mid-training, and post-training phases, allowing intervention at optimal stages rather than waiting for full training completion. Proprietary data can be blended with Amazon Nova curated data throughout training phases using proven recipes on SageMaker AI’s fully managed infrastructure. This data mixing strategy substantially reduces catastrophic forgetting compared to training with raw data alone. It helps maintain foundational skills, including core intelligence, general instruction following, and safety benefits, while integrating specialized knowledge. Nova Forge offers a straightforward and cost-effective method for building custom frontier models.
The following video provides an introduction to Amazon Nova Forge.
Nova Forge is designed for organizations possessing proprietary or industry-specific data that aim to build AI with a deep understanding of their domain, including:
- Manufacturing and automation – Building models that understand specialized processes and equipment data
- Research and development – Creating models trained on proprietary research data
- Content and media – Developing models that understand brand voice and content standards
- Specialized industries – Training models on industry-specific terminology, regulations, and best practices
Companies such as Nomura Research Institute are utilizing Amazon Nova Forge to develop industry-specific large language models (LLMs) by merging Amazon Nova curated data with their proprietary datasets.
Takahiko Inaba, Head of AI and Managing Director at Nomura Research Institute, Ltd., commented: “Nova Forge allows for the creation of industry-specific LLMs, presenting a strong alternative to open-weight models. Operating on SageMaker AI with managed training infrastructure, specialized models, such as their Japanese financial services LLM, can be efficiently developed by combining Amazon Nova curated data with proprietary datasets.”
Elastic training for intelligent resource management at scale
The need for AI accelerators continuously varies as inference workloads adjust to traffic, completed experiments free up resources, and new training jobs alter priorities. Conventional training workloads are often fixed to their initial compute allocation, unable to utilize idle capacity without manual intervention, a task that can consume many engineering hours weekly.
Elastic training on Amazon SageMaker HyperPod changes this situation. Training jobs now automatically scale according to compute resource availability, expanding to use idle AI accelerators and optimizing infrastructure utilization. If higher-priority workloads, like inference or evaluation, require resources, training gracefully scales down, continuing with fewer resources instead of stopping completely.
The technical architecture ensures training quality during scaling transitions by maintaining global batch size and learning rate across diverse data-parallel configurations. This guarantees consistent convergence properties irrespective of the current scale. The SageMaker HyperPod training operator manages scaling decisions by integrating with the Kubernetes control plane, constantly monitoring cluster status via pod lifecycle events, node availability changes, and resource scheduler priority signals.
Initiating this is straightforward. New elastic SageMaker HyperPod recipes for publicly available FMs, including Meta’s Llama and GPT-OSS, require no code alterations—only YAML configuration updates to define the elastic policy.
Salesforce utilizes elastic training to automatically scale workloads and absorb idle GPUs as they become available. The company explained that elastic training “will enable workloads to automatically scale to absorb idle GPUs as they become available and seamlessly yield resources, all without disrupting development cycles. Most importantly, it will save hours spent manually reconfiguring jobs to match available compute, time that can be reinvested in innovation.”
Minimizing recovery downtime with checkpointless training
Infrastructure failures have historically hindered progress in large-scale training. Training runs lasting weeks can be disrupted by a single node failure, necessitating a restart from the last checkpoint and resulting in the loss of hours or days of costly GPU time. Conventional checkpoint-based recovery involves several sequential stages: job termination and restart, process discovery and network setup, checkpoint retrieval, GPU context reinitialization, and training loop resumption. When failures happen, the entire cluster must await the completion of each stage before training can recommence.
Checkpointless training on Amazon SageMaker HyperPod eliminates this obstacle. The system continuously preserves model state across distributed clusters, automatically replacing faulty components and restoring training via peer-to-peer transfer of model states from healthy AI accelerators. When infrastructure faults occur, recovery takes seconds with no manual intervention. The following video introduces checkpointless training.
This results in over 95% training goodput on clusters with thousands of AI accelerators, indicating that compute infrastructure is actively used for training jobs up to 95% of the time. This allows for a focus on innovation rather than infrastructure management, speeding up time-to-market by weeks.
Intercom is integrating checkpointless training into its pipelines to eliminate manual checkpoint recovery, stating: “At Intercom, new models are constantly trained to improve Fin, and the integration of checkpointless training into pipelines is highly anticipated. This will completely eliminate the need for manual checkpoint recovery. Combined with elastic training, it will allow for the delivery of improvements to Fin faster and with lower infrastructure costs.”
Serverless MLflow: Observability for every AI developer
Regardless of whether models are being customized or trained at scale, capabilities are needed to track experiments, observe behavior, and evaluate performance. However, managing MLflow infrastructure typically demands that administrators continuously maintain and scale tracking servers, make intricate capacity planning decisions, and deploy separate instances for data isolation. This infrastructure overhead diverts resources from primary AI development.
Amazon SageMaker AI now provides a serverless MLflow capability that eliminates this complexity. Users can start tracking, comparing, and evaluating experiments without waiting for infrastructure setup. MLflow scales dynamically to provide rapid performance for demanding and unpredictable model development tasks, then scales down during periods of inactivity. The following screenshot displays the MLFlow application within the SageMaker AI UI. 
This capability integrates natively with Amazon SageMaker AI serverless model customization, allowing visualization of in-progress training jobs and evaluations through a unified interface. Advanced tracing features assist in quickly identifying bugs or unexpected behaviors in agentic workflows and multi-step applications. Teams can leverage the MLflow Prompt Registry to version, track, and reuse prompts across organizations, ensuring consistency and enhancing collaboration.
Integration with SageMaker Model Registry ensures seamless model governance, automatically synchronizing models registered in MLflow with the production lifecycle. Once models meet desired accuracy and performance goals, they can be deployed to SageMaker AI inference endpoints with just a few clicks.
Administrators can boost productivity by configuring cross-account access using AWS Resource Access Manager (AWS RAM), simplifying collaboration across organizational limits. The serverless MLflow capability is provided at no extra cost and automatically updates to the latest MLflow version, granting access to new features without maintenance windows or migration efforts.
The Wildlife Conservation Society is utilizing the new serverless capability to improve productivity and speed up time-to-insights. Kim Fisher, MERMAID Lead Software Engineer at WCS, stated: “WCS advances global coral reef conservation through MERMAID, an open-source platform that uses ML models to analyze coral reef photos from scientists worldwide. Amazon SageMaker with MLflow has enhanced productivity by removing the necessity to configure MLflow tracking servers or manage capacity as infrastructure needs evolve. By enabling the team to focus entirely on model innovation, time-to-deployment is accelerated to deliver critical cloud-driven insights to marine scientists and managers.”
Accelerating AI innovation at every level
These announcements signify more than just individual feature enhancements; they create a comprehensive system for AI model development, supporting builders at every stage of their journey. From natural language-guided customization to self-directed workflows, intelligent resource management to fault-tolerant training, and experiment tracking to production deployment, Amazon SageMaker AI offers a full toolkit for bringing AI concepts to production.
Getting started
The new SageMaker AI model customization and SageMaker HyperPod capabilities are currently available in AWS Regions globally. Existing SageMaker AI customers can access these features via the SageMaker AI console, while new customers can begin with the AWS Free Tier.
For additional details on the latest Amazon SageMaker AI capabilities, visit aws.amazon.com/sagemaker/ai.
