Using Azure Machine Learning (AML) for Medical Imaging Vision Model Training and Fine-tuning | Microsoft Community Hub (2025)

Vision Model Architectures

At present, Transformer-based vision model architecture is considered the forefront of advanced vision modeling. These models are exceptionally versatile, capable of handling a wide range of applications, from object detection and image segmentation to contextual classification.

Two popular Transformer-based model implementations are often used in real-world applications. These are:

1. Masked Autoencoders (MAE) and
2. Vision Transformer (ViT)

Masked Autoencoders (MAE)

Masked Autoencoders (MAE) are a type of Transformer-based vision model architecture. They are designed to handle large-scale vision tasks by leveraging the power of self-supervised learning. The key idea behind MAE is to mask a portion of the input image and then train the model to reconstruct the missing parts. This approach forces the model to learn meaningful representations of the visual data, which can be useful for various downstream tasks such as object detection, image segmentation, and contextual classification.

MAE models are highly flexible and can be adapted to different applications by fine-tuning on specific datasets. Their ability to learn from large amounts of unlabeled data makes them particularly effective for tasks where labeled data is scarce or expensive to obtain.

Vision Transformer (ViT)

The Vision Transformer (ViT) is a cutting-edge model architecture designed for computer vision tasks. Unlike traditional convolutional neural networks (CNNs), ViT leverages the power of transformers, which have been highly successful in natural language processing. The key innovation of ViT is its ability to process images as sequences of patches, similar to how transformers handle sequences of words in text.

ViT divides an input image into fixed-size patches and then embeds each patch into a vector. These vectors are then fed into a transformer model, which processes them to capture long-range dependencies and complex patterns within the image. This approach allows ViT to achieve impressive performance on various vision tasks, such as image classification, object detection, and segmentation.

One of the main advantages of ViT is its scalability. It can be trained on large datasets and benefit from the transformer architecture's ability to handle vast amounts of data efficiently. Additionally, ViT models can be fine-tuned on specific datasets to adapt to different applications, making them highly versatile for various vision-related tasks.

Medical Imaging

Medical imaging applications often require highly specialized models to accurately interpret complex visual data. Custom image model training, using architectures like Masked Autoencoders (MAE) and Vision Transformers (ViT), can be particularly beneficial in the following areas:

1. Radiology: Custom models can be trained to detect and classify abnormalities in X-rays, CT scans, and MRI images. These models can help radiologists identify conditions such as tumors, fractures, and infections with greater precision.
2. Pathology: In digital pathology, custom models can analyze tissue samples to detect cancerous cells, classify different types of tumors, and assess the severity of diseases. This can improve diagnostic accuracy and speed up the analysis process.
3. Ophthalmology: Custom models can be used to analyze retinal images for early detection of eye diseases such as diabetic retinopathy, glaucoma, and age-related macular degeneration. These models can assist ophthalmologists in providing timely and accurate diagnoses.
4. Cardiology: Custom models can be trained to interpret echocardiograms and other cardiac imaging modalities to detect heart conditions such as arrhythmias, valve disorders, and congenital heart defects. This can enhance the accuracy of cardiac assessments and treatment planning.
5. Oncology: Custom models can be used to analyze PET and CT scans to monitor the progression of cancer and evaluate the effectiveness of treatments. These models can help oncologists make informed decisions about patient care.
6. Neurology: Custom models can be trained to analyze brain scans for the detection of neurological disorders such as Alzheimer's disease, Parkinson's disease, and multiple sclerosis. This can aid neurologists in diagnosing and managing these conditions.

MAE Base Model Training for Medical Imaging

There are several key challenges in training a Medical Accessory Expert (MAE) model for medical imaging. These challenges include dealing with a large unlabeled dataset of x-rays, handling data in various types of storage locations, the difficulty of local testing, managing a large number of parameters, and the requirement for a significant amount of GPU cores/memory.

Some of these challenges may be alleviated by using industry-grade Machine Learning tools such as Azure Machine Learning (AML). Azure Machine Learning offers a robust platform for training large-scale Transformer-based vision models like Masked Autoencoders (MAE). Several key aspects of training MAE models using Azure Machine Learning may be simplified with AML. These include:

• Data Access: Efficiently accessing large datasets stored in various types of storage.
• SKU Sizing: Selecting appropriate compute resources.
• Debugging: Resolving issues related to large parameter spaces and training efficiency.
• Deepspeed: Utilizing Deepspeed for managing large parameter spaces and embeddings

Data Access

Often, medical image training data consists of large number of relatively small images, such as X-rays. While AML offers multiple types of storage interfaces, such as MLTable, in practice, training performance seems to benefit from data access via the Premium Blob Storage type. Premium block blob storage accounts make data available via high-performance hardware. Data is stored on solid-state drives (SSDs), providing higher throughput compared to traditional hard drives. This results in faster file transfer because data is stored on instantly accessible memory chips. Premium block blob storage accounts are ideal for workloads that require fast and consistent response times and/or have a high number of input-output operations per second (IOP), such medical imagining AI/ML model training.

SKU Sizing

Training large-scale vision models, such as MAE, for medical imaging applications presents several challenges, including handling large unlabeled datasets, managing data in various storage locations, and requiring significant GPU cores and memory. Azure Machine Learning can help alleviate these challenges by providing on-demand access to industry-grade preconfigured GPU-accelerated Virtual Machines, significantly simplifying development efforts.

Debugging

AML provides several features to simplify the debugging process for large-scale vision models like Masked Autoencoders (MAE). These include:

• Log Search: AML exposes an intuitive interface to search and view job logs from multiple AML nodes in a single place, making it easier to debug distributed training jobs.
• Built-in Profiler: The profiler is a no-code tool that helps understand and optimize GPU operations for distributed training.
• Alerting on Job Metrics: AML can generate alerts on specific job metrics, helping monitor the performance and identify issues in real-time

Deepspeed

DeepSpeed is a powerful tool that can significantly enhance the training of large-scale Transformer-based vision models like Masked Autoencoders (MAE). Here’s how it can be utilized:

DeepSpeed can be used to manage large parameter spaces and embeddings during MAE training. It offers several key features that can help alleviate challenges associated with training large-scale vision models, including:

• Memory Efficiency: DeepSpeed's ZeRO (Zero Redundancy Optimizer) technology allows for efficient memory usage by partitioning model states across multiple GPUs. This enables the training of models with large parameter spaces without running into memory limitations.
• Scalability: DeepSpeed supports 3D parallelism, which combines tensor slicing, pipeline parallelism, and data parallelism. This allows for efficient scaling of training workloads across multiple GPUs, improving training throughput and efficiency1.
• Optimized Communication: DeepSpeed provides optimized communication strategies to reduce the overhead associated with data transfer between GPUs. This helps in maintaining high training speeds and efficiency

ViT fine-tuning for Classification

The process of fine-tuning Vision Transformers (ViT) for classification tasks include several key steps:

1. base model selection,
2. validation set partitioning,
3. job configuration,
4. infrastructure validation,
5. ACPT configuration

Base Model Selection

Selecting the right model to serve as the basis for ViT fine-tuning is a critical aspect application success. Some of the ViT models available on Hugging Face:

• google/vit-base-patch16-224: This model is designed for image classification tasks and is trained on ImageNet-21k. It uses a patch size of 16x16 and an image resolution of 224x2241.
• google/vit-base-patch16-384: Similar to the previous model but with a higher image resolution of 384x384, making it suitable for tasks requiring higher detail2.
• google/vit-large-patch16-224: A larger version of the base model with more parameters, designed for more complex image classification tasks2.
• google/vit-large-patch32-384: This model uses a larger patch size of 32x32 and a higher image resolution of 384x384, providing a balance between detail and computational efficiency2.
• google/vit-huge-patch14-224: This model has an even larger architecture with a patch size of 14x14, designed for very detailed image classification tasks2.

These models are highly versatile and can be fine-tuned for various computer vision applications, including object detection, image segmentation, and contextual classification. They leverage the power of transformers to capture long-range dependencies and complex patterns within images, making them highly effective for a wide range of vision tasks.

Validation Set Partitioning

When training a vision model, it is important to appropriately partition the training data in such a way as to minimize validation noise. Inappropriately sampled validation dataset may lead to significant challenges during training. Below example shows how a particular training job performance was improved by appropriately sized validation dataset sample:

Job Configuration

Azure Machine Learning supports running distributed jobs using PyTorch native distributed training capabilities. However, care must be taken to configure training code in a way that takes advantage of these capabilities. For example, when implementing custom training scripts, it is important to specify Local and Global GPU Rank in a way that corresponds to training infrastructure topology:

Infrastructure Validation

For NVIDIA GPU workloads, NCCL stand-alone library is used to implement communication routines for multi-GPU training. The following operations should be tested on NVIDA GPU-accelerate compute clusters to ensure best performance for distributed jobs:

1. all-reduce,
2. all-gather,
3. reduce,
4. broadcast,
5. reduce-scatter,

ACPT Configuration

ACPT is available in a curated environment and a data science virtual machine. It supports Nvidia GPUs within AzureML, including state-of-the-art H100s, A100s and V100s, and can also run on AMD GPUs like ROCm. ACPT is integrated with AzureML SDK, AzureML CLI, and AzureML Studio, making it easily accessible for various deep learning tasks.

While ACPT simplifies development by offering out-of-the-box interaction with GPUs, fine-tuning advanced vision models such as ViT might require specific versions of CUDA and PyTorch to function properly. Therefore, care must be taken to select the correct ACPT image flavor to ensure compatibility with the training code.

Conclusion

In conclusion, the use of Azure Machine Learning (AML) for training and fine-tuning advanced vision models, particularly in the medical imaging domain, offers significant advantages. The ability to leverage Transformer-based architectures such as Masked Autoencoders (MAE) and Vision Transformers (ViT) allows for the development of highly specialized models that can accurately interpret complex visual data. AML provides robust tools and features to address the challenges associated with training large-scale vision models, including efficient data access, appropriate SKU sizing, and comprehensive debugging facilities.

By utilizing AML, developers can streamline the training process, optimize resource usage, and enhance model performance. The integration of DeepSpeed further improves scalability and memory efficiency, making it possible to train models with large parameter spaces. Additionally, the curated environments like Azure Container for PyTorch (ACPT) simplify the setup and deployment of deep learning tasks, ensuring compatibility with the latest technologies. Overall, AML empowers researchers and practitioners to advance the field of medical imaging through the development of cutting-edge vision models.