SageMaker training job not using managed spot instances
- Severity
- Medium
- Service
- SageMaker
- Check ID
- SAGEMAKER_NO_SPOT_TRAINING
What this check finds
A SageMaker training job ran without managed spot training enabled. Spot instances reduce training costs by up to 90% with automatic checkpointing on interruption.
Passing looks like: SageMaker training uses managed spot.
How to fix it
Enable managed spot training: set EnableManagedSpotTraining=true and configure StoppingCondition.MaxWaitTimeInSeconds when submitting training jobs.
Compliance
This is a cost check. It flags spend that buys nothing, which no compliance framework asks about, so it is not mapped to a control.
Checked on every scan
KloudLytics runs this check each time it scans a connected AWS account, through a read-only role, and lists every affected resource with its region. On Pro and Business a fix is written for the specific resource rather than the general case above. The exact access it needs
More SageMaker checks
- HighSageMaker notebook instance has direct internet access
- HighSageMaker training job has no checkpoint configuration
- MediumSageMaker endpoint not deployed inside VPC
- MediumSageMaker notebook instance has root access enabled
- MediumSageMaker notebook instance running for 7+ days without modification
- MediumSageMaker resource not encrypted with CMK