SageMaker training job has no checkpoint configuration
- Severity
- High
- Service
- SageMaker
- Check ID
- SAGEMAKER_NO_CHECKPOINTING
What this check finds
A SageMaker training job was submitted without a checkpoint configuration. Without checkpoints, a spot interruption loses all training progress and the full job must restart.
Passing looks like: SageMaker training has checkpointing.
How to fix it
Add CheckpointConfig with an S3 URI when submitting training jobs: --checkpoint-config S3Uri=s3://bucket/checkpoints/. Required when using spot instances.
Compliance
This is a cost check. It flags spend that buys nothing, which no compliance framework asks about, so it is not mapped to a control.
Checked on every scan
KloudLytics runs this check each time it scans a connected AWS account, through a read-only role, and lists every affected resource with its region. On Pro and Business a fix is written for the specific resource rather than the general case above. The exact access it needs
More SageMaker checks
- HighSageMaker notebook instance has direct internet access
- MediumSageMaker endpoint not deployed inside VPC
- MediumSageMaker notebook instance has root access enabled
- MediumSageMaker notebook instance running for 7+ days without modification
- MediumSageMaker resource not encrypted with CMK
- MediumSageMaker training job not using managed spot instances