A competition team's hardware budget almost always goes to the part that flies. What's left
for training is usually a laptop and a Google Colab tab. And Colab does the job —
right up until hour three of a run, when the tab says "runtime disconnected" and wipes everything.

That is exactly how we lost our first runs. We were training YOLO11m at imgsz=1280,
the training outputs were being written to Google Drive, and the dataset was sitting on Drive too.
When the session dropped, we had neither a usable weights file nor any way to pick the
training up where it left off. We started over.

This post describes the setup we built so we would never make that mistake again: which GPU we
pick and why, where we put the data, how we take checkpoints (intermediate save files), and
how we resume training after a disconnect. It's Colab-specific, but the logic holds in any
environment that can cut you off.

Colab tiers and GPU types

On Colab's free tier you usually get an NVIDIA T4. The paid tiers unlock L4 and A100
access; which card you actually get depends on availability at that moment — Colab's own
documentation states plainly that GPU types vary over time. So there's no such thing as "I paid
for Pro, my A100 is guaranteed."

The official specs of the three cards, and what each is actually good for:

Here's how we use them: we validate the data pipeline and augmentation settings with a few
epochs on a T4, then run the long final training on a card with more VRAM. At imgsz=1280, the T4's
16 GB noticeably limits the batch size.

The paid tiers run on "compute unit" consumption, and a more powerful card burns
more units per hour. Prices vary by country and change over time, so we're not
quoting numbers here; check Colab's own pricing page (as of 2026 the plan structure
is Free / Pro / Pro+).

Session and idle limits

The most important thing to know here: Google does not officially publish these limits. The Colab FAQ
says the limits aren't published precisely because they can change over time. The
"90 minutes idle, 12 hours maximum" figures floating around the internet are based on observation; all
the FAQ actually says is that on the free tier notebooks can run for "at most 12 hours, depending on
availability and your usage patterns."

The one point that can be confirmed officially is background execution: Colab Pro+ supports
up to 24 hours of uninterrupted code execution as long as you have enough compute units — meaning
training keeps going even if you close the tab. On the other tiers, if the browser connection
drops, the session goes with it.

The practical takeaway: instead of trying to guess the limit, drive the cost of a disconnect close to zero.
Your training should be built on the assumption that it can be cut off at any moment.

Don't leave your data on Drive, copy it to /content

Mounting Drive is a one-liner:

from google.colab import drive
drive.mount('/content/drive')

from google.colab import drive
drive.mount('/content/drive')

A mounted Drive looks like a local folder, but it isn't — every read goes over the network
through FUSE. With one large file you won't notice; with tens of thousands of small JPEGs it turns
into a clear bottleneck. Community benchmarks show reads from Drive running several times slower
than local disk, and that means your epoch time is set by the data pipeline, not the GPU.

The right layout: keep the dataset on Drive as a single zip, then at the start of the session copy it under /content
and unzip it there.

!cp /content/drive/MyDrive/suas/dataset.zip /content/
!unzip -q /content/dataset.zip -d /content/dataset
!df -h /content          # how much room is left on the ephemeral disk?
!nvidia-smi --query-gpu=name,memory.total --format

!cp /content/drive/MyDrive/suas/dataset.zip /content/
!unzip -q /content/dataset.zip -d /content/dataset
!df -h /content          # how much room is left on the ephemeral disk?
!nvidia-smi --query-gpu=name,memory.total --format

The fastest way to find out where the bottleneck is: run nvidia-smi during training.
If GPU utilization stays low, the problem isn't the model, it's data loading.

cache='disk' or cache=True?

Ultralytics' cache argument caches images. Its default value is False.

At our imgsz=1280 scale, cache=True filled the session's memory and crashed the runtime —
and the error message doesn't say so outright, the session just quietly restarts. cache='disk'
is the safer middle ground at that scale.

Checkpointing: taking intermediate saves with save_period

By default (save=True) Ultralytics updates last.pt and best.pt at the end
of every epoch. The save_period argument, on the other hand, produces numbered intermediate copies, and its default is -1,
meaning off.

from ultralytics import YOLO

model = YOLO("yolo11m.pt")
model.train(
    data="/content/dataset/data.yaml",
    epochs=300,          # the starting point the docs recommend
    imgsz=1280,
    batch=-1,            # auto mode targeting ~60% GPU memory
    cache="disk",
    save_period=10,      # a separate checkpoint every 10 epochs
    project="/content/runs",
    name="yolo11m_1280",
)

from ultralytics import YOLO

model = YOLO("yolo11m.pt")
model.train(
    data="/content/dataset/data.yaml",
    epochs=300,          # the starting point the docs recommend
    imgsz=1280,
    batch=-1,            # auto mode targeting ~60% GPU memory
    cache="disk",
    save_period=10,      # a separate checkpoint every 10 epochs
    project="/content/runs",
    name="yolo11m_1280",
)

batch=-1 makes life easier: per the docs, it picks a batch size that targets roughly 60% GPU memory
utilization. It saves you from hand-tuning every time you move between cards.

Note that the project path is under /content, not on Drive. We leave backing up to Drive as a
separate job:

import subprocess
subprocess.Popen(["bash", "-c",
    "while true; do rsync -a /content/runs/ "
    "/content/drive/MyDrive/suas/runs/; sleep 600; done"])

import subprocess
subprocess.Popen(["bash", "-c",
    "while true; do rsync -a /content/runs/ "
    "/content/drive/MyDrive/suas/runs/; sleep 600; done"])

This loop copies the outputs to Drive every 10 minutes. Training keeps writing to local disk while
the backup runs in the background. When the session drops, you lose at most 10 minutes of progress.

Picking up where you left off with resume=True

When a disconnect hits, what you do is resume training from last.pt:

from ultralytics import YOLO

model = YOLO("/content/drive/MyDrive/suas/runs/yolo11m_1280/weights/last.pt")
model.train(resume=True)

from ultralytics import YOLO

model = YOLO("/content/drive/MyDrive/suas/runs/yolo11m_1280/weights/last.pt")
model.train(resume=True)

The command-line equivalent:

yolo train model=.../weights/last.pt resume

yolo train model=.../weights/last.pt resume

resume=True restores not just the weights but also the optimizer state, the learning-rate scheduler and
the epoch counter. That's why "resume" really does mean resume; it isn't the same thing as loading the
weights by hand and starting a fresh run.

Two important notes: at least one epoch must have completed (otherwise there's no state to
restore), and the training arguments are read from the record inside the checkpoint — passing values like epochs
again won't have the effect you expect.

Common pitfalls

1. Writing training output straight to Drive. Passing project=/content/drive/... means a write over the
network at the end of every epoch. Besides being slow, if a disconnect lands mid-write,
last.pt can be left corrupted. Write locally, copy periodically.

2. Not taking numbered checkpoints. With the default save_period=-1, all you have is a
last.pt that keeps getting overwritten. If it goes bad, there's no point left to fall back to.

3. The folder name silently changing. Because exist_ok=False is the default, a second run
lands in a new folder like yolo11m_12802. Then the "which last.pt?" question starts. Manage the folder
name deliberately.

4. Blaming the GPU. If epoch time is longer than you expected, look at the data pipeline first. Low GPU
utilization doesn't mean the card is slow; it means the data isn't arriving fast enough.

5. A powerful card sitting idle. An A100 session left open after training has finished quietly burns
through compute units. Add a step at the end of training that shuts the session down.

6. Assuming you can close the tab. Background execution is only possible on Pro+ and only with enough
compute units. Otherwise the browser has to stay open.

Practical summary

1. Keep the data on Drive as a zip and unzip it into /content at the start of the session. Reading directly
from Drive turns the data pipeline into a bottleneck.

2. Start with cache='disk'. cache=True is the fastest, but with large images it takes the
session down.

3. Set save_period. Trusting a single last.pt is trusting a backup with exactly one copy.

4. Write the output locally and copy it to Drive periodically. You decide how much progress you can afford to lose —
for us it's 10 minutes.

5. On a disconnect: last.pt + resume=True. The optimizer and scheduler state come back too;
don't try to pass the arguments again.

6. Do a 2-3 epoch dry run before you launch the long one. Path errors, class-count
mismatches and memory problems show up in the first epoch — not in hour three.