<@ULVA73B9P> we load a huggingface one shot classi...
# ask-marvint
Theo Bouwman
12/15/2025, 7:40 PM@Marvin we load a huggingface one shot classification model in our task:
@task(name="initialize_classifier", cache_policy=NONE, cache_key_fn=None, retries=2)
def initialize_classifier_task():
"""Initialize the zero-shot classification model"""
logger = get_run_logger()
# Device setup
if torch.backends.mps.is_available():
device = "mps"
logger.info("Using MPS (Metal) acceleration")
elif torch.cuda.is_available():
device = 0
logger.info("Using CUDA GPU")
else:
device = -1
logger.info("Using CPU")
logger.info(f"Loading model: {MODEL_NAME}...")
classifier = pipeline(
"zero-shot-classification",
model=MODEL_NAME,
device=device,
torch_dtype=torch.float32,
)
logger.info("Model loaded successfully!")
return classifier, device
But sometimes it loads forever, longer than 1 minute, and never stops the task.
But normally it loads in 30 seconds an proceeds wuth the flow as usual.
Also out server and worker memory usage in k8s shows normal values (under 50%)
m
Marvin
12/15/2025, 7:40 PM✅ thought for 207.6 seconds
t
Theo Bouwman
12/15/2025, 7:42 PMBut, i see in k8s that i get a "Job has reached the specified backoff limit" error:
kind: Event
apiVersion: v1
metadata:
name: eggplant-trout-mfqn7.18817aa3533305a0
namespace: prefect
uid: f8d93e47-a4d9-4b8c-a4ff-af8d0c463ef0
resourceVersion: '117497973'
creationTimestamp: '2025-12-15T193610Z'
managedFields:
- manager: kube-controller-manager
operation: Update
apiVersion: v1
time: '2025-12-15T193610Z'
fieldsType: FieldsV1
fieldsV1:
fcount {}
ffirstTimestamp {}
finvolvedObject {}
flastTimestamp {}
fmessage {}
freason {}
freportingComponent {}
fsource
fcomponent {}
ftype {}
involvedObject:
kind: Job
namespace: prefect
name: eggplant-trout-mfqn7
uid: 77a2c4cf-101e-4613-9a76-ed9fdc5304a9
apiVersion: batch/v1
resourceVersion: '117497970'
reason: BackoffLimitExceeded
message: Job has reached the specified backoff limit
source:
component: job-controller
firstTimestamp: '2025-12-15T193610Z'
lastTimestamp: '2025-12-15T193610Z'
count: 1
type: Warning
eventTime: null
reportingComponent: job-controller
reportingInstance: ''
m
Marvin
12/15/2025, 7:44 PMThanks for the detail — this is a common gotcha when loading large HF/torch models inside a task.
What’s happening
- Prefect will only stop a task if you set a timeout. However, even with
timeout_seconds Copy code
@task(
name="initialize_classifier",
retries=2,
retry_delay_seconds=[15, 60],
retry_jitter_factor=0.3,
timeout_seconds=90, # soft timeout
)
def initialize_classifier_task():
logger = get_run_logger()
if torch.backends.mps.is_available():
device = "mps"
<http://logger.info|logger.info>("Using MPS (Metal) acceleration")
elif torch.cuda.is_available():
device = 0
<http://logger.info|logger.info>("Using CUDA GPU")
else:
device = -1
<http://logger.info|logger.info>("Using CPU")
<http://logger.info|logger.info>(f"Loading model: {MODEL_NAME}...")
# Strongly consider preventing network on init to avoid hangs
classifier = pipeline(
"zero-shot-classification",
model=MODEL_NAME,
device=device,
torch_dtype=torch.float32,
# local_files_only=True, # enable if you pre-bundle weights
)
<http://logger.info|logger.info>("Model loaded successfully!")
return classifier, device Copy code
@task(retries=2, retry_delay_seconds=30)
def warmup_model_hard_timeout(timeout=60):
import sys, subprocess, textwrap
code = textwrap.dedent(f"""
import torch
from transformers import pipeline
pipe = pipeline("zero-shot-classification", model="{MODEL_NAME}", device=-1, torch_dtype=torch.float32)
out = pipe("warmup", candidate_labels=["a","b"])
print("READY")
""")
try:
p = subprocess.run([sys.executable, "-c", code], timeout=timeout, capture_output=True, text=True)
p.check_returncode()
if "READY" not in p.stdout:
raise RuntimeError(f"Warmup failed: {p.stdout} {p.stderr}")
return True
except subprocess.TimeoutExpired:
raise RuntimeError(f"Model warmup exceeded {timeout}s (killed)")Marvin
12/15/2025, 7:44 PM- Or restructure each classification to run inside the subprocess (load + infer), accepting the load cost and caching weights on disk.
3) Reduce the chance of hangs during load
- Pre-bundle weights: Bake the model into your image or mount a persistent volume with the HF cache, then set
HF_HOMElocal_files_only=TruepipelinerevisionTRANSFORMERS_VERBOSITY=infoHF_HUB_ENABLE_HF_TRANSFER=1HF_HUB_DISABLE_TELEMETRY=1MODEL_NAMEpipeline(...)t
Theo Bouwman
12/15/2025, 10:12 PM@Marvin What do you mean with this?
Do you need to reuse the in-memory classifier across multiple tasks, or could you move loading + inference into a single step (or a model-serving sidecar)?
m
Marvin
12/15/2025, 10:12 PM✅ thought for 69.3 seconds
Marvin
12/15/2025, 10:13 PMGreat question — I meant the trade-offs between keeping a model in memory inside your Prefect run vs. isolating it in another process/service so you can enforce hard timeouts.
Here are the options and what they imply:
1) Reuse an in-memory classifier inside the Prefect run
- Pattern: One task loads the model, later tasks reuse it (by returning it or via a module-level singleton).
- Pros: Fast inference after the first load; simplest code.
- Cons:
- If the load hangs inside torch/transformers C code, Prefect’s
timeout_seconds6 Views