https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> In self hosted prefect version 3.7.0 ...
# ask-marvin
r
@Marvin In self hosted prefect version 3.7.0 in kubernetes, jobs run in pods. But I now have a peer certificate error issuer not found, but perhaps the real insight is that the pod's log has "creating virtual environment at: .venv". I never saw there before. what do you think it going on here?
m
✅ thought for 171.3 seconds
Yeah — I think those two symptoms are probably connected. What changed: in newer Prefect 3.x behavior, if the run environment contains a project setup that Prefect can execute with `uv`/`pyproject.toml`, the flow run may start by invoking 
uv
, which creates a local virtualenv like 
.venv
inside the job pod. So seeing:
Copy code
text
creating virtual environment at: .venv
usually means the pod is no longer just running your already-baked Python environment directly — it is trying to materialize a project environment at runtime. Why that matters for your cert error: creating that 
.venv
often means 
uv
is also trying to resolve/install dependencies from PyPI or a private package index. If the pod image or cluster environment does not trust the CA chain for that endpoint, you get errors like peer certificate issuer not found. So my read is: 1. The cert error may not be about Prefect API connectivity itself 2. It may be happening during dependency resolution / package download inside the run pod 3. The 
.venv
message is a strong clue that 
uv
is doing work it didn’t do before Most likely causes - Your deployment source now includes a 
pyproject.toml
- 
prefect
is listed as a dependency there, which can trigger 
uv
-based execution - 
uv
exists in the image / PATH - The pod cannot validate TLS for: - PyPI - an internal package index - a git source over HTTPS What I’d check first 1. Inspect the full pod logs around that message and look for what URL/certificate check is failing 2. Exec into a pod and check:
Copy code
text
   env | grep -i -E 'ssl|cert|uv|pip'
   ls /etc/ssl/certs
3. Check whether your image actually has CA certs installed 4. Check whether your flow code package now has a 
pyproject.toml
Very plausible explanation If you upgraded to 3.7.0 and didn’t otherwise change much, Prefect may now be detecting the project and launching it with 
uv
, which is new enough that it explains “I never saw 
.venv
there before”. Good fixes - Best simple fix: make the job image contain all needed dependencies already, so the run pod does not need to build a fresh env at runtime - If possible, avoid having the run depend on runtime installs - Ensure the pod trusts your CA bundle if you use a private registry/index - If this is triggered by project metadata, consider whether 
prefect
needs to be in the project dependencies versus installed in the base image If you want, I can help you narrow it down quickly if you paste: - the full cert error text - whether your deployment uses 
pyproject.toml
- your Kubernetes work pool job image / base image - whether you pull from public PyPI or a private package index Relevant background: - Issue mentioning uv behavior in run environments - Prefect repository
r
@Marvin The deployment source is defined in a prefect.yaml pull step that uses git clone. The image does seems to have the certs in it when I do your env grep and ls (above). The pod's log do not say which cert is failing but it would look for the packages in a private pypi. The image does seems to have the python dependencies already there if I docker run into it and do a uv pip list.
m
✅ thought for 26.2 seconds
That makes the picture a lot clearer. My best guess: the pod is cloning your code, then Prefect/uv is treating that cloned repo as a project and creating a fresh 
.venv
anyway — even though your image already has the dependencies installed. So you likely have two environments in play: 1. Your container image environment - already has Python deps - works when you 
docker run
into it 2. A new runtime 
.venv
created in the working directory after 
git clone
 - this is the one trying to talk to your private PyPI - this is the one failing certificate validation That would explain all of your observations. Why the git clone matters With a 
prefect.yaml
pull step using 
git clone
, the run pod gets a checked-out project directory at runtime. If that repo contains a 
pyproject.toml
, Prefect 3.7.x may decide to launch the flow with 
uv run --project ...
, which creates 
.venv
in that cloned workspace. So even if the base image is fine, the runtime project env creation can still fail against private PyPI. Why certs can still fail even if cert files exist Having files in 
/etc/ssl/certs
is necessary, but not always sufficient. Common gotchas: - the private PyPI uses an internal CA not in that trust bundle - 
uv
is using a cert path different from what your shell/Python test used - your image environment works, but the newly created 
.venv
/ runtime subprocess inherits different env - your private index settings in 
pyproject.toml
/ uv config point to a host whose cert chain is incomplete What I think is happening, concretely Something like:
Copy code
text
pod starts
-> prefect pull step runs git clone
-> repo includes pyproject.toml
-> prefect decides to use uv for project execution
-> uv creates .venv
-> uv attempts to resolve/sync deps from private pypi
-> TLS validation fails: issuer not found
Most useful next checks 1. Look in the cloned repo for 
pyproject.toml
 - especially whether 
prefect
is listed in dependencies - also check for 
[tool.uv]
index settings 2. Compare package sources in the image vs runtime project - 
uv pip list
in the image only proves the image env is good - it does not prove runtime 
uv run --project
won’t try to sync again 3. Check whether the pod has env vars for certs/indexes for example:
Copy code
text
   SSL_CERT_FILE
   REQUESTS_CA_BUNDLE
   UV_INDEX
   UV_DEFAULT_INDEX
   PIP_INDEX_URL
   PIP_EXTRA_INDEX_URL
4. Inspect whether the private PyPI CA is actually in the trust chain used by the pod If the issuer is internal, you may need to mount/add that CA explicitly, not just rely on system certs. What I would try first Option A: stop the runtime project env creation If your image already contains everything needed, this is usually the cleanest path. In practice that means making sure the flow run uses the image’s existing Python environment instead of a repo-managed 
uv
project environment. The strongest clue/trigger is usually the presence of 
pyproject.toml
in the pulled repo. A practical test: - temporarily deploy from code that does not include 
pyproject.toml
- or remove 
prefect
from that project dependency declaration if it’s there - rerun and see if the 
.venv
creation message disappears If 
.venv
disappears and the run works, you’ve confirmed the cause. Option B: make the runtime 
.venv
trust your private PyPI If you actually want this 
uv
behavior, then fix the trust path for the private index by explicitly setting the CA bundle env vars in the job/pod spec. For example, in your job template/container env:
Copy code
text
SSL_CERT_FILE=/path/to/your/ca-bundle.crt
REQUESTS_CA_BUNDLE=/path/to/your/ca-bundle.crt
CURL_CA_BUNDLE=/path/to/your/ca-bundle.crt
And ensure that bundle includes the issuer for your private PyPI cert. What I suspect is most likely in your case Since: - git clone pull step is used - private PyPI is involved - image already has deps
- 
.venv
creation is new …I’d bet the failure is not your original image env. It’s the new uv-managed project env created after clone. Recommended direction If you already ship a fully baked image, I would strongly favor: - don’t rely on runtime dependency installation - use the git clone only for source if necessary - avoid the repo triggering 
uv run --project
in the pod If you want, paste your relevant: - 
prefect.yaml
pull step - 
pyproject.toml
- work pool job variables/template env and I can tell you exactly which part is causing the 
.venv
creation and what minimal change should stop it.
r
@Marvin After a Prefect 3.7.0 Variable.get or Variable.aget when I try to use the variable, let's say it's a dict used as a mapping table, I get this error 
TypeError: 'coroutine' object is not subscriptable
It was called from an async flow.
m
✅ thought for 183.5 seconds
Yep — in an async flow, you need to 
await
it. In Prefect 3.7.0, 
Variable.get()
is context-sensitive. In sync code it returns the value directly, but in async code it dispatches to the async implementation, so without 
await
you get a coroutine object. That’s why this fails:
Copy code
python
from prefect import flow
from prefect.variables import Variable

@flow
async def my_flow():
    mapping = Variable.get("mapping_table")
    value = mapping["some_key"]  # TypeError: 'coroutine' object is not subscriptable
Use either of these instead:
Copy code
python
from prefect import flow
from prefect.variables import Variable

@flow
async def my_flow():
    mapping = await Variable.get("mapping_table")
    value = mapping["some_key"]
    return value
or explicitly:
Copy code
python
from prefect import flow
from prefect.variables import Variable

@flow
async def my_flow():
    mapping = await Variable.aget("mapping_table")
    value = mapping["some_key"]
    return value
Rule of thumb: - sync flow / sync function: 
Variable.get("x")
- async flow / async function: 
await Variable.get("x")
or 
await Variable.aget("x")
The same pattern applies to other variable methods too: - 
await Variable.set(...)
/ 
await Variable.aset(...)
- 
await Variable.unset(...)
/ 
await Variable.aunset(...)
If you want, I can also help you grep for the other 3.7.0 async-dispatch places that might bite you after upgrade.
r
@Marvin but the variable.get and variable.aget is in a utility functions (not a task for flow) that is meant to be multiple purpose and called by both async and normal flows. the utility function goes on to make a notification. So the async flow does not get the variable directly. Can I just add an await in front of the Variable.aget inside the utility function? Or is there a better way to do this?
m
✅ thought for 18.5 seconds
Yes — but only if that utility function itself is 
async
. The key rule is: - if a function calls 
await Variable.aget(...)
, that function must be declared 
async def
- and then its caller must also 
await
it So if you make your shared utility async, it will work great for async flows, but sync callers can’t call it directly without bridging async/sync. What not to do If you have:
Copy code
python
def send_notification(...):
    mapping = await Variable.aget("mapping")  # invalid in a normal def
that won’t work, because 
await
is only legal inside 
async def
. --- ## Best approach If the utility is truly used by both sync and async flows, the cleanest approach is usually: ### Option 1: provide two wrappers - one sync utility - one async utility - both call a shared pure helper for the business logic Example:
Copy code
python
from prefect.variables import Variable

def _build_notification(mapping, key):
    value = mapping[key]
    return f"notify about {value}"

def send_notification(key: str):
    mapping = Variable.get("mapping_table")
    return _build_notification(mapping, key)

async def asend_notification(key: str):
    mapping = await Variable.aget("mapping_table")
    return _build_notification(mapping, key)
Then:
Copy code
python
@flow
def sync_flow():
    msg = send_notification("foo")
    print(msg)

@flow
async def async_flow():
    msg = await asend_notification("foo")
    print(msg)
This is usually the safest and clearest pattern. --- ## Why not rely on one function for both? You can try to make one function work in both sync and async contexts, but it gets awkward fast because: - a sync 
def
cannot 
await
- an async 
def
always returns a coroutine, so sync callers must explicitly run it somehow - “notification” code often also involves async I/O, which makes mixed behavior even trickier Prefect’s own 
Variable.get()
does context dispatch, but your utility function won’t magically do that unless you build similar behavior. --- ## Option 2: make the utility async-only If notification sending is naturally async anyway, another good choice is to standardize on:
Copy code
python
async def send_notification(...):
    mapping = await Variable.get("mapping_table")
    ...
Then: - async flows do 
await send_notification(...)
- sync flows would need a sync wrapper that runs the coroutine But I’d only do that if most callers are async. --- ## Recommended pattern for your case Because you said: - it is a reusable utility - called by both async and normal flows - it fetches a variable and then makes a notification I’d recommend:
Copy code
python
from prefect.variables import Variable

def _prepare_notification(mapping: dict, event_type: str) -> str:
    target = mapping[event_type]
    return f"Send notification to {target}"

def send_notification(event_type: str) -> str:
    mapping = Variable.get("notification_mapping")
    return _prepare_notification(mapping, event_type)

async def asend_notification(event_type: str) -> str:
    mapping = await Variable.get("notification_mapping")
    return _prepare_notification(mapping, event_type)
If the actual send is also different for sync vs async, split that too. --- ## Short answer to your exact question Can I just add an await in front of 
Variable.aget
inside the utility function? - Yes, if you change the utility to 
async def
- No, not if you want the same 
def
function to still be callable directly from sync flows So the better way is usually: - keep the shared logic in a pure helper - expose both a sync and async wrapper If you want, paste your utility function and I can rewrite it into a clean sync/async pair.
2 Views
Open in Slack
PreviousNext
Powered by