<@ULVA73B9P> I want to use the prefect client in m...
# ask-marvin
r
@Marvin I want to use the prefect client in my python code using
Copy code
async with get_client() as client:...
But my prefect self hosted server isn't located at localhost. How can I provide the prefect client, the API url and auth string ? I've tried doing
Copy code
os.environ["PREFECT_PROFILES_PATH"] = "/dev/null"

    # Set the Prefect API URL
    os.environ["PREFECT_API_URL"] = PREFECT_API_URL

    # Set the Basic Auth string (username:password format)
    # Prefect will use this for Basic Authentication
    os.environ["PREFECT_API_AUTH_STRING"] = f"{PREFECT_USERNAME}:{PREFECT_PASSWORD}"
But in the end, it seems like it's still trying to contact localhost as defined in my prefect profile :
Copy code
🚀 you are connected to:
<http://127.0.0.1:4200>
PREFECT_PROFILE='ephemeral'
PREFECT_API_AUTH_STRING='********' (from profile)
PREFECT_API_URL='<http://127.0.0.1:4200/api>' (from profile)
PREFECT_RESULTS_PERSIST_BY_DEFAULT='true' (from profile)
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE='true' (from profile)
m
âś… thought for 385.6 seconds
You’re on the right track—`get_client()` pulls its URL/auth from Prefect’s settings system. In Prefect 3.x, those settings come from (highest → lowest): direct constructor args, environment variables, .env/file secrets, local config files, then profiles. If you’re still seeing “(from profile)”, it means the settings were already loaded before your env vars were set. Here are reliable ways to point the client at a non-localhost API and set auth: Option 1: Set env vars before Prefect settings are loaded - Make sure you set env vars before any code that might touch Prefect settings (including some imports that may access settings).
Copy code
import os

# Set these BEFORE importing or calling anything that uses Prefect settings
os.environ["PREFECT_API_URL"] = "<http://your-host:4200/api>"
os.environ["PREFECT_API_AUTH_STRING"] = "username:password"  # or use PREFECT_API_KEY

from prefect import get_client

async with get_client() as client:
    ...
Tip: The URL should include
/api
for Prefect Server. Option 2: Override settings just for this block - This guarantees a fresh override even if settings were already cached.
Copy code
from prefect.settings import temporary_settings, PREFECT_API_URL, PREFECT_API_AUTH_STRING
from prefect import get_client

with temporary_settings({
    PREFECT_API_URL: "<http://your-host:4200/api>",
    PREFECT_API_AUTH_STRING: "username:password",  # or use PREFECT_API_KEY
}):
    async with get_client() as client:
        ...
Option 3: Bypass
get_client()
and instantiate the client explicitly - This gives you full control without relying on profiles or env vars.
Copy code
from prefect.client.orchestration import PrefectClient

async with PrefectClient(
    api="<http://your-host:4200/api>",
    auth_string="username:password",  # or api_key="..."
) as client:
    ...
Verify what Prefect will use - From your shell, you can confirm the sources:
Copy code
prefect config view --show-sources
- If you prefer to persist the change in your profile (instead of env vars), set it:
Copy code
prefect config set PREFECT_API_URL="<http://your-host:4200/api>"
prefect config set PREFECT_API_AUTH_STRING="username:password"  # or PREFECT_API_KEY="..."
Notes - In Prefect 3.x, use
PREFECT_API_URL
,
PREFECT_API_AUTH_STRING
(Basic), or
PREFECT_API_KEY
(Bearer). There’s no
PREFECT_API_ROOT_URL
. - If both Basic and API key are set, Basic (
PREFECT_API_AUTH_STRING
) will be used. -
get_client()
does not accept a base URL/auth parameter; it only takes
httpx_settings
. Use
temporary_settings
or instantiate
PrefectClient
directly to override. If you share a small code snippet showing where you set the env vars relative to your Prefect imports/usage, I can point out exactly why it’s still reading from the profile.
r
@Marvin It works thank you, I've another question : I've setup flows persistance using a shared storage on s3. I can see that the data is there in the s3 bucket. When I retrieve my flow's data using the prefect client, I can get the storage key which is the right one. But when I try retrieving state result using the client, It raises a "All connection attempts failed" error, why ? Here's my code :
Copy code
flow_run = await client.read_flow_run(UUID(flow_run_id))
state = flow_run.state
result = await state.result() -> raised here
m
âś… thought for 399.8 seconds
Short answer: Prefect doesn’t proxy results through the API. When you call
state.result()
, your Python process downloads the payload directly from the configured result storage (S3 in your case) using s3fs/fsspec. If your current environment can’t reach that bucket or doesn’t have credentials/deps, you’ll see “All connection attempts failed” even though the storage key is correct. What to check - Use the async method in async code - Prefer
await state.aresult(...)
in async code.
state.result()
is the sync helper. - Ensure required deps are installed where you call result() - Install s3fs (and boto3):
pip install s3fs boto3
(or include them in your project env) - Without s3fs/boto3, S3 reads will fail. - Ensure AWS credentials are available in the environment running
state.result()
- The worker that wrote the result may have had creds, but your current process may not. - Provide credentials via env vars (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, optional AWS_SESSION_TOKEN, AWS_REGION) or store them in the block (if appropriate). - If the bucket is private, anonymous access will fail with a similar message. - If using an S3-compatible service (MinIO/Ceph), set endpoint/config in the block - Your RemoteFileSystem / S3 block should include endpoint_url and addressing style if needed, for example:
Copy code
RemoteFileSystem(
      basepath="<s3://my-bucket/prefix/>",
      settings={
          "key": "...", "secret": "...",
          "client_kwargs": {
              "endpoint_url": "<https://minio.example.com>",
              "region_name": "us-east-1"
          },
          # For some S3-compatible services you may need:
          # "config_kwargs": {"s3": {"addressing_style": "path"}}
      }
  )
- Network/VPC routing - If your bucket is reachable only from a VPC/private network, a local laptop (or another host) may fail to connect. Quick diagnostics - Confirm the result metadata and referenced storage block
Copy code
flow_run = await client.read_flow_run(UUID(flow_run_id))
state = flow_run.state

# Inspect where Prefect thinks the result lives
meta = state.data.result_metadata  # contains storage_key and storage_document_id
print(meta.storage_key, meta.storage_document_id)

# Optional: fetch the storage block doc to verify basepath/settings
block_doc = await client.read_block_document(meta.storage_document_id)
print(block_doc.block_type_slug, block_doc.name)
print(block_doc.data)  # shows basepath/settings that will be used for S3
- Test S3 access directly from this environment
Copy code
import s3fs
fs = s3fs.S3FileSystem()  # picks up your AWS_* env vars or ~/.aws/credentials
# Does the object exist?
print(fs.exists("<s3://your-bucket/path/from/meta.storage_key>"))
# Try to read a few bytes
with fs.open("<s3://your-bucket/path/from/meta.storage_key>", "rb") as f:
    print(f.read(16))
If that fails with a similar error, it’s a credentials/connectivity issue outside Prefect. - Retrieve result with retries (default is 10 attempts)
Copy code
# Async
value = await state.aresult(raise_on_failure=True, retry_result_failure=True)

# Sync
value = state.result(raise_on_failure=True, retry_result_failure=True)
Common fixes - Install deps in the reading environment:
pip install s3fs boto3
- Provide AWS credentials where you call
state.result()
- If using S3-compatible storage, ensure your block’s settings include the correct endpoint and addressing style. - If you don’t want to put credentials in the block, ensure the environment running the read has them via env/instance profile. If you can share: - The block type/slug used for result storage - Whether this environment has AWS_* env vars or an IAM role - Whether you’re using AWS S3 or an S3-compatible service …I can suggest the exact configuration to resolve the connection error.
r
@Marvin I've retrieve the complete error, here are infos : ConnectError('All econnection attempts failed') request = <Request('GET', 'http://127.0.0.1:4200/api/block_documents/fd335865-b732-479f-b092-df8f7b09ffcf?include_secrets=true')> Is it normal that the url tries to contact localhost while the prefect client was configured to use another prefect api ?
m
âś… thought for 77.2 seconds
Great detail — that error explains it. What’s happening: - When you call `state.result()`/`state.aresult()`, Prefect needs to fetch the result-storage block (with secrets) to read from S3. It does this by calling the Prefect API at
/block_documents/{id}?include_secrets=true
. - That internal call uses the active Prefect settings context. It does not reuse your explicitly created
client
instance. - In your case, that internal call is picking up the profile default (localhost), hence the GET to
<http://127.0.0.1:4200/api/>...
. So yes, seeing localhost there means the settings used during
state.result()
are not the same as the ones you configured for your earlier client call. Fixes Pick one of these patterns so the settings are correct when
state.aresult()
runs: 1) Use temporary_settings around the call
Copy code
from uuid import UUID
from prefect import get_client
from prefect.settings import temporary_settings, PREFECT_API_URL, PREFECT_API_AUTH_STRING

with temporary_settings({
    PREFECT_API_URL: "<http://your-host:4200/api>",
    PREFECT_API_AUTH_STRING: "username:password",  # or use PREFECT_API_KEY
}):
    async with get_client() as client:
        flow_run = await client.read_flow_run(UUID(flow_run_id))
        state = flow_run.state
        result = await state.aresult()  # async version
2) Set environment variables before Prefect settings are used
Copy code
import os
os.environ["PREFECT_API_URL"] = "<http://your-host:4200/api>"
os.environ["PREFECT_API_AUTH_STRING"] = "username:password"  # or PREFECT_API_KEY

from uuid import UUID
from prefect import get_client

async with get_client() as client:
    flow_run = await client.read_flow_run(UUID(flow_run_id))
    state = flow_run.state
    result = await state.aresult()
Notes - Prefer
await state.aresult()
in async code;
state.result()
is the sync helper. - If you instantiated
PrefectClient(api=...)
directly without setting
PREFECT_API_URL
,
state.aresult()
will still open its own client using settings (and hit the profile’s URL). Use one of the two patterns above so the settings context is correct. - Ensure s3fs/boto3 and AWS creds are available in the environment running this code so the S3 read can succeed once the API call is fixed. Quick sanity check - Run:
Copy code
prefect config view --show-sources
and confirm
PREFECT_API_URL
is coming from env/temporary settings (not profile) when you execute the code that calls
state.aresult()
. If you’re still seeing localhost after wrapping with
temporary_settings
, share the snippet around the
aresult()
call and how you’re currently constructing the client, and I’ll pinpoint the mismatch.