< Marvin> is it possible to run a flow locally as a simple p Prefect Community #ask-marvin

Join Slack

<@ULVA73B9P> is it possible to run a flow locally ...

# ask-marvin

gee

07/16/2025, 2:00 PM

@Marvin is it possible to run a flow locally as a simple python script, without logging to prefect?

Marvin

07/16/2025, 2:00 PM

✅ thought for 65.4 seconds

Marvin

07/16/2025, 2:01 PM

Yes, it's absolutely possible to run a Prefect flow locally as a simple Python script! Here's a straightforward example:

Copy code

python
from prefect import flow, task

@task
def say_hello(name):
    return f"Hello {name}!"

@flow
def my_flow(name: str = "world"):
    message = say_hello(name)
    print(message)

if __name__ == "__main__":
    my_flow()

You can save this as

my_flow.py

and run it directly with:

Copy code

bash
python my_flow.py

The flow will execute just like any other Python script. While Prefect will still maintain some local logging (it's part of how the framework tracks execution), you don't need to connect to any Prefect server or API. This is one of the nice features of Prefect 3.x - flows are just Python functions with superpowers. You can develop and test them locally without any additional infrastructure or configuration. If you want to see the flow in action with different parameters, you can also call it directly in a Python REPL or Jupyter notebook:

Copy code

python
my_flow("Marvin")  # Will print "Hello Marvin!"

The

@flow

decorator adds functionality like automatic logging and state tracking, but it doesn't require any external connections to work. This makes it great for local development and testing. Would you like me to explain more about what functionality is still available when running flows locally, or would you like to see more complex examples?

gee

07/16/2025, 2:02 PM

@Marvin when I run my flow locally it still logs it on prefect server.

Marvin

07/16/2025, 2:02 PM

✅ thought for 40.0 seconds

Nate

07/16/2025, 2:03 PM

hmm we might be missing this in the docs but

Copy code

my_flow.fn()

Marvin

07/16/2025, 2:03 PM

To prevent your flow from logging to the Prefect server, you have a few options: 1. Set the environment variable before running your script:

Copy code

bash
export PREFECT_LOGGING_TO_API_ENABLED=False
python your_flow.py

2. Set it in your Python script before defining your flow:

Copy code

python
import os
os.environ["PREFECT_LOGGING_TO_API_ENABLED"] = "False"

from prefect import flow, task
# ... rest of your flow code

3. Use Prefect's settings API in your code:

Copy code

python
from prefect.settings import Setting
from prefect.context import settings

with settings.update({"PREFECT_LOGGING_TO_API_ENABLED": False}):
    my_flow()

If you want to make this change permanent for your local development environment, you can add the environment variable to your shell's configuration file (like

.bashrc

.zshrc

Copy code

bash
echo "export PREFECT_LOGGING_TO_API_ENABLED=False" >> ~/.bashrc

This will prevent all your local flow runs from being logged to the server while still maintaining local execution capabilities and local logging. Note that this only affects the logging to the API - your flow will still work normally and maintain local logs, but won't send any information to the Prefect server. Would you like me to explain more about other logging configurations you can customize, or would you like to know how to selectively enable/disable this for specific flows?

Nate

07/16/2025, 2:04 PM

there's also a

disable_logging

context manager and oh yeah you can disable

PREFECT_LOGGING_TO_API_ENABLED

with

temporary_settings

i guess too

gee

07/16/2025, 2:04 PM

Thanks. Anything that doesn't involve changing the script code?

Nate

07/16/2025, 2:05 PM

well do you just care about logging or do you want to avoid saving any data to the db

gee

07/16/2025, 2:05 PM

Yeah any data to the db. Just want a fully local mode

Nate

07/16/2025, 2:05 PM

maybe just switch to a local profile?

gee

07/16/2025, 2:06 PM

Ok haven't had a look at profiles yet

Nate

07/16/2025, 2:08 PM

yea so you can make one that points at your server and then have another one called

local

or something that either uses an ephemeral server or a different database or whatever you want

Untitled

👍 1

gee

07/16/2025, 2:14 PM

For future reference

prefect profile populate-defaults

creates cloud/local/test profiles. The test profile starts and stop a server before and after running the flow making it portable. The only way to achieve what I wanted here (don't send anything to any server) is by changing the script code and call flow.fn() instead. Note that this could be easily done either with an if in the

if __name__ == "__main__"

block or importing. the flow into another script.

Nate

07/16/2025, 2:16 PM

yea there's also

prefect_test_harness

which we use for testing. but thats still writing to a tmp db. the only way to totally avoid writing data to the configured API is to call the underlying functions via

.fn

or by literally removing the decorator

✅ 1

gee

07/16/2025, 2:16 PM

@Marvin what is the ephemeral profile? It seems to be a default but if I try to switch to it, I get the following error

Copy code

⠹ Checking API connectivity...
Error authenticating with Prefect Cloud using profile 'ephemeral'

Marvin

07/16/2025, 2:16 PM

✅ thought for 41.2 seconds

Marvin

07/16/2025, 2:17 PM

✅ thought for 26.7 seconds

Marvin

07/16/2025, 2:17 PM

Let me clarify what the ephemeral profile is and what's likely happening with your error. The ephemeral profile is a special default profile in Prefect that: 1. Runs a temporary, in-memory server 2. Doesn't require authentication 3. Is perfect for local development and testing The error you're seeing suggests that the system is trying to authenticate with Prefect Cloud when using the ephemeral profile, which isn't its intended use. This typically happens when there's a configuration mismatch. To fix this, you have a few options: 1. Use the ephemeral profile correctly (for local development):

Copy code

bash
prefect profile use ephemeral

And ensure these environment variables are NOT set: -

PREFECT_API_KEY

PREFECT_API_URL

(should be local) 2. Create a new profile for Prefect Cloud (if that's what you want):

Copy code

bash
prefect profile create cloud
prefect profile use cloud
prefect cloud login

3. Check your current configuration:

Copy code

bash
prefect profile inspect

This will show you what settings are active in your current profile. The ephemeral profile is designed to work out-of-the-box for local development without any authentication. If you're seeing an authentication error, it likely means that: - There might be conflicting environment variables set - The profile might have been modified from its default state - You might have Prefect Cloud settings that are interfering Would you like me to help you: 1. Reset your ephemeral profile to its default state? 2. Set up a proper Prefect Cloud profile? 3. Check your current configuration to identify what's causing the conflict? Just let me know which path you'd like to explore and I'll provide more specific guidance.

Marvin

07/16/2025, 2:17 PM

Ah, now I can explain what's happening. The error you're seeing occurs because the ephemeral profile is trying to authenticate with Prefect Cloud, which isn't its intended use case. The ephemeral profile is a special profile that: 1. Runs a temporary, in-memory Prefect server 2. Is meant for local development and testing 3. Doesn't require authentication 4. Is automatically used after a fresh Prefect installation The error you're seeing suggests that your ephemeral profile might be incorrectly configured to try connecting to Prefect Cloud instead of running in local mode. Here's how to fix this: 1. First, let's check your current profiles:

Copy code

bash
prefect profile ls

2. You can create a new local profile:

Copy code

bash
prefect profile create local

3. Then switch to it:

Copy code

bash
prefect profile use local

4. To ensure it's set up for local development, you can set these settings:

Copy code

bash
prefect config set PREFECT_API_URL="<http://127.0.0.1:4200/api>"
prefect config set PREFECT_SERVER_ALLOW_EPHEMERAL_MODE=true

If you want to inspect your current profile settings:

Copy code

bash
prefect profile inspect

The ephemeral profile should work without authentication when properly configured for local development. If you're seeing authentication errors, it likely means the profile is trying to connect to Prefect Cloud instead of running in local mode. Would you like me to help you set up a proper local development profile, or would you like to see more details about how the ephemeral profile works?

gee

07/16/2025, 2:21 PM

@Marvin Here is my profiles.toml file it does look like the ephemeral profile is pointing to prefect-cloud. Any idea what might have happened?

Copy code

[profiles.ephemeral]
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE = "true"
PREFECT_API_KEY = "pnu_<NOPE>
PREFECT_API_URL = "<https://api.prefect.cloud/api/accounts/5f98...4fb99/workspaces/8aaf2787-...4cf52a5>"

[profiles.local]
PREFECT_API_URL = "<http://127.0.0.1:4200/api>"

[profiles.test]
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE = "true"
PREFECT_API_DATABASE_CONNECTION_URL = "sqlite+aiosqlite:///:memory:"

[profiles.cloud]

[profiles.prefect-cloud]
PREFECT_API_URL = "<https://api.prefect.cloud/api/accounts/5f98.....fb99/workspaces/8aaf27.....f52a5>"
PREFECT_API_KEY = "pnu_<NOPE>"

Marvin

07/16/2025, 2:21 PM

✅ thought for 21.2 seconds

Marvin

07/16/2025, 2:21 PM

Copy code

bash
prefect profile populate-defaults

This will reset the default profiles (including ephemeral) to their original state while preserving your custom profiles. Your profiles.toml should then look more like this:

Copy code

toml
[profiles.ephemeral]
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE = "true"
PREFECT_API_URL = "<http://127.0.0.1:4200/api>"

[profiles.local]
PREFECT_API_URL = "<http://127.0.0.1:4200/api>"

[profiles.test]
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE = "true"
PREFECT_API_DATABASE_CONNECTION_URL = "sqlite+aiosqlite:///:memory:"

[profiles.cloud]
PREFECT_API_URL = "<https://api.prefect.cloud/api/accounts/...>"
PREFECT_API_KEY = "your-api-key"

[profiles.prefect-cloud]
PREFECT_API_URL = "<https://api.prefect.cloud/api/accounts/...>"
PREFECT_API_KEY = "your-api-key"

Looking at your current setup: 1. Your

ephemeral

profile has Cloud settings that should be in the

cloud

prefect-cloud

profile 2. You have both a

cloud

and

prefect-cloud

profile (you might want to consolidate these) 3. Your

local

profile looks correct 4. Your

test

profile looks correct After running

populate-defaults

, try switching to the ephemeral profile again:

Copy code

bash
prefect profile use ephemeral

This should now work without any authentication errors since it will be properly configured for local development. Would you like me to explain how to properly manage multiple profiles or help you consolidate your cloud profiles?

Open in Slack

Previous Next