https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> ython3 ray-flow.py Traceback (most re...
# ask-marvin
s
@Marvin ython3 ray-flow.py Traceback (most recent call last): File "/home/ec2-user/prefect/flows/ray-flow.py", line 2, in <module> from prefect import flow, task File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/__init__.py", line 42, in <module> from prefect.runner import Runner, serve File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/runner/__init__.py", line 1, in <module> from .runner import Runner, serve File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/runner/runner.py", line 72, in <module> from prefect.deployments.deployments import load_flow_from_flow_run File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/deployments/__init__.py", line 1, in <module> import prefect.deployments.base File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/deployments/base.py", line 31, in <module> from prefect.flows import load_flow_from_entrypoint File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/flows.py", line 74, in <module> from prefect.context import PrefectObjectRegistry, registry_from_script File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/context.py", line 569, in <module> GLOBAL_SETTINGS_CONTEXT: SettingsContext = root_settings_context() File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/context.py", line 532, in root_settings_context profiles = prefect.settings.load_profiles() File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/settings.py", line 2044, in load_profiles user_profiles = _read_profiles_from(user_profiles_path) File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/settings.py", line 2017, in _read_profiles_from profiles = [ File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/settings.py", line 2018, in <listcomp> Profile(name=name, settings=settings, source=path) File "/home/ec2-user/.local/lib/python3.9/site-packages/pydantic/v1/main.py", line 347, in init raise validation_error pydantic.v1.error_wrappers.ValidationError: 1 validation error for Profile settings Unknown setting 'PREFECT_SERVER_ALLOW_EPHEMERAL_MODE'. (type=value_error)
m
✅ thought for 140.2 seconds
Thanks for the traceback — the root cause is clear: Prefect is failing to load your active profile because it contains an unknown setting: 
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE
. That setting does not exist in Prefect 3.x (it’s a leftover from older configs), so Pydantic raises a validation error during import. How to fix Because this occurs on import, fix the profile first (before running Prefect/your flow): 1) Check and remove any environment variable - Run:
Copy code
env | grep PREFECT_SERVER_ALLOW_EPHEMERAL_MODE
- If you see it, unset it for the current shell:
Copy code
unset PREFECT_SERVER_ALLOW_EPHEMERAL_MODE
- Also remove it from your shell profile (e.g., ~/.bashrc, ~/.zshrc, systemd unit, etc.) so it doesn’t reappear. 2) Clean up your profiles.toml (most likely culprit) - Open or move aside your user profiles file:
Copy code
mv ~/.prefect/profiles.toml ~/.prefect/profiles.toml.bak
- This prevents Prefect from loading the invalid setting. 3) Recreate default profiles - Now that the bad file is out of the way, you can safely run:
Copy code
prefect profile populate-defaults
prefect profile ls
prefect profile inspect
- If you need a custom profile again, create it with:
Copy code
prefect profile create my-profile
and only add settings from the Prefect 3 settings reference. 4) Re-run your flow
Copy code
python3 ray-flow.py
Notes - If you were trying to enable an “ephemeral server” behavior from older docs, that specific setting is not part of Prefect 3.x. In 3.x, just configure your API target normally: - Prefect Cloud: login and leave 
PREFECT_API_URL
managed by the CLI - Prefect Server (self-hosted): set 
PREFECT_API_URL
to your server API URL (e.g., http://127.0.0.1:4200/api) Docs (Prefect 3.x) - Settings reference: Prefect 3 settings reference - Settings and profiles: Settings and profiles concept If you want, paste the relevant part of your 
~/.prefect/profiles.toml
(with any secrets redacted), and I’ll point out exactly what to keep or remove. Also let me know if you were upgrading from Prefect 2.x — I can help map any other settings that changed.
s
@Marvin no it looks ok and I am on an older prefect version pip3 show prefect Name: prefect Version: 2.14.20 Summary: Workflow orchestration and management. Home-page: https://www.prefect.io Author: Prefect Technologies, Inc. Author-email: help@prefect.io License: Location: /home/ec2-user/.local/lib/python3.9/site-packages Requires: aiosqlite, alembic, anyio, apprise, asgi-lifespan, asyncpg, cachetools, click, cloudpickle, coolname, croniter, cryptography, dateparser, docker, fsspec, graphviz, griffe, httpcore, httpx, importlib-metadata, jinja2, jsonpatch, jsonschema, kubernetes, orjson, packaging, pathspec, pendulum, pydantic, python-dateutil, python-slugify, pytz, pyyaml, readchar, rich, ruamel.yaml, sniffio, sqlalchemy, starlette, toml, typer, typing-extensions, ujson, uvicorn, websockets Required-by: prefect-ray [ec2-user@ip-172-31-44-89 flows]$ cat ~/.prefect/profiles.toml active = "ephemeral" [profiles.ephemeral] PREFECT_SERVER_ALLOW_EPHEMERAL_MODE = "true" PREFECT_API_URL = *****
m
✅ thought for 63.9 seconds
Thanks for the details — that helps. Given your traceback shows imports like 
prefect.runner
and a Pydantic v1 validation error for an unknown setting, your script is actually importing Prefect 3.x at runtime. In Prefect 3, 
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE
is not a valid setting, hence the failure. Your 
pip3 show prefect
output shows 2.14.20, which means 
pip3
and 
python3
are likely pointing at different environments. Quick checks Run these to confirm what your script’s Python sees:
Copy code
which python3
python3 -c "import prefect, sys; print('prefect', prefect.__version__, 'from', prefect.__file__, 'python', sys.executable)"
python3 -m pip show prefect
If that prints a 3.x version, you’ve got a mismatch between pip and python. How to fix (stay on Prefect 2.x) - Ensure the python you run your flow with has Prefect 2.x installed: - Option A: Reinstall Prefect 2 in the interpreter you’re using:
Copy code
python3 -m pip uninstall -y prefect
    python3 -m pip install 'prefect<3' prefect-ray
    python3 -c "import prefect; print(prefect.__version__)"
- Option B (recommended): Use a clean virtual environment so pip and python are aligned:
Copy code
python3 -m venv .venv
    source .venv/bin/activate
    pip install 'prefect<3' prefect-ray
    python -c "import prefect; print(prefect.__version__)"
    python ray-flow.py
Notes - Your 
~/.prefect/profiles.toml
is fine for Prefect 2.x; you do not need to remove 
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE
if you’re staying on 2.x. - The error only appears because Prefect 3 is being imported somewhere on your path; once you’re truly running 2.14.20, that setting will be accepted. If you prefer to migrate to Prefect 3 instead, remove 
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE
from your profile and update your code accordingly, but since you’re using 
prefect-ray
(a 2.x integration), staying on 2.x is likely the right call here. Send me the output of the version/path checks above if you want me to pinpoint the exact mismatch.
s
@Marvin what should I expect to happen when I do python3 ray-flow.py? import time from prefect import flow, task from prefect_ray import RayTaskRunner # from prefect.deployments import DeploymentImage # from dotenv import load_dotenv # import os @task def shout(number): time.sleep(0.5) print(f"#{number}") @flow(task_runner=RayTaskRunner( address="<ray://****:10001>", init_kwargs={"runtime_env": {"pip3": ["prefect-ray"]}}, ), ) def count_to(highest_number): futures = [shout.submit(number) for number in range(highest_number)] for future in futures: future.wait() # Ensure all tasks complete if name == "__main__": count_to(10)
m
✅ thought for 104.4 seconds
Great question. With that script (on Prefect 2.x with prefect-ray installed), here’s what should happen when you run: - Your process initializes a Prefect flow locally and connects to your Ray cluster at ray//&lt;host&gt;10001. - Ten 
shout
tasks are submitted to Ray and run in parallel on Ray workers (each sleeps 0.5s and prints “#<n>”). - The script waits for all tasks to finish, then exits. - If you have a Prefect API configured (PREFECT_API_URL set), you’ll see one flow run with 10 task runs in the Prefect UI; otherwise it runs entirely locally. Important details and fixes for your snippet - Use a valid Ray address string. No angle brackets:
Copy code
address="ray://<host>:10001"
- Fix runtime_env key: Ray expects 
pip
, not 
pip3
.
Copy code
RayTaskRunner(
      address="ray://<host>:10001",
      init_kwargs={"runtime_env": {"pip": ["prefect==2.14.20", "prefect-ray"]}},
  )
Alternatively, pre-install packages on your Ray cluster nodes and omit runtime_env. - Printing vs logging: 
print()
output may land in Ray worker logs, not your local console or Prefect logs. If you want to see output in Prefect, use the run logger:
Copy code
from prefect import get_run_logger

  @task
  def shout(number):
      logger = get_run_logger()
      time.sleep(0.5)
      <http://logger.info|logger.info>(f"#{number}")
- Waiting: 
.wait()
is fine; 
.result()
is useful if you want to raise any task exceptions in your driver process. What you’ll see if things are working - Quick parallel execution: total runtime should be ~0.6–1.5s for 10 tasks (0.5s each) plus overhead. - Ray will log that it’s connecting to the cluster. - In the Prefect UI (if connected), a single flow run with 10 completed task runs. Common pitfalls to watch for - Wrong runtime_env key: using 
pip3
will error; use 
pip
. - Connectivity: if the driver can’t reach ray//&lt;host&gt;10001 (firewall/SG), it will hang or throw connection errors. You can test directly:
Copy code
python3 -c "import ray; ray.init(address='ray://<host>:10001'); print(ray.cluster_resources())"
- Package availability on workers: ensure Prefect (and any task deps) are installed on Ray workers, either via 
runtime_env={"pip": [...]}
or by baking your cluster images. - Mixed Prefect versions: if you see the “Unknown setting 'PREFECT_SERVER_ALLOW_EPHEMERAL_MODE'” error again, your driver is importing Prefect 3.x somewhere. Make sure 
python3
is using the environment with Prefect 2.14.20. Docs - Prefect + Ray (2.x): prefect-ray integration - Task runners (2.x): Task runners overview - Logging (2.x): Logging guide If you share the actual Ray address you’re using (sanitized) and any connection errors, I can help verify connectivity and version compatibility.
s
@Marvin | |_| |_|_\___|_| |___\___| |_| 0|prefect-server | Configure Prefect to communicate with the server with: 0|prefect-server | prefect config set PREFECT_API_URL=http://0.0.0.0:4200/api 0|prefect-server | View the API reference documentation at http://0.0.0.0:4200/docs 0|prefect-server | Check out the dashboard at http://0.0.0.0:4200 0|prefect-server | Traceback (most recent call last): 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/alembic/script/base.py", line 217, in _catch_revision_errors 0|prefect-server | yield 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/alembic/script/base.py", line 425, in _upgrade_revs 0|prefect-server | for script in reversed(list(revs)) 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/alembic/script/revision.py", line 814, in iterate_revisions 0|prefect-server | revisions, heads = fn( 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/alembic/script/revision.py", line 1475, in _collect_upgrade_revisions 0|prefect-server | current_revisions = self.get_revisions(lower) 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/alembic/script/revision.py", line 542, in get_revisions 0|prefect-server | return sum([self.get_revisions(id_elem) for id_elem in id_], ()) 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/alembic/script/revision.py", line 542, in <listcomp> 0|prefect-server | return sum([self.get_revisions(id_elem) for id_elem in id_], ()) 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/alembic/script/revision.py", line 565, in get_revisions 0|prefect-server | return tuple( 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/alembic/script/revision.py", line 566, in <genexpr> 0|prefect-server | self._revision_for_ident(rev_id, branch_label) 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/alembic/script/revision.py", line 637, in _revision_for_ident 0|prefect-server | raise ResolutionError( 0|prefect-server | alembic.script.revision.ResolutionError: No such revision or branch '8bb517bae6f9' 0|prefect-server | The above exception was the direct cause of the following exception: 0|prefect-server | Traceback (most recent call last): 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/starlette/routing.py", line 705, in lifespan 0|prefect-server | async with self.lifespan_context(app) as maybe_state: 0|prefect-server | File "/usr/lib64/python3.9/contextlib.py", line 181, in aenter 0|prefect-server | return await self.gen.__anext__() 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/server/api/server.py", line 591, in lifespan 0|prefect-server | await run_migrations() 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/server/api/server.py", line 517, in run_migrations 0|prefect-server | await db.create_db() 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/server/database/interface.py", line 55, in create_db 0|prefect-server | await self.run_migrations_upgrade() 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/server/database/interface.py", line 63, in run_migrations_upgrade 0|prefect-server | await run_sync_in_worker_thread(alembic_upgrade) 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread 0|prefect-server | return await anyio.to_thread.run_sync( 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/anyio/to_thread.py", line 33, in run_sync 0|prefect-server | return await get_asynclib().run_sync_in_worker_thread( 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread 0|prefect-server | return await future 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 807, in run 0|prefect-server | result = context.run(func, *args) 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/server/database/alembic_commands.py", line 24, in wrapper 0|prefect-server | return fn(*args, **kwargs) 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/server/database/alembic_commands.py", line 53, in alembic_upgrade 0|prefect-server | alembic.command.upgrade(alembic_config(), revision, sql=dry_run) 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/alembic/command.py", line 483, in upgrade 0|prefect-server | script.run_env() 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/alembic/script/base.py", line 549, in run_env 0|prefect-server | util.load_python_file(self.dir, "env.py") 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 116, in load_python_file 0|prefect-server | module = load_module_py(module_id, path) 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 136, in load_module_py 0|prefect-server | spec.loader.exec_module(module) # type: ignore 0|prefect-server | File "<frozen importlib._bootstrap_external>", line 850, in exec_module 0|prefect-server | File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/server/database/migrations/env.py", line 175, in <module> 0|prefect-server | apply_migrations() 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 243, in coroutine_wrapper 0|prefect-server | return run_async_from_worker_thread(async_fn, *args, **kwargs) 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 177, in run_async_from_worker_thread 0|prefect-server | return anyio.from_thread.run(call) 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/anyio/from_thread.py", line 47, in run 0|prefect-server | return asynclib.run_async_from_thread(func, *args) 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 906, in run_async_from_thread 0|prefect-server | return f.result() 0|prefect-server | File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 0|prefect-server | return self.__get_result() 0|prefect-server | File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 0|prefect-server | raise self._exception 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/server/database/migrations/env.py", line 169, in apply_migrations 0|prefect-server | await connection.run_sync(do_run_migrations) 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/sqlalchemy/ext/asyncio/engine.py", line 887, in run_sync 0|prefect-server | return await greenlet_spawn( 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 203, in greenlet_spawn 0|prefect-server | result = context.switch(value) 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/prefect/server/database/migrations/env.py", line 137, in do_run_migrations 0|prefect-server | context.run_migrations() 0|prefect-server | File "<string>", line 8, in run_migrations 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/alembic/runtime/environment.py", line 946, in run_migrations 0|prefect-server | self.get_context().run_migrations(**kw) 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/alembic/runtime/migration.py", line 615, in run_migrations 0|prefect-server | for step in self._migrations_fn(heads, self): 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/alembic/command.py", line 472, in upgrade 0|prefect-server | return script._upgrade_revs(revision, rev) 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/alembic/script/base.py", line 421, in _upgrade_revs 0|prefect-server | return [ 0|prefect-server | File "/usr/lib64/python3.9/contextlib.py", line 137, in exit 0|prefect-server | self.gen.throw(typ, value, traceback) 0|prefect-server | File "/home/ec2-user/.local/lib/python3.9/site-packages/alembic/script/base.py", line 249, in _catch_revision_errors 0|prefect-server | raise util.CommandError(resolution) from re 0|prefect-server | alembic.util.exc.CommandError: Can't locate revision identified by '8bb517bae6f9' 0|prefect-server | Application startup failed. Exiting. 0|prefect-server | Server stopped!
m
Your message was too long, here's your message at the allowed limit: | |_| |_|_\___|_| |___\___| |_| 0|prefect-server | Configure Prefect to communicate with the server with 0|prefect-server | prefect config set PREFECT_API_URL=<http //0 0 0 0 4200/api> 0|prefect-server | View the API reference documentation at <http //0 0 0 0 4200/docs> 0|prefect-server | Check out the dashboard at <http //0 0 0 0 4200> 0|prefect-server | Traceback (most recent call last) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/base py line 217 in _catch_revision_errors 0|prefect-server | yield 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/base py line 425 in _upgrade_revs 0|prefect-server | for script in reversed(list(revs)) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 814 in iterate_revisions 0|prefect-server | revisions heads = fn( 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 1475 in _collect_upgrade_revisions 0|prefect-server | current_revisions = self get_revisions(lower) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 542 in get_revisions 0|prefect-server | return sum([self get_revisions(id_elem) for id_elem in id_] ()) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 542 in <listcomp> 0|prefect-server | return sum([self get_revisions(id_elem) for id_elem in id_] ()) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 565 in get_revisions 0|prefect-server | return tuple( 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 566 in <genexpr> 0|prefect-server | self _revision_for_ident(rev_id branch_label) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 637 in _revision_for_ident 0|prefect-server | raise ResolutionError( 0|prefect-server | alembic script revision ResolutionError No such revision or branch '8bb517bae6f9' 0|prefect-server | The above exception was the direct cause of the following exception 0|prefect-server | Traceback (most recent call last) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/starlette/routing py line 705 in lifespan 0|prefect-server | async with self lifespan_context(app) as maybe_state 0|prefect-server | File /usr/lib64/python3 9/contextlib py line 181 in aenter 0|prefect-server | return await self gen __anext__() 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/server/api/server py line 591 in lifespan 0|prefect-server | await run_migrations() 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/server/api/server py line 517 in run_migrations 0|prefect-server | await db create_db() 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/server/database/interface py line 55 in create_db 0|prefect-server | await self run_migrations_upgrade() 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/server/database/interface py line 63 in run_migrations_upgrade 0|prefect-server | await run_sync_in_worker_thread(alembic_upgrade) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/utilities/asyncutils py line 91 in run_sync_in_worker_thread 0|prefect-server | return await <http //anyio to|anyio to>_thread run_sync( 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/anyio/to_thread py line 33 in run_sync 0|prefect-server | return await get_asynclib() run_sync_in_worker_thread( 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/anyio/_backends/_asyncio py line 877 in run_sync_in_worker_thread 0|prefect-server | return awa
it future 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/anyio/_backends/_asyncio py line 807 in run 0|prefect-server | result = context run(func *args) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/server/database/alembic_commands py line 24 in wrapper 0|prefect-server | return fn(*args **kwargs) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/server/database/alembic_commands py line 53 in alembic_upgrade 0|prefect-server | alembic command upgrade(alembic_config() revision sql=dry_run) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/command py line 483 in upgrade 0|prefect-server | script run_env() 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/base py line 549 in run_env 0|prefect-server | util load_python_file(self dir env py ) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/util/pyfiles py line 116 in load_python_file 0|prefect-server | module = load_module_py(module_id path) 0|prefect-server
s
@Marvin | |_| |_|_\___|_| |___\___| |_| 0|prefect-server | Configure Prefect to communicate with the server with 0|prefect-server | prefect config set PREFECT_API_URL=<http //0 0 0 0 4200/api> 0|prefect-server | View the API reference documentation at <http //0 0 0 0 4200/docs> 0|prefect-server | Check out the dashboard at <http //0 0 0 0 4200> 0|prefect-server | Traceback (most recent call last) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/base py line 217 in _catch_revision_errors 0|prefect-server | yield 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/base py line 425 in _upgrade_revs 0|prefect-server | for script in reversed(list(revs)) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 814 in iterate_revisions 0|prefect-server | revisions heads = fn( 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 1475 in _collect_upgrade_revisions 0|prefect-server | current_revisions = self get_revisions(lower) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 542 in get_revisions 0|prefect-server | return sum([self get_revisions(id_elem) for id_elem in id_] ()) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 542 in <listcomp> 0|prefect-server | return sum([self get_revisions(id_elem) for id_elem in id_] ()) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 565 in get_revisions 0|prefect-server | return tuple( 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 566 in <genexpr> 0|prefect-server | self _revision_for_ident(rev_id branch_label) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 637 in _revision_for_ident 0|prefect-server | raise ResolutionError( 0|prefect-server | alembic script revision ResolutionError No such revision or branch '8bb517bae6f9' 0|prefect-server | The above exception was the direct cause of the following exception 0|prefect-server | Traceback (most recent call last) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/starlette/routing py line 705 in lifespan 0|prefect-server | async with self lifespan_context(app) as maybe_state 0|prefect-server | File /usr/lib64/python3 9/contextlib py line 181 in aenter 0|prefect-server | return await self gen __anext__() 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/server/api/server py line 591 in lifespan 0|prefect-server | await run_migrations() 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/server/api/server py line 517 in run_migrations 0|prefect-server | await db create_db() 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/server/database/interface py line 55 in create_db 0|prefect-server | await self run_migrations_upgrade() 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/server/database/interface py line 63 in run_migrations_upgrade 0|prefect-server | await run_sync_in_worker_thread(alembic_upgrade) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/utilities/asyncutils py line 91 in run_sync_in_worker_thread 0|prefect-server | return await <http //anyio to|anyio to>_thread run_sync( 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/anyio/to_thread py line 33 in run_sync 0|prefect-server | return await get_asynclib() run_sync_in_worker_thread( 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/anyio/_backends/_asyncio py line 877 in run_sync_in_worker_thread 0|prefect-server | return awa [4:10 PM] it future 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/anyio/_backends/_asyncio py line 807 in run 0|prefect-server | result = context run(func *args) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/server/database/alembic_commands py line 24 in wrapper 0|prefect-server | return fn(*args **kwargs) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/server/database/alembic_commands py line 53 in alembic_upgrade 0|prefect-server | alembic command upgrade(alembic_config() revision sql=dry_run) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/command py line 483 in upgrade 0|prefect-server | script run_env() 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/base py line 549 in run_env 0|prefect-server | util load_python_file(self dir env py ) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/util/pyfiles py line 116 in load_python_file 0|prefect-server | module = load_module_py(module_id path) 0|prefect-server
m
Your message was too long, here's your message at the allowed limit: | |_| |_|_\___|_| |___\___| |_| 0|prefect-server | Configure Prefect to communicate with the server with 0|prefect-server | prefect config set PREFECT_API_URL=<http //0 0 0 0 4200/api> 0|prefect-server | View the API reference documentation at <http //0 0 0 0 4200/docs> 0|prefect-server | Check out the dashboard at <http //0 0 0 0 4200> 0|prefect-server | Traceback (most recent call last) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/base py line 217 in _catch_revision_errors 0|prefect-server | yield 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/base py line 425 in _upgrade_revs 0|prefect-server | for script in reversed(list(revs)) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 814 in iterate_revisions 0|prefect-server | revisions heads = fn( 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 1475 in _collect_upgrade_revisions 0|prefect-server | current_revisions = self get_revisions(lower) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 542 in get_revisions 0|prefect-server | return sum([self get_revisions(id_elem) for id_elem in id_] ()) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 542 in <listcomp> 0|prefect-server | return sum([self get_revisions(id_elem) for id_elem in id_] ()) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 565 in get_revisions 0|prefect-server | return tuple( 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 566 in <genexpr> 0|prefect-server | self _revision_for_ident(rev_id branch_label) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/revision py line 637 in _revision_for_ident 0|prefect-server | raise ResolutionError( 0|prefect-server | alembic script revision ResolutionError No such revision or branch '8bb517bae6f9' 0|prefect-server | The above exception was the direct cause of the following exception 0|prefect-server | Traceback (most recent call last) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/starlette/routing py line 705 in lifespan 0|prefect-server | async with self lifespan_context(app) as maybe_state 0|prefect-server | File /usr/lib64/python3 9/contextlib py line 181 in aenter 0|prefect-server | return await self gen __anext__() 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/server/api/server py line 591 in lifespan 0|prefect-server | await run_migrations() 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/server/api/server py line 517 in run_migrations 0|prefect-server | await db create_db() 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/server/database/interface py line 55 in create_db 0|prefect-server | await self run_migrations_upgrade() 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/server/database/interface py line 63 in run_migrations_upgrade 0|prefect-server | await run_sync_in_worker_thread(alembic_upgrade) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/utilities/asyncutils py line 91 in run_sync_in_worker_thread 0|prefect-server | return await <http //anyio to|anyio to>_thread run_sync( 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/anyio/to_thread py line 33 in run_sync 0|prefect-server | return await get_asynclib() run_sync_in_worker_thread( 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/anyio/_backends/_asyncio py line 877 in run_sync_in_worker_thread 0|pre
fect-server | return awa [4 10 PM] it future 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/anyio/_backends/_asyncio py line 807 in run 0|prefect-server | result = context run(func *args) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/server/database/alembic_commands py line 24 in wrapper 0|prefect-server | return fn(*args **kwargs) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/prefect/server/database/alembic_commands py line 53 in alembic_upgrade 0|prefect-server | alembic command upgrade(alembic_config() revision sql=dry_run) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/command py line 483 in upgrade 0|prefect-server | script run_env() 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/script/base py line 549 in run_env 0|prefect-server | util load_python_file(self dir env py ) 0|prefect-server | File /home/ec2-user/ local/lib/python3 9/site-packages/alembic/util/pyfiles py line 116 in load_python_file 0|prefect-server | module
s
@Marvin what security settings do I need to apply for my cli on my personal laptop to be able to communicate with my prefect server on ec2?
m
✅ thought for 153.1 seconds
Short answer: open network access from your laptop to the Prefect API on your EC2 instance and point your CLI at it. The safest path is an SSH tunnel; if you must expose it, lock it to your IP and use TLS. Two secure patterns you can choose from: Option A — SSH tunnel (recommended, no public exposure) 1) On EC2, run Prefect Server bound to localhost:
Copy code
prefect server start --host 127.0.0.1 --port 4200
2) From your laptop, create a tunnel:
Copy code
ssh -i /path/to/key.pem -N -L 4200:127.0.0.1:4200 ec2-user@<ec2-public-dns>
3) Point your CLI at the tunneled API:
Copy code
prefect config set PREFECT_API_URL="<http://127.0.0.1:4200/api>"
prefect config view | grep PREFECT_API_URL
curl -s <http://127.0.0.1:4200/api/health>
Security group requirements: only inbound SSH (22) from your IP. No 4200 open to the internet. Option B — Publicly expose the API (lock down to your IP and add TLS) 1) On EC2, run Prefect Server listening on all interfaces:
Copy code
prefect server start --host 0.0.0.0 --port 4200
2) AWS Security Group for the instance: - Inbound: - TCP 4200 from your laptop’s public IP only (x.x.x.x/32) - TCP 22 from your IP for SSH - Outbound: allow all (default) 3) OS firewall: ensure no local firewall blocks 4200 (Amazon Linux typically allows by default). 4) Point your CLI at the EC2 address:
Copy code
prefect config set PREFECT_API_URL="http://<ec2-public-dns>:4200/api"
prefect config view | grep PREFECT_API_URL
curl -s http://<ec2-public-dns>:4200/api/health
5) Strongly recommended: put Nginx/ALB in front with HTTPS - Terminate TLS on 443 and proxy to 127.0.0.1:4200. - Then set:
Copy code
prefect config set PREFECT_API_URL="https://<your-domain>/api"
- Restrict inbound 443 to your IP(s) or your VPN. What your CLI needs to work - The CLI/SDK on your laptop must be able to reach 
PREFECT_API_URL
over TCP. - In Prefect 2.x, the correct value is the full API endpoint with 
/api
, e.g. 
<http://host:4200/api>
or 
<https://domain/api>
. - You can validate with:
Copy code
prefect config view
prefect config validate
curl -s $PREFECT_API_URL/health
Notes and gotchas - Prefect 2 Server has no built-in auth/TLS. If you expose it, use a reverse proxy and restrict ingress. - If you only need the CLI (not the UI), you still only need port 4200 (or 443 via proxy). - If workers/flow-runners live in the same VPC, you can keep the API private and access via VPN/SSH tunnel instead of opening to the internet. Docs (Prefect 2.x) - Hosting Prefect Server: Self-hosting guide - Settings: Settings guide If you tell me whether you prefer SSH tunnel, VPN, or public exposure with TLS, I can share an exact command/proxy config and the precise Security Group rule to add.
s
@Marvin prefect.exceptions.PrefectHTTPStatusError: Client error '409 Conflict' for url 'http://ec2-*-*-*-*.ap-south-1.compute.amazonaws.com:4200/api/deployments/' Response: {'detail': 'Error creating deployment: <ValidationError: "None is not of type \'string\'">'}
m
✅ thought for 106.0 seconds
That 409 with “None is not of type 'string'” means the Server rejected your deployment payload because at least one required field that must be a string is coming through as None. In Prefect 2.x this typically happens when building/applying a deployment with a missing or null value for one of these: - name - entrypoint (e.g., path/to/file.py:flow_func) - work_pool_name or work_queue_name (depending on whether you use workers/pools or agents/queues) - version/description when explicitly set to None instead of omitted - job_variables for workers (should be an object, not null) Can you share how you’re creating the deployment? The exact CLI command (or the Python code) and, if you’re applying a YAML, the YAML contents (with secrets redacted). That will let me point out the offending field. Known-good examples for Prefect 2.x A) Using agents/queues (older pattern)
Copy code
prefect deployment build flows/ray-flow.py:count_to \
  -n ray-count \
  -q default \
  -o ray-deploy.yaml

# Inspect the YAML to ensure no null/None values
cat ray-deploy.yaml

# Then apply
prefect deployment apply ray-deploy.yaml
B) Using workers/work pools (newer 2.x pattern)
Copy code
# Ensure a work pool exists, e.g., base worker pool name 'default-agent-pool'
prefect work-pool ls

prefect deployment build flows/ray-flow.py:count_to \
  -n ray-count \
  --pool default-agent-pool \
  -o ray-deploy.yaml

cat ray-deploy.yaml
prefect deployment apply ray-deploy.yaml
Minimal deployment YAML template you can use to test
Copy code
name: ray-count
version: "1"
work_pool_name: "default-agent-pool"     # or remove this and use 'work_queue_name: default' if you use agents
job_variables: {}                        # keep as {}, not null
schedule: null
tags: []
parameters: {}
entrypoint: flows/ray-flow.py:count_to
description: ""
Checklist to resolve - Ensure you’re on Prefect 2.x on both client and server (2.14.20). Run 
prefect version
on your laptop and on EC2. - If you’re using workers/pools, include a non-empty 
work_pool_name
and ensure the pool exists (
prefect work-pool ls
). - If you’re using agents/queues, include a non-empty 
work_queue_name
(e.g., 
default
) and have an agent pointed at that queue. - Do not set optional strings to None in YAML; either provide a string or omit the field. - If you attempted to “recreate” an existing deployment, you might also see 409 due to slug conflict. You can delete the old one and re-apply:
Copy code
prefect deployment ls
  prefect deployment delete <flow-name>/<deployment-name>
  prefect deployment apply ray-deploy.yaml
- Server logs on EC2 will usually include which exact field failed validation. If you can capture that log line, I can pinpoint the fix. Docs (Prefect 2.x) - Deployments overview: Deployments (2.x) - Build/apply flow deployments: Block-based deployments - Work pools and workers: Work pools - Prefect + Ray integration: prefect-ray If you paste your command and the YAML (or Python snippet) you used to create the deployment, I’ll tell you exactly which field is coming through as None and how to correct it.
s
@Marvin this is my code import os import time from dotenv import load_dotenv from prefect import flow, task from prefect_ray import RayTaskRunner from prefect.deployments import DeploymentImage from prefect.tasks import task_input_hash from prefect_ray.context import remote_options import pandas as pd import random PANDAS_VERSION = "2.1.3" # Load both .env files to access all environment variables # TEAM: Make sure you have .env and .env.generated files in the ray-prefect-final directory load_dotenv('../.env') load_dotenv('../.env.generated') def get_ray_image_uri(): """Get the Docker image URI for Ray tasks.""" print("Call to get_ray_image_uri, shall be at build time.") # Use environment variable or fallback to hardcoded registry registry = os.getenv('AWS_ECR_REGISTRY') uri = f"{registry}:pd213" print(f"Image URI: {uri}") return uri @task(_cache_key_fn_=task_input_hash) def test(_in_arg_): print("test task started", _in_arg_) # print(pd.DataFrame([1,2,3])) # Validate Python version import sys print(f"Python version: {sys.version}") print(f"Python version info: {sys.version_info}") # Check pandas version _# print(f"Pandas version: {pd.version}")_ print(f"✅ Pandas version is correct: {PANDAS_VERSION}") _# if pd.version == PANDAS_VERSION:_ _# print(f"_✅ _Pandas version is correct: {PANDAS_VERSION}")_ # else: _# print(f"_❌ _Pandas version mismatch! Expected: {PANDAS_VERSION}, Got: {pd.version}")_ # raise Exception("Pandas version mismatch") time.sleep(100) print("test task finished", _in_arg_) @flow( _log_prints_=True, _task_runner_=RayTaskRunner( address=f"ray://{os.getenv('RAY_HEAD_NODE_IP')}:10001", # this shall enable Docker image to be used in the Ray cluster # but it is not working for some reason, pandas is not available. _# init_kwargs={"runtime_env": {"image_uri": get_ray_image_uri()}} # for ray 2.48.0_ _# init_kwargs={"runtime_env": {"container": {"image": get_ray_image_uri()}}} # for ray 2.9.2_ ) ) def test_ray(): args = [random.random() for _ in range(10)] args.append(0) # task that shall be always cached print("args", args) # Apply runtime environment to each task to run in separate containers with remote_options(_num_cpus_=1, resources={"worker_node_1": 0.001},_runtime_env_={"image_uri": get_ray_image_uri()}): _# with remote_options(num_cpus=1, resources={"worker_node_1": 0.001}):_ test.map(args) if name == "__main__": # Check for required environment variables required_vars = { 'AWS_ECR_REGISTRY': os.getenv('AWS_ECR_REGISTRY'), 'PREFECT_DEPLOYMENT_NAME': os.getenv('PREFECT_DEPLOYMENT_NAME'), 'PREFECT_WORKPOOL_NAME': os.getenv('PREFECT_WORKPOOL_NAME'), } print("required_vars", required_vars) missing_vars = [var for var, value in required_vars.items() if not value] if missing_vars: print(f"❌ ERROR: Missing required environment variables: {', '.join(missing_vars)}") print("Please set these variables in your .env file or environment.") exit(1) test_ray.deploy( name="test_ray_1", _work_pool_name_=os.getenv('PREFECT_WORKPOOL_NAME'), description="Test Ray deployment", image=DeploymentImage( name=get_ray_image_uri(), platform="linux/amd64", dockerfile="Dockerfile", buildargs={ "BUILD_TIME": str(int(time.time())), # Force rebuild with timestamp } ), _job_variables_={ "cpu": 1024, "memory": 1024, "env": { "AWS_DEFAULT_REGION": os.getenv("AWS_DEFAULT_REGION"), "AWS_DEFAULT_BUCKET": os.getenv("AWS_DEFAULT_BUCKET"), "AWS_ECR_REGISTRY": os.getenv("AWS_ECR_REGISTRY"), "RAY_HEAD_NODE_IP": os.getenv("RAY_HEAD_NODE_IP"), } } )
m
✅ thought for 137.1 seconds
That 409 “None is not of type 'string'” means the deployment payload your client sent to the server contains a field that must be a string but is actually None. In your code, the most likely culprit is the env you pass inside job_variables: You build env from os.getenv(...). Any missing variable will be None, and Prefect validates job_variables.env as a mapping of string->string. A None value will trigger exactly this error. What to change - Filter out missing env vars (or provide defaults) before calling .deploy. - Also print the values you’re about to send so you can spot any None. Example patch:
Copy code
def build_job_env():
    env = {
        "AWS_DEFAULT_REGION": os.getenv("AWS_DEFAULT_REGION"),
        "AWS_DEFAULT_BUCKET": os.getenv("AWS_DEFAULT_BUCKET"),
        "AWS_ECR_REGISTRY": os.getenv("AWS_ECR_REGISTRY"),
        "RAY_HEAD_NODE_IP": os.getenv("RAY_HEAD_NODE_IP"),
    }
    # Warn and drop any unset values
    missing = [k for k, v in env.items() if v in (None, "")]
    if missing:
        print(f"Warning: missing env vars for job_variables.env: {missing} (they will be omitted)")
    return {k: v for k, v in env.items() if v not in (None, "")}

# ...

wp = os.getenv('PREFECT_WORKPOOL_NAME')
print("Deploying to work pool:", wp)

img = get_ray_image_uri()
print("Image URI to use:", img)

test_ray.deploy(
    name="test_ray_1",
    work_pool_name=wp,                        # must be a non-empty string
    description="Test Ray deployment",
    image=DeploymentImage(
        name=img,                             # must be a non-empty string
        platform="linux/amd64",
        dockerfile="Dockerfile",
        buildargs={"BUILD_TIME": str(int(time.time()))},
    ),
    job_variables={
        "cpu": 1024,
        "memory": 1024,
        "env": build_job_env(),               # ensure only strings here
    }
)
Also double-check these are non-empty strings at runtime: - work_pool_name (from PREFECT_WORKPOOL_NAME) - image.name (comes from AWS_ECR_REGISTRY) - name (you hard-coded "test_ray_1", good) - Any other optional strings you pass (description is fine) Important version note - Your code uses Flow.deploy(...) and DeploymentImage, which are Prefect 3-style APIs. If you truly want to stay on Prefect 2.14.20 end-to-end, you’d typically use the 2.x deployment flow (build/apply YAML) rather than Flow.deploy. Please confirm the versions on both your laptop and the EC2 server: - On laptop and EC2:
Copy code
prefect version
- If they don’t both show a 3.x version, you may be mixing client/server versions which can also cause schema errors. Next steps - Add the filtering patch above and re-run; if it still fails, please share the stdout prints for: - work_pool_name value - image URI - the final env dict you’re sending in job_variables - If you prefer to stick with Prefect 2.x, I can show the equivalent “prefect deployment build/apply” approach for your flow. If you’re moving to 3.x, we’ll keep using Flow.deploy but ensure your server is also 3.x. If you can also grab the server log line around the 409 on EC2, it often includes the exact field that failed validation and we can pinpoint it immediately.
5 Views
Open in Slack
PreviousNext
Powered by