Amir
10/18/2022, 5:00 PMpoetry run python src/driver/__main__.py
shell_run_command. This has been a slow process as there's been a bunch of little issues arising, etc.
My question: What would be the easiest way of doing all this? ie. what would be the best method to approach this problem? I feel like my method is a bit... patchy?dammy arinde
10/18/2022, 5:18 PMAndrei Tulbure
10/18/2022, 5:26 PMJosh Paulin
10/18/2022, 6:21 PMdisable_run_logger
Esdras Lopes Nani
10/18/2022, 9:07 PM.map
?
I've tried using CLI prefect concurrency-limit
tagging the task but apperently there was no effect and the limit was not respectedTony Piazza
10/18/2022, 9:13 PMFROM python:3.9.13
COPY requirements.txt .
RUN pip install -r requirements.txt
Prefect is included in the requirements. When the job is launched, I get the following error:
Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 247, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "/usr/local/lib/python3.9/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/prefect/deployments.py", line 159, in load_flow_from_flow_run
await storage_block.get_directory(from_path=deployment.path, local_path=".")
File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 144, in get_directory
shutil.copytree(from_path, local_path, dirs_exist_ok=True)
File "/usr/local/lib/python3.9/shutil.py", line 566, in copytree
with os.scandir(src) as itr:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/flows'
How do I resolve this?Kevin Grismore
10/18/2022, 10:23 PMrun_deployment()
in a flow, and I want to run multiple deployments concurrently, then move on to the next task once all the deployments have finished running. How might I do that?Adam Green
10/19/2022, 1:17 AMNace Plesko
10/19/2022, 3:27 AMAccount
tab it says first 20k task runs are free and the pricing page that is linked there doesn't mention anything at all about number of task runs. Could someone help me understand please how pricing works? The descriptions of our plan in the Account
tab and the plans in the pricing page are completely different.
I would love to make Prefect work for us but I don't have an approval for the paid version until I'm confident that Prefect is the right tool for us. With the current setup we have plenty of issues and I'd love to break down the current tasks into smaller ones. However, with that I'd most likely go over the 20k monthly limit already at this point even before we start growing.Alex
10/19/2022, 8:01 AMMatthieu Lhonneux
10/19/2022, 8:33 AMredsquare
10/19/2022, 8:36 AM{"detail":"Not Found"}
Florian Kühnlenz
10/19/2022, 8:53 AMDeepanshu Aggarwal
10/19/2022, 11:01 AMflapili
10/19/2022, 2:21 PMflapili
10/19/2022, 2:22 PMChris Gunderson
10/19/2022, 2:47 PMprefect deployment build -n fidelity-allocations-deployment -q default -sb s3/prefect-training -ib docker-container/prefect-training -a src/main/prefect/flows/allocations/prefect_fidelity_allocations.py:FidelityAllocationsFlow --cron "13 21 * * 1-5"
Error message from the flow:
Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "<frozen importlib._bootstrap_external>", line 846, in exec_module
File "<frozen importlib._bootstrap_external>", line 982, in get_code
File "<frozen importlib._bootstrap_external>", line 1039, in get_data
FileNotFoundError: [Errno 2] No such file or directory: 'src\\main\\prefect\\flows\\allocations\\prefect_fidelity_allocations.py'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/engine.py", line 257, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/client/orion.py", line 82, in with_injected_client
return await fn(*args, **kwargs)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/deployments.py", line 70, in load_flow_from_flow_run
flow = await run_sync_in_worker_thread(import_object, str(import_path))
File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 57, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(call, cancellable=True)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/opt/pysetup/.venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/opt/pysetup/.venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/utilities/importtools.py", line 193, in import_object
module = load_script_as_module(script_path)
File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/utilities/importtools.py", line 156, in load_script_as_module
raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'src\\main\\prefect\\flows\\allocations\\prefect_fidelity_allocations.py' encountered an exceptionDavid Elliott
10/19/2022, 2:56 PMDaskTaskRunner()
😢 It can’t pickle the task futures, which is sadly blocking our migration to 2.0. Have tried this both in Dask locally and also on k8s. Any ideas? MRE in 🧵Evan Curtin
10/19/2022, 2:57 PMResult
in 2.6+ ? I am thinking of passing data between tasks, persisted into remote storage as parquet, for example (e.g. using spark)redsquare
10/19/2022, 3:13 PMKelvin DeCosta
10/19/2022, 4:16 PMcreate_deployments
script multiple times and overwrite the deployments (similar to blocks).
I'm currently using the .build_from_flow
and .apply
methods but I keep running into warningsRahul Kadam
10/19/2022, 4:33 PMClaire Herdeman
10/19/2022, 4:42 PMDavid Beck
10/19/2022, 5:10 PMKubernetesJob.job_from_file
function to read a yaml file with our full k8s manifest. The function works, however I noticed that we have fields that get overwritten with default values in this call for _shortcut_customizations
in build_job
, specifically the namespace
and image
properties. I know that I can use an infrastructure_override
when setting up the deployment, however there should be option to simply read a yaml file with those fields.Erik Amundson
10/19/2022, 5:11 PMJehan Abduljabbar
10/19/2022, 5:22 PMMatt Denno
10/19/2022, 6:53 PMprefecthq/prefect:2.6.1-python3.10-conda
as a base image. Couple of questions:
• When installing a python environment in the container does it mater what it is called? I see the image has a base
env and a prefect
env. Should packages be installed in prefect
env of can I create a new one?
• If I create a new env should the Dockerfile include a line to activate the correct env.?
• Is there a recipe for creating a custom image from a conda base image? I looked around and did find one, but maybe I missed it?Matt Denno
10/19/2022, 7:05 PMdeploy_import_1 = Deployment.build_from_flow(
flow=import_1,
name="import1",
work_queue_name="imports",
schedule=IntervalSchedule(
interval=timedelta(hours=1),
anchor_date=datetime(2021, 1, 1, 2, 30, tzinfo=pytz.UTC)
),
infrastructure=DockerContainer(
image="path_to_image/import-prefect:python3.10",
image_pull_policy="NEVER",
auto_remove=False
)
)
Do I need to include remote storage to test? I am a bit unclear on all the pieces of the puzzle still.Kalise Richmond
10/19/2022, 7:06 PM