Santiago Gonzalez
04/07/2023, 3:02 AMChris White
04/07/2023, 3:09 AMdeployment.yaml
file and running
prefect project init --recipe s3
# inspect your deployment.yaml and prefect.yaml files here
prefect deploy --help # will show you your options
For more information on how this all works, you can reference our new tutorial: https://docs.prefect.io/latest/tutorials/projects/Nelson Griffiths
04/07/2023, 11:40 AMChris White
04/07/2023, 2:06 PMNelson Griffiths
04/07/2023, 2:08 PMChris White
04/07/2023, 2:08 PMSantiago Gonzalez
04/07/2023, 2:12 PMprefect project
doesn’t work. I just installed prefect==2.9.0
(I was using 2.8.3
). thanks for your replyChris White
04/07/2023, 2:12 PM2.10.1
Santiago Gonzalez
04/07/2023, 2:25 PMChris White
04/07/2023, 2:29 PMprefect deploy ./path/to/flow_file.py:flow_func -n name-of-deployment
Nelson Griffiths
04/07/2023, 2:30 PMChris White
04/07/2023, 2:31 PMSantiago Gonzalez
04/07/2023, 3:18 PMprefect deploy flows/flows/landbaron.py:generate_lb_sites_flow -n landbaron_sites_generation -p dev-pool -v 1.0.0
Traceback (most recent call last):
File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
return fn(*args, **kwargs)
File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/utilities/asyncutils.py", line 260, in coroutine_wrapper
return call()
File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 245, in __call__
return self.result()
File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 173, in result
return self.future.result(timeout=timeout)
File "/usr/local/opt/python@3.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 428, in result
return self.__get_result()
File "/usr/local/opt/python@3.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 218, in _run_async
result = await coro
File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/cli/root.py", line 364, in deploy
step_outputs.update(await run_step(step))
File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/projects/steps/core.py", line 69, in run_step
return step_func(**inputs)
File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect_aws/projects/steps.py", line 102, in push_project_to_s3
client.upload_file(str(local_file_path), bucket, str(remote_file_path))
File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/boto3/s3/inject.py", line 148, in upload_file
callback=Callback,
File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/boto3/s3/transfer.py", line 285, in upload_file
filename, bucket, key, extra_args, subscribers
File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/s3transfer/manager.py", line 322, in upload
self._validate_if_bucket_supported(bucket)
File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/s3transfer/manager.py", line 490, in _validate_if_bucket_supported
match = pattern.match(bucket)
TypeError: expected string or bytes-like object
Chris White
04/07/2023, 3:21 PMprefect.yaml
file?Santiago Gonzalez
04/07/2023, 3:28 PMpip
), but is not availableChris White
04/07/2023, 3:33 PMSantiago Gonzalez
04/07/2023, 3:39 PMChris White
04/07/2023, 3:39 PMSantiago Gonzalez
04/07/2023, 3:40 PMAWS_PROFILE=data prefect deploy flows/flows/landbaron.py:generate_lb_sites_flow -n landbaron_sites_generation -p dev-pool -v 1.0.0
Unable to load step function: {fully_qualified_name}. Attempting install of {requires}.
Collecting prefect-aws>=0.3.0
Using cached prefect_aws-0.3.0-py3-none-any.whl (39 kB)
Installing collected packages: prefect-aws
Attempting uninstall: prefect-aws
Found existing installation: prefect-aws 0.2.4
Uninstalling prefect-aws-0.2.4:
Successfully uninstalled prefect-aws-0.2.4
Successfully installed prefect-aws-0.3.0
WARNING: You are using pip version 22.0.4; however, version 23.0.1 is available.
You should consider upgrading via the '/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/bin/python -m pip install --upgrade pip' command.
Traceback (most recent call last):
File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
return fn(*args, **kwargs)
File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/utilities/asyncutils.py", line 260, in coroutine_wrapper
return call()
File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 245, in __call__
return self.result()
File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 173, in result
return self.future.result(timeout=timeout)
File "/usr/local/Cellar/python@3.7/3.7.16/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 428, in result
return self.__get_result()
File "/usr/local/Cellar/python@3.7/3.7.16/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 218, in _run_async
result = await coro
File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/cli/root.py", line 368, in deploy
key, value = variable.split("=", 1)
ValueError: not enough values to unpack (expected 2, got 1)
An exception occurred.
Chris White
04/07/2023, 3:59 PM-v
flag is for work pool job variables, not version; I'll update the documentation to reflect thatSantiago Gonzalez
04/07/2023, 4:13 PMAWS_PROFILE=data prefect deploy flows/flows/landbaron.py:generate_lb_sites_flow -n landbaron_sites_generation -p dev-pool
Cannot deploy project with work pool of type 'prefect-agent'.
Chris White
04/07/2023, 4:14 PMprefect worker start -t process -p new-pool-name
Santiago Gonzalez
04/07/2023, 4:21 PMFile "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/utilities/asyncutils.py", line 92, in run_sync_in_worker_thread
call, cancellable=True, limiter=get_thread_limiter()
File "/home/sgonzalez/venv/lib/python3.7/site-packages/anyio/to_thread.py", line 32, in run_sync
func, *args, cancellable=cancellable, limiter=limiter
File "/home/sgonzalez/venv/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/home/sgonzalez/venv/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/flows.py", line 809, in load_flow_from_entrypoint
flow = import_object(entrypoint)
File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/utilities/importtools.py", line 201, in import_object
module = load_script_as_module(script_path)
File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/utilities/importtools.py", line 164, in load_script_as_module
raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'flows/flows/landbaron.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
Chris White
04/07/2023, 4:23 PMprefect.yaml
file?Santiago Gonzalez
04/07/2023, 4:25 PM# File for configuring project / deployment build, push and pull steps
# Generic metadata about this project
name: automation-flows-v2
prefect-version: 2.10.2
# build section allows you to manage and build docker images
build: null
# push section allows you to manage if and how this project is uploaded to remote locations
push:
- prefect_aws.projects.steps.push_project_to_s3:
requires: prefect-aws>=0.3.0
bucket: wp-databricks-trial
folder: automation-flows-v2
# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect_aws.projects.steps.pull_project_from_s3:
requires: prefect-aws>=0.3.0
bucket: wp-databricks-trial
folder: prefect-flows-v2
alex
04/07/2023, 4:30 PMSantiago Gonzalez
04/07/2023, 4:30 PMprefect worker start -t process -p new-pool-name
alex
04/07/2023, 4:31 PMfolder
is different between your push
and pull
actions. I think that’s why your worker is unable to find the code for your flow. to sync up your push
and pull
actions, you can update your prefect.yaml
to look like this:
# File for configuring project / deployment build, push and pull steps
# Generic metadata about this project
name: automation-flows-v2
prefect-version: 2.10.2
# build section allows you to manage and build docker images
build: null
# push section allows you to manage if and how this project is uploaded to remote locations
push:
- prefect_aws.projects.steps.push_project_to_s3:
requires: prefect-aws>=0.3.0
bucket: wp-databricks-trial
folder: automation-flows-v2
# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect_aws.projects.steps.pull_project_from_s3:
requires: prefect-aws>=0.3.0
bucket: "{{ bucket }}"
folder: "{{ folder }}"
What happening here is the push_project_to_s3
step is outputting the bucket and folder that the flow code is pushed to. You can then use those values as placeholders in your pull_project_from_s3
step to make sure that you’re pulling from the same bucket and folder. Let me know if that works for you!Santiago Gonzalez
04/07/2023, 4:43 PMChris White
04/07/2023, 4:44 PMs3
recipes to use these bucket and folder placeholders so future users don't run into this 🙌Santiago Gonzalez
04/07/2023, 6:05 PMChris White
04/07/2023, 6:54 PMprefect.yaml
file to your repository and run mulitple prefect deploy
commands, one for each flow you need to deploy.
In the very near future (hopefully next week!) we'll make this much easier to do programmaticallySantiago Gonzalez
04/07/2023, 7:05 PMdeploy
command where I can switch between recipes?Chris White
04/07/2023, 7:06 PM