https://prefect.io logo
Powered by
Title
s

Santiago Gonzalez

04/07/2023, 3:02 AM
Hi, I am facing some troubles with a deployment with Prefect 2. The deployment succeed and the flow is on the API, but when I try to run it fails because the agent can’t download de flow from S3 storage I set up before (which path has been set to a folder where the flow file has been uploaded at the moment that the deployment happened). The error is FileNotFoundException. Can you send me an easy example that I can reproduce with S3 location (not cli command, but Deployment.build_from_flow line)? Thanks in advance. Regards
✅ 1
c

Chris White

04/07/2023, 3:09 AM
Hi Santiago - we just released a new way to manage deployments that I think is much more transparent and should help you resolve this; it's currently a CLI-first approach, but you could at least experiment with it by removing your 
deployment.yaml
file and running 
prefect project init --recipe s3 
# inspect your deployment.yaml and prefect.yaml files here
prefect deploy --help # will show you your options
For more information on how this all works, you can reference our new tutorial: https://docs.prefect.io/latest/tutorials/projects/
n

Nelson Griffiths

04/07/2023, 11:40 AM
@Chris White it seems like when these projects were released, the former method for deployments broke. Patterns we have been using to deploy things are no longer working. Is that true? We were storing things in GCS and now nothing that gets deployed can be found
c

Chris White

04/07/2023, 2:06 PM
I've heard a few reports of this, currently investigating
n

Nelson Griffiths

04/07/2023, 2:08 PM
@Chris White Update: If I switch my storage over to Bitbucket from GCS and change nothing else, it works again. Seems like something about the path in GCS (or S3 from others) might be messing things up. Let me know if I can help with any other info
c

Chris White

04/07/2023, 2:08 PM
That's incredibly helpful info thank you!
s

Santiago Gonzalez

04/07/2023, 2:12 PM
Hey, the command 
prefect project
doesn’t work. I just installed 
prefect==2.9.0
(I was using 
2.8.3
). thanks for your reply
c

Chris White

04/07/2023, 2:12 PM
Ah yes this command is only available on version 
2.10.1
👍 1
We believe we have found the issue and plan to cut a patch release to resolve
👍 1
s

Santiago Gonzalez

04/07/2023, 2:25 PM
What I don’t understand is how can I do a reference to the flow itself with that command
c

Chris White

04/07/2023, 2:29 PM
you can reference it with an entrypoint, similar to `prefect deployment build`: 
prefect deploy ./path/to/flow_file.py:flow_func -n name-of-deployment
👍 1
n

Nelson Griffiths

04/07/2023, 2:30 PM
@Chris White will that patch go out today?
c

Chris White

04/07/2023, 2:31 PM
yup, should be out within the hour - will update here when it's out
👍 1
s

Santiago Gonzalez

04/07/2023, 3:18 PM
I executed it, and I got an error
prefect deploy flows/flows/landbaron.py:generate_lb_sites_flow -n landbaron_sites_generation -p dev-pool -v 1.0.0 

Traceback (most recent call last):
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/utilities/asyncutils.py", line 260, in coroutine_wrapper
    return call()
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 245, in __call__
    return self.result()
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 173, in result
    return self.future.result(timeout=timeout)
  File "/usr/local/opt/python@3.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/usr/local/opt/python@3.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 218, in _run_async
    result = await coro
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/cli/root.py", line 364, in deploy
    step_outputs.update(await run_step(step))
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/projects/steps/core.py", line 69, in run_step
    return step_func(**inputs)
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect_aws/projects/steps.py", line 102, in push_project_to_s3
    client.upload_file(str(local_file_path), bucket, str(remote_file_path))
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/boto3/s3/inject.py", line 148, in upload_file
    callback=Callback,
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/boto3/s3/transfer.py", line 285, in upload_file
    filename, bucket, key, extra_args, subscribers
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/s3transfer/manager.py", line 322, in upload
    self._validate_if_bucket_supported(bucket)
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/s3transfer/manager.py", line 490, in _validate_if_bucket_supported
    match = pattern.match(bucket)
TypeError: expected string or bytes-like object
c

Chris White

04/07/2023, 3:21 PM
did you provide a bucket name in your 
prefect.yaml
file?
@Nelson Griffiths 2.10.2 is out and should resolve this issue
❤️ 1
s

Santiago Gonzalez

04/07/2023, 3:28 PM
@Chris White I tried to download that version (through 
pip
), but is not available
c

Chris White

04/07/2023, 3:33 PM
Can you try again? You can see it here: https://pypi.org/project/prefect/
s

Santiago Gonzalez

04/07/2023, 3:39 PM
Now I could. trying to deploy again with this version, should I use this new way or Deployment.build_from_flow?
c

Chris White

04/07/2023, 3:39 PM
i highly recommend the new way - it's more flexible and easier to debug, and is the mode we'll begin documenting more as the recommendation
s

Santiago Gonzalez

04/07/2023, 3:40 PM
Ok, thanks
👍 1
AWS_PROFILE=data prefect deploy flows/flows/landbaron.py:generate_lb_sites_flow -n landbaron_sites_generation -p dev-pool -v 1.0.0
Unable to load step function: {fully_qualified_name}. Attempting install of {requires}.
Collecting prefect-aws>=0.3.0
  Using cached prefect_aws-0.3.0-py3-none-any.whl (39 kB)
Installing collected packages: prefect-aws
  Attempting uninstall: prefect-aws
    Found existing installation: prefect-aws 0.2.4
    Uninstalling prefect-aws-0.2.4:
      Successfully uninstalled prefect-aws-0.2.4
Successfully installed prefect-aws-0.3.0
WARNING: You are using pip version 22.0.4; however, version 23.0.1 is available.
You should consider upgrading via the '/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/bin/python -m pip install --upgrade pip' command.
Traceback (most recent call last):
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/utilities/asyncutils.py", line 260, in coroutine_wrapper
    return call()
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 245, in __call__
    return self.result()
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 173, in result
    return self.future.result(timeout=timeout)
  File "/usr/local/Cellar/python@3.7/3.7.16/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/usr/local/Cellar/python@3.7/3.7.16/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 218, in _run_async
    result = await coro
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/cli/root.py", line 368, in deploy
    key, value = variable.split("=", 1)
ValueError: not enough values to unpack (expected 2, got 1)
An exception occurred.
c

Chris White

04/07/2023, 3:59 PM
yes - the 
-v
flag is for work pool job variables, not version; I'll update the documentation to reflect that
s

Santiago Gonzalez

04/07/2023, 4:13 PM
AWS_PROFILE=data prefect deploy flows/flows/landbaron.py:generate_lb_sites_flow -n landbaron_sites_generation -p dev-pool         
Cannot deploy project with work pool of type 'prefect-agent'.
c

Chris White

04/07/2023, 4:14 PM
correct, please see the project documentation: https://docs.prefect.io/latest/tutorials/projects/ Your work pool needs to be a process work pool; this will be auto-created for you when you run 
prefect worker start -t process -p new-pool-name
s

Santiago Gonzalez

04/07/2023, 4:21 PM
The flow has been deploy, however this is what I got on the worker 
File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/utilities/asyncutils.py", line 92, in run_sync_in_worker_thread
    call, cancellable=True, limiter=get_thread_limiter()
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/anyio/to_thread.py", line 32, in run_sync
    func, *args, cancellable=cancellable, limiter=limiter
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/flows.py", line 809, in load_flow_from_entrypoint
    flow = import_object(entrypoint)
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/utilities/importtools.py", line 201, in import_object
    module = load_script_as_module(script_path)
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/utilities/importtools.py", line 164, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'flows/flows/landbaron.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
c

Chris White

04/07/2023, 4:23 PM
i'm going to need a lot more information if i'm going to debug step-by-step with you - what are the contents of your 
prefect.yaml
file?
s

Santiago Gonzalez

04/07/2023, 4:25 PM
# File for configuring project / deployment build, push and pull steps

# Generic metadata about this project
name: automation-flows-v2
prefect-version: 2.10.2

# build section allows you to manage and build docker images
build: null

# push section allows you to manage if and how this project is uploaded to remote locations
push:
- prefect_aws.projects.steps.push_project_to_s3:
    requires: prefect-aws>=0.3.0
    bucket: wp-databricks-trial
    folder: automation-flows-v2

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect_aws.projects.steps.pull_project_from_s3:
    requires: prefect-aws>=0.3.0
    bucket: wp-databricks-trial
    folder: prefect-flows-v2
a

alex

04/07/2023, 4:30 PM
Hey @Santiago Gonzalez, what type of worker are you using?
s

Santiago Gonzalez

04/07/2023, 4:30 PM
this one 
prefect worker start -t process -p new-pool-name
a

alex

04/07/2023, 4:31 PM
Thanks! I’m going to try to reproduce the error that you’re seeing and I’ll report back with my findings
👍 1
@Santiago Gonzalez I noticed that the value for 
folder
is different between your 
push
and 
pull
actions. I think that’s why your worker is unable to find the code for your flow. to sync up your 
push
and 
pull
actions, you can update your 
prefect.yaml
to look like this: 
# File for configuring project / deployment build, push and pull steps

# Generic metadata about this project
name: automation-flows-v2
prefect-version: 2.10.2

# build section allows you to manage and build docker images
build: null

# push section allows you to manage if and how this project is uploaded to remote locations
push:
- prefect_aws.projects.steps.push_project_to_s3:
    requires: prefect-aws>=0.3.0
    bucket: wp-databricks-trial
    folder: automation-flows-v2

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect_aws.projects.steps.pull_project_from_s3:
    requires: prefect-aws>=0.3.0
    bucket: "{{ bucket }}"
    folder: "{{ folder }}"
What happening here is the 
push_project_to_s3
step is outputting the bucket and folder that the flow code is pushed to. You can then use those values as placeholders in your 
pull_project_from_s3
step to make sure that you’re pulling from the same bucket and folder. Let me know if that works for you!
:upvote: 1
s

Santiago Gonzalez

04/07/2023, 4:43 PM
Hey, I could do it
Thank you very much for your help, all of you, I really appreciate it.
🙌 1
You are awesome
c

Chris White

04/07/2023, 4:44 PM
Thanks for the feedback - we are updating the default values in our 
s3
recipes to use these bucket and folder placeholders so future users don't run into this 🙌
✅ 1
s

Santiago Gonzalez

04/07/2023, 6:05 PM
By the way, is there a way to perform this programmatically? I ask you this, because I have a requirement to deploy all the flows contained in the repo, to prefect every time the PRs got merged.
Thanks again
c

Chris White

04/07/2023, 6:54 PM
Right now you should commit your 
prefect.yaml
file to your repository and run mulitple 
prefect deploy
commands, one for each flow you need to deploy. In the very near future (hopefully next week!) we'll make this much easier to do programmatically
👍 2
s

Santiago Gonzalez

04/07/2023, 7:05 PM
That’s nice. Let’s say that for dev deployment, we want to do deploy with S3 Storage, and for prod with GitHub Storage (just pulling from there). Is there a parameter can be set through 
deploy
command where I can switch between recipes?
c

Chris White

04/07/2023, 7:06 PM
That's a great question! @alex is working on that right now, but yes that is part of the plan
🙌 1
#prefect-cloudJoin Slack
Powered by