https://prefect.io logo
Join Slack
Powered by
Hi, I am facing some troubles with a deployment wi...
# prefect-cloud
s
Hi, I am facing some troubles with a deployment with Prefect 2. The deployment succeed and the flow is on the API, but when I try to run it fails because the agent can’t download de flow from S3 storage I set up before (which path has been set to a folder where the flow file has been uploaded at the moment that the deployment happened). The error is FileNotFoundException. Can you send me an easy example that I can reproduce with S3 location (not cli command, but Deployment.build_from_flow line)? Thanks in advance. Regards
✅ 1
c
Hi Santiago - we just released a new way to manage deployments that I think is much more transparent and should help you resolve this; it's currently a CLI-first approach, but you could at least experiment with it by removing your 
deployment.yaml
file and running
Copy code
prefect project init --recipe s3 
# inspect your deployment.yaml and prefect.yaml files here
prefect deploy --help # will show you your options
For more information on how this all works, you can reference our new tutorial: https://docs.prefect.io/latest/tutorials/projects/
n
@Chris White it seems like when these projects were released, the former method for deployments broke. Patterns we have been using to deploy things are no longer working. Is that true? We were storing things in GCS and now nothing that gets deployed can be found
c
I've heard a few reports of this, currently investigating
n
@Chris White Update: If I switch my storage over to Bitbucket from GCS and change nothing else, it works again. Seems like something about the path in GCS (or S3 from others) might be messing things up. Let me know if I can help with any other info
c
That's incredibly helpful info thank you!
s
Hey, the command 
prefect project
doesn’t work. I just installed 
prefect==2.9.0
(I was using 
2.8.3
). thanks for your reply
c
Ah yes this command is only available on version 
2.10.1
👍 1
We believe we have found the issue and plan to cut a patch release to resolve
👍 1
s
What I don’t understand is how can I do a reference to the flow itself with that command
c
you can reference it with an entrypoint, similar to `prefect deployment build`:
Copy code
prefect deploy ./path/to/flow_file.py:flow_func -n name-of-deployment
👍 1
n
@Chris White will that patch go out today?
c
yup, should be out within the hour - will update here when it's out
👍 1
s
I executed it, and I got an error
Copy code
prefect deploy flows/flows/landbaron.py:generate_lb_sites_flow -n landbaron_sites_generation -p dev-pool -v 1.0.0 

Traceback (most recent call last):
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/utilities/asyncutils.py", line 260, in coroutine_wrapper
    return call()
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 245, in __call__
    return self.result()
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 173, in result
    return self.future.result(timeout=timeout)
  File "/usr/local/opt/python@3.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/usr/local/opt/python@3.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 218, in _run_async
    result = await coro
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/cli/root.py", line 364, in deploy
    step_outputs.update(await run_step(step))
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/projects/steps/core.py", line 69, in run_step
    return step_func(**inputs)
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect_aws/projects/steps.py", line 102, in push_project_to_s3
    client.upload_file(str(local_file_path), bucket, str(remote_file_path))
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/boto3/s3/inject.py", line 148, in upload_file
    callback=Callback,
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/boto3/s3/transfer.py", line 285, in upload_file
    filename, bucket, key, extra_args, subscribers
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/s3transfer/manager.py", line 322, in upload
    self._validate_if_bucket_supported(bucket)
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/s3transfer/manager.py", line 490, in _validate_if_bucket_supported
    match = pattern.match(bucket)
TypeError: expected string or bytes-like object
c
did you provide a bucket name in your 
prefect.yaml
file?
@Nelson Griffiths 2.10.2 is out and should resolve this issue
❤️ 1
s
@Chris White I tried to download that version (through 
pip
), but is not available
c
Can you try again? You can see it here: https://pypi.org/project/prefect/
s
Now I could. trying to deploy again with this version, should I use this new way or Deployment.build_from_flow?
c
i highly recommend the new way - it's more flexible and easier to debug, and is the mode we'll begin documenting more as the recommendation
s
Ok, thanks
👍 1
Copy code
AWS_PROFILE=data prefect deploy flows/flows/landbaron.py:generate_lb_sites_flow -n landbaron_sites_generation -p dev-pool -v 1.0.0
Unable to load step function: {fully_qualified_name}. Attempting install of {requires}.
Collecting prefect-aws>=0.3.0
  Using cached prefect_aws-0.3.0-py3-none-any.whl (39 kB)
Installing collected packages: prefect-aws
  Attempting uninstall: prefect-aws
    Found existing installation: prefect-aws 0.2.4
    Uninstalling prefect-aws-0.2.4:
      Successfully uninstalled prefect-aws-0.2.4
Successfully installed prefect-aws-0.3.0
WARNING: You are using pip version 22.0.4; however, version 23.0.1 is available.
You should consider upgrading via the '/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/bin/python -m pip install --upgrade pip' command.
Traceback (most recent call last):
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/utilities/asyncutils.py", line 260, in coroutine_wrapper
    return call()
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 245, in __call__
    return self.result()
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 173, in result
    return self.future.result(timeout=timeout)
  File "/usr/local/Cellar/python@3.7/3.7.16/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/usr/local/Cellar/python@3.7/3.7.16/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/_internal/concurrency/calls.py", line 218, in _run_async
    result = await coro
  File "/Users/santiago.gonzalez/git-python/automation/automation-flows-v2/venv/lib/python3.7/site-packages/prefect/cli/root.py", line 368, in deploy
    key, value = variable.split("=", 1)
ValueError: not enough values to unpack (expected 2, got 1)
An exception occurred.
c
yes - the 
-v
flag is for work pool job variables, not version; I'll update the documentation to reflect that
s
Copy code
AWS_PROFILE=data prefect deploy flows/flows/landbaron.py:generate_lb_sites_flow -n landbaron_sites_generation -p dev-pool         
Cannot deploy project with work pool of type 'prefect-agent'.
c
correct, please see the project documentation: https://docs.prefect.io/latest/tutorials/projects/ Your work pool needs to be a process work pool; this will be auto-created for you when you run 
prefect worker start -t process -p new-pool-name
s
The flow has been deploy, however this is what I got on the worker
Copy code
File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/utilities/asyncutils.py", line 92, in run_sync_in_worker_thread
    call, cancellable=True, limiter=get_thread_limiter()
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/anyio/to_thread.py", line 32, in run_sync
    func, *args, cancellable=cancellable, limiter=limiter
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/flows.py", line 809, in load_flow_from_entrypoint
    flow = import_object(entrypoint)
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/utilities/importtools.py", line 201, in import_object
    module = load_script_as_module(script_path)
  File "/home/sgonzalez/venv/lib/python3.7/site-packages/prefect/utilities/importtools.py", line 164, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'flows/flows/landbaron.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
c
i'm going to need a lot more information if i'm going to debug step-by-step with you - what are the contents of your 
prefect.yaml
file?
s
Copy code
# File for configuring project / deployment build, push and pull steps

# Generic metadata about this project
name: automation-flows-v2
prefect-version: 2.10.2

# build section allows you to manage and build docker images
build: null

# push section allows you to manage if and how this project is uploaded to remote locations
push:
- prefect_aws.projects.steps.push_project_to_s3:
    requires: prefect-aws>=0.3.0
    bucket: wp-databricks-trial
    folder: automation-flows-v2

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect_aws.projects.steps.pull_project_from_s3:
    requires: prefect-aws>=0.3.0
    bucket: wp-databricks-trial
    folder: prefect-flows-v2
a
Hey @Santiago Gonzalez, what type of worker are you using?
s
this one
Copy code
prefect worker start -t process -p new-pool-name
a
Thanks! I’m going to try to reproduce the error that you’re seeing and I’ll report back with my findings
👍 1
@Santiago Gonzalez I noticed that the value for 
folder
is different between your 
push
and 
pull
actions. I think that’s why your worker is unable to find the code for your flow. to sync up your 
push
and 
pull
actions, you can update your 
prefect.yaml
to look like this:
Copy code
# File for configuring project / deployment build, push and pull steps

# Generic metadata about this project
name: automation-flows-v2
prefect-version: 2.10.2

# build section allows you to manage and build docker images
build: null

# push section allows you to manage if and how this project is uploaded to remote locations
push:
- prefect_aws.projects.steps.push_project_to_s3:
    requires: prefect-aws>=0.3.0
    bucket: wp-databricks-trial
    folder: automation-flows-v2

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect_aws.projects.steps.pull_project_from_s3:
    requires: prefect-aws>=0.3.0
    bucket: "{{ bucket }}"
    folder: "{{ folder }}"
What happening here is the 
push_project_to_s3
step is outputting the bucket and folder that the flow code is pushed to. You can then use those values as placeholders in your 
pull_project_from_s3
step to make sure that you’re pulling from the same bucket and folder. Let me know if that works for you!
upvote 1
s
Hey, I could do it
Thank you very much for your help, all of you, I really appreciate it.
🙌 1
You are awesome
c
Thanks for the feedback - we are updating the default values in our 
s3
recipes to use these bucket and folder placeholders so future users don't run into this 🙌
✅ 1
s
By the way, is there a way to perform this programmatically? I ask you this, because I have a requirement to deploy all the flows contained in the repo, to prefect every time the PRs got merged.
Thanks again
c
Right now you should commit your 
prefect.yaml
file to your repository and run mulitple 
prefect deploy
commands, one for each flow you need to deploy. In the very near future (hopefully next week!) we'll make this much easier to do programmatically
👍 2
s
That’s nice. Let’s say that for dev deployment, we want to do deploy with S3 Storage, and for prod with GitHub Storage (just pulling from there). Is there a parameter can be set through 
deploy
command where I can switch between recipes?
c
That's a great question! @alex is working on that right now, but yes that is part of the plan
🙌 1
Open in Slack
PreviousNext
Powered by