Blake Stefansen
08/20/2024, 3:13 PMFlow could not be retrieved from deployment.
Can you help me?Marvin
08/20/2024, 3:14 PMFileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp9pq0n7nsprefect/subdir'
This suggests an issue with the path configuration in the deployment manifest. Ensure the S3 bucket and path are correctly set and accessible.
For local storage:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp9pq0n7nsprefect/subdir'
Ensure that the path to the stored flow is correct and absolute.
### Further References:
- GitHub Issue: Detailed case of this error with a private GitHub repository and local file system issues
https://github.com/PrefectHQ/prefect/issues/8310
- Documentation on Deployments and Storage:
- Concepts: https://docs.prefect.io/latest/concepts/deployments-overview/
- Integrations: https://docs.prefect.io/latest/integrations/prefect-gcp/deployments/steps/
Ensure all configurations are correct, and if the issue persists, consider providing more specific details about your deployment setup for more targeted troubleshooting.Blake Stefansen
08/20/2024, 3:16 PMDownloading flow code from storage at ''
09:35:45 AM
prefect.flow_runs
Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/prefect/engine.py", line 427, in retrieve_flow_then_begin_flow_run
else await load_flow_from_flow_run(
File "/usr/local/lib/python3.10/dist-packages/prefect/client/utilities.py", line 100, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/prefect/deployments/deployments.py", line 303, in load_flow_from_flow_run
await storage_block.get_directory(from_path=from_path, local_path=".")
File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 496, in get_directory
return await self.filesystem.get_directory(
File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 330, in get_directory
return self.filesystem.get(from_path, local_path, recursive=True)
File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 976, in get
self.get_file(rpath, lpath, callback=child, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 893, in get_file
os.makedirs(lpath, exist_ok=True)
File "/usr/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: ''
09:35:48 AM
prefect.flow_runs
Downloading flow code from storage at '/'
11:04:28 AM
prefect.flow_runs
Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/prefect/engine.py", line 427, in retrieve_flow_then_begin_flow_run
else await load_flow_from_flow_run(
File "/usr/local/lib/python3.10/dist-packages/prefect/client/utilities.py", line 100, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/prefect/deployments/deployments.py", line 303, in load_flow_from_flow_run
await storage_block.get_directory(from_path=from_path, local_path=".")
File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 496, in get_directory
return await self.filesystem.get_directory(
File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 330, in get_directory
return self.filesystem.get(from_path, local_path, recursive=True)
File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 976, in get
self.get_file(rpath, lpath, callback=child, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 893, in get_file
os.makedirs(lpath, exist_ok=True)
File "/usr/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: ''
Blake Stefansen
08/20/2024, 3:18 PMMarvin
08/20/2024, 3:18 PMBlake Stefansen
08/20/2024, 3:19 PMDownloading flow code from storage at ''
09:35:45 AM
prefect.flow_runs
Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/prefect/engine.py", line 427, in retrieve_flow_then_begin_flow_run
else await load_flow_from_flow_run(
File "/usr/local/lib/python3.10/dist-packages/prefect/client/utilities.py", line 100, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/prefect/deployments/deployments.py", line 303, in load_flow_from_flow_run
await storage_block.get_directory(from_path=from_path, local_path=".")
File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 496, in get_directory
return await self.filesystem.get_directory(
File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 330, in get_directory
return self.filesystem.get(from_path, local_path, recursive=True)
File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 976, in get
self.get_file(rpath, lpath, callback=child, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 893, in get_file
os.makedirs(lpath, exist_ok=True)
File "/usr/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: ''
09:35:48 AM
prefect.flow_runs
Downloading flow code from storage at '/'
11:04:28 AM
prefect.flow_runs
Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/prefect/engine.py", line 427, in retrieve_flow_then_begin_flow_run
else await load_flow_from_flow_run(
File "/usr/local/lib/python3.10/dist-packages/prefect/client/utilities.py", line 100, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/prefect/deployments/deployments.py", line 303, in load_flow_from_flow_run
await storage_block.get_directory(from_path=from_path, local_path=".")
File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 496, in get_directory
return await self.filesystem.get_directory(
File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 330, in get_directory
return self.filesystem.get(from_path, local_path, recursive=True)
File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 976, in get
self.get_file(rpath, lpath, callback=child, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 893, in get_file
os.makedirs(lpath, exist_ok=True)
File "/usr/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: ''
Marvin
08/20/2024, 3:19 PMhere is more contextso the last successful flow run for this deployment was in march. the deployment yaml has not changed since then. I thought it was due to the path="", but that seemed to work back in march. I changed the path to "/" this time, but that didnt fix it. ```Downloading flow code from storage at '' 093545 AM prefect.flow_runs Flow could not be retrieved from deployment. Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/prefect/engine.py", line 427, in retrieve_flow_then_begin_flow_run else await load_flow_from_flow_run( File "/usr/local/lib/python3.10/dist-packages/prefect/client/utilities.py", line 100, in with_injected_client return await fn(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/prefect/deployments/deployments.py", line 303, in load_flow_from_flow_run await storage_block.get_directory(from_path=from_path, local_path=".") File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 496, in get_directory return await self.filesystem.get_directory( File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 330, in get_directory return self.filesystem.get
Blake Stefansen
08/20/2024, 3:20 PMMarvin
08/20/2024, 3:20 PMyaml
storage:
- block_type_slug: local
attributes:
basepath: /absolute/path/to/your/flows
3. Deployment File:
- Ensure that the --path
specified while deploying matches the storage block location.
- Example command:
bash
prefect deployment build your_flow.py:flow_function --name your_deployment --storage-block local/your_storage_block --path /absolute/path/to/your/flows
4. Adjust Manifest:
- If needed, adjust the manifest to reflect the absolute path.
For further details:
- Deployment and Storage Documentation: https://docs.prefect.io/latest/concepts/deployments-block-based/
- Storage Guide: https://docs.prefect.io/latest/guides/deployment/storage-guide/
Validate these configurations, and if discrepancies are found, adjust them to ensure correct pathing and accessibility.Blake Stefansen
08/20/2024, 3:23 PMDownloading flow code from storage at ''
09:35:45 AM
prefect.flow_runs
Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/prefect/engine.py", line 427, in retrieve_flow_then_begin_flow_run
else await load_flow_from_flow_run(
File "/usr/local/lib/python3.10/dist-packages/prefect/client/utilities.py", line 100, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/prefect/deployments/deployments.py", line 303, in load_flow_from_flow_run
await storage_block.get_directory(from_path=from_path, local_path=".")
File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 496, in get_directory
return await self.filesystem.get_directory(
File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 330, in get_directory
return self.filesystem.get(from_path, local_path, recursive=True)
File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 976, in get
self.get_file(rpath, lpath, callback=child, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 893, in get_file
os.makedirs(lpath, exist_ok=True)
File "/usr/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: ''
Marvin
08/20/2024, 3:23 PMwhat about this?```Downloading flow code from storage at '' 093545 AM prefect.flow_runs Flow could not be retrieved from deployment. Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/prefect/engine.py", line 427, in retrieve_flow_then_begin_flow_run else await load_flow_from_flow_run( File "/usr/local/lib/python3.10/dist-packages/prefect/client/utilities.py", line 100, in with_injected_client return await fn(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/prefect/deployments/deployments.py", line 303, in load_flow_from_flow_run await storage_block.get_directory(from_path=from_path, local_path=".") File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 496, in get_directory return await self.filesystem.get_directory( File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 330, in get_directory return self.filesystem.get(from_path, local_path, recursive=True) File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 976, in get self.get_file(rpath, lpath, callback=child, **kwargs) File "/usr
Blake Stefansen
08/20/2024, 3:24 PMDownloading flow code from storage at ''
Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/prefect/engine.py", line 427, in retrieve_flow_then_begin_flow_run
else await load_flow_from_flow_run(
File "/usr/local/lib/python3.10/dist-packages/prefect/client/utilities.py", line 100, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/prefect/deployments/deployments.py", line 303, in load_flow_from_flow_run
await storage_block.get_directory(from_path=from_path, local_path=".")
File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 496, in get_directory
return await self.filesystem.get_directory(
File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 330, in get_directory
return self.filesystem.get(from_path, local_path, recursive=True)
File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 976, in get
self.get_file(rpath, lpath, callback=child, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 893, in get_file
os.makedirs(lpath, exist_ok=True)
File "/usr/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: ''
Marvin
08/20/2024, 3:24 PMwhat about this?```Downloading flow code from storage at '' Flow could not be retrieved from deployment. Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/prefect/engine.py", line 427, in retrieve_flow_then_begin_flow_run else await load_flow_from_flow_run( File "/usr/local/lib/python3.10/dist-packages/prefect/client/utilities.py", line 100, in with_injected_client return await fn(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/prefect/deployments/deployments.py", line 303, in load_flow_from_flow_run await storage_block.get_directory(from_path=from_path, local_path=".") File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 496, in get_directory return await self.filesystem.get_directory( File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 330, in get_directory return self.filesystem.get(from_path, local_path, recursive=True) File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 976, in get self.get_file(rpath, lpath, callback=child, **kwargs) File "/usr/local/lib/python3.10/dist-packages/fsspec
Blake Stefansen
08/20/2024, 3:24 PMDownloading flow code from storage at ''
09:35:45 AM
prefect.flow_runs
Flow could not be retrieved from deployment.
Traceback (most recent call last):
....
await storage_block.get_directory(from_path=from_path, local_path=".")
File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 496, in get_directory
return await self.filesystem.get_directory(
File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 330, in get_directory
return self.filesystem.get(from_path, local_path, recursive=True)
File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 976, in get
self.get_file(rpath, lpath, callback=child, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 893, in get_file
os.makedirs(lpath, exist_ok=True)
File "/usr/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: ''
Marvin
08/20/2024, 3:24 PMfrom_path
parameter in your deployment is empty, leading to a FileNotFoundError
. This can happen if the storage path in your deployment YAML is not set correctly or is missing.
### Steps to Fix:
1. Check Deployment YAML:
- Ensure that the path
specified in your YAML is correct and not empty.
2. Verify Storage Block:
- Check the configuration of the storage block you are using to ensure it is correctly set up. For example, if you're using a LocalFileSystem
block, verify its basepath
.
### Example Deployment Configuration:
Your deployment YAML should look something like this:
yaml
name: example-flow-deployment
description: An example deployment for flow
storage:
- block_type_slug: local-file-system
attributes:
basepath: /path/to/your/flow/code
path: /path/to/your/flow/code
### Create Deployment Command:
Ensure you correctly specify the --path
during deployment creation:
bash
prefect deployment build your_flow.py:flow_function --name your_deployment --storage-block local/your_storage_block --path /path/to/your/flow/code
### Adjust Manifest if Needed:
If the automatic deployment creation set an incorrect path, manually adjust the deployment manifest file (deployment.yaml
) to reflect the correct path.
### Documentation and References:
- Deployment and Storage Blocks:
https://docs.prefect.io/latest/concepts/deployments-block-based/
- Storage Guide:
https://docs.prefect.io/latest/guides/deployment/storage-guide/
Ensure these configurations are correct, and it should resolve the FileNotFoundError
due to an empty path.Kevin Grismore
08/20/2024, 3:25 PMKevin Grismore
08/20/2024, 3:25 PMBlake Stefansen
08/20/2024, 3:28 PM###
### A complete description of a Prefect Deployment for flow 'isp-portal-submission'
###
name: isp_portal_submission_yaml_deploy_k8
description: "This flow takes parameters specifiying the \nlocation of the isp file\
\ to be processed, extracts\nthe file from s3, then transforms and loads the file\
\ within our \nRDS instance of SQL Server. \n\n\nsubmission_key: Unique key associated\
\ with each isp submission\ns3_filepath: s3 filepath location for file submitted through the\
\ isp portal\n\nisp_portal_submission(\n \"123\",\n \"<S3://sslv-prod-sftp/bstefansen/TO_LIGHTBOX/TX_CSV_MOCK.csv>\"\
\n)"
version: e9eec5047a532d89afa5c4de4d4ec92d
# The work queue that will handle this deployment's runs
work_queue_name: kubernetes-broadband
tags: []
parameters: {}
schedule: null
infra_overrides: {}
infrastructure:
type: kubernetes-job
env: {}
labels: {}
name: null
command: null
image: digmap/broadband_prefect_jobs:1.081624.0
namespace: prefect2
service_account_name: null
image_pull_policy: null
cluster_config: null
job:
apiVersion: batch/v1
kind: Job
metadata:
labels: {}
spec:
template:
spec:
parallelism: 1
completions: 1
restartPolicy: Never
containers:
- name: prefect-job
env: []
imagePullSecrets:
- name: dockercloud-secret
customizations: []
job_watch_timeout_seconds: null
pod_watch_timeout_seconds: 300
stream_output: true
finished_job_ttl: null
block_type_slug: kubernetes-job
_block_type_slug: kubernetes-job
###
### DO NOT EDIT BELOW THIS LINE
###
flow_name: isp-portal-submission
manifest_path: null
storage:
bucket_path: lightbox-prefect/Staging/Broadband/isp_flow
aws_access_key_id: '**********'
aws_secret_access_key: '**********'
_block_document_id: ad04fc33-9c67-4966-8179-6f8fa49a35e5
_block_document_name: isp-flow-filesystem
_is_anonymous: false
block_type_slug: s3
_block_type_slug: s3
path: ''
entrypoint: flows/isp_portal_submission.py:isp_portal_submission
parameter_openapi_schema:
title: Parameters
type: object
properties:
submission_key:
title: submission_key
s3_filepath:
title: s3_filepath
required:
- submission_key
- s3_filepath
definitions: null
Blake Stefansen
08/20/2024, 3:29 PMpath
but that didn't seem to work.
The deployment as it is now worked back in march of this year.
It no longer works as of today.Blake Stefansen
08/20/2024, 3:39 PMBlake Stefansen
08/20/2024, 4:47 PMMarvin
08/20/2024, 4:47 PMFROM prefecthq/prefect:latest
COPY . /opt/prefect/flows/
WORKDIR /opt/prefect/flows/
2. Deploy the Flow:
- When deploying the flow, specify the storage block and path.
- Example command:
prefect deployment build your_flow.py:flow_function --name your_deployment --storage-block local/your_storage_block --path /path/to/your/flow/code
3. Worker Downloads Flow Code:
- The worker retrieves the flow code using the storage configuration, ensuring the code is present inside the container where the flow will run.
Further details and full guide can be found here.
Ensure the deployment YAML and Dockerfile are correctly set up to facilitate the seamless transfer of flow code into the Docker container. This guarantees that the code is available and executable when the container starts.Blake Stefansen
08/20/2024, 5:00 PMMarvin
08/20/2024, 5:02 PMget_directory
method is called, it downloads the contents of the specified path in the storage to the local directory.
Example:
python
await storage_block.get_directory(from_path="path/to/your/code", local_path=".")
This copies all files from path/to/your/code
in the bucket to the current working directory inside the Docker container.
### Entrypoint and Flow Execution:
- Entrypoint: Specifies the location and function within your code that Prefect should run as the flow.
- It uses the Python path-to-object syntax: path/to/file.py:function_name
.
- The worker/container uses this to locate the flow's entry function and execute it.
Example in YAML:
yaml
entrypoint: path/to/flow.py:flow_function
### Path in Deployment Configuration:
- Path: This refers to where the code is stored within your storage system. It must align with the from_path
used in the storage block configuration.
Purpose:
- During Build: This specifies where the deployment code will be stored.
- During Execution: It helps the storage block to retrieve and download the flow code to the worker/container environment.
Example Command:
bash
prefect deployment build your_flow.py:flow_function --name your_deployment --storage-block local/your_storage_block --path /absolute/path/to/your/flow/code
### Practical Example:
Here is a combined example that covers these concepts:
1. Deployment YAML Example:
yaml
name: example-flow-deployment
description: An example deployment for flow
storage:
- block_type_slug: local-file-system
attributes:
basepath: /path/to/your/flow/code
path: /path/to/your/flow/code
entrypoint: path/to/flow.py:flow_function
2. Deployment Command:
bash
prefect deployment build your_flow.py:flow_function --name your_deployment --storage-block local/your_storage_block --path /path/to/your/flow/code
By correctly setting the bucket path, entrypoint, and path properties, you ensure that Prefect knows where to get the code, which code to execute, and where to store or retrieve this code during deployment and execution.
### Additional Reading:
- Prefect Deployments: https://docs.prefect.io/latest/concepts/deployments/
- GCS Integration: https://docs.prefect.io/latest/integrations/prefect-gcp/cloud_storage/
These configurations work together to ensure your flow runs smoothly within a Docker container or any other specified infrastructure.Dave D
10/09/2024, 8:15 PMFlow could not be retrieved from deployment.
Dave D
10/09/2024, 8:16 PM