https://prefect.io logo
Join Slack
Powered by
Hello! I need help to configure s3 storage for flo...
# ask-community
e
Hello! I need help to configure s3 storage for flows I have an not-AWS, S3-compatible provider (already works great with minio client and such), but i failed to configure it with prefect blocks as storage provider I used the s3-bucket block with aws-credentials, configured through the UI with my custom endpoint, then I built with : 
prefect deployment build flow.py:check -n k8sjob -sb s3-bucket/my-s3 -i kubernetes-job --override env.EXTRA_PIP_PACKAGES="s3fs prefect-aws" -a
The build ends with success, and I can see the flow.py being uploaded in my bucket. When I launch the flow, it fails to retrieve the code, stack trace in thread!
✅ 1
Error when launching the flow :
Copy code
Downloading flow code from storage at ''
04:47:41 PM
prefect.flow_runs

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 839, in exec_module
  File "<frozen importlib._bootstrap_external>", line 975, in get_code
  File "<frozen importlib._bootstrap_external>", line 1032, in get_data
FileNotFoundError: [Errno 2] No such file or directory: 'flow.py'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 274, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.8/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/deployments.py", line 187, in load_flow_from_flow_run
    flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.8/site-packages/prefect/flows.py", line 809, in load_flow_from_entrypoint
    flow = import_object(entrypoint)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/importtools.py", line 197, in import_object
    module = load_script_as_module(script_path)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/importtools.py", line 160, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'flow.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
04:47:42 PM
prefect.flow_runs

Downloading flow code from storage at ''
04:48:20 PM
prefect.flow_runs

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 839, in exec_module
  File "<frozen importlib._bootstrap_external>", line 975, in get_code
  File "<frozen importlib._bootstrap_external>", line 1032, in get_data
FileNotFoundError: [Errno 2] No such file or directory: 'flow.py'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 274, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.8/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/deployments.py", line 187, in load_flow_from_flow_run
    flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.8/site-packages/prefect/flows.py", line 809, in load_flow_from_entrypoint
    flow = import_object(entrypoint)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/importtools.py", line 197, in import_object
    module = load_script_as_module(script_path)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/importtools.py", line 160, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'flow.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
The flow if relevant, just copied from a tutorial :
Copy code
from prefect import flow, get_run_logger
from platform import node, platform

@flow
def check():
    logger = get_run_logger()
    <http://logger.info|logger.info>(f"Network: {node()}. ✅")
    <http://logger.info|logger.info>(f"Instance: {platform()}. ✅")

if __name__ == "__main__":
    check()
and the deployment.yaml just in case i missed something:
Copy code
###
### A complete description of a Prefect Deployment for flow 'check'
###
name: k8sjob
description: null
version: e433c7d8474dc025c33ba4ec3a690abe
# The work queue that will handle this deployment's runs
work_queue_name: default
work_pool_name: default-agent-pool
tags: []
parameters: {}
schedule: null
is_schedule_active: true
infra_overrides:
  env.EXTRA_PIP_PACKAGES: s3fs prefect-aws
infrastructure:
  type: kubernetes-job
  env: {}
  labels: {}
  name: null
  command: null
  image: prefecthq/prefect:2.8.7-python3.8
  namespace: default
  service_account_name: null
  image_pull_policy: null
  cluster_config: null
  job:
    apiVersion: batch/v1
    kind: Job
    metadata:
      labels: {}
    spec:
      template:
        spec:
          parallelism: 1
          completions: 1
          restartPolicy: Never
          containers:
          - name: prefect-job
            env: []
  customizations: []
  job_watch_timeout_seconds: null
  pod_watch_timeout_seconds: 60
  stream_output: true
  finished_job_ttl: null
  block_type_slug: kubernetes-job
  _block_type_slug: kubernetes-job

###
### DO NOT EDIT BELOW THIS LINE
###
flow_name: check
manifest_path: null
storage:
  bucket_name: prefect
  minio_credentials: null
  aws_credentials: null
  basepath: ''
  endpoint_url: null
  credentials:
    aws_access_key_id: ###
    aws_secret_access_key: '**********'
    aws_session_token: null
    profile_name: null
    region_name: eu-west-1
    aws_client_parameters:
      api_version: null
      use_ssl: true
      verify: true
      verify_cert_path: null
      endpoint_url: ###
      config: null
    _block_document_id: ###
    _block_document_name: s3-credentials
    _is_anonymous: false
    block_type_slug: aws-credentials
  bucket_folder: prefect
  _block_document_id: ###
  _block_document_name: my-s3
  _is_anonymous: false
  block_type_slug: s3-bucket
  _block_type_slug: s3-bucket
path: ''
entrypoint: flow.py:check
parameter_openapi_schema:
  title: Parameters
  type: object
  properties: {}
  required: null
  definitions: null
timestamp: '2023-03-30T14:45:30.993032+00:00'
When checking the documentation, it seems that S3 Bucket block is used inside tasks, not as storage configuration, I wonder if it's then compatible for this purpose, but the S3 Filesystem doesn't provide an endpoint parameter either
r
the S3 block is the one you need, it has a bucket path
c
add a “/” at the end of your path
e
@Christopher Boyd thanks that was it ! 🙂
✅ 1
🙌 1
Open in Slack
PreviousNext
Powered by