<@ULVA73B9P> I am having trouble with the followin...
# marvin-in-the-wild
f
@Marvin I am having trouble with the following code:
Copy code
from prefect import flow, get_run_logger
from prefect_aws.s3 import S3Bucket

if __name__ == "__main__":
    flow = flow.from_source(
        source=S3Bucket.load("s3-storage-block"), entrypoint="FILE_PATH:flow_function"
    )
    print(flow)
    flow.deploy(name="test-deploy-s3", work_pool_name="local-pool")
The code is not able to move past the flow.from_source part. However, if I try the following approach, it works:
Copy code
s3_bucket = S3Bucket.load("s3-storage-block")
with open("LOCAL_FILE", "wb") as f:
    s3_bucket.download_object_to_file_object("FILE_PATH", f)
It successfully downloads the file from S3, but getting the flow from S3 does not work Could you please help me understand why the flow.from_source part is not working and how I can fix it? Using Prefect3
m
It appears that your issue with
flow.from_source
and S3 may be related to a bug or a limitation where the dependencies for the flow need to be present in the environment. Here are a few crucial points: 1. Dependencies Issue: The flow's dependencies must be present in the environment where
flow.from_source().deploy()
is executed. If there are missing modules in your S3 bucket (as highlighted in the issue https://github.com/PrefectHQ/prefect/issues/14464), the flow cannot be created from the source. 2. Alternative Method: You can manually download the file from S3 (as in your second example) and ensure that all dependencies are present locally. Then, call
flow.deploy()
. 3. Using Remote Storage: The use of
RemoteStorage
can help if your dependencies are spread out. The Prefect documentation and issues indicate improvements in handling storage blocks. You can refer to this document. 4. Advanced Configurations: Prefect has improved support for remote sources and various storage solutions. Refer to this guide for additional approaches. For now, I would recommend ensuring that your environment has all the necessary dependencies and avoid relying solely on the
flow.from_source
if the dependencies are complex. Here are some more relevant links: - Issue regarding missing dependencies - Added support for storage blocks I hope this helps you troubleshoot the problem more effectively.
It seems there might be a couple of issues causing the
flow.from_source
to fail in your Prefect 3 setup. ### Key Points: 1. Dependency Issues: The environment where
flow.from_source
is executed should have all dependencies present. 2. Storage Block Support: Ensure you are using the correct method to pull configurations from S3. 3. Prefect Configuration: Deploy commands should be run from the root level of the repository. ### Suggestions: 1. Check Dependencies: Ensure that the system executing
flow.from_source
has all the necessary dependencies (libraries, configurations). 2. Manual Pull: Alternatively, you could manually download the flow file from S3 as illustrated in your example and then execute the flow using
flow.deploy()
. 3. Correct Usage: Ensure the
S3Bucket
block is correctly configured and try to use more verbose error logging to understand what’s failing. Refer to: - Upgrading Agents to Workers: https://docs.prefect.io/latest/guides/upgrade-guide-agents-to-workers/ - AWS S3 Integration Documentation: https://docs.prefect.io/latest/integrations/prefect-aws/s3/ If the problem persists, manual fetching and setting the flow might be a workaround until all elements align.
f
I dont get any type of error the code just stays running for several minutes without no response