https://prefect.io
Join Slack
`prefect deploy build` with an S3 upload block is hanging unless I use `--skip-upload` ```$ prefect...
# ask-community
a
prefect deploy build
with an S3 upload block is hanging unless I use 
--skip-upload
Copy code
$ prefect deployment build flows/helloworld/helloworld_flow.py:helloworld -sb s3/prod -n helloworld --skip-upload
Found flow 'Hello World'
Deployment YAML created at '/home/aradox/code/wfp/wfp-prefect/helloworld-deployment.yaml'.

$ prefect deployment build flows/helloworld/helloworld_flow.py:helloworld -sb s3/prod -n helloworld
Found flow 'Hello World'
✅ 1
I realized it's not actually hanging, but it's slowly uploading the entire repository to the s3 bucket. Is that the intended behavior? If so, why?
I actually still think it hangs, the upload is definitely complete but the method doesn't finish running
j
The command uploads all files/folders starting from the present working directory. Whatever directory you use the command in, you will want to make a .prefectignore file and add to it files you don't want uploaded. https://docs.prefect.io/concepts/deployments/#create-a-deployment-on-the-cli
a
thank you!
Any idea why it would hang after upload?
j
What output are you seeing from the terminal when the hanging is happening?
Without seeing anything my hunch is that the upload isn't fully complete and it's still working on uploading files that should be ignored
👍 1
a
no output from the terminal
Ok ill try ignoring a bunch of stuff and seeing if that helps
👍 1
the output from the terminal is what i posted above - the final command never finishes
r
I saw the same thing, I think it’s just extremely slow at uploading to S3. If I limit the files sent using .prefectignore (eg no libraries or pycache) it’s still slow but finishes after several seconds
👍 1
j
Here's my .prefectignore. Maybe there's something you're not catching like .git/ files?
Copy code
# Project-specific
data/
infrastructure/
utilities/

# General files
.secrets
Dockerfile
*.toml
*.md
*.*sh
*.h5ad
*.parquet
*.png
*.svg

# python artifacts
__pycache__/
*.py[cod]
*$py.class
*.egg-info/
*.egg

# Type checking artifacts
.mypy_cache/
.dmypy.json
dmypy.json
.pyre/

# IPython
profile_default/
ipython_config.py
*.ipynb_checkpoints/*

# Environments
.python-version
.env
.venv
env/
venv/

# MacOS
.DS_Store

# Dask
dask-worker-space/

# Editors
.idea/
.vscode/

# VCS
.*ignore
.git/
.github/
🙏 1
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by