So, Deployments are taking a realllly long time on our CI/CD machine. They're reasonable-ish when I run them interactively, but in CI/CD a super-stripped down "Hello World" flow took about 20 minutes to deploy. I'm using
ECSTask
and
s3
storage, and an
s3
filesystem. And while it's deploying it keeps repeating
botocore.credentials - Found credentials in environment variables.
n
Nico Neumann
06/17/2023, 6:26 AM
Are you uploading many small files?
Maybe there are files/dirs which you could add to the .prefectignore file if you don’t need them
m
Matt Alhonte
06/21/2023, 8:22 PM
Oh, I didn't realize the behavior now was to upload everything in the Working Directory. So the solution will be
Find every file in our Flows directory
Make a new folder for each one
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.