Karen
07/15/2020, 1:12 AMChris White
07/15/2020, 4:09 AMflow.build()
to produce the flow artifact in the s3 bucket, and then save the output of flow.serialize()
to store the JSON metadata representation of the Flow that is necessary to make the registration call with the API in the future.
3. This is handled by using unique filenames within your s3 bucket; if you use default Prefect settings for S3 storage, we typically use a timestamp so that different versions can be distinguished. This S3 file location is stored in the backend to distinguish between the versions. In particular, if you want to update one of your tasks without re-registering, you will need to ensure you store the updated Flow in the exact same location that the API is aware of.
4. I’m actually not 100% sure what you mean here; labels will ensure that only certain agents run your Flow, but whether the agent is truly compatible with your Flow is ultimately user responsibility.
There are two additional things worth mentioning based on your questions:
- you will have to do some heavy lifting to deploy the open source API in the way you describe; specifically, you’ll need to make sure all agents / CI jobs can communicate with your self-hosted API. This is one of the major benefits of the Cloud API is that is accessible from anywhere (with an auth token)
- the Flow artifact that is ultimately stored in the S3 bucket does not contain all possible dependencies of your Flow. You’ll need to make sure that any imports the Flow relies on are available at runtime; the most common way to achieve this is to run your flow within a Docker image that has all files / packages on the import pathKaren
07/15/2020, 5:55 AM