Hey everyone, it looks like you've done a lot with prefect server! Thanks for the awesome tool. I have a few quick questions about the updates to server/cloud:
1. While the free (dev) version of cloud has a limit of 1 concurrent flow run, is there any limit to concurrency in server? I realize the custom concurrency feature isn't available, but I'm shooting for an unlimited concurrency here.
2. Originally, agents for prefect server had to be on the same device. Is that still the case? Or can I now have any number of agents distributed accross devices & clusters -- just like with cloud? Also shoutout to making Agent requests/signaling one directional (from Agent to Prefect API). This helps soooo much with not needing to deal with firewalls and opening ports on HPC clusters. Thank you!
3. Because I'm still early in my codes development, I need to reset my Postgres database often and then start from scratch for testing. This is pretty easy for server, but what is the easiest way for cloud? The graphql interface ("interactive API")? There used to be an option to reset your data in the user settings that I liked to use.
01/14/2021, 6:49 PM
Hi @Jack Sundberg and thanks for the kind feedback!!
1. There are no concurrency limits in Server; also, it’s worth calling out that the free developer version of Cloud is going to be undergoing a pricing update in the next few weeks (it’s imminent!) which will allow for unlimited flow concurrency on the free tier so stay tuned for that change
2. Glad you’re finding the hybrid approach useful! The single node constraint for server is more about what we formally support and less about an inherent constraint of the setup. If you know how to network to the Server API then you could run flows / agents on diverse infrastructure, but you’d have to configure it and set it up. There has been some great work on a server helm chart that you might find useful here: https://github.com/PrefectHQ/server/tree/master/helm/prefect-server
3. I think I’ll need some more context on this particular issue - for Cloud, what is the reason for clearing your data? You can definitely use the graphQL interface for running delete commands.
01/14/2021, 7:21 PM
Awesome, thank you! That's awesome to hear. And for question 3, I'm in a unique spot compared to your typical users haha. I'm making a software repo that has predefined workflows users may need (say a chemist wants to calculate a specific material property, they can use a standard workflow with their personal structure as the parameter). Since its a repo, I want to automate the users cloud configuration and workflow registration -- each lab will want their personal setup as to not share data until it's ready. So to test automated configuration of the cloud, I need to constantly reset my projects/workflows in cloud and then test my setup scripts. This project is also why I contributed module path as an option for local storage -- you can access my workflow library via an import in python such as "from myrepo.workflows import calculateconductivitywf"
Sorry if that's too much detail lol. But that's the reasoning why I'd like to reset all of my projects and database for testing.
01/14/2021, 8:16 PM
Very interesting, yea I see; we don’t currently have a top level “delete all data” button (for most of our users with large quantities of runs, deleting can take a long time), so you’d want to create a utility for yourself to do so. It wouldn’t be too difficult, I’d recommend starting at the low level of deleting task runs, then individual flows, then projects last
01/14/2021, 8:34 PM
Alright gotcha. So I'll just write a function that executes the proper graphql mutations on the server. Easy enough! Based on your suggestion, deleting the project first won't cascade and delete the data associated with it (flow and task runs). Is that right? In that case, I'll have to execute a mutation for each table from bottom-up.
01/14/2021, 8:37 PM
Good question - it will cascade, but depending on how big your flows are and how often you run them, the cascade might timeout (it has to cascade through ~6 different tables of varying sizes), so you can definitely try that first and if it doesn’t complete move onto the more granular calls
01/14/2021, 8:40 PM
Sweet, that helps a lot and I have a clear idea of what to do now. Thanks for answering all of my questions!!
01/14/2021, 8:40 PM
02/10/2021, 9:58 AM
@Chris White Hi, you mentioned the "unlimited concurrency in free tier" Prefect Cloud update in your first message (about 27 days ago). You said that the update shall take place during the next weeks. Our engineering team is looking forward to this update, but it is somewhat unclear how much of a delay in production waiting it will cause.
Simply put: can you name some prescise period during wich we shall wait for the update to take place? Maybe date or something...