Morning, quick question does prefect have to be in...
# prefect-communityb
Byron Kilian
08/19/2022, 7:10 AMMorning, quick question does prefect have to be installed where the agent is running from? If so does it not create 2 database instances?. When prefect is installed on host running the webservice the .prefect directory and orion.db instance gets created. And then when installing prefect again on the host from where the flows are being run then another .prefect directory and orion.db instance gets created?
✅ 1
b
Bianca Hoch
08/19/2022, 2:30 PMHello Byron, in short, yes. Prefect must be installed in any environment in which you intend to run the agent or execute a flow run.
Bianca Hoch
08/19/2022, 2:41 PMIn regard to whether or not it creates a new instance, let me consult with the team and we'll get back to you here.
b
Byron Kilian
08/20/2022, 8:05 AMHi Bianca thanks for the reply. I hope you dont mind, but I think this thread might get a bit lengthy as I have lots of confusion around the setup and documentation. I don't know if there are concepts introduced in Prefect V1, but I'm starting from the documentation on Prefect V2. The context I have are just based on what I'm reading within this documentation. Example for work-queue setup, it speaks to agent configuration, but its not clear how the agent is configured. The backend seems to expect certain fields that describes the concept of an agent, but I cannot seem to find what that is from the PrefectV2 documentation. From my screenshot it looks like "prefect work-queue" CLI command has to run first in order to create the entry on table "work_queue" which is then a key into the "agent" table (work_queue_id), but I can run "prefect agent start" command even though the agent on table "agent" does not exist yet. I need to understand how to configure the agent first and only then it should allow me to start the agent schedule. The command "prefect agent start <queue-id>" is also confusing. When the agent is created it should already be associated with a queue, and the command prefect agent start <agent-id> make more sense. Please let me know your thoughts
Byron Kilian
08/20/2022, 8:09 AMthis is something in addition to the multiple environments that require prefect to be installed. There is the agent process that could run on hostA. There is the execution environment where the agent deploys the flow run which also requires prefect to be installed (lets say on hostB), then there is the orion service (webservice) that runs on a hostC that also requires prefect to be installed. In each of these cases the .prefect project directory gets created, and I'm confused as to why and what prefect environment variables to set for each of these hosts. Will continue to discover in the meantime while I await feedback.
Byron Kilian
08/20/2022, 8:57 AMHi Bianca, some feedback on the definitions used. The term "concurrency_limit" is used in two different configurations. It refers to both queue concurrency and task concurrency limits which is confusing. My suggestion here is to use task_concurrency_limits for tasks and also rename the database tbl and prefect CLI command accordingly (+ doc update) and the concurrency_limit field on the work_queue table to remain as is, since it is an attribute of the work-queue object and makes sense when configuring it
Byron Kilian
08/20/2022, 9:58 AMHi Bianca, I notice when a flow is configured (within your code base) and then run the flow manually with (python my_flow.py) it will automatically populate the table "flow" on the backend. However the DB model object does not align with what can be configured from the documentation explaining the flow concept. image below shows the configuration vs the backend
Byron Kilian
08/20/2022, 10:00 AMFrom the documentation these configurations can be set, but where does it persist?
5 Views