Kazushi Nagayama
08/18/2023, 1:37 PMMarvin
08/18/2023, 1:38 PMFROM prefecthq/prefect:2-python3.9 # example base image
RUN pip install s3fs prefect-aws
2. Resource Allocation: Make sure to allocate enough CPU and memory to your agent. Your agent needs enough resources to provision infrastructure for your flow runs and to monitor their execution. Otherwise, your flow runs may get stuck in a pending state. You can also set a work-queue concurrency limit to ensure that the agent will not try to process all runs at the same time.
3. API URL Setting: When using workers, agents, and work pools that can create flow runs for deployments in remote environments, PREFECT_API_URL
must be set for the environment in which your worker or agent is running. For example, if you're using Prefect Cloud, your PREFECT_API_URL
might look something like this:
PREFECT_API_URL="<https://api.prefect.cloud/api/accounts/[ACCOUNT-ID]/workspaces/[WORKSPACE-ID]>"
You can find your Account ID and Workspace ID in your browser URL when logged into Prefect Cloud.
4. Retries: Consider adding retries to handle potential failures in API calls to provision infrastructure.
For more detailed information, you might want to visit the Prefect documentation. But then again, you might not. Life is full of such difficult choices. Infrastructure and Settings might be good places to start.Kazushi Nagayama
08/18/2023, 1:39 PMMarvin
08/18/2023, 1:39 PMFROM prefecthq/prefect:2-python3.9 # example base image
RUN pip install google-cloud-storage prefect-gcp
2. Resource Allocation: Make sure to allocate enough CPU and memory to your agent. Your agent needs enough resources to provision infrastructure for your flow runs and to monitor their execution. Otherwise, your flow runs may get stuck in a pending state. You can also set a work-queue concurrency limit to ensure that the agent will not try to process all runs at the same time.
3. API URL Setting: When using workers, agents, and work pools that can create flow runs for deployments in remote environments, PREFECT_API_URL
must be set for the environment in which your worker or agent is running. For example, if you're using Prefect Cloud, your PREFECT_API_URL
might look something like this:
PREFECT_API_URL="<https://api.prefect.cloud/api/accounts/[ACCOUNT-ID]/workspaces/[WORKSPACE-ID]>"
You can find your Account ID and Workspace ID in your browser URL when logged into Prefect Cloud.
4. Retries: Consider adding retries to handle potential failures in API calls to provision infrastructure.
For more detailed information, you might want to visit the Prefect documentation. But then again, you might not. Life is full of such difficult choices. Infrastructure and Settings might be good places to start.