https://prefect.io logo
Join Slack
Channels
Powered by Linen
ask-marvin
  • c

    Chris White

    01/11/2023, 5:15 PM
    @Marvin can you tell me how to set retries on tasks?
    m
    n
    • 3
    • 7
  • j

    Jeremiah

    01/11/2023, 5:21 PM
    @Marvin whats up
    m
    • 2
    • 3
  • k

    Kyle McChesney

    01/11/2023, 6:07 PM
    @Marvin what is the meaning of life?
    m
    j
    • 3
    • 2
  • d

    Dmytro Ponomarchuk

    01/12/2023, 1:17 PM
    @Marvin How to run a subtask from a task in Prefect 2.0?
    m
    • 2
    • 1
  • k

    Karim Amrani

    01/13/2023, 7:56 AM
    @Marvin what's the strangest question you've ever had to answer?
    m
    • 2
    • 3
  • t

    Tim-Oliver

    01/13/2023, 9:51 AM
    @Marvin How can I fix this error 
    Task run 'f4eca18a-0509-46e9-b3ea-832074006308' received abort during orchestration: This run cannot transition to the RUNNING state from the RUNNING state. Task run is in RUNNING state.
    ?
    m
    j
    a
    • 4
    • 10
  • e

    Emma Rizzi

    01/26/2023, 9:36 AM
    @Marvin are you an open-source bot ?
    m
    j
    • 3
    • 8
  • d

    Dmytro Ponomarchuk

    01/26/2023, 12:39 PM
    @Marvin Why manual retries from the UI work only for flow runs triggered from deployments?
    m
    j
    • 3
    • 5
  • d

    Dmytro Ponomarchuk

    01/26/2023, 12:43 PM
    @Marvin How to retry not deployed sub-flows from the UI?
    m
    • 2
    • 1
  • s

    Slackbot

    02/01/2023, 12:29 PM
    This message was deleted.
    m
    l
    • 3
    • 2
  • l

    Luca Vehbiu

    02/01/2023, 12:32 PM
    @Marvin can you suggest a project template for a prefect etl repository
    m
    • 2
    • 1
  • e

    Emil Sorensen

    02/01/2023, 9:08 PM
    @Marvin how do I get started with prefect?
    m
    c
    • 3
    • 2
  • n

    Nate

    02/02/2023, 12:55 AM
    @Marvin how can I define a parent deployment that kicks off a bunch of child flow runs from different deployments and uses the child flow run results in the parent flow run?
    m
    • 2
    • 2
  • d

    Dmytro Ponomarchuk

    02/09/2023, 8:08 AM
    @Marvin How to stop using inactive AWS ECS TaskDefinition by a Prefect agent?
  • b

    Bigya Man Pradhan

    02/09/2023, 9:38 AM
    @Marvin How to fetch "flow name" in a running task method in Prefect 2 ?
  • d

    Dmytro Ponomarchuk

    02/09/2023, 11:45 AM
    @Marvin Are you alive?
    m
    • 2
    • 1
  • d

    Dmytro Ponomarchuk

    02/09/2023, 11:45 AM
    @Marvin How to stop using inactive AWS ECS TaskDefinition revisions by a Prefect agent?
    a
    • 2
    • 2
  • a

    Anna Geller

    02/09/2023, 12:11 PM
    @Marvin tell me how to stop using inactive AWS ECS TaskDefinition by a Prefect agent
    😆 1
  • a

    Alireza

    02/14/2023, 2:04 PM
    @Marvin I’ve got too many 429 errors on my local agent, is it because of my rate limit to call the api?
    m
    m
    • 3
    • 6
  • a

    Alireza

    02/14/2023, 8:07 PM
    @Marvin Is it possible to pass the number of DaskTaskrunner workers as a parameter of a flow?
    m
    • 2
    • 7
  • m

    Mrityunjoy Das

    02/16/2023, 7:14 AM
    @Marvin what's the possible cause for the following issue ?: Task run 'fc1c29d3-2aab-47ae-b898-c5d7604861e7' received abort during orchestration: This run cannot transition to the RUNNING state from the RUNNING state. Task run is in RUNNING state.
    m
    • 2
    • 1
  • s

    Slackbot

    02/16/2023, 12:49 PM
    This message was deleted.
    m
    • 2
    • 2
  • j

    Jen Evenson

    02/16/2023, 9:50 PM
    @Marvin How do we get 
    prefect.context.flow_name
    in Prefect 2?
    m
    n
    • 3
    • 5
  • t

    Tameem Syed

    02/18/2023, 7:40 AM
    @Marvin What is equivalent of apply_map in Prefect 2?
    m
    • 2
    • 1
  • t

    Tameem Syed

    02/18/2023, 2:11 PM
    @Marvin How to programmatically create Service Accounts and Customs Roles in Prefect 2?
    m
    • 2
    • 1
  • t

    Tameem Syed

    02/18/2023, 2:16 PM
    @Marvin How to deploy Prefect 2 Agent in Azure Kubernetes Cluster using Helm Charts?
    m
    • 2
    • 1
  • s

    Slackbot

    02/18/2023, 7:15 PM
    This message was deleted.
    m
    • 2
    • 1
  • t

    Tameem Syed

    02/19/2023, 2:31 PM
    @Marvin Does Prefect 2 support imperative API?
    m
    • 2
    • 1
  • b

    Bigya Man Pradhan

    02/21/2023, 9:07 AM
    @Marvin Does prefect 2 have LOOP context?
  • j

    Joshua Grant

    02/21/2023, 9:54 PM
    @Marvin in prefect 2 what is the environmental variable to set to specify the flow path?
    m
    m
    a
    • 4
    • 27
Title
j

Joshua Grant

02/21/2023, 9:54 PM
@Marvin in prefect 2 what is the environmental variable to set to specify the flow path?
m

Marvin

02/21/2023, 9:54 PM
It looks like the environment variable you're asking about is 
PATH
and 
PREFECT__FLOW_PATH
. You can find more detailed information about this in the [Prefect Documentation](https://docs.prefect.io/ui/flows/), especially in the topics on Flows, Deployments, and Logging. Hope this helps!
m

Michael Adkins

02/21/2023, 10:05 PM
There is no environment variable for this in Prefect 2
What are you trying to do?
j

Joshua Grant

02/21/2023, 10:06 PM
I think it is 
PREFECT_LOCAL_STORAGE_PATH=
, I am trying to tell prefect where to look for a flow when building a deployment from flow without specifying storage (flow logic is in the container, as is the flow files).
error: 
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/flows'
m

Michael Adkins

02/21/2023, 10:07 PM
What’s the entrypoint for your deployment?
j

Joshua Grant

02/21/2023, 10:08 PM
there isn't one. we found setting one caused execution of the entrypoint and the flow when the flow ran on the deployment infrastructure (
ECSTask
)
m

Michael Adkins

02/21/2023, 10:09 PM
That’s a little confusing — the entrypoint is how we load the flow for execution
Are you saying it ran twice?
Once at load, then again after?
j

Joshua Grant

02/21/2023, 10:10 PM
correct.
m

Michael Adkins

02/21/2023, 10:10 PM
Sounds like you’re calling your flow in the file then?
You should guard that with 
if __name__ == "__main__":
   flow_call()
j

Joshua Grant

02/21/2023, 10:11 PM
it was guarded. No matter what we put in the entrypoint, it was executed along with the flow.
but that was back when we were using github storage. We want to use local storage now.
possible incorrect configuration of the 
Dockerfile
?
FROM public.ecr.aws/bitnami/python:3.9-prod
WORKDIR /app
RUN apt-get update && apt-get install git libmagic-dev -y && \
      apt-get clean && \
      pip install --upgrade pip && \
      pip install pipenv
COPY . .
ARG GIT_API_KEY
ARG COMMON_PKG_VERSION="dev"
ENV PREFECT_LOCAL_STORAGE_PATH="/app"
RUN pipenv lock --clear && pipenv install --system
m

Michael Adkins

02/21/2023, 10:12 PM
Setting the local storage path won’t make us look for a flow there
We use this function https://github.com/PrefectHQ/prefect/blob/main/src/prefect/deployments.py#L150
There needs to be an entrypoint of some sort, I’m pretty confused about your setup tbh.
j

Joshua Grant

02/21/2023, 10:15 PM
well, it's running fine no entrypoint with github storage, but with I think Prefect 2.5.x > < 2.7.x it reported a 
shutil.copytree
error when using local storage
a

Anna Geller

02/22/2023, 1:09 AM
it would be easiest to use Prefect base image and then using /opt/prefect/flows as work dir should make Prefect look in the right place, could you try that? 
FROM prefecthq/prefect:2-python3.9
WORKDIR /opt/prefect/flows
also --path CLI argument will point your deployment to the path within the docker image
j

Joshua Grant

02/22/2023, 3:33 PM
Interesting. We are probably not using prefect as intended then. We now have it working with the above Dockerfile (removal of the 
PREFECT_LOCAL_STORAGE_PATH
envar breaks the flow. Here is a brief overview of how we are deploying Prefect v2 with this method: 1. a flow-centric image is created in AWS CodeBuild based on Dockerfile and is pushed to ECR. 2. CodeBuild fires off a script that spins up an ECS Task in the targeted AWS environment 3. ECS Task creates a task definition using the image created in CodeBuild, it overrides some things and adds sidecars to the new task definition 4. Above task definition is passed to prefect deployment script, which creates an 
ECSTask
infrastructure block and then builds the flow (or flows) and registers them with prefect
some business requirements: • all of the deployment logic is handled in Python • build environment is siloed from deploy environments; we use (dev, alpha, beta, prod) • shared `prefect.task`s and deployment functions (and data like a 
task_definitoin.yaml
that can be parsed with 
pyaml-env
) are stored in a common python package We are using 
prefect==0.15.2
simultaneously, but run into some issues with GraphQL. We are migrating from that prefect version to prefect 2
a

Anna Geller

02/22/2023, 5:56 PM
Prefect 1 and 2 are entirely different products and they have very different deployment patterns • v1 ECS example • v2 ECS example
j

Joshua Grant

02/22/2023, 6:12 PM
I think our biggest challenge migrating to v2 was to allow the addition of sidecars and avoiding getting the too-many-revisions to a task definition issue. Our current deployment strategy is working as expected albeit probably not what it was designed to do. Some things with 
ECSTask
creation had to be modified (providing 
task_definition_arn
and 
auto_deregister_task=False
) to allow for us to use sidecars for logging to DataDog (among other things) and avoid the too-many-revisions. In our workflow, once a flow is "deployed" it is the source of truth until overwritten by another "deployment". The source of all this discussion was removing the need for a 
Github
block and using the flow stored in the image, which the addition of the envar 
PREFECT_LOCAL_STORAGE_PATH
solves (magically). This strategy allows us to deploy multiple 
ECSTask
infrastructure blocks (`EC2`|`Fargate`) with varying amounts of memory and cpu. Downstream workflow logic determines the infrastructure flow to run (e.g. 
-ec2-light
or 
-fg-heavy
) in the flow name, which for light processing that is needed to complete quickly we use 
EC2
with low resource request.
View count: 2