• a

    Alex Furrier

    1 year ago
    I'm trying to run a "flow of flows" following the documentation on dependent flows here. I'm trying to parametrize this flow and pass those down to the dependent flows. I can pass params to the first flow but when it tries to run the dependent flow with params they all come back as null and the sub flow fails.
    flow_a = StartFlowRun(flow_name="flow a", project_name="my project", wait=True)
    flow_b = StartFlowRun(flow_name="flow b", project_name="my project", wait=True)
    
    with Flow('flow-of-flows') as flow_of_flows:
        param_a = Parameter('param_a', 'a')
        param_b = Parameter('param_b', 'b')
    
        flow_a(parameters={"param_a": param_a})
        flow_b(parameters={"param_b": param_b}, upstream_tasks=[flow_a])
    In the UI if I attempt to run this flow with params like
    {'param_a': 'parametrized_1', 'param_b':'parametrized_b'}
    it will create a task running
    flow_a
    and fail. Viewing the initiated flow for
    flow_a
    it shows the parameters being
    {'param_a': null}
    .
    flow_a
    fails and
    flow_b
    fails before it even runs since it's dependent on
    flow_a
    and says a reference task failed. Is there a proper way to pass parameters between parent and child flows?
    a
    Kevin Kho
    5 replies
    Copy to Clipboard
  • h

    Hugo Polloli

    1 year ago
    Hi, I'm trying to setup my prefect backend and one agent to be ever-running on my EC2 instance, docker is already automatically starting and since restart is set to true in docker-server's compose.yml, prefect server also automatically starts ever since I launched it once with --detach. But how do I make the agent automatically start on boot too ? according to the docs I could use supervisor with the local agent, is there any alternative using the docker agent ? Should I make my own compose.yml for the agent ?
    h
    Michael Adkins
    +2
    64 replies
    Copy to Clipboard
  • d

    Daniel Davee

    1 year ago
    If I have a docker agent running on a vm with a local docker repo and am using dockers stoage and a docker run, it should pickup the job?
    d
    Kevin Kho
    6 replies
    Copy to Clipboard
  • Amogh Kulkarni

    Amogh Kulkarni

    1 year ago
    Hi I want to deploy prefect-server on local docker how we can do that?
    Amogh Kulkarni
    1 replies
    Copy to Clipboard
  • YD

    YD

    1 year ago
    If I have a prefect.io running on a VM and I want to test the cloud server, do I need to stop the agent running on my laptop first, or can I have the agent for the cloud server and the agent for the VM server running at the same time ?
    YD
    Kevin Kho
    8 replies
    Copy to Clipboard
  • Jonas

    Jonas

    1 year ago
    Hello, I am currently trying to register a flow to a local prefect server, but I ran into the following issue:
    TypeError: cannot pickle 'weakref' object (running this with flow.run worked fine)
    I narrowed it down to the use of a sqlalchemy session object I use to send and retrieve data from a database and that the object can't be serialized. Now I'm a bit clueless as to how I should solve this because I'm unsure with what steps to take next.
    from prefect import task, Flow,case, Parameter
    from sqlalchemy import orm
    from sqlalchemy import orm
    
    engine = create_engine("mysql+mysqldb://{USER}:{PASSWORD}@{HOST}:{PORT}/{NAME}".format(**json.load(f)))
    session = orm.Session(bind=engine)
    
    @task(max_retries=3, retry_delay=timedelta(seconds=10))
    def commit_to_db(object_list: List):
    
        session.add_all(object_list)
        try:
            session.commit()
    
        except exc.SQLAlchemyError as error:
            session.rollback()
            #logger.warning(error)
            raise Exception(error)
    
        session.close()
    Jonas
    Kevin Kho
    12 replies
    Copy to Clipboard
  • c

    Chris Bowen

    1 year ago
    Hello, not sure if this will be helpful for anyone else, but figured I'd share. Not sure where else to post it. A couple weeks ago I posted here asking how to manage the Markdown for flow groups in a more programmatic way than just writing it in the UI (I want to keep mine in
    .md
    files in a repo).@Zach Angell provided me with the GraphQL to start out and I expanded on it to make a solution that works for me. I'm storing mine in a few functions, but I'll just provide the entire thing as a script here. This script assumes a few things (I'll edit/update if I think of any more or anyone points any out):1. This script is running in a directory that contains all of your
    .md
    files that you want to deploy 2. Your
    .md
    files are named the same thing as your flows (see the
    active_flows
    object returned from the GraphQL query) 3. Any
    \
    characters in your
    .md
    files need to be escaped with a second
    \
    or the deployment won't work 4. I believe including any "characters in the
    .md
    files also blew up the API call, but
    '
    was fine CODE IN THREAD I've got this divided up into three functions, but posting it in this format seemed like it might be easier to read. One thing I did notice is that the Prefect UI doesn't seem to handle code blocks very elegantly. I tried using the standard three ticks (`), tildas (~), with/without specifying the code base, and using four spaces. I attached a couple pictures that show the input and output. Not a huge deal, but curious why it doesn't "block" like most other platforms I've seen. Anyways, happy to hear any feedback, suggestions, etc. Also just happy to share it if it helps anyone else at all!
    c
    Kevin Kho
    +1
    8 replies
    Copy to Clipboard
  • Nathan Walker

    Nathan Walker

    1 year ago
    Ohhhh, I did a bad and I need some help. What I did: I created a Flow that creates other Flows and haven't implemented the "don't do this infinitely" part yet, so it grew and grew and now I have a few dozen Flows in "Running" and "Scheduled" statuses. I took my Agent offline, so it's not creating new Flows any more, but I need to cancel all the Flows. Going through each one and clicking "Cancel" in the UI will take quite some time. What I need: Some GraphQL query to cancel all Running and Scheduled Flows. I can query for them and I can call the mutation to cancel them, but I don't know how to have GraphQL iterate through each returned id and run a mutation on it.
    Nathan Walker
    Kevin Kho
    2 replies
    Copy to Clipboard
  • Mateusz Pazdzior

    Mateusz Pazdzior

    1 year ago
    Hello! Do you know if it's possible to pay via bank wire? I've already contacted sales@prefect.io but I have no answer yet.
    Mateusz Pazdzior
    Kevin Kho
    +1
    4 replies
    Copy to Clipboard
  • Alfie

    Alfie

    1 year ago
    Hi Team, now I’m using (Prefect core 0.13) flow.state_handlers to do some actions once the state is changed, and the call back method takes three params: flow, old_state, new_state. What I want is to access some data generated during the flow execution in the handler and it’s used to assistant the triggered action. Any suggestions about how can I make the data accessible in the handler? Thanks
    Alfie
    Kevin Kho
    9 replies
    Copy to Clipboard