• d

    Daniel

    2 years ago
    Hi, just joined this communit. Have started using Prefect in the past month. Have worked with different ETL tools in the past, and really think this is the best so far! My question for now: I am building an open data set for the Netherlands (also pertaining to mapping the spread of COVID-19) and have about 5 smaller flows that pulls data from public sources (bureau of statistics, database of addresses, reported cases etc.). What is the most idiomatic way of running these independent flows in parallel (prior to the actual modeling that needs to be done)? Just execute the data collection flows all at once on a DaskExecutor? Or is there a way to combine e.g. four flows into a fifth that is dependent on the four independent ones?
    d
    Jeremiah
    +1
    6 replies
    Copy to Clipboard
  • z

    Ziyao Wei

    2 years ago
    Q: is there a way to cache output persistently?
    z
    Laura Lorenz (she/her)
    4 replies
    Copy to Clipboard
  • k

    Kostas Chalikias

    2 years ago
    Hello there, last time we checked it was not possible to kill a running flow/task from the UI. Has this been fixed now or is there a workaround I can use?
    k
    Zachary Hughes
    2 replies
    Copy to Clipboard
  • k

    Kamil Okáč

    2 years ago
    Hello again, guys. Do I understand correctly that it's not possible to restart server and persist its state (information about past flow/job runs etc.)?
    k
    Jeremiah
    2 replies
    Copy to Clipboard
  • d

    David Ojeda

    2 years ago
    Hi, I have a question that is a follow-up from one question I asked on April 19th. I managed to deploy a prefect server and to run flows with docker storage and a DaskKubernetes environment. The only hurdle I have left (I hope!) is that an important part of our Task depend on some common configuration set on
    prefect.context
    . This includes some secrets and some URLs and object ids on some internal rest services. Before we used the prefect server, we had a small cli that would populate the
    prefect.context
    and use it when running the flow:
    with prefect.context(**our_custom_vars):
        flow_state = flow.run(parameters=flow_parameters, ...)
    My ultimate objective is to have a flow with a default context that I can run from the UI or schedule it with that context when deploying it. In my question of April 19th, @Jeremiah pointed out that there is no way to set these contexts at the moment and you will discuss it internally later… is there any update on this front? Otherwise, I am looking for alternatives or workarounds: One workaround would be to understand where the context would be set on an agent, worker, or runner (I am not sure which one). Jeremiah also pointed our that the FlowRunner requests the context from the server, but I can’t find where or even if a flow has its context saved anywhere (there does not seem to be a field named context on the
    flow_by_pk
    query). Another workaround would be to populate environment variables or change the default
    config.toml
    of the agent, worker, or runner (I am not sure which one) so that the
    prefect.context
    is populated with these values. I am not sure if this would work. Another workaround may be to override the
    __setstate__
    and
    __getstate__
    method, so that the flow can retrieve the context when unpickled. I am not sure if this would work either. Any ideas on which of these workarounds may be the best bet here?
    d
    Jeremiah
    +1
    17 replies
    Copy to Clipboard
  • j

    Jacques

    2 years ago
    Hi everyone, quick question - it seems from docs (https://docs.prefect.io/core/concepts/mapping.html) like setting a trigger on a Task should still work if I use map on that Task. I'm trying to do that but doesn't seem to work. Do I need to do something special?
  • j

    Jacques

    2 years ago
    I want to do
    trigger=all_successful
    for the mapped function, but it only works for unmapped tasks
    j
    Kyle Moon-Wright
    13 replies
    Copy to Clipboard
  • a

    alvin goh

    2 years ago
    Hi I understand that depth first execution was being worked on in the past, is it released yet? If not, is there a workaround I can do to have a depth first execution??
    a
    1 replies
    Copy to Clipboard
  • a

    An Hoang

    2 years ago
    Hello, my work’s IT team is not keen on docker apparently so I can’t run prefect server on work’s machine. Here is what they said, quoted directly since I don’t know much about this stuff. Is there a way to deploy prefect without docker?
    X does want to push back because docker is a involved, and has many unsolved aspects since the daemons run as root. It didn’t look like the prefect ui could run in singularity which is the user-space container management thing that they were talking about.
    If it is possible to install the prefect ui as a non-docker, either as a library, or on a server. I think that has many fewer concerns from the systems team.
    a
    Jeremiah
    +1
    9 replies
    Copy to Clipboard
  • Matias Godoy

    Matias Godoy

    2 years ago
    Hi guys! I'd like to start a flow execution from another flow. Is it possible? I have flow A that is scheduled and runs every 5 minutes. When it finds something pending in our DB it should start flow B with some parameters. Can I do that?
    Matias Godoy
    Kyle Moon-Wright
    +3
    14 replies
    Copy to Clipboard