• j

    Jack Sundberg

    1 year ago
    Hey everyone, is there a good way to access flow-run metadata within a flow context? For example, a task that grabs the current flow-run-id or even an upstream task's duration. Or will pulling metadata need to be done in a separate flow?
    j
    Kevin Kho
    5 replies
    Copy to Clipboard
  • Jonathan Chu

    Jonathan Chu

    1 year ago
    hi guys, we had a docker agent fail over the weekend, so there were like 100 late scheduled runs as soon as i restarted the agent, it immediately queued and started all 100 runs these are pretty expensive tasks, so firing them all up at once immediately ground the server to a halt, couldn't ssh in to try killing things so my questions coming out of this are: • How do you configure the scheduled services to just give up if they get missed? We run the task every hour, so it's fine to just run it the next hour instead. Having it give up after, say, 30 minutes of waiting, would be very elegant, but just failing immediately is perfectly fine • How do you mass cancel scheduled jobs that have missed their time? • How do you mass cancel running jobs? I manually clicked open every job and pressed cancel until the system starting responding enough to ssh and I could go in and start killing things via the docker cli. With how unresponsive the UI is in general, this was pretty painful. • Is there some way to kill the agent to stop everything? This is the idea of some kind of breaker switch -- prefect just started going crazy on this server, shut it all down so we're back to a working state, and then we can sort it out afterwards. • After all the jobs had been marked as Cancelled or Failed, there were still containers running on the server. Are these just all the Failed containers that lost their heartbeat? Is there a nicer way to clean them up, or are Failed containers just abandoned by Prefect at that point, and the answer is to just go in and manually remove them like I did? Does that mean that any Failed tasks should always be manually reviewed for any hanging containers that need to be stopped and removed? • How do you configure a limit for the agent, so it can't get bad enough to make the server inaccessible to repair? I know based on tags that this agent is only going to run jobs of a certain type, so I could set the limit to, say, 5, and it'd just have 5 running 95 queued, and then I'd just need to mass-cancel those queued jobs.
    Jonathan Chu
    nicholas
    +1
    17 replies
    Copy to Clipboard
  • x

    xyzz

    1 year ago
    Does the cost calculator at https://www.prefect.io/pricing/ work for anyone? I can't enter numbers higher than 9999
    x
    Kevin Kho
    +1
    9 replies
    Copy to Clipboard
  • x

    xyzz

    1 year ago
    Also it would be great if the blue keywords would show detailed description of the feature on mouse over or link to a page with it
  • Rob Fowler

    Rob Fowler

    1 year ago
    I just ran one of my new flows in production, scripts run on 3500 machines
  • Dotan Asselmann

    Dotan Asselmann

    1 year ago
    hey everyone, is it possible to capture default ‘logging’ logs on prefect run? we’re using many libraries that print valuable logs but don’t have the context of the prefect logger. we want instead of pushing forward the logger object to be able to use the global python logging object
    Dotan Asselmann
    1 replies
    Copy to Clipboard
  • Pedro Machado

    Pedro Machado

    1 year ago
    Hi everyone. I am working with a data science team that wants to orchestrate some workflows. They already have scripts that run inside of a docker container. I was thinking that the easiest way to add these to the flow would be to use the Docker Tasks (
    CreateContainer
    ,
    StartContainer
    , etc.). However, it seems that the logs would need to be retrieved after the container finishes running with
    GetContainerLogs
    . Is this correct? If so, this is less than ideal for our use case given that these are long-running processes we'd want to see the logs in real-time. So far, I've thought about a couple of alternatives: 1. Modify the Docker Tasks to use the
    logs
    method with
    stream=True
    (I haven't tested this yet, but the docs suggest this could work) 2. Add prefect to their docker image and create a flow that runs inside of the image Do you see another option? What would you recommend? Thanks!
    Pedro Machado
    nicholas
    3 replies
    Copy to Clipboard
  • k

    Kelby

    1 year ago
    Is it possible to change the start time of a scheduled workflow?
    k
    nicholas
    4 replies
    Copy to Clipboard
  • Kayvan Shah

    Kayvan Shah

    1 year ago
    Is there any way to define which task executes first and second task has to be executed only after first task is executed Both of these tasks loads same data from a loader function
    Kayvan Shah
    nicholas
    +1
    8 replies
    Copy to Clipboard
  • m

    Mitchell Bregman

    1 year ago
    Hello, I am using KubernetesRun for the flow run configs.. I am trying to specify a
    service_account_name
    , but it seems the pods are still using the default service account to the namespace.. my agent is also deployed with an undefined
    service_account_name
    .. any thoughts?
    flow.run_configs = KubernetesRun(
        image_pull_secrets=["xxx"],
        job_template=config.KUBE_JOB_TEMPLATE,
        cpu_limit=config.KUBE_CPU_LIMIT,
        cpu_request=config.KUBE_CPU_REQUEST,
        memory_limit=config.KUBE_MEMORY_LIMIT,
        memory_request=config.KUBE_MEMORY_REQUEST,
        service_account_name="my-service-account",
    )
    m
    Kevin Kho
    +1
    8 replies
    Copy to Clipboard