• t

    trapped

    2 years ago
    hi guys, any way to speed up (i.e. avoid serializing the whole result and sending it back to Prefect) task mapping? I've got a ~1M item list/tuple computed with Dask Bags which I'd like to map over, and between completing computation of the bag itself and the first mapped task I'm observing a good deal of time spent waiting for the complete result list
    t
    Zachary Hughes
    +1
    10 replies
    Copy to Clipboard
  • r

    RyanB

    2 years ago
    I am getting Event loop was unresponsive in Scheduler for 18.15s. followed by distributed.scheduler - ERROR - Couldn’t gather keys for a large data set this causes a loop in the prefect flow as it reschedules the tasks it couldn’t gather and eventually hits the same error. Any ideas on what to look for of any configuration options? I am using dask-kubernetes 0.10.1 and dask 2.11.0 in AWS.
  • l

    Luke Orland

    2 years ago
    I have some questions about
    FargateTaskEnvironment
    . 1. Can it just run an existing ECS task definition, or does it always register a new task definition based on the specified family/taskDefinition? 2. If it can just run an existing ECS task definition, what would the minimal necessary arguments be, something like
    FargateTaskEnvironment(taskDefinition='my-ecr-task-definition')
    , assuming AWS credentials are accessible by boto3 via the environment? 3. If it always generates/registers a new task definition, what would the minimal set of required arguments? I could probably go through the process of figuring that out by running
    boto3.client('ecs').run_task(*args)
    until I get the desired task definition running and then translate that into
    FargateTaskEnvironment
    arguments. Curious if someone has gone through this process already.
    l
    Braun Reyes
    +1
    12 replies
    Copy to Clipboard
  • Pete Fein

    Pete Fein

    2 years ago
    How do you unit test flows? I didn't see anything about it in the documentation...
    Pete Fein
    Jeff Brainerd
    +1
    3 replies
    Copy to Clipboard
  • n

    Nate Joselson

    2 years ago
    Hi! I have some flows that run every hour according to a cron job. My question is whether it is possible to have one of the flows repeat again after 30 minutes (so essentially running that particular flow twice in the hour). I understand that I could use the scheduler in prefect, or just create a separate cron job that runs half-hourly, but I was wondering if there was a third option as well in prefect. Thanks!
    n
    j
    4 replies
    Copy to Clipboard
  • i

    itay livni

    2 years ago
    Hi - I have
    task
    in a
    Flow
    that takes a
    list
    . How do I set rhe
    List
    Trigger
    to run on
    any_successful
    upstream tasks? https://docs.prefect.io/core/concepts/tasks.html#collections I tried a couple approaches including def_list = List(trigger=any_successful) and then calling
    munged_defs_df = definitions.munge_dfs(def_list([df1, df2, df3])
    i
    Alex Goodman
    6 replies
    Copy to Clipboard
  • j

    Justin Shimkovitz

    2 years ago
    Hi, I am new to prefect and just finished reading through the documentation. Question about
    map
    responses. When I send a list of elements to a task and one element sent to the task finishes and returns a response, is it possible to move that response to the next task before the rest of the original elements finish in the first task? Thanks in advance!
    j
    Chris White
    2 replies
    Copy to Clipboard
  • Jeff Brainerd

    Jeff Brainerd

    2 years ago
    Random PSA: our team ran into issues installing
    psutil
    (a dependency of
    distributed
    ) on Catalina (OSX 10.15). This thread helped us get past it: https://github.com/giampaolo/psutil/issues/1632
  • Jeremiah

    Jeremiah

    2 years ago
    has renamed the channel from "prefect-core" to "prefect-community"
  • l

    Luke Orland

    2 years ago
    I'm a little confused about the
    upstream_tasks
    arg on
    Task.map
    . "a list of upstream dependencies to map over". What would mapping over the upstream dependencies do?
    l
    nicholas
    4 replies
    Copy to Clipboard