https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • l

    Leo Meyerovich (Graphistry)

    04/11/2020, 4:56 AM
    We are close getting our initial orchestration pipeline ported 🙂 we're a bit confused on how to get long jobs running. tips appreciated. setup: -- server 'ui': running the ui container -- server 'gpu': running a prefect agent as well. registers with ui so it can pick up gpu jobs. -- server 'nb': jupyter notebooks we're using to submit jobs. has a local prefect agent installed that points to 'ui' so we can submit jobs. notebooks often die we can do quick one-offs fine. hurray! tricky case 1: long historic job we want to do a ~3 day job that processes 200 files, one at a time sequentially in sorted order. the problem is notebook server that runs the job will periodically stop, so we really want to submit a job like
    seq([ task_1(file_1), task_2(file_2), ... task_n(file_n)])
    . as soon as the meta-task is submitted, the notebook (and its local agent) can stop. however, for the next 3 days, we want those tasks to run one at a time, and we see status in the ui (incl. fails/retries). if we ever want to, we can rerun the flow to add/swap tasks.
    b
    • 2
    • 2
  • l

    Leo Meyerovich (Graphistry)

    04/11/2020, 4:56 AM
    is there anything like
    seq
    we should look for?
  • l

    Leo Meyerovich (Graphistry)

    04/11/2020, 4:56 AM
    or we should think about it differently?
  • l

    Leo Meyerovich (Graphistry)

    04/11/2020, 5:02 AM
    tricky case 2: streaming job similar, except now we want the notebook to kick off a scheduled flow. we see the
    schedule
    primitive, so looks like we can schedule something every say 30s - 2min. except if the jupuyter notebook process stops... will the schedule still be running?
  • l

    Leo Meyerovich (Graphistry)

    04/11/2020, 5:36 AM
    Digging through more: it sounds like we need to create a Storage config for the central scheduler, such as Azure blob store, and figure out how to use the task dependency stuff, and then all of the above should Just Work (tm). once the flow gets registered, the agent who kicked it off can terminate.
  • o

    Omar Sultan

    04/11/2020, 5:14 PM
    Hello Everyone, wanted to see if i can get some help 🙂 so we're evaluating prefect to replace our current implementation that's build on another platform and one of our use cases is to start a flow from inside a flow. I've been through the documentation and can't seem to find something that covers this part. Any help in pointing me in the right direction would be great thanks
    c
    • 2
    • 3
  • z

    Ziyao Wei

    04/11/2020, 5:47 PM
    I’m trying to get
    prefect agent start
    to connect to Azure - I am passing in a
    AZURE_STORAGE_CONNECTION_STRING
    but it complains:
    Unexpected error: HTTPError('400 Client Error: Bad Request for url: http://<URL>:4200/graphql/alpha')
    Traceback (most recent call last):
      File "/home/<username>/.local/lib/python3.7/site-packages/prefect/client/secrets.py", line 121, in get
        value = secrets[self.name]
    KeyError: 'AZ_CREDENTIALS'
    j
    • 2
    • 14
  • z

    Ziyao Wei

    04/11/2020, 5:47 PM
    Is there something else I need to pass in?
  • z

    Ziyao Wei

    04/11/2020, 5:48 PM
    Or I guess - could Prefect get the creds from the conn string itself?
  • z

    Ziyao Wei

    04/11/2020, 5:49 PM
    Also - bug report - https://docs.prefect.io/orchestration/execution/storage_options.html#local the code samples on this page largely don’t work and Python says
    storage
    isn’t defined
    c
    • 2
    • 4
  • z

    Ziyao Wei

    04/11/2020, 7:17 PM
    Another issue trying to get Prefect to work with Azure - it seems Prefect is using both
    BlockBlobService
    and
    BlobServiceClient
    , but the former is for <=v2.1.0 and the latter >=v12.0.0 AFAIK - should the former be migrated to the latter?
    j
    • 2
    • 1
  • z

    Ziyao Wei

    04/11/2020, 7:18 PM
    I cannot seem to get the Azure agent to work with either version FWIW
    j
    • 2
    • 9
  • l

    Leo Meyerovich (Graphistry)

    04/12/2020, 9:10 PM
    Is there a way to see what agents are (successfully) registered w/ the UI server for receiving tasks to execute?
    j
    • 2
    • 1
  • l

    Leo Meyerovich (Graphistry)

    04/12/2020, 9:11 PM
    and otherwise diagnose an agent's configuration for recieving tasks? we're using a UI server + AWS storage
  • b

    Brad

    04/13/2020, 7:39 AM
    Has anyone had any issues running a docker agent on a linux server?
    j
    l
    • 3
    • 7
  • a

    alexandre

    04/13/2020, 10:55 AM
    Hello guys, I was wondering if it is possible to move the log level in a subflow (to have it on INFO on my main flow and have it on ERROR on my subflow). Do you know any methods to do this ? 🙂
    c
    • 2
    • 2
  • a

    Adam Roderick

    04/13/2020, 1:13 PM
    Getting close to 2,000 stars. Come on, quit slacking and get there already! https://github.com/prefecthq/prefect
    😂 5
    🤩 8
  • a

    Adam Roderick

    04/13/2020, 2:11 PM
    Hey prefect, I am trying to run
    prefect server start
    for the first time and running into some errors with the postgres step. Can you help troubleshoot?
    z
    j
    • 3
    • 36
  • n

    Nathan Molby

    04/13/2020, 2:53 PM
    Hey all. I am attempting to use a DaskExecutor to run a flow that includes a mapped tasked. When the mapped task gets executed, I keep getting an error that "There is no current event loop in thread "Dask-Worker-Threads". Any suggestions?
    j
    • 2
    • 1
  • a

    Adam Roderick

    04/13/2020, 3:31 PM
    Is there a feature comparison between what is available in the OS UI vs Prefect Cloud?
    j
    • 2
    • 9
  • m

    Manuel AristarĂĄn

    04/13/2020, 3:41 PM
    Hey, has anyone tried to run the server in a Fargate container?
    a
    p
    d
    • 4
    • 12
  • b

    Ben Fogelson

    04/13/2020, 7:48 PM
    If I run a flow with a non-parallel executor is the task execution order guaranteed to be the same from run to run?
    n
    • 2
    • 1
  • a

    Adam Roderick

    04/14/2020, 5:39 AM
    I'm running into an issue with
    flow.register()
    and not sure how to troubleshoot. Can anyone help?
    n
    • 2
    • 16
  • n

    Niclas Roos

    04/14/2020, 11:14 AM
    Hi all! I’m evaluating Prefect for a new job that I’am starting shortly. If I want to use Prefect Cloud to trigger Lambda functions in AWS, do I need a separate Python environment to act as the agent?
    j
    • 2
    • 1
  • a

    Adam Roderick

    04/14/2020, 12:27 PM
    Does anyone know why an Fargate agent would immediately fail when a flow run is triggered, with the message "An error occurred (InvalidParameterException) when calling the RegisterTaskDefinition operation: Invalid 'cpu' setting for task."
    j
    • 2
    • 19
  • a

    Adam Roderick

    04/14/2020, 1:38 PM
    I'm working on using the Dockerfile method to create Docker storage using prefect 10.1. I believe I am running into an issue due to python pathing on windows.
    z
    • 2
    • 6
  • j

    Jeff Brainerd

    04/14/2020, 2:50 PM
    Hello Prefect team — I’m looking for documentation that details the permissions that each type of cloud user has (admin, user, restricted), as well as permissions for API tokens. For example, what types of users can start or cancel a flow, both via the UI and API? Thanks! 🙏
    k
    • 2
    • 2
  • e

    emre

    04/14/2020, 3:27 PM
    Hello everyone, I am thinking about adding a small feature to
    ShellTask
    , but in my case, one of the tests (test_shell_logs_error_on_non_zero_exit) fails without me touching anything. It is the only test in
    test_<http://shell.py|shell.py>
    that uses
    caplog
    , and
    caplog
    seems to be capturing 0 log lines. Does anybody have an idea? I have tried
    caplog.set_level(logging.DEBUG)
    , to no avail.
    🧐 1
    l
    • 2
    • 5
  • v

    Viv Ian

    04/14/2020, 4:18 PM
    Couple questions: • Is a flow being evaluated even if it’s not running? Airflow automatically evaluates a DAG every ~5 minutes, and I’m wondering if Prefect does something similar (I prefer it doesn’t) • I may be looking at this incorrectly, but it seems like when using
    map
    , each complete
    map
    task must complete before the next task starts. Using the example provided on the Prefect website (https://docs.prefect.io/core/examples/twitter_to_airtable.html), replies are received, all replies must be transformed before being loaded into the table. Is it possible to not wait for all of the replies to be transformed? So after replies are extracted…each one just goes through the transform and load steps without having to wait for the complete for other replies? THANKS!
    c
    j
    m
    • 4
    • 5
  • a

    An Hoang

    04/14/2020, 6:43 PM
    Hi, when I run the following flow using the local executor all tasks succeed, but when I run it with the dask executor, there are import errors. Is it because the workers don’t have the same environment as the scheduler? How can I fix it?
    c
    j
    m
    • 4
    • 3
Powered by Linen
Title
a

An Hoang

04/14/2020, 6:43 PM
Hi, when I run the following flow using the local executor all tasks succeed, but when I run it with the dask executor, there are import errors. Is it because the workers don’t have the same environment as the scheduler? How can I fix it?
c

Chris White

04/14/2020, 7:31 PM
Hi @An Hoang - you are correct, your workers need to have all the packages your flow / tasks rely on. To fix it you should run your workers within a python environment matching the one you use to build your flow
j

Jeremiah

04/14/2020, 10:13 PM
@Marvin archive “My flow runs with the local executor but has an import error in Dask”
m

Marvin

04/14/2020, 10:13 PM
https://github.com/PrefectHQ/prefect/issues/2335
View count: 1