https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • k

    kevin

    09/10/2020, 7:36 PM
    Hey guys, when a task fails on prefect cloud, is there anywhere I can look at the full stack trace?
    d
    • 2
    • 6
  • j

    John Song

    09/10/2020, 8:07 PM
    Hi guys,
    d
    • 2
    • 1
  • j

    John Song

    09/10/2020, 8:08 PM
    I am new to Prefect but experienced with airflow, I wrote a simple script but found my flow run multiple times automatically, sometimes 3 other times 6. Can't find anything in the community, please help. from prefect import Parameter, Flow, task print("***************") @task def get_bbp_data():     return "bbp_data" @task def get_sem_data():     return "sem_data" @task def align_bbp_sem_data(bbp, sem):     return "align_data" @task def generate_train_data(align):     return "train_data" with Flow("I5 ETL") as flow:     bbp_data = get_bbp_data()     sem_data = get_sem_data()     align_data = align_bbp_sem_data(bbp_data, sem_data)     train_data = generate_train_data(align_data) flow.visualize() flow.run()
    d
    • 2
    • 21
  • z

    Zach

    09/10/2020, 9:01 PM
    I am getting an error when trying to use pipenv with prefect installed. This has worked for months and suddenly isn't anymore. I have attached the pipfile and here is the error output:
    Pipfile
    👀 1
  • z

    Zach

    09/10/2020, 9:02 PM
    Here is the error as a txt doc
    pipenv_error.txt
    c
    d
    • 3
    • 4
  • z

    Zach

    09/10/2020, 9:05 PM
    It looks like prefect needs
    cloudpickle<1.5
    , but prefect needs
    dstributed==2.25.0
    and
    distributed
    needs
    cloudpickle>=1.5.0
  • d

    Dylan

    09/10/2020, 9:07 PM
    Is there something blocking you from upgrading to Prefect Core’s latest version?
  • j

    Julian

    09/10/2020, 9:44 PM
    Hey, I'm running prefect version 0.13.5 and want to have a flow that triggers two other flows, which involves scaling a presto cluster. Here is an exemplary code snippet
    with Flow("scale_presto_cluster", result=get_default_local_result()) as flow:
        scale_down = Parameter("scale_down_capacity", required=False)
        scale_up = Parameter("scale_up_capacity", required=False)
        with PrestoCluster(n_worker_low=scale_down, n_worker_high=scale_up) as client:
            run_flow_task_stats_weekly = FlowRunTask(flow_name="dummy-weekly", project_name='default', wait=True)
            run_flow_task_stats_weekly(parameters=get_weekly_fromto())
    
            with case(check_if_monthly_run, True):
                run_flow_task_stats_monthly = FlowRunTask(flow_name="dummy-monthly", project_name='default', wait=True)
                run_flow_task_stats_monthly(parameters=get_monthly_fromto(), upstream_tasks=[run_flow_task_stats_weekly])
    I used a ResourceManager as documented in https://docs.prefect.io/api/latest/tasks/resources.html#functions I noticed that the cleanup task wasn't called after I cancelled
    scale_presto_cluster
    flow run during it's execution, but was cancelled instead as well. Since this flow will be run nightly, I mus't ensure that the
    cleanup
    task of the Flow will be triggered no matter what. How can I achieve this? Is a task even the right approach?
    d
    s
    n
    • 4
    • 12
  • r

    Rob Fowler

    09/10/2020, 10:05 PM
    it would be good to include a mention of num_workers in the LocalDaskExecutor page. Many might assume pool=ThreadPool(100) as a parameter might work without looking at the code.
    j
    • 2
    • 2
  • r

    Rob Fowler

    09/10/2020, 10:08 PM
    works awesome once you know that.
  • g

    Gitanshu Sardana

    09/11/2020, 1:20 AM
    Hi. I am a bit confused about the graphql endpoint config. The prefect ui says "you can modify this and other settings from your Server's ~/.prefect/config.toml or in the input box below". The input box works but setting it in Server's config.toml doesn't seem to work for me
    backend = "server"
    
    [server]
    host = "<http://172.18.1.3>"
    
        [server.ui]
        graphql_url = "<http://172.18.1.3:4200/graphql>"
    
    [logging]
    # The logging level: NOTSET, DEBUG, INFO, WARNING, ERROR, or CRITICAL
    level = "WARNING"
    this is what my config.toml looks like
    c
    n
    • 3
    • 4
  • g

    Gitanshu Sardana

    09/11/2020, 2:33 AM
    Another question: How do I reduce the verbosity of server logs? Setting logging level doesn't work for that. I guess that's for the task loggers. I see a constant barrage of request logs even when nothing's running and the docker container keep growing
    c
    • 2
    • 3
  • v

    Vipul

    09/11/2020, 7:22 AM
    I am ready for contest @Marvin 🙂
  • m

    Marvin

    09/11/2020, 7:22 AM
    I’d give you advice, but you wouldn’t listen. No one ever does. Good luck in the contest @Vipul.
  • l

    Leonard Marcq

    09/11/2020, 7:57 AM
    Hi there - I am a bit lost when it comes to deploying the server ( 😞 ). For those of you deploying Prefect server with a dedicated Postgres database sitting on an external host, how do you do it? Do you just pull out the
    docker-compose.yml
    from
    prefect/cli/
    , modify it to point to your db and
    up
    , or do you pull the whole repo (prefect or prefect_server?), or install prefect via
    pip
    and pass it a custom
    .toml
    config somehow and spin up your server with `prefect server start`; or do you do something else entirely?
    n
    • 2
    • 2
  • k

    karteekaddanki

    09/11/2020, 8:28 AM
    Hello everyone! Is there any way to set a priority for upcoming flows so that they take higher priority? More concretely, we have some scheduled jobs that need to run daily and a limited number of flow slots. Normally, this isn't a problem but sometimes when we are backfilling we have a lot of flows scheduled which end up in front of these scheduled daily jobs delaying the results. Is there any way to specify flow priority or have scheduled flows run at a higher priority than one in the queue?
    n
    • 2
    • 2
  • s

    Sven Teresniak

    09/11/2020, 8:49 AM
    Hi there, we need some synchronization between flow runs. It is possible to use
    tag
    s for tasks,
    label
    s for flows, but not possible to use tags or labels for flow-runs, right? Now, the idea is to write a ressourcemanager as kind of Lock (reentrant or not, semaphore or whatever flavor). Then you could (hopefully) enter a RessourceManager
    with
    -block the same way you would use a
    Lock
    in Python code. Something like 1.
    setup()
    searches (like in "asking Prefect") for all flows-runs with a given tag (that is, a run with a flow that has this certain tag set) 2. if
    setup()
    does NOT find another flow-run with that tag, the tag is set (i don't know yet where and how) 3. if
    setup()
    DOES find find another matching flow-run (or RM): then do some polling or return with successful error (configuration) 4. then all the tasks in the RM's
    with
    environment run (this is the part that should be synchronized/locked) 5. on exit the RMs
    cleanup()
    is called to remove the tag/label/whatever from the current flow (to allow other flows to set the tag) Do you think this is a good idea? Which/where is the best way to store that tag? Using the Context, Parameters, Tags, Labels? If you think this could be a nice feature we could create a prototype and pull request. We need to prevent a race condition here and for that we need to make the execution of several task isolated (the I in ACID in RDBS)
    m
    n
    • 3
    • 36
  • j

    Jordan

    09/11/2020, 9:49 AM
    @Marvin ❤️
  • m

    Marvin

    09/11/2020, 9:49 AM
    I think you ought to know I’m feeling very depressed. Also, you're in the contest, @Jordan.
    p
    r
    • 3
    • 4
  • j

    Juan

    09/11/2020, 1:50 PM
    Hi, is it possible to register a flow with a project, send it to a Dask cluster and keep track of it in Prefect GUI ?
    n
    • 2
    • 6
  • k

    Kyle Pierce

    09/11/2020, 2:08 PM
    @Marvin 😄
  • m

    Marvin

    09/11/2020, 2:09 PM
    I think you ought to know I’m feeling very depressed. Also, you're in the contest, @Kyle Pierce.
  • n

    Nate Joselson

    09/11/2020, 2:24 PM
    Tried to resist the urge, but… @Marvin 😅
    j
    • 2
    • 1
  • m

    Marvin

    09/11/2020, 2:24 PM
    Here I am, brain the size of a planet, and they ask me to enter you in a contest. Call that job satisfaction? ’Cos I don’t. I'll still enter you in the contest though, @Nate Joselson
  • j

    Joe Lyman

    09/11/2020, 4:15 PM
    What is the correct way to actually fire off a flow run from a separate script? I have defined and registered a flow in `a.py`:
    with Flow("myflow") as myflow:
        # tasks
    myflow.register(project_name="myproj")
    Now, separately in
    b.py
    I run a file watcher that needs to trigger "myflow" (in a non-blocking fashion) when a file arrives. I stumbled across:
    FlowRunTask(project_name="myproj", flow_name="myflow")
    but I don't know if this is correct. I considered
    from a import myflow; myflow.run()
    in
    b.py
    also - although the flows seemed to run, they didn't appear in the UI.
    n
    j
    • 3
    • 6
  • a

    ARun

    09/11/2020, 4:17 PM
    Hey guys, I'm trying to map an array of dictionary items into a function. My tasks keep failing from random errors, but it looks like the task isn't actually taking in the dictionary, instead it's taking in the first letter of every object. for example, I have an array in the format [{},{},{},{}...] the dict contains about a dozen attributes I am trying to map the array to the function like function.map(array) but i keep getting random errors like file not found(one of the attributes). Instead of using the actual fully qualified file name, what's being passed in is just a letter
    n
    • 2
    • 8
  • s

    Skip Breidbach

    09/11/2020, 6:12 PM
    Swag me, @Marvin, even though I know that all the diodes down my left side will still hurt.
    😆 3
    :marvin: 2
  • m

    Marvin

    09/11/2020, 6:12 PM
    It gives me a headache just trying to think down to your level. I'll still enter you in the contest, @Skip Breidbach.
  • n

    Nick McCullum

    09/11/2020, 7:06 PM
    Yo @Marvin - thnks for having me in the community!
    🇨🇦 1
  • m

    Marvin

    09/11/2020, 7:06 PM
    I would like to say that it is a very great pleasure, honour and privilege for me to enter @Nick McCullum in this contest, but I can’t because my lying circuits are all out of commission.
Powered by Linen
Title
m

Marvin

09/11/2020, 7:06 PM
I would like to say that it is a very great pleasure, honour and privilege for me to enter @Nick McCullum in this contest, but I can’t because my lying circuits are all out of commission.
View count: 1