https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • e

    Eric

    09/25/2020, 11:53 PM
    I'm trying to figure out how to run some Tasks in parallel, like something like this in the second stage. There is no data passed between the stages, the tasks in the previous stage just have to complete with Success I know how to run them sequentailly like task_2(task_1()) but unclear if I should be using .map?
    c
    13 replies · 2 participants
  • n

    Newskooler

    09/26/2020, 10:33 AM
    Hi everyone 👋, Is it possible to create a task which is triggered based on some external event (e.g. email received, or website updated specific data point, etc)? I wonder how would one go about creating such a flow?:) 🤔
    👍 1
    👋 1
    c
    v
    +1
    6 replies · 4 participants
  • v

    Vitaly Shulgin

    09/28/2020, 8:39 AM
    Hello team, previously I was asking to clarify why it is impossible to register new flow from parent flow, here https://prefect-community.slack.com/archives/CL09KU1K7/p1600240447350500?thread_ts=1600209791.334200&cid=CL09KU1K7 Could you bring some light on that case? Like cold system start, where the one parent flow creates many new flows, initially stored in git or database. What will be the way to accomplish it with current prefect funcionality?
    c
    1 reply · 2 participants
  • a

    Adrien Boutreau

    09/28/2020, 8:52 AM
    Hello! Thanks again for your product! It's amazing! We did installation of Prect Core in AWS and we want to do : 1. SSL certificated added on prefect 2. Have a login page to the prefect UI Do you know where I can documentation about theses 2 points ? Thanks!
    👀 2
    p
    d
    +1
    7 replies · 4 participants
  • j

    Jovan Visnjic

    09/28/2020, 10:12 AM
    Hi everyone. I have a question about
    LocalResult
    and exceptions. We use
    LocalResult
    for checkpointing, so only failed tasks get executed on a workflow re-run. So far we execute the workflow without a server, using just the basic CLI. If
    LocalResult
    exists, the task is skipped and just tasks with missing results get executed. We would also like to store error information on the disk. But if write exception's content as a
    LocalResult
    through task's
    target
    parameter, it gets treated as successful, because something exists at the result location, whatever that may be. Is there some established way of dumping errors to disk, so they don't just remain in the console's
    stdout
    , but that at the same time they don't obstruct the logic of what is successful and what not? Any ideas or comments appreciated. Jovan
    k
    2 replies · 2 participants
  • j

    Jonas Bernhard

    09/28/2020, 2:36 PM
    Hi team, thanks a lot for Prefect, having a lot of fun working with it so far! I have two questions 1. When mapping over results of a previous task, is it possible to only continue with the ones that were successful? In my use case, I first fetch feeds for a list of urls with
    fetch_feed_for_url
    and then save them with
    save_feed_entries
    (with trigger
    any_successful
    ). However, some of the urls might not work leading to all subsequent tasks mapping over the result (and with that the flow) also failing. What I'm looking for is to map only over "successful
    feeds_list
    entries"
    feeds_list = fetch_feed_for_url.map(url=feed_urls)
        save_feed_entries = save_feed_entries_to_db.map(feed=feeds_list)
    2. Is it possible to initialize a task from a Task class using
    EnvVarSecret
    ? For example, creating a Task using the Task Library often requires the credentials upfront. Are all the secrets required to be available at "flow building time" to work with Task classes?
    fetch_mysql = MySQLFetch(
      user=EnvVarSecret("USER)
      password=EnvVarSecret("PASSWORD)
    )
    Thanks!
    k
    5 replies · 2 participants
  • e

    Eric

    09/28/2020, 7:20 PM
    I'm trying to use the files{} parameter with the Docker storage class, it says it can take files or directories, however it only deploys if i have a file specified, not a directory. When pointed at the directory I get a "permission denied" error, https://stackoverflow.com/questions/11278066/using-shutil-copyfile-i-get-a-python-ioerror-errno-13-permission-denied/11278116 • `files (dict, optional)`: a dictionary of files or directories to copy into the image when building. Takes the format of 
    {'src': 'dest'}
    c
    r
    +1
    8 replies · 4 participants
  • v

    Von Christian Pizarro

    09/28/2020, 9:16 PM
    Hi Everyone, Has anyone experienced their flow running an hour later after DST? Just wanted to know if there's a way to disable it or would I need to update our flows manually? Thanks!
    k
    3 replies · 2 participants
  • e

    EdCode

    09/28/2020, 10:53 PM
    Hi, I have prefect on a RHEL box with a windows drive mounted via smb. I would like airflow to trigger some python code using the WINDOWS process, not the linux process. Is this possible?
    k
    1 reply · 2 participants
  • r

    Rob Fowler

    09/29/2020, 7:03 AM
    can you mark a task as always_run in a flow? For example I get some results but sometimes I want the later tasks to run, sometimes not, so I don't want to decorate the actual task as allways_run
  • r

    Robin

    09/29/2020, 9:36 AM
    Hi, I have prefect 0.13.8 installed and running on Win 10, Python 3.7.9. At the very end of trying to registering the flow (
    INFO - prefect.Docker | Pushing image to the registry...
    ) I get a
    shutil PermissionError: [WinError 5] Access is denied
    . Has anybody had the same issues? We are using Python 3.7.9 due to some issues with Python 3.8 of another third party package ...
    ✔️ 1
    n
    32 replies · 2 participants
  • r

    Ralph Willgoss

    09/29/2020, 9:40 AM
    Hi, We've setup a workflow with the goal of parallelizing some behaviour. I've attached the DAG from the
    visualize
    command. The issue is that individual
    individual_preprocessing
    task should be done in parallel as well, however its not. What have we have done wrong here in our task setup?
  • r

    Ralph Willgoss

    09/29/2020, 10:14 AM
    Attached is the code for the above
    demo.py
  • r

    Rob Fowler

    09/29/2020, 11:52 AM
    Still stuck on this. I have a task that is the source of some data but it may fail to gather some of it but I want the rest of the flow to run.
  • r

    Rob Fowler

    09/29/2020, 11:58 AM
    another way too look at it is I have a reference task that I want to feed a pipeline, even it is has some failures
  • r

    Rob Fowler

    09/29/2020, 12:24 PM
    worked out a way around it, I made my initial discovery task catch all errors and simply made a return item that indicated the exception, then, in the following filter I removed those items.
    n
    3 replies · 2 participants
  • n

    Nuno Silva

    09/29/2020, 12:29 PM
    Hi, quick question: I deployed prefect server in a VM in azure and everything works fine except I cannot change the default
    apollo_url
    like described here: https://docs.prefect.io/orchestration/server/deploy-local.html#ui-configuration I do this in `~/.prefect/config.toml`:
    [server]
      [server.ui]
      apollo_url="http://<server_url>:4200/graphql"
    then:
    prefect server start
    and it still uses the default
    localhost
    hence I have to manually change it in the server UI. Any idea why?
    n
    17 replies · 2 participants
  • l

    Lars Corneliussen

    09/29/2020, 12:46 PM
    Just encountered a late run for the first time. I did everything like yesterday, but somehow my agent isn't picking up the task... Any ideas?
    j
    3 replies · 2 participants
  • m

    Mikael

    09/29/2020, 2:13 PM
    Any news about how to run prefect with a predefined postgres database? I have a client running on windows and it works pretty well except that all flow and logs disappear on restart due to the mount volume parameter doesn’t work on windows. This is a general docker on windows bug but if you can specify your own postgres db prefect would run fine.
    j
    c
    14 replies · 3 participants
  • j

    Johnny

    09/29/2020, 8:34 PM
    Hello! Having an issue with Kubernetes cluster autoscaler for long running (> 21 min) flows similar to issue 3058. I noticed the issue has been marked "closed". What was the solution?
    👀 1
    j
    5 replies · 2 participants
  • a

    Andy Dyer

    09/29/2020, 9:40 PM
    hey all. I am trying to register a flow but its having difficulty pickling a dependency of
    paramiko
    which I am using for sftp. See below stack trace. it makes sense and other people are having difficulties pickling the same object in different applications. Is the best bet to use ShellTasks to sftp?
    <https://github.com/paramiko/paramiko/issues/753>
    <https://stackoverflow.com/questions/39321606/cant-pickle-an-rsa-key-to-send-over-a-socket>
    stack trace : https://pastebin.com/8hyAdpx1
    👍 1
    j
    19 replies · 2 participants
  • i

    itay livni

    09/29/2020, 9:59 PM
    Hi - I am running into an error where a merge statement gets triggered in a case branch unnecessarily causing the flow to fail. Is this expected behavior? https://github.com/PrefectHQ/prefect/issues/3399
    c
    j
    3 replies · 3 participants
  • m

    Matt

    09/29/2020, 10:36 PM
    Hi! I'm new to Prefect, working on my first flow, and struggling a bit with config and context in Prefect Core. I'm calling this from the command line and have the parameters going in fine and am grabbing secrets from env settings. I also understand that Prefect returns these as Tasks and not strings. Are both of these (Parameters and EnvVarSecrets) only available to the Flow? I don't seem to be able to access them from the Tasks themselves and it doesn't seem like they get automatically added to prefect.context or prefect.config. Is the only way to do this to pass them to the Task (socrata.download_dataset in this case) as parameters? I've tried adding the Secrets to both context and config programmatically but they don't seem to appear when I do this although the docs suggest this should work. My first attempt at a flow had parameters getting passed every which way and I'm trying to clean this up. Would love some guidance from the community. Thanks!
    Untitled
    n
    c
    10 replies · 3 participants
  • j

    JC Garcia

    09/29/2020, 10:40 PM
    Hey guys, got a noob question here. When running:
    prefect create project "Test Project X"
    I assume that under the hood prefect is hitting localhost:4200, is there any way to change that url?
    c
    2 replies · 2 participants
  • j

    John Song

    09/29/2020, 11:49 PM
    A dumb question, how can I redirect or config prefect logging to a local file? I read to doc but didn't find any example to do it
    c
    7 replies · 2 participants
  • m

    ms16

    09/30/2020, 3:05 AM
    Greetings ! Prefect Community Im using the Prefect for the first time and trying to figure out how to get the flow to execute on the Prefect Cloud . Here is what I have done. 1. Create RUNNER Token as an Admin user 2. Build a docker container to install the prefect pip pkg and run prefect agent start at the ENTRYPOINT
    ENTRYPOINT prefect agent start --token  $PREFECT_RUNNER_TOKEN -l fargate-dev
    3. When I run the docker container locally - The Precct agent starts inside the container and can see the output
    [2020-09-30 02:53:34,447] INFO - agent | Waiting for flow runs...
    4. But the flows registered in the project never start to execute I have RTFM but still missing a link here 😕 Here is the code of my registered flow
    import prefect
    from prefect import task, Flow
    @task
    def hello_task():
        print("hello")
    flow = Flow("hello-flow", tasks=[hello_task])
    flow.register(project_name="Demo",labels = "fargate-dev")
    j
    4 replies · 2 participants
  • m

    ms16

    09/30/2020, 3:05 AM
    tldr; registered the agent , cant run flow
  • p

    Prathamesh

    09/30/2020, 8:46 AM
    Hi all, I am new to Prefect. I have a simple Hello World python script that I'm orchestrating via Prefect. In my local Prefect Core server it takes 1 sec to run from submission to execution. But from Prefect Cloud, it takes about 6-9 minutes to complete. Any help/pointers will be helpful, thanks in advance.
    n
    9 replies · 2 participants
  • r

    Robin

    09/30/2020, 9:10 AM
    Hey all, is there a way to set the
    min_workers
    and
    max_workers
    as flow parameters, such that they can be changed from the cloud ui?
    flow.environment = DaskKubernetesEnvironment(
            min_workers=20, max_workers=30, labels=["k8s"]
        )
    ✔️ 1
    j
    5 replies · 2 participants
  • n

    Nuno Silva

    09/30/2020, 12:08 PM
    Hello. When using mapped tasks, is it possible provide a custom name to each child so that in prefect server instead of showing e.g.
    table_copy 0....table_copy n
    would show
    table_copy table_name_1....table_copy table_name_n
    ? Thank you
    n
    j
    +1
    8 replies · 4 participants
Powered by Linen
Title
n

Nuno Silva

09/30/2020, 12:08 PM
Hello. When using mapped tasks, is it possible provide a custom name to each child so that in prefect server instead of showing e.g.
table_copy 0....table_copy n
would show
table_copy table_name_1....table_copy table_name_n
? Thank you
n

nicholas

09/30/2020, 12:58 PM
Hi @Nuno Silva ! This is a really good question, and something we'll support in the very near future. @josh has a draft PR open for this but ran into some issues; that said, we recently released a new API for setting task run names (
set_task_run_name
mutation) which enables this in a manual way. The UI doesn't pull the task run name yet but will very soon!
j

josh

09/30/2020, 1:08 PM
Yeah we now have a path forward with this using the run name API! Haven’t gotten around to implementing it yet but it’s on the list 🙂
🙌 1
p

Pedro Machado

09/30/2020, 1:15 PM
What's the recommended pattern to populate the names? I suppose this has to be done at run time. Also, can this API be used to rename the flow runs that are generated by the scheduler to have deterministic names based on clock parameters?
j

josh

09/30/2020, 1:27 PM
Currently the implementation of templating run names has been proposed and I encourage you to voice your use case here https://github.com/PrefectHQ/prefect/issues/3010 🙂 Yeah at runtime is when the names need to be populated because users want to base the task run names on the task inputs and now that the API is exposed we can safely do it
n

Nuno Silva

09/30/2020, 2:47 PM
Sounds good, like @Pedro Machado says, it would be easier in the gant and schematic views to have this. When the UI starts to pull this, how exactly can I use
set_task_run_name
to set the desired name:
set_task_run_name=lambda **kwargs: kwargs["val"]
? Thanks
j

josh

09/30/2020, 2:48 PM
Yeah something like that! I’m thinking we’ll expose a kwarg on the task where you can provide a callable and grab information like that on run
p

Pedro Machado

09/30/2020, 5:58 PM
Hi Josh, I am curious, if we contribute a PR for templating flow run names, would that be eventually available on cloud or just on server?
j

josh

09/30/2020, 6:46 PM
Yes! It will eventually be promoted to both services
View count: 1