• e

    Eric

    2 years ago
    I'm trying to figure out how to run some Tasks in parallel, like something like this in the second stage. There is no data passed between the stages, the tasks in the previous stage just have to complete with Success I know how to run them sequentailly like task_2(task_1()) but unclear if I should be using .map?
    e
    Chris White
    13 replies
    Copy to Clipboard
  • Newskooler

    Newskooler

    2 years ago
    Hi everyone 👋, Is it possible to create a task which is triggered based on some external event (e.g. email received, or website updated specific data point, etc)? I wonder how would one go about creating such a flow?😃 🤔
    Newskooler
    Chris White
    +1
    6 replies
    Copy to Clipboard
  • v

    Vitaly Shulgin

    1 year ago
    Hello team, previously I was asking to clarify why it is impossible to register new flow from parent flow, herehttps://prefect-community.slack.com/archives/CL09KU1K7/p1600240447350500?thread_ts=1600209791.334200&cid=CL09KU1K7 Could you bring some light on that case? Like cold system start, where the one parent flow creates many new flows, initially stored in git or database. What will be the way to accomplish it with current prefect funcionality?
    v
    1 replies
    Copy to Clipboard
  • a

    Adrien Boutreau

    1 year ago
    Hello! Thanks again for your product! It's amazing! We did installation of Prect Core in AWS and we want to do :1. SSL certificated added on prefect 2. Have a login page to the prefect UI Do you know where I can documentation about theses 2 points ? Thanks!
    a
    Pedro Machado
    +2
    7 replies
    Copy to Clipboard
  • Jovan Visnjic

    Jovan Visnjic

    1 year ago
    Hi everyone. I have a question about
    LocalResult
    and exceptions. We use
    LocalResult
    for checkpointing, so only failed tasks get executed on a workflow re-run. So far we execute the workflow without a server, using just the basic CLI. If
    LocalResult
    exists, the task is skipped and just tasks with missing results get executed. We would also like to store error information on the disk. But if write exception's content as a
    LocalResult
    through task's
    target
    parameter, it gets treated as successful, because something exists at the result location, whatever that may be. Is there some established way of dumping errors to disk, so they don't just remain in the console's
    stdout
    , but that at the same time they don't obstruct the logic of what is successful and what not? Any ideas or comments appreciated. Jovan
    Jovan Visnjic
    Kyle Moon-Wright
    2 replies
    Copy to Clipboard
  • j

    Jonas Bernhard

    1 year ago
    Hi team, thanks a lot for Prefect, having a lot of fun working with it so far! I have two questions1. When mapping over results of a previous task, is it possible to only continue with the ones that were successful? In my use case, I first fetch feeds for a list of urls with
    fetch_feed_for_url
    and then save them with
    save_feed_entries
    (with trigger
    any_successful
    ). However, some of the urls might not work leading to all subsequent tasks mapping over the result (and with that the flow) also failing. What I'm looking for is to map only over "successful
    feeds_list
    entries"
    feeds_list = fetch_feed_for_url.map(url=feed_urls)
        save_feed_entries = save_feed_entries_to_db.map(feed=feeds_list)
    2. Is it possible to initialize a task from a Task class using
    EnvVarSecret
    ? For example, creating a Task using the Task Library often requires the credentials upfront. Are all the secrets required to be available at "flow building time" to work with Task classes?
    fetch_mysql = MySQLFetch(
      user=EnvVarSecret("USER)
      password=EnvVarSecret("PASSWORD)
    )
    Thanks!
    j
    Kyle Moon-Wright
    5 replies
    Copy to Clipboard
  • e

    Eric

    1 year ago
    I'm trying to use the files{} parameter with the Docker storage class, it says it can take files or directories, however it only deploys if i have a file specified, not a directory. When pointed at the directory I get a "permission denied" error, https://stackoverflow.com/questions/11278066/using-shutil-copyfile-i-get-a-python-ioerror-errno-13-permission-denied/11278116 files (dict, optional): a dictionary of files or directories to copy into the image when building. Takes the format of 
    {'src': 'dest'}
    e
    Chris White
    +2
    8 replies
    Copy to Clipboard
  • v

    Von Christian Pizarro

    1 year ago
    Hi Everyone, Has anyone experienced their flow running an hour later after DST? Just wanted to know if there's a way to disable it or would I need to update our flows manually? Thanks!
    v
    Kyle Moon-Wright
    3 replies
    Copy to Clipboard
  • e

    EdCode

    1 year ago
    Hi, I have prefect on a RHEL box with a windows drive mounted via smb. I would like airflow to trigger some python code using the WINDOWS process, not the linux process. Is this possible?
    e
    1 replies
    Copy to Clipboard
  • Rob Fowler

    Rob Fowler

    1 year ago
    can you mark a task as always_run in a flow? For example I get some results but sometimes I want the later tasks to run, sometimes not, so I don't want to decorate the actual task as allways_run