• Will Milner

    Will Milner

    1 year ago
    what's the best way to get a url back to a flow run when running locally? for context I'm trying to include this url inside of an email message I send on flow failure
    Will Milner
    Kevin Kho
    12 replies
    Copy to Clipboard
  • Ismail Cenik

    Ismail Cenik

    1 year ago
    Hello everyone, I hope you are well. We have an AWS Kinesis Data Analytics (KDA) application. A task in one of my flows is responsible for starting the KDA via the Boto3 library (start_application). My next task will start when the KDA stops, so I need to know the app status, so there is another API called “describe_application” to see its status. If the app status is “READY”, then I can continue with the next task. What is the best way to design that flow?1. Using "waiter" to see the status, but I could not find related waiters for KDAs. There are good examples for Amazon Batch. 2. Creating a separate task to call the “describe_application” in a scheduled way. As far as I understand schedules are generally used for flows. Could you please advise me on this?
    Ismail Cenik
    Kevin Kho
    +1
    10 replies
    Copy to Clipboard
  • Jacob Blanco

    Jacob Blanco

    1 year ago
    Is it possible to create new schedules for existing flows from GraphQL with default parameters? I know about
    set_flow_group_schedule
    . edit: Nevermind, I found it here: https://github.com/PrefectHQ/server/blob/4a9887091ffabe1a8c7098d6603663779d63e8eb/tests/graphql/test_flow_groups.py#L155 Leaving this here for posterity
    Jacob Blanco
    1 replies
    Copy to Clipboard
  • h

    Hawkar Mahmod

    1 year ago
    Morning all. I have a bit of confusion around registering my flows in our CI/CD process, and it boils down I think to this simple question - do I need prefect AND all my flow's dependencies installed when I make the call to
    flow.register()
    or
    prefect register
    ? I'm using
    S3
    Storage and
    ECS
    Run Config. I was previously using
    Docker
    Storage but didn't like the rebuilding of the image each time the flow code changed. I am using AWS CodePipeline/CodeBuild, so I have Docker daemon at my disposal, and I have a base image that contains the whole repo the flow sits in. Ideally, in my build environment I can build the base image if necessary, and register my flow with Prefect Cloud. However it seems I can't register unless I have all my dependencies, which happen to sit in the base image anyway. So perhaps I
    docker run
    and register this way? I've read the GitHub discussion on flow deployment patterns but I am not fully clear on this particular question.
    h
    Kevin Kho
    7 replies
    Copy to Clipboard
  • Varun Joshi

    Varun Joshi

    1 year ago
    Hey Prefecters, I don't want my flows to run between 2 AM to 10 AM from Mondays to Thursdays and 2 AM to 8 AM from Fridays to Sundays. I'm using the filters module provided by prefect for this. Assuming that I can do this,
    from prefect.schedules import IntervalSchedule, filters
    I made the below change in my code
    ist = pendulum.timezone('Asia/Calcutta')
    schedule = Schedule(
        clocks=[IntervalClock(interval=datetime.timedelta(minutes=1))],
        filters = filters.between_times(pendulum.time(6),pendulum.time(10))
    However, this doesn't seem to be working. Could someone please guide me on how to import filters and apply them to my schedule? Thank you, Varun
    Varun Joshi
    Kevin Kho
    11 replies
    Copy to Clipboard
  • tash lai

    tash lai

    1 year ago
    Morning. I'm looking for a way to manually trigger a Cached state from within a task, is it at all possible?
    tash lai
    Kevin Kho
    +1
    10 replies
    Copy to Clipboard
  • Aurélien Vallée

    Aurélien Vallée

    1 year ago
    Hmm, the concept of "building" flows when registering is a bit vague and I cannot find documentation on this. What does it actually mean to provide
    build=True
    ? The doc states:
    if
    True
    , the flow's environment is built prior to serialization; defaults to
    True
    Basically, I'm trying to automate the registration of flow in a CI environment, and I get warnings such as:
    UserWarning: A flow with the same name is already contained in storage; if you changed your Flow since the last build, you might experience unexpected issues and should re-create your storage object.
      serialized_flow = flow.serialize(build=build)
    Well, basically, since I'm automating the registration of flows, I do not really know which ones are new, which ones are identical, and which ones changed.
    Aurélien Vallée
    Kevin Kho
    3 replies
    Copy to Clipboard
  • Aurélien Vallée

    Aurélien Vallée

    1 year ago
    providing
    build=False
    does not trigger the warning, though I don't really understand the impact it has. Should I somehow manage to provide
    build=True
    for the first time a flow is registered and then
    build=False
    during updates?
  • Robin

    Robin

    1 year ago
    Dear prefect folks, I get an error when trying to schedule one prefect flow from the UI, while another one works fine. 😮 Setting a schedule for the same flow during flow registration also works fine. 🤔
    There was a problem creating your schedule, please try again shortly. Error message: Error: GraphQL error: Invalid clock provided for schedule: {'type': 'CronClock', 'cron': '0 * * * *', 'parameter_defaults': {...}, 'start_date': {'dt': '2021-04-14T14:21:57.622602', 'tz': 'Europe/Amsterdam'}}
    Robin
    Jenny
    +1
    6 replies
    Copy to Clipboard
  • tash lai

    tash lai

    1 year ago
    Hey prefectians I have a yet another question regarding memory usage. Not sure if it's a prefect problem or dask problem or just my ignorance. Here is the most useless flow humanity ever created:
    @task
    def donothing(x):
        pass
    
    with Flow('useless') as flow:
        lst = list(range(4000))
        donothing.map(lst)
    
    
    flow.executor = DaskExecutor('<tcp://localhost:8786>')
    dask-worker --nthreads=50
    Thing is the worker quickly eats up a lot of memory with each mapped task run, up to a gigabyte at the end of the flow, and that memory is not cleared when the flow finishes. The project i'm working on implies running up to ~100000 io-heavy tasks, so seeing this i'm a little worried that prefect might not be a right tool for the job. But maybe it's me doing something wrong?
    tash lai
    Kevin Kho
    +1
    6 replies
    Copy to Clipboard