• a

    Alex Post

    2 years ago
    Hi - I'm having an issue with setting a parameter according to the documentation found here: https://docs.prefect.io/core/concepts/parameters.html My flow looks like this:
    with Flow('Kafka Data Quality Test') as flow:
        full_run = Parameter('full_run', default=False)
    
    flow.run(parameters=dict(full_run=True))
    and I keep getting this error:
    Traceback (most recent call last):
      File "src/app.py", line 243, in <module>
        flow.run(parameters=dict(full_run=True))
      File "/Users/apost/ccde/ccde-kafka-data-quality/venv/lib/python3.7/site-packages/prefect/core/flow.py", line 991, in run
        fmt_params
    ValueError: Flow.run received the following unexpected parameters: full_run
    Has anyone seen this before?
    a
    j
    13 replies
    Copy to Clipboard
  • Dylan

    Dylan

    2 years ago
    Hey Prefect Community! For those of you using
    LOOP
    signals in your Flows, would you be up to sharing what you use them for? We’re considering a change to the behavior and we’re interested in your thoughts /thread
    Dylan
    emre
    3 replies
    Copy to Clipboard
  • n

    Nat Busa

    2 years ago
    Hi, I hope this is the right place to post it ... I understand your opencore philosophy, but I am having a hard time choosing between prefect and two other projects: dagster and metaflow ... Also just using dask distributed does not sound bad ... Any thoughts on this?
    n
    Chris White
    +1
    5 replies
    Copy to Clipboard
  • a

    ARun

    2 years ago
    hey guys, got a basic questions. Just found out about prefect, and started messing around with it today. I've been following along the docs. And I've tried using this locally, and using the dask executor. I think I'm missing something. I've made a simple job to move file from a database into a sftp. However, when I run the file, I get this message. If I ctrl-c out of it, it crashes the program. Do i have to do it in nohup, or is there a way i can execute the script in the background. So it just runs, and follows the schedule? If there is any documentation on this, that would be of great help. Thanks!
    [2019-12-05 18:38:18,226] INFO - prefect.Flow: sf -> sftp | Waiting for next scheduled run at 2019-12-05T21:00:00+00:00
    a
    j
    9 replies
    Copy to Clipboard
  • l

    Luke Orland

    2 years ago
    I am developing a workflow, and I'm having trouble writing tests that mock out the
    run()
    method of tasks in the workflow.
    l
    Chris White
    15 replies
    Copy to Clipboard
  • k

    Kevin Hill

    2 years ago
    I'm speaking in hypotheticals here, I haven't dug entirely through the framework (minus the easy first example), though I wanted to ask a few questions: 1. Can if I had a hypothetical Reinforcement Learning training pipeline run through prefect, could it handle millions of interactions over the course of a training period? 2. Is it possible to push information from the end of the flow to the beginning if a condition is met? 3. If so, how fast would that be? Can I use Dask to split certain tasks up between machines and concurrently process more tasks? 4. Would that use the
    LOOP
    API?
    k
    Dylan
    12 replies
    Copy to Clipboard
  • n

    Nat Busa

    2 years ago
    Hi community: Anybody working on a papermill task?
  • k

    Kevin Hill

    2 years ago
    Is it possible to put a task inside of another task?
    k
    1 replies
    Copy to Clipboard
  • k

    Kevin Hill

    2 years ago
    You know, I'll try checking myself tomorrow
  • k

    Kevin Hill

    2 years ago
    So, I saw one of your blog articles about loops. Could you go through a usecase like I'm asking for in another one? It's pretty simple. You need to send events through a pipeline, you'd identify what data you'd need to process on by ID, then you'd log the result in another step, then check for resending a task with that given id to the beginning of the pipeline