• s

    simone

    1 year ago
    Hi I have a function wrapped in a
    @task
    decorator. When the function is executed in a flow everything runs fine. If I write the same code by subclassing
    Task
    I get a pickling error:
    pickle.UnpicklingError: NEWOBJ class argument isn't a type object
    The issue happened in a couple of other functions. I guess there is something basic in the Task construction that I am missing. I will appreciate if you can shine some light on this issue! Thanks Example of the code: Gist Tasks code
    s
    s
    +1
    25 replies
    Copy to Clipboard
  • k

    Kevin Weiler

    1 year ago
    hi prefecters - I’m having an issue with docker storage. I’m using a base image that has the environment and all the necessary code to run my flow. This is a conda environment that has prefect installed. When I attempt to store the flow container, it tries to install prefect over top of my conda installed version. This issue is compounded by the fact that the environment is owned by root and not the user of the container. Is there some way to tell prefect to not install the prefect package via pip?
    k
    Dylan
    6 replies
    Copy to Clipboard
  • m

    Mike Grabenstein

    1 year ago
    Hi New to Prefect, wondering if anyone is using Prefect to manage data flows with tasks written in Node, Java or Clojure. And how would you do it? Shell task that kicks the Node or Java/Clojure code? Or Container autorun like task? Thanks.
    m
    Dylan
    11 replies
    Copy to Clipboard
  • Walt Wells

    Walt Wells

    1 year ago
    for any great expectations users - how are teams making their GE configuration directory available in their task worker environments? https://docs.prefect.io/api/latest/tasks/great_expectations.html#rungreatexpectationscheckpoint
    Walt Wells
    r
    2 replies
    Copy to Clipboard
  • Saulius Beinorius

    Saulius Beinorius

    1 year ago
    Hi, is there a way to have task timeouts - to mark the task as failed if it runs too long (and then execute some code to stop it, for example)? In my case I can code that into the task itself, but it would be nice if there was some built-in mechanism.
    Saulius Beinorius
    nicholas
    4 replies
    Copy to Clipboard
  • Klemen Strojan

    Klemen Strojan

    1 year ago
    What is the best way to tell Kubernetes Agent to utilise only the nodes from a specific Node pool? We are running the agent on AKS in a separate namespace.
    Klemen Strojan
    j
    2 replies
    Copy to Clipboard
  • Alexander

    Alexander

    1 year ago
    Is it possible to re-run tasks in already running flow? I have quite a big DAG of tasks. If one task failed, its downstream tasks are triggerfailed. But the run still continues because there are many other parallel tasks which are running. So imagine i fixed the issue because of my task failed. How do i re-run it within existing running flow? Or should i wait for flow run to complete and only then restart flow run from failed?
  • Alexander

    Alexander

    1 year ago
    So i was able to narrow it down to a such case: imagine you have a flow with two branches, one branch is a long-running and other is short lived and may fail. So if short branch fails, its dependencies are triggerfailed. If i re-run failed task in a still running flow, its state is set to pending and nothing happens. Flow finishes when long task fininished and this task is still in pending state. But, if before re-running failed task i set downstream triggerfailed task to pending state, re-running failed task executes as expected. Is this a bug?
    Alexander
    nicholas
    20 replies
    Copy to Clipboard
  • z

    Zach

    1 year ago
    How do I unpickle a file that I used the
    PickleSerializer()
    to store during a prefect flow?
  • z

    Zach

    1 year ago
    The task decorator on the function looked like this:
    @task(
        checkpoint=True,
        result=GCSResult(
            bucket="my-bucket",
            location=RESULT_LOCATION_FORMAT,
            serializer=PickleSerializer(),
        )
    )
    z
    Jim Crist-Harif
    2 replies
    Copy to Clipboard