• m

    Michael Ludwig

    2 years ago
    We just got this error in our logs? Anybody knows what this means?
    Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID 0c1f4327-19fe-4522-8f8c-ad4c2fb7dc45: provided a running state but associated flow run c4d445b2-8d4e-4863-9ba5-b89b5addeba3 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
    Traceback (most recent call last):
      File "/opt/app/.venv/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 112, in call_runner_target_handlers
        state = self.client.set_task_run_state(
      File "/opt/app/.venv/lib/python3.8/site-packages/prefect/client/client.py", line 1295, in set_task_run_state
        result = self.graphql(
      File "/opt/app/.venv/lib/python3.8/site-packages/prefect/client/client.py", line 287, in graphql
        raise ClientError(result["errors"])
    prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID 0c1f4327-19fe-4522-8f8c-ad4c2fb7dc45: provided a running state but associated flow run c4d445b2-8d4e-4863-9ba5-b89b5addeba3 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
    m
    nicholas
    6 replies
    Copy to Clipboard
  • Miecio

    Miecio

    2 years ago
    Hello everyone! I'm completely new to prefect (in fact I discover it couple days ago) and I wonder if my use case fits in what prefect can deliver. I need to run some kind of reporting pipeline working similarly to ETL schema. I need to extract ~1M of records from postgresql than for each of this record I need to query redis database, run some processing and save results to db. I manage to POC it as prefect flow but I have some questions. Does prefect support some concurrency for tasks similar to MapReduce? I can imagine that my data extracted from psql by extraction task can be splited and processed separately on multiple agents.
    Miecio
    i
    +1
    7 replies
    Copy to Clipboard
  • m

    Michael Ludwig

    2 years ago
    If we start to use Prefect
    Results
    like
    S3Result
    do I need to use them for all tasks to be able to restart failed flows? Not totally clear to me
    m
    nicholas
    3 replies
    Copy to Clipboard
  • m

    Michael Reeves

    2 years ago
    If I want my flow to run a series of tasks in an exact order ignoring data dependencies, is there an easier way to do that other than manually adding an edge between all the tasks?
    m
    nicholas
    11 replies
    Copy to Clipboard
  • j

    Jared

    2 years ago
    Does the
    late_work_seconds
    setting shown in the [prefect server config](https://github.com/PrefectHQ/server/blob/master/src/prefect_server/config.toml) actually do anything? As far as I can tell there's nothing referencing
    config.services.sla
    in the open-source server code, but this would be really useful to have access to
    j
    1 replies
    Copy to Clipboard
  • bral

    bral

    2 years ago
    Good day ! noticed that while executing a flow, the delay in transitioning from an upstream task to a downstream task is about 10 seconds. Is there some way to lessen this delay? Deployed local prefect server. Server resources are not used at all. a local agent is used, with an executor on the dask cluster.
    bral
    1 replies
    Copy to Clipboard
  • m

    Marwan Sarieddine

    2 years ago
    Hi folks, moving my issue post to the prefect-community channel. I am trying the S3 flow storage option - using a kubernetes agent and a DaskKubernetesEnvironment execution environment. I was facing aws s3 authentication issues but thanks to Josh - I got them resolved. I am now facing a new issue which is being raised by the prefect-job pod before the flow ever gets to a running state ... Here is the error I am getting:
    [5:57 PM] $ kubectl logs pod/prefect-job-e74114b4-26n6v 
    [2020-08-21 21:55:08] INFO - prefect.S3 | Downloading simple-flow-8/2020-08-21t21-54-33-484226-00-00 from infima-etl-flows
    Traceback (most recent call last):
      File "/usr/local/bin/prefect", line 8, in <module>
        sys.exit(cli())
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
        return callback(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/prefect/cli/execute.py", line 80, in cloud_flow
        raise exc
      File "/usr/local/lib/python3.8/site-packages/prefect/cli/execute.py", line 69, in cloud_flow
        flow = storage.get_flow(storage.flows[flow_data.name])
      File "/usr/local/lib/python3.8/site-packages/prefect/environments/storage/s3.py", line 101, in get_flow
        return cloudpickle.loads(output)
    TypeError: an integer is required (got type bytes)
    I am wondering if anyone else faced something similar
    m
    a
    3 replies
    Copy to Clipboard
  • m

    Maxwell Dylla

    2 years ago
    Hi Community! Is there any established best practices for developing a library of prefect flows? One of the struggles I have is instituting standards for a CI process for a library of flows. Part of the challenge is that code for flows represents a delayed execution of tasks, so it is difficult to test flows without deploying them to a specific environment
    m
    Chris White
    2 replies
    Copy to Clipboard
  • Kyle McEntush

    Kyle McEntush

    2 years ago
    Is there a way to run functions that are decorated with @task outside of prefect? I'm trying to rapidly develop individual task stages but I'd prefer to not have two functions defined. Currently, calling a function decorated with @task in a jupyter notebook throws a context error.
    Kyle McEntush
    Chris White
    +1
    26 replies
    Copy to Clipboard
  • Kyle McEntush

    Kyle McEntush

    2 years ago
    Next newbie question: What's the best way to apply a task over a list? The task also takes additional static arguments that don't change for each element. I tried to use map but then it tried to access indices of the static args
    Kyle McEntush
    Chris White
    +1
    5 replies
    Copy to Clipboard