• Emma Willemsma

    Emma Willemsma

    1 year ago
    Does anyone know if there's a way to set up workflow dependency imports so that they're only imported during flow runs? Say I have something like this:
    from prefect import task, Flow
    from prefect.environments.storage import Docker
    import boto3
    
    @task(log_stdout=True)
    def use_boto3():
        print('Using boto3 version {}'.format(boto3.__version__))
    
    with Flow('Sample Flow') as flow:
        use_boto3()
    
    storage = Docker(python_dependencies=['boto3'])
    storage.add_flow(flow)
    storage.build()
    flow.register(project_name='Dev', build=False)
    I would like to be able to run this to register my flow without having to have boto3 installed (since it will only be used during the flow run anyway)
    Emma Willemsma
    Dylan
    14 replies
    Copy to Clipboard
  • Robin

    Robin

    1 year ago
    We are having some major roadblocks when using prefect and snowflake. We got the following error when running a flow with ~35000 embarassingly parallel tasks (one for each of ~35000 systems):
    Failed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID 3417baee-44a2-4b39-82f4-c6ac6d073d1e: provided a running state but associated flow run 51cc335d-f029-45c6-80b4-8c88a0173dbc is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/prefect/engine/cloud/task_runner.py", line 128, in call_runner_target_handlers
        cache_for=self.task.cache_for,
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1321, in set_task_run_state
        version=version,
      File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 294, in graphql
        raise ClientError(result["errors"])
    prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID 3417baee-44a2-4b39-82f4-c6ac6d073d1e: provided a running state but associated flow run 51cc335d-f029-45c6-80b4-8c88a0173dbc is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
    Has anyone already experienced this? How to debug it? 😕
    Robin
    Chris White
    +2
    76 replies
    Copy to Clipboard
  • h

    Hui Zheng

    1 year ago
    hello, Prefect, we have a scheduled flow that runs on k8e agent. Sometimes, the run started 10 mins later than its scheduled time. It seems related to some irresponsiveness of the tasks and The prefect-server has attempted
    Rescheduled by a Lazarus process.
    For example, the one in the screenshot is scheduled for 11:10, but actually didn’t start until 11:20. Could anyone help understand why this happen and how to prevent it? Because we are building a new flow which need to run every 10 minutes with a very strict SLA. a 10 minute delay would be fatal to the new flow. Thank you
    h
    t
    +1
    3 replies
    Copy to Clipboard
  • ale

    ale

    1 year ago
    Hi folks, question related to flow scheduling: is it possible to tell Prefect to start a schedule only if there are no other running instances of that flow? Basically I want to prevent concurrent runnings of the same flow
    ale
    Raphaël Riel
    +2
    20 replies
    Copy to Clipboard
  • Robin

    Robin

    1 year ago
    Hey folks, I am asking myself how to set the logging level to DEBUG on prefect cloud when using dask-kubernetes environment on AWS EKS? We successfully added the env variable to the docker container using
    env_vars={"PREFECT__LOGGING__LEVEL": "DEBUG"},
    in
    flow.storage = Docker(...)
    . However, we still don't see the debug level log messages in cloud 🤔 Do we need to change anything else?
    Robin
    nicholas
    +2
    40 replies
    Copy to Clipboard
  • ale

    ale

    1 year ago
    Hi folks, how would you configure PostgresFetch when the connection details are available into environment variables which exists only at runtime when the flow is executed on Fargate?
    ale
    nicholas
    3 replies
    Copy to Clipboard
  • James Phoenix

    James Phoenix

    1 year ago
    Hey everyone should I start with prefect cloud or prefect server?
    James Phoenix
    nicholas
    16 replies
    Copy to Clipboard
  • flavienbwk

    flavienbwk

    1 year ago
    Hi ! I'm just starting off with Prefect and I was wondering if I can set up the Prefect server, Prefect UI and 1 Prefect worker directly in my own docker-compose file. Moreover, where do I find the ports to open and volumes to set up ?
    flavienbwk
    nicholas
    +1
    24 replies
    Copy to Clipboard
  • Pedro Machado

    Pedro Machado

    1 year ago
    Hi. I am trying to get the docker agent running on an ubuntu machine that has been somewhat restricted by the administrator. I was trying to run
    prefect agent start docker
    (passing several env variables) with the recipe created by
    prefect agent install local
    but I get a permission denied error and the agent never starts. I am able to start the agent without
    supervisord
    and it works fine. Any ideas? Logs in thread.
    Pedro Machado
    nicholas
    4 replies
    Copy to Clipboard
  • j

    John Song

    1 year ago
    Does Prefect store the data between tasks somewhere? If not, when task retries in a cluster of nodes, how the data is resent to the task? Does Prefect guaranteed the retried task goes to the same node? Sorry for so many questions
    j
    nicholas
    2 replies
    Copy to Clipboard