• Nathan Walker

    Nathan Walker

    1 year ago
    Hey folks, We're currently using Server until we can get Cloud approved at a client. There's some GraphQL queries/mutations we're looking at adding to Server (more metadata aggregation, schedule updates, etc. - probably no new resolvers) and we're wondering if adding to the GraphQL Schema will be an option in Cloud as well.
    Nathan Walker
    m
    4 replies
    Copy to Clipboard
  • Ismail Cenik

    Ismail Cenik

    1 year ago
    Hello, we are intermittently getting “ No heartbeat detected from the remote task, marking the run as failed.” When we look at the slack channel, we detect two solutions.1. Adding the "cluster-autoscaler.kubernetes.io/safe-to-evict": "false." 2. Another solution might be disabling the Lazarus toggle in the Prefect UI for our Plan Data Loading Flow -> Done We applied them but still getting the same error. What is your recommendation?
    Ismail Cenik
    Chris White
    3 replies
    Copy to Clipboard
  • Fabrice Toussaint

    Fabrice Toussaint

    1 year ago
    Hey everyone, Can someone explain to me why I keep getting this error in my flow runs 
    requests.exceptions.ReadTimeout: HTTPConnectionPool(host='prefect-apollo.prefect', port=4200): Read timed out. (read timeout=15)
    ? It does not happen all the time but it happens so often that a lot of tasks fail because of it. Our Prefect server is deployed using Kubernetes on GCP and the pod seems to be getting enough resources.
    Fabrice Toussaint
    Kevin Kho
    +1
    38 replies
    Copy to Clipboard
  • Fabrice Toussaint

    Fabrice Toussaint

    1 year ago
    Hey everyone, Does anyone know why I get this errors (sometimes) with scheduled tasks?
    Failed to retrieve task state with error: ClientError([{'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 5}], 'path': ['get_or_create_task_run_info'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: '}}}])
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 154, in initialize_run
        task_run_info = self.client.get_task_run_info(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1399, in get_task_run_info
        result = self.graphql(mutation)  # type: Any
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 319, in graphql
        raise ClientError(result["errors"])
    prefect.utilities.exceptions.ClientError: [{'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 5}], 'path': ['get_or_create_task_run_info'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: '}}}]
    Our Prefect server is deployed using Kubernetes on GCP with autoscaling enabled, maybe it has something to do with that?
    Fabrice Toussaint
    Kevin Kho
    20 replies
    Copy to Clipboard
  • Niko Usai

    Niko Usai

    1 year ago
    Hi everyone I've got a prefect agent running on a Kubernetes cluster for a year or so and everything was fine, then it started to give me this error few days ago, without any change in anything:
    [2021-05-07 08:16:03,101] ERROR - agent | Error while deploying flow: ValidationError({'_schema': 'Invalid data type: None'},)
    I've tried to search, but couldn't find any hint on what to look, anyone could help?
    Niko Usai
    Kevin Kho
    7 replies
    Copy to Clipboard
  • d

    Devin McCabe

    1 year ago
    Does anyone have any tips on managing environments with Prefect Cloud? Did you set up different projects for dev/qa/prod? Different flows within a single project? I'd love to hear about some examples of successful architectures if anyone has any.
    d
    Kevin Kho
    3 replies
    Copy to Clipboard
  • Andy Dyer

    Andy Dyer

    1 year ago
    Hi all, Does anyone have a good recipe that is pretty plug&play CI for deploying flows either with dind or mounting the docker sock. (preferably using gitlab.com runners)
  • d

    Devin McCabe

    1 year ago
    Does anyone know if the local Prefect server is supposed to work with
    DockerAgent
    ? I'm trying the example flow from the tutorial:
    from prefect import Flow
    from prefect import task
    from prefect.run_configs import DockerRun
    
    
    @task
    def add(x, y=1):
        return x + y
    
    
    with Flow("My first flow!", run_config=DockerRun()) as flow:
        first_result = add(1, y=2)
        second_result = add(x=first_result, y=100)
    
    flow.register(project_name="test-project")
    I submit the flow using the "Quick Run" button in the UI and start a Docker agent:
    prefect agent docker start --show-flow-logs --log-level DEBUG
    But the run forever stays in the pending state and I only ever see
    Querying for flow runs / No flow runs found
    in the agent logs. I've tried
    DockerRun()
    as above and also using a custom image containing only this:
    FROM python:3.8-buster
    RUN pip install prefect
    The UI shows it can reach my agent, though. Any ideas?
    d
    Kevin Kho
    11 replies
    Copy to Clipboard
  • m

    Marwan Sarieddine

    1 year ago
    Hi folks, we experienced a flow run failure that seems to stem from prefect cloud given the failure messages we see. Re-running the flow resolves the issue but I thought I would flag it in case it helps.
    m
    Kevin Kho
    +1
    12 replies
    Copy to Clipboard
  • n

    Noah Guilbault

    1 year ago
    Help needed / Question -- I have a local agent running in a docker container that is properly picking up a flow i publish from another container but failing with the following error message. "Failed to load and execute Flow's environment: ValueError('Flow is not contained in this Storage')" The flow is assigned a UniversalRun (i tried a LocalRun with the same effect), a LocalExecutor, LocalStorage, and labels ABC and DEF. The Flow has labels ABC, DEF and is properly picked up by the agent.
    n
    Kevin Kho
    15 replies
    Copy to Clipboard