https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • b

    Bob De Schutter

    05/20/2022, 2:58 PM
    Hello, I'm trying to register flows that are generated dynamically based on a yaml file and that are stored in a list. In short, I'm using code like this (flows is a simple python list of Flow objects):
    for flow in flows:
            flow.register(
                    project_name=f"Data Platform - {os.environ.get('PREFECT_TARGET_ENV', 'Develop')}",
                    add_default_labels=False,
                    set_schedule_active=True if os.environ.get('PREFECT_TARGET_ENV', 'Develop') == 'Production' else False
                )
    This has worked for me before but since I upgraded prefect to 1.2.1, I'm getting an error message related to the Azure storage I'm using for the flows:
    azure.core.exceptions.ResourceExistsError: The specified blob already exists
    Any idea why this strategy is not working anymore?
    k
    k
    • 3
    • 24
  • j

    James Keegan

    05/20/2022, 3:47 PM
    Hello! I'm trying to clone a bitbucket repository into a Prefect Docker image (using the
    ShellTask
    task) but keep running into this error. Any suggestions? The SSH key is set up correctly, but there doesn't appear to be a user associated with the
    ShellTask
    commands. This wasn't an issue previously, as older flow images can pull git projects just fine. Only came up recently after I rebuilt the image again (Prefect version & python version are still the same).
    Cloning into './dbt_project'...
    No user exists for uid 1000190000
    fatal: Could not read from remote repository.
    
    Please make sure you have the correct access rights
    and the repository exists.
    k
    • 2
    • 6
  • j

    James Keegan

    05/20/2022, 5:00 PM
    cloudpickle
    v2.1.0 was just released an hour ago and it has broken the healthcheck when we try to register a flow. We're using the
    prefecthq/prefect:0.15.6-python3.7
    as a base image. We now get this error when building the docker image.
    Step 21/21 : RUN python /opt/prefect/healthcheck.py '["/opt/prefect/flows/prefect-dbt-run-modelling.prefect"]' '(3, 7)'
     ---> Running in 1d14c333ced9
    Beginning health checks...
    System Version check: OK
    Traceback (most recent call last):
      File "/opt/prefect/healthcheck.py", line 152, in <module>
        flows = cloudpickle_deserialization_check(flow_file_paths)
      File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
        flows.append(cloudpickle.loads(flow_bytes))
    AttributeError: Can't get attribute '_make_function' on <module 'cloudpickle.cloudpickle' from '/usr/local/lib/python3.7/site-packages/cloudpickle/cloudpickle.py'>
    k
    m
    +2
    • 5
    • 13
  • f

    Florian Guily

    05/20/2022, 5:21 PM
    Hello, more a python issue but encountered when trying to execute this tuto. I have a project structures that looks like the attached image. This structure is the same as @Anna Geller github repo for the tuto. In her flow codes, she imports functions from a
    config.py
    file in
    flow_utilities
    folder like this
    from flow_utilities.config import funcA funcB
    . When i do so, i get an error. So i tried to reproduced it with the dummy code you have on the image. I get the same error which is
    ModuleNotFoundError: No module named 'flow_utilities'
    . This really seems stupid to me as it is just importing some func from local file but somehow i can't make it work with this file architecture. What am i doing wrong ?
    ✅ 1
    a
    • 2
    • 4
  • f

    Florian Guily

    05/20/2022, 5:46 PM
    New issue: When using
    Secret.get()
    it apparently only search for local secrets. The doc says i have to change
    prefect.config.use_local_secrets
    to false but i can't understand how... I suppose it is in the config.toml file but as the flow has to run on an eks cluster, how can i specify this for the cluster ?
    k
    • 2
    • 2
  • m

    Mike Vanbuskirk

    05/20/2022, 7:24 PM
    does anyone have a link to an example Dockerfile for a custom image that a flow might run on?
    a
    k
    • 3
    • 8
  • m

    Malthe Karbo

    05/20/2022, 7:54 PM
    When attaching a log handler (e.g., slack) to prefect logger, then whenever running a flow with dask, it will log multiple times with the handler - at least this is what we see with our implementation - Is there a good example for attaching a custom log handler to prefect orion logging?
    k
    • 2
    • 2
  • a

    Alvaro Durán Tovar

    05/20/2022, 8:42 PM
    if I want to create flows dynamically like this (registration happens after)
    def build(...):
        with Flow(...) as Flow:
            ...
        return flow
    is it possible to use docker storage? thinking on possible issues trying to find the path of the flow inside the docker file, there won't any "flow" variable on the module level
    k
    a
    • 3
    • 12
  • p

    Philip MacMenamin

    05/20/2022, 9:42 PM
    Hello - can I get some help with debugging a failing workflow?
    k
    • 2
    • 22
  • n

    Nash Taylor

    05/20/2022, 10:55 PM
    I feel like I'm going to gain a reputation as the guy who only ever asks about type safety but... I've got pyright integrated into my environment now, which is working far better with Prefect than Mypy did. It just requires one small hack, and it makes me wonder why this isn't part of the library: I had to add
    py.typed
    on my own. I see that there is a
    py.typed
    on the master branch; was there a conscious decision made not to include this marker file in 2.0? If so, why? Without the marker file, I get:
    from prefect.flows import flow
    
    reveal_type(flow)  # Unknown
    With the marker file, I get:
    from prefect.flows import flow
    
    reveal_type(flow)  # Overload[(__fn: (**P@flow) -> R@flow, /) -> Flow[P@flow, R@flow], (*, name: str = None, version: str = None, task_runner: BaseTaskRunner = ConcurrentTaskRunner, description: str = None, timeout_seconds: int | float = None, validate_parameters: bool = True) -> (((**P@flow) -> R@flow) -> Flow[P@flow, R@flow]), (__fn: Unknown | None = None, *, name: str = None, version: str = None, task_runner: BaseTaskRunner = ConcurrentTaskRunner, description: str = None, timeout_seconds: int | float = None, validate_parameters: bool = True) -> (Flow[P, R] | (((**P) -> R) -> Flow[P, R]))]
    which is much better. For example, now when I decorate a function with
    @task
    and use it in an
    @flow
    , the return type of the task is known to have a
    .result()
    method. When it was just an Unknown, pyright would complain that I'm accessing a method that doesn't exist. For someone who enforces fully passing mypy (now pyright) checks on every pull request, this is kind of a necessity.
    m
    a
    • 3
    • 8
  • g

    GGK Kellie

    05/21/2022, 2:34 AM
    Hello, when I run prefect 2.0b5 on windows machine, there is an error message "ValidationError: 1 validation error for Settings PREFECT_ORION_API_PORT value is not a valid integer (type=type_error.integer)". I try to reset it with "prefect config set PREFECT_ORION_API_PORT=4200". It showed the same error message. Any idea to fix it? Thanks.
    k
    • 2
    • 4
  • d

    Daniel Sääf

    05/21/2022, 11:45 AM
    Hi. I continue to ask my questions here - super grateful for the help i got so far. I have now setup a flow that launches a number of subflows that fetches data from Google Cloud Storage and writes the data to Big Query. I’m running the flow locally (prefect 2.0) and it’s connected to prefect cloud. However, i sometimes run into an error i cannot really understand and i don’t know how to troubleshoot it. It only happens occasionally and i cannot really reproduce it. I moved the trace to the thread. The logging message @
    16:34:10.826
    is the last thing that happens in the task read_blob in which the error occurs in. So it looks to me that something goes wrong when reporting the task. The error message doens’t tell me that much - so if you have any advices on how i should troubleshoot this i would be really helpful (or if you can guess on what might be wrong?)
    a
    m
    • 3
    • 6
  • c

    Constantino Schillebeeckx

    05/21/2022, 2:08 PM
    is something funky going on with logging in the prefect UI? for flows that failed, i'm not getting the full logs (in the UI) - I send those logs to AWS cloudwatch and can confirm that full expected logs are there 😞
    a
    k
    • 3
    • 10
  • c

    Clément VEROVE

    05/21/2022, 3:11 PM
    Hi everyone 👋 I have some troubles with the running of my flow on kubernetes. My flow includes docker command such as
    docker volume create
    /
    docker-compose up
    so i need docker daemon but it cannot be outside my job. Here is my job template
    apiVersion: batch/v1
    kind: Job
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: flow-container
            - name: dind-daemon
              image: docker:stable-dind
              env:
                - name: DOCKER_TLS_CERTDIR
                  value: ""
              securityContext:
                privileged: true
          imagePullSecrets:
            - name: regcred
    It works but my docker daemon container never stop...... any ideas ?
    a
    • 2
    • 3
  • d

    Daniel Saxton

    05/21/2022, 4:04 PM
    any suggestions / best practices for triggering Prefect jobs within a CI/CD pipeline? suppose we're building a Docker image within the pipeline and pushing it to a container registry, and want to execute a flow using that container image, guaranteeing that we're always using the latest (can we do this with a Docker agent?)
    e
    a
    • 3
    • 3
  • j

    Joshua Greenhalgh

    05/21/2022, 4:22 PM
    Hi wonder if anyone could help me with a problem I have working with the Dask KubeCluster? So the issue I am having is that various secrets that I have mounted to the usual flow jobs don't get carried over to the pods that are started by dask - there is an added complexity that I am using two images a dev one and a non dev one tied to two different prefect projects - I am able to do something like this to switch the image;
    DEV_TAG = os.environ.get("DEV", "") != ""
    
    JOB_IMAGE_NAME = f"blah/flows{':dev' if DEV_TAG else ''}"
    And then in each flow I ref the
    JOB_IMAGE_NAME
    - this just changes the image but otherwise uses the job template I have defined on the agent;
    apiVersion: batch/v1
    kind: Job
    spec:
      template:
        spec:
          containers:
            - name: flow
              imagePullPolicy: Always
              env:
                - name: SOME_ENV
                  valueFrom:
                    secretKeyRef:
                      name: secret-env-vars
                      key: some_env
                      optional: false
    Now when I specify the dask setup I do the following;
    executor=DaskExecutor(
            cluster_class=lambda: KubeCluster(make_pod_spec(image=JOB_IMAGE_NAME)),
            adapt_kwargs={"minimum": 2, "maximum": 3},
        )
    But this is obviously missing the env part of my default template - I would like to not have to respecify it (its much bigger then the above snippet) - is it possible to grab a handle on the default template and just override the image name?
    a
    • 2
    • 11
  • k

    Kayvan Shah

    05/21/2022, 4:57 PM
    I am trying to write a DeploymentSpec YAML config file referring to this example:
    $ prefect deployment inspect 'hello-world/hello-world-daily'
    {
        'id': '710145d4-a5cb-4e58-a887-568e4df9da88',
        'created': '2022-04-25T20:23:42.311269+00:00',
        'updated': '2022-04-25T20:23:42.309339+00:00',
        'name': 'hello-world-daily',
        'flow_id': '80768746-cc02-4d25-a01c-4e4a92797142',
        'flow_data': {
            'encoding': 'blockstorage',
            'blob': '{"data": "\\"f8e7f81f24512625235fe5814f1281ae\\"", "block_id":
    "c204821d-a44f-4b9e-aec3-fcf24619d22f"}'
        },
        'schedule': {
            'interval': 86400.0,
            'timezone': None,
            'anchor_date': '2020-01-01T00:00:00+00:00'
        },
        'is_schedule_active': True,
        'parameters': {},
        'tags': ['earth'],
        'flow_runner': {'type': 'universal', 'config': {'env': {}}}
    }
    Is there any extensive example available to write the complete config for a flow??
    a
    • 2
    • 3
  • k

    Kayvan Shah

    05/21/2022, 6:29 PM
    Can't get the reason why there are so many late runs piling up Have scheduled about 6-7 flows on single node cluster via minikube
    k
    a
    • 3
    • 12
  • j

    Jan Domanski

    05/21/2022, 6:43 PM
    Hi there what’s the best practice for passing database parameters into the prefect flows? I have a flow that I want to connect to different DBs (alpha/beta/prod), should I just use the Parameter mechanism for this?
    k
    • 2
    • 3
  • m

    Marwan Sarieddine

    05/21/2022, 8:48 PM
    Not sure if others have faced this as well, but we experienced an issue at 4:00pm EST where all flows that were scheduled to run on prefect cloud at 4:00 pm EST were only picked up by our agents at 4:25 pm EST. It seems that things are back to normal now, given runs are being picked up by our agents promptly
    ✅ 1
    a
    • 2
    • 1
  • h

    Hui Zheng

    05/21/2022, 8:56 PM
    FYI, we experience flow run issues from 1PM - 1:30 PDT time. The scheduled flow runs were not triggered or picked up to run.
    ✅ 1
    m
    • 2
    • 1
  • a

    Anna Geller

    05/21/2022, 9:15 PM
    @Marwan Sarieddine and @Hui Zheng We've seen some spike in traffic during that time, but the issue has been resolved now. We apologize, and we'll keep you up-to-date as we monitor the services.
    m
    • 2
    • 1
  • q

    Quan Cao

    05/22/2022, 5:14 AM
    Hi. I'm using client 2.0b4 and orion cloud. Got problem recently with the same setup, my agent fails to pick up runs from queue. All IDs are correct (was working fine for weeks), yet it keeps on failing. What's wrong?
    httpx.HTTPStatusError: Server error '500 Internal Server Error' for url '<https://api-beta.prefect.io/api/accounts/ff4de07a-3c0a-4831-a96f-236ce6513b52/workspaces/c36fd28e-d969-4a32-bb11-0f7712f96635/work_queues/cfbcef84-6668-41ff-83dd-0053055deffb/get_runs>'
    ✅ 1
    a
    • 2
    • 11
  • j

    jars

    05/22/2022, 10:49 AM
    Hey @Prefect Cloud - our production alert system just alerted us. It looks like Flows are not kicking off in Prefect Cloud. They are getting stuck in the Scheduled and Submitted States. Potential repeat of this afternoon?
    👍 1
    ✅ 1
    q
    a
    • 3
    • 4
  • a

    Anna Geller

    05/22/2022, 10:56 AM
    Hi, everyone. We see some performance degradation in Prefect Cloud. We are aware of the issue and working on a resolution. We apologize, and we'll keep you up-to-date. cc @jars @Quan Cao
    👀 1
    q
    j
    • 3
    • 5
  • j

    jars

    05/22/2022, 10:59 AM
    Thanks @Anna Geller
    👍 1
  • k

    Kayvan Shah

    05/22/2022, 11:10 AM
    I observed that the pods start and keep failing and restarting While no resources show up when we get pods for default namespace
    error.txt.py
    a
    • 2
    • 30
  • a

    Assaf Ben Shimon

    05/22/2022, 11:12 AM
    Hi 🙂 I'm getting error 500 when running my flow using Orion. Any idea what's wrong?
    httpx.HTTPStatusError: Server error '500 Internal Server Error' for url '<https://api-beta.prefect.io/api/accounts/927cf1b8-198e-4236-841e-36098d433977/workspaces/be783b36-6bf5-4c3e-b919-8da5b8ea57cc/task_runs/277f8cff-c1aa-47b4-a5d7-126ce2957ad3/set_state>'
    ✅ 1
    a
    • 2
    • 4
  • j

    jars

    05/22/2022, 11:26 AM
    Starting to see Flow Runs kick off again now...
    ✅ 1
    a
    • 2
    • 4
  • a

    Anna Geller

    05/22/2022, 11:52 AM
    A short update on the above: all systems are back to normal since 11:31 AM UTC - the status page is updated and we keep monitoring the services. Thanks to everyone reporting the issue!
    f
    • 2
    • 9
Powered by Linen
Title
a

Anna Geller

05/22/2022, 11:52 AM
A short update on the above: all systems are back to normal since 11:31 AM UTC - the status page is updated and we keep monitoring the services. Thanks to everyone reporting the issue!
f

Florian Kühnlenz

05/22/2022, 12:41 PM
We are still seeing some fallout, of flows getting canceled or killed by Lazarus.
a

Anna Geller

05/22/2022, 2:25 PM
Thanks Florian, could you post more info of what you see? This might be unrelated to the incident, I'm afk but can check in the evening if you post more info incl.flow run ID
f

Florian Kühnlenz

05/22/2022, 2:29 PM
Sure.
cb076c5-f718-4225-b9e9-bb15e0d19665
This is probably the one. For some reason we were notified about it twice. The other errors were from before 11:31 UTC I just realized.
👍 1
a

Anna Geller

05/22/2022, 6:31 PM
weird, I can't find this run in the system
are you sure you sent the right flow run ID?
so it might be related to the brief API issue we had today - LMK if not and we can investigate more
f

Florian Kühnlenz

05/22/2022, 6:33 PM
I will check again tomorrow, but I copied the id from a notification link.
👍 1
Somehow the first character went missing. Sorry about that:
1cb076c5-f718-4225-b9e9-bb15e0d19665
a

Anna Geller

05/24/2022, 1:39 PM
I checked the logs and it doesn't seem related to the outage but to task runs losing heartbeats in general I was trying to find out more about the task runs that lost heartbeat - e.g. task run ID 40497634-4fe7-4005-93c8-f0e6d4a06a2b but it doesn't show any log messages in our system I don't even know how to help here honestly, the logs show very little and it looks like your tasks lost communication with the API and as a result, we don't have any state updates and logs in the system and Lazarus was triggered to reschedule the flow run - Lazarus was trying to be helpful here in a way that it wanted to reschedule the work that couldn't be finished
👍 1
View count: 4