https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-server
  • e

    Edvard Kristiansen

    05/05/2022, 8:41 AM
    Hey, I recently ran into this issue:
    Failed to load and execute flow run: RuntimeError("can't start new thread")
    Restarting the VM resolved the issue but no flows were able to run at all until it was restarted, all with the same or similar errors. Is there any way to fix this without restarting the VM?
    a
    • 2
    • 9
  • m

    Michael Hadorn

    05/05/2022, 11:56 AM
    Hey, sometimes we miss some CloudTaskRunner Errors in the output of
    prefect agent docker start --show-flow-logs
    . We can see them in the GUI, but not in the terminal output. Is this a known behavior?
    a
    m
    • 3
    • 14
  • l

    Lana Dann

    05/05/2022, 5:59 PM
    do you have a code example of setting a filter in a certain timezone? i think they default to UTC but i’d like to set them in PST (since the start time is in PST)
    k
    • 2
    • 5
  • m

    Michael Smith

    05/06/2022, 1:58 PM
    Hello, I'm starting to test the Kubernetes Agent on Prefect 1.x....our requirement is to run up to 1000 concurrent jobs. With the standard executor I'm seeing a memory requirement of about 200MB per job, so we'd struggle to reach 1000 concurrent jobs on our moderately sized K8s cluster. Would the Static Dask Cluster approach be a better option? Are there any trade-offs in terms of reliability compared to using Kubernetes jobs - our workload is expected to be a mixture of short running (a minute) and long running (a few minutes to an hour) jobs - which might influence our approach towards reliability.
    k
    • 2
    • 13
  • e

    Edvard Kristiansen

    05/06/2022, 2:21 PM
    Hey, I am trying to send a notification with the state of a task when it finishes/fails. Something like "X completed with status: success". I didn't really find anything similar here other than the get_task_run_result but that seemed overkill? Also not sure how that would work to get the status? Is there any way I can call the state of the task like below?
    run = DbtShellTask()
    
        send_notification = send_message(title = "Run Completed" , body = "The run has completed with status: "  + run.status)
    At the same time, it would also be great to return any error messages on a failed run. Any idea how to do that?
    k
    • 2
    • 5
  • m

    madhav

    05/06/2022, 7:11 PM
    Hi - we are running into a problem with running a flow on a docker container. We get the following error:
    Failed to load and execute flow run: ModuleNotFoundError("No module named '/root/'")
    Our setup: • a GitHub agent packages and deploys a docker container to ECR • ECS pulls the ECR image and deploys a fargate instance • (For our proof of concept) When the machine runs, we register a flow that lives in the image/fargate instance • After that we run
    prefect local agent start
    (both flow registration and starting the agent are run from a script that our ECS task definition runs on boot) • We do a “quick run” and get the module not found error. Other things to note: • all of our flows and supporting modules are on this container. • I’ve read some of the existing discussion surrounding this error and was not sure if the solution applied to our situation - given that we are running everything on the same Fargate instance
    k
    m
    • 3
    • 12
  • l

    Luke Orland

    05/06/2022, 9:42 PM
    Is it possible to update a registered flow's project value (effectively moving it to a different project without changing anything else, like the flow's run history, etc.)?
    k
    • 2
    • 9
  • i

    Ievgenii Martynenko

    05/09/2022, 8:49 AM
    Hi, Yesterday we faced 2 issues with Prefect flow which uses K8s job_template. When job started it started to fail with not very useful:
    'str' object has no attribute 'setdefault'
    after examining code we understood that issue comes from https://github.com/PrefectHQ/prefect/blob/9b5fb783d64299a180e5a54ed88ca6d4a1f8fb90/src/prefect/agent/kubernetes/agent.py#L490 . It could be an access issue or missing file, or even a typo in yaml file. There is no error handling or checks to point to root cause properly. MIght be a good minor story to add checks and throw proper exception.
    a
    • 2
    • 5
  • a

    Architha Rao

    05/09/2022, 11:27 AM
    Hi. We are using prefect 0.15.13 server version. The graphql (v2.1.1) throws the below error. It persists even after multiple restarts.
    a
    k
    • 3
    • 5
  • a

    Alexis Lucido

    05/09/2022, 4:56 PM
    Hi, I have been asking for help some time ago, regarding a issue of tasks and flows getting longer and longer to complete until the whole application freeze (my processors being filled with many processes). I have been told to dockerize the app through the guide provided in this link, which I have done, also because it made our deployment process cleaner and more robust. That was quite some work but it has been making our job a lot easier. Unfortunately, we can still observe the same pattern today, though less extreme. I have joined a screenshot of a flow. It almost always fails because it maps a task to fetch data from different sources, and one of them is down at the moment. The flow takes longer to complete not because of the volume of data to fetch. Indeed, it gathers data on a daily basis: every half-hour, it fetches all the data, accumulating throughout the day, for the current date, and it takes longer today at 1am (without much data then) than yesterday at 11pm. When i redeploy the application, the flows gets back to the normal in terms of speed. I've done that this morning, which is why the bars are shorter at the end of the screenshot. I am kinda lost, as we risk, without close monitoring, a failure of our whole environment. The issue makes me think of the Airflow's instable scheduler one. As a matter of fact, the fix I've thought of is to redeploy the application nightly, but I would by far better understand the root cause of the issue. If you have any idea, I'd be delighted to make some further research with you, and maybe we could help Prefect become an even better product. Alexis
    k
    • 2
    • 10
  • e

    ek

    05/09/2022, 8:28 PM
    Has anyone trying to build/send a flow(docker storage) from github-runner? Basically, this is what I’m doing: • prefect up and running on k8s • trigger a build by pushing a flow to github on specific branch • a self hosted github-runner on k8s that is waiting to exec a job ◦ github runner has docker in docker enabled and can communicate with prefect-server-apollo api ◦ prefect is installed on my github-runner ◦
    ~/.prefect/config.toml
    looks like this:
    backend = "server"
    [server]
    endpoint = "<http://prefect-server-apollo.prefect-server:4200>"
    • in
    .github/workflows/build.yml
    , execute
    python flow.py
    which it should build as docker storage when ever my github-runner exec
    python flow.py
    , it won’t build the docker image for me here is the error message:
    <http://localhost:2376/version>: Bad Request ("b'Client sent an HTTP request to an HTTPS server.'")
    Any thoughts/suggestions would be much appreciate it!
    k
    m
    • 3
    • 8
  • l

    Lana Dann

    05/10/2022, 6:26 PM
    is there a way to set dependencies between two flows (not tasks)?
    a
    k
    • 3
    • 6
  • j

    Jeff Kehler

    05/11/2022, 3:09 AM
    I'm running prefect server on a EC2 machine using
    --expose
    and connecting to the UI using the server IP address on port
    8080
    . All that works fine, however I noticed that the url in Slack notifications are using
    <http://localhost:4200>
    . How can I configure this part?
    k
    a
    • 3
    • 8
  • m

    Maria

    05/11/2022, 9:10 AM
    Hello, is it possible to use multiple different images for tasks in a prefect flow? And with a Kubernetes runner or Dask runner?
    a
    • 2
    • 6
  • i

    iñigo

    05/11/2022, 4:15 PM
    Hello community, I have prefect prefect in a docker over a linux VM. is it really necesary or better to move to prefect 2.0 I already have around 20 scripts that are loaded in v0.15.3. If so what is the best way to upgrade to v2 or v1?
    a
    • 2
    • 9
  • b

    Binoy Shah

    05/11/2022, 6:46 PM
    When is Prefect 2.0 Cloud going to be GA ?
    a
    • 2
    • 2
  • b

    Binoy Shah

    05/11/2022, 6:47 PM
    Does Prefect Cloud have any certifications FedRAMP or otherwise ?
    a
    • 2
    • 1
  • l

    Luiz Felipe

    05/12/2022, 6:53 PM
    Hello! I'm running a Prefect Server on K8s and instantiating a Dask cluster to run my flows. It seems to be working well (the flow finishes successfully), but it is not storing the task logs on the UI. Is there any way to guarantee it is going to be there? When I've added a
    sleep(60)
    before the end of the task, the task logs appeared, but without it it does not work.
    k
    • 2
    • 12
  • m

    Michael Smith

    05/12/2022, 7:10 PM
    Hi, on prefect 1.x could you confirm when task run state is updated in the core prefect database. I'd like to understand if we have an agent failure/restart what would happen to "in-flight" workflows - would the agent be able to process the workflow from where it left off (re-running the last task run)
    k
    • 2
    • 2
  • b

    Binoy Shah

    05/12/2022, 7:44 PM
    few Questions about Prefect 2.0 [Orion]. 1. Why is there no Helm chart for Prefect 2.0 Orion 2. How many components are then in Prefect 2.0. Orion 3. Where can I find the components and Architecture of Prefect 2.0 Orion
    k
    m
    a
    • 4
    • 20
  • t

    Tom Baldwin

    05/12/2022, 9:25 PM
    I have a self-hosted prefect server running 0.15.13 that I am trying to set up a Slack integration for via webhook. I am clicking on the "secret link" in the docs to get the webhook URL. But when I paste this into the Prefect UI and click Save, I get the error:
    Variable "$input" got invalid value null at "input.tenant_id"; Expected non-nullable type UUID! not to be null.
    I searched the community and saw some recent discussion about
    tenant_id
    in this forum related to the 1.0 release, but none in this exact context I think. Do I need to install a newer version of Prefect, or click an older version of the secret link, to make this work?
    k
    • 2
    • 22
  • c

    Cory Jez

    05/13/2022, 3:29 PM
    hi all - i am getting a pickling error on something related to SQLAlchemy when i go to register my flows to s3 (AWS prefect cloud setup). I recently tweaked some db connections (snowflake) to make them more performant; but can't find anything in SO related to this. has anyone ever seen something like this before?
    k
    • 2
    • 3
  • j

    jack

    05/14/2022, 2:45 PM
    Seeing a new pattern today running tasks on ECS. The prefect logs show
    Flow run SUCCESS: all reference tasks succeeded
    and then the next log line says
    No heartbeat detected from the remote task; marking the run as failed.
    a
    • 2
    • 20
  • y

    YD

    05/16/2022, 11:35 PM
    Hi, How to register,
    flow.register(...)
    , a docker agent ? Can I have different docker agents running on a single server?
    k
    • 2
    • 9
  • m

    Maria

    05/17/2022, 2:14 PM
    Hi, my flow of flows is returning a flow like the image attached. Is there a way to get the names of what it’s doing? e.g. sub flow’s name or task names?
    k
    • 2
    • 2
  • k

    Kayvan Shah

    05/17/2022, 4:56 PM
    I have setup docker and Prefect on remote on-premise server. How should I keep the Prefect server with docker agent always running?
    k
    • 2
    • 4
  • k

    Kayvan Shah

    05/18/2022, 6:45 AM
    Getting these errors & warning while starting the server Env: python 3.6.9
    Untitled.txt
    ✅ 1
    👀 1
    a
    l
    • 3
    • 10
  • r

    Roey Brecher

    05/18/2022, 6:59 AM
    What’s the best way to prune\cleanup old data on Prefect Server? the
    .prefect
    directory reached a size of 428G.
    ✅ 1
    a
    • 2
    • 3
  • s

    Sebastian Beschke

    05/18/2022, 6:03 PM
    EDIT It seems I was seeing something else, please disregard :) Hi! I’m not sure if this has been discussed before… I’m running Prefect Server on EC2 along with an ECS agent deployed as a fargate task. I’m seeing the issue that if the server ever goes down (or the connection is lost otherwise), the agent will show a
    requests.exceptions.ConnectionError
    , but does not exit. However it’s also does not seem to reestablish the connection once the server becomes available again. Is there some way to either 1. have the agent retry connecting some time after encountering a connection error, or 2. exit after a connection error so ECS can detect that and restart it? The way it looks now I have to manually restart the agent in this situation, which I don’t think is ideal… Am I missing something? Thanks in advance!
    ✅ 1
    k
    • 2
    • 2
  • d

    David Beck

    05/19/2022, 3:39 PM
    Hi all, I have having an issue with our logging implementation changes are not holding when registered to and running in Prefect Cloud. Essentially, the custom log handler (which is just opentelemetry-sdk) enforces a new
    _log_record_context_injector
    function to modify the names of
    PREFECT_LOG_RECORD_ATTRIBUTES
    for our purposes which is then added to the root Prefect logger. This function call occurs in a Task class I created which is set as an
    upstream_task.
    When this is run locally, all log records have the corrected attribute names. When run in Cloud, only the task run to initialize the log handler enforces the changes. Given that all tasks should inherent from the root logger, why are the changes not being implemented across the tasks/flow? Further info: we are using KubernetesRun with Docker flow storage. Also the changes hold when the implementation is run locally.
    k
    • 2
    • 9
Powered by Linen
Title
d

David Beck

05/19/2022, 3:39 PM
Hi all, I have having an issue with our logging implementation changes are not holding when registered to and running in Prefect Cloud. Essentially, the custom log handler (which is just opentelemetry-sdk) enforces a new
_log_record_context_injector
function to modify the names of
PREFECT_LOG_RECORD_ATTRIBUTES
for our purposes which is then added to the root Prefect logger. This function call occurs in a Task class I created which is set as an
upstream_task.
When this is run locally, all log records have the corrected attribute names. When run in Cloud, only the task run to initialize the log handler enforces the changes. Given that all tasks should inherent from the root logger, why are the changes not being implemented across the tasks/flow? Further info: we are using KubernetesRun with Docker flow storage. Also the changes hold when the implementation is run locally.
k

Kevin Kho

05/19/2022, 3:47 PM
So the logger is part of
context
and modifications to the
context
only live inside that task. So for example if you do:
@task
def abc():
    prefect.context.x =3
    return ..
This x does not exist in downstream tasks. Instead, the better attempt here would be to attach it outside of the tasks and Flow if possible
logger = prefect.context.get("logger")
logger.addHandler()

@task
def abc():
    ...
and then used script based storage so that it’s not serialized and it’s run during script start. Still unsure if this will work. Were you the one who asked about the
PREFECT_LOG_RECORD_ATTRIBUTES
before as well? I think those are hardcoded
d

David Beck

05/19/2022, 5:22 PM
Yes I am the one that asked about
PREFECT_LOG_RECORD_ATTRIBUTES
before. I'm basically aliasing those values which does work. I'm curious though why this wouldn't work to access the root Prefect logger for changes?
# Set up custom logger LogRecord Factory
        logging.setLogRecordFactory(_log_record_context_injector)

        # Attach OTLP handler to root Prefect logger
        logger = get_logger()
        logger.addHandler(handler)
        <http://logger.info|logger.info>("OTEL Log Handler is set up!")
the
get_logger
function should supply the root logger which is the context logger as well
Also I attempted using
logger = prefect.context.get("logger")
logger.addHandler()
before and that did not seem to work.
Also I could discuss with our team if script based storage is an option, but we are using Docker storage now and I think that figuring out the path of the flow once constructed might be fairly tricky
k

Kevin Kho

05/19/2022, 5:58 PM
I will need to ask some people to get more detailed answers. I don’t know this much detail about the logger myself. The attempt with
logger = prefect.context.get("logger")
logger.addHandler()
was that inside or outside a task?
d

David Beck

05/19/2022, 6:16 PM
Likely within a task if I remember correctly which I learned would only affect that task context. That's why I started using
get_logger
from the utility module instead. I think I should also clarify that the log handler I add to the prefect logger IS attaching. The issue I'm encountering is that the
_log_record_context_injector
function to modify the names of
PREFECT_LOG_RECORD_ATTRIBUTES
is somehow not holding. I find that curious because all tasks should be calling
get_logger(self.name)
which should yield a child logger from root. So it seems to me that the original
_log_record_context_injector
is being applied to those child loggers. Not sure why that is if the root logger has been changed
I figured out an alternative solution. On the handler I add to the root Prefect logger, I added a filter that changes the context values added to the log records. A simple enough work around. However I do think that some additional features to customize logging within Prefect would be ideal. Or for that matter, OpenTelemetry should be packaged into Prefect in the future. It's becoming an industry standard across so many platforms.
k

Kevin Kho

05/20/2022, 3:14 PM
Looks like we don’t have an existing issue to track OpenTelemetry. Would you be willing to write one in the repo? I can then direct future users to chime in there as well.
View count: 3