https://prefect.io
Join Slack
I want to ask the Dask Experts whether they can help identify a scheduler bottleneck I have experien...
v

Vincent

over 4 years ago
I want to ask the Dask Experts whether they can help identify a scheduler bottleneck I have experienced. I have a flow that schedules 22 parallel mapped jobs (each with 65 items) allowing for 1430 parallel jobs. When I schedule the job on an k8s cluster with 375 nodes (each with a worker that runs 4 threads), my scheduler throws messages out like the below. The CPU utilization of the scheduler also goes to 100%. These jobs ultimately die when all the workers get disconnected.
distributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
...
distributed.core - INFO - Event loop was unresponsive in Scheduler for 7.07s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
...
distributed.comm.tcp - INFO - Connection closed before handshake completed
...
distributed.scheduler - INFO - Close client connection: Client-worker-3c6d8642-53b5-11eb-800e-32b98c347770
When I scale the job down to 250 nodes and 3 threads per worker, I still get 100% utilization, but it is slightly more stable. where the only warning messages is
distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
Thanks for any advice!
v
m
d
  • 3
  • 21
  • 274
Using prefect, is it possible to implement tasks in different programming languages, we already hav...
h

Hari Krishna Sunkari

about 4 years ago
Using prefect, is it possible to implement tasks in different programming languages, we already have our business login in Nodejs and I couldn't find any docs related to this
h
j
k
  • 3
  • 2
  • 273
Hi all, I'm having an issue authenticating to Prefect through `prefect cloud login` . If I do browse...
m

Moe

over 1 year ago
Hi all, I'm having an issue authenticating to Prefect through
prefect cloud login
. If I do browser authentication it goes through the process, says successful and then aborts in terminal without asking me to select workspace. I also tried
prefect cloud login -k <key> -w <handle>/<workspace>
which just returns
"Unable to authenticate with Prefect Cloud. Your key is not in our expected format."
This is happening on one of my EC2 servers. Locally it is fine, I can authenticate with web browser authentication and it asks me to select workspace. Version is 2.10.21. What could be the issue? @Marvin
m
m
  • 2
  • 2
  • 271
I'm curious if there is a way to use the `prefect deploy` CLI command non-interactively? I've define...
l

Luke Orland

over 1 year ago
I'm curious if there is a way to use the
prefect deploy
CLI command non-interactively? I've defined deployments in
./prefect.yaml
and would like the deployments to be created without answering these questions interactively, e.g. in a continuous deployment setting: •
Would you like to build a custom Docker image for this deployment? [y/n] (n):
•
Would you like to save configuration for this deployment for faster deployments in the future? [y/n]:
l
n
  • 2
  • 2
  • 270
Hi, I’m trying to get a Datadog agent running in k8s to collect application metrics from my Prefect ...
d

Doug Balog

almost 4 years ago
Hi, I’m trying to get a Datadog agent running in k8s to collect application metrics from my Prefect Flows. Just wondering if anybody has this working and if they could share how they did it.
d
k
+2
  • 4
  • 16
  • 270
I'm trying to use the Scrapy framework in conjunction with Prefect 2.0, and Scrapy automatically set...
k

Kevin Grismore

about 3 years ago
I'm trying to use the Scrapy framework in conjunction with Prefect 2.0, and Scrapy automatically sets up its own core Python Loggers. When I run a flow in the terminal, I can see the logs coming from Scrapy's loggers like so:
13:37:49.758 | DEBUG   | scrapy.core.scraper - Scraped from <200 <https://quotes.toscrape.com/page/9/>>
However, those logs don't appear in the flow run logs UI. I'm guessing I need to set
PREFECT_LOGGING_EXTRA_LOGGERS
, but I'm not certain what the right way to do that is.
prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="scrapy"
seems to have no effect. I also tried
prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="scrapy.core.engine"
using the name of the specific logger I wanted to see, but that didn't work either.
✅ 1
k
a
  • 2
  • 10
  • 269
i keep getting RuntimeError: Failed to reach API at <http://127.0.0.1:4200> when i run my prefect fl...
c

cauchymike

11 months ago
i keep getting RuntimeError: Failed to reach API at http://127.0.0.1:4200 when i run my prefect flow. In my profiles.toml, i have set the PREFECT_API_URL. but i still keep getting the same error. I am currently using prefect==3.0.8
c
d
  • 2
  • 2
  • 267
<@ULVA73B9P>, show me please example of base job template for Docker work pool type, where custom en...
m

Michal

about 1 year ago
@Marvin, show me please example of base job template for Docker work pool type, where custom envvars, image and volumes are customized
m
m
  • 2
  • 12
  • 267
Hello. I'm using `subprocess` to run my dbt project locally (Prefect 1.2.2, Windows). I create my co...
m

Mansour Zayer

about 3 years ago
Hello. I'm using
subprocess
to run my dbt project locally (Prefect 1.2.2, Windows). I create my command (
dbt run --vars '{data_processing_start_date: 2022-07-20, data_processing_end_date: 2022-07-20}' --profiles-dir ./
) like this:
command = (
        f"dbt run --vars '{{"
        f"data_processing_start_date: {data_processing_start_date}, "
        f"data_processing_end_date: {data_processing_end_date}}}' --profiles-dir ./ "
    )
The command is created correctly, but dbt gives me this error
dbt: error: unrecognized arguments: 2022-07-20, data_processing_end_date: 2022-07-20}'
Seems like dbt interprets
2022-07-20
as an argument instead of the value for
data_processing_start_date
variable. Keep in mind that when I run the same command in my CLI, dbt works fine. But when it's provided to dbt through
subprocess
this occurs. This is my subprocess:
subprocess.run(
        command,
        check=True,
        stderr=True,
        stdout=True,
        shell=True,
        cwd="dbt",
    )
Any idea what might cause this, and how to solve this? Thank you
✅ 1
m
a
  • 2
  • 4
  • 266
is there a way programatically to get a list of deployments in prefect? i know we can do ```prefect ...
r

Rajan Subramanian

over 3 years ago
is there a way programatically to get a list of deployments in prefect? i know we can do
prefect deployment ls
but i want to get all my created deployments...1000 of them and run them all. so im assuming i have to programatically execute prefect deployment run deploymenet_name for each deployment in my deployments
discourse 1
r
k
  • 2
  • 5
  • 266
Previous151617Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by