Jeremiah
Nakul Gowdra
09/30/2020, 3:23 AMEmma Willemsma
10/01/2020, 4:02 PMgunarevuri
10/06/2020, 6:14 AMclient.create_project("my_first_flow")
to create it." And here is my basic code from prefect import task, Flow
@task
def hello_world():
return "hello prefect"
@task
def prefect_say(s:str):
print(s)
with Flow("my_first_flow") as f:
r = hello_world()
s = prefect_say(r)
# f.run()
f.visualize()
f.register("my_first_flow")gunarevuri
10/06/2020, 6:14 AMVipul
10/06/2020, 4:15 PMKfir Stri
10/22/2020, 7:18 AMKfir Stri
10/22/2020, 7:20 AMMichael Kennely
10/22/2020, 9:07 PM.py
file locally to test that it works as intended before pushing to GH without changing any code? From my understanding you shouldn’t have flow.run()
in your file when using Prefect cloud, but that’s how I started off running things locally
Simple hello world script for reference
import prefect
from prefect import task, Flow
from prefect.environments.storage import GitHub
@task
def hello_task():
logger = prefect.context.get("logger")
<http://logger.info|logger.info>("Hello world!")
hello_flow = Flow("gh_storage_flow", tasks=[hello_task])
hello_flow.storage = GitHub(
repo="{redacted}",
path="{redacted}",
secrets=["{redacted}"]
)
# Basically, I want the functionality of the below line so I can run this locally via Pycharm and then push to GH, merge to main branch, register, etc, and schedule via the UI to execute in our dedicated execution environment, but when you aren't running this as a script my understanding is that you need to remove the following line
hello_flow.run()
Zach
10/23/2020, 4:02 PMZach
10/23/2020, 4:03 PMSteven Hamblin
11/10/2020, 1:33 PM.info
log, but not the debug
log in this snippet? (Using the logging example from the docs here: https://docs.prefect.io/core/concepts/logging.html)
@task
def my_task(data):
from prefect import context
logger = context.get("logger")
<http://logger.info|logger.info>("info log inside mytask")
logger.debug("debug log inside mytask")
Mitchell Bregman
11/13/2020, 4:09 PMZach
11/17/2020, 3:40 PMAlexander
11/23/2020, 1:33 PMupdated
field. So does that mean that after upgrade all my flows runs two week old got updated for some reason?Andrew Hannigan
11/25/2020, 6:50 PMNewskooler
11/26/2020, 8:56 PMKevin Weiler
12/08/2020, 4:20 PMjack
12/10/2020, 8:08 PMjack
12/10/2020, 9:19 PMMichael Kennely
12/17/2020, 8:10 PMping source system for dataset -> send dataset to pandas DF -> store dataset as a file in S3 -> push contents of said file to our datastore
I’m using the LocalDaskExecutor
to do DFE and that’s helped a ton as the flow used to fail at the first step because it was doing Breadth first and holding all of the datasets in memory. I expected that once the task tree was completed for each dataset that it’d release that memory but that doesn’t seem to be the case. Am I doing something wrong/poorly or is there a way to free up that memory once the terminal task has completed for each branch?Joseph
12/17/2020, 10:28 PMSagun Garg
12/21/2020, 7:27 AM$ prefect agent kubernetes install -t <MY_TOKEN> --rbac | kubectl apply -f -
Warning Unhealthy kubelet Liveness probe failed :8080/api/health
Logs of the failing pod made to restart by K8s
Warning LoggingDisabled 48m fargate-scheduler Disabled logging because aws-logging configmap was not found. configmap "aws-logging" not found
Normal Scheduled 48m fargate-scheduler Successfully assigned default/prefect-agent-7bcdb4f975-pftnw to fargate-ip-10-0-231-193.ap-southeast-1.compute.internal
Normal Created 43m (x4 over 47m) kubelet Created container agent
Normal Started 43m (x4 over 47m) kubelet Started container agent
Normal Killing 42m (x2 over 45m) kubelet Container agent failed liveness probe, will be restarted
Normal Pulling 41m (x5 over 48m) kubelet Pulling image "prefecthq/prefect:0.14.0-python3.6"
Normal Pulled 41m (x5 over 47m) kubelet Successfully pulled image "prefecthq/prefect:0.14.0-python3.6"
Warning Unhealthy 12m (x13 over 46m) kubelet Liveness probe failed: Get <http://10.0.231.193:8080/api/health>: dial tcp 10.0.231.193:8080: connect: connection refused
Warning BackOff 3m3s (x113 over 43m) kubelet Back-off restarting failed container
Pedro Martins
12/23/2020, 2:35 PMMinakshi
01/01/2021, 6:53 AMMinakshi
01/12/2021, 9:05 AMprefect auth create-token -n my-runner-token -s RUNNER
I am getting an error "Error: Auth commands with server are not currently supported."
Also, what is the easiest way to run a sample flow on prefect cloud? I have this flow which i am running using repl.it https://repl.it/@MinakshiChandra/PrefectDemo#main.py and trying to see this flow on prefect cloudVincent
01/13/2021, 3:08 PMRuslan
01/14/2021, 7:52 PMSonny
01/16/2021, 4:53 AMAlex Rud
01/16/2021, 5:31 AM