Carlos Paiva
12/01/2021, 4:27 PMFailed to load and execute Flow's environment: ValueError('Flow is not contained in this Storage')
My setup is on a AWS EC2 machine and I am running the Flow with a Docker Agent, as per below.
from prefect import Parameter, Flow, task
from prefect.storage import GitHub
from prefect.run_configs.docker import DockerRun
@task(log_stdout=True)
def test(e):
return e.get("body")
storage = GitHub(repo=repo,
path="flows/flow.py",
access_token_secret="GITHUB_ACCESS_TOKEN")
run_config = DockerRun(labels=["L1", "L2", "L3"],
env={"EXTRA_PIP_PACKAGES": "zcrmsdk zohocrm-prefect-tasks"})
with Flow(name="Name", storage=storage, run_config=run_config) as flow:
# Pipeline parameters
event = Parameter('event', required=True)
test(event)
Any idea about what I am doing wrong?Tom Shaffner
12/01/2021, 4:37 PMAleksandr Liadov
12/01/2021, 4:38 PMrun config
in flow (I do the local run with the local agent):
run_config=UniversalRun(env={
"PREFECT__LOGGING__EXTRA_LOGGERS": '["lib1", "lib2"]',
"PREFECT__LOGGING__LEVEL": "DEBUG",
})
I can ONLY observe info
level from my libs even if I try to switch on debug
level!
(However the prefect logger shows the information related to CloudFlowRunner
from debug
level without any problem).
What do I wrong?Jason Motley
12/01/2021, 5:07 PM@task(name = "Renamed Transform", log_stdout=True, checkpoint=False)
, can I set upstream_task = ["Renamed Transform"]) in a downstream task?Thomas Furmston
12/01/2021, 6:59 PMThomas Furmston
12/01/2021, 6:59 PMThomas Furmston
12/01/2021, 7:00 PMA -> B -> C -> D
. Now suppose I run the flow and tasks A
and B
complete, but C
fails because of a bug. I therefore want to fix the code in task C and redeploy the flow.Thomas Furmston
12/01/2021, 7:00 PMA
& B
are expensive to run, so I would rather not re-run them from scratch. Is it possible to re-run the flow with the newly deployed code from task C
onwards?Thomas Furmston
12/01/2021, 7:01 PMTilak Maddy
12/01/2021, 8:48 PMprefect auth login --key <MY_KEY>
even though my key was valid and authorized. I created so many keys later on still didn't work ....and in the end it turned out that I had already logged in with another key couple days ago. So i had to run prefect auth logout
before logging in with the new key .Tom Shaffner
12/01/2021, 8:55 PMprefect server config
to generate a docker compose file, and I'm partway down the path of trying to set that up to auto-start (still working through https://github.com/flavienbwk/prefect-docker-compose, which was linked in it), but so far it's not working and it seems like there should be a simpler answer here. Docker has restart policies already (https://docs.docker.com/config/containers/start-containers-automatically/), is there not some flag I can pass to prefect server start
to use them?Jason Motley
12/01/2021, 9:37 PMflow = Flow("myflow")
with flow:
df = extract(stuff)
print(df.head(10)) # This does not work
volkerjaenisch
12/01/2021, 10:54 PMhttps://upload.wikimedia.org/wikipedia/commons/8/89/Kozloduy_Nuclear_Power_Plant_-_Control_Room_of_Unit_5.jpg▾
brian
12/01/2021, 11:03 PMWilhelm Su
12/01/2021, 11:17 PMArun Giridharan
12/02/2021, 1:04 AM张强
12/02/2021, 6:20 AMBruno Murino
12/02/2021, 9:43 AMTilak Maddy
12/02/2021, 10:29 AMZheng Xie
12/02/2021, 10:52 AMprefect register --project yourproject -p yourflow.py
It seems that perfect creates a tmp{randomNumber}
folder and puts healthcheck.py and a Dockerfile, which participate the docker build.
But I accidentally deleted the tmp
folder, even after I recovered it, it still throws the following error. What happened was that Prefect expected a tmp
folder with a new random_number as the suffix. How can I recover from it?
https://stackoverflow.com/questions/70198040/error-in-prefect-when-build-docker-image-what-triggers-prefect-to-create-the-tmChris L.
12/02/2021, 11:01 AMClientError
when using Prefect Cloud for a mapped tasks. The inputs to each task are identical, but some mapped tasks are successful while a few raise DeprecationWarning: Using 'method_whitelist' with Retry is deprecated and will be removed in v2.0. Use 'allowed_methods' instead
(see screenshot for trace). Has anybody experienced something similar? I'm using urllib3 v1.26.7 and prefect
v0.15.9Aqib Fayyaz
12/02/2021, 11:33 AMhelm repo add prefecthq <https://prefecthq.github.io/server/>
i have cloned the repo
<https://github.com/PrefectHQ/server/tree/master/helm/prefect-server>
and so that i have this helm chart prefect-server locally and when i do helm install demo1 prefect-server i get the following error
Error: INSTALLATION FAILED: found in Chart.yaml, but missing in charts/ directory: postgresql
Tilak Maddy
12/02/2021, 11:53 AMflow.storage=Local()
to my Hello world flow ? Everything seems to be the samePinakpani Mukherjee
12/02/2021, 12:07 PMjcozar
12/02/2021, 1:01 PMArnon Kimhi
12/02/2021, 1:25 PMArnon Kimhi
12/02/2021, 1:27 PMif new_state.is_successful():
flow_run_id = prefect.context.flow_run_id
flow_run_view = FlowRunView.from_flow_run_id(flow_run_id)
b_task_run = flow_run_view.get_task_run("b-1")
Arnon Kimhi
12/02/2021, 1:28 PMJacob Blanco
12/02/2021, 1:30 PM{
task_run(
where: {_and: [{state: {_eq: "Running"}}, {flow_run: {state: {_neq: "Running"}}}]}
) {
id
start_time
}
}
Maybe unrelated but for tasks which have a lot of runs, the dashboard seems to hit some kind of query limit and crashes when you try to see the list of Task Runs. The stuck ghost tasks don’t always have a lot of task runs related to them.Tom Klein
12/02/2021, 2:20 PMPrefect
for the various benefits it could offer
i saw that there's support for "docker" and "kubernetes" tasks but since my DevOps knowledge is kind of limited i was wondering if (by any chance) there's some examples of that kind of usage laying around somewhere or if you could at least give your thoughts about whether what I'm thinking even makes sense?Tom Klein
12/02/2021, 2:20 PMPrefect
for the various benefits it could offer
i saw that there's support for "docker" and "kubernetes" tasks but since my DevOps knowledge is kind of limited i was wondering if (by any chance) there's some examples of that kind of usage laying around somewhere or if you could at least give your thoughts about whether what I'm thinking even makes sense?alex
12/02/2021, 2:25 PMStartContainer
task to run a container from a pre-built Docker image.Anna Geller
12/02/2021, 2:26 PMRunNamespacedJob
task and here are two examples of how it can be used: https://github.com/anna-geller/packaging-prefect-flows/tree/master/flows_task_libraryTom Klein
12/02/2021, 2:39 PMPrefect
but i'm fine with the build-process being handled by a CI/CD system unrelated to Docker and wired into our GitHub accountKevin Kho
12/02/2021, 3:02 PMTom Klein
12/02/2021, 3:07 PMCodeFresh
) which i'm not really involved in, i.e. - our DevOps (manually) wire some github repo to be built on every commit, and automatically be pushed into ECR upon success
They then (manually) create (upon request from us) some K8s job (or long-running service, or whatever) based on said image (this is done once per job/service/etc. , as part of its initial setup)
and then when we "deploy" (using our deploy dashboard) - the latest image is taken from ECR and pushed to k8s using their DevOps magic
at which point is the "ahead of time" you're referring to, relative to this process?Kevin Kho
12/02/2021, 3:09 PMTom Klein
12/02/2021, 3:10 PMgit clone
, docker build
,push to ECR
) that somehow "sends this" (this being the image path) somewhere that Prefect will know of, right?Kevin Kho
12/02/2021, 3:12 PMTom Klein
12/02/2021, 3:13 PMKevin Kho
12/02/2021, 3:14 PMTom Klein
12/02/2021, 3:15 PMKevin Kho
12/02/2021, 3:22 PMTom Klein
12/02/2021, 3:30 PMAnna Geller
12/02/2021, 3:59 PMsteps:
- run: pip install prefect
- run: prefect auth login --key $PREFECT_API_KEY
- run: export PREFECT__CLOUD__USE_LOCAL_SECRETS=false && prefect kv set YOUR_FLOW_ECR_IMAGE_URL "<http://XXXX.dkr.ecr.us-east-1.amazonaws.com/image_name:tag|XXXX.dkr.ecr.us-east-1.amazonaws.com/image_name:tag>"
Tom Klein
12/02/2021, 6:08 PMAnna Geller
12/02/2021, 6:11 PMTom Klein
12/02/2021, 6:13 PMwhich
things run, or when
or in what
order, but not where
or how
...Anna Geller
12/02/2021, 6:15 PMTom Klein
12/02/2021, 6:18 PMservice-name
to its ECR image, so i guess we can talk to our DevOps to make it so we utilize the same mapping for Prefect...
and then we'll just need to know the name/type of the container we're interested in, and not the actual ECR pathKevin Kho
12/02/2021, 6:51 PM