Oscar Krantz
06/17/2022, 1:33 PMHalvar Trøyel Nerbø
06/17/2022, 2:02 PMXavier Babu
06/17/2022, 2:11 PMJosh Paulin
06/17/2022, 2:42 PMPREFECT__CLOUD__USE_LOCAL_SECRETS
and PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN
set on my agent, but not on the job. Trying to run the flow just errors out at
Failed to load and execute flow run: KeyError('The secret GITLAB_ACCESS_TOKEN was not found. Please ensure that it was set correctly in your tenant: <https://docs.prefect.io/orchestration/concepts/secrets.html>')
Florian Guily
06/17/2022, 3:07 PMAlfred Martinez
06/17/2022, 3:32 PMBenny Warlick
06/17/2022, 3:41 PMjack
06/17/2022, 5:23 PMChristian Nuss
06/17/2022, 5:38 PMDeepak Pilligundla
06/17/2022, 7:03 PMHemabh Kamboj
06/17/2022, 9:15 PMprefect_flow_run_id
and prefect_flow_version
and inserting those params in the dbMatt Alhonte
06/17/2022, 10:27 PMWilliam Jamir
06/18/2022, 7:31 AMupstream_tasks
Does someone know what I’m doing wrong?
My intention is to execute task_3
only after task_1
and `task_2`is finished (since it depends on the output of them) and start task_4
only after the task_3
is finished
How can I accomplish that?
(Code and output error in thread)Daniel
06/19/2022, 3:44 AMprefecthq/prefect:latest-python3.10
docker image with some additional packages including the snowflake python connector and it's dependencies. Flows without snowflake interactions work perfectly when run from Prefect cloud however my EL flows which insert data into snowflake tables fail to load and execute returning >ModuleNotFoundError("No module named 'snowflake'").
Haven't been able to fine any similar reports among the community so wondering if someone could suggest what I may have done wrong.
With thanks,
Daniel.Xavier Witdouck
06/19/2022, 4:45 PMmarque
06/20/2022, 3:44 AMprefect.exceptions.ClientError: [{'path': ['create_project'], 'message': 'Uniqueness violation.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
when I ran prefect delete project
--> prefect create project
(using Prefect v1.0, with Prefect Cloud as backend). Can't seem to recreate a project with the same nameBharadwaj Yadati
06/20/2022, 5:41 AMTarek
06/20/2022, 8:23 AMMichal Zawadzki
06/20/2022, 10:39 AMdev_queue
with the label dev
and supporting all flow runners. It's the only work queue I have set up.
2. I ran an agent with prefect agent start dev_queue
3. I created a deployment with the tag `dev`:
name: test_platform_flow_first_deployment
flow_name: Data Platform Demo
flow_location: ./test_platform_flow.py
parameters:
to_print: "Hello from first deployment!"
tags:
- dev
I verified in the UI that it has the right tag and flow runner.
However, when I run the deployment, the flow run is never picked up.
One suspicious thing I noticed is that the flow run doesn't inherit the dev
tag from the deployment (although I don't know if it's a bug or a feature).
When I remove all labels from the work queue, the flow runs are picked up correctly.Michal Zawadzki
06/20/2022, 10:52 AMflow_runner
in the deployment? I get `ValueError: Unregistered flow runner 'DockerFlowRunner'`when running prefect deployment create my_deployment.yaml
. My deployment looks like this:
name: test_platform_flow_first_deployment
flow_name: Data Platform Demo
flow_location: ./test_platform_flow.py
parameters:
to_print: "Hello from first deployment!"
tags:
- dev
flow_runner:
type: DockerFlowRunner
config:
image: viadot:orion
Unfortunately the flow_runner
config is not documented anywhere so it's hard for me to say if I'm specifying it incorrectly or it's not supported at all.Roman Jendrusch
06/20/2022, 3:41 PMOfek Katriel
06/20/2022, 8:48 PMbuild-and-deploy-production:
stage: build
# image: prefecthq/prefect:latest-python3.9
variables:
DOCKER_HOST: <tcp://localhost:2375>
DOCKER_TLS_CERTDIR: ""
# image: python:3.9.13-buster
image: google/cloud-sdk
services:
- docker:19.03.1-dind
script:
- pip3 install prefect
- echo $PREFECT_PRODUCTION_SERVICE_ACCOUNT > /tmp/$CI_PIPELINE_ID.json
- gcloud auth activate-service-account --key-file /tmp/$CI_PIPELINE_ID.json
- gcloud config set project my project
# - gcloud components install docker-credential-gcr
- gcloud auth configure-docker --quiet
- prefect auth login -k mytoken
- cd ./asin2vec_pipeline/asin2vec
- pip3 install -r requirements.txt
- export GOOGLE_APPLICATION_CREDENTIALS=/tmp/$CI_PIPELINE_ID.json
- prefect register --project "onboarding" --path flow.py --name "asin2vec_v2"
as you can see, tried with images: prefecthq/prefect:latest-python3.9, cloudsdk, and python.
with all had the same error: InterruptedError: denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/myproject/locations/us-central1/repositories/data-pipelines" (or it may not exist)
to be clear, locally it works, when I authenticate with this service account.Ankur Sheth
06/20/2022, 9:17 PMJack Sundberg
06/20/2022, 11:44 PMGol Bahar
06/21/2022, 2:14 AMyu zeng
06/21/2022, 3:08 AMApostolos Papafragkakis
06/21/2022, 10:56 AMJessica Smith
06/21/2022, 2:04 PMSlackbot
06/21/2022, 2:35 PMHoratiu Bota
06/21/2022, 3:10 PM