Roman Jendrusch
06/20/2022, 3:41 PMOfek Katriel
06/20/2022, 8:48 PMbuild-and-deploy-production:
stage: build
# image: prefecthq/prefect:latest-python3.9
variables:
DOCKER_HOST: <tcp://localhost:2375>
DOCKER_TLS_CERTDIR: ""
# image: python:3.9.13-buster
image: google/cloud-sdk
services:
- docker:19.03.1-dind
script:
- pip3 install prefect
- echo $PREFECT_PRODUCTION_SERVICE_ACCOUNT > /tmp/$CI_PIPELINE_ID.json
- gcloud auth activate-service-account --key-file /tmp/$CI_PIPELINE_ID.json
- gcloud config set project my project
# - gcloud components install docker-credential-gcr
- gcloud auth configure-docker --quiet
- prefect auth login -k mytoken
- cd ./asin2vec_pipeline/asin2vec
- pip3 install -r requirements.txt
- export GOOGLE_APPLICATION_CREDENTIALS=/tmp/$CI_PIPELINE_ID.json
- prefect register --project "onboarding" --path flow.py --name "asin2vec_v2"
as you can see, tried with images: prefecthq/prefect:latest-python3.9, cloudsdk, and python.
with all had the same error: InterruptedError: denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/myproject/locations/us-central1/repositories/data-pipelines" (or it may not exist)
to be clear, locally it works, when I authenticate with this service account.Ankur Sheth
06/20/2022, 9:17 PMJack Sundberg
06/20/2022, 11:44 PMGol Bahar
06/21/2022, 2:14 AMyu zeng
06/21/2022, 3:08 AMApostolos Papafragkakis
06/21/2022, 10:56 AMJessica Smith
06/21/2022, 2:04 PMSlackbot
06/21/2022, 2:35 PMHoratiu Bota
06/21/2022, 3:10 PMMansour Zayer
06/21/2022, 3:45 PMWei Mei
06/21/2022, 4:27 PMAlfred Martinez
06/21/2022, 5:05 PMibrahem
06/21/2022, 9:31 PMobject is not iterable
is there anyway to return the result of a task as a normal python object list, dict and not a task (please note i don't know the length of list and its very dynamic)Josh
06/21/2022, 10:16 PMUnexpected error: KilledWorker('{{TaskRunName}}', <WorkerState '<tcp://127.0.0.1:44295>', name: 1, status: closed, memory: 0, processing: 72>)
Any idea what might be causing this?Kyle McChesney
06/21/2022, 10:42 PM@task
def required():
pass
@task
def optional():
pass
@task
def final():
pass
with Flow('example'):
res = required()
with case(some_case, True):
optional_res = optional()
final(res, upstream_tasks=[optional_res])
If optional is skipped, its okay to run final. If required fails, or optional fails, I dont want to run finalKyle McChesney
06/21/2022, 10:42 PMChris Goddard
06/22/2022, 1:58 AMBen Muller
06/22/2022, 2:01 AMau_harness_win_place = CronClock(
cron="33 * * * *",
parameter_defaults={
"race_types": ["Harness"],
"market_type_codes": ["WIN", "PLACE"],
"event_type_ids": None,
"market_countries": ["AU"],
"table_name": "harness_market_catalogue_au"
},
start_date=datetime.now(tz=timezone("Australia/Brisbane")),
)
that is called like so
with Flow(
name="betfair_market_catalogues",
storage=Storage.in_a_s3_bucket(),
run_config=RunConfig().fargate_on_ecs(),
schedule=Schedule(clocks=[au_nz_greys_win_place, au_harness_win_place]),
executor=LocalDaskExecutor(scheduler="threads"),
) as flow:
If I want to call this via the cli it doesnt like the None values and the python objects, do I have to accept stringified versions of this or does prefect handle this on its own?
eg, how would I run this locally
prefect run -p flows/market_catalogues/betfair_market_catalogues.py --param table_name="harness_market_catalogue_au" --param race_types=["Harness"] --param market_type_codes=["WIN", "PLACE"] --param event_type_ids=[] --param market_countries=["AU"]
doesnt do the jobRussell Brooks
06/22/2022, 7:08 AMyair friedman
06/22/2022, 11:27 AMGeorge Shishorin
06/22/2022, 12:20 PMSuresh R
06/22/2022, 12:23 PMMichael Hadorn
06/22/2022, 12:27 PMToby Rahloff
06/22/2022, 12:55 PMAditi Bhalawat
06/22/2022, 1:16 PMJason White
06/22/2022, 1:34 PMThe Prefect 2.0 beta period is expected to last for at least the second quarter of 2022.
And that 2.0 is not recommended for production at this time. Is this still accurate?
For context, we are looking to build an internal tool using prefect as the orchestrator and are debating whether to start with 2.0 or use 1.0 and migrate. We would be hoping to start using the tool Q3 this year.Abhishek Mitra
06/22/2022, 1:45 PMSubmitted for execution: Task arn:
Can anybody suggest a solution?Zheng Cheng
06/22/2022, 2:00 PMZheng Cheng
06/22/2022, 2:01 PMZheng Cheng
06/22/2022, 2:01 PMKevin Kho
06/22/2022, 2:07 PM