Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
a
Alexander Butler
04/14/2022, 11:55 PMprefect deployment createk
Kevin Kho
04/14/2022, 11:55 PMWhat error do you get and what version are you on?
a
Alexander Butler
04/14/2022, 11:58 PM2.0b3The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/alexanderbutler/Library/Caches/pypoetry/virtualenvs/analytics-pipelines-nKjHKDtD-py3.9/lib/python3.9/site-packages/prefect/cli/deployment.py",
line 251, in create
await spec.create_deployment(validate=False)
File "/Users/alexanderbutler/Library/Caches/pypoetry/virtualenvs/analytics-pipelines-nKjHKDtD-py3.9/lib/python3.9/site-packages/prefect/client.py", line 82,
in with_injected_client
return await fn(*args, **kwargs)
File "/Users/alexanderbutler/Library/Caches/pypoetry/virtualenvs/analytics-pipelines-nKjHKDtD-py3.9/lib/python3.9/site-packages/prefect/deployments.py", line
280, in create_deployment
self.flow_storage._block_id = await client.create_block(
File "/Users/alexanderbutler/Library/Caches/pypoetry/virtualenvs/analytics-pipelines-nKjHKDtD-py3.9/lib/python3.9/site-packages/prefect/client.py", line 856,
in create_block
raise prefect.exceptions.ObjectAlreadyExists(http_exc=e) from e
prefect.exceptions.ObjectAlreadyExists
Failed to create deployment 'elt-salesforce/primary'k
Kevin Kho
04/15/2022, 12:02 AMI’ll try to replicate one second
Are you running against Cloud 2.0 or local orion?
a
Alexander Butler
04/15/2022, 12:13 AMLocal
k
Kevin Kho
04/15/2022, 12:14 AMI can’t replicate. Could you show me the DeploymentSpec? Just omit if you have sensitive info
a
Alexander Butler
04/15/2022, 12:14 AMnp, its very barebones so should be easy to replicate in theory
from datetime import timedelta
from pathlib import Path
from prefect.deployments import DeploymentSpec
from prefect.orion.schemas.schedules import IntervalSchedule, CronSchedule
from prefect.flow_runners import SubprocessFlowRunner, DockerFlowRunner
FLOW_DIR = Path(__file__).parent / "flows"
# Salesforce ELT
DeploymentSpec(
flow_location=str((FLOW_DIR / "salesforce.py").absolute()),
name="primary",
schedule=IntervalSchedule(interval=timedelta(minutes=5)),
tags=["pipeline"],
flow_runner=SubprocessFlowRunner()
)from prefect import flow, task
@task(name="extract", description="This task triggers an airbyte extraction job for Salesforce data to BigQuery.", tags=["airbyte", "extract"])
def extract():
print("Data moved.")
@flow(name="elt-salesforce")
def elt():
extract()k
Kevin Kho
04/15/2022, 12:17 AMYikes your code worked for me. I’d need to ask a colleague tom since we’re at end of day here. I’ll leave a message and get that to you
Wait what storage are you using? Local?
a
Alexander Butler
04/15/2022, 12:20 AMLocal yes
k
Kevin Kho
04/15/2022, 12:21 AMOk will leave I message. I was on Local too. I was wondering if you were on a storage that was blocking overwriting
a
Alexander Butler
04/15/2022, 12:28 AMOh also the whole deployment was done via 2 stage docker so it could be a permission thing
k
Kevin Kho
04/15/2022, 12:29 AMCould you give me more details on that? What do you mean by 2 stage docker? 😆
I was debugging the issue by Ryan below yours and I found how to replicate
a
Alexander Butler
04/15/2022, 1:21 AMDoes it seem easily remediable? Curious on RC.
2 stage docker uses a builder image, then a deployment image. Actually my deployment pattern for Prefect is pretty cool. I'll share when back at PC
k
Kevin Kho
04/15/2022, 1:45 AMIf not remediable by a setting we expose, it might be remediable by a guide how to do it. Will update you tom
Hey @Alexander Butler, I replicated this and I tried it again in front the team and it worked. It seems this weirdness might be related to Prefect version and the database version. Resetting the database or having consistent Prefect versions might get you past this but it was able to work!
and expected to be working