https://prefect.io logo
Channels
Powered by
#prefect-community
Title
# prefect-community
a

Alexander Butler

04/14/2022, 11:55 PM
prefect deployment create
says it should create or update a deployment but it is failing when the deployment exists (prefect 2.0) ?
k

Kevin Kho

04/14/2022, 11:55 PM
What error do you get and what version are you on?
a

Alexander Butler

04/14/2022, 11:58 PM
2.0b3
Copy code
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/alexanderbutler/Library/Caches/pypoetry/virtualenvs/analytics-pipelines-nKjHKDtD-py3.9/lib/python3.9/site-packages/prefect/cli/deployment.py", 
line 251, in create
    await spec.create_deployment(validate=False)
  File "/Users/alexanderbutler/Library/Caches/pypoetry/virtualenvs/analytics-pipelines-nKjHKDtD-py3.9/lib/python3.9/site-packages/prefect/client.py", line 82, 
in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/alexanderbutler/Library/Caches/pypoetry/virtualenvs/analytics-pipelines-nKjHKDtD-py3.9/lib/python3.9/site-packages/prefect/deployments.py", line
280, in create_deployment
    self.flow_storage._block_id = await client.create_block(
  File "/Users/alexanderbutler/Library/Caches/pypoetry/virtualenvs/analytics-pipelines-nKjHKDtD-py3.9/lib/python3.9/site-packages/prefect/client.py", line 856,
in create_block
    raise prefect.exceptions.ObjectAlreadyExists(http_exc=e) from e
prefect.exceptions.ObjectAlreadyExists

Failed to create deployment 'elt-salesforce/primary'
prefect.exceptions.ObjectAlreadyExists
k

Kevin Kho

04/15/2022, 12:02 AM
I’ll try to replicate one second
Are you running against Cloud 2.0 or local orion?
a

Alexander Butler

04/15/2022, 12:13 AM
Local
k

Kevin Kho

04/15/2022, 12:14 AM
I can’t replicate. Could you show me the DeploymentSpec? Just omit if you have sensitive info
a

Alexander Butler

04/15/2022, 12:14 AM
np, its very barebones so should be easy to replicate in theory
Copy code
from datetime import timedelta
from pathlib import Path

from prefect.deployments import DeploymentSpec
from prefect.orion.schemas.schedules import IntervalSchedule, CronSchedule
from prefect.flow_runners import SubprocessFlowRunner, DockerFlowRunner


FLOW_DIR = Path(__file__).parent / "flows"

# Salesforce ELT
DeploymentSpec(
    flow_location=str((FLOW_DIR / "salesforce.py").absolute()),
    name="primary",
    schedule=IntervalSchedule(interval=timedelta(minutes=5)),
    tags=["pipeline"],
    flow_runner=SubprocessFlowRunner()
)
deployment
Copy code
from prefect import flow, task


@task(name="extract", description="This task triggers an airbyte extraction job for Salesforce data to BigQuery.", tags=["airbyte", "extract"])
def extract():
    print("Data moved.")

@flow(name="elt-salesforce")
def elt():
    extract()
flow
k

Kevin Kho

04/15/2022, 12:17 AM
Yikes your code worked for me. I’d need to ask a colleague tom since we’re at end of day here. I’ll leave a message and get that to you
Wait what storage are you using? Local?
a

Alexander Butler

04/15/2022, 12:20 AM
Local yes
k

Kevin Kho

04/15/2022, 12:21 AM
Ok will leave I message. I was on Local too. I was wondering if you were on a storage that was blocking overwriting
a

Alexander Butler

04/15/2022, 12:28 AM
Oh also the whole deployment was done via 2 stage docker so it could be a permission thing
k

Kevin Kho

04/15/2022, 12:29 AM
Could you give me more details on that? What do you mean by 2 stage docker? 😆
I was debugging the issue by Ryan below yours and I found how to replicate
a

Alexander Butler

04/15/2022, 1:21 AM
Does it seem easily remediable? Curious on RC. 2 stage docker uses a builder image, then a deployment image. Actually my deployment pattern for Prefect is pretty cool. I'll share when back at PC
k

Kevin Kho

04/15/2022, 1:45 AM
If not remediable by a setting we expose, it might be remediable by a guide how to do it. Will update you tom
Hey @Alexander Butler, I replicated this and I tried it again in front the team and it worked. It seems this weirdness might be related to Prefect version and the database version. Resetting the database or having consistent Prefect versions might get you past this but it was able to work!
and expected to be working
26 Views
Powered by