Thread
#prefect-community
    Alexander Butler

    Alexander Butler

    5 months ago
    prefect deployment create
    says it should create or update a deployment but it is failing when the deployment exists (prefect 2.0) ?
    Kevin Kho

    Kevin Kho

    5 months ago
    What error do you get and what version are you on?
    Alexander Butler

    Alexander Butler

    5 months ago
    2.0b3
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/Users/alexanderbutler/Library/Caches/pypoetry/virtualenvs/analytics-pipelines-nKjHKDtD-py3.9/lib/python3.9/site-packages/prefect/cli/deployment.py", 
    line 251, in create
        await spec.create_deployment(validate=False)
      File "/Users/alexanderbutler/Library/Caches/pypoetry/virtualenvs/analytics-pipelines-nKjHKDtD-py3.9/lib/python3.9/site-packages/prefect/client.py", line 82, 
    in with_injected_client
        return await fn(*args, **kwargs)
      File "/Users/alexanderbutler/Library/Caches/pypoetry/virtualenvs/analytics-pipelines-nKjHKDtD-py3.9/lib/python3.9/site-packages/prefect/deployments.py", line
    280, in create_deployment
        self.flow_storage._block_id = await client.create_block(
      File "/Users/alexanderbutler/Library/Caches/pypoetry/virtualenvs/analytics-pipelines-nKjHKDtD-py3.9/lib/python3.9/site-packages/prefect/client.py", line 856,
    in create_block
        raise prefect.exceptions.ObjectAlreadyExists(http_exc=e) from e
    prefect.exceptions.ObjectAlreadyExists
    
    Failed to create deployment 'elt-salesforce/primary'
    prefect.exceptions.ObjectAlreadyExists
    Kevin Kho

    Kevin Kho

    5 months ago
    I’ll try to replicate one second
    Are you running against Cloud 2.0 or local orion?
    Alexander Butler

    Alexander Butler

    5 months ago
    Local
    Kevin Kho

    Kevin Kho

    5 months ago
    I can’t replicate. Could you show me the DeploymentSpec? Just omit if you have sensitive info
    Alexander Butler

    Alexander Butler

    5 months ago
    np, its very barebones so should be easy to replicate in theory
    from datetime import timedelta
    from pathlib import Path
    
    from prefect.deployments import DeploymentSpec
    from prefect.orion.schemas.schedules import IntervalSchedule, CronSchedule
    from prefect.flow_runners import SubprocessFlowRunner, DockerFlowRunner
    
    
    FLOW_DIR = Path(__file__).parent / "flows"
    
    # Salesforce ELT
    DeploymentSpec(
        flow_location=str((FLOW_DIR / "salesforce.py").absolute()),
        name="primary",
        schedule=IntervalSchedule(interval=timedelta(minutes=5)),
        tags=["pipeline"],
        flow_runner=SubprocessFlowRunner()
    )
    deployment
    from prefect import flow, task
    
    
    @task(name="extract", description="This task triggers an airbyte extraction job for Salesforce data to BigQuery.", tags=["airbyte", "extract"])
    def extract():
        print("Data moved.")
    
    @flow(name="elt-salesforce")
    def elt():
        extract()
    flow
    Kevin Kho

    Kevin Kho

    5 months ago
    Yikes your code worked for me. I’d need to ask a colleague tom since we’re at end of day here. I’ll leave a message and get that to you
    Wait what storage are you using? Local?
    Alexander Butler

    Alexander Butler

    5 months ago
    Local yes
    Kevin Kho

    Kevin Kho

    5 months ago
    Ok will leave I message. I was on Local too. I was wondering if you were on a storage that was blocking overwriting
    Alexander Butler

    Alexander Butler

    5 months ago
    Oh also the whole deployment was done via 2 stage docker so it could be a permission thing
    Kevin Kho

    Kevin Kho

    5 months ago
    Could you give me more details on that? What do you mean by 2 stage docker? 😆
    I was debugging the issue by Ryan below yours and I found how to replicate
    Alexander Butler

    Alexander Butler

    5 months ago
    Does it seem easily remediable? Curious on RC. 2 stage docker uses a builder image, then a deployment image. Actually my deployment pattern for Prefect is pretty cool. I'll share when back at PC
    Kevin Kho

    Kevin Kho

    5 months ago
    If not remediable by a setting we expose, it might be remediable by a guide how to do it. Will update you tom
    Hey @Alexander Butler, I replicated this and I tried it again in front the team and it worked. It seems this weirdness might be related to Prefect version and the database version. Resetting the database or having consistent Prefect versions might get you past this but it was able to work!
    and expected to be working