https://prefect.io logo
Powered by
Title
h

Horatiu Bota

05/16/2022, 10:35 AM
hi community! i'm trying to set up bitbucket storage (using bitbucket cloud) for my flows (working off this doc https://docs.prefect.io/api/latest/storage.html#bitbucket), but i'm getting a 404 when trying to execute the flow 
404 Client Error: Not Found for url
. I've tried various combinations of repo/project/auth credentials for Bitbucket storage, but all with the same issue - has anyone successfully set up Bitbucket storage? (using Prefect 1.1.0)
a

Anna Geller

05/16/2022, 11:43 AM
I currently don't have a Bitbucket account so a bit hard to replicate. Can you clone the repository from your agent without using Prefect? Can you share how you configured your Bitbucket storage on your flow?
keep in mind that 
project
is required, this is sort of special for Bitbucket
h

Horatiu Bota

05/16/2022, 11:45 AM
sure: 
flow.storage = Bitbucket(
    project="LEAF",
    repo="hbota/prefect-storage-test",
    path="flow.py",
    ref="main",
    cloud_username_secret="BITBUCKET_USERNAME",
    cloud_app_password_secret="BITBUCKET_APP_PASSWORD"
)
i've been trying to make requests to the bitbucket api locally and this seems to work fine (i.e., i can get the flow file using the username / app password auth method): 
url = "<https://api.bitbucket.org/2.0/repositories/{workspace}/{repo_slug}/path/to/file.py>"
url = "<https://api.bitbucket.org/2.0/repositories/hbota/prefect-storage-test/src/main/flow.py>"
a

Anna Geller

05/16/2022, 11:48 AM
what's your run config? the agent must be able to pull the repo
h

Horatiu Bota

05/16/2022, 11:48 AM
whereas i can't use the URL built by the Bitbucket storage object to get the same file: 
url = "<https://bitbucket.org/rest/api/2.0/projects/LEAF/repos/prefect-storage-test/raw/flow.py?at=main>"
i'm starting the agent with: 
prefect agent ecs start \
 --cluster arn:aws:ecs:us-east-2:478904141323:cluster/<cluster-name> \
 --env PREFECT_CONTEXT_SECRETS__BITBUCKET_USERNAME="<username>" \
 --env PREFECT_CONTEXT_SECRETS__BITBUCKET_APP_PASSWORD="<bitbucket-app-password>"
a

Anna Geller

05/16/2022, 11:51 AM
but you're on Prefect Cloud you said?
h

Horatiu Bota

05/16/2022, 11:51 AM
no, sorry, running my Prefect Server on AWS EC2
a

Anna Geller

05/16/2022, 11:53 AM
in that case, you may need extra variables, formatted as you did --env X=y: 
env:
        - name: PREFECT__CLOUD__USE_LOCAL_SECRETS
          value: true
        - name: PREFECT__CLOUD__API
          value: "<http://some_ip:4200/graphql>" # paste your GraphQL Server endpoint here
        - name: PREFECT__BACKEND
          value: server
h

Horatiu Bota

05/16/2022, 11:56 AM
thank you! but it looks like i already have those set in my 
config.toml
a

Anna Geller

05/16/2022, 11:58 AM
config.toml is relevant to register flows from your local Prefect installation; for the agent you may need to set it separately, depending on where you run this command
btw how far are you in your Prefect adoption? storage may be easier to configure with <http://v|Prefect 2.0> and Orion API server rather than Prefect Server (1.0)
h

Horatiu Bota

05/16/2022, 12:10 PM
👍 thank you, i'll take a look! but it seems there's not github/bitbucket storage yet for Prefect 2.0
looks like i needed to set the Bitbucket workspace name in the flow storage config: 
flow.storage = Bitbucket(
    project="<project_name>",
    workspace="<workspace_id>",
    repo="<repository_name>",
    path="flow.py",
    ref="main",
    cloud_username_secret="BITBUCKET_USERNAME",
    cloud_app_password_secret="BITBUCKET_APP_PASSWORD"
)
or pass 
cloud=True
when initing the Bitbucket object
👍 1
maybe this helps elsewhere 🙂
thank you for your help!
a

Anna Geller

05/16/2022, 12:24 PM
it seems there's not github/bitbucket storage yet for Prefect 2.0
that's true, but you may use any storage supported by fsspec e.g. s3, gcs, Azure blob - super easy to configure
h

Horatiu Bota

05/16/2022, 12:26 PM
👍
a

Anna Geller

05/16/2022, 12:26 PM
oh so you figured it out for 1.0 with Bitbucket? was workspace all you needed?
nice work!
maybe this helps elsewhere 🙂
for sure! thanks for sharing!
h

Horatiu Bota

05/16/2022, 12:27 PM
i'm reluctant to switch to orion because we use 1.1 in our team right now and it's unclear how much work is required to switch
a

Anna Geller

05/16/2022, 12:28 PM
sure, definitely no rush 👍
h

Horatiu Bota

05/16/2022, 12:28 PM
also, given it's in beta right now, i think switching to orion would be a hard sell to my manager 😬
a

Anna Geller

05/16/2022, 12:32 PM
that's understandable
4 Views
#prefect-communityJoin Slack
Powered by