https://prefect.io
Join Slack
hi community! i'm trying to set up bitbucket storage (using bitbucket cloud) for my flows (working o...
# ask-community
h
hi community! i'm trying to set up bitbucket storage (using bitbucket cloud) for my flows (working off this doc https://docs.prefect.io/api/latest/storage.html#bitbucket), but i'm getting a 404 when trying to execute the flow 
404 Client Error: Not Found for url
. I've tried various combinations of repo/project/auth credentials for Bitbucket storage, but all with the same issue - has anyone successfully set up Bitbucket storage? (using Prefect 1.1.0)
a
I currently don't have a Bitbucket account so a bit hard to replicate. Can you clone the repository from your agent without using Prefect? Can you share how you configured your Bitbucket storage on your flow?
keep in mind that 
project
is required, this is sort of special for Bitbucket
h
sure:
Copy code
flow.storage = Bitbucket(
    project="LEAF",
    repo="hbota/prefect-storage-test",
    path="flow.py",
    ref="main",
    cloud_username_secret="BITBUCKET_USERNAME",
    cloud_app_password_secret="BITBUCKET_APP_PASSWORD"
)
i've been trying to make requests to the bitbucket api locally and this seems to work fine (i.e., i can get the flow file using the username / app password auth method):
Copy code
url = "<https://api.bitbucket.org/2.0/repositories/{workspace}/{repo_slug}/path/to/file.py>"
url = "<https://api.bitbucket.org/2.0/repositories/hbota/prefect-storage-test/src/main/flow.py>"
a
what's your run config? the agent must be able to pull the repo
h
whereas i can't use the URL built by the Bitbucket storage object to get the same file:
Copy code
url = "<https://bitbucket.org/rest/api/2.0/projects/LEAF/repos/prefect-storage-test/raw/flow.py?at=main>"
i'm starting the agent with:
Copy code
prefect agent ecs start \
 --cluster arn:aws:ecs:us-east-2:478904141323:cluster/<cluster-name> \
 --env PREFECT_CONTEXT_SECRETS__BITBUCKET_USERNAME="<username>" \
 --env PREFECT_CONTEXT_SECRETS__BITBUCKET_APP_PASSWORD="<bitbucket-app-password>"
a
but you're on Prefect Cloud you said?
h
no, sorry, running my Prefect Server on AWS EC2
a
in that case, you may need extra variables, formatted as you did --env X=y:
Copy code
env:
        - name: PREFECT__CLOUD__USE_LOCAL_SECRETS
          value: true
        - name: PREFECT__CLOUD__API
          value: "<http://some_ip:4200/graphql>" # paste your GraphQL Server endpoint here
        - name: PREFECT__BACKEND
          value: server
h
thank you! but it looks like i already have those set in my 
config.toml
a
config.toml is relevant to register flows from your local Prefect installation; for the agent you may need to set it separately, depending on where you run this command
btw how far are you in your Prefect adoption? storage may be easier to configure with <http://v|Prefect 2.0> and Orion API server rather than Prefect Server (1.0)
h
👍 thank you, i'll take a look! but it seems there's not github/bitbucket storage yet for Prefect 2.0
looks like i needed to set the Bitbucket workspace name in the flow storage config:
Copy code
flow.storage = Bitbucket(
    project="<project_name>",
    workspace="<workspace_id>",
    repo="<repository_name>",
    path="flow.py",
    ref="main",
    cloud_username_secret="BITBUCKET_USERNAME",
    cloud_app_password_secret="BITBUCKET_APP_PASSWORD"
)
or pass 
cloud=True
when initing the Bitbucket object
👍 1
maybe this helps elsewhere 🙂
thank you for your help!
a
it seems there's not github/bitbucket storage yet for Prefect 2.0
that's true, but you may use any storage supported by fsspec e.g. s3, gcs, Azure blob - super easy to configure
h
👍
a
oh so you figured it out for 1.0 with Bitbucket? was workspace all you needed?
nice work!
maybe this helps elsewhere 🙂
for sure! thanks for sharing!
h
i'm reluctant to switch to orion because we use 1.1 in our team right now and it's unclear how much work is required to switch
a
sure, definitely no rush 👍
h
also, given it's in beta right now, i think switching to orion would be a hard sell to my manager 😬
a
that's understandable
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by