https://prefect.io
Join Slack
Hello, I am wondering if anyone has an example of getting a flow’s ID using the flow name and proje...
# ask-community
m
Hello, I am wondering if anyone has an example of getting a flow’s ID using the flow name and project name to execute a pre-registered flow from within a web loop? I’m not entirely sure on the best approach and how to do this: Option 1: if there is a route that I can make a GET request too, how would I find that? (been searching the repo but no luck yet) Option 2: I would write this as a graphql query. How would I know what to write? Thanks - any help is much appreciated!
k
Hey @Matthew Millendorf! The query is something like this:
Copy code
query {
  flow (where: {project: {name:{_eq: "testing-tasks"}}, name:{_eq:"snowflake-test"}}){
    name
    id
    project {
      id
      name
    }
  }
}
m
Thanks @Kevin Kho !
k
The request with Python looks something like this:
Copy code
query = """query {
  flow (where: {project: {name:{_eq: "testing-tasks"}}, name:{_eq:"snowflake-test"}}){
    name
    id
    project {
      id
      name
    }
  }
}"""

url = '<https://api.prefect.io>'
r = requests.get(url, params={'query': query}, headers={"authorization": "Bearer insert_token_here"})
print(r.status_code)
print(r.text)
m
HI @Kevin Kho I’m getting 403 and ive tried:
Copy code
Bearer <api_key>
<api_key>

Bearer <service account key>
<service account key>
any ideas?
AH I see token is not the same as API Key (not a web developer whatsoever here). How do I figure out my token?
k
Hey sorry was wrapped up. I just got an api token from the service account and this worked. Have you moved to service accounts?
m
Mm not entirely sure - I created the a service account token following these steps. https://docs.prefect.io/orchestration/tutorial/overview.html#create-a-service-account-key
k
I think that might be for the agent. Could you try going to Prefect Cloud, going to the Team tab, going to Service Accounts, then creating a service account there and an API token. That should give you a token with access for the graphQL endpoint
m
Still getting a 403 - Current session has been terminated
k
Are you using Python requests?
m
Yep
k
Can you give me that error?
m
oh man totally missed this message one second….
Copy code
/home:prefect_management matthewmillendorf$ python hello_flow.py 
Flow URL: <https://cloud.prefect.io/raptormaps/flow/a39210bb-5598-4cee-ab8b-b9da2f4f0a3a>
 └── ID: bb7d1f40-7203-4c6f-918e-18bae14b0081
 └── Project: tutorial_matt
 └── Labels: ['my-macbook']
403
<!DOCTYPE html><html><head><title>Error Page</title><style type="text/css">html{font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;font-size:1em}.center-box{margin: 20% auto auto auto;width: 50%;border: 1px solid #dcdcdc;padding: 1em;}
</style><title>Security Violation (403)</title></head></head><body>
<div class="center-box">
    <h3><http://api.prefect.io|api.prefect.io>&nbsp;|&nbsp;Access denied (403)</h3>
    <h4>Current session has been terminated.</h2>
    <p>For further information, do not hesitate to contact us.</p>
    <p>Ref: <span id="addr">65.96.168.232</span>&nbsp;<span id="time">1620741471</span></p>
</div></body><script>document.getElementById("time").innerHTML = (new Date()).toISOString()</script>
</html>
k
Thank you!
The fix for you might be to add the 
User-Agent
under your headers
{'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"}
m
mm didnt work for me. Error:
Copy code
403
<!DOCTYPE html><html><head><title>Error Page</title><style type="text/css">html{font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;font-size:1em}.center-box{margin: 20% auto auto auto;width: 50%;border: 1px solid #dcdcdc;padding: 1em;}
</style><title>Security Violation (403)</title></head></head><body>
<div class="center-box">
    <h3><http://api.prefect.io|api.prefect.io>&nbsp;|&nbsp;Access denied (403)</h3>
    <h4>Current session has been terminated.</h2>
    <p>For further information, do not hesitate to contact us.</p>
    <p>Ref: <span id="addr">65.96.168.232</span>&nbsp;<span id="time">1620752288</span></p>
</div></body><script>document.getElementById("time").innerHTML = (new Date()).toISOString()</script>
</html>
Code:
Copy code
r = requests.get(url, params={'query': query}, headers={"authorization": "Bearer <my token>",
                                                            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36'})
k
We’ll look into this more. Man the user-agent fixed my Azure thing.
m
Damn. Another thing Im coming across that Im a bit confused on - havent seen this on GitHub or Slack yet… my logger doesnt log when I deploy to Fargate. The logs dont show up in cloud… any idea? But they show when I run the pipeline lcoally - its only when deployed to fargate. This is how im instantiating the logger.
Copy code
self.logger = prefect.context.get("logger")
does something need to change perhaps? so weird it works locally
python code hasnt changed so figured it was something regarding the env potentially - wondering if you had a quick idea
t
hiya Matthew, I'm helping look into this 403. re: logs, what log level are you using? your logs may be swallowed by the logging level
m
yeah thats a good point @Tyler Wanner , ive tried both info and error. its the same python code as when running it locally.
t
is there a reason you're setting 
self.logger
and not 
logger
m
yeah i want to log various class functions
ill give just instantiating it right before i log a try righ tnow
k
Is your class a Task? You can use the logger directly .
m
the class is getting called from within the task but the class itself is not a task. ive tried every way of instantiating the logger but no matter what, when i deploy to fargate, nothing logs
deploy with fargate**
running locally totally fine - no code change
k
Oh wait. Are you using a DaskExecutor?
m
i am not im using local executor
or whatever is default
k
I’ll look into this more
Do you see your logs on cloudwatch? this thread has some details.
m
checking this out now
i didnt set up logs for cloudwatch - is this something I should do - im not sure if that’s the best thread since its not that I don’t know whats happening within the container - its just that any time i make a call to the logger within the prefect context, no matter the logger set level and the message type, the logs dont show up in the UI. And this only happens for ECS tasks - not when running the pipeline locally.
k
Oh if you dont have cloudwatchs logs, I wouldn’t set it up for this. I think dig a bit more about this I think I’ve come across some threads.
m
okay cool ill keep digging
hey yeah ive looked through a bunch of github issues and slack threads - cant find too many similar threads. Been trying a bunch of things like logging levels, how im instantiating the logger, etc. and not having success
t
I'm very sorry about that--that is very frustrating
on the other note, the request that is receiving a 403 is violating some endpoint protection we have set up to protect our API
k
Yeah sorry I’m not finding much myself on the Fargate logs. I think this is your best bet. Have you seen it?
I think your Prefect setup looks right
m
cool cool yeah i looked through and i was having poor docker builds my bad. so @Tyler Wanner how do i proceed with hitting the API?
🚀 1
and thanks for the help guys appreciate it!
t
are you able to make the request using the prefect client? you should be able to just use client.graphql(query)
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by