Hi Everyone, I am trying to trigger prefect flow deployed on cloud whenever file is landed to S3 buc...
j
Hi Everyone, I am trying to trigger prefect flow deployed on cloud whenever file is landed to S3 bucket using AWS Lambda function by referring this documentation from Chris. Lambda function is failing with
"errorMessage": "HTTP Error 404: Not Found"
I am passing API URL
<https://api.prefect.io>
. I understand that this documentation is not latest and I also tried URL
<https://api.prefect.cloud/>
and getting same error. Request your inputs regarding this error in Prefect version 2.0
1
r
Hi JV, The ideal way to do this looks a bit different in Prefect 2. Instead of calling the API, consider setting the PREFECT_API_URL and PREFECT_API_KEY environment variables and then calling your Prefect flow function from your Lambda. It will run and record the flow run results in Prefect Cloud. You might find this post helpful, as well as the GitHub repository that goes with it (it doesn't sound like you need the GitHub Actions part of the article, but I think it's still useful if you skip that part). This S3 event flow in particular sounds like it is close to what you need.
If you have more questions about this or anything AWS-related, I suggest posting to the #C048K0MGHNK channel. That help ensure it gets seen ASAP 😄
j
Hi @Ryan Peden, I am setting those parameters as environment variables and then accessing in the code, lambda is failing with the earlier mentioned error. I have referred the posts which you have shared and I was not able to find resolution. Status code 404 indicates page not available so I wanted to check if the API URL which I am using is correct.
n
Hi @JV - your URL should be structured like this
Copy code
<https://api.prefect.cloud/api/accounts/UUID/workspaces/UUID>
is that how yours looks?
j
Hi @Nate no, I have not structured URL in that format, I am just passing
Copy code
<https://api.prefect.cloud/>
can you please let me know if I need to dynamically pass UUID to the URL you have mentioned above and where will I get that?
n
if you run
prefect config view
locally you can see what your URL should be (provided you've setup a profile, otherwise here are the docs) and that's what you should set as the PREFECT_API_URL in the env vars for your lambda
j
Hi @Nate, Thanks for this information, its working. Now my API call resulted in
Copy code
"HTTP Error 405: Method Not Allowed"
Do we need to set any permissions in prefect cloud to allow POST requests on cloud API?
n
can you show how you're making your api call?
j
Hi @Nate, I am making api call as mentioned in the post, I want to pass s3 file path which is captured as event to the flow
Copy code
bucket = event['Records'][0]['s3']['bucket']['name']
key = event['Records'][0]['s3']['object']['key']
input_file = "s3://"+bucket+"/"+key

create_mutation = """
mutation($input: create_flow_run_input!){
	create_flow_run(input: $input){
		id
	}
}
"""
inputs = dict(flow_id=os.getenv("PREFECT__FLOW_ID"))
inputs['file'] = input_file

variables = dict(input=inputs)

data = json.dumps(
	dict(query=create_mutation, variables=json.dumps(variables))
).encode("utf-8")

## prepare the request
req = urllib.request.Request(os.getenv("PREFECT__CLOUD__API"), data=data)
req.add_header("Content-Type", "application/json")
req.add_header(
	"Authorization", "Bearer {}".format(os.getenv("PREFECT__CLOUD__API_KEY"))
)

## send the request and return the response
resp = urllib.request.urlopen(req)
return json.loads(resp.read().decode())
n
@JV because that article you linked above is for prefect 1, it shows how to interact with the graphQL API we had in prefect 1. Prefect 2 has a REST API, so you will need to hit the appropriate REST endpoint to trigger flow runs Let me find a discourse link
you can do something like this in prefect 2
1
j
Hi @Nate , Its working, Thanks for your inputs.
🦜 1
591 Views