https://prefect.io logo
Join Slack
Powered by
OK, I'm trying to get Prefect Cloud v2 working ......
# prefect-cloud
c
OK, I'm trying to get Prefect Cloud v2 working .... and learning as I go .... I have a script that is working perfectly (finally), now created a deployment with an "every 30 mins schedule" .... but does that mean that every workflow in the workspace runs on that schedule?
r
No just that flow you deployed
c
I can't get the flow linked to the deployment
r
What does your deployment look like
c
in what regard ? I'm using Cloud v2. I defined the deployment, added a schedule of 0,30 * * * *
it's enabled, and creates the runs ... but nothing actually runes
runs 😛
r
Have you got an agent running to pick up the flow run?
c
yes
running via supervisord
well, I THINK it's running ... maybe not 😞
r
check the agent logs
it should show in the cloud ui as being healthy if connected
c
where is that in v2 ?
hmm m... even with Cloud do I need to do :
prefect deployment build
I'm always confused what is meant for when you're using Cloud or not
r
in v2 there is a work pools menu on the left hand nav
inside the pool there is work queues showing your agent(s)
e.g
c
UNHEALTHY 😞
r
you need to do build and apply
on your deployment
if its unhealthy - are you sure the api key and url env vars are set correctly
check the logs
c
I remember pasting an API this morning 🙂
Basically I created a new VM this morning, installed prefect and did all the setup
copied over my scripts and converted to v2
if I run manually they work fine, I just can't schedule them
on my old (v1) VM that I was using with Prefect Cloud v1 I had to run supervisord to make sure that the agent was alive all the time
r
the agent needs to be healthy to poll for flows on the schedule
c
so I copied that over to the new VM (the config file after installing supervisord) but there's still something that I'm missing (in my brain, in the config, somewhere)
r
agent needs PREFECT_API_KEY and PREFECT_API_URL
c
hmm ... I just started the agent on the CLI and it's running all the flows that hadn't started yet
agent is now showing as Healthy in the Work Queues tab
I just need to get the agent working all the time
I think that might be the missing piece
r
systemd
c
not supervisord ?
r
it should work - not tried it
the supervisord logs should say what is happening if it cant keep the agent healthy
c
ummm ... that looked like it worked 🙂
agent is reporting healthy now
r
which bit 😁
c
no, I spoke too soon 😞
in Work Pools > Work Queues the agent is healthy
in Work Pools > Runs the deployment runs
but the flow still isn't linked to the deployment
is this a CLI thing I need, or something in the actual code ?
I'm missing the yaml file I think
r
you build the yaml file then apply to deploy
c
I still need to do that with Cloud ?
r
yes
you do using python to deploy
cli is --apply
c
can I have 2 flows in a deployment ?
I can't even get the yaml file to be created, or it's not where I think it should be 😞
r
you can bundle a second flow and call it as a subflow
use --output to specify the folder for the yaml
c
OK, so 1 flow per deployment is fine, I just can't get even 1 deployment to work
r
if you specify the output path it should create the yaml, assume you have specified storage/infrastructure etc
c
I'm running on a linode VM
so it's just like a normal machine
r
so you want it running as a sub process on the agent
c
I don't know what the right terminology is - I just want my 2 scripts running every 30 mins like they did on Prefect Cloud v1
it just feels like there's far too many hoops to jump through to even get one flow running on a schedule 😞
and I know it's just me ... surely it can't be the product
r
yeah so youll need a process infrastructure block to it knows how you want the flow to be ran once the agent collects it
and youll need a storage block so it can download the flow files
c
so I need to write the code, then make it a flow then create a deployment then assign the flow to a deployment (I still can't do that), but now I need a storage block and ... andd ... and.... it's just too much 😞
the deployment works, it just doesn't know what to run
r
it's all pretty logical, it has to upload the files somewhere for them to be downloaded and ran
c
but they run fine on the VM
can't they just run "in situ"
r
then a localstorage block
c
why a block at all?
I'm reading all that doco about deployments and I haven't seen any mention of a block yet
but rewinding back to the start .... I have a VM that has 2 python scripts. Both scripts have to run every 30 mins (in cron terms 0,30 **)
that's it, that's all I need
r
i thick local storage is the default
c
Prefect 1 had a great gui and was very simple to configure and use - v2 is a nightmare
r
as long as its in the default prefect path
p.s I dont work for Prefect:)
and I didnt use v1
c
yeah, I know 😞 .... I've had meetings with them as well - hence why I'm here because they stuffed my v1 setup this morning 😞
r
c
so this MUST be fixed today and it still doesn't work
I can run the scripts manually, it works fine and shows the results in the GUI
surely the "scheduling" side should be a breeze to configure and use - that's the whole point right ?
r
it is, just a cron or rrule
c
I mean ... get a script to run on the cron.
my deployment is working fine, just not actually doing anything
r
so if your flows in the vm are in PREFECT_LOCAL_STORAGE_PATH it should find them and invoke them
so that is ~/.prefect/storage
by default
c
I don't even have a storage folder in .prefect
and no PREFECT_LOCAL_STORAGE_PATH defined
in .prefect there is only a profiles.toml
r
did you install in a venv
c
no, just a brand new VM
only for this purpose
r
what instructions did you follow
c
LOL ... the instructions from Prefect employees and the website
pip install prefect, etc....
convert v1 to v2 doing this, etc..
"should be easy" .. just not so easy
I think I'm close, but at the same time I don't know how big the gap is - the script runs, you can see progress in the GUI
it just doesn't run every 30 mins and only runs when I manually run the script or the agent
actually ... if the agent is healthy why doesn't it work ?
r
the agent logs should give an indication if its picking up a flow run and then not finding the flow locally
c
where are the agent logs ?
r
Logs to console by default - you can specify create a logging config to write it to disk
but lets start the agent in interactive mode and see what is happening
stop the systemd prefect service and start the agent from the command line
c
like this ? systemctl stop prefect-agent.service
r
systemctl stop prefect-agent
c
ok
r
then export PREFECT_LOGGING_LEVEL="DEBUG"
then start the agent interactively prefect agent start
c
I have to do prefect agent start -p default-agent-pool
to provide the work queue
r
yeah work pool
c
just constantly pinging
r
right so you have not set your cloud url
because it should tell you the workspace your connected to
export PREFECT_API_KEY="[YOUR-API-KEY]"
c
it does, I just didn't capture that 😉
r
ha ok
so have you got a deployment scheduled
c
r
so 28 minutes
c
hang on, it jsut caught this ...
but the flows still didn't run
r
says they completed
what does the flow do
c
reads an API and writes the results
so I can check that no results have been written
those times match the most recent data, which is when I ran the scripts manually
times are Sydney time, so 5 hrs ago
r
have you got persist_result on the flow
c
probably not, I don't know what that means at all
r
c
I just ran the script manually at the command line - worked perfectly
r
it only persists if you tell it to - or set the env var to always persist
c
do I need that though ?
r
dunno - you mentioned "no results have been written"
I assumed you meant flow result storage
c
sorry, the data that is produced in the script
the script reads an API and writes data to a database
and I can check what has been written to the database
there is only something written when I run the script manually
even though the deployment is running every 30 mins
r
is that the flow that logs out Hi From Prefect
I would add some more logs to the flow before+after trying to write to the db to see if it gets to those points during the scheduled run
the scheduled ones seem to run for a second versus the 30+ of your manual ones
c
that's the manual run, I just can't get the same thing on the schedule
r
yeah so log some stuff out so we can identify why as the agent log just says its completed
c
hmmm the Hi is something I was trying earlier
r
seems to fall out quickly
c
r
all the flow is doing is logging hi
c
yeah, but at least it works
maybe that code should go in my script ?
r
where is the api call etc
c
the Hi is just some dummy code I found today
I didn't even realise it was running properly
r
ok - so yeah - slap the actual work in the flow and you are good
c
OK, that looked like it worked
at least they ran on scheduled
OK, THANK YOU SO MUCH for your help and time
I'll let it run overnight and see that it's still stable in the morning
r
cool, fingers crossed
c
AGAIN ... thank you so much for your help yesterday, it's now working perfectly every 30 mins
r
hey @Chris McClellan, glad we sorted 👍
✅ 1
Open in Slack
PreviousNext
Powered by