https://prefect.io logo
Powered by
Title
k

Kurt Rhee

08/04/2021, 4:01 PM
Hello, I have reached this point in the prefect documentation and I was wondering where I should be looking to be able to host my dashboard in the cloud instead of from my local server? https://docs.prefect.io/orchestration/server/deploy-local.html#ui-configuration
👋 1
k

Kevin Kho

08/04/2021, 4:05 PM
Hey @Kurt Rhee, as in you want to migrate from Prefect Cloud from Prefect Server? Or you want to host the UI somewhere like an EC2 instance?
k

Kurt Rhee

08/04/2021, 4:06 PM
I currently have my prefect dashboard hoste on localhost 80:80, is it simple to host it at cloud.prefect.io?
k

Kevin Kho

08/04/2021, 4:08 PM
We don’t host UIs in 
<http://cloud.prefect.io|cloud.prefect.io>
, you would create something like an EC2 instance, and then start server there, and then the UI would be 
<VM-IP>:8080
.
k

Kurt Rhee

08/04/2021, 4:08 PM
ah gotcha
k

Kevin Kho

08/04/2021, 4:13 PM
Just wanna clarify, Prefect Server is just the open source version of Prefect Cloud. We host it all for you whereas Prefect Server would be hosted on your own infrastructure.
k

Kurt Rhee

08/04/2021, 5:31 PM
understood
Hey Kevin I've set up a flow and I've also deployed the local host server, but I'm not sure how to connec tthe dots between them
is this something you can help me with?
I tried using prefect agent local start, but I get an prefect.exeptions.authorizationerror no agent api token provided
k

Kevin Kho

08/04/2021, 7:03 PM
Did you register the flow? and did you do 
prefect backend server
?
k

Kurt Rhee

08/04/2021, 7:06 PM
Yes I did prefect backend server, this fixed the authentication problem
I'm not sure what registering the flow is
ah i found the documentation
k

Kevin Kho

08/04/2021, 7:08 PM
There’s a bunch so just wanna make sure you have the right one
k

Kurt Rhee

08/04/2021, 7:08 PM
awesome thank you!
Kevin you are the most helpful person
k

Kevin Kho

08/04/2021, 7:09 PM
No problem it’s my job lol
k

Kurt Rhee

08/04/2021, 7:09 PM
welp thanks for being good at your job!
:thank-you: 1
can you help me delete a project?
swear last question for the day
k

Kevin Kho

08/04/2021, 7:22 PM
Haha no worries, go to Team -> Projects in the UI and you can delete the projects there
k

Kurt Rhee

08/04/2021, 7:23 PM
awesome thanks again!
have a nice rest of your day
👍 1
I was wondering do I need to register a flow each time I edit it?
k

Kevin Kho

08/05/2021, 8:57 PM
For the most part yes, unless you use a script based storage like S3 or Github where there is no serialization. If you dont edit the DAG significantly, this will work but if you add a new task or change a task name, then you need to re-register
k

Kurt Rhee

08/05/2021, 8:58 PM
awesome thank you!
Have you seen this error before?
My flow seems to work fine in python console
but the UI seems to think that it failed
k

Kevin Kho

08/05/2021, 8:59 PM
Yeah what storage do you use? Do you have one defined?
Or is botany something you import?
k

Kurt Rhee

08/05/2021, 9:50 PM
botany is the top level directory that I am using for my flows
I'm using parquet to store data from the flows, but that is it
k

Kevin Kho

08/05/2021, 9:58 PM
Are you running this on a different machine than where you registered from?
k

Kurt Rhee

08/05/2021, 11:08 PM
nope same machine
same virtual environment too
k

Kevin Kho

08/06/2021, 1:14 AM
I see so you when don’t choose the 
Storage
class for Prefect, it automatically serializes your file and puts it in the 
.prefect
folder. So something with the way you are registering is ruining that path. I think it might be a bit weird that your code is in the 
___init___.py
, not sure why you would do that what is happening is the path to the flow in the 
.prefect
folder is not working well for you. So here are a couple of things you can do: 1. Use 
Local
storage explicitly. There are two arguments you can use. First is the 
directory
so you can control where the pickle is saved. Second, you can point it to a script by using `stored_as_script=True`and then supplying a 
path_to_file
, which points to the Python file. 2. After this, you can also use the 
LocalRun
to specify a 
working_dir
so that if your flow as other imports in the same directory, you can choose the directory to start the flow from. How did you register this? Using the 
flow.register
in the 
init.py
?
I think you can try not doing it in an init file and that might work for you?
Local storage docs and local run docs
k

Kurt Rhee

08/06/2021, 2:04 PM
Interesting, yes I registered the path here:
k

Kevin Kho

08/06/2021, 2:07 PM
That is not the path. That is the project name. The default 
Local()
storage will get this, serialize it, and then save it under your 
.prefect
folder. When you run this, can you find the file under 
flows
under 
.prefect
in the home directory?
k

Kurt Rhee

08/06/2021, 2:32 PM
I tried moving outside of init and got the same issue
I can't find my .prefect folder
k

Kevin Kho

08/06/2021, 2:35 PM
It’s a hidden folder so maybe it’s just not showing in the UI? How do you register? Do you do 
python /path/to/file/__ _init__.py_
or are you in the same folder as the init and it;s just 
python __init __.py
?
k

Kurt Rhee

08/06/2021, 2:37 PM
Tried adding this on top of flow.register and it also failed 
flow.run_config = LocalRun()
looking for hidden folders now
i looked inside of my project directory and virtual enviornment
should i be looking someplace else?
k

Kevin Kho

08/06/2021, 2:38 PM
.prefect
is in the home directory of your machine
k

Kurt Rhee

08/06/2021, 2:39 PM
as for register it is in the same file as the flow, just under a if name == main statement
I can see the flows inside of .prefect/flows
k

Kevin Kho

08/06/2021, 2:42 PM
One sec will run some tests
k

Kurt Rhee

08/06/2021, 2:42 PM
okay ty
k

Kevin Kho

08/06/2021, 2:43 PM
Can you post your whole script?
k

Kurt Rhee

08/06/2021, 2:43 PM
import os
import json
import warnings
import pendulum

from datetime import datetime, timedelta

from prefect import Flow, Parameter
from prefect.schedules import IntervalSchedule


class ValentineEtl:
    """
    Valentine ETL Class
    """

    def __init__(self):

        # --- original data period ---
        self.start = '2019-09-11 00:00'
        self.end = datetime.now().strftime('%Y-%m-%d %H:%M')

        # --- get localized time ---
        self.site_code = 'VLT1'
        dirname = os.path.dirname(__file__)
        filename = os.path.join(dirname, fr'../intake/config/{self.site_code}.json')
        try:
            self.config = json.load(open(filename))
        except FileNotFoundError:
            self.config = warnings.warn(
                f"Could not find {filename} "
                "Please utilize botany.intake.geocode "
                "to create a config.json file for this project")



    # --- ghi ---
    from botany.etl.generic_ghi import (
        extract_ghi, extract_ghi_tilt,
        transform_ghi,
        load_ghi, load_ghi_tilt
        )


# --- Schedule ---
schedule = IntervalSchedule(
    start_date=datetime.utcnow() + timedelta(seconds=1),
    interval=timedelta(days=1)
    )

# --- Flow ---
with Flow(
        'Valentine_GHI',
        schedule=schedule) as flow:

    # --- Configurations ---
    dag = ValentineEtl()

    # --- Extracts ---
    # tags
    ghi_tags = [
        'VLT1SMET001_Pyranometer_1_SR30_Irradiance', 'VLT1SMET002_Pyranometer_1_SR30_Irradiance',
        'VLT1SMET003_Pyranometer_1_SR30_Irradiance', 'VLT1SMET004_Pyranometer_1_SR30_Irradiance']
    ghi_tilt_tags = [
        'VLT1SMET001_MVTiltAngle', 'VLT1SMET002_MVTiltAngle',
        'VLT1SMET003_MVTiltAngle', 'VLT1SMET004_MVTiltAngle']

    # extractions
    ghi = dag.extract_ghi(dag, ghi_tags)
    ghi_tilt = dag.extract_ghi_tilt(dag, ghi_tilt_tags)

    # --- Transforms ---
    ghi = dag.transform_ghi(dag, ghi, ghi_tilt)

    # --- Loads ---
    dag.load_ghi(dag, ghi)
    dag.load_ghi_tilt(dag, ghi_tilt)


if __name__ == '__main__':
    flow.register(project_name="APPA")
    flow.run()
    flow.visualize()
    pass
k

Kevin Kho

08/06/2021, 2:45 PM
This is your error I think: 
# --- ghi ---
    from botany.etl.generic_ghi import (
        extract_ghi, extract_ghi_tilt,
        transform_ghi,
        load_ghi, load_ghi_tilt
        )
It doesn’t know where to import these from. I suggest you can try 
LocalRun(working_dir="dir_above_botany")
to that these imports can work. Is botany Python module or just a collection of scripts?
k

Kurt Rhee

08/06/2021, 2:51 PM
oo nice i'll give that a shot
i actually don't know the difference between a module and a collection of scripots
I think it is a module, there are basically a lot of different functions / classes in there that we use for work
k

Kevin Kho

08/06/2021, 2:53 PM
A module is actually like 
pip installed
so that it can be used in other projects on the same machine that are in different directories
k

Kurt Rhee

08/06/2021, 2:53 PM
ah gotcha
then no it is not a module
k

Kevin Kho

08/06/2021, 2:56 PM
If you start your local agent in the directory above 
botany
, that import will work also. It’s just where the Python process starts, or you could install it as a module to make it available no matter where you run the script. The 
pip install
adds the project to the path so it can be imported
k

Kurt Rhee

08/06/2021, 2:59 PM
if __name__ == '__main__':
    flow.run_config = LocalRun(working_dir=r'../..')
    flow.register(project_name="APPA")
    flow.run()
    flow.visualize()
    pass
Tried this one, it dind't work
k

Kevin Kho

08/06/2021, 3:00 PM
I think because that “”../..” is applied relative to where the agent is running from. Not from where the flow is registered
k

Kurt Rhee

08/06/2021, 3:01 PM
ohh
if __name__ == '__main__':
    dirname = os.path.dirname(__file__)
    filename = os.path.join(dirname, r'../..')
    flow.run_config = LocalRun(working_dir=filename)
    flow.register(project_name="APPA")
    flow.run()
    flow.visualize()
tried this one that failed too
same module not found error
k

Kevin Kho

08/06/2021, 3:14 PM
Actually I was wrong. It should be evaluated and then saved so it’s relative to the file path. What does 
os.path.dirname(__file__)
give you though when you print it? It gives me nothing
k

Kurt Rhee

08/06/2021, 3:22 PM
os.path.dirname gives me the path to the file
I was able to change the location where i start the agent to the top directory
and how i am getting this error which seems like progress
k

Kevin Kho

08/06/2021, 3:23 PM
That is progress. I can’t see the error immediately. Does Flow.run() work for you?
I’ll be out of office today, but if you leave messages I can get back to you when I’m on
k

Kurt Rhee

08/06/2021, 3:31 PM
Nice sounds good man, thanks so much for the help so far
flow.run does work within the console
k

Kevin Kho

08/06/2021, 3:32 PM
Ah if you can post your traceback here I can look
k

Kurt Rhee

08/06/2021, 3:32 PM
Flow URL: <http://localhost:8080/default/flow/cea14a36-26b1-4058-85bc-d9b792aafb08>
 └── ID: f88652b7-44c5-4487-b66c-a4f16826c1d8
 └── Project: APPA
 └── Labels: ['sdhqragsolpy02']
[2021-08-06 08:19:06-0700] INFO - prefect.Pendleton_GHI | Waiting for next scheduled run at 2021-08-06T15:19:07.129768+00:00
[2021-08-06 08:19:07-0700] INFO - prefect.FlowRunner | Beginning Flow run for 'Pendleton_GHI'
[2021-08-06 08:19:07-0700] INFO - prefect.TaskRunner | Task 'extract_ghi': Starting task run...
Current Tag:  PND1SMET001_GHI
Current Tag:  PND1SMET002_GHI
[2021-08-06 08:19:08-0700] INFO - prefect.TaskRunner | Task 'extract_ghi': Finished task run for task with final state: 'Success'
[2021-08-06 08:19:09-0700] INFO - prefect.TaskRunner | Task 'extract_ghi_tilt': Starting task run...
Current Tag:  PND1SMET001_GHI_TILT
Current Tag:  PND1SMET002_GHI_TILT
[2021-08-06 08:19:10-0700] INFO - prefect.TaskRunner | Task 'extract_ghi_tilt': Finished task run for task with final state: 'Success'
[2021-08-06 08:19:10-0700] INFO - prefect.TaskRunner | Task 'load_ghi_tilt': Starting task run...
[2021-08-06 08:19:10-0700] INFO - prefect.TaskRunner | Task 'load_ghi_tilt': Finished task run for task with final state: 'Success'
[2021-08-06 08:19:10-0700] INFO - prefect.TaskRunner | Task 'transform_ghi': Starting task run...
[2021-08-06 08:19:10-0700] INFO - prefect.TaskRunner | Task 'transform_ghi': Finished task run for task with final state: 'Success'
[2021-08-06 08:19:10-0700] INFO - prefect.TaskRunner | Task 'load_ghi': Starting task run...
[2021-08-06 08:19:10-0700] INFO - prefect.TaskRunner | Task 'load_ghi': Finished task run for task with final state: 'Success'
[2021-08-06 08:19:10-0700] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
[2021-08-06 08:19:10-0700] INFO - prefect.Pendleton_GHI | Waiting for next scheduled run at 2021-08-07T15:19:07.129768+00:00
k

Kevin Kho

08/06/2021, 3:33 PM
That looks like a success right? Thought you had the error with the write
k

Kurt Rhee

08/06/2021, 3:33 PM
yes it works in console, fails in ui
Here is the UI failed message
k

Kevin Kho

08/06/2021, 3:34 PM
Oh I think I know. Can you try removing 
flow.run()
when you register? You might be running into issues with that. When you register, try having only the registration as the last line of your code.
k

Kurt Rhee

08/06/2021, 3:36 PM
Same error
actually i think i may have got it
amazing
thanks for all of your help Kevin, I was just dumb and was using self as an argument to pass into a function
k

Kevin Kho

08/06/2021, 3:44 PM
then how did it work with flow.run?
k

Kurt Rhee

08/06/2021, 4:29 PM
no idea
2 Views
#prefect-serverJoin Slack
Powered by