k

    Kurt Rhee

    1 year ago
    Hello, I have reached this point in the prefect documentation and I was wondering where I should be looking to be able to host my dashboard in the cloud instead of from my local server?https://docs.prefect.io/orchestration/server/deploy-local.html#ui-configuration
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Kurt Rhee, as in you want to migrate from Prefect Cloud from Prefect Server? Or you want to host the UI somewhere like an EC2 instance?
    k

    Kurt Rhee

    1 year ago
    I currently have my prefect dashboard hoste on localhost 80:80, is it simple to host it at cloud.prefect.io?
    Kevin Kho

    Kevin Kho

    1 year ago
    We don’t host UIs in
    <http://cloud.prefect.io|cloud.prefect.io>
    , you would create something like an EC2 instance, and then start server there, and then the UI would be
    <VM-IP>:8080
    .
    k

    Kurt Rhee

    1 year ago
    ah gotcha
    Kevin Kho

    Kevin Kho

    1 year ago
    Just wanna clarify, Prefect Server is just the open source version of Prefect Cloud. We host it all for you whereas Prefect Server would be hosted on your own infrastructure.
    k

    Kurt Rhee

    1 year ago
    understood
    Hey Kevin I've set up a flow and I've also deployed the local host server, but I'm not sure how to connec tthe dots between them
    is this something you can help me with?
    I tried using prefect agent local start, but I get an prefect.exeptions.authorizationerror no agent api token provided
    Kevin Kho

    Kevin Kho

    1 year ago
    Did you register the flow? and did you do
    prefect backend server
    ?
    k

    Kurt Rhee

    1 year ago
    Yes I did prefect backend server, this fixed the authentication problem
    I'm not sure what registering the flow is
    ah i found the documentation
    Kevin Kho

    Kevin Kho

    1 year ago
    There’s a bunch so just wanna make sure you have the right one
    k

    Kurt Rhee

    1 year ago
    awesome thank you!
    Kevin you are the most helpful person
    Kevin Kho

    Kevin Kho

    1 year ago
    No problem it’s my job lol
    k

    Kurt Rhee

    1 year ago
    welp thanks for being good at your job!
    can you help me delete a project?
    swear last question for the day
    Kevin Kho

    Kevin Kho

    1 year ago
    Haha no worries, go to Team -> Projects in the UI and you can delete the projects there
    k

    Kurt Rhee

    1 year ago
    awesome thanks again!
    have a nice rest of your day
    I was wondering do I need to register a flow each time I edit it?
    Kevin Kho

    Kevin Kho

    1 year ago
    For the most part yes, unless you use a script based storage like S3 or Github where there is no serialization. If you dont edit the DAG significantly, this will work but if you add a new task or change a task name, then you need to re-register
    k

    Kurt Rhee

    1 year ago
    awesome thank you!
    Have you seen this error before?
    My flow seems to work fine in python console
    but the UI seems to think that it failed
    Kevin Kho

    Kevin Kho

    1 year ago
    Yeah what storage do you use? Do you have one defined?
    Or is botany something you import?
    k

    Kurt Rhee

    1 year ago
    botany is the top level directory that I am using for my flows
    I'm using parquet to store data from the flows, but that is it
    Kevin Kho

    Kevin Kho

    1 year ago
    Are you running this on a different machine than where you registered from?
    k

    Kurt Rhee

    1 year ago
    nope same machine
    same virtual environment too
    Kevin Kho

    Kevin Kho

    1 year ago
    I see so you when don’t choose the
    Storage
    class for Prefect, it automatically serializes your file and puts it in the
    .prefect
    folder. So something with the way you are registering is ruining that path. I think it might be a bit weird that your code is in the
    ___init___.py
    , not sure why you would do that what is happening is the path to the flow in the
    .prefect
    folder is not working well for you. So here are a couple of things you can do: 1. Use
    Local
    storage explicitly. There are two arguments you can use. First is the
    directory
    so you can control where the pickle is saved. Second, you can point it to a script by using stored_as_script=Trueand then supplying a
    path_to_file
    , which points to the Python file. 2. After this, you can also use the
    LocalRun
    to specify a
    working_dir
    so that if your flow as other imports in the same directory, you can choose the directory to start the flow from. How did you register this? Using the
    flow.register
    in the
    init.py
    ?
    I think you can try not doing it in an init file and that might work for you?
    Local storage docs and local run docs
    k

    Kurt Rhee

    1 year ago
    Interesting, yes I registered the path here:
    Kevin Kho

    Kevin Kho

    1 year ago
    That is not the path. That is the project name. The default
    Local()
    storage will get this, serialize it, and then save it under your
    .prefect
    folder. When you run this, can you find the file under
    flows
    under
    .prefect
    in the home directory?
    k

    Kurt Rhee

    1 year ago
    I tried moving outside of init and got the same issue
    I can't find my .prefect folder
    Kevin Kho

    Kevin Kho

    1 year ago
    It’s a hidden folder so maybe it’s just not showing in the UI? How do you register? Do you do
    python /path/to/file/__ _init__.py_
    or are you in the same folder as the init and it;s just
    python __init __.py
    ?
    k

    Kurt Rhee

    1 year ago
    Tried adding this on top of flow.register and it also failed
    flow.run_config = LocalRun()
    looking for hidden folders now
    i looked inside of my project directory and virtual enviornment
    should i be looking someplace else?
    Kevin Kho

    Kevin Kho

    1 year ago
    .prefect
    is in the home directory of your machine
    k

    Kurt Rhee

    1 year ago
    as for register it is in the same file as the flow, just under a if name == main statement
    I can see the flows inside of .prefect/flows
    Kevin Kho

    Kevin Kho

    1 year ago
    One sec will run some tests
    k

    Kurt Rhee

    1 year ago
    okay ty
    Kevin Kho

    Kevin Kho

    1 year ago
    Can you post your whole script?
    k

    Kurt Rhee

    1 year ago
    import os
    import json
    import warnings
    import pendulum
    
    from datetime import datetime, timedelta
    
    from prefect import Flow, Parameter
    from prefect.schedules import IntervalSchedule
    
    
    class ValentineEtl:
        """
        Valentine ETL Class
        """
    
        def __init__(self):
    
            # --- original data period ---
            self.start = '2019-09-11 00:00'
            self.end = datetime.now().strftime('%Y-%m-%d %H:%M')
    
            # --- get localized time ---
            self.site_code = 'VLT1'
            dirname = os.path.dirname(__file__)
            filename = os.path.join(dirname, fr'../intake/config/{self.site_code}.json')
            try:
                self.config = json.load(open(filename))
            except FileNotFoundError:
                self.config = warnings.warn(
                    f"Could not find {filename} "
                    "Please utilize botany.intake.geocode "
                    "to create a config.json file for this project")
    
    
    
        # --- ghi ---
        from botany.etl.generic_ghi import (
            extract_ghi, extract_ghi_tilt,
            transform_ghi,
            load_ghi, load_ghi_tilt
            )
    
    
    # --- Schedule ---
    schedule = IntervalSchedule(
        start_date=datetime.utcnow() + timedelta(seconds=1),
        interval=timedelta(days=1)
        )
    
    # --- Flow ---
    with Flow(
            'Valentine_GHI',
            schedule=schedule) as flow:
    
        # --- Configurations ---
        dag = ValentineEtl()
    
        # --- Extracts ---
        # tags
        ghi_tags = [
            'VLT1SMET001_Pyranometer_1_SR30_Irradiance', 'VLT1SMET002_Pyranometer_1_SR30_Irradiance',
            'VLT1SMET003_Pyranometer_1_SR30_Irradiance', 'VLT1SMET004_Pyranometer_1_SR30_Irradiance']
        ghi_tilt_tags = [
            'VLT1SMET001_MVTiltAngle', 'VLT1SMET002_MVTiltAngle',
            'VLT1SMET003_MVTiltAngle', 'VLT1SMET004_MVTiltAngle']
    
        # extractions
        ghi = dag.extract_ghi(dag, ghi_tags)
        ghi_tilt = dag.extract_ghi_tilt(dag, ghi_tilt_tags)
    
        # --- Transforms ---
        ghi = dag.transform_ghi(dag, ghi, ghi_tilt)
    
        # --- Loads ---
        dag.load_ghi(dag, ghi)
        dag.load_ghi_tilt(dag, ghi_tilt)
    
    
    if __name__ == '__main__':
        flow.register(project_name="APPA")
        flow.run()
        flow.visualize()
        pass
    Kevin Kho

    Kevin Kho

    1 year ago
    This is your error I think:
    # --- ghi ---
        from botany.etl.generic_ghi import (
            extract_ghi, extract_ghi_tilt,
            transform_ghi,
            load_ghi, load_ghi_tilt
            )
    It doesn’t know where to import these from. I suggest you can try
    LocalRun(working_dir="dir_above_botany")
    to that these imports can work. Is botany Python module or just a collection of scripts?
    k

    Kurt Rhee

    1 year ago
    oo nice i'll give that a shot
    i actually don't know the difference between a module and a collection of scripots
    I think it is a module, there are basically a lot of different functions / classes in there that we use for work
    Kevin Kho

    Kevin Kho

    1 year ago
    A module is actually like
    pip installed
    so that it can be used in other projects on the same machine that are in different directories
    k

    Kurt Rhee

    1 year ago
    ah gotcha
    then no it is not a module
    Kevin Kho

    Kevin Kho

    1 year ago
    If you start your local agent in the directory above
    botany
    , that import will work also. It’s just where the Python process starts, or you could install it as a module to make it available no matter where you run the script. The
    pip install
    adds the project to the path so it can be imported
    k

    Kurt Rhee

    1 year ago
    if __name__ == '__main__':
        flow.run_config = LocalRun(working_dir=r'../..')
        flow.register(project_name="APPA")
        flow.run()
        flow.visualize()
        pass
    Tried this one, it dind't work
    Kevin Kho

    Kevin Kho

    1 year ago
    I think because that “”../..” is applied relative to where the agent is running from. Not from where the flow is registered
    k

    Kurt Rhee

    1 year ago
    ohh
    if __name__ == '__main__':
        dirname = os.path.dirname(__file__)
        filename = os.path.join(dirname, r'../..')
        flow.run_config = LocalRun(working_dir=filename)
        flow.register(project_name="APPA")
        flow.run()
        flow.visualize()
    tried this one that failed too
    same module not found error
    Kevin Kho

    Kevin Kho

    1 year ago
    Actually I was wrong. It should be evaluated and then saved so it’s relative to the file path. What does
    os.path.dirname(__file__)
    give you though when you print it? It gives me nothing
    k

    Kurt Rhee

    1 year ago
    os.path.dirname gives me the path to the file
    I was able to change the location where i start the agent to the top directory
    and how i am getting this error which seems like progress
    Kevin Kho

    Kevin Kho

    1 year ago
    That is progress. I can’t see the error immediately. Does Flow.run() work for you?
    I’ll be out of office today, but if you leave messages I can get back to you when I’m on
    k

    Kurt Rhee

    1 year ago
    Nice sounds good man, thanks so much for the help so far
    flow.run does work within the console
    Kevin Kho

    Kevin Kho

    1 year ago
    Ah if you can post your traceback here I can look
    k

    Kurt Rhee

    1 year ago
    Flow URL: <http://localhost:8080/default/flow/cea14a36-26b1-4058-85bc-d9b792aafb08>
     └── ID: f88652b7-44c5-4487-b66c-a4f16826c1d8
     └── Project: APPA
     └── Labels: ['sdhqragsolpy02']
    [2021-08-06 08:19:06-0700] INFO - prefect.Pendleton_GHI | Waiting for next scheduled run at 2021-08-06T15:19:07.129768+00:00
    [2021-08-06 08:19:07-0700] INFO - prefect.FlowRunner | Beginning Flow run for 'Pendleton_GHI'
    [2021-08-06 08:19:07-0700] INFO - prefect.TaskRunner | Task 'extract_ghi': Starting task run...
    Current Tag:  PND1SMET001_GHI
    Current Tag:  PND1SMET002_GHI
    [2021-08-06 08:19:08-0700] INFO - prefect.TaskRunner | Task 'extract_ghi': Finished task run for task with final state: 'Success'
    [2021-08-06 08:19:09-0700] INFO - prefect.TaskRunner | Task 'extract_ghi_tilt': Starting task run...
    Current Tag:  PND1SMET001_GHI_TILT
    Current Tag:  PND1SMET002_GHI_TILT
    [2021-08-06 08:19:10-0700] INFO - prefect.TaskRunner | Task 'extract_ghi_tilt': Finished task run for task with final state: 'Success'
    [2021-08-06 08:19:10-0700] INFO - prefect.TaskRunner | Task 'load_ghi_tilt': Starting task run...
    [2021-08-06 08:19:10-0700] INFO - prefect.TaskRunner | Task 'load_ghi_tilt': Finished task run for task with final state: 'Success'
    [2021-08-06 08:19:10-0700] INFO - prefect.TaskRunner | Task 'transform_ghi': Starting task run...
    [2021-08-06 08:19:10-0700] INFO - prefect.TaskRunner | Task 'transform_ghi': Finished task run for task with final state: 'Success'
    [2021-08-06 08:19:10-0700] INFO - prefect.TaskRunner | Task 'load_ghi': Starting task run...
    [2021-08-06 08:19:10-0700] INFO - prefect.TaskRunner | Task 'load_ghi': Finished task run for task with final state: 'Success'
    [2021-08-06 08:19:10-0700] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
    [2021-08-06 08:19:10-0700] INFO - prefect.Pendleton_GHI | Waiting for next scheduled run at 2021-08-07T15:19:07.129768+00:00
    Kevin Kho

    Kevin Kho

    1 year ago
    That looks like a success right? Thought you had the error with the write
    k

    Kurt Rhee

    1 year ago
    yes it works in console, fails in ui
    Here is the UI failed message
    Kevin Kho

    Kevin Kho

    1 year ago
    Oh I think I know. Can you try removing
    flow.run()
    when you register? You might be running into issues with that. When you register, try having only the registration as the last line of your code.
    k

    Kurt Rhee

    1 year ago
    Same error
    actually i think i may have got it
    amazing
    thanks for all of your help Kevin, I was just dumb and was using self as an argument to pass into a function
    Kevin Kho

    Kevin Kho

    1 year ago
    then how did it work with flow.run?
    k

    Kurt Rhee

    1 year ago
    no idea