https://prefect.io logo
Powered by
x

Xavier Babu

04/26/2022, 4:44 PM
We are not using any deployment container like Docker. We use the storage just mounted in the Linux Server. How can we create a local storage for prefect instead of using /tmp. When I run the the command "prefect storage create" I don't get any list to choose for storage type and whatever value I put, it says "invalid selection". Please help. Found the following storage types: Select a storage type to create: 4 Invalid selection 4
k

Kevin Kho

04/26/2022, 4:56 PM
Is this on local Orion or is this on Cloud 2.0?
x

Xavier Babu

04/26/2022, 4:57 PM
local Orion
k

Kevin Kho

04/26/2022, 4:58 PM
Are you on 2.0b3?
x

Xavier Babu

04/26/2022, 4:58 PM
yes
let me check
2.0b2
k

Kevin Kho

04/26/2022, 4:58 PM
Could you try 2.0b3 and see if it persists?
x

Xavier Babu

04/26/2022, 4:59 PM
how can I upgrade 2.0b2 to b3?
k

Kevin Kho

04/26/2022, 4:59 PM
pip install prefect==2.0b3
x

Xavier Babu

04/26/2022, 5:00 PM
same command, got it. I thought we have any specific upgrade command.
Let me try
I am running (prefect dev UI with source code). Will this upgrade impact that dev UI?
a

Anna Geller

04/26/2022, 5:05 PM
when you run it in dev mode, you would also need to install it from source I think I had a similar issue when doing it while some session was already open in my browser. Can you try closing all Orion UI browser windows and then try creating storage and starting Orion?
x

Xavier Babu

04/26/2022, 5:08 PM
sure
that trick worked. Thanks Anna.
🙌 1
@flow(name="XBCustom_Flow2") def custom_flow2(sql): logger = get_run_logger() log_message(sql) logger.debug("Entering Custom FLow") if not sql : sql = "SELECT count(*) from rpt_analytics.dcm_conversions_combined_p2c;" r1 = hello_world() Am I doing anything wrong in this code? I don't see the log in the UI.
k

Kevin Kho

04/26/2022, 5:25 PM
How are you running it? Just by calling 
custom_flow2
?
x

Xavier Babu

04/26/2022, 5:26 PM
DeploymentSpec( flow=custom_flow2, name="XBparent_flow4_deploy", tags=['waves_q'], parameters={'sql': 'SELECT count(*) from rpt_analytics.dcm_conversions_combined_p2c;'}, flow_runner=SubprocessFlowRunner(condaenv="/localpart0/aop-shared/WAVES/prefect2", stream_output=True) )
By deploy and run
k

Kevin Kho

04/26/2022, 5:32 PM
Do you see logs that the flow started and ended but not logs from the logger? Or do you see 0 logs completely?
x

Xavier Babu

04/26/2022, 5:38 PM
Yes, I see the logs (flow start and end etc., too) while running the flow in CLI using prefect deployment execute
Only in the UI I can't see it.
'log_message-718c3f46-0' - Finished in state Completed(None)
a

Anna Geller

04/26/2022, 5:43 PM
can you share your log_message() code?
x

Xavier Babu

04/26/2022, 5:44 PM
@flow(name="XBCustom_Flow2") def custom_flow2(sql): logger = get_run_logger() log_message(sql) logger.debug("Entering Custom FLow") if not sql : sql = "SELECT count(*) from rpt_analytics.dcm_conversions_combined_p2c;" r1 = hello_world()
Output comes in the command line, but, not in the UI
Do I need to do any additional config?
k

Kevin Kho

04/26/2022, 5:48 PM
Sorry I’m confused what you see in the UI. Could you share how it looks there?
a

Anna Geller

04/26/2022, 5:54 PM
you still shared custom_flow2 - I asked for log_message(sql) definition. Can you share the full flow code? hard to help otherwise
x

Xavier Babu

04/26/2022, 5:54 PM
from prefect import task, flow from prefect.task_runners import DaskTaskRunner from prefect.deployments import DeploymentSpec from prefect.flow_runners import SubprocessFlowRunner import aops1.aopsconn.db_connect as dc import psycopg2 from prefect.tasks import task_input_hash from datetime import timedelta from prefect import get_run_logger from prefect.orion.schemas.schedules import RRuleSchedule from dateutil.rrule import rrule, MINUTELY, DAILY, MO, TU, WE, TH, FR import pendulum from prefect.orion.schemas.schedules import IntervalSchedule from datetime import timedelta import platform @task def say_hi(): logger = get_run_logger() logger.info("Hello world!") @task def print_platform_info(): logger = get_run_logger() logger.info( "Platform information: IP = %s, Python = %s, Platform type = %s, OS Version = %s", platform.node(), platform.python_version(), platform.platform(), platform.version(), ) @task def log_message(name): logger = get_run_logger() logger.info(f"Hello {name}!") return @flow(name="XBCustom_Flow2") def custom_flow2(sql): hi = say_hi() print_platform_info(wait_for=[hi]) logger = get_run_logger() log_message(sql) logger.debug("Entering Custom FLow") if not sql : sql = "SELECT count(*) from rpt_analytics.dcm_conversions_combined_p2c;" r1 = hello_world()
k

Kevin Kho

04/26/2022, 5:58 PM
Thanks! I can try to replicate this. This is backed by Cloud 2.0 right?
x

Xavier Babu

04/26/2022, 5:59 PM
All in local Linux Server. No cloud involved.
Prefect Orion 2.0
k

Kevin Kho

04/26/2022, 6:00 PM
Ok will try
a

Anna Geller

04/26/2022, 6:00 PM
you may want to change the sql as a default argument on your flow function and refactor it this way:
Copy code
from prefect import task, flow
from prefect import get_run_logger
import platform


@task
def say_hi():
    logger = get_run_logger()
    <http://logger.info|logger.info>("Hello world!")


@task
def print_platform_info():
    logger = get_run_logger()
    <http://logger.info|logger.info>(
        "Platform information: IP = %s, Python = %s, Platform type = %s, OS Version = %s",
        platform.node(),
        platform.python_version(),
        platform.platform(),
        platform.version(),
    )


@task
def log_message(name):
    logger = get_run_logger()
    <http://logger.info|logger.info>(f"Hello {name}!")


@flow(name="XBCustom_Flow2")
def community_custom_flow2(
    sql: str = "SELECT count(*) from rpt_analytics.dcm_conversions_combined_p2c;",
):
    hi = say_hi()
    print_platform_info(wait_for=[hi])
    log_message(sql)
    logger = get_run_logger()
    logger.debug("Exiting FLow")


if __name__ == "__main__":
    community_custom_flow2()
all logs will appear in the UI - I think you are just a bit too fast 🙂
my debug log doesn't show up because my log level is set to INFO - you can change it using:
Copy code
export PREFECT_LOGGING_LEVEL=DEBUG
or even better:
Copy code
prefect config set PREFECT_LOGGING_LEVEL=DEBUG
x

Xavier Babu

04/26/2022, 6:07 PM
Done the config. Executed again. Still the log is not coming in the UI.
But I can see the log in CLI
180616.670 | DEBUG | Task run 'log_message-718c3f46-0' - Beginning execution... 180616.671 | INFO | Task run 'log_message-718c3f46-0' - Hello SELECT count(*) from rpt_analytics.dcm_conversions_combined_p2c;! 180616.707 | INFO | Task run 'say_hi-93ad9977-0' - Finished in state Completed(None)
180616.746 | DEBUG | Flow run 'vivacious-dogfish' - Submitted task run 'prefect_say-6e4453ba-0' to task runner 180616.761 | DEBUG | Task run 'print_platform_info-340ece69-0' - Beginning execution... 180616.763 | INFO | Task run 'print_platform_info-340ece69-0' - Platform information: IP = dtord03aop02d.dc.dotomi.net, Python = 3.10.1, Platform type = Linux-3.10.0-1160.11.1.el7.x86_64-x86_64-with-glibc2.17, OS Version = #1 SMP Fri Dec 18 163456 UTC 2020 1
k

Kevin Kho

04/26/2022, 6:31 PM
I can’t seem to replicate as well. I get logs in the UI and in the CLI
x

Xavier Babu

04/26/2022, 6:34 PM
Is it something wrong with my UI Config?
BTW, I also don't see Add Work-Queue Button in my UI.
k

Kevin Kho

04/26/2022, 6:36 PM
You don’t see these?
x

Xavier Babu

04/26/2022, 6:36 PM
Correct
k

Kevin Kho

04/26/2022, 6:37 PM
What do you mean when you said you used the dev UI?
x

Xavier Babu

04/26/2022, 6:37 PM
I created the build from the source code and run it
k

Kevin Kho

04/26/2022, 6:38 PM
Why did you do that, and when did you get your source? Are the logs missing for just this flow or for all flows? Because I think the logs were not there a couple of monthsa go
x

Xavier Babu

04/26/2022, 6:39 PM
because I would like to access the UI using my hostname instead of using localhost
It was not supported in 2.0
k

Kevin Kho

04/26/2022, 6:40 PM
Ahh I see. I would suggest you update the UI source again and try that?
x

Xavier Babu

04/27/2022, 2:49 PM
What is the REST API equivalent to run "prefect deployment run flow_name/deploy_name"?
k

Kevin Kho

04/27/2022, 2:50 PM
Do you need REST API? You can use the Orion client like this and it will hit the API for you
x

Xavier Babu

04/27/2022, 3:09 PM
I have deployment spec in the python code itself and passing the file via blob in the REST API to deploy which is similar to "prefect deployment create", but the deployment_spec in the .py file is either not used or overwritten by the params in the REST API Payload.
BTW, we are using the REST APIS, left and right 🙂
we are good at it, but I see some discrepancies.
k

Kevin Kho

04/27/2022, 3:14 PM
Uhh…that’s amazing…but not a pattern we recommend for Orion😅. Specifically because there is a lot of handling to shape stuff for the API request. You can trace the code from here though for the CLI command
x

Xavier Babu

04/27/2022, 3:25 PM
@deployment_app.command()async def create(path: str): In the source code you showed, the above portion of the code is what I am talking about. How REST API runs this function internally?
k

Kevin Kho

04/27/2022, 3:26 PM
I think you are more concerned about the post here
a

Anna Geller

04/27/2022, 3:26 PM
I answered already here - can we continue the discussion on Discourse?
it's definitely confusing to talk about the same on two channels (Slack and Discourse) at the same time
x

Xavier Babu

04/27/2022, 3:30 PM
Yes, I will continue in the discourse. Sorry.
👍 1
Kevin/Ann, Still I can't see the log in the Orion UI. Are there any other config I miss, because of that the UI is not reading the log correctly from SQLite, or any config to write the log in SQLite to bring it in the UI?
BTW, I see the log data stored in the log table in orion.db
somehow the UI is not fetching it
a

Anna Geller

04/29/2022, 7:51 PM
can you share your flow code? if you use:
Copy code
from prefect import get_run_logger
it should work fine
if local is too hard to debug, you can use a free tier of Cloud 2.0 https://discourse.prefect.io/t/how-to-get-started-with-prefect-cloud-2-0/539
maybe it's a matter of a log level?
x

Xavier Babu

05/05/2022, 11:36 PM
I tried all known possibilities, still I can't get the log in Orion UI.
Also the Flows Tab in the UI is not showing the list of flows but it shows only the count. But all other tables like Deployments, Flow Runs and Task Runs it shows the list when I click it.
I am getting "No logs to show". Even I tried all log levels
PREFECT_PROFILE='default' PREFECT_HOME='/localpart0/aop-shared/WAVES/prefect2' PREFECT_API_URL='http://dtord03aop02:5200/api' PREFECT_LOGGING_LEVEL='DEBUG' PREFECT_LOGGING_SETTINGS_PATH='${PREFECT_HOME}/lib/python3.10/site-packages/prefect/logging/logging.yml' PREFECT_ORION_DATABASE_CONNECTION_URL='sqlite+aiosqlite://///localpart0/aop-shared/WAVES/prefect2/orion.db' PREFECT_ORION_API_HOST='dtord03aop02' PREFECT_ORION_API_PORT='5200' PREFECT_AGENT_PREFETCH_SECONDS='10' PREFECT_AGENT_QUERY_INTERVAL='5.0' PREFECT_API_KEY='None' PREFECT_API_REQUEST_TIMEOUT='30.0' PREFECT_API_URL='None' PREFECT_CLOUD_URL='https://api-beta.prefect.io/api' PREFECT_DEBUG_MODE='False' PREFECT_HOME='/home/xbabu/.prefect' PREFECT_LOGGING_EXTRA_LOGGERS='' PREFECT_LOGGING_LEVEL='INFO' PREFECT_LOGGING_ORION_BATCH_INTERVAL='2.0' PREFECT_LOGGING_ORION_BATCH_SIZE='4000000' PREFECT_LOGGING_ORION_ENABLED='True' PREFECT_LOGGING_ORION_MAX_LOG_SIZE='1000000' PREFECT_LOGGING_SERVER_LEVEL='WARNING' PREFECT_LOGGING_SETTINGS_PATH='${PREFECT_HOME}/logging.yml' PREFECT_ORION_ANALYTICS_ENABLED='True' PREFECT_ORION_API_DEFAULT_LIMIT='200' PREFECT_ORION_API_HOST='127.0.0.1' PREFECT_ORION_API_PORT='4200' PREFECT_ORION_DATABASE_CONNECTION_TIMEOUT='5.0' PREFECT_ORION_DATABASE_CONNECTION_URL='sqlite+aiosqlite:////${PREFECT_HOME}/orion.db' PREFECT_ORION_DATABASE_ECHO='False' PREFECT_ORION_DATABASE_MIGRATE_ON_START='True' PREFECT_ORION_DATABASE_TIMEOUT='1.0' PREFECT_ORION_SERVICES_LATE_RUNS_AFTER_SECONDS='00005' PREFECT_ORION_SERVICES_LATE_RUNS_ENABLED='True' PREFECT_ORION_SERVICES_LATE_RUNS_LOOP_SECONDS='5.0' PREFECT_ORION_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE='100' PREFECT_ORION_SERVICES_SCHEDULER_ENABLED='True' PREFECT_ORION_SERVICES_SCHEDULER_INSERT_BATCH_SIZE='500' PREFECT_ORION_SERVICES_SCHEDULER_LOOP_SECONDS='60.0' PREFECT_ORION_SERVICES_SCHEDULER_MAX_RUNS='100' PREFECT_ORION_SERVICES_SCHEDULER_MAX_SCHEDULED_TIME='100 days, 00000' PREFECT_ORION_UI_ENABLED='True' PREFECT_PROFILES_PATH='${PREFECT_HOME}/profiles.toml' PREFECT_TEST_MODE='False'
prefect default settings
where prefect stores the default settings. I don't see it in the sqlite DB.
I also don't see option to create API KEY from the UI using Settings
Please help me to resolve these issues.
I don't think the DEV Orion UI code has write build.
a

Anna Geller

05/06/2022, 12:53 PM
Re: "all known possibilities" - Can you list what exactly have you tried so far?
Copy code
PREFECT_HOME='/localpart0/aop-shared/WAVES/prefect2'
PREFECT_ORION_DATABASE_CONNECTION_URL='sqlite+aiosqlite://///localpart0/aop-shared/WAVES/prefect2/orion.db'
It looks like you run Orion from some FTP location. This may be the source of your issue. Could you try running Orion from your local machine or use Cloud 2.0?
x

Xavier Babu

05/06/2022, 2:32 PM
Anna, Thank you for your reply. Actually, it is not a FTP, it is a local storage mounted in our linux server.
I don't see any performance issue.
a

Anna Geller

05/06/2022, 2:43 PM
but this won't work due to WAL
x

Xavier Babu

05/06/2022, 2:45 PM
I am setting the cloud as per your info. Let me update you soon. I am getting an error while giving the workspace during cloud login. (I have created a workspace already)
👍 1
Cloud is working fine for me. What is the constraint or limitation on using this Cloud? Can we use it for real business? Is there any cost involved?
Instead of using SQLite if I use Postgres DB, will this problem be resolved?
k

Kevin Kho

05/06/2022, 3:33 PM
Cloud has no cost for now. The pricing model is not public yet. I don’t think your problem will be resolved by chaging the database because you have a UI issue right? Cloud would be a good option though for a remote UI
x

Xavier Babu

05/06/2022, 3:47 PM
Can I get an example or tutorial to use Postgres, please?
k

Kevin Kho

05/06/2022, 3:48 PM
You mean this ?
x

Xavier Babu

05/06/2022, 3:54 PM
It helps. Thanks Kevin.
👍 1
The Cloud Prefect is only taking care of SQLite DB and update from my local server goes to the cloud? OR the entire workflow is running at the cloud?
k

Kevin Kho

05/06/2022, 6:27 PM
If you are using the cloud endpoint, the database used is the cloud hosted one, but the workflow is running locally
x

Xavier Babu

05/06/2022, 6:29 PM
I don't need to run "prefect orion start", only I need to run "prefect agent start"?
in my local machine
k

Kevin Kho

05/06/2022, 6:29 PM
Yes exactly. Agent pointing to Cloud
x

Xavier Babu

05/06/2022, 6:30 PM
the python code I deploy doesn't go to the cloud, however the log info goes to the cloud which may contain some of my code info during exceptions, right?
k

Kevin Kho

05/06/2022, 6:33 PM
I guess that is possible yes. Would you like to turn off logging?
x

Xavier Babu

05/06/2022, 8:01 PM
how to get the bearer token to call cloud prefect 2.0 API?
do we need to have that bearer token to call the APIs?
a

Anna Geller

05/06/2022, 11:15 PM
it depends on what you try to do. If you want to e.g. trigger a flow run from deployment via an API call, then yes, you would need to create an API key in the UI and then authenticate e.g. via terminal
x

Xavier Babu

05/07/2022, 3:02 AM
Thank you, Anna! Thanks, Xavier Babu
23 Views
Powered by