https://prefect.io
Join Slack
We are not using any deployment container like Docker. We use the storage just mounted in the Linux ...
# ask-community
x
We are not using any deployment container like Docker. We use the storage just mounted in the Linux Server. How can we create a local storage for prefect instead of using /tmp. When I run the the command "prefect storage create" I don't get any list to choose for storage type and whatever value I put, it says "invalid selection". Please help. Found the following storage types: Select a storage type to create: 4 Invalid selection 4
k
Is this on local Orion or is this on Cloud 2.0?
x
local Orion
k
Are you on 2.0b3?
x
yes
let me check
2.0b2
k
Could you try 2.0b3 and see if it persists?
x
how can I upgrade 2.0b2 to b3?
k
pip install prefect==2.0b3
x
same command, got it. I thought we have any specific upgrade command.
Let me try
I am running (prefect dev UI with source code). Will this upgrade impact that dev UI?
a
when you run it in dev mode, you would also need to install it from source I think I had a similar issue when doing it while some session was already open in my browser. Can you try closing all Orion UI browser windows and then try creating storage and starting Orion?
x
sure
that trick worked. Thanks Anna.
🙌 1
@flow(name="XBCustom_Flow2") def custom_flow2(sql): logger = get_run_logger() log_message(sql) logger.debug("Entering Custom FLow") if not sql : sql = "SELECT count(*) from rpt_analytics.dcm_conversions_combined_p2c;" r1 = hello_world() Am I doing anything wrong in this code? I don't see the log in the UI.
k
How are you running it? Just by calling 
custom_flow2
?
x
DeploymentSpec( flow=custom_flow2, name="XBparent_flow4_deploy", tags=['waves_q'], parameters={'sql': 'SELECT count(*) from rpt_analytics.dcm_conversions_combined_p2c;'}, flow_runner=SubprocessFlowRunner(condaenv="/localpart0/aop-shared/WAVES/prefect2", stream_output=True) )
By deploy and run
k
Do you see logs that the flow started and ended but not logs from the logger? Or do you see 0 logs completely?
x
Yes, I see the logs (flow start and end etc., too) while running the flow in CLI using prefect deployment execute
Only in the UI I can't see it.
'log_message-718c3f46-0' - Finished in state Completed(None)
a
can you share your log_message() code?
x
@flow(name="XBCustom_Flow2") def custom_flow2(sql): logger = get_run_logger() log_message(sql) logger.debug("Entering Custom FLow") if not sql : sql = "SELECT count(*) from rpt_analytics.dcm_conversions_combined_p2c;" r1 = hello_world()
Output comes in the command line, but, not in the UI
Do I need to do any additional config?
k
Sorry I’m confused what you see in the UI. Could you share how it looks there?
a
you still shared custom_flow2 - I asked for log_message(sql) definition. Can you share the full flow code? hard to help otherwise
x
from prefect import task, flow from prefect.task_runners import DaskTaskRunner from prefect.deployments import DeploymentSpec from prefect.flow_runners import SubprocessFlowRunner import aops1.aopsconn.db_connect as dc import psycopg2 from prefect.tasks import task_input_hash from datetime import timedelta from prefect import get_run_logger from prefect.orion.schemas.schedules import RRuleSchedule from dateutil.rrule import rrule, MINUTELY, DAILY, MO, TU, WE, TH, FR import pendulum from prefect.orion.schemas.schedules import IntervalSchedule from datetime import timedelta import platform @task def say_hi(): logger = get_run_logger() logger.info("Hello world!") @task def print_platform_info(): logger = get_run_logger() logger.info( "Platform information: IP = %s, Python = %s, Platform type = %s, OS Version = %s", platform.node(), platform.python_version(), platform.platform(), platform.version(), ) @task def log_message(name): logger = get_run_logger() logger.info(f"Hello {name}!") return @flow(name="XBCustom_Flow2") def custom_flow2(sql): hi = say_hi() print_platform_info(wait_for=[hi]) logger = get_run_logger() log_message(sql) logger.debug("Entering Custom FLow") if not sql : sql = "SELECT count(*) from rpt_analytics.dcm_conversions_combined_p2c;" r1 = hello_world()
k
Thanks! I can try to replicate this. This is backed by Cloud 2.0 right?
x
All in local Linux Server. No cloud involved.
Prefect Orion 2.0
k
Ok will try
a
you may want to change the sql as a default argument on your flow function and refactor it this way:
Copy code
from prefect import task, flow
from prefect import get_run_logger
import platform


@task
def say_hi():
    logger = get_run_logger()
    <http://logger.info|logger.info>("Hello world!")


@task
def print_platform_info():
    logger = get_run_logger()
    <http://logger.info|logger.info>(
        "Platform information: IP = %s, Python = %s, Platform type = %s, OS Version = %s",
        platform.node(),
        platform.python_version(),
        platform.platform(),
        platform.version(),
    )


@task
def log_message(name):
    logger = get_run_logger()
    <http://logger.info|logger.info>(f"Hello {name}!")


@flow(name="XBCustom_Flow2")
def community_custom_flow2(
    sql: str = "SELECT count(*) from rpt_analytics.dcm_conversions_combined_p2c;",
):
    hi = say_hi()
    print_platform_info(wait_for=[hi])
    log_message(sql)
    logger = get_run_logger()
    logger.debug("Exiting FLow")


if __name__ == "__main__":
    community_custom_flow2()
all logs will appear in the UI - I think you are just a bit too fast 🙂
my debug log doesn't show up because my log level is set to INFO - you can change it using:
Copy code
export PREFECT_LOGGING_LEVEL=DEBUG
or even better:
Copy code
prefect config set PREFECT_LOGGING_LEVEL=DEBUG
x
Done the config. Executed again. Still the log is not coming in the UI.
But I can see the log in CLI
180616.670 | DEBUG | Task run 'log_message-718c3f46-0' - Beginning execution... 180616.671 | INFO | Task run 'log_message-718c3f46-0' - Hello SELECT count(*) from rpt_analytics.dcm_conversions_combined_p2c;! 180616.707 | INFO | Task run 'say_hi-93ad9977-0' - Finished in state Completed(None)
180616.746 | DEBUG | Flow run 'vivacious-dogfish' - Submitted task run 'prefect_say-6e4453ba-0' to task runner 180616.761 | DEBUG | Task run 'print_platform_info-340ece69-0' - Beginning execution... 180616.763 | INFO | Task run 'print_platform_info-340ece69-0' - Platform information: IP = dtord03aop02d.dc.dotomi.net, Python = 3.10.1, Platform type = Linux-3.10.0-1160.11.1.el7.x86_64-x86_64-with-glibc2.17, OS Version = #1 SMP Fri Dec 18 163456 UTC 2020 1
k
I can’t seem to replicate as well. I get logs in the UI and in the CLI
x
Is it something wrong with my UI Config?
BTW, I also don't see Add Work-Queue Button in my UI.
k
You don’t see these?
x
Correct
k
What do you mean when you said you used the dev UI?
x
I created the build from the source code and run it
k
Why did you do that, and when did you get your source? Are the logs missing for just this flow or for all flows? Because I think the logs were not there a couple of monthsa go
x
because I would like to access the UI using my hostname instead of using localhost
It was not supported in 2.0
k
Ahh I see. I would suggest you update the UI source again and try that?
x
What is the REST API equivalent to run "prefect deployment run flow_name/deploy_name"?
k
Do you need REST API? You can use the Orion client like this and it will hit the API for you
x
I have deployment spec in the python code itself and passing the file via blob in the REST API to deploy which is similar to "prefect deployment create", but the deployment_spec in the .py file is either not used or overwritten by the params in the REST API Payload.
BTW, we are using the REST APIS, left and right 🙂
we are good at it, but I see some discrepancies.
k
Uhh…that’s amazing…but not a pattern we recommend for Orion😅. Specifically because there is a lot of handling to shape stuff for the API request. You can trace the code from here though for the CLI command
x
@deployment_app.command()async def create(path: str): In the source code you showed, the above portion of the code is what I am talking about. How REST API runs this function internally?
k
I think you are more concerned about the post here
a
I answered already here - can we continue the discussion on Discourse?
it's definitely confusing to talk about the same on two channels (Slack and Discourse) at the same time
x
Yes, I will continue in the discourse. Sorry.
👍 1
Kevin/Ann, Still I can't see the log in the Orion UI. Are there any other config I miss, because of that the UI is not reading the log correctly from SQLite, or any config to write the log in SQLite to bring it in the UI?
BTW, I see the log data stored in the log table in orion.db
somehow the UI is not fetching it
a
can you share your flow code? if you use:
Copy code
from prefect import get_run_logger
it should work fine
if local is too hard to debug, you can use a free tier of Cloud 2.0 https://discourse.prefect.io/t/how-to-get-started-with-prefect-cloud-2-0/539
maybe it's a matter of a log level?
x
I tried all known possibilities, still I can't get the log in Orion UI.
Also the Flows Tab in the UI is not showing the list of flows but it shows only the count. But all other tables like Deployments, Flow Runs and Task Runs it shows the list when I click it.
I am getting "No logs to show". Even I tried all log levels
PREFECT_PROFILE='default' PREFECT_HOME='/localpart0/aop-shared/WAVES/prefect2' PREFECT_API_URL='http://dtord03aop02:5200/api' PREFECT_LOGGING_LEVEL='DEBUG' PREFECT_LOGGING_SETTINGS_PATH='${PREFECT_HOME}/lib/python3.10/site-packages/prefect/logging/logging.yml' PREFECT_ORION_DATABASE_CONNECTION_URL='sqlite+aiosqlite://///localpart0/aop-shared/WAVES/prefect2/orion.db' PREFECT_ORION_API_HOST='dtord03aop02' PREFECT_ORION_API_PORT='5200' PREFECT_AGENT_PREFETCH_SECONDS='10' PREFECT_AGENT_QUERY_INTERVAL='5.0' PREFECT_API_KEY='None' PREFECT_API_REQUEST_TIMEOUT='30.0' PREFECT_API_URL='None' PREFECT_CLOUD_URL='https://api-beta.prefect.io/api' PREFECT_DEBUG_MODE='False' PREFECT_HOME='/home/xbabu/.prefect' PREFECT_LOGGING_EXTRA_LOGGERS='' PREFECT_LOGGING_LEVEL='INFO' PREFECT_LOGGING_ORION_BATCH_INTERVAL='2.0' PREFECT_LOGGING_ORION_BATCH_SIZE='4000000' PREFECT_LOGGING_ORION_ENABLED='True' PREFECT_LOGGING_ORION_MAX_LOG_SIZE='1000000' PREFECT_LOGGING_SERVER_LEVEL='WARNING' PREFECT_LOGGING_SETTINGS_PATH='${PREFECT_HOME}/logging.yml' PREFECT_ORION_ANALYTICS_ENABLED='True' PREFECT_ORION_API_DEFAULT_LIMIT='200' PREFECT_ORION_API_HOST='127.0.0.1' PREFECT_ORION_API_PORT='4200' PREFECT_ORION_DATABASE_CONNECTION_TIMEOUT='5.0' PREFECT_ORION_DATABASE_CONNECTION_URL='sqlite+aiosqlite:////${PREFECT_HOME}/orion.db' PREFECT_ORION_DATABASE_ECHO='False' PREFECT_ORION_DATABASE_MIGRATE_ON_START='True' PREFECT_ORION_DATABASE_TIMEOUT='1.0' PREFECT_ORION_SERVICES_LATE_RUNS_AFTER_SECONDS='00005' PREFECT_ORION_SERVICES_LATE_RUNS_ENABLED='True' PREFECT_ORION_SERVICES_LATE_RUNS_LOOP_SECONDS='5.0' PREFECT_ORION_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE='100' PREFECT_ORION_SERVICES_SCHEDULER_ENABLED='True' PREFECT_ORION_SERVICES_SCHEDULER_INSERT_BATCH_SIZE='500' PREFECT_ORION_SERVICES_SCHEDULER_LOOP_SECONDS='60.0' PREFECT_ORION_SERVICES_SCHEDULER_MAX_RUNS='100' PREFECT_ORION_SERVICES_SCHEDULER_MAX_SCHEDULED_TIME='100 days, 00000' PREFECT_ORION_UI_ENABLED='True' PREFECT_PROFILES_PATH='${PREFECT_HOME}/profiles.toml' PREFECT_TEST_MODE='False'
prefect default settings
where prefect stores the default settings. I don't see it in the sqlite DB.
I also don't see option to create API KEY from the UI using Settings
Please help me to resolve these issues.
I don't think the DEV Orion UI code has write build.
a
Re: "all known possibilities" - Can you list what exactly have you tried so far?
Copy code
PREFECT_HOME='/localpart0/aop-shared/WAVES/prefect2'
PREFECT_ORION_DATABASE_CONNECTION_URL='sqlite+aiosqlite://///localpart0/aop-shared/WAVES/prefect2/orion.db'
It looks like you run Orion from some FTP location. This may be the source of your issue. Could you try running Orion from your local machine or use Cloud 2.0?
x
Anna, Thank you for your reply. Actually, it is not a FTP, it is a local storage mounted in our linux server.
I don't see any performance issue.
a
but this won't work due to WAL
x
I am setting the cloud as per your info. Let me update you soon. I am getting an error while giving the workspace during cloud login. (I have created a workspace already)
👍 1
Cloud is working fine for me. What is the constraint or limitation on using this Cloud? Can we use it for real business? Is there any cost involved?
Instead of using SQLite if I use Postgres DB, will this problem be resolved?
k
Cloud has no cost for now. The pricing model is not public yet. I don’t think your problem will be resolved by chaging the database because you have a UI issue right? Cloud would be a good option though for a remote UI
x
Can I get an example or tutorial to use Postgres, please?
k
You mean this ?
x
It helps. Thanks Kevin.
👍 1
The Cloud Prefect is only taking care of SQLite DB and update from my local server goes to the cloud? OR the entire workflow is running at the cloud?
k
If you are using the cloud endpoint, the database used is the cloud hosted one, but the workflow is running locally
x
I don't need to run "prefect orion start", only I need to run "prefect agent start"?
in my local machine
k
Yes exactly. Agent pointing to Cloud
x
the python code I deploy doesn't go to the cloud, however the log info goes to the cloud which may contain some of my code info during exceptions, right?
k
I guess that is possible yes. Would you like to turn off logging?
x
how to get the bearer token to call cloud prefect 2.0 API?
do we need to have that bearer token to call the APIs?
a
it depends on what you try to do. If you want to e.g. trigger a flow run from deployment via an API call, then yes, you would need to create an API key in the UI and then authenticate e.g. via terminal
x
Thank you, Anna! Thanks, Xavier Babu
23 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by