Hi guys I ve installer docker and docker compose ``` base th

Question

Hi guys I ve installer docker and docker compose ``` base thomas Ifremer ~ docker v Docker version 20 10 8 build 3967b7d base thomas Ifremer ~ docker compose version docker compose version 1 25 0 build unknown``` I m trying to start prefect with the command `prefect server start ` but i obtain this error ```ERROR Couldn t connect to Docker daemon at is it running If it s at a non standard location specify the URL with the DOCKER HOST environment variable Exception caught killing services press ctrl C to force ERROR Couldn t connect to Docker daemon at is it running If it s at a non standard location specify the URL with the DOCKER HOST environment variable Traceback most recent call last File home thomas miniconda3 lib python3 8 site packages prefect cli server py line 623 in start subprocess check call File home thomas miniconda3 lib python3 8 subprocess py line 364 in check call raise CalledProcessError retcode cmd subprocess CalledProcessError Command docker compose pull returned non zero exit status 1 During handling of the above exception another exception occurred Traceback most recent call last File home thomas miniconda3 bin prefect line 10 in lt module gt sys exit cli File home thomas miniconda3 lib python3 8 site packages click core py line 829 in call return self main args kwargs File home thomas miniconda3 lib python3 8 site packages click core py line 782 in main rv = self invoke ctx File home thomas miniconda3 lib python3 8 site packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File home thomas miniconda3 lib python3 8 site packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File home thomas miniconda3 lib python3 8 site packages click core py line 1066 in invoke return ctx invoke self callback ctx params File home thomas miniconda3 lib python3 8 site packages click core py line 610 in invoke return callback args kwargs File home thomas miniconda3 lib python3 8 site packages prefect cli server py line 664 in start subprocess check output File home thomas miniconda3 lib python3 8 subprocess py line 411 in check output return run popenargs stdout=PIPE timeout=timeout check=True File home thomas miniconda3 lib python3 8 subprocess py line 512 in run raise CalledProcessError retcode process args subprocess CalledProcessError Command docker compose down returned non zero exit status 1 ```

Anna Geller · Answer

looks like you need to start your docker btw can you move the code blocks into the main thread to keep the channel cleaner

Thomas Mignon · Answer

Thanks for the fast reply I have something particular to do in order to start it They are not in the main thread already

Pierre Monico · Answer

Try this <https docs docker com engine install linux postinstall manage docker as a non root user>

Thomas Mignon · Answer

Hi Pierre thanks a lot it seems to work slightly smiling face

Thomas Mignon · Answer

I have another question Is it possible to have the interface on my PC then acceed in ssh to the HPC from the UI

Kevin Kho · Answer

If you mean connecting to the server from a remote machine through the UI yes you can You just need to point your UI to the GraphQL API

Thomas Mignon · Answer

Have you a link in the docs who treat of this slightly smiling face

Thomas Mignon · Answer

I have another more question see no evil If i want to launch a flow without the UI without filling the inputs forms how can i pass the parameters `workspace command semaphore audience version command parameter` then retrieving it in the code when i m launching the following prefect code with this command `python myflow py` ```import time import prefect from dask jobqueue import PBSCluster from prefect executors import DaskExecutor from prefect import Flow task Parameter task def template runner workspace command semaphore audience version command parameter workspaceVar Path = Path os path join workspace f command os environ PBS JOBID template runner TemplateRunner command=command version=semaphore audience version command parameter=command parameter workspace=workspaceVar skip signal=True run exit=False with Flow template runner flow as flow worskpace = Parameter worskpace default= command = Parameter command default= semaphore audience version = Parameter people default= command parameter = Parameter people default= Map `say hello` across the list of names template runner map worskpace command semaphore audience version command parameter if name == main flow run executor=DaskExecutor cluster class=PBSCluster cluster kwargs= cores 1 memory 24GB queue sequentiel walltime 00 05 00 interface ib0 n workers 1 ```

Kevin Kho · Answer

For the connection check <https medium com the prefect blog prefect server 101 deploying to google cloud platform 47354b16afe2|this blog> The last section after `prefect server start` shows how to connect You need to edit the `config toml` on the machine you are connecting from To open outside connections you need to start it with `prefect server start expose`

Kevin Kho · Answer

If you are mapping and something stays the same throughout the mapped calls you need to wrap it with `unmapped`

Kevin Kho · Answer

As long as downstream tasks use the Parameter populating them in the UI will populate them during runtime

Thomas Mignon · Answer

Thanks a lot for the doc about the connection i will check this out just after i ve resolved my parameters issue slightly smiling face If i m understanding correctly i have to use `unmapped` to pass multiple arg to the mapping

Kevin Kho · Answer

Check <https docs prefect io core concepts mapping html unmapped inputs|this> for unmapped

Thomas Mignon · Answer

Then how can i pass the args on the start via `python myflow py` I was thinking of a syntax with something like `python myflow py workspace= myworkspace command= mycommand `

Kevin Kho · Answer

`prefect run param workspace=myworkspace` I think

Kevin Kho · Answer

More info <https docs prefect io api latest cli run html run 2|here>

Thomas Mignon · Answer

Ok thank you a lot for all the info i will try that stuck out tongue

Thomas Mignon · Answer

Hi < Kevin Kho> I m struggling retrieving my parameters passed on the start command who is `python home1 datawork semexp workspace common bin prefectflow py param workspace= home1 datawork semexp workspace dagit running param command=purify raw logs param command parameter= home datawork semaphore exp spool input ftp ftp2 2021 ftp2 f access log 20211201 gz param version=3 4 9` When i m printing the parameter workspace in my flow for example ```with Flow template runner flow as flow workspace = Parameter workspace print workspace workspace dict ``` I obtain this ```workspace required True default None name workspace slug workspace logger lt Logger prefect workspace INFO gt tags set max retries 0 retry delay None timeout None trigger lt function all successful at 0x7fffeaefde50 gt skip on upstream skip True cache for None cache key None cache validator lt function never use at 0x7fffeb3eeaf0 gt checkpoint True result lt PrefectResult None gt target None task run name None state handlers auto generated False log stdout False nout None ``` So the param doesn t seems to be passed correctly

Anna Geller · Answer

You don t need to build your own CLI arg parser to do that because already provides that functionality You can use ```prefect run name yourflow param name=Marvin watch``` Also check out this about parameters

Thomas Mignon · Answer

Thanks a lot < Anna Geller> for the blog post that was exactly what i was searching for

Thomas Mignon · Answer

Do you know DaskExecutor a little Here s how my flow looks like now ```import time import prefect import os from pathlib import Path from dask jobqueue import PBSCluster from prefect executors import DaskExecutor from prefect import Flow task Parameter unmapped task log stdout=True def template runner workspace command semaphore audience version command parameter print workspace command workspaceVar Path = Path os path join workspace f command os environ PBS JOBID template runner TemplateRunner command=command version=semaphore audience version command parameter=command parameter workspace=workspaceVar skip signal=True run exit=False with Flow template runner flow as flow workspace = Parameter workspace command = Parameter command semaphore audience version = Parameter semaphore audience version command parameter = Parameter command parameter template runner workspace=workspace command=command semaphore audience version=semaphore audience version command parameter=command parameter if name == main flow run workspace= test executor=DaskExecutor cluster class=PBSCluster cluster kwargs= cores 1 memory 24GB queue sequentiel walltime 00 05 00 interface ib0 n workers 1 ``` The comportment i should have is that the daskExecutor should have submit to my PBSCluster a qsub with the cluster kwargs isn t it I got this error because no qsub was done so no PBS JOBID is set as env var ``` 09 06 20 | ERROR | Task template runner Exception encountered during task execution Traceback most recent call last File home datawork semaphore exp conda conda envs prefect lib python3 9 site packages prefect engine task runner py line 876 in get task run state value = prefect utilities executors run task with timeout File home datawork semaphore exp conda conda envs prefect lib python3 9 site packages prefect utilities executors py line 467 in run task with timeout return task run args kwargs type ignore File home1 datawork semexp workspace common bin prefectflow py line 16 in template runner os path join workspace f command os environ PBS JOBID File home datawork semaphore exp conda conda envs prefect lib python3 9 os py line 679 in getitem raise KeyError key from None KeyError PBS JOBID ```

Thomas Mignon · Answer

I launch my flow with prefect CLI with ```prefect run path home1 datawork semexp workspace common bin prefectflow py param workspace= home1 datawork semexp workspace dagit running param command= purify raw logs param command parameter= home datawork semaphore exp spool input ftp ftp2 2021 ftp2 f access log 20211201 gz param semaphore audience version= 3 4 9 watch```

Anna Geller · Answer

< Thomas Mignon> your flow definition looks fine I usually see people using DaskExecutor when they leverage mapping i e when they parallelize the execution of a function over a list of various inputs But even without mapping the executor should submit the task to he cluster It s hard to give any suggestions since I can t reproduce this myself locally shares how one user has done it with HPC cluster perhaps it gives you some info A good way to debug this would be to start with something simpler Can you try running this simple flow on your PBS cluster Does this work ```from prefect import task Flow from prefect executors import DaskExecutor from dask jobqueue import PBSCluster task def generate random numbers return list range 1 200 task def add one x return x + 1 task log stdout=True def print results res print res with Flow mapping executor=DaskExecutor cluster class=PBSCluster cluster kwargs= cores 1 memory 24GB queue sequentiel walltime 00 05 00 interface ib0 n workers 1 as flow numbers = generate random numbers result = add one map numbers print results result ```

Thomas Mignon · Answer

Hi < Anna Geller> The little code you gave me has worked a qsub was created on my PBS Cluster slightly smiling face

Anna Geller · Answer

that s amazing I guess this might be then an issue with the template runner class Perhaps some statefulness issue bound to using classes in parallel processing

Thomas Mignon · Answer

I ve commented the line with template runner and the problem remain the same

Thomas Mignon · Answer

But if i use the way you write the flow it s good slightly smiling face

Thomas Mignon · Answer

```import time import prefect import os from pathlib import Path from semaphore scripts import template runner from dask jobqueue import PBSCluster from prefect executors import DaskExecutor from prefect import Flow task Parameter unmapped task log stdout=True def template runner task workspace command semaphore audience version command parameter workspaceVar Path = Path os path join workspace f command os environ PBS JOBID os path join workspace f command template runner TemplateRunner command=command version=semaphore audience version command parameter=command parameter workspace=workspaceVar skip signal=True run exit=False with Flow template runner flow executor=DaskExecutor cluster class= dask jobqueue PBSCluster cluster kwargs= cores 1 memory 24GB queue sequentiel walltime 00 05 00 interface ib0 n workers 1 as flow workspace = Parameter workspace command = Parameter command semaphore audience version = Parameter semaphore audience version command parameter = Parameter command parameter template runner task workspace=workspace command=command semaphore audience version=semaphore audience version command parameter=command parameter ``` So my flow when looks like that start a qsub

Thomas Mignon · Answer

When the executor is passed as args on the flow

Thomas Mignon · Answer

So this is a great news it works stuck out tongue

Thomas Mignon · Answer

Thank you a lot < Anna Geller>

Kevin Kho · Answer

< An Hoang> you were asking who else is on HPC

An Hoang · Answer

< Thomas Mignon> I m making a list of people who are on HPC to eventually write a blogpost about HPC + Prefect It seems like you are on PBS I m on LSF transitioning to SLURM Can I DM you