Vaibhav Ariyur
01/05/2022, 4:26 PMMargaret Walter
01/06/2022, 5:01 PMCorris Randall
01/06/2022, 6:26 PMElliot Oram
01/07/2022, 12:21 PMFailed to load and execute Flow's environment: ModuleNotFoundError("No module named '/home/ubuntu/'")
The agent is running in a python virtual environment (I have also tried this without the venv and the result is the same) the agent is started with the following command:
prefect agent local start --api <http://my-server.com:4200>
Any ideas on what might be the cause, or how to start debugging this?Pedro Martins
01/07/2022, 1:30 PMclient.create_flow_run(flow_id, parameters)
and flow.run(parameters)
. Are they equivalent?
Would the flow.run
connect to the client and run the latest version of the flow?Carrie Bohinc
01/07/2022, 5:44 PMprefect-server-postgres
and the prefect-server-postgres-headless
services running. I was under the impression that the helm chart would automatically start a postgres instance to use if you weren't connecting to an outside db. Did I miss a step somewhere? Any guidance is appreciated.Cristian Toma
01/11/2022, 8:14 AMstate_handler
for a flow. When running locally in the new_state
object the result
property has values in it, but when I run it on the cluster, the result
property is empty. Does someone know why is that empty, and how can I get the errors of a flow run in the state handler on a cluster env?
Thanks.Tony Liberato
01/11/2022, 3:03 PMWill Skelton
01/12/2022, 4:30 PMStƩphan Taljaard
01/13/2022, 5:17 AMSam Werbalowsky
01/13/2022, 9:59 PM<https://data:XXXXXXXXXXXX@https>://<my-repo>/data/scratch-data.git
Itās unexpected to see that <https://data>ā¦@
portionDenys Volokh
01/14/2022, 2:25 PMBradley Hurley
01/14/2022, 3:22 PMhost.docker.internal
. I dug into the source and realize the IP only gets set properly if the sys.platform
starts with linux
. My host machine is OSX, but docker is running on Linux. Here is my prefect agent start command
prefect agent docker start -b <tcp://192.168.64.2:2375> -l bh-docker --network prefect-server --log-level DEBUG --show-flow-logs --no-pull
Ahmed Ezzat
01/16/2022, 4:34 AMtask = mytask(...)
other1 = other1.map(task)
other2.map(other1)
Pierre Monico
01/17/2022, 10:09 AMSamuel James
01/17/2022, 11:24 AMSantiago Gonzalez
01/17/2022, 4:54 PMec2_instances
and others). In ec2_instances.py
there are 3 task methods, and in this way it works. However, since I made a refactor in that file in order to add some behavior I needed, itās failing. I added some non-task methods that are going to be invoked in the existing task methods. The error that is being thrown is:
Failed to load and execute Flow's environment: StorageError('An error occurred while unpickling the flow:\n AttributeError("Can\'t get attribute \'get_boto_client_with_auth\' on <module \'...ec2_instances\' from \'/usr/local/lib/python3.7/site-packages/automation_library/ec2_instances.py\'>")\nThis may be due to one of the following version mismatches between the flow build and execution environments:\n - cloudpickle: (flow built with \'1.6.0\', currently running with \'2.0.0\')\n - python: (flow built with \'3.7.9\', currently running with \'3.7.12\')')
Why is it failing only when I add non-task methods? Does it make any sense?Michael Moscater
01/17/2022, 7:37 PMCƓme Arvis
01/18/2022, 11:57 AMUnexpected error: AttributeError("partially initialized module 'prefect' has no attribute 'schedules' (most likely due to a circular import)")
during our flows executions.
This seems to occur randomly between tasks executions.
Is there some known issue about it? Thanks!Pierre Monico
01/18/2022, 1:02 PMThomas Mignon
01/18/2022, 1:42 PM(base) thomas@Ifremer ī° ~ ī° docker -v
Docker version 20.10.8, build 3967b7d
(base) thomas@Ifremer ī° ~ ī° docker-compose --version
docker-compose version 1.25.0, build unknown
I'm trying to start prefect with the command prefect server start
but i obtain this error :
ERROR: Couldn't connect to Docker daemon at <http+docker://localhost> - is it running?
If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
Exception caught; killing services (press ctrl-C to force)
ERROR: Couldn't connect to Docker daemon at <http+docker://localhost> - is it running?
If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
Traceback (most recent call last):
File "/home/thomas/miniconda3/lib/python3.8/site-packages/prefect/cli/server.py", line 623, in start
subprocess.check_call(
File "/home/thomas/miniconda3/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker-compose', 'pull']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/thomas/miniconda3/bin/prefect", line 10, in <module>
sys.exit(cli())
File "/home/thomas/miniconda3/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/thomas/miniconda3/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/thomas/miniconda3/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/thomas/miniconda3/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/thomas/miniconda3/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/thomas/miniconda3/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/thomas/miniconda3/lib/python3.8/site-packages/prefect/cli/server.py", line 664, in start
subprocess.check_output(
File "/home/thomas/miniconda3/lib/python3.8/subprocess.py", line 411, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/home/thomas/miniconda3/lib/python3.8/subprocess.py", line 512, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['docker-compose', 'down']' returned non-zero exit status 1.
Boggdan Barrientos
01/18/2022, 7:10 PMpip install prefect
Inside python idle I was able to run successfully:
import prefect
from prefect import task, Flow
@task
def hello_task():
logger = prefect.context.get("logger")
<http://logger.info|logger.info>("Hello world!")
with Flow("hello-flow") as flow:
hello_task()
flow.run()
But in the cli, I try to run and nothing.
I try to run in the cli prefect version
and results.
ubuntu@ip-20-0-0-33:~$ prefect version
prefect: command not found
I dont know what I'm doing wrong. I would appreciate your help. Thanks.Will Skelton
01/19/2022, 2:19 AMPaulo Benatto
01/19/2022, 3:04 PMclient = Client(api_token=token)
client.register(flow, project_name=project)
from prefect import config
# this looks wrong
config.cloud.agent.auth_token = token
LocalAgent(name=name).start()
using command line to start agent works fine:
⢠prefect auth login -t TOKEN
⢠prefect agent local start
I would like to run the agent from python and inject the token in my agent š¤
thanksMaximilian Rausch
01/20/2022, 3:26 AMMoss Ebeling
01/20/2022, 6:33 AMfrom common_etl.data_sources import fetch_sales
from common_etl.util import enrich_with_user_details
# in flows/agg_sales.py
@task
def aggregate_sales():
sales = fetch_sales()
enriched_sales = enrich_with_user_details(sales)
# in flows/publish_sales_report.py
@task
def publish_to_wiki():
sales = fetch_sales()
sales.to_wiki()
We are using docker storage when deploying flows and copy the relevant package into the docker image and pip install it using the final commands options. Doing this means that we're able to deploy and run using a LocalExecutor
that has the correct version of our package in its execution environment.
However, since our jobs are large and can exhaust the resources on a single host, we're trying to use the DaskExecutor
instead, and point to a multi-host cluster. At this point we see failures because the dask workers in that cluster do not have this package installed in their execution environments. If we were to try to install it in their environments it would require stopping dask workers, installing the package in their environment, then restarting them. This would also seem to restrict all deployed flows to be using the exact same release of the common package (since the dask workers can only have one version installed?).
Has anyone had similar experience dealing with docker storage and dask executors or suggestions on this design?Raviraj Dixit
01/20/2022, 7:15 AMVaibhav Shetye
01/20/2022, 12:22 PMGian Piero Izzo
01/20/2022, 4:22 PMstorage = Git(
repo="org/repo",
flow_path="flows/my_flow.py"
)
The problem is that the docker agent is not able to get the repository, it gives the error:
Failed to load and execute Flow's environment: NotGitRepository('No git repository was found at https://@github.com/org/repo.git')
is there any way to avoid this "@" ?
Thanks in advanceChristopher Chong Tau Teng
01/21/2022, 2:41 AM