Marc Lipoff
12/18/2020, 5:29 PM/
needs to be removed.Pedro Machado
12/21/2020, 12:06 AMprefect server stop
command had been added. I lost the connection to the server. Is there a graceful way to stop it once it's running?
If I wanted to use the docker compose file directly to run server, what is the easiest way to set all the variables that are needed in this file?Joël Luijmes
12/21/2020, 3:36 PMFrank Cheng
12/21/2020, 6:59 PMRon Gross
12/22/2020, 8:03 AMwith case
and upstream_tasks
together
we have a set of 3 tasks that need to run one after the other (no values shared between them)
A -> B -> C
we also would like to have the ability to run only:
A -> C
that means that B
should go inside with case
, but then what happens is that C
is skipped, because it has upstream_tasks
related to B
any solutions available?Konstantin Tretiakov
12/22/2020, 8:26 AMnib
12/22/2020, 10:17 AMMassoud Mazar
12/24/2020, 12:05 AMNo module named 'prefect.engine.result_handlers'
on the agentJoël Luijmes
12/24/2020, 5:13 PMfblaze f
12/26/2020, 5:07 PM#!/usr/bin/python3
from prefect import task, Flow
import boto3
import json
client = boto3.client('sagemaker', region_name='us-east-1'
)
@task
def list_training_jobs():
response = client.list_training_jobs()
print(response)
with Flow("SageMakerFlow") as flow:
training_jobs = list_training_jobs()
print(training_jobs)
flow.register(project_name="tutorial")
flow.run()fblaze f
12/26/2020, 5:26 PMimport prefect
from prefect import task, Flow
@task
def say_hello():
logger = prefect.context.get("logger")
<http://logger.info|logger.info>("Hello, Cloud!")
with Flow("hello-flow") as flow:
say_hello()
# Register the flow under the "tutorial" project
flow.register(project_name="tutorial")
flow.run()
Matthew Blau
12/28/2020, 2:03 PMPedro Machado
12/31/2020, 3:00 AMPREFECT_SERVER__TELEMETRY__ENABLED=false
Thanks!fblaze f
12/31/2020, 5:40 PM#!/usr/bin/python3
from datetime import datetime
from prefect import task, Flow, Parameter, context, case
from prefect.utilities.debug import is_serializable
logger = context.get("logger")
timestamp = datetime.now().strftime("%Y-%m-%dT%H-%M-%S")
@task
def print_timestamp():
<http://logger.info|logger.info>(timestamp)
with Flow("Test") as flow:
print_timestamp()
serializable = is_serializable(flow)
if serializable:
flow.register(project_name="Default",
idempotency_key=flow.serialized_hash())
flow.run()
Attached screenshot from two runs. As you could see, the value of the global parameter timestamp
hasn't changedPedro Martins
01/04/2021, 12:46 PMkevin
01/04/2021, 9:18 PMIntervalClock
or a CronClock
? Which is more in line with what Prefect expects?jack
01/06/2021, 4:51 PMArnaud Fombellida
01/07/2021, 7:48 AMprefect backend server
, I start the server with prefect server start --postgres-port 4301 --hasura-port 4302 --graphql-port 4303 --ui-port 4304 --use-volume --volume-path /home/arnaud/mco/prefect/pgdata
(Mapping ports as some ports are already in use).
In the config.toml, I changed the following:
[server]
[server.ui]
apollo_url = "<http://192.168.176.193>:${server.host_port}/graphql"
After starting the server, all services seem to start properly and docker ps
indicates all services are up (and all ports mapped correctly). However, as you can see below, the are no logs for the apollo server.
Running prefect server create-tenant --name default --slug default
results in
requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.176.193', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4b14a1dc70>: Failed to establish a new connection: [Errno 111] Connection refused'))
and I cannot connect to http://192.168.176.193:4200/graphql (nor can the UI).
Can anyone help me with this ?Francisco Tanudjaja
01/07/2021, 8:38 PM$docker run -it prefecthq/prefect:latest bash
... within the container
root@8afe017ae8ef:/# prefect backend server
Backend switched to server
root@8afe017ae8ef:/# prefect server start
Exception caught; killing services (press ctrl-C to force)
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/cli/server.py", line 347, in start
["docker-compose", "pull"], cwd=compose_dir_path, env=env
File "/usr/local/lib/python3.7/subprocess.py", line 358, in check_call
retcode = call(*popenargs, **kwargs)
File "/usr/local/lib/python3.7/subprocess.py", line 339, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/local/lib/python3.7/subprocess.py", line 800, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python3.7/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'docker-compose': 'docker-compose'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/prefect", line 8, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/cli/server.py", line 385, in start
["docker-compose", "down"], cwd=compose_dir_path, env=env
File "/usr/local/lib/python3.7/subprocess.py", line 411, in check_output
**kwargs).stdout
File "/usr/local/lib/python3.7/subprocess.py", line 488, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/local/lib/python3.7/subprocess.py", line 800, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python3.7/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'docker-compose': 'docker-compose'
It's likely I'm missing something basic. Any tips appreciated. Thanks.Greg Roche
01/08/2021, 11:33 AM0.13.x
to 0.14.2
by running these commands:
pip install --upgrade prefect
prefect server stop
prefect server start --detach
Unfortunately, after the upgrade had finished all of the projects, registered flows, and flow run history have been lost. Is there any way to prevent this from happening after future upgrades?kevin
01/08/2021, 8:38 PMPostgresFetch
task to accept an optional boolean arg that indicates whether you want the column names returned with the query or not?Ajith Kumara Beragala Acharige Lal
01/11/2021, 11:56 AMscheduled
forever without getting triggered, , any idea?
import prefect
from prefect.tasks.shell import ShellTask
from prefect import task, Flow
task = ShellTask(helper_script="cd /prefect")
with Flow("ShellScriptTest") as f:
mv_file = task(command='cp -rf /prefect/test.py /prefect/kubernetes/')
out = f.register(project_name="ShellScriptTest")
Jeremy Phelps
01/11/2021, 6:38 PMagent@prefect-agent:~/.prefect/results$ prefect run flow --name 'My first Flow' --project test
Traceback (most recent call last):
File "/home/agent/.pyenv/versions/3.7.3/bin/prefect", line 10, in <module>
sys.exit(cli())
File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/cli/run.py", line 186, in flow
run_name=run_name,
File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/client/client.py", line 1094, in create_flow_run
res = self.graphql(create_mutation, variables=dict(input=inputs))
File "/home/agent/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/client/client.py", line 319, in graphql
raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'message': "Required parameters were not supplied: {'numbers'}", 'locations': [{'line': 2, 'column': 5}], 'path': ['create_flow_run'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': "Required parameters were not supplied: {'numbers'}"}}}]
I checked the Docker logs and found this GraphQL error:
GraphQL request:2:3
1 | mutation ($input: create_flow_run_input!) {
2 | create_flow_run(input: $input) {
| ^
3 | id
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 628, in await_result
return await result
File "/prefect-server/src/prefect_server/graphql/extensions.py", line 52, in resolve
result = await result
File "/prefect-server/src/prefect_server/graphql/runs.py", line 119, in resolve_create_flow_run
run_config=input.get("run_config"),
File "/prefect-server/src/prefect_server/api/runs.py", line 74, in create_flow_run
run_config=run_config,
File "/prefect-server/src/prefect_server/api/runs.py", line 215, in _create_flow_run
raise ValueError(f"Required parameters were not supplied: {missing}")
ValueError: Required parameters were not supplied: {'numbers'}
Eric Sawler
01/12/2021, 2:49 PMJordan Stewart
01/12/2021, 3:48 PMFROM python:3
ADD hello_world/hello_world.py /~
RUN pip3 install prefect
RUN pip3 install numpy
RUN pip3 install pandas
CMD python3 /hello_world/hello_world.py
this is an example, but do the dockerfile's need a CMD line since Prefect Agent is what handles running the code? it seems like we would just need ADD and RUN statements, and then Prefect agent would handle executionRob Fowler
01/13/2021, 2:43 AMJC Garcia
01/13/2021, 10:26 PMnow = datetime.datetime.utcnow()
dev_clock = clocks.IntervalClock(start_date=now,
interval=datetime.timedelta(minutes=10),
parameter_defaults={"some_prop": "someval"})
schedule = Schedule(clocks=[dev_clock])
flow.schedule = schedule
and it works! the problem is that it only schedules 10 runs, and no more. Any pointers?Jack Sundberg
01/14/2021, 6:11 PMRuslan
01/15/2021, 10:09 AMRuslan
01/15/2021, 10:09 AMGreg Roche
01/15/2021, 10:22 AMLocalExecutor
) doesn't run tasks in parallel. Have you set the executor for your flow to a Dask executor? https://docs.prefect.io/orchestration/tutorial/flow_config.html#enable-parallel-execution If you've done that and the tasks still aren't running in parallel then you probably also need to change your flow's logic, most likely by mapping over the tasks: https://docs.prefect.io/core/concepts/mapping.htmlRuslan
01/15/2021, 10:28 AM