Vitaly Shulgin
09/28/2020, 8:39 AMAdrien Boutreau
09/28/2020, 8:52 AMJovan Visnjic
09/28/2020, 10:12 AMLocalResult
and exceptions. We use LocalResult
for checkpointing, so only failed tasks get executed on a workflow re-run. So far we execute the workflow without a server, using just the basic CLI. If LocalResult
exists, the task is skipped and just tasks with missing results get executed. We would also like to store error information on the disk. But if write exception's content as a LocalResult
through task's target
parameter, it gets treated as successful, because something exists at the result location, whatever that may be. Is there some established way of dumping errors to disk, so they don't just remain in the console's stdout
, but that at the same time they don't obstruct the logic of what is successful and what not?
Any ideas or comments appreciated.
JovanJonas Bernhard
09/28/2020, 2:36 PMfetch_feed_for_url
and then save them with save_feed_entries
(with trigger any_successful
). However, some of the urls might not work leading to all subsequent tasks mapping over the result (and with that the flow) also failing. What I'm looking for is to map only over "successful feeds_list
entries"
feeds_list = fetch_feed_for_url.map(url=feed_urls)
save_feed_entries = save_feed_entries_to_db.map(feed=feeds_list)
2. Is it possible to initialize a task from a Task class using EnvVarSecret
? For example, creating a Task using the Task Library often requires the credentials upfront. Are all the secrets required to be available at "flow building time" to work with Task classes?
fetch_mysql = MySQLFetch(
user=EnvVarSecret("USER)
password=EnvVarSecret("PASSWORD)
)
Thanks!Eric
09/28/2020, 7:20 PM{'src': 'dest'}
Von Christian Pizarro
09/28/2020, 9:16 PMEdCode
09/28/2020, 10:53 PMRob Fowler
09/29/2020, 7:03 AMRobin
09/29/2020, 9:36 AMINFO - prefect.Docker | Pushing image to the registry...
) I get a shutil PermissionError: [WinError 5] Access is denied
.
Has anybody had the same issues?
We are using Python 3.7.9 due to some issues with Python 3.8 of another third party package ...Ralph Willgoss
09/29/2020, 9:40 AMvisualize
command.
The issue is that individual individual_preprocessing
task should be done in parallel as well, however its not.
What have we have done wrong here in our task setup?Ralph Willgoss
09/29/2020, 10:14 AMRob Fowler
09/29/2020, 11:52 AMRob Fowler
09/29/2020, 11:58 AMRob Fowler
09/29/2020, 12:24 PMNuno Silva
09/29/2020, 12:29 PMapollo_url
like described here: https://docs.prefect.io/orchestration/server/deploy-local.html#ui-configuration
I do this in `~/.prefect/config.toml`:
[server]
[server.ui]
apollo_url="http://<server_url>:4200/graphql"
then: prefect server start
and it still uses the default localhost
hence I have to manually change it in the server UI.
Any idea why?Lars Corneliussen
09/29/2020, 12:46 PMMikael
09/29/2020, 2:13 PMJohnny
09/29/2020, 8:34 PMAndy Dyer
09/29/2020, 9:40 PMparamiko
which I am using for sftp. See below stack trace.
it makes sense and other people are having difficulties pickling the same object in different applications. Is the best bet to use ShellTasks to sftp?
<https://github.com/paramiko/paramiko/issues/753>
<https://stackoverflow.com/questions/39321606/cant-pickle-an-rsa-key-to-send-over-a-socket>
stack trace : https://pastebin.com/8hyAdpx1itay livni
09/29/2020, 9:59 PMMatt
09/29/2020, 10:36 PMJC Garcia
09/29/2020, 10:40 PMprefect create project "Test Project X"
I assume that under the hood prefect is hitting localhost:4200, is there any way to change that url?John Song
09/29/2020, 11:49 PMms16
09/30/2020, 3:05 AMENTRYPOINT prefect agent start --token $PREFECT_RUNNER_TOKEN -l fargate-dev
3. When I run the docker container locally - The Precct agent starts inside the container and can see the output [2020-09-30 02:53:34,447] INFO - agent | Waiting for flow runs...
4. But the flows registered in the project never start to execute
I have RTFM but still missing a link here 😕
Here is the code of my registered flow
import prefect
from prefect import task, Flow
@task
def hello_task():
print("hello")
flow = Flow("hello-flow", tasks=[hello_task])
flow.register(project_name="Demo",labels = "fargate-dev")
ms16
09/30/2020, 3:05 AMPrathamesh
09/30/2020, 8:46 AMRobin
09/30/2020, 9:10 AMmin_workers
and max_workers
as flow parameters, such that they can be changed from the cloud ui?
flow.environment = DaskKubernetesEnvironment(
min_workers=20, max_workers=30, labels=["k8s"]
)
Nuno Silva
09/30/2020, 12:08 PMtable_copy 0....table_copy n
would show table_copy table_name_1....table_copy table_name_n
? Thank youRobin
09/30/2020, 12:11 PMwrite_task_result_to_table
task that gets the result of other tasks and writes e.g. into a table whether the task was a success
, failed
, missing data
or empty (= task was not yet executed).
We have a flow that runs a several tasks for many systems.
So the table we would like to have as result would look like:
system id, task_1, task_2, task_3, ...
and then one row for each system.
Is this there already a more elegant way to write all the results of a flow into a table or something like this?
Or is the above described way the way to go?
Cheers 🙂Adam
09/30/2020, 12:35 PM