Hwi Moon
01/11/2022, 6:09 PMDaniel Kornhauser
01/11/2022, 6:13 PMquery {
flow_run{
logs {
message
level
info
created
}
}
}
Of course a totally different query to get error logs from graphql would also be welcomedMatthew Seligson
01/11/2022, 7:18 PMHeeje Cho
01/11/2022, 9:51 PMLeon Kozlowski
01/11/2022, 10:31 PMSidney Fox
01/11/2022, 11:13 PMFailed to scan table: Unable to locate credentials
I have AWS_CREDENTIALS
stored in Prefect cloud as a Secret, and I’ve tried passing credentials as envs passed to KuberenetesRun:
env={
"ACCESS_KEY": Secret("AWS_CREDENTIALS").get().get("ACCESS_KEY"),
"SECRET_ACCESS_KEY": Secret("AWS_CREDENTIALS").get().get("SECRET_ACCESS_KEY")
}
Returns the same error. What’s the best / preferred approach to authenticate a Kubernetes agent against AWS?Jeff Wiens
01/12/2022, 12:42 AMM. Siddiqui
01/12/2022, 8:33 AMDekel R
01/12/2022, 10:07 AMif pendulum.today('America/New_York').weekday() == 2: # Monday is 0 so Wednesday is 2
x_flow = create_flow_run(flow_name=PREFECT_TRAIN_FLOW_NAME, project_name=PREFECT_TRAIN_PROJECT_NAME)
wait_for_flow_a = wait_for_flow_run(x_flow, raise_final_state=True)
This code is of course inside my “with Flow….” code block.
Now when running this code alone (in a dummy flow) - it works and x_flow gets invoked.
But when running this code in my real flow, after some other tasks - nothing happens.
I cannot even see the task of “wait_for_flow” in prefect cloud(flow -> tasks tab) - seems like its getting ignored.
What am I missing here?
ThanksRam Vuppaladadiyam
01/12/2022, 12:44 PMTony Liberato
01/12/2022, 2:50 PMJustin Green
01/12/2022, 3:00 PMRouven
01/12/2022, 4:11 PMQwame
01/12/2022, 4:21 PMSuresh R
01/12/2022, 4:57 PMJason May
01/12/2022, 5:30 PMDidier Marin
01/12/2022, 5:32 PMJawaad Mahmood
01/12/2022, 6:46 PM### THIS CODE ALLOWS ME TO BIND A LOCAL FILE PATH
from prefect.run_configs import DockerRun
import docker
flow.run_config = DockerRun(labels=['my-label']
,host_config={'mounts':[docker.types.Mount(target='/public'
,source=r'//c/some/local/path'
,type='bind')
]
}
)
### THIS CODE THROWS ERROR
flow.run_config = DockerRun(labels=['my-label']
,host_config={'mounts':[docker.types.Mount(target='/public'
,source=r'\\path\to\windows\network\shared\drive'
,type='bind')
]
}
)
Chris Reuter
01/12/2022, 7:57 PMFilipe Reis
01/12/2022, 8:32 PMChristoph Deil
01/12/2022, 9:39 PMschedule = IntervalSchedule(interval=datetime.timedelta(seconds=10))
deployment_spec = DeploymentSpec(name="hola", flow=greetings_flow, schedule=schedule)
Do I now use OrionClient and some methods to deploy?
We currently use Prefect core in a pod and simply do flow.run() with a schedule attached, and I’m looking for a working example to do the equivalent in Orion (even if I gather behind the scenes it will do something else via a server and DB).
Basically I’m looking for this: https://orion-docs.prefect.io/concepts/deployments/#running-deployments-with-the-api
🙂Daniel Komisar
01/12/2022, 10:10 PMcreated
(or any other field) guaranteed to be returned in the same order? Thanks!Jason Motley
01/12/2022, 10:54 PMdf_2015 = extract_past(connection, start_date="2015-01-01", end_date = "2015-12-31", task_args={"name": "Extract 2015"})
Anh Nguyen
01/13/2022, 12:59 AMAnh Nguyen
01/13/2022, 5:37 AMAqib Fayyaz
01/13/2022, 7:53 AMJamie McDonald
01/13/2022, 11:40 AMmap()
functionality? My scenario is a list of URLs that should be used for making requests to but want to perform the requests in batches of 'n' rather than overwhelming a server with them all at once.Martim Lobao
01/13/2022, 11:49 AMtask_run_name
and name
args in the task
decorator to provide more context, but neither works the way I’d like it to.
as a MWE, here’s a sketch of what I’d like to happen:
with Flow() as flow:
task_a(entity="foo") # shows up as "foo_task" in the DAG
task_a(entity="bar") # shows up as "bar_task" in the DAG
the issue is that name
only takes in static strings (so name="{entity}"
doesn’t work) and task_run_name
only sets the task run name, meaning it will never show up in the schematic outside of flow runs (even in flow runs, the name is only shown when clicking on each individual task card, making it hard to see an overall picture). is there any way to achieve what i’d like to do?davzucky
01/13/2022, 12:32 PM@task
def list_file_from_storate(prefix: str) -> List[str]:
result = prefect.context["result"]
return result.list_files(prefix)
Looking at the context doc I cannot see the result been available here https://docs.prefect.io/api/latest/utilities/context.html
We we want is to be able to interact with the result using task from the flow setup.Ahmed Rafik
01/13/2022, 12:39 PMimport pandas as pd
from prefect.deployments import DeploymentSpec
from prefect.orion.schemas.schedules import IntervalSchedule
from pred.predict import predict_flow
DeploymentSpec(
flow=predict_flow,
name="Prediction"
)
I create the deployment without a problem and I can see it in the UI. I also can run it successfully from CLI using:
prefect deployment execute 'Prediction flow/Prediction'
But When I try to run it using the “Quick Run” in the UI, a “Scheduled” task is created and is never run. I can see it in the lateness graph. one task didn’t run for over an hour during my lunch break. Same happens if I add a schedule to the deployment. any ideas why that happens or how to fix it?