Jacqueline Riley Garrahan
03/01/2022, 9:46 PMWesley Jin
03/01/2022, 11:45 PMflow.register
vs. on the run_config
object e.g. ECSRun
? If not, is there a preferred place to set it?soner
03/01/2022, 11:50 PMMax Lei
03/01/2022, 11:58 PMDockerRun
with LocalStorage
but I have a config.py
file that is not packaged in my package, but it is beside the flow.py
however seems that flow.py
was not able to access this. The working directory in docker is set to the same place where both config.py
and flow.py
. Is there a setting I can configure?Aqib Fayyaz
03/02/2022, 6:31 AMYas Mah
03/02/2022, 8:50 AMEdvard Kristiansen
03/02/2022, 10:44 AMAqib Fayyaz
03/02/2022, 11:35 AMTomer Cagan
03/02/2022, 12:25 PMAyah Safeen
03/02/2022, 1:53 PMpip install -U "prefect>=2.0a"
pip install -U "prefect==2.0a9"
Do anyone have any idea about this,,
ERROR: Could not find a version that satisfies the requirement prefect==2.0a9 (from versions: 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.5.5, 0.6.0, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.6.6, 0.6.7, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.9.2, 0.9.3, 0.9.4, 0.9.5, 0.9.6, 0.9.7, 0.9.8, 0.10.0, 0.10.1, 0.10.2, 0.10.3, 0.10.4, 0.10.5, 0.10.6, 0.10.7, 0.11.0, 0.11.1, 0.11.2, 0.11.3, 0.11.4, 0.11.5, 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.12.4, 0.12.5, 0.12.6, 0.13.0, 0.13.1, 0.13.2, 0.13.3, 0.13.4, 0.13.5, 0.13.6, 0.13.7, 0.13.8, 0.13.9, 0.13.10, 0.13.11, 0.13.12, 0.13.13, 0.13.14, 0.13.15, 0.13.16, 0.13.17, 0.13.18, 0.13.19, 0.14.0, 0.14.1, 0.14.2, 0.14.3, 0.14.4, 0.14.5, 0.14.6, 0.14.7, 0.14.8, 0.14.9, 0.14.10, 0.14.11, 0.14.12, 0.14.13, 0.14.14, 0.14.15, 0.14.16, 0.14.17, 0.14.18, 0.14.19, 0.14.20, 0.14.21, 0.14.22, 0.15.0, 0.15.1, 0.15.2, 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7, 0.15.8, 0.15.9, 0.15.10, 0.15.11, 0.15.12, 0.15.13, 1.0rc1, 1.0.0)
ERROR: No matching distribution found for prefect==2.0a9
Brett Naul
03/02/2022, 1:53 PMEven though the user didn't create them explicitly, the children tasks of a mapped task are first-class Prefect tasks. They can do anything a "normal" task can do, including succeed, fail, retry, pause, or skip.this seems kinda true, but when you try to raise PAUSE inside a mapped task it seems to just loop infinitely, whereas for a normal task you see
<Task: pause> is currently Paused; enter 'y' to resume:
. anyone have a strong opinion on whether this is a bug or just not a supported usage?
from prefect import Flow, task
from prefect.engine.signals import PAUSE
@task
def pause(i):
import prefect
if not prefect.context.get('resume'):
raise PAUSE("pausing")
return i
with Flow("f") as f:
# pause.map(i=[1]) # infinite loop
pause(i=1) # works normally
f.run()
Daniel Nilsen
03/02/2022, 2:59 PMStep 15/15 : RUN python /opt/prefect/healthcheck.py '["flow-data_transformation.py"]' '(3, 10)'
---> Running in 2853f6886cd5
File "/opt/prefect/healthcheck.py", line 17
def system_check(python_version: str):
^
SyntaxError: invalid syntax
Adi Gandra
03/02/2022, 4:16 PMRajan Subramanian
03/02/2022, 4:22 PMXavier Babu
03/02/2022, 5:55 PMRajan Subramanian
03/02/2022, 6:23 PMfrom prefect.orion.schemas.schedules import IntervalSchedule
schedule = IntervalSchedule(interval=timedelta(minutes=5))
@flow(
name="redis_to_postgres_pipeline", task_runner=DaskTaskRunner(), schedule=schedule
)
def run_flow(shell_task: str):
cmd = build_command(shell_task)
run_in_shell(command=cmd)
Getting an error, flow doesn't accept schedule as argument anymoreVamsi Reddy
03/02/2022, 7:42 PMAn error occurred (ThrottlingException) when calling the DeregisterTaskDefinition operation (reached max retries: 2): Rate exceeded
does anyone know why some of the runs failed?Chris Reuter
03/02/2022, 7:45 PMLeo Kacenjar
03/02/2022, 8:02 PMPatrick Tan
03/02/2022, 8:29 PMChristian Nuss
03/02/2022, 8:33 PMRunning
state, then we do a cancel via the API, which leaves them in Cancelling stateHedgar
03/02/2022, 9:31 PMflow.register(projectname, flowname e.t.c
Would it be necessary to do again on the command line prefect register...
before doing prefect agent local start
?Jason Noxon
03/02/2022, 9:40 PMMax Lei
03/03/2022, 4:03 AM"ephemeralStorage": {
"sizeInGiB": 200
}
Sudharshan B
03/03/2022, 4:38 AMDaniel Nilsen
03/03/2022, 9:53 AMlocal agent
. The fix for that was to run the agent from the folder of the flow. How do I fix this for the docker agent when the flow is dockerized?
Failed to load and execute flow run: ModuleNotFoundError("No module named 'parameters'")
Valantis Hatzimagkas
03/03/2022, 10:08 AMSen
03/03/2022, 10:28 AMNico Neumann
03/03/2022, 12:37 PMYas Mah
03/03/2022, 1:57 PM@task
def get_access_paths(base_path:Path):
return base_path
with Flow("flow") as flow:
base_path = Parameter("base_path", default=pathlib.Path(__file__).parent.parent.resolve())
data_access = get_access_paths(base_path)
files = [str(Path.joinpath(data_access, x)) for x in data_access.glob('*') if x.is_file()]
input = Parameter("input", default=files)
Yas Mah
03/03/2022, 1:57 PM@task
def get_access_paths(base_path:Path):
return base_path
with Flow("flow") as flow:
base_path = Parameter("base_path", default=pathlib.Path(__file__).parent.parent.resolve())
data_access = get_access_paths(base_path)
files = [str(Path.joinpath(data_access, x)) for x in data_access.glob('*') if x.is_file()]
input = Parameter("input", default=files)
Kevin Kho
03/03/2022, 2:04 PMYas Mah
03/03/2022, 2:49 PM