Aqib Fayyaz
03/02/2022, 11:35 AMTomer Cagan
03/02/2022, 12:25 PMAyah Safeen
03/02/2022, 1:53 PMpip install -U "prefect>=2.0a"
pip install -U "prefect==2.0a9"
Do anyone have any idea about this,,
ERROR: Could not find a version that satisfies the requirement prefect==2.0a9 (from versions: 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.5.5, 0.6.0, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.6.6, 0.6.7, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.9.2, 0.9.3, 0.9.4, 0.9.5, 0.9.6, 0.9.7, 0.9.8, 0.10.0, 0.10.1, 0.10.2, 0.10.3, 0.10.4, 0.10.5, 0.10.6, 0.10.7, 0.11.0, 0.11.1, 0.11.2, 0.11.3, 0.11.4, 0.11.5, 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.12.4, 0.12.5, 0.12.6, 0.13.0, 0.13.1, 0.13.2, 0.13.3, 0.13.4, 0.13.5, 0.13.6, 0.13.7, 0.13.8, 0.13.9, 0.13.10, 0.13.11, 0.13.12, 0.13.13, 0.13.14, 0.13.15, 0.13.16, 0.13.17, 0.13.18, 0.13.19, 0.14.0, 0.14.1, 0.14.2, 0.14.3, 0.14.4, 0.14.5, 0.14.6, 0.14.7, 0.14.8, 0.14.9, 0.14.10, 0.14.11, 0.14.12, 0.14.13, 0.14.14, 0.14.15, 0.14.16, 0.14.17, 0.14.18, 0.14.19, 0.14.20, 0.14.21, 0.14.22, 0.15.0, 0.15.1, 0.15.2, 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7, 0.15.8, 0.15.9, 0.15.10, 0.15.11, 0.15.12, 0.15.13, 1.0rc1, 1.0.0)
ERROR: No matching distribution found for prefect==2.0a9
Brett Naul
03/02/2022, 1:53 PMEven though the user didn't create them explicitly, the children tasks of a mapped task are first-class Prefect tasks. They can do anything a "normal" task can do, including succeed, fail, retry, pause, or skip.this seems kinda true, but when you try to raise PAUSE inside a mapped task it seems to just loop infinitely, whereas for a normal task you see
<Task: pause> is currently Paused; enter 'y' to resume:
. anyone have a strong opinion on whether this is a bug or just not a supported usage?
from prefect import Flow, task
from prefect.engine.signals import PAUSE
@task
def pause(i):
import prefect
if not prefect.context.get('resume'):
raise PAUSE("pausing")
return i
with Flow("f") as f:
# pause.map(i=[1]) # infinite loop
pause(i=1) # works normally
f.run()
Daniel Nilsen
03/02/2022, 2:59 PMStep 15/15 : RUN python /opt/prefect/healthcheck.py '["flow-data_transformation.py"]' '(3, 10)'
---> Running in 2853f6886cd5
File "/opt/prefect/healthcheck.py", line 17
def system_check(python_version: str):
^
SyntaxError: invalid syntax
Adi Gandra
03/02/2022, 4:16 PMRajan Subramanian
03/02/2022, 4:22 PMXavier Babu
03/02/2022, 5:55 PMRajan Subramanian
03/02/2022, 6:23 PMfrom prefect.orion.schemas.schedules import IntervalSchedule
schedule = IntervalSchedule(interval=timedelta(minutes=5))
@flow(
name="redis_to_postgres_pipeline", task_runner=DaskTaskRunner(), schedule=schedule
)
def run_flow(shell_task: str):
cmd = build_command(shell_task)
run_in_shell(command=cmd)
Getting an error, flow doesn't accept schedule as argument anymoreVamsi Reddy
03/02/2022, 7:42 PMAn error occurred (ThrottlingException) when calling the DeregisterTaskDefinition operation (reached max retries: 2): Rate exceeded
does anyone know why some of the runs failed?Chris Reuter
03/02/2022, 7:45 PMLeo Kacenjar
03/02/2022, 8:02 PMPatrick Tan
03/02/2022, 8:29 PMChristian Nuss
03/02/2022, 8:33 PMRunning
state, then we do a cancel via the API, which leaves them in Cancelling stateHedgar
03/02/2022, 9:31 PMflow.register(projectname, flowname e.t.c
Would it be necessary to do again on the command line prefect register...
before doing prefect agent local start
?Jason Noxon
03/02/2022, 9:40 PMMax Lei
03/03/2022, 4:03 AM"ephemeralStorage": {
"sizeInGiB": 200
}
Sudharshan B
03/03/2022, 4:38 AMDaniel Nilsen
03/03/2022, 9:53 AMlocal agent
. The fix for that was to run the agent from the folder of the flow. How do I fix this for the docker agent when the flow is dockerized?
Failed to load and execute flow run: ModuleNotFoundError("No module named 'parameters'")
Valantis Hatzimagkas
03/03/2022, 10:08 AMSen
03/03/2022, 10:28 AMNico Neumann
03/03/2022, 12:37 PMYas Mah
03/03/2022, 1:57 PM@task
def get_access_paths(base_path:Path):
return base_path
with Flow("flow") as flow:
base_path = Parameter("base_path", default=pathlib.Path(__file__).parent.parent.resolve())
data_access = get_access_paths(base_path)
files = [str(Path.joinpath(data_access, x)) for x in data_access.glob('*') if x.is_file()]
input = Parameter("input", default=files)
Jake
03/03/2022, 2:43 PMKeith Veleba
03/03/2022, 3:15 PMXavier Babu
03/03/2022, 3:52 PMDominick Olivito
03/03/2022, 3:54 PM/home/flex/.local/bin/prefect: line 3: import: command not found
/home/flex/.local/bin/prefect: line 4: import: command not found
/home/flex/.local/bin/prefect: line 5: from: command not found
/home/flex/.local/bin/prefect: prefect: line 7: syntax error near unexpected token `('
/home/flex/.local/bin/prefect: prefect: line 7: ` sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])'
it looks like it's finding and parsing the prefect
executable file but not running it with python
. when I run a local container using the image, i'm able to successfully call the command prefect
and run a flow. prefect
is in the path of active user (flex
).
i'm also able to run basic flows successfully on GKE using prefect's base image, so the issue is specific to our custom image. do you have any suggestions on what we can check in our custom image?Vadym Dytyniak
03/03/2022, 4:50 PMChristian Nuss
03/03/2022, 4:51 PMKubernetesRun
defininig the job_template
as a dict?Prasanth Kothuri
03/03/2022, 5:15 PM# upload to s3
write_to_s3 = S3Upload(
bucket=s3_bucket,
boto_kwargs=dict(
endpoint_url=os.getenv("s3_endpoint"),
aws_access_key_id=os.getenv("s3_access_key"),
aws_secret_access_key=os.getenv("s3_secret_key")
)
)
output = write_to_s3(results.to_csv(index=False), key=file_name)