Evan Curtin
10/19/2022, 2:57 PMResult
in 2.6+ ? I am thinking of passing data between tasks, persisted into remote storage as parquet, for example (e.g. using spark)redsquare
10/19/2022, 3:13 PMKelvin DeCosta
10/19/2022, 4:16 PMcreate_deployments
script multiple times and overwrite the deployments (similar to blocks).
I'm currently using the .build_from_flow
and .apply
methods but I keep running into warningsRahul Kadam
10/19/2022, 4:33 PMClaire Herdeman
10/19/2022, 4:42 PMDavid Beck
10/19/2022, 5:10 PMKubernetesJob.job_from_file
function to read a yaml file with our full k8s manifest. The function works, however I noticed that we have fields that get overwritten with default values in this call for _shortcut_customizations
in build_job
, specifically the namespace
and image
properties. I know that I can use an infrastructure_override
when setting up the deployment, however there should be option to simply read a yaml file with those fields.Erik Amundson
10/19/2022, 5:11 PMJehan Abduljabbar
10/19/2022, 5:22 PMMatt Denno
10/19/2022, 6:53 PMprefecthq/prefect:2.6.1-python3.10-conda
as a base image. Couple of questions:
• When installing a python environment in the container does it mater what it is called? I see the image has a base
env and a prefect
env. Should packages be installed in prefect
env of can I create a new one?
• If I create a new env should the Dockerfile include a line to activate the correct env.?
• Is there a recipe for creating a custom image from a conda base image? I looked around and did find one, but maybe I missed it?Matt Denno
10/19/2022, 7:05 PMdeploy_import_1 = Deployment.build_from_flow(
flow=import_1,
name="import1",
work_queue_name="imports",
schedule=IntervalSchedule(
interval=timedelta(hours=1),
anchor_date=datetime(2021, 1, 1, 2, 30, tzinfo=pytz.UTC)
),
infrastructure=DockerContainer(
image="path_to_image/import-prefect:python3.10",
image_pull_policy="NEVER",
auto_remove=False
)
)
Do I need to include remote storage to test? I am a bit unclear on all the pieces of the puzzle still.Kalise Richmond
10/19/2022, 7:06 PMMichał Augoff
10/19/2022, 7:43 PMscikit-learn
providing a flow to train a machine learning model and you can deploy it with your own model class like
Deployment.build_from_flow(flow=sklearn.flows.training, …, parameters={"model": SomePythonClass})
Is that even feasible?
Trevor Kramer
10/19/2022, 7:51 PMfrom prefect import flow, task
@task
def task1(x: int) -> int:
return x + 10
@task
def task2(x: int) -> int:
return -x
@flow()
def run_my_flow(n: int):
task2.map(task1.map(range(n)))
if __name__ == "__main__":
n = 500
print(run_my_flow(n))
David Beck
10/19/2022, 8:13 PMis_anonymous
set to True
. Is it important to also set the flag for overwrite=True
if the anonymous is one that is saved numerous times ala a CI/CD process?Atul Vig
10/19/2022, 8:44 PMDavid Beck
10/19/2022, 9:39 PMMichał
10/19/2022, 10:40 PMJohn Ramey
10/20/2022, 1:19 AMTypeError: task() got an unexpected keyword argument 'nout'
— is the nout
concept longer required?Thomas Pedersen
10/20/2022, 7:27 AMVadym Dytyniak
10/20/2022, 8:16 AMwonsun
10/20/2022, 9:41 AMFailed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State payload is too large.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
File "/home/da/enviorments/bdi/lib/python3.10/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
state = self.client.set_task_run_state(
File "/home/da/enviorments/bdi/lib/python3.10/site-packages/prefect/client/client.py", line 1604, in set_task_run_state
result = self.graphql(
File "/home/da/enviorments/bdi/lib/python3.10/site-packages/prefect/client/client.py", line 464, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State payload is too large.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
Actually, it wasn't the first time that task failed, and before that, it ran for about 3 minutes and then informed me that no heartbeat detected. (No heartbeat detected from the remote task; marking the run as failed.
) So, the solution I found was configure heartbeats to use threads instead of processes and worte about the flow run config in the .py file. When I did that, the task of receiving parameters was performed longer than the first... (1st try : 3 minutes running -> 2nd try: 12 minutes running) Although the task was executed for a longer time, it was still a failure. 😞
How can i solve this problem? What's the problem of my engineering? This flow may have been written in the wrong way, so I also wrote the flow code below..
import...
from prefect.run_configs import UniversalRun
def custom_function():
'''some works'''
return output
@task
def parsing_waveforms(download):
processing_target = download
'''some works by using above custom_function'''
with Flow('flow_waveforms')as flow:
heir = Parameter('download')
task1 = parsing_waveforms(download=heir)
flow.run_config = UniversalRun(env={'PREFECT__CLOUD__HEARTBEAT_MODE'}:'thread')
flow.register(project_name='data_factory')
Sudharshan B
10/20/2022, 11:23 AMKlemen Strojan
10/20/2022, 12:02 PMdev-prefect-2
.
Which makes no sense - why would I need privileges in the default
namespace? I can run this with Prefect 2.3 without issues.Lennert Van de Velde
10/20/2022, 12:06 PMTrevor Kramer
10/20/2022, 1:42 PMJessica Smith
10/20/2022, 3:14 PMAshoka Sangapallar
10/20/2022, 4:28 PMAshoka Sangapallar
10/20/2022, 4:28 PMAshoka Sangapallar
10/20/2022, 4:34 PMKelvin DeCosta
10/20/2022, 4:42 PMECSTask
.
My question: Do I need to install dependencies like prefect-aws
, s3fs
etc. and register the blocks (eg: prefect block register -m prefect_aws.ecs
) in the container for the agent? Or can I just use the latest prefecthq/prefect
image?Kelvin DeCosta
10/20/2022, 4:42 PMECSTask
.
My question: Do I need to install dependencies like prefect-aws
, s3fs
etc. and register the blocks (eg: prefect block register -m prefect_aws.ecs
) in the container for the agent? Or can I just use the latest prefecthq/prefect
image?FROM prefecthq/prefect:2-python3.10
# Install dependencies
COPY requirements.txt .
RUN pip install -r requirements.txt --no-cache-dir
# Register AWS block types
RUN prefect block register -m prefect_aws.ecs
But I think this might be unused / redundantAnna Geller
10/20/2022, 4:43 PMprefecthq/prefect
image for your agentKelvin DeCosta
10/20/2022, 4:47 PMAnna Geller
10/20/2022, 4:52 PMKelvin DeCosta
10/25/2022, 5:26 PMprefect-aws
if you plan to use ECSTask
.
I ran into a KeyError: ecs-task
could not be dispatched
Also, I recently switched from S3
to private GitHub
and I had to install git
on the prefect
container's image in my ECSTask
definition