15:58:05.322 | INFO    | prefect.agent - Reported flow run 'dfc9e73f-e303-4068-9bc9-6b8d8fec51da' as crashed: Flow run infrastructure exited with non-zero status code 143.

Hi everyone, can anyone please explain why this flow crashed

hi @Nimesh Kumar - this is not enough information to know what is going on. can you provide agent logs?

Thats all i am getting in the agent logs, rest of the logs are the prints that i have added in the code

sorry, but im not sure without more information. could you detail how you're creating your deployment, where your agent is running etc?

so i am running prefect on a Google cloud VM using docker and using kubernetes as a infrastructure to run each flows in a pod

hi @Nimesh Kumar - this seems to have to do with your kubernetes pod's ephemeral storage limit you should be able to adjust that somehow depending on how you're managing kubernetes something that looks like this someplace

resources:
  limits:
    ephemeral-storage: 2Gi

Let me check, Do you think i should write the above config in the image that i am using here for pod

not directly no, you'd have to make sure its nested properly within your job config. what I sent above would be for the pod spec its also up to you to decide whether the new ephemeral storage limit would apply to everything on your cluster or only stuff associated with this infra block

i m sorry i am kindof new to kubernetes and feels a bit complicated , is there any resources where i can read about this

no worries, yep! kubernetes docs on this are pretty good

Prefect does support creating multiple storage configurations and allows you to switch between them as needed. When it comes to Kubernetes, Prefect utilizes the concept of storage blocks to facilitate storage configuration. Here's a brief overview of how storage is configured with Prefect: - Using Default Local Storage: When you build a deployment without specifying the

--storage-block

flag, Prefect defaults to a LocalFileSystem block, which utilizes your current working directory as its basepath. This option is typically not suited for Kubernetes deployments where the local file system of other machines isn't accessible. - Preconfigured Storage Blocks: You can preconfigure a storage block (for example, cloud storage options) to be used with your deployments. This provides more flexibility when running workflows across different environments. Since you're asking about Kubernetes local ephemeral storage, keep in mind that Prefect does not detail specifics on configuring Kubernetes storage specifics, such as ephemeral storage, within its documentation. For more detailed information please refer to Prefect's documentation on storage and the Kubernetes deployment guide. As the information on Kubernetes local ephemeral storage is quite dated (pre 2021), you might find more current practices and configurations by directly consulting Kubernetes' own documentation or community-driven resources. Ephemeral storage is typically managed through Kubernetes itself and configuring it might not be directly related to Prefect's storage configuration.

@Marvin take a break from prefect stuff, tell me what you know about kubernetes local ephemeral storage

Local ephemeral storage in Kubernetes refers to the temporary storage that is provided to a pod on the node where it is running. This kind of storage is tied to the lifecycle of the pod and is used for scratch space, caching, and other similar use cases where data does not need to be persistent across pod restarts or rescheduling. Kubernetes allows pods to specify their temporary storage needs using ephemeral-storage requests and limits in the pod specification. This storage is provisioned from the local storage of the node (such as its disk or SSD) and is managed by the kubelet service on the node. Key points about Kubernetes local ephemeral storage include: - Ephemeral by Nature: As the name suggests, local ephemeral storage is meant for temporary data. When a pod is removed from a node for any reason, the data in its local ephemeral storage is deleted forever. - Resource Accounting: Kubernetes tracks the usage of local ephemeral storage per pod. This helps in maintaining the Quality of Service (QoS) and ensuring that a single pod does not consume all the available storage on the node, which could affect other pods. - EmptyDir Volumes: One common way to utilize local ephemeral storage is through the use of

emptyDir

volumes. An

emptyDir

volume is created when a pod gets assigned to a node and exists as long as that pod is running on that node. - Resource Scheduling: When scheduling pods, Kubernetes will take into account the storage requirements specified in the pod specification to ensure that nodes have sufficient storage for the pod's needs. - Limits and Requests: Similar to CPU and memory, Kubernetes allows you to specify requests and limits for storage. Requests guarantee a minimum amount of ephemeral storage for the pod, and limits ensure a pod does not exceed a specified amount of storage. Local ephemeral storage is not suitable for any data that must be retained across pod restarts, crashes, or deletions. For persistent storage needs, Kubernetes offers a PersistentVolume subsystem that provides an API for users and administrators that abstracts details of how storage is provided and how it is consumed. More detailed information can be found in the Kubernetes documentation on ephemeral storage.

Thanks a lot Nate.

Why do you need ephemeral storage for a K8S workers at first place? Use in-memory storage between tasks/flows like dataframes, dicts etc. If you really need a storage for a temp data, use S3 or Minio etc.