https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I have issues with prefect assets. I ...
# marvin-ai
v
@Marvin I have issues with prefect assets. I have updated our codebase and removed all materializations to tasks, removed dependencies and manually deleted all the assets. But everyday the same assets are being built. One thing i dont understand is that it shows the materializations that occured 48days ago.
m
✅ thought for 175.5 seconds
What you’re seeing is usually explained by two separate things: 1. Historical asset/materialization events are still retained, so the UI can still show materializations from weeks ago. 2. Something is still emitting new asset events every day, which is why the same assets keep coming back. A few important details for Prefect 3.x: Assets are event-driven. They are not really “standalone records” that disappear forever once deleted in the UI. If Prefect still has historical asset/materialization events, the UI can continue to show those assets/history. And if any scheduled run still emits those events again, the assets will reappear. Why you still see materializations from 48 days ago - Prefect keeps event history for some retention period. - If your server / Cloud environment is retaining those events that long, the UI can still show a materialization that happened 48 days ago. - So removing the code now does not erase old materialization history. Why the same assets are rebuilt every day The most likely cause is that some deployment is still running on a daily schedule and is still emitting asset/materialization events. Common sources: - an old deployment still exists - a schedule is still active on that deployment - an automation is triggering runs - a different branch/image/repo revision is still deployed than the code you edited - another flow/task still contains the asset/materialization logic So even if you updated your current codebase, Prefect may still be running an older deployed version. What I’d check first 1. Look for active deployments/schedules - Make sure there isn’t an older deployment still enabled and running daily. 2. Check the code version/source of the deployment - Confirm the deployment is actually using the updated code you expect. 3. Check automations - An automation may be triggering runs that recreate the assets. 4. Check flow run history - Open one of the daily runs and inspect which deployment / work pool / image / source it came from. If you want to inspect with the CLI, verify the available commands in your environment, but generally you’ll want to look at deployments, schedules, and automations. A practical mental model is: - Deleting assets in the UI removes the current visible asset entry - Old event history can still make historical materializations visible - Any new run that emits the same asset key will recreate the asset So the real fix is usually: - stop the process still generating those events, and - wait for event retention to expire, or reduce retention if that’s under your control If you’re on self-hosted Prefect Server, event retention is configurable. That would explain the 48-day-old materializations still appearing. Could you share: - whether you’re on Prefect Cloud or self-hosted Server - whether these daily asset rebuilds are tied to a scheduled deployment - and whether the runs that recreate them are coming from the same deployment name as before If you want, I can help you narrow this down step-by-step. Relevant docs: - Assets concepts - Deployments - Automations
v
we are using prefect-cloud
@Marvin to add context { "key": "athena://layered_data_dev/a47c9a3e8ecf24e7e8681503edfa0e85d_matched_matcher_manual_domain", "properties": { "name": "Matched on Matcher Manual Domain Only", "url": "", "description": "The domain input list matched to infobel unique ids based on the manual matches, but only using the domain to join on and not include the country. This is used as a fallback for when we have a manual match on the domain but the countries do not match. So for example a domain on a Ducth IP can than be matched to a German holding when a manual match exists in Germany. This is done to provide a good as possible match", "owners": [ "pim@leadinfo.com" ] }, "last_seen": "2025-10-31T144931.704129Z", "latest_reference": { "event_id": "019a3abe-c338-799b-aacd-84c6c6ba4d75", "occurred": "2025-10-31T144931.704129Z", "flow_run_id": "06904cc1-501e-7495-8000-1bc51e677ebd", "task_run_id": "019a3abe-71d9-7c7f-803d-249140b0d5f1", "originating_workspace_id": null, "metadata": {} }, "latest_materialization": { "event_id": "019a3abd-7f7d-77ab-86f7-60b5f9cc0a46", "occurred": "2025-10-31T144808.829006Z", "flow_run_id": "06904cc1-501e-7495-8000-1bc51e677ebd", "task_run_id": "019a3abd-5ac6-7621-9a49-cd0f6d0a5103", "originating_workspace_id": null, "metadata": {}, "by_tools": [], "status": "succeeded" } } this is being generated and we use prefect-clould
m
✅ thought for 18.6 seconds
Thanks — that payload is very helpful. The key detail is this: This asset is not just historical. It has: - 
last_seen: 2025-10-31T14:49:31...
- 
latest_materialization.occurred: 2025-10-31T14:48:08...
- a concrete 
flow_run_id
- a concrete 
task_run_id
That means Prefect Cloud received a new asset event recently. So this is not just old retained history from 48 days ago — something is still actively emitting this asset. Also, the asset key: 
<athena://layered_data_dev/a47c9a3e8ecf24e7e8681503edfa0e85d_matched_matcher_manual_domain>
suggests the event is being emitted by code that still knows about that asset key, likely from a task or materialization wrapper in a deployed flow. What this tells us Because there is a fresh: - 
flow_run_id
- 
task_run_id
- 
latest_materialization
the root cause is almost certainly one of these: 1. An old deployment is still running - The deployment may still point to older code. 2. The worker is using an old image / old checked-out code - For example, your deployment definition changed, but the runtime image/container did not. 3. Another flow or task still emits the same asset key - Maybe not the one you edited. 4. A scheduled run or automation is recreating it - The daily cadence strongly suggests a schedule. Most important next step Open that exact 
flow_run_id
in Prefect Cloud and inspect: - deployment name - work pool / worker - deployment version or code source - container image / infrastructure block / pull step - which flow file was actually executed - task run name for 
task_run_id
That will tell you which runtime is still producing the asset event. How to reason about the payload You pasted both: - 
latest_materialization
- 
latest_reference
That means the flow is not only materializing the asset, but something in the run is also referencing it afterward. So there is definitely asset-aware code still executing in that flow path. Because you are using Prefect Cloud The “48 days ago” part is best understood as history retention in Cloud UI/events. But the new daily reappearance is not caused by retention. Retention explains why old entries are visible; it does not explain why 
latest_materialization
keeps updating. A live run is doing that. What I would check in order 1. Search all deployments for the flow run that owns 
flow_run_id = 06904cc1-501e-7495-8000-1bc51e677ebd
2. Check if that deployment still has an active schedule 3. Confirm what code source that deployment is using - remote git source - Docker image tag - local worker filesystem - baked image with stale code 4. Check whether multiple deployments exist for the same flow - often an “old” deployment is forgotten 5. Search your codebase for the asset key suffix - 
matched_matcher_manual_domain
- and also search for the human-readable name 6. Check if a helper abstraction still emits asset events - sometimes materializations are not directly visible as 
@materialize
anymore because they happen through a wrapper/util Likely scenario The most likely explanation is: - you removed the materialization code in your current branch/repo - but a deployment in Prefect Cloud is still running older code every day - each run emits the same asset key again - Cloud then shows the asset as active again What would confirm this immediately If you click that flow run and the code/deployment points to an older image tag, commit SHA, or deployment version, that’s your answer. If you want, send me any of the following and I can help pinpoint it quickly: - the flow run details for 
06904cc1-501e-7495-8000-1bc51e677ebd
- the deployment configuration - how you deploy today (
prefect deploy
, Docker image, git-based source, etc.) - whether workers pull latest code or run a fixed image And one extra note: if you share the deployment method, I can tell you the most likely stale-code failure mode for that setup.
v
@Marvin I checked the flows with the existing id and there seems to be nothing that exists
m
✅ thought for 17.1 seconds
If the 
flow_run_id
from the asset event doesn’t correspond to a run you can find in Prefect Cloud, then the most likely explanation is stale/foreign event data rather than a currently existing flow run. A few possibilities: 1. The event is old/stale history - Asset views are built from event data. - If the event store still has asset events, the UI can show 
latest_materialization
/ 
latest_reference
even when the linked flow run record is no longer available in the UI. - That can happen if flow runs were deleted/pruned but the asset event history remains. 2. The event came from a different workspace/account context Your payload includes: - 
originating_workspace_id: null
That’s a little notable. It may indicate the event metadata doesn’t have a workspace association in the way you’d expect, so the asset history can outlive or be disconnected from the current visible run records. 3. The flow run record no longer exists, but the asset event does This is probably the cleanest explanation if: - the asset still shows a 
latest_materialization
- but the run ID cannot be opened anywhere That means the asset page is rendering from the event stream, not from a live run lookup. So at this point, I would not assume a currently active deployment unless you also see new timestamps continuing to advance every day. Important question: Are the timestamps like 
2025-10-31T14:48:08Z
continuing to move forward each day, or are they fixed on the same recent date? Because there are two very different cases: Case A — timestamps keep changing daily - then something is definitely still emitting events - and we need to find the source Case B — timestamps are static, but the assets remain visible - then this is almost certainly retained historical event data / UI persistence - not an active job recreating them Given your latest message, I’d lean more toward Case B unless you’ve confirmed those timestamps keep updating. What to check next 1. Pick one asset and record: - 
last_seen
- 
latest_materialization.occurred
2. Check it again after the next daily window. 3. If those values change, then events are still being emitted. 4. If they do not change, then the asset is just being shown from retained history. If they are not changing Then the behavior is expected from retained asset events in Cloud, and manual asset deletion may not fully remove what can be reconstructed from historical event data. If they are changing Then we need to look for the actual emitter outside the obvious flow list: - another workspace - another deployment name - a worker still sending events from old code - direct event emission / asset instrumentation outside the flow you checked One other subtle possibility: if your team has changed workspaces/environments, the asset key may be the same but the event provenance may not line up with the workspace you’re currently inspecting. So my recommendation is: First confirm whether 
last_seen
is still advancing. If you want, paste: - whether that timestamp changes tomorrow - and whether searching Prefect Cloud for the task run id also returns nothing If both flow run and task run IDs are missing and timestamps are no longer moving, then this looks like retained/disconnected historical asset events rather than an active deployment.
v
@Marvin no, the last_seen doesn't change. we get the same asset data everyday, even after manully deleting them
m
✅ thought for 18.3 seconds
Got it — that changes the diagnosis quite a bit. If: - 
last_seen
does not change - 
latest_materialization.occurred
does not change - the 
flow_run_id
no longer resolves - but the same asset reappears every day even after you manually delete it then this does not look like an active flow recreating the asset. It looks much more like the asset view is being reconstructed from retained historical event data in Prefect Cloud, and the manual delete is not removing the underlying event history that the UI uses to rebuild that asset. So in plain terms: You are deleting the visible asset, but not the historical materialization/reference events behind it. Then some background refresh/reindex/UI rebuild causes the asset to show up again from those same old events. Why this fits your symptoms - The timestamps stay frozen -> no new events are arriving - The run/task IDs are effectively dead -> these are old events - The asset returns after deletion -> the source of truth is still historical event data, not a live deployment That means this is likely either: 1. expected behavior of the current asset/event model in Cloud, or 2. a Cloud-side bug in asset deletion, where deletion hides the asset temporarily but a later refresh reconstructs it from retained events Given your description, I would strongly suspect #2 or a product limitation around deletion semantics. Important distinction There are really two layers here: - Asset event history — old 
prefect.asset.*
events - Current asset listing — what the UI renders If the UI can derive the asset from old events, then deleting only the listing won’t be durable. What you can do right now Since you’re on Prefect Cloud, you do not control event-retention internals the same way as self-hosted Server. So there probably isn’t a self-serve cleanup knob for you to permanently purge those old asset events. At this point, the most accurate guidance is: - this does not appear to be caused by currently running code - this does not appear to be caused by a still-active deployment - it does appear that old asset events are still present in Cloud and are enough to recreate the asset after deletion Most likely conclusion This is likely a Prefect Cloud issue / limitation with asset deletion when historical asset events still exist. Because I can’t file this for you, the best next step is to contact Prefect directly and include exactly what you already collected: - the asset 
key
- the frozen 
last_seen
- the frozen 
latest_materialization.occurred
- the non-resolving 
flow_run_id
- the fact that the asset reappears daily after manual deletion without timestamp changes Contact: - Prefect contact If helpful, you can send them a concise report like this:
Copy code
We are using Prefect Cloud.

We deleted several assets manually, and they reappear every day in the UI.

However, these assets do not appear to be actively rematerialized:
- `last_seen` does not change
- `latest_materialization.occurred` does not change
- the referenced `flow_run_id` and `task_run_id` do not resolve to existing runs

Example asset:
- key: `<athena://layered_data_dev/a47c9a3e8ecf24e7e8681503edfa0e85d_matched_matcher_manual_domain>`
- last_seen: `2025-10-31T14:49:31.704129Z`
- latest_materialization.occurred: `2025-10-31T14:48:08.829006Z`

This suggests the asset is being reconstructed from retained historical asset events after deletion, rather than from active runs. Can you confirm whether this is expected behavior or a bug in Prefect Cloud asset deletion?
If you want, I can also help you turn this into a shorter bug report for GitHub discussion/issues, but based on what you’ve shown, I would describe it as historical asset events resurrecting deleted assets in Cloud.
v
@Marvin could you help me raise this issue for more help
m
✅ thought for 18.1 seconds
Yes — I can help you prepare it. I can’t submit a support request or route it internally, but since I didn’t find an existing GitHub discussion/issue matching this, you have two good options: - Contact Prefect directly for Cloud help: prefect.io/contact - Open a GitHub issue/discussion with the summary below: - Issues: Prefect issues - Discussions: Prefect discussions For this one, I’d suggest a GitHub issue if you believe asset deletion is behaving incorrectly, or contact if you want faster Cloud-specific confirmation. Suggested issue title 
Deleted assets reappear in Prefect Cloud without new materialization events
Suggested issue body
Copy code
### Description

In Prefect Cloud, we manually delete assets from the UI, but the same assets reappear later even though they do not appear to be actively rematerialized.

### What we observed

For affected assets:

- `last_seen` does not change
- `latest_materialization.occurred` does not change
- the referenced `flow_run_id` and `task_run_id` do not resolve to existing runs in the workspace
- after manual deletion, the same assets reappear later with the exact same historical data

This suggests the assets may be getting reconstructed from retained historical asset events rather than from active flow runs.

### Example asset payload
{ "key": "athena://layered_data_dev/a47c9a3e8ecf24e7e8681503edfa0e85d_matched_matcher_manual_domain", "properties": { "name": "Matched on Matcher Manual Domain Only", "url": "", "description": "The domain input list matched to infobel unique ids based on the manual matches, but only using the domain to join on and not include the country. This is used as a fallback for when we have a manual match on the domain but the countries do not match. So for example a domain on a Ducth IP can than be matched to a German holding when a manual match exists in Germany. This is done to provide a good as possible match", "owners": [ "pim@leadinfo.com" ] }, "last_seen": "2025-10-31T144931.704129Z", "latest_reference": { "event_id": "019a3abe-c338-799b-aacd-84c6c6ba4d75", "occurred": "2025-10-31T144931.704129Z", "flow_run_id": "06904cc1-501e-7495-8000-1bc51e677ebd", "task_run_id": "019a3abe-71d9-7c7f-803d-249140b0d5f1", "originating_workspace_id": null, "metadata": {} }, "latest_materialization": { "event_id": "019a3abd-7f7d-77ab-86f7-60b5f9cc0a46", "occurred": "2025-10-31T144808.829006Z", "flow_run_id": "06904cc1-501e-7495-8000-1bc51e677ebd", "task_run_id": "019a3abd-5ac6-7621-9a49-cd0f6d0a5103", "originating_workspace_id": null, "metadata": {}, "by_tools": [], "status": "succeeded" } }
Copy code
### Expected behavior

When an asset is manually deleted, it should either:
- remain deleted unless new events are emitted, or
- the product should clearly indicate that historical retained events will recreate the asset

### Actual behavior

The asset reappears later with unchanged historical timestamps and references to flow/task runs that do not exist anymore.

### Environment

- Prefect Cloud
- Prefect 3.x
- No evidence of new materialization events for the affected assets
- `last_seen` remains unchanged between deletions/reappearances
If you want, I can also make this: 1. shorter and more GitHub-ready, or 2. more support-ticket style for the contact form.
v
@Marvin look at the script I ran! to monitor the assets 2026-05-05T072917Z New layered_data_dev assets detected: 10 2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_xing", "last_seen": "2026-04-15T181909.792945Z", "latest_materialization": "2026-04-28T134903.765353Z"} 2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_domain", "last_seen": "2026-04-15T181906.607191Z", "latest_materialization": "2026-04-28T134859.746878Z"} 2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_domain_country", "last_seen": "2026-04-15T181900.656565Z", "latest_materialization": "2026-04-28T134854.515172Z"} 2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_vat", "last_seen": "2026-04-15T181855.305439Z", "latest_materialization": "2026-04-28T134848.960320Z"} 2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_coc", "last_seen": "2026-04-15T181843.456839Z", "latest_materialization": "2026-04-28T134837.230402Z"} 2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_phone", "last_seen": "2026-04-15T181833.060078Z", "latest_materialization": "2026-04-28T134825.696189Z"} 2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_phone_country", "last_seen": "2026-04-15T181823.098659Z", "latest_materialization": "2026-04-28T134816.912536Z"} 2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_matcher_manual_domain", "last_seen": "2026-04-15T181814.702194Z", "latest_materialization": "2026-04-28T134805.232442Z"} 2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_matcher_manual", "last_seen": "2026-04-15T181804.191546Z", "latest_materialization": "2026-04-28T134754.590706Z"} 2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_creditsafe", "last_seen": "2026-04-15T181759.943897Z", "latest_materialization": "2026-04-28T134750.202717Z"} 2026-05-05T073115Z Total assets: 355, layered_data_dev matches: 158 So, maybe this could be of some help to narrow down the issue
m
✅ thought for 32.7 seconds
Yes — this is very helpful, and it narrows it down a lot. This strongly supports that you’re not seeing new events, but that deleted assets are being repopulated from older asset history/state in Prefect Cloud. Why this output matters: - Your monitor reports “New layered_data_dev assets detected: 10” on 
2026-05-05
- But each asset’s timestamps are old: - `last_seen`: 
2026-04-15...
- `latest_materialization`: 
2026-04-28...
- That means they are new to your current asset listing snapshot, not newly materialized on May 5 - In other words: the assets are reappearing, but their underlying event timestamps are not being updated That is exactly the pattern you’d expect if: - the assets were deleted from the visible asset list - Prefect Cloud later rebuilt/re-surfaced them from retained historical asset/event data It does not fit an active-rematerialization explanation. There’s also another notable clue: For many of these assets, - 
latest_materialization
is later than 
last_seen
That suggests 
last_seen
may not simply mean “last time this asset existed in the UI.” It may reflect a reference-type event or another event-derived timestamp, while the materialization history is preserved separately. That makes the case even stronger that the asset UI is being synthesized from event history rather than a single canonical asset row. What this means for the report You now have much better evidence that: 1. The assets reappear after deletion 2. They reappear with unchanged historical timestamps 3. They are not being freshly materialized at the time they reappear 4. Therefore, deletion is likely not removing the historical records Cloud uses to reconstruct assets That’s a much stronger bug report. Recommended issue title 
Deleted assets reappear in Prefect Cloud from historical state/events without new materializations
Updated issue body
Copy code
### Description

In Prefect Cloud, manually deleted assets reappear later even though they are not being newly materialized.

### What we observed

We monitor our asset list externally. After manually deleting assets, the same assets show up again later, but with unchanged historical timestamps.

Example monitor output:
2026-05-05T072917Z New layered_data_dev assets detected: 10 2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_xing", "last_seen": "2026-04-15T181909.792945Z", "latest_materialization": "2026-04-28T134903.765353Z"} 2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_domain", "last_seen": "2026-04-15T181906.607191Z", "latest_materialization": "2026-04-28T134859.746878Z"} 2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_domain_country", "last_seen": "2026-04-15T181900.656565Z", "latest_materialization": "2026-04-28T134854.515172Z"} 2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_vat", "last_seen": "2026-04-15T181855.305439Z", "latest_materialization": "2026-04-28T134848.960320Z"} 2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_coc", "last_seen": "2026-04-15T181843.456839Z", "latest_materialization": "2026-04-28T134837.230402Z"} 2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_phone", "last_seen": "2026-04-15T181833.060078Z", "latest_materialization": "2026-04-28T134825.696189Z"} 2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_phone_country", "last_seen": "2026-04-15T181823.098659Z", "latest_materialization": "2026-04-28T134816.912536Z"} 2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_matcher_manual_domain", "last_seen": "2026-04-15T181814.702194Z", "latest_materialization": "2026-04-28T134805.232442Z"}
2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_matcher_manual", "last_seen": "2026-04-15T181804.191546Z", "latest_materialization": "2026-04-28T134754.590706Z"} 2026-05-05T072917Z {"key": "athena://layered_data_dev/output_table_prefix/matched_creditsafe", "last_seen": "2026-04-15T181759.943897Z", "latest_materialization": "2026-04-28T134750.202717Z"} 2026-05-05T073115Z Total assets: 355, layered_data_dev matches: 158
Copy code
### Why this seems incorrect

These assets are detected as newly present in the asset list on 2026-05-05, but their timestamps are still old:
- `last_seen` remains in April
- `latest_materialization` remains in April

So the assets appear to be reappearing from retained historical asset/event state, not from new materialization events.

We also checked referenced flow/task run IDs for some assets, and they do not resolve to currently visible runs.

### Expected behavior

If an asset is manually deleted, it should either:
- stay deleted unless a new event/materialization recreates it, or
- the product should clearly document that historical retained asset events will recreate deleted assets

### Actual behavior

Deleted assets reappear in Prefect Cloud with old timestamps and without evidence of new materialization activity.

### Environment

- Prefect Cloud
- Prefect 3.x

### Additional context

We removed asset materializations and dependencies from our codebase, and we do not see evidence that these assets are being actively rebuilt now. The reappearing assets seem tied to historical state rather than current executions.
My recommendation Use that as either: - a GitHub issue: Prefect issues - or a Cloud support/contact request: prefect.io/contact For Cloud-specific behavior, the contact form is probably the better first step. If you want, I can do one more pass and turn this into: - a very short support-form version, or - a polished GitHub issue version with less repetition.
6 Views
Open in Slack
PreviousNext
Powered by