Hey, I am running into issues with using blocks for s3 storage and I can't find anything about it on google. I am using the following block for storage (see attached image). The issue I have is that when I make changes to my folder structure, the old versions of files are not deleted. They just get left behind and the new versions of files get uploaded. This can cause things to get out of sync very quickly and cause bugs. For instance, I am using prefect for dbt. I keep getting the following error after moving

fact_twoway_pl_mv.sql

from the

pipeline_events

folder into the

twoway

folder. It seems like the block storage does not effectively account for the fact that I moved this file. It did nothing to delete the old file in the

pipeline_events

folder and just uploaded a new copy into

twoway

. Since both files exist now, it causes confusion in dbt and it throws an error.

exception_text: Compilation Error
  dbt found two models with the name "fact_twoway_pl_mv".
  Since these resources have the same name, dbt will be unable to find the correct resource
  when looking for ref("fact_twoway_pl_mv").
  To fix this, change the name of one of these resources:
  - model.rn_dw_dbt.fact_twoway_pl_mv (models/pipeline_events/fact_twoway_pl_mv.sql)
  - model.rn_dw_dbt.fact_twoway_pl_mv (models/twoway/fact_twoway_pl_mv.sql)

Does anyone know where I am going wrong here? It feels like this block storage should be doing a better job of syncing. Did I configure something incorrectly? Thanks

@Jeff Rose Did you ever have any resolution for this? I am running into the same issue on my system. Was thinking I will need to upload to a new S3 directory every time I create a deployment, but that feels like a bad solution.

@dathey Nope. This seems like just how it is. I had a meeting with prefect support about this and they just suggested alternative storage solutions.

Jeez, that is frustrating. Thanks for getting back to me!