https://prefect.io logo
Join Slack
Powered by
:wave::skin-tone-2: Is anyone else having issues w...
# ask-community
e
👋🏻 Is anyone else having issues with Prefect 3.0 and the prefect-dbt integration? There seems to be version conflicts that pop up when trying to use the latest python package versions of each
n
hi @Eric Albanese - can you be more specific about what you're seeing? in particular the output of 
prefect version
should be helpful
e
Ok sorry, been testing more. Not quite version conflicts, but in the process of migrating to a self hosted 3.0, having the exact same configs are throwing some unclear errors.
Copy code
RuntimeError: Unable to load 'dbt-profile-prod-warehouse' of block type 'dbt-cli-profile' due to failed validation. To load without validation, try loading again with `validate=False`.

pydantic_core._pydantic_core.ValidationError: 2 validation errors for DbtCliProfile
target_configs.PostgresTargetConfigs.credentials.connection_info
  Field required [type=missing, input_value={'host': 'warehouse-db.cr..., 'username': 'prefect'}, input_type=dict]
    For further information visit <https://errors.pydantic.dev/2.9/v/missing>
target_configs.TargetConfigs.type
  Field required [type=missing, input_value={'schema': 'marts', 'thre... 'username': 'prefect'}}, input_type=dict]
    For further information visit <https://errors.pydantic.dev/2.9/v/missing>
I can't quite figure out what's "missing", since the same config is working on Prefect Cloud
Screenshot 2024-10-18 at 1.30.45 PM.png,Screenshot 2024-10-18 at 1.31.02 PM.png,Screenshot 2024-10-18 at 1.31.13 PM.png
trying the CLI route - type is definitely missing
Copy code
(venv) ➜  prefect git:(eric/prefect) ✗ prefect blocks inspect database-credentials/warehouse-db 
                                                                    database-credentials/warehouse-db                                                                    
┌──────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Block Type                   │ Database Credentials                                                                                                                   │
│ Block id                     │ 11463590-3eb2-4c0a-8a5d-5b91e7f390ad                                                                                                   │
├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ host                         │ <http://warehouse-xxxxx.com|warehouse-xxxxx.com>                                                                                  │
│ port                         │ 5432                                                                                                                                   │
│ driver                       │ postgresql+pygresql                                                                                                                    │
│ database                     │ warehouse                                                                                                                              │
│ password                     │ ********                                                                                                                               │
│ username                     │ prefect                                                                                                                                │
└──────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
n
can you share the output of 
prefect version
?
e
Copy code
(venv) ➜  prefect git:(eric/airbyte_api_replacement) ✗ prefect version
Version:             3.0.10
API version:         0.8.4
Python version:      3.10.12
Git commit:          3aa2d893
Built:               Tue, Oct 15, 2024 1:31 PM
OS/Arch:             darwin/arm64
Profile:             default
Server type:         server
Pydantic version:    2.9.2
Integrations:
  prefect-dbt:       0.6.1
  prefect-slack:     0.3.0
  prefect-aws:       0.5.1
  prefect-shell:     0.3.0
  prefect-docker:    0.6.1
  prefect-sqlalchemy: 0.5.1
n
oh, i think I know whats happening you have block documents from 2.x where blocks where pydantic 1 basemodels, and are now trying to load them into the new schema (blocks are pydantic 2 basemodels now) are you able to try recreating these block documents with the same values?
e
Yeah I think I did that through the UI - I loaded up our fresh 3.0 instance, and manually re-created all of the same dbt/postgres blocks through there
n
hmm. if you're willing, can you try 
prefect block type delete dbt-cli-profile
and then
Copy code
prefect block register -m prefect_dbt
and then try again?
this might be an issue with the default block types registered to the server
e
sure trying now
hm same exact error
Copy code
(venv) ➜  prefect git:(eric/airbyte_api_replacement) ✗ prefect block type delete dbt-cli-profile
Are you sure you want to delete block with id <built-in function id>? [y/N]: y
Deleted Block Type 'dbt-cli-profile'.
(venv) ➜  prefect git:(eric/airbyte_api_replacement) ✗ prefect block register -m prefect_dbt 
Successfully registered 6 blocks

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Registered Blocks               ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ dbt CLI Profile                 │
│ dbt Cloud Credentials           │
│ dbt Core Operation              │
│ dbt CLI Global Configs          │
│ dbt CLI Postgres Target Configs │
│ dbt CLI Target Configs          │
└─────────────────────────────────┘

 To configure the newly registered blocks, go to the Blocks page in the Prefect UI: <https://prefect.compaoffers.com/blocks/catalog>

(venv) ➜  prefect git:(eric/airbyte_api_replacement) ✗ 
(venv) ➜  prefect git:(eric/airbyte_api_replacement) ✗ 
(venv) ➜  prefect git:(eric/airbyte_api_replacement) ✗ python flows/import_core.py                                     
13:49:43.136 | INFO    | prefect.engine - Created flow run 'amber-shellfish' for flow 'compa_core_sync'
13:49:43.139 | INFO    | prefect.engine - View at <https://prefect.compaoffers.com/runs/flow-run/c4eb103c-d289-49b9-ad49-64e68058792c>
13:49:45.286 | INFO    | apprise - Notifying 1 service(s) asynchronously.
13:49:45.566 | INFO    | apprise - Sent Slack notification.
13:49:45.617 | INFO    | Task run 'compa_core_dbt_run-559' - >> INFO Running Compa Core / DBT
13:49:45.885 | ERROR   | Task run 'compa_core_dbt_run-559' - Task run failed with exception: ValueError('Unable to find block document named dbt-profile-prod-warehouse for block type dbt-cli-profile') - Retries are exhausted
Traceback (most recent call last):
  File "/Users/albanese/prefect/venv/lib/python3.10/site-packages/prefect/client/orchestration.py", line 1557, in read_block_document_by_name
    response = await self._client.get(
  File "/Users/albanese/prefect/venv/lib/python3.10/site-packages/httpx/_client.py", line 1814, in get
    return await self.request(
  File "/Users/albanese/prefect/venv/lib/python3.10/site-packages/httpx/_client.py", line 1585, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/Users/albanese/prefect/venv/lib/python3.10/site-packages/prefect/client/base.py", line 361, in send
    response.raise_for_status()
  File "/Users/albanese/prefect/venv/lib/python3.10/site-packages/prefect/client/base.py", line 174, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url '<https://prefect.compaoffers.com/api/block_types/slug/dbt-cli-profile/block_documents/name/dbt-profile-prod-warehouse?include_secrets=true>'
Response: {'detail': 'Block document not found'}
For more information check: <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404>
Then I re-created the block in the UI
Copy code
(venv) ➜  prefect git:(eric/airbyte_api_replacement) ✗ 
(venv) ➜  prefect git:(eric/airbyte_api_replacement) ✗ 
(venv) ➜  prefect git:(eric/airbyte_api_replacement) ✗ python flows/import_core.py
13:51:06.819 | INFO    | prefect.engine - Created flow run 'hospitable-porpoise' for flow 'compa_core_sync'
13:51:06.823 | INFO    | prefect.engine - View at <https://prefect.compaoffers.com/runs/flow-run/55111420-dd21-4363-bf1e-559d8742f51a>
13:51:09.009 | INFO    | apprise - Notifying 1 service(s) asynchronously.
13:51:09.294 | INFO    | apprise - Sent Slack notification.
13:51:09.345 | INFO    | Task run 'compa_core_dbt_run-e36' - >> INFO Running Compa Core / DBT
13:51:09.575 | ERROR   | Task run 'compa_core_dbt_run-e36' - Task run failed with exception: KeyError('target_configs') - Retries are exhausted
Traceback (most recent call last):
  File "/Users/albanese/prefect/venv/lib/python3.10/site-packages/prefect/blocks/core.py", line 1016, in _load_from_block_document
    return cls._from_block_document(block_document)
  File "/Users/albanese/prefect/venv/lib/python3.10/site-packages/prefect/blocks/core.py", line 688, in _from_block_document
    block = block_cls.model_validate(block_document.data)
  File "/Users/albanese/prefect/venv/lib/python3.10/site-packages/prefect/blocks/core.py", line 1304, in model_validate
    return super().model_validate(
  File "/Users/albanese/prefect/venv/lib/python3.10/site-packages/pydantic/main.py", line 596, in model_validate
    return cls.__pydantic_validator__.validate_python(
pydantic_core._pydantic_core.ValidationError: 2 validation errors for DbtCliProfile
target_configs.PostgresTargetConfigs.credentials.connection_info
  Field required [type=missing, input_value={'host': 'warehouse-db.cr..., 'username': 'prefect'}, input_type=dict]
    For further information visit <https://errors.pydantic.dev/2.9/v/missing>
target_configs.TargetConfigs.type
  Field required [type=missing, input_value={'schema': 'marts', 'thre... 'username': 'prefect'}}, input_type=dict]
    For further information visit <https://errors.pydantic.dev/2.9/v/missing>
n
hmm okay. do you mind creating an issue about this? https://github.com/PrefectHQ/prefect/issues/new/choose
e
Yeah can do later today
n
thank you!
20 Views
Open in Slack
PreviousNext
Powered by