Emerson Franks
07/12/2022, 5:51 PMKevin Grismore
07/12/2022, 5:58 PM13:37:49.758 | DEBUG | scrapy.core.scraper - Scraped from <200 <https://quotes.toscrape.com/page/9/>>
However, those logs don't appear in the flow run logs UI. I'm guessing I need to set PREFECT_LOGGING_EXTRA_LOGGERS
, but I'm not certain what the right way to do that is.
prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="scrapy"
seems to have no effect. I also tried prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="scrapy.core.engine"
using the name of the specific logger I wanted to see, but that didn't work either.Jacob Bedard
07/12/2022, 5:59 PMFailed to load and execute flow run: FlowStorageError('An error occurred while unpickling the flow:\n ModuleNotFoundError("No module named \'snowflake\'")\nThis may be due to one of the following version mismatches between the flow build and execution environments:\n - python: (flow built with \'3.8.13\', currently running with \'3.8.10\')\nThis also may be due to a missing Python module in your current environment. Please ensure you have all required flow dependencies installed.')
I created another flow that logged the environment, then ran that one via the UI too. It reported that it was running in the right env.
Where is this 3.8.10 version coming from? And what's the solution for the issue of not being able to pull up snowflake? I didn't have this issue with my last prefect setup, so I'm just kinda scratching my head hereJosh
07/12/2022, 6:11 PMUnexpected error: TypeError("InvalidSchema.__init__() missing 2 required positional arguments: 'local_schema' and 'remote_schema'")
I’m passing dataframes down to another task and for some reason it’s not able to startup the task at all.Constantino Schillebeeckx
07/12/2022, 6:16 PMAndrew Pruchinski
07/12/2022, 6:30 PMTenantView
class but for security purposes we now have all environments (projects) under one tenant/team. Appreciate the help!Irisa Shrestha
07/12/2022, 7:03 PMJohn Mil
07/12/2022, 8:27 PMKevin Grismore
07/12/2022, 9:26 PMflow.py
is contained in?davzucky
07/13/2022, 12:07 AMwonsun
07/13/2022, 1:55 AMTraceback (most recent call last):
File "C:\Users\user\anaconda3\lib\site-packages\prefect\client\client.py", line 551, in _send_request
response.raise_for_status()
File "C:\Users\user\anaconda3\lib\site-packages\requests\models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: <http://localhost:4200/>
This is the part of my code.
from prefect import Flow, task, resource_manager, case
from prefect.tasks.control_flow import merge
from prefect import Client
from prefect.tasks.secrets import PrefectSecret
client = Client()
client.set_secret(name="secret", value="<mysql+pymysql://wonsun>:RKSek849200!@192.168.0.13/metadata")
wonsun
07/13/2022, 2:16 AMTraceback (most recent call last):
File "C:\Users\user\anaconda3\lib\site-packages\prefect\client\client.py", line 551, in _send_request
response.raise_for_status()
File "C:\Users\user\anaconda3\lib\site-packages\requests\models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: <http://localhost:4200/>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File ".\label_test.py", line 19, in <module>
client.set_secret(name="mysql", value="<mysql+pymysql://user:pwd@host/database>")
File "C:\Users\user\anaconda3\lib\site-packages\prefect\client\client.py", line 1643, in set_secret
result = self.graphql(
File "C:\Users\user\anaconda3\lib\site-packages\prefect\client\client.py", line 452, in graphql
result = <http://self.post|self.post>(
File "C:\Users\user\anaconda3\lib\site-packages\prefect\client\client.py", line 407, in post
response = self._request(
File "C:\Users\user\anaconda3\lib\site-packages\prefect\client\client.py", line 641, in _request
response = self._send_request(
File "C:\Users\user\anaconda3\lib\site-packages\prefect\client\client.py", line 564, in _send_request
raise ClientError(f"{exc}\n{graphql_msg}") from exc
prefect.exceptions.ClientError: 400 Client Error: Bad Request for url: <http://localhost:4200/>
The following error messages were provided by the GraphQL server:
GRAPHQL_VALIDATION_FAILED: Unknown type "set_secret_input". Did you mean
"agent_set_input", "edge_set_input", "log_set_input", "task_set_input", or
"agent_insert_input"?
GRAPHQL_VALIDATION_FAILED: Cannot query field "set_secret" on type "Mutation".
The GraphQL query was:
mutation($input: set_secret_input!) {
set_secret(input: $input) {
success
}
}
The passed variables were:
{"input": {"name": "mysql", "value": "<mysql+pymysql://user:pwd@host/database>"}}
Rajeshwar Agrawal
07/13/2022, 10:05 AMconfig.toml
placed at /home/.prefect/config.toml
Is there a way to check via prefect-ui or apollo whether the config items from this toml are applied?Tom Klein
07/13/2022, 10:38 AMEnrique
07/13/2022, 10:39 AMJason Thomas
07/13/2022, 12:11 PMAwaitingRetry
- retry 1:
- flow filters the list, leaving 1 file to process
- flow iterates over the filtered list, but does not call task
- flow finishes in state Completed
I’m guessing task is not called on the retry because the result has been cached and there is no reason to try again. That’s fine. However, I expected the final state of the task to be Failed
. Is this behaving as intended?Tom Klein
07/13/2022, 12:40 PMNo heartbeat detected from the remote task; marking the run as failed
- this happend for tasks being run in parallel via the dask KubeCluster
looking at our (internal) logs - it seems like all of them died simultaneously, 20+ minutes into their run
there’s a total of 58 tasks being run, with 4 workers
the first 4 tasks took approx. 1 hour to run and completed successfully, and then the next 4 that were run all seem to have failed simultaneously
any idea / help?Shivam Bhatia
07/13/2022, 2:12 PMibrahem
07/13/2022, 2:20 PMKevin Grismore
07/13/2022, 3:35 PMtutorial
that's in the flow's directory, and when I run prefect deployment create deployment.py
I get ModuleNotFoundError: No module named 'tutorial'
Josh Paulin
07/13/2022, 3:55 PMwith case(cond, True):
val1_if_true = action1_if_true()
val2_if_true = action2_if_true()
with case(cond, False):
val_if_false = action_if_false()
What (if anything) do I pass to the merge function?Sabir Ali
07/13/2022, 5:21 PMTraceback (most recent call last):
File "/Users/sabirali/PycharmProjects/ETL/ElasticSearchClientTest.py", line 5, in <module>
print(<http://client.info|client.info>())
File "/Users/sabirali/PycharmProjects/ETL/venv/lib/python3.8/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped
return api(*args, **kwargs)
File "/Users/sabirali/PycharmProjects/ETL/venv/lib/python3.8/site-packages/elasticsearch/_sync/client/__init__.py", line 2277, in info
return self.perform_request( # type: ignore[return-value]
File "/Users/sabirali/PycharmProjects/ETL/venv/lib/python3.8/site-packages/elasticsearch/_sync/client/_base.py", line 332, in perform_request
raise UnsupportedProductError(
elasticsearch.UnsupportedProductError: The client noticed that the server is not Elasticsearch and we do not support this unknown product
I used following command to install elasticsearch client
(venv) sabirali@Sabirs-MacBook-Pro ETL % pip install elasticsearch
Collecting elasticsearch
Using cached elasticsearch-8.3.1-py3-none-any.whl (382 kB)
Requirement already satisfied: elastic-transport<9,>=8 in ./venv/lib/python3.8/site-packages (from elasticsearch) (8.1.2)
Requirement already satisfied: urllib3<2,>=1.26.2 in ./venv/lib/python3.8/site-packages (from elastic-transport<9,>=8->elasticsearch) (1.26.9)
Requirement already satisfied: certifi in ./venv/lib/python3.8/site-packages (from elastic-transport<9,>=8->elasticsearch) (2022.6.15)
Installing collected packages: elasticsearch
Successfully installed elasticsearch-8.3.1
WARNING: You are using pip version 21.3.1; however, version 22.1.2 is available.
You should consider upgrading via the '/Users/sabirali/PycharmProjects/ETL/venv/bin/python -m pip install --upgrade pip' command.
(venv) sabirali@Sabirs-MacBook-Pro ETL %
Divya
07/13/2022, 6:33 PMFrank Hardisty
07/13/2022, 8:25 PMVictoria Alvarez
07/13/2022, 8:47 PMRyan
07/14/2022, 2:05 AMAmanda Wee
07/14/2022, 2:46 AMМаксим Пышный
07/14/2022, 2:50 AMShivam Bhatia
07/14/2022, 7:46 AM400 List of found errors: 1.Field: job_spec.worker_pool_specs[0].container_spec.env[12].value; Message: Required field is not set. 2.Field: job_spec.worker_pool_specs[0].container_spec.env[3].value; Message: Required field is not set. 3.Field: job_spec.worker_pool_specs[0].container_spec.env[4].value; Message: Required field is not set. [field_violations {
field: "job_spec.worker_pool_specs[0].container_spec.env[12].value"
description: "Required field is not set."
}
field_violations {
field: "job_spec.worker_pool_specs[0].container_spec.env[3].value"
description: "Required field is not set."
}
field_violations {
field: "job_spec.worker_pool_specs[0].container_spec.env[4].value"
description: "Required field is not set."
}
]
Do i need to configure vertex jobs manually?Faheem Khan
07/14/2022, 8:54 AM_pickle.PicklingError: Can't pickle <function db_con at 0x7f85c986f880>: it's not the same object as __main__.db_con
when I run a task with querying a database. the code runs fine without that db connection task.