Christian Juhl
09/27/2022, 3:34 PMfrom prefect import flow, task, get_run_logger
from prefect.orion.schemas.states import Failed
def my_function():
raise Exception('This is an exception')
@flow
def run_flow():
logger = get_run_logger()
try:
result = my_function()
<http://logger.info|logger.info>(result)
except Exception as e:
errmsg = f"Error! Type: {e.__class__.__name__} Detail: {e}"
logger.error(errmsg)
return Failed()
if __name__ == "__main__":
run_flow()
However, I get the following error;
Exception has occurred: TypeError
Unexpected result for failure state: None —— NoneType cannot be resolved into an exception
File "main.py", line 21, in <module>
run_flow()
I'm thinking I might be doing more than one thing wrong here. Can anyone enlighten me, please? Thanks!Joe Schueller
09/27/2022, 3:58 PMprefect-dbt
and can successfully execute a flow using the trigger_dbt_cli_command
task, but dbt doesn't actually do anything.Tim Helfensdörfer
09/27/2022, 4:40 PM[]
) in my yaml file the command points to my prefect entry script but it needs the UUID of the flow run. So it boils down to two questions:
1. How does the queue worker provide the UUID into the ecs task
2. What does Command
do?
(in the second screenshot I changed the task definition command from [] to python3 /app/tasks/run.py
- but it still needs the flow run uuid)chicago-joe
09/27/2022, 4:42 PM"containerDefinitions": [
"entryPoint": [
"bash",
"-c"
],
"command": [
"prefect agent start -q ecs-worker"
],
results in error:
Usage: prefect agent [OPTIONS] COMMAND [ARGS]...
Try 'prefect agent -h' for help.
Error: No such command 'start'.
"containerDefinitions": [
"entryPoint": [
"bash",
"-c"
],
"command": [
"prefect","agent","start","-q","ecs-worker"
],
results in what appears to be calling "prefect" with nothing else, as the logs just keep repeating the menu stdout:Bradley McLaughlin
09/27/2022, 4:54 PMBrett Naul
09/27/2022, 4:56 PMStephen Herron
09/27/2022, 5:12 PMConfiguration:
profiles.yml file [[32mOK found and valid[0m]
dbt_project.yml file [[32mOK found and valid[0m]
Required dependencies:
- git [[32mOK found[0m]
Nick DeCraene
09/27/2022, 6:55 PMJacob Blanco
09/27/2022, 8:44 PM[{'path': ['secret_value'], 'message': 'An unknown error occurred.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
The interesting part is that this only seems to have to jobs at 9AM JST and 9PM JST. Which would be midday and midnight UTC.Chris Gunderson
09/27/2022, 9:10 PM@task(name='Get Email Credentials',
retries = 2,
retry_delay_seconds = 10)
async def GetEmailCredentials(aws_credentials: AwsCredentials) -> EmailCredentials:
session = aws_credentials.get_boto3_session()
secrets_client = session.client(service_name='secretsmanager')
aws_secret = secrets_client.get_secret_value(SecretId='DEV_EMAIL')
secret_json = aws_secret['SecretString']
emailDict = json.loads(secret_json)
return EmailCredentials(emailDict)
Florian Giroud
09/27/2022, 10:57 PMMichael Hadorn
09/28/2022, 5:48 AMRobin Eklund
09/28/2022, 8:14 AMprefect deployment build ./log_flow.py:log_flow -n log-simple -q test
which generate the yaml file. But it looks slightly different for infrastructure
where in the tutorial they have:
infrastructure:
type: process
env: {}
labels: {}
name: null
command:
- python
- -m
- prefect.engine
stream_output: true
and i have:
infrastructure:
type: process
env: {}
labels: {}
name: null
command: null
stream_output: true
block_type_slug: process
_block_type_slug: process
And when i run prefect deployment apply log_flow-deployment.yaml
I get this error:
prefect.exceptions.PrefectHTTPStatusError: Client error '403 Forbidden' for url '<prefect_domain>/api/block_types/016c45c9-6346-49c1-a1cb-da3021f915b2'
Response: {'detail': 'protected block types cannot be updated.'}
For more information check: <https://httpstatuses.com/403>
An exception occurred.
Anyone knows what i need to do?Giorgio Basile
09/28/2022, 10:09 AMCarlos Paiva
09/28/2022, 10:44 AMINFO: pip is looking at multiple versions of feedparser to determine which version is compatible with other requirements. This could take a while.
Collecting feedparser>=5.0.1
Using cached feedparser-6.0.10-py3-none-any.whl (81 kB)
Using cached feedparser-6.0.9-py3-none-any.whl (81 kB)
Using cached feedparser-6.0.8-py3-none-any.whl (81 kB)
Using cached feedparser-6.0.7-py3-none-any.whl (81 kB)
Using cached feedparser-6.0.6-py3-none-any.whl (80 kB)
Using cached feedparser-6.0.5-py3-none-any.whl (80 kB)
Using cached feedparser-6.0.4-py3-none-any.whl (80 kB)
INFO: pip is looking at multiple versions of feedparser to determine which version is compatible with other requirements. This could take a while.
Using cached feedparser-6.0.3-py3-none-any.whl (80 kB)
Using cached feedparser-6.0.2-py3-none-any.whl (80 kB)
Using cached feedparser-5.2.1.zip (1.2 MB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [1 lines of output]
error in feedparser setup command: use_2to3 is invalid.
[end of output]
1. How can I solve this?
2. Can’t we upgrade the feedparser requirement?Thomas Fredriksen
09/28/2022, 11:22 AMagent-default
, agent-small
, agent-medium
, agent-large
and agent-huge
, each with different amounts of available RAM and CPU.
Now that we with Orion can set KubernetesJob
-infrastructure blocks in the Deployment
-object, having different agents or queues for the different resource tiers no longer seems necessary.
Does anyone know of examples where agents/queues have been used explicitly? I noticed for examples the documentation mention agent queue prioritiesDavid
09/28/2022, 12:32 PMVille Parkkinen
09/28/2022, 1:00 PMtask_input_hash
hashes the block differently on each flow run, and therefore causing cache misses. I’ll attach code snippets needed to reproduce the issue in this thread.eddy davies
09/28/2022, 1:23 PMhelm install --values k8s-agent.yaml prefect-agent prefect/prefect-agent && \
helm install --values k8s-orion.yaml prefect-orion prefect/prefect-orion
Adam Eury
09/28/2022, 2:02 PMset_state
flow run endpoint would be used for?
https://docs.prefect.io/api-ref/rest-api/#/Flow%20Runs/set_flow_run_state_flow_runs__id__set_state_postHawkar Mahmod
09/28/2022, 2:23 PM.fn
that contains a call to get_run_logger
. I get a RuntimeError: There is no active flow or task run context
exceptionMatthew Millendorf
09/28/2022, 3:11 PMAlexey Stoletny
09/28/2022, 3:26 PMraise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '403 Forbidden' for url '<https://api.prefect.cloud/api/accounts/43db7ccd-9f39-41f2-8989-000b28747858/workspaces/cedd89e9-9f12-421e-a17b-94045c976a2a/block_types/e91e7544-7ecd-4fa6-b6d7-a53068cb67fc>'
Response: {'detail': 'protected block types cannot be updated.'}
For more information check: <https://httpstatuses.com/403>
Alexey Stoletny
09/28/2022, 3:26 PMAlexey Stoletny
09/28/2022, 3:27 PMTomás Emilio Silva Ebensperger
09/28/2022, 3:57 PMprefect agent start --work-queue blabla
Guillaume G
09/28/2022, 4:05 PMget_directory
I got
Exception has occurred: TypeError
object of type 'PosixPath' has no len()
Do you know how to use SMB block ?
I also get this issue https://github.com/PrefectHQ/prefect/issues/6971 when I try to put_directory
Thanks 🙂Ilya Galperin
09/28/2022, 4:05 PM2.4.1
release which include the Flow run URL on Cloud? Our notifications seem to still be coming through in the “old style” i.e.David Cupp
09/28/2022, 4:20 PMLance Cuthbert
09/28/2022, 6:09 PMLance Cuthbert
09/28/2022, 6:09 PMMason Menges
09/28/2022, 6:41 PMLance Cuthbert
09/28/2022, 6:43 PM[2022-09-28 18:02:59+0000] ERROR - prefect.S3 | Error downloading Flow from S3: deprecated() got an unexpected keyword argument 'name'
deprecated() got an unexpected keyword argument 'name'
/usr/local/lib/python3.7/site-packages/prefect/client/client.py:177: UserWarning: Client was created with an API token configured for authentication. API tokens are deprecated, please use API keys instead.
"Client was created with an API token configured for authentication. "
Traceback (most recent call last):
File "/usr/local/bin/prefect", line 8, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 96, in flow_run
raise exc
File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 73, in flow_run
flow = storage.get_flow(flow_data.name)
File "/usr/local/lib/python3.7/site-packages/prefect/storage/s3.py", line 89, in get_flow
obj = self._boto3_client.get_object(Bucket=self.bucket, Key=key)
File "/usr/local/lib/python3.7/site-packages/prefect/storage/s3.py", line 219, in _boto3_client
from prefect.utilities.aws import get_boto_client
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/aws.py", line 10, in <module>
import boto3
File "/usr/local/lib/python3.7/site-packages/boto3/__init__.py", line 16, in <module>
from boto3.session import Session
File "/usr/local/lib/python3.7/site-packages/boto3/session.py", line 17, in <module>
import botocore.session
File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 29, in <module>
import botocore.credentials
File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 35, in <module>
from botocore.config import Config
File "/usr/local/lib/python3.7/site-packages/botocore/config.py", line 16, in <module>
from botocore.endpoint import DEFAULT_TIMEOUT, MAX_POOL_CONNECTIONS
File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 22, in <module>
from botocore.awsrequest import create_request_object
File "/usr/local/lib/python3.7/site-packages/botocore/awsrequest.py", line 24, in <module>
import botocore.utils
File "/usr/local/lib/python3.7/site-packages/botocore/utils.py", line 32, in <module>
import botocore.httpsession
File "/usr/local/lib/python3.7/site-packages/botocore/httpsession.py", line 28, in <module>
from urllib3.contrib.pyopenssl import orig_util_SSLContext as SSLContext
File "/usr/local/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 50, in <module>
import OpenSSL.SSL
File "/usr/local/lib/python3.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
from OpenSSL import SSL, crypto
File "/usr/local/lib/python3.7/site-packages/OpenSSL/SSL.py", line 19, in <module>
from OpenSSL.crypto import (
File "/usr/local/lib/python3.7/site-packages/OpenSSL/crypto.py", line 3232, in <module>
name="load_pkcs7_data",
TypeError: deprecated() got an unexpected keyword argument 'name'
`Mason Menges
09/28/2022, 6:58 PMLance Cuthbert
09/28/2022, 7:02 PMUserWarning: Client was created with an API token configured for authentication. API tokens are deprecated, please use API keys instead.
"Client was created with an API token configured for authentication. "
Mason Menges
09/28/2022, 8:17 PMstorage = S3(bucket="********bucket", key="test_flow")
@task
def some_task():
print("test")
with Flow(name="test_flow",
storage=storage,
run_config=UniversalRun(env=dict(AWS_ACCESS_KEY_ID=Secret("aws_personal_key_id").get(),
AWS_SECRET_ACCESS_KEY=Secret("aws_personal_api_key").get()))) as flow:
some_task()
flow.register(project_name="Local-test")
with prefect version 1.3.0Lance Cuthbert
09/28/2022, 8:27 PMprefect agent docker start -t EdYAD_SW9wvRsPfsf42yQQ -n DockerAgent -l <http://prefect.segmint.com|prefect.segmint.com> --volume /opt/segmint:/opt/segmint --show-flow-logs
____ __ _ _ _
| _ \ _ __ ___ / _| ___ ___| |_ / \ __ _ ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __| / _ \ / _` |/ _ \ '_ \| __|
| __/| | | __/ _| __/ (__| |_ / ___ \ (_| | __/ | | | |_
|_| |_| \___|_| \___|\___|\__| /_/ \_\__, |\___|_| |_|\__|
|___/
[2022-09-28 20:34:21,989] INFO - DockerAgent | Starting DockerAgent with labels ['<http://prefect.segmint.com|prefect.segmint.com>']
[2022-09-28 20:34:21,989] INFO - DockerAgent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2022-09-28 20:34:21,989] INFO - DockerAgent | Agent connecting to the Prefect API at <https://api.prefect.io>
[2022-09-28 20:34:22,076] INFO - DockerAgent | Waiting for flow runs...
[2022-09-28 20:34:22,300] INFO - DockerAgent | Found 1 flow run(s) to submit for execution.
[2022-09-28 20:34:22,437] INFO - DockerAgent | Deploying flow run 437b9710-93ce-49ca-a014-16495d381681
[2022-09-28 20:34:22,439] INFO - DockerAgent | Pulling image prefecthq/prefect:1.3.0-python3.7...
[2022-09-28 20:34:37,925] INFO - DockerAgent | Successfully pulled image prefecthq/prefect:1.3.0-python3.7...
+pip install snowflake-connector-python
Collecting snowflake-connector-python
Downloading snowflake_connector_python-2.8.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (21.3 MB)
Collecting oscrypto<2.0.0
Downloading oscrypto-1.3.0-py2.py3-none-any.whl (194 kB)
Requirement already satisfied: cffi<2.0.0,>=1.9 in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (1.15.1)
Requirement already satisfied: charset-normalizer<3,>=2 in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (2.1.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (3.3)
Requirement already satisfied: typing-extensions<5,>=4.3 in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (4.3.0)
Requirement already satisfied: requests<3.0.0 in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (2.28.1)
Collecting filelock<4,>=3.5
Downloading filelock-3.8.0-py3-none-any.whl (10 kB)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (2022.6.15)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (1.26.11)
Collecting pycryptodomex!=3.5.0,<4.0.0,>=3.2
Downloading pycryptodomex-3.15.0-cp35-abi3-manylinux2010_x86_64.whl (2.3 MB)
Requirement already satisfied: setuptools>34.0.0 in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (57.5.0)
Requirement already satisfied: pyjwt<3.0.0 in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (2.4.0)
Collecting cryptography<37.0.0,>=3.1.0
Downloading cryptography-36.0.2-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.7 MB)
Collecting asn1crypto<2.0.0,>0.24.0
Downloading asn1crypto-1.5.1-py2.py3-none-any.whl (105 kB)
Requirement already satisfied: pytz in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (2022.1)
Collecting pyOpenSSL<23.0.0,>=16.2.0
Downloading pyOpenSSL-22.1.0-py3-none-any.whl (57 kB)
Requirement already satisfied: pycparser in /usr/local/lib/python3.7/site-packages (from cffi<2.0.0,>=1.9->snowflake-connector-python) (2.21)
Installing collected packages: asn1crypto, oscrypto, filelock, pycryptodomex, cryptography, pyOpenSSL, snowflake-connector-python
Attempting uninstall: cryptography
Found existing installation: cryptography 37.0.4
Uninstalling cryptography-37.0.4:
Successfully uninstalled cryptography-37.0.4
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.
We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.
pyopenssl 22.1.0 requires cryptography<39,>=38.0.0, but you'll have cryptography 36.0.2 which is incompatible.
Successfully installed asn1crypto-1.5.1 cryptography-36.0.2 filelock-3.8.0 oscrypto-1.3.0 pyOpenSSL-22.1.0 pycryptodomex-3.15.0 snowflake-connector-python-2.8.0
WARNING: You are using pip version 20.2.4; however, version 22.2.2 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
Traceback (most recent call last):
File "/usr/local/bin/prefect", line 8, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 53, in flow_run
result = client.graphql(query)
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 464, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['flow_run'], 'message': 'Missing Authorization header in JWT authentication mode', 'extensions': {'path': '$', 'code': 'invalid-headers', 'exception': {'message': 'Missing Authorization header in JWT authentication mode'}}}]
Mason Menges
09/28/2022, 8:38 PMLance Cuthbert
09/29/2022, 4:13 PMMason Menges
09/29/2022, 4:16 PMLance Cuthbert
09/29/2022, 4:18 PMMason Menges
09/29/2022, 4:19 PMBianca Hoch
09/29/2022, 6:02 PMLance Cuthbert
09/29/2022, 6:25 PMenv={"EXTRA_PIP_PACKAGES": "snowflake-connector-python==2.4.3"}
Bianca Hoch
09/29/2022, 6:27 PM