Saksham Dixit
05/10/2021, 4:51 PMSaksham Dixit
05/10/2021, 4:52 PMSaksham Dixit
05/10/2021, 4:52 PMSaksham Dixit
05/10/2021, 4:53 PMMatthew Millendorf
05/10/2021, 5:17 PMBelal Aboabdo
05/10/2021, 6:10 PMprefect build -p
with the following error. Anyone experience this issue?
Beginning health checks...
System Version check: OK
Traceback (most recent call last):
File "/opt/prefect/healthcheck.py", line 150, in <module>
flows = import_flow_from_script_check(flow_file_paths)
File "/opt/prefect/healthcheck.py", line 63, in import_flow_from_script_check
flows.append(extract_flow_from_file(file_path=flow_file_path))
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/storage.py", line 86, in extract_flow_from_file
exec(contents, exec_vals)
File "<string>", line 129, in <module>
NameError: name '__file__' is not defined
Enda Peng
05/10/2021, 6:44 PMfoo.py
where flowA
is encoded. I can easily run into following cases:
• register with -p ./foo.py
, it doesn’t work while -p foo.py
works.
• Under same folder /pipelines/
there are two files flow.py, common_task.py
, I can also run into import error, sometimes from .common_task import xxx
works sometimes not, depending on how PYTHONPATH is set.
So far my experience is to make sure local agent run on the same level where my resiger
command is called, no leading dot in register command etc. Curious about whether ppl have better ideas on how to manage the entire process.Fabrice Toussaint
05/11/2021, 8:00 AMSalohy
05/11/2021, 9:00 AMprefect.tasks.sql_server
requires Prefect to be installed with the "sql_server" extra`.
I am using, docker for storage using a docker file as explained here. My code looks like this
from prefect.tasks.sql_server import SqlServerExecute
STORAGE = Docker(registry_url="<http://xxx.azurecr.io|xxx.azurecr.io>",
image_name="prefect/test",
dockerfile="Dockerfile",
python_dependencies=["pyodbc"])
SQL = SqlServerExecute(db_name="myDb",
user="user",
host="<tcp:xxx.database.windows.net>",
port=1433,
driver="ODBC Driver 17 for SQL Server")
QUERY = """
CREATE TABLE [dbo].[myTable] (
[id] VARCHAR(22) COLLATE Latin1_General_100_BIN2 NOT NULL,
[FirstName] VARCHAR(222) NULL,
[LastName] VARCHAR(222) NULL
)
"""
@task
def create_table():
logger = prefect.context.get('logger')
<http://logger.info|logger.info>("Create table in DW for loading the results")
SQL.run(query=QUERY, password='myPass')
with Flow("candidates-flow", storage=STORAGE, run_config=RUN_CONFIG, executor=EXECUTOR) as flow:
create_table()
Can someone help? I do not understand the error. Many thanks already 🙏David
05/11/2021, 10:25 AMAn error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.
The ECS agent logs (verbose) isn’t very helpful and I can’t debug the error as it happens when the system at its peak.
And I have couple of questions -
1. Why does the flow register itself on AWS only when its running and not when it was registered?
2. Can I ask prefect to just dockerize my flow instead of register a new task definition (because nothing changed in the task definition), I saw the parameter of idempotency_key
but I don’t understand if this is helpful in my situation
3. Should I call register
if my docker is the only thing changed? or should I call flow,storage.build
instead?
I can provide code if needed,
Thank you in advanceSaulius Beinorius
05/11/2021, 10:39 AMRunConfig/ECSRun
). When I am trying to start the agent, it tries to build the default parameters for a task definition and fails (since we don't have a default VPC). Is there a way to start the ECS agent without providing a task template and to only rely on the flow configuration instead?Ian Harvey
05/11/2021, 3:37 PMPREFECT__LOGGING__EXTRA_LOGGERS
accept wildcards? So instead of something like
flow.run_config = LocalRun(
env={
"PREFECT__LOGGING__EXTRA_LOGGERS": json.dumps([
'foo.bar.aaa',
'foo.bar.bbb',
])
}
)
I could use
flow.run_config = LocalRun(
env={
"PREFECT__LOGGING__EXTRA_LOGGERS": json.dumps([
'foo.bar.*'
])
}
)
With larger projects it's quite time consuming to add each class individually.Matej
05/11/2021, 3:56 PMprefect.utilities.exceptions.AuthorizationError: [{'path': ['auth_info'], 'message': 'AuthenticationError: Forbidden', 'extensions': {'code': 'UNAUTHENTICATED'}}]
I am unsure on how to set up the yaml variables:
- name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
value: ''
- name: PREFECT__CLOUD__API
value: <https://api.prefect.io>
- name: NAMESPACE
value: default
- name: IMAGE_PULL_SECRETS
value: ''
- name: PREFECT__CLOUD__AGENT__LABELS
value: '[]'
- name: JOB_MEM_REQUEST
value: ''
- name: JOB_MEM_LIMIT
value: ''
- name: JOB_CPU_REQUEST
value: ''
- name: JOB_CPU_LIMIT
value: ''
- name: IMAGE_PULL_POLICY
value: ''
- name: SERVICE_ACCOUNT_NAME
value: ''
- name: PREFECT__BACKEND
value: cloud
- name: PREFECT__CLOUD__AGENT__AGENT_ADDRESS
value: http://:8080
image: prefecthq/prefect:0.14
specifically the prefect__cloud___agent__address seem to be incorrect.
Q: Is there a better documentation on how to set up prefect-agent ?Christian
05/11/2021, 3:58 PMJoseph Loss
05/11/2021, 5:15 PMMatej
05/11/2021, 5:17 PM[2021-05-11 19:11:14+0200] ERROR - prefect.FlowRunner | Unexpected error: ModuleNotFoundError("No module named 'prefect'")
it seems that the worker does not have the prefect installed.
Dask is running in a kubernetes cluster so is there a smart way on how to install prefect module on all the dask workers?
Msapatil
05/11/2021, 5:41 PMSalohy
05/11/2021, 5:54 PMTypeError: execute() takes no keyword arguments
When running sql query using SqlServerExecute. Here is my task code following the doc explained here https://docs.prefect.io/api/latest/tasks/sql_server.html#sqlserverexecute
from prefect.tasks.sql_server import SqlServerExecute
SQL = SqlServerExecute(db_name="my"Db",
user="user",
host="<tcp:xxxs.database.windows.net>",
port=1433,
driver="ODBC Driver 17 for SQL Server")
QUERY = "CREATE TABLE [dbo].[CandidateDetails] ([CandidateId] VARCHAR(22) COLLATE Latin1_General_100_BIN2 NOT NULL,[FirstName] VARCHAR(222) NULL,[LastName] VARCHAR(222) NULL)"
@task
def create_table():
logger = prefect.context.get('logger')
<http://logger.info|logger.info>("Create table in DW for loading the results")
SQL.run(query=QUERY, password='myPass')
Can you help me on this? many thanks already 🙂Matthew Blau
05/11/2021, 6:19 PMZach Hodowanec
05/11/2021, 10:59 PMprefect --help
results.Anze Kravanja
05/12/2021, 3:30 AMfrom prefect import Task, Flow, task
@task()
def task_success():
print("Success!")
return {'p': 1, 'k': 2, 'j': 3}
@task()
def task_to_fail(p, k):
print(f"p: {p}; k: {k}")
raise NotImplementedError()
with Flow("catch-all-failed-flow", state_handlers=[STATE_HANDLER_CATCH_ALL_FAILED]) as flow:
ts = task_success()
tf = task_to_fail(ts) # how can I map p and k params to the task_to_fail task instead of passing the whole dict in p but nothing as k
flow.run()
I know I can do ts[‘p’] and ts[‘k’], but I’m looking for something like **ts
Thank you!Kamil Okáč
05/12/2021, 8:02 AMfrom prefect.configuration import config
config.tasks.defaults.max_retries = 2
config.tasks.defaults.retry_delay = datetime.timedelta(seconds=10)
Sven Teresniak
05/12/2021, 11:52 AM[~] -> docker run -it prefecthq/<server:core-0.14.18> pip show prefect
Name: prefect
Version: 0.14.16+95.g377f2fe7a
holy crap!
what is the purpose of the version number in the tag if not the prefect version?Sven Teresniak
05/12/2021, 12:25 PMclick >= 7.0.0
will install click v8.0.0
. This is not compatible with click v7.1.2
which is part of the prefect-server docker image (with tag core-0.14.18
)Joseph Ellis
05/12/2021, 12:27 PMAttributeError: ‘Context’ object has no attribute ‘flow_run_id’Any ideas?
Joseph Ellis
05/12/2021, 12:28 PMprefect.context.flow_run_id
Vincent
05/12/2021, 1:10 PMnvidia-docker
with the prefect docker agent?Tim Enders
05/12/2021, 1:20 PMRobert Bastian
05/12/2021, 1:40 PMale
05/12/2021, 3:39 PM