Brian Phillips
10/31/2022, 4:57 PMeddy davies
10/31/2022, 4:58 PMconfig
file in my .kube
file but doesn't seem to work, any advice?Meghan Franklin
10/31/2022, 7:04 PMimport matplotlib; matplotlib.use('agg')
I get a segfault like this:
./run_local.sh: line 14: 34109 Segmentation fault: 11 python cli.py local -pfile developer/test_input.json
make: *** [run_local] Error 139
(p39_ngs) MB-MFRANKLIN:amplicon-analysis meghanfranklin$ /Users/meghanfranklin/opt/anaconda3/envs/p39_ngs/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
but if I don’t use that, python crashes 🙃
/Users/meghanfranklin/opt/anaconda3/envs/p39_ngs/lib/python3.9/site-packages/amp_analysis/plots.py:425: UserWarning: Starting a Matplotlib GUI outside of the main thread will likely fail.
fig, ax = plt.subplots()
[2022-10-31 15:02:54-0400] INFO - prefect.TaskRunner | Task 'create_pipeline_run': Finished task run for task with final state: 'Success'
2022-10-31 15:02:54.121 python[34449:328037] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'NSWindow drag regions should only be invalidated on the Main Thread!'
*** First throw call stack:
(
0 CoreFoundation 0x00007ff80bf1e7c3 __exceptionPreprocess + 242
1 libobjc.A.dylib 0x00007ff80bc7ebc3 objc_exception_throw + 48
2 CoreFoundation 0x00007ff80bf47076 -[NSException raise] + 9
[a lot more lines here]
)
libc++abi: terminating with uncaught exception of type NSException
./run_local.sh: line 14: 34449 Abort trap: 6 python cli.py local -pfile developer/test_input.json
make: *** [run_local] Error 134
(p39_ngs) MB-MFRANKLIN:amplicon-analysis meghanfranklin$ /Users/meghanfranklin/opt/anaconda3/envs/p39_ngs/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
-> cue Mac popup window about sending crash report to apple
David Cupp
10/31/2022, 7:54 PMBen Muller
10/31/2022, 10:33 PMprefect run -p test_flow.py
Is this an option in 2.0 on the cli? or do I have to initate through python every time?Ben Muller
10/31/2022, 11:31 PMSequentialTaskRunner
with .submit
.
Would this just be the same as not specifying a task_runner
and just running the tasks without .submit
or am I missing something?Tony Yun
11/01/2022, 2:46 AMTim Galvin
11/01/2022, 3:01 AMdask_jobqeue.SLURMCluster
) with a DaskTaskExecutor
being used, with 10 separate compute nodes being pulled into the distributed dask schedular.
• A single large Flow with ~7 tasks, with each task calling a separate python script's main
I am just running the postgres database remotely. I have not set an prefect orion
server running remotely - although I can try this.
I found that regular running of the pipeline would often raise a TimeoutError
in an unpredictable manner. Setting ``PREFECT_ORION_DATABASE_CONNECTION_TIMEOUT=20` eliminated these. Now my problems seem to be TimeoutErrors
when the flow is closing. I can confirm that all expected data products expected by the pipeline have been created and stored on disk, and all log messages have been issued. The flow is essentially finished executing and wrapping up and then the error is raised. The traceback is too long to post as a comment ( 😢 ) , so it is below.
File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
self._handle_dbapi_exception(
File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2128, in _handle_dbapi_exception
util.raise_(exc_info[1], with_traceback=exc_info[2])
File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
raise exception
File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
self.dialect.do_execute(
File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 479, in execute
self._adapt_connection.await_(
File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 68, in await_only
return current.driver.switch(awaitable)
File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 121, in greenlet_spawn
value = await result
File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 408, in _prepare_and_execute
await adapt_connection._start_transaction()
File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 716, in _start_transaction
self._handle_exception(error)
File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 684, in _handle_exception
raise error
File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 714, in _start_transaction
await self._transaction.start()
File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/asyncpg/transaction.py", line 138, in start
await self._connection.execute(query)
File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/asyncpg/connection.py", line 318, in execute
return await self._protocol.query(query, timeout)
File "asyncpg/protocol/protocol.pyx", line 338, in query
asyncio.exceptions.TimeoutError
Ben Muller
11/01/2022, 6:09 AMECSTask
that I have successfully deployed :
import sys
import prefect
from prefect import flow, task, get_run_logger
from utilities import AN_IMPORTED_MESSAGE
from prefect_aws.ecs import ECSTask
ecs_task_block = ECSTask.load("staging-test")
@task
def log_task(name):
logger = get_run_logger()
<http://logger.info|logger.info>("Hello %s!", name)
<http://logger.info|logger.info>("Prefect Version = %s 🚀", prefect.__version__)
logger.debug(AN_IMPORTED_MESSAGE)
@flow()
def log_flow(name: str):
log_task(name)
if __name__ == "__main__":
name = sys.argv[1]
log_flow(name)
Our org uses 1.0 at present and we have never had to pass AWS Credentials - we dont use credentials like this as we usually use roles that you can assume. I believe this was the case in Prefect 1.0 if I remember correctly. All my agent has to do is assume a role ( and it has access to whatever I need it to with its task role that I set up in deployment ).
So with these blocks all requiring credentials, I am wondering if these are optional and would be picked up by the default AWS environment variables if I leave them blank or I will need to configure some type of user access for a prefect machine user?
Secondly: I am trying a deployment with storage - but it is coming up with an error:
from flows.log_flow import log_flow
from prefect.deployments import Deployment
from prefect.filesystems import S3
storage_block = S3(bucket_path="prefect-2-test")
deployment = Deployment.build_from_flow(
flow=log_flow,
name="log-simple",
parameters={"name": "Marvin"},
infra_overrides={"env": {"PREFECT_LOGGING_LEVEL": "DEBUG"}},
work_queue_name="staging",
storage=S3.load("staging-test-block"),
)
if __name__ == "__main__":
storage_block.save("staging-test-block", overwrite=True)
deployment.apply()
ERROR:
File "/usr/local/Caskroom/miniconda/base/envs/prefect2/lib/python3.10/site-packages/aiobotocore/client.py", line 82, in create_client
self._register_s3_control_events(
TypeError: ClientCreator._register_s3_control_events() takes 2 positional arguments but 6 were given
Any ideas to help me overcome this?eddy davies
11/01/2022, 10:39 AMaws-iam-authenticator
, any ideas?Michael Hadorn
11/01/2022, 10:58 AMtrigger=always_run, skip_on_upstream_skip=False
, but currently we have to abort the full flow run. So this task will never be executed.Oscar Björhn
11/01/2022, 12:53 PMPatrick Alves
11/01/2022, 1:45 PM❯ PREFECT_API_URL=<https://prefect.xxx.xxx/api> prefect deployment ls
I am getting:
File "/home/patrick/miniconda3/envs/prefect/lib/python3.8/ssl.py", line 944, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)
An exception occurred.
• I've tried set envs to disable certificate verification: PYTHONHTTPSVERIFY=false
• Tried to add the server certificate (*.xxx.xxx.crt) on the server copying the CRT file /usr/local/share/ca-certificates/
and update the CA store: sudo update-ca-certificates
Nothing works.
Any tip to solve this?Roger Webb
11/01/2022, 2:01 PMKalise Richmond
11/01/2022, 3:03 PMAusten Bouza
11/01/2022, 4:00 PMNoam Banay
11/01/2022, 4:26 PMJavier Ruere
11/01/2022, 5:46 PMJavier Ruere
11/01/2022, 5:47 PM<http://anyio.to|anyio.to>_thread
internally.
I'm using the ConcurrentTaskRunner.
The flow is ran like this:
anyio.run(
generate_report,
datetime.date.fromisoformat(run_date),
backend_options={"debug": True, "use_uvloop": True},
)
The problem I' having is that some Tasks finish but they remain in state Running forever. There's no output I could see to understand the problem.
Help, please.merlin
11/01/2022, 6:02 PMAwaitingRetry
?
@task
# some code, oh found out a blocker I'd like to wait for
time_2_hours_from_now= .... some date function...
if function_is_blocked:
return AwaitingRetry(scheduled_time=time_2_hours_from_now
Jon Young
11/01/2022, 6:43 PMwith prefect.Flow(
name="SOME_DYNAMIC_NAME" # instead of hardcoding this, i want to pass a value when registering / calling it
Carlo
11/01/2022, 6:48 PMVadym Dytyniak
11/01/2022, 7:28 PMJon Young
11/01/2022, 8:31 PMmerlin
11/01/2022, 9:04 PM@task
def fail_task():
return Failed()
@flow
def fail_flow_i_hope():
fail_task()
fail_flow_i_hope()
11:42:32.292 | DEBUG | Task run 'always_fail_task-d83fd752-0' - Beginning execution...
11:42:32.302 | INFO | Task run 'always_fail_task-d83fd752-0' - Finished in state Completed()
...
11:42:32.313 | INFO | Flow run 'ludicrous-lemur' - Finished in state Completed('All states completed.')
I'd like a task to fail, and then do retries (or not) but end the steps in the flow, with the flow also failed.Jon Young
11/01/2022, 9:53 PMTraceback (most recent call last):
File "/Users/jonyoung/Dev/workflows/.venv/lib/python3.9/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/Users/jonyoung/Dev/workflows/.venv/lib/python3.9/site-packages/prefect/engine/flow_runner.py", line 569, in get_flow_run_state
executors.prepare_upstream_states_for_mapping(
File "/Users/jonyoung/Dev/workflows/.venv/lib/python3.9/site-packages/prefect/utilities/executors.py", line 682, in prepare_upstream_states_for_mapping
value = upstream_state.result[i]
KeyError: 0
└── 17:51:25 | ERROR | Unexpected error occured in FlowRunner: KeyError(0)
any help what this error means?Aaron
11/01/2022, 10:01 PMAhmed Ezzat
11/01/2022, 10:37 PMBen Muller
11/02/2022, 2:20 AMprefect-aws
deps installed.
I have an infrastructure ECSTask
block that pulls a custom image:
FROM prefecthq/prefect:2.6.5-python3.9
RUN pip install prefect-aws s3fs==2022.5.0
This installs everything I need to run a demo flow that is:
import prefect
from prefect import flow, task, get_run_logger
from utilities import AN_IMPORTED_MESSAGE
from prefect_aws.ecs import ECSTask
ecs_task_block = ECSTask.load("staging-test")
@task
def log_task():
logger = get_run_logger()
<http://logger.info|logger.info>("Hello %s!", "BEN")
<http://logger.info|logger.info>("Prefect Version = %s 🚀", prefect.__version__)
logger.debug(AN_IMPORTED_MESSAGE)
@flow
def my_test_flow(name: str):
log_task(name)
if __name__ == "__main__":
my_test_flow()
This flow is using S3 Block Storage, now when I trigger this flow from the cloud UI the task is generated in my cluster, it just fails with a ModuleNotFoundError: No module named 'prefect_aws.ecs'
More details in 🧵iKeepo w
11/02/2022, 6:36 AMiKeepo w
11/02/2022, 6:36 AMTim-Oliver
11/02/2022, 8:02 AMBianca Hoch
11/02/2022, 9:08 PM