Is there a recommended way to produce a LOT of graphs in a p

Question

Is there a recommended way to produce a LOT of graphs in a prefect flow with matplotlib If I use `import matplotlib matplotlib use agg ` I get a segfault like this ``` run local sh line 14 34109 Segmentation fault 11 python cli py local pfile developer test input json make run local Error 139 p39 ngs MB MFRANKLIN amplicon analysis meghanfranklin$ Users meghanfranklin opt anaconda3 envs p39 ngs lib python3 9 multiprocessing resource tracker py 216 UserWarning resource tracker There appear to be 2 leaked semaphore objects to clean up at shutdown warnings warn resource tracker There appear to be d ``` but if I don t use that python crashes upside down face ``` Users meghanfranklin opt anaconda3 envs p39 ngs lib python3 9 site packages amp analysis plots py 425 UserWarning Starting a Matplotlib GUI outside of the main thread will likely fail fig ax = plt subplots 2022 10 31 15 02 54 0400 INFO prefect TaskRunner | Task create pipeline run Finished task run for task with final state Success 2022 10 31 15 02 54 121 python 34449 328037 Terminating app due to uncaught exception NSInternalInconsistencyException reason NSWindow drag regions should only be invalidated on the Main Thread First throw call stack 0 CoreFoundation 0x00007ff80bf1e7c3 exceptionPreprocess + 242 1 libobjc A dylib 0x00007ff80bc7ebc3 objc exception throw + 48 2 CoreFoundation 0x00007ff80bf47076 NSException raise + 9 a lot more lines here libc++abi terminating with uncaught exception of type NSException run local sh line 14 34449 Abort trap 6 python cli py local pfile developer test input json make run local Error 134 p39 ngs MB MFRANKLIN amplicon analysis meghanfranklin$ Users meghanfranklin opt anaconda3 envs p39 ngs lib python3 9 multiprocessing resource tracker py 216 UserWarning resource tracker There appear to be 2 leaked semaphore objects to clean up at shutdown warnings warn resource tracker There appear to be d gt cue Mac popup window about sending crash report to apple```

Meghan Franklin · Answer

I get the same kind of error but without the segfault if I reboot my computer which is not a real solution Running in a brand new 10 mins ago environment so it s also possible there s a tiny mismatch in some underlying library python=3 9 prefect=0 15 13

Meghan Franklin · Answer

If I build a new docker container w o setting the matplotlib backend and run the flow from there I get this error ``` 2022 10 31 19 07 35+0000 ERROR prefect TaskRunner | Unexpected error Done lt matplotlib backends backend agg RendererAgg object at 0x7f337ba30c10 gt Traceback most recent call last File opt conda lib python3 9 site packages prefect engine runner py line 48 in inner new state = method self state args kwargs File opt conda lib python3 9 site packages prefect engine task runner py line 876 in get task run state value = prefect utilities executors run task with timeout File opt conda lib python3 9 site packages prefect utilities executors py line 467 in run task with timeout return task run args kwargs type ignore File opt conda lib python3 9 site packages amp analysis analyze py line 48 in analyze sam info process sam stats sample 0 sam df summary folder graph folder File opt conda lib python3 9 site packages amp analysis sam py line 103 in process sam stats phred histogram mut phreds f graph base Qscores mismatches png File opt conda lib python3 9 site packages amp analysis plots py line 71 in phred histogram plt savefig outfile File opt conda lib python3 9 site packages matplotlib pyplot py line 958 in savefig res = fig savefig args kwargs File opt conda lib python3 9 site packages matplotlib figure py line 3019 in savefig self canvas print figure fname kwargs File opt conda lib python3 9 site packages matplotlib backend bases py line 2319 in print figure result = print method File opt conda lib python3 9 site packages matplotlib backend bases py line 1648 in wrapper return func args kwargs File opt conda lib python3 9 site packages matplotlib api deprecation py line 412 in wrapper return func inner args inner kwargs File opt conda lib python3 9 site packages matplotlib backends backend agg py line 540 in print png FigureCanvasAgg draw self File opt conda lib python3 9 site packages matplotlib backends backend agg py line 436 in draw self figure draw self renderer File opt conda lib python3 9 site packages matplotlib backend bases py line 1533 in draw def draw renderer raise Done renderer matplotlib backend bases get renderer lt locals gt Done lt matplotlib backends backend agg RendererAgg object at 0x7f337ba30c10 gt 2022 10 31 19 07 35+0000 ERROR prefect TaskRunner | Task analyze sam info 1 Unexpected error while running task Done lt matplotlib backends backend agg RendererAgg object at 0x7f337ba30c10 gt ``` At least there s no segfault

Zanie · Answer

Hm this is peculiar I ve seen issues without Agg but with the Agg backend it should be fine <https github com PrefectHQ prefect issues 5991 issuecomment 1213310687>

Zanie · Answer

Are you getting any warnings from matplotlib about threads

Meghan Franklin · Answer

No other warnings than the ones I posted

Meghan Franklin · Answer

I am on a Mac too so I m in the crash category of your post

Meghan Franklin · Answer

Ugh if I set the matplolib backend build a new docker run the flow in the docker segfault

Zanie · Answer

Can you open an issue with a MRE

Zanie · Answer

I ll be working on a fix for this soon In the meantime using an `async` task or flow will run things in the main thread

Meghan Franklin · Answer

I haven t done that before can you point me to where to do that

Jeff Hale · Answer

Sure Meghan <https github com PrefectHQ prefect issues new choose>