Hi Brian, thanks for your response. I was referring specifically to parallelizing your Python code with Dask. But it's interesting that you achieved parallelism using SubDags - afaik SubDags are run by SequentialExecutor to prevent from deadlocks.

--

Lead Community Engineer at Prefect, Data Professional, Cloud & .py fan. www.annageller.com. Get my articles via email: https://annageller.medium.com/subscribe

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store