I'm using jaydebeapi on windows and prefect keeps throwing the error java.sql.SQLException: No suitable driver found for. This flow works when not using prefect server. However, it appears it can't find the jar file possibly. Where is the best place to put the file? This is also using dask distributed

Hi @Matthew Maldonado - this is most likely a PATH issue; I recommend: - determining which location you can run your flows from without using server - append that location to your local agent’s import PATH with the

-p

flag:

prefect agent start -p /full/path/to/stuff

Thank you, I'll give it a shot

interesting; so the agent only tracks PYTHONPATH, you might need to manually append the relevant locations to your PATH before starting the agent. Honestly though, given that your flow has non python dependencies I’d recommend using the Docker agent and Docker storage instead of Local, as that gives you more control over the runtime environment of your flow

Ok, so I need to get a jvm on the docker file prefect builds. Do i prebuild a docker file with a JVM and specify that as the docker file?

yea you have a few options here: - you can specify a

base_image

on the Docker storage option (using something you pre-built) - you can specify a

dockerfile

with the location of a Dockerfile so that you only have to build one complete image (note that if you’re running on Windows there is a possible bug with this kwarg that we will be fixing very soon: https://github.com/PrefectHQ/prefect/issues/2332)

I figured out one workaround which was putting the Jar file in the CLASSPATH environment variable. However, I don't think running dask distributed for processing works with JayDeBeApi because I ekpt getting the error TypeError: can't pickle JavaField objects

Ah, interesting — glad you made some progress

So i made some progress but the flow seemed to work the first time. It then failed after that saying it can not find the jar anymore when I use prefect server. When I run it in a jupyter notebook it fails to run. I get distributed.protocol.core - CRITICAL - Failed to deserialize. A restart fixes it but thats not really practical. I think it has to possibly do with th jdbc package I'm using. Not sure though