https://prefect.io logo
Title
m

Matthew Maldonado

04/17/2020, 8:31 PM
I'm using jaydebeapi on windows and prefect keeps throwing the error java.sql.SQLException: No suitable driver found for. This flow works when not using prefect server. However, it appears it can't find the jar file possibly. Where is the best place to put the file? This is also using dask distributed
c

Chris White

04/17/2020, 8:59 PM
Hi @Matthew Maldonado - this is most likely a PATH issue; I recommend: - determining which location you can run your flows from without using server - append that location to your local agent’s import PATH with the
-p
flag:
prefect agent start -p /full/path/to/stuff
m

Matthew Maldonado

04/17/2020, 9:13 PM
Thank you, I'll give it a shot
I can find a json file when I use the full path but jars seem to fail
at least in the case of using jdbc
c

Chris White

04/17/2020, 10:23 PM
interesting; so the agent only tracks PYTHONPATH, you might need to manually append the relevant locations to your PATH before starting the agent. Honestly though, given that your flow has non python dependencies I’d recommend using the Docker agent and Docker storage instead of Local, as that gives you more control over the runtime environment of your flow
m

Matthew Maldonado

04/18/2020, 12:57 AM
Ok, so I need to get a jvm on the docker file prefect builds. Do i prebuild a docker file with a JVM and specify that as the docker file?
c

Chris White

04/18/2020, 12:59 AM
yea you have a few options here: - you can specify a
base_image
on the Docker storage option (using something you pre-built) - you can specify a
dockerfile
with the location of a Dockerfile so that you only have to build one complete image (note that if you’re running on Windows there is a possible bug with this kwarg that we will be fixing very soon: https://github.com/PrefectHQ/prefect/issues/2332)
m

Matthew Maldonado

04/19/2020, 1:13 AM
I figured out one workaround which was putting the Jar file in the CLASSPATH environment variable. However, I don't think running dask distributed for processing works with JayDeBeApi because I ekpt getting the error TypeError: can't pickle JavaField objects
Thanks, for your help
c

Chris White

04/19/2020, 1:15 AM
Ah, interesting — glad you made some progress
m

Matthew Maldonado

04/19/2020, 4:09 AM
So i made some progress but the flow seemed to work the first time. It then failed after that saying it can not find the jar anymore when I use prefect server. When I run it in a jupyter notebook it fails to run. I get distributed.protocol.core - CRITICAL - Failed to deserialize. A restart fixes it but thats not really practical. I think it has to possibly do with th jdbc package I'm using. Not sure though