https://prefect.io
Join Slack
Hey Everyone. We are running a ShellTask that just calls another python file in the directory to e...
# ask-community
i
Hey Everyone. We are running a ShellTask that just calls another python file in the directory to execute. This works when running the task, but when we register the flow and run from the cloud the ShellTask fails. The python code itself is importing a module that needs to authenticate to azure, and that is where the cloud failure comes, however we use the same dependecy in other flows with out the ShellTask and the authentication is fine. Any Ideas?
k
It sounds like you have multiple environments and the one running the shell task doesn’t have the dependencies installed? Or does the agent have it installed but it seems that the ShellTask is not using the right environment?
i
in the shell task command we are sourcing the venv ahead of the bash command
to make sure of that
k
Could you show me how you call the ShellTask?
i
Copy code
shell_task = ShellTask(stream_output=True)

with Flow("Barra Data") as flow:
   generate_risk_tables = shell_task(
         command="source /home/stromadmin/strom-neutron/.venv/bin/activate; python /home/stromadmin/strom-neutron/save_risk_tables.py")
k
Maybe you can try seeing if the venv is activates by doing:
Copy code
import sys
print(sys.prefix)
inside the python file that you call?
i
ya it says that is activated
k
Then I guess maybe the library is just not getting installed in the venv right? What library is it?
i
its a library we created
and installed in the environment
we have other tasks
in the code
that use that environment too
and then use the shell task to run another script
its only in the shell task where we are calling that file where it cant get the library
and the call actually can find the library
the issue is that it doesnt recognize our azure authentication in the shellTask
but it does in the main call
k
You mean the import works but the authentication step does not? Does it rely on env variables?
i
shell_task = ShellTask( stream_output=True, env=os.environ.copy(), )
this solved it
k
Ah gotcha ok
i
it wasnt obvious that it sourcing the .venv
wouldnt maintain the sambe base env
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by