Milton
03/18/2022, 10:01 PMKevin Kho
Milton
03/19/2022, 12:03 AMKevin Kho
Milton
03/21/2022, 3:57 PMKevin Kho
pygitlab
version for me?Milton
03/21/2022, 4:27 PMrun_config = KubernetesRun(image="path/to/prefect:0.14.11",
labels= ["your", "label"],
cpu_limit="1",
cpu_request="0.5",
memory_limit="200Mi",
memory_request="100Mi",
)
``````Kevin Kho
Milton
03/21/2022, 4:30 PMKevin Kho
Milton
03/21/2022, 4:33 PMAnna Geller
Milton
03/21/2022, 4:38 PMprefect diagnostics
?Anna Geller
Milton
03/21/2022, 4:43 PMAnna Geller
Milton
03/21/2022, 4:55 PMFROM <internal-python-base-image>
ENV PYTHONPATH /home/svc_app
ENV PREFECT__USER_CONFIG_PATH /home/svc_app/.prefect/config.toml
# copy over files needed for init
COPY --chown=svc_app \
entrypoint.sh \
/home/svc_app/
# # pre-install base dependencies
USER root
RUN curl -OL "<https://github.com/krallin/tini/releases/download/v0.19.0/tini_0.19.0-amd64.rpm>"
RUN yum localinstall tini_0.19.0-amd64.rpm -y
RUN rm tini_0.19.0-amd64.rpm
# Environment Setup
USER svc_app
RUN pip install --upgrade pip
WORKDIR /home/svc_app
RUN mkdir .prefect
RUN pip uninstall prefect -y
RUN pip install
'prefect[orchestration_extras]==0.14.11'
ENTRYPOINT ["tini", "-g", "--", "/home/svc_app/entrypoint.sh"]
run_config = KubernetesRun(image="path/to/prefect:0.14.11",
labels= ["your", "label"],
cpu_limit="1",
cpu_request="0.5",
memory_limit="200Mi",
memory_request="100Mi",
)
$ prefect diagnostics
{
"config_overrides": {
"run_config": {
"cpu_limit": true,
"cpu_request": true,
"env": {
...
},
"image": true,
"labels": true,
"memory_limit": true,
"memory_request": true
},
"schedule": {
"cron": true,
"tz": true
},
"storage": {
"access_token_secret": true,
"host": true,
"path": true,
"ref": true,
"repo": true
}
},
"env_vars": [
"PREFECT__STORAGE__REF",
"PREFECT__USER_CONFIG_PATH"
],
"system_information": {
"platform": "Linux-3.10.0-1062.el7.x86_64-x86_64-with-redhat-7.7-Maipo",
"prefect_backend": "cloud",
"prefect_version": "0.14.11",
"python_version": "3.7.3"
}
}
Kevin Kho
Milton
03/21/2022, 5:13 PMprefect execute flow-run
? Trying to pinpoint which line of code that does the downloadingKevin Kho
Milton
03/21/2022, 5:18 PMprefect execute flow-run
? Trying to pinpoint which line of code that does the downloadingKevin Kho
prefect execute flow-run
will already have the Flow downloaded so i’s not there. Anything related to cloning will be found here . I don’t see enough debug level logs thoughMilton
03/21/2022, 5:20 PMprefect execute flow-run
somehowprefect execute flow-run
Kevin Kho
storage.get_flow
called here that then goes to the Gitlab get_flow
. I know these links are Prefect 1.1 but this file hasn’t changed significantly. Can you try running with the base Prefect image?prefect execute flow-run
Milton
03/21/2022, 5:35 PMprefecthq/prefect 0.14.11
Kevin Kho
ENTRYPOINT
of the image to download the Git repo and pip install -e .
on the image. That will get your up to date dependencies during execution. I am positive you have something like that going on, and it should be possible in 1.0Milton
03/22/2022, 9:14 PMstorage = Git(
repo="org/repo", # name of repo
flow_path="flows/my_flow.py", # location of flow file in repo
repo_host="<http://gitlab.com|gitlab.com>", # repo host name, which may be custom
git_token_secret_name="MY_GIT_ACCESS_TOKEN",# name of Secret containing Deploy Token
git_token_username="myuser" # username associated with the Deploy Token
)
the flow run triggered returns
└── 204545 | INFO | Entered state <Scheduled>: Flow run scheduled.
└── 204553 | INFO | Entered state <Submitted>: Submitted for execution
└── 204553 | INFO | Submitted for execution: Job prefect-job-36cd933b
└── 204557 | INFO | Entered state <Failed>: Failed to load and execute flow run: HangupException(‘ssh: Could not resolve hostname https: Name or service not known\r’)
└── 204557 | ERROR | Failed to load and execute flow run: HangupException(‘ssh: Could not resolve hostname https: Name or service not known\r’)
Flow run failed!
ssh
would be used at all and we have use_ssh set to false by default in Git StorageKevin Kho
.sql
or .yaml
files but Python imports won’t work because you’d need some heavy Python path manipulation. It might not even be doable. I’ve seen people try. Not discourage you to try on that. I just really don’t know if it can be done.Milton
03/22/2022, 9:19 PMssh
would be used in the first place?Kevin Kho
use_ssh
was a flag and I thought it was set to False.Milton
03/22/2022, 9:34 PMKevin Kho
Milton
03/22/2022, 9:36 PMKevin Kho
Milton
03/22/2022, 9:46 PMKevin Kho
Milton
03/22/2022, 10:08 PMfrom prefect.utilities.git import TemporaryGitRepo
with TemporaryGitRepo(
git_clone_url=“https//<user><pass>@self-hosted-gitlab.com/<repo>.git",
branch_name=“master”,
tag=None,
commit=None,
clone_depth=1,
) as temp_repo:
print(temp_repo.temp_dir.name)
Traceback (most recent call last):
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 1091, in fetch_pack
refs, server_capabilities = read_pkt_refs(proto)
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 234, in read_pkt_refs
for pkt in proto.read_pkt_seq():
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/protocol.py”, line 287, in read_pkt_seq
pkt = self.read_pkt_line()
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/protocol.py”, line 232, in read_pkt_line
raise HangupException()
dulwich.errors.HangupException: The remote server unexpectedly closed the connection.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “test.py”, line 8, in <module>
clone_depth=1,
File “/home/<user>/miniconda3/lib/python3.7/site-packages/prefect/utilities/git.py”, line 48, in enter
source=self.git_clone_url, target=self.temp_dir.name, depth=self.clone_depth
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/porcelain.py”, line 451, in clone
depth=depth,
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 535, in clone
result = self.fetch(path, target, depth=depth)
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 607, in fetch
depth=depth,
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 1093, in fetch_pack
raise _remote_error_from_stderr(stderr)
dulwich.errors.HangupException: ssh: Could not resolve hostname https: Name or service not known
Kevin Kho
git_clone_url_secret_name
instead to create the stringMilton
03/22/2022, 10:25 PMgit clone https://<user>:<pass>@self-hosted-gitlab.com/<repo>.git
worksAnna Geller
Milton
03/22/2022, 10:37 PMAnna Geller
Milton
03/22/2022, 10:41 PMfrom prefect.utilities.git import TemporaryGitRepo
with TemporaryGitRepo(
git_clone_url=“https//<user><pass>@self-hosted-gitlab.com/<repo>.git",
branch_name=“master”,
tag=None,
commit=None,
clone_depth=1,
) as temp_repo:
print(temp_repo.temp_dir.name)gives
Traceback (most recent call last):
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 1091, in fetch_pack
refs, server_capabilities = read_pkt_refs(proto)
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 234, in read_pkt_refs
for pkt in proto.read_pkt_seq():
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/protocol.py”, line 287, in read_pkt_seq
pkt = self.read_pkt_line()
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/protocol.py”, line 232, in read_pkt_line
raise HangupException()
dulwich.errors.HangupException: The remote server unexpectedly closed the connection.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “test.py”, line 8, in <module>
clone_depth=1,
File “/home/<user>/miniconda3/lib/python3.7/site-packages/prefect/utilities/git.py”, line 48, in enter
source=self.git_clone_url, target=self.temp_dir.name, depth=self.clone_depth
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/porcelain.py”, line 451, in clone
depth=depth,
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 535, in clone
result = self.fetch(path, target, depth=depth)
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 607, in fetch
depth=depth,
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 1093, in fetch_pack
raise _remote_error_from_stderr(stderr)
dulwich.errors.HangupException: ssh: Could not resolve hostname https: Name or service not known
Anna Geller
with TemporaryGitRepo(
git_clone_url="https://<user>:<pass>@self-hosted-gitlab.com/<repo>.git",
branch_name="master", # try without that since your git clone command that worked didn't have that
Kevin Kho
Milton
03/23/2022, 3:40 PMKevin Kho