Milton
03/18/2022, 10:01 PMKevin Kho
03/18/2022, 11:35 PMMilton
03/19/2022, 12:03 AMKevin Kho
03/21/2022, 2:12 PMMilton
03/21/2022, 3:57 PMKevin Kho
03/21/2022, 3:58 PMpygitlab
version for me?Milton
03/21/2022, 4:27 PMrun_config = KubernetesRun(image="path/to/prefect:0.14.11",
labels= ["your", "label"],
cpu_limit="1",
cpu_request="0.5",
memory_limit="200Mi",
memory_request="100Mi",
)
``````Kevin Kho
03/21/2022, 4:28 PMMilton
03/21/2022, 4:30 PMKevin Kho
03/21/2022, 4:31 PMMilton
03/21/2022, 4:33 PMAnna Geller
03/21/2022, 4:36 PMMilton
03/21/2022, 4:38 PMprefect diagnostics
?Anna Geller
03/21/2022, 4:39 PMMilton
03/21/2022, 4:43 PMAnna Geller
03/21/2022, 4:49 PMMilton
03/21/2022, 4:55 PMFROM <internal-python-base-image>
ENV PYTHONPATH /home/svc_app
ENV PREFECT__USER_CONFIG_PATH /home/svc_app/.prefect/config.toml
# copy over files needed for init
COPY --chown=svc_app \
entrypoint.sh \
/home/svc_app/
# # pre-install base dependencies
USER root
RUN curl -OL "<https://github.com/krallin/tini/releases/download/v0.19.0/tini_0.19.0-amd64.rpm>"
RUN yum localinstall tini_0.19.0-amd64.rpm -y
RUN rm tini_0.19.0-amd64.rpm
# Environment Setup
USER svc_app
RUN pip install --upgrade pip
WORKDIR /home/svc_app
RUN mkdir .prefect
RUN pip uninstall prefect -y
RUN pip install
'prefect[orchestration_extras]==0.14.11'
ENTRYPOINT ["tini", "-g", "--", "/home/svc_app/entrypoint.sh"]
run_config = KubernetesRun(image="path/to/prefect:0.14.11",
labels= ["your", "label"],
cpu_limit="1",
cpu_request="0.5",
memory_limit="200Mi",
memory_request="100Mi",
)
$ prefect diagnostics
{
"config_overrides": {
"run_config": {
"cpu_limit": true,
"cpu_request": true,
"env": {
...
},
"image": true,
"labels": true,
"memory_limit": true,
"memory_request": true
},
"schedule": {
"cron": true,
"tz": true
},
"storage": {
"access_token_secret": true,
"host": true,
"path": true,
"ref": true,
"repo": true
}
},
"env_vars": [
"PREFECT__STORAGE__REF",
"PREFECT__USER_CONFIG_PATH"
],
"system_information": {
"platform": "Linux-3.10.0-1062.el7.x86_64-x86_64-with-redhat-7.7-Maipo",
"prefect_backend": "cloud",
"prefect_version": "0.14.11",
"python_version": "3.7.3"
}
}
Kevin Kho
03/21/2022, 5:08 PMMilton
03/21/2022, 5:13 PMprefect execute flow-run
? Trying to pinpoint which line of code that does the downloadingKevin Kho
03/21/2022, 5:16 PMMilton
03/21/2022, 5:18 PMprefect execute flow-run
? Trying to pinpoint which line of code that does the downloadingKevin Kho
03/21/2022, 5:18 PMprefect execute flow-run
will already have the Flow downloaded so i’s not there. Anything related to cloning will be found here . I don’t see enough debug level logs thoughMilton
03/21/2022, 5:20 PMprefect execute flow-run
somehowprefect execute flow-run
Kevin Kho
03/21/2022, 5:28 PMstorage.get_flow
called here that then goes to the Gitlab get_flow
. I know these links are Prefect 1.1 but this file hasn’t changed significantly. Can you try running with the base Prefect image?prefect execute flow-run
Milton
03/21/2022, 5:35 PMprefecthq/prefect 0.14.11
Kevin Kho
03/21/2022, 5:39 PMENTRYPOINT
of the image to download the Git repo and pip install -e .
on the image. That will get your up to date dependencies during execution. I am positive you have something like that going on, and it should be possible in 1.0Milton
03/22/2022, 9:14 PMstorage = Git(
repo="org/repo", # name of repo
flow_path="flows/my_flow.py", # location of flow file in repo
repo_host="<http://gitlab.com|gitlab.com>", # repo host name, which may be custom
git_token_secret_name="MY_GIT_ACCESS_TOKEN",# name of Secret containing Deploy Token
git_token_username="myuser" # username associated with the Deploy Token
)
the flow run triggered returns
└── 20:45:45 | INFO | Entered state <Scheduled>: Flow run scheduled.
└── 20:45:53 | INFO | Entered state <Submitted>: Submitted for execution
└── 20:45:53 | INFO | Submitted for execution: Job prefect-job-36cd933b
└── 20:45:57 | INFO | Entered state <Failed>: Failed to load and execute flow run: HangupException(‘ssh: Could not resolve hostname https: Name or service not known\r’)
└── 20:45:57 | ERROR | Failed to load and execute flow run: HangupException(‘ssh: Could not resolve hostname https: Name or service not known\r’)
Flow run failed!
ssh
would be used at all and we have use_ssh set to false by default in Git StorageKevin Kho
03/22/2022, 9:17 PM.sql
or .yaml
files but Python imports won’t work because you’d need some heavy Python path manipulation. It might not even be doable. I’ve seen people try. Not discourage you to try on that. I just really don’t know if it can be done.Milton
03/22/2022, 9:19 PMssh
would be used in the first place?Kevin Kho
03/22/2022, 9:29 PMuse_ssh
was a flag and I thought it was set to False.Milton
03/22/2022, 9:34 PMKevin Kho
03/22/2022, 9:36 PMMilton
03/22/2022, 9:36 PMKevin Kho
03/22/2022, 9:41 PMMilton
03/22/2022, 9:46 PMKevin Kho
03/22/2022, 9:47 PMMilton
03/22/2022, 10:08 PMfrom prefect.utilities.git import TemporaryGitRepo
with TemporaryGitRepo(
git_clone_url=“https😕/<user>:<pass>@self-hosted-gitlab.com/<repo>.git",
branch_name=“master”,
tag=None,
commit=None,
clone_depth=1,
) as temp_repo:
print(temp_repo.temp_dir.name)
Traceback (most recent call last):
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 1091, in fetch_pack
refs, server_capabilities = read_pkt_refs(proto)
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 234, in read_pkt_refs
for pkt in proto.read_pkt_seq():
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/protocol.py”, line 287, in read_pkt_seq
pkt = self.read_pkt_line()
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/protocol.py”, line 232, in read_pkt_line
raise HangupException()
dulwich.errors.HangupException: The remote server unexpectedly closed the connection.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “test.py”, line 8, in <module>
clone_depth=1,
File “/home/<user>/miniconda3/lib/python3.7/site-packages/prefect/utilities/git.py”, line 48, in enter
source=self.git_clone_url, target=self.temp_dir.name, depth=self.clone_depth
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/porcelain.py”, line 451, in clone
depth=depth,
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 535, in clone
result = self.fetch(path, target, depth=depth)
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 607, in fetch
depth=depth,
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 1093, in fetch_pack
raise _remote_error_from_stderr(stderr)
dulwich.errors.HangupException: ssh: Could not resolve hostname https: Name or service not known
Kevin Kho
03/22/2022, 10:09 PMgit_clone_url_secret_name
instead to create the stringMilton
03/22/2022, 10:25 PMgit clone https://<user>:<pass>@self-hosted-gitlab.com/<repo>.git
worksAnna Geller
03/22/2022, 10:35 PMMilton
03/22/2022, 10:37 PMAnna Geller
03/22/2022, 10:39 PMMilton
03/22/2022, 10:41 PMfrom prefect.utilities.git import TemporaryGitRepo
with TemporaryGitRepo(
git_clone_url=“https😕/<user>:<pass>@self-hosted-gitlab.com/<repo>.git",
branch_name=“master”,
tag=None,
commit=None,
clone_depth=1,
) as temp_repo:
print(temp_repo.temp_dir.name)gives
Traceback (most recent call last):
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 1091, in fetch_pack
refs, server_capabilities = read_pkt_refs(proto)
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 234, in read_pkt_refs
for pkt in proto.read_pkt_seq():
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/protocol.py”, line 287, in read_pkt_seq
pkt = self.read_pkt_line()
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/protocol.py”, line 232, in read_pkt_line
raise HangupException()
dulwich.errors.HangupException: The remote server unexpectedly closed the connection.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “test.py”, line 8, in <module>
clone_depth=1,
File “/home/<user>/miniconda3/lib/python3.7/site-packages/prefect/utilities/git.py”, line 48, in enter
source=self.git_clone_url, target=self.temp_dir.name, depth=self.clone_depth
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/porcelain.py”, line 451, in clone
depth=depth,
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 535, in clone
result = self.fetch(path, target, depth=depth)
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 607, in fetch
depth=depth,
File “/home/<user>/miniconda3/lib/python3.7/site-packages/dulwich/client.py”, line 1093, in fetch_pack
raise _remote_error_from_stderr(stderr)
dulwich.errors.HangupException: ssh: Could not resolve hostname https: Name or service not known
Anna Geller
03/22/2022, 10:44 PMwith TemporaryGitRepo(
git_clone_url="https://<user>:<pass>@self-hosted-gitlab.com/<repo>.git",
branch_name="master", # try without that since your git clone command that worked didn't have that
Kevin Kho
03/23/2022, 4:06 AMMilton
03/23/2022, 3:40 PMKevin Kho
03/23/2022, 3:45 PM