John Mizerany
08/03/2022, 7:06 PMBilly McMonagle
08/03/2022, 9:41 PMDmitrii Egunov
08/03/2022, 11:43 PMDarshan
08/04/2022, 12:48 AMMichael W
08/04/2022, 1:22 AMprefect cloud login --key...
it shows the following:Oleg Sheyner
08/04/2022, 2:03 AMKeith
08/04/2022, 6:54 AMTraceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 193, in wrapper
return run_async_in_new_loop(async_fn, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop
return anyio.run(partial(__fn, *args, **kwargs))
File "/usr/local/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
return future.result()
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
return await func(*args)
File "/usr/local/lib/python3.10/site-packages/prefect/cli/agent.py", line 93, in start
async with OrionAgent(
File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 249, in __aenter__
await self.start()
File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 237, in start
await self.default_infrastructure._save(is_anonymous=True)
File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 618, in _save
await self.register_type_and_schema(client=client)
File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 560, in register_type_and_schema
block_type = await client.read_block_type_by_slug(
File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 1093, in read_block_type_by_slug
return BlockType.parse_obj(response.json())
File "/usr/local/lib/python3.10/site-packages/httpx/_models.py", line 743, in json
return jsonlib.loads(self.text, **kwargs)
File "/usr/local/lib/python3.10/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Priyank
08/04/2022, 7:14 AMdocker ps -a
output)
3f832d699b5e prefecthq/server:core-0.15.11
this one occupied nearly 40G
42G 3f832d699b5edd87e39af62f724bb143826f2c5ae929a0565d0ceb479d2befde-json.log
• These logs are taking so much space that our root gets full and our local prefect server goes down, so is there any way we can remove old logs, or divide these logs in dated files and remove old ones as per need, or can we set size limit for these logs file ?
• and also is there any command to get prefect server status and agent status, so that we can send some alerts as per server status and agents' status.jaehoon
08/04/2022, 7:15 AMGaetan Dumortier
08/04/2022, 9:46 AMTarek
08/04/2022, 10:02 AMimage="<http://prefectdemos.azurecr.io/community/flows|prefectdemos.azurecr.io/community/flows>"
)? I understand that the default storage is local and if we don’t set up a storage then our kubernetes agent can fail, but I just want to understand why we need storage in the first place when we have a KubernetesRun
Mohamed Alaa
08/04/2022, 10:07 AMTraceback (most recent call last):
File "/usr/lib/python3.8/asyncio/selector_events.py", line 848, in _read_ready__data_received
data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
From my understanding, this means that there was some connection problems with a TCP/IP connection and it wasn't able to send an acknowledgment message, however, I dont understand why it happens and it seems to me that it happens randomly (probably not true). Does it have to do with the size of data being transmitted (I am extracting data from an S3 bucket for processing)?
Thanks in Advance!Iuliia Volkova
08/04/2022, 10:20 AMflow_data
key, so we deployed flows this way and in our case it was very useful. We need to deploy flow with REST API without using Prefect cli or prefect as a python package. Prefect runs on a totally separate server. Now in 2.0.2 I cannot find how to do this. I read about blocks but I cannot find that block used as default in Prefect to store flow data. So can anyone explain how implement same flow as with flow_data
in version 2.0b8 and early? How can I just submit Flow code with rest api to deploy it on Server? Thanks in Advance!Riccardo Tesselli
08/04/2022, 10:21 AMprefect deployment ls
to get only the deployment’s ids without the fancy drawing of the table? (it makes it difficult to parse the output to use with xargs
).
Then another thing, I’ve noticed that when doing prefect deployment apply
if the deployment already exists in the cloud, it doesn’t get updated. Is there a way to force the re-deployment of an existing one?haris khan
08/04/2022, 10:44 AMname
parameter in the flow definition:
@flow(name='my_unique_name', ...)
warnings.warn(
Found flow 'gen_venue'
Manifest created at '/home/haris/Desktop/snowflake/gen_venue_flow-manifest.json'.
Traceback (most recent call last):
File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
return fn(*args, **kwargs)
File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 193, in wrapper
return run_async_in_new_loop(async_fn, *args, **kwargs)
File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 140, in run_async_in_new_loop
return anyio.run(partial(__fn, *args, **kwargs))
File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/anyio/_core/_eventloop.py", line 56, in run
return asynclib.run(func, *args, **backend_options) # type: ignore
File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 233, in run
return native_run(wrapper(), debug=debug)
File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 228, in wrapper
return await func(*args)
File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/prefect/cli/deployment.py", line 502, in build
file_count = await storage.put_directory()
File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/prefect/filesystems.py", line 428, in put_directory
return await self.filesystem.put_directory(
File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/prefect/filesystems.py", line 250, in put_directory
self.filesystem.put_file(f, fpath)
File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/fsspec/asyn.py", line 86, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/fsspec/asyn.py", line 66, in sync
raise return_result
File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/fsspec/asyn.py", line 26, in _runner
result[0] = await coro
File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/gcsfs/core.py", line 1045, in _put_file
await simple_upload(
File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/gcsfs/core.py", line 1596, in simple_upload
j = await fs._call(
File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/gcsfs/core.py", line 392, in _call
status, headers, info, contents = await self._request(
File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/decorator.py", line 221, in fun
return await caller(func, *(extras + args), **kw)
File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/gcsfs/retry.py", line 115, in retry_request
return await func(*args, **kwargs)
File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/gcsfs/core.py", line 384, in _request
validate_response(status, contents, path, args)
File "/home/haris/Desktop/snowflake/venv/lib/python3.8/site-packages/gcsfs/retry.py", line 100, in validate_response
raise ValueError("Bad Request: %s\n%s" % (path, msg))
ValueError: Bad Request: https://storage.googleapis.com/upload/storage/v1/b/prefect2-snowflake/o
Payload size invalid. Expected size: 2. Actual size: 3.
An exception occurred.Flavio Oliveira
08/04/2022, 10:45 AMFelix Sonntag
08/04/2022, 11:23 AMupdate_flow_run
in engine.py
.
I’m running a flow via the docker-container infrastructure type. It seems to start, but there’s no output on the UI, in fact the flow run status stays in “Pending” state, even after it fails on the work queue.
I also see some TF log outputs which appear when importing TF, but the actual log output of my script is not showing up.
Are there any hints on how to solve this?Tim Helfensdörfer
08/04/2022, 11:59 AM<https://app.prefect.cloud/account/400***/workspace/8d8***/deployment/2f5***>
- it has the staging
tag. We have a running agent:
$ /usr/bin/python3 /usr/local/bin/prefect agent start 360***
Starting agent connected to
<https://api.prefect.cloud/api/accounts/400***/workspaces/8d8***>...
___ ___ ___ ___ ___ ___ _____ _ ___ ___ _ _ _____
| _ \ _ \ __| __| __/ __|_ _| /_\ / __| __| \| |_ _|
| _/ / _|| _|| _| (__ | | / _ \ (_ | _|| .` | | |
|_| |_|_\___|_| |___\___| |_| /_/ \_\___|___|_|\_| |_|
Agent started! Looking for work from queue
'360***'...
The work queue also has the tag staging
. When I click on "Run" in the Cloud UI in the deployment, it is scheduled but the flow run has no tags assigned. And I guest this is the reason why it is never picked up by the agent. Or how can I debug this issue?Steph Clacksman
08/04/2022, 12:39 PMprefect deployment apply deployment.yaml
but I keep getting an error from sqlalchemy: sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) FOREIGN KEY constraint failed
Owen Cook
08/04/2022, 12:48 PMprefect deployment build file.py:flow_name --name some-deployment -sb azure/block-name
I get the error TypeError: put_directory() got an unexpected keyword argument 'ignore_file'.
I then went into prefect/cli/deployment.py and removed the ignore_file arg in line 470 and ran the command again and only file.py
is added to my azure container and everything seems to be stuck in an endless loop. Has anyone had this problem/know what could have gone wrong? Cheers 🙂
TLDR: prefect deployment build
stuck in endless loop and not accepting ignore_file arg.Florian Kühnlenz
08/04/2022, 2:02 PMMatt Delacour
08/04/2022, 2:25 PMTony Yun
08/04/2022, 3:18 PMprefect.tasks.sftp.sftp
I’ve tried 1.20, 2.0 version but all don’t have this module available.Matt Delacour
08/04/2022, 3:38 PMDylan McReynolds
08/04/2022, 3:46 PMChris Reuter
08/04/2022, 3:55 PMGeorvic Tur
08/04/2022, 3:56 PMWilliam Jamir
08/04/2022, 4:37 PMMike Kovetsky
08/04/2022, 4:40 PMprefect kubernetes manifest orion
to k8s_deployment.yaml
(please add the ability to set non-default k8s namespace)
• change PREFECT_API_URL from http://orion:4200/api to http://0.0.0.0:4200/api
• kubectl apply -f k8s_deployment.yaml
(it already has command: [“prefect”, “orion”, “start”, “--host”, “0.0.0.0”, “--log-level”, “WARNING”] for API container)
• check the GCP admin. The agent/ui/api seems to be healthy.
• kubectl port-forward deployment/orion 4200:4200
• check the UI is healthy via http://127.0.0.1:4200/.
• prefect config set PREFECT_API_URL=<http://0.0.0.0:4200/api>
• register GCS block via admin panel. name=prefect. path: gs://{my-path}. (prefect block register -f gcs_block.py didn’t work for me for some reason)
• prefect deployment build ../flows/backtest.py:backtest_flow --name dev --tag dev --infra kubernetes-job --storage-block gcs/prefect
(should i run it from root of the project?)
• checked my GCS bucket. The files did appear in the local folder and GCS.
• prefect deployment apply backtest_flow-deployment.yaml
• check UI http://127.0.0.1:4200/deployments. deployment did appear
• create kubernetes
queue manually. The name seems to be hardcoded according to the error logs in GCP agent. This step fixes the error.
• prefect deployment run Backtest/dev
• the flow run was created. But it is stuck in pending.
• The appropriate KubernetesJob was created. But it is stuck in error state with the error All connection attempts failed. and ConnectionRefusedError: [Errno 111] Connect call failed (‘0.0.0.0’, 4200).
Please help! Thank you in advance :)William Jamir
08/04/2022, 5:09 PMprefect agent start -t test
I can see that a new “Work Queue” is added to the UI (Agent queue test
)
But if I start a new agent (on another tab) with the same command, the UI keeps the same.
Is this the expected behavior? Would it be possible to add a --name
parameter on the start command to allow starting multiples agents to consume the same tag?