Roger Webb
08/09/2022, 1:22 PMPedro Machado
08/09/2022, 2:01 PMRoger Webb
08/09/2022, 2:15 PMFlow_A = create_flow_run(
flow_name="Flow A",
project_name="Project A",
task_args=dict(name="Flow A (Execution)"),
scheduled_start_time=pendulum.now().add(minutes=60),
parameters={"Parameter1":"Flow A Parameter 1"}
)
Flow_A_Flag = wait_for_flow_run(
Flow_A,
raise_final_state=True,
stream_logs=True
)
I would expect the parent flow to kick off the flow run.. and then the wait for to wait for an hour before that wait for would succeed. But, it appears that the Flow A actually executes immediately.. not waiting the hour, so the wait_for succeeds after minutes. Is my misunderstanding in the scheduled_Start_time.. or the wait_for?Jessica Smith
08/09/2022, 2:48 PMclone_url = "XXXXXXX"
with TemporaryGitRepo(clone_url) as temp_repo:
flow = extract_flow_from_file(
file_path=os.path.join(
temp_repo.temp_dir.name,
"path/to/flow/fil.py",
),
flow_name="flow_name",
)
Patrick Tan
08/09/2022, 3:01 PMdatamongus
08/09/2022, 3:26 PMSam Garvis
08/09/2022, 3:28 PM$ prefect profile use development_service_account
â ‹ Connecting...
Error authenticating with Prefect Cloud using profile 'development_service_account'
No matter what I do I cannot connect to our Prefect 2.0 Cloud environment from my terminal.
When I try to run a flow locally I get
RuntimeError: Cannot create flow run. Failed to reach API at <https://api.prefect.cloud/api/accounts> ...
My account_id and workspace_id are correct, I have checked.
I don't understand why I cannot connectTarek
08/09/2022, 4:02 PMTony Yun
08/09/2022, 4:08 PM.prefect/config.toml
file to runtime environment? For example, I’m writing pytest
in VScode, when I run the test, it always fail for not finding the secrets that specified in that config file. I can only run in CLI python flow.py
fine.Oscar Björhn
08/09/2022, 4:29 PMOscar Björhn
08/09/2022, 4:49 PMMars
08/09/2022, 6:42 PMPREFECT_API_URL
?) Are there additional agent settings in 2.0 I should be aware of for running a production k8s deployment, like the work queue name? If so, do they have envvars, and where in the docs can I read about them?Patrick Tan
08/09/2022, 7:42 PMMatt Delacour
08/09/2022, 7:55 PMSimon Macklin
08/09/2022, 8:16 PMRoss Teach
08/09/2022, 8:26 PMMatt Delacour
08/09/2022, 8:42 PMJohn Archer
08/09/2022, 11:35 PMmanifest.json
file that is created from the deployment build
cli command. I see you can set the --output
for the deployment.yaml
file. I have a number of flows in a single repo and would like to keep the structure clean, I know I can run the command from within the directory that I have the flow in but am looking to run the commands from the project root. This is also so I can simplify CI/CD jobs in the future. Any help would be appreciated.Thuy Tran
08/10/2022, 4:46 AM00:35:13.071 | DEBUG | Flow run 'logical-lemming' - Resolving inputs to 'update-record'
00:35:14.896 | DEBUG | prefect.client - Connecting to API at <http://127.0.0.1:4200/api/>
00:35:17.563 | DEBUG | prefect.agent - Checking for flow runs...
00:35:22.591 | DEBUG | prefect.agent - Checking for flow runs...
--- Orion logging error ---
The log worker encountered a fatal error.
Traceback (most recent call last):
File "/Applications/anaconda3/envs/datapipeline/lib/python3.9/site-packages/prefect/logging/handlers.py", line 82, in _send_logs_loop
anyio.run(self.send_logs)
File "/Applications/anaconda3/envs/datapipeline/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/Applications/anaconda3/envs/datapipeline/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/Applications/anaconda3/envs/datapipeline/lib/python3.9/asyncio/runners.py", line 47, in run
_cancel_all_tasks(loop)
File "/Applications/anaconda3/envs/datapipeline/lib/python3.9/asyncio/runners.py", line 56, in _cancel_all_tasks
to_cancel = tasks.all_tasks(loop)
File "/Applications/anaconda3/envs/datapipeline/lib/python3.9/asyncio/tasks.py", line 53, in all_tasks
tasks = list(_all_tasks)
File "/Applications/anaconda3/envs/datapipeline/lib/python3.9/_weakrefset.py", line 65, in __iter__
for itemref in self.data:
RuntimeError: Set changed size during iteration
Worker information:
Approximate queue length: 0
Pending log batch length: 0
Pending log batch size: 0
00:35:27.618 | DEBUG | prefect.agent - Checking for flow runs...
00:35:32.655 | DEBUG | prefect.agent - Checking for flow runs...
00:35:37.680 | DEBUG | prefect.agent - Checking for flow runs...
00:35:42.713 | DEBUG | prefect.agent - Checking for flow runs...
00:35:46.427 | DEBUG | prefect.flows - Parameter 'target_collection' for flow 'update-record' is of unserializable type 'Collection' and will not be stored in the backend.
00:35:46.427 | DEBUG | prefect.flows - Parameter 'sys_collection' for flow 'update-record' is of unserializable type 'Collection' and will not be stored in the backend.
00:35:46.427 | DEBUG | prefect.flows - Parameter 'metadata_collection' for flow 'update-record' is of unserializable type 'Collection' and will not be stored in the backend.
00:35:47.751 | DEBUG | prefect.agent - Checking for flow runs...
00:35:48.177 | INFO | Flow run 'logical-lemming' - Created subflow run 'knowing-avocet' for flow 'update-record'
00:35:48.395 | ERROR | Flow run 'knowing-avocet' - Received invalid parameters
Traceback (most recent call last):
File "/Applications/anaconda3/envs/datapipeline/lib/python3.9/site-packages/prefect/engine.py", line 433, in create_and_begin_subflow_run
parameters = flow.validate_parameters(parameters)
File "/Applications/anaconda3/envs/datapipeline/lib/python3.9/site-packages/prefect/flows.py", line 275, in validate_parameters
raise validation_err
prefect.exceptions.ParameterTypeError: 1 validation error for UpdateRecord
end_date
str type expected (type=type_error.str)
00:35:48.435 | INFO | Flow run 'logical-lemming' - Created task run 'update_record_in_db-a12634ec-0' for task 'update_record_in_db'
00:35:48.435 | INFO | Flow run 'logical-
Vadym Dytyniak
08/10/2022, 8:15 AMWARNING - agent | Job 'prefect-job-623be8f6' is for flow run '958c388f-f155-4ee8-a279-40c546b99808' which does not exist. It will be ignored.
Benjamin.bgx
08/10/2022, 9:16 AMblock = Azure(azure_storage_connection_string="paste_the_string_here")
block.save("dev")
and with that commande line I finalize the storage information for the flow :
prefect deployment build flow.py:flowname \
--name deploy_name --tag dev -sb azure/dev
Can I do the same things from the UI ? In this case, I just need to use the command line and reference the block define in the UI.
Am I right with all this assumptions ?
And so, how I specify the storage for the persistent logs ?
Thank you ! 🙂Vincent Chéry
08/10/2022, 11:36 AMSudharshan B
08/10/2022, 12:11 PMChristian Vogel
08/10/2022, 12:18 PM(begin_task_run pid=81126) from prefect.packaging.docker import DockerPackager
(begin_task_run pid=81126) ImportError: cannot import name 'DockerPackager' from partially initialized module 'prefect.packaging.docker' (most likely due to a circular import) (/home/christian/Documents/ray_and_prefect/temp/temp-venv/lib/python3.9/site-packages/prefect/packaging/docker.py)
Paul Lucas
08/10/2022, 12:35 PMtrigger=any_failed
which works fine when one of those does fail. But when those upstream tasks are all successful, I seem to get a TriggerFailed
on that task and therefore the flow run is FAILED
. Is that expected behaviour and if so, how do I instead get it to just skip that task instead of marking it as Failed and therefore the whole flow as failed? Thanks in advanceKelvin
08/10/2022, 12:37 PMLucien Fregosi
08/10/2022, 12:54 PM--schedule
option (like we do for storage block for instance) ?
In my automated process I can’t edit the deployment.yaml file as I re-build this file every time in case of something has changedVlad Tudor
08/10/2022, 12:59 PMdocker-compose.yaml
from running prefect server config
(posted below). I should now add a new service at the end that runs a Local Agent and a service that registers my flow at startup - is that correct?
For now if I add a service that runs prefect agent local start
I get ValueError: You have not set an API key for authentication.
when running docker-compose up
.Matt Delacour
08/10/2022, 1:13 PMNikhil Joseph
08/10/2022, 1:43 PMNikhil Joseph
08/10/2022, 1:43 PMTaylor Curran
08/10/2022, 1:47 PMNikhil Joseph
08/10/2022, 1:49 PM