Adam Brusselback
01/28/2021, 11:43 PMJan Marais
01/29/2021, 6:05 AMDJ Erraballi
01/29/2021, 6:42 AMTim Pörtner
01/29/2021, 9:33 AMBorut Hafner
01/29/2021, 1:29 PMAlex Furrier
01/29/2021, 3:57 PM@task
def get_array_input(db_client, query):
return db_client.query(query)
@task
def compute_array_write_to_db(input):
array = requests.get_huge_array()
db_client.write(array)
with Flow("generate-and-write-arrays",) as generate_and_write_arrays:
lots_of_inputs = get_array_input(db_client, query)
lots_of_inputs.map(compute_array_write_to_db)
# Crashes after certain amount of arrays
# due to memory limit
generate_and_write_arrays.run()
jeff n
01/29/2021, 4:33 PMwith Flow("Example") as flow:
....
def __main__():
if os.getenv("production") == True:
flow.schedule(my_scheduler)
flow.register()
If I register a flow with that in the main would I be able to control if it has a schedule?Levi Leal
01/29/2021, 4:45 PMJosh
01/29/2021, 5:06 PMFailed to write log with error: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: /graphql (Caused by ProtocolError('Connection aborted.', BrokenPipeError(32, 'Broken pipe')))
In Prefect it says the task failed, but it still seems to be executing in the agent. There were logs before from the agent to the cloud, but no logs afterwardsKieran
01/29/2021, 5:24 PMPrefect automatically gathers mapped results into a list if they are needed by a non-mapped task
I have two mapped tasks, one feeding the other, then a third non-mapped task which expects a list. The logger context doesn't appear to be getting passed so I cant get any insight. The UI is showing the child-mapped tasks and I have logged out their content but their results are not being gathered together as suggested in the docs.
Any pointers to help would be amazing!Raphaël Riel
01/29/2021, 7:25 PM<http://univeral.prefet.io|univeral.prefet.io>
D) Click on Google Again.
E) Now I have Google’s prompt to select which account to use (Personal, Work, etc.)
F) Then I’m logged in.
Any idea? Can this be related to https://prefect-community.slack.com/archives/CL09KU1K7/p1611786977326500 ?Josh
01/30/2021, 3:32 AMFailed to write log with error: 413 Client Error: Request Entity Too Large for url: <https://api.prefect.io/graphql>
Peter Roelants
01/30/2021, 8:36 AM"test-volume"
created with docker volume create test-volume
If I want to mount the named docker volume "test-volume"
with prefect agent docker start --volume test-volume:$DOCKER_DIR ...
, the named volume is never mounted into the runtime started by the agent, a new volume seems to be created and mounted each time.jspeis
01/31/2021, 1:43 AMRyan Kelly
01/31/2021, 3:29 PMAnatoliy Zhyzhkevych
01/31/2021, 9:51 PMPeyton Runyan
01/31/2021, 10:58 PMmode
as a parameter instead of messing with it in the config and re-registering my flow.
mode = "hard_coded_mode"
[sql_server]
server = server
driver = driver
dsn = "MYMSSQL"
user = "${sql_server.${mode}.user}"
database = "${sql_server.${mode}.database}"
[sql_server.dev]
user = dev user
database = "dev db
[sql_server.hub]
user = app user
database = app db
[sql_server.prod]
user = prod user
database = prod db
Yash Bhandari
02/01/2021, 1:34 AMMichael Hadorn
02/01/2021, 8:25 AMAdam
02/01/2021, 12:15 PMGiovanni Giacco
02/01/2021, 1:06 PMAdrien Boutreau
02/01/2021, 3:17 PMJoël Luijmes
02/01/2021, 3:27 PM@pytest.fixture
def mock_resource_manager(monkeypatch):
mock = MagicMock(return_value=None)
monkeypatch.setattr("prefect.tasks.core.resource_manager.ResourceManager.__call__", mock)
return mock
But this gives weird errors down the road, as I also want to mock prefect.config/context for other purposes.Josh Greenhalgh
02/01/2021, 4:16 PMKilian
02/01/2021, 4:43 PMMorgan Omind
02/01/2021, 4:58 PMPedro Martins
02/01/2021, 6:27 PMMatthew Blau
02/01/2021, 7:40 PMException raised while calling state handlers: ClientError('400 Client Error: Bad Request for url: <http://host.docker.internal:4200/graphql>\n\nThe following error messages were provided by the GraphQL server:\n\n GRAPHQL_VALIDATION_FAILED: Cannot query field "secret_value" on type "Query".\n\nThe GraphQL query was:\n\n query($name: String!) {\n secret_value(name: $name)\n }\n\nThe passed variables were:\n\n {"name": "SLACK_WEBHOOK_URL"}\n')
what am I doing wrong? I have @task(state_handlers=[slack_notifier]) set for the task I am testing out the notification for.Arun Giridharan
02/01/2021, 8:44 PMBK Lau
02/01/2021, 9:34 PMApache Beam
cluster using Prefect. If so, like to get some input and lessons from your experience.