Adam Brusselback
05/05/2021, 4:27 PMcould not translate host name "<Task: get_value>" to address: Unknown host
Adam Brusselback
05/05/2021, 4:29 PMwith Flow("test") as flow:
client_name = Parameter('client_name', default='client_demo')
client_context = get_client_context("C:/.secrets/client/", client_name)
db_exec = PostgresExecute(
host=get_value(client_context, 'oltp_host'), db_name=get_value(client_context, 'oltp_database'), user=get_value(client_context, 'oltp_user'), query="SELECT do_something()"
)
result = db_exec(password=get_value(client_context, 'oltp_pass'))
bastianwegge
05/05/2021, 5:53 PMPREFECT__CLOUD__AUTH_TOKEN
is a bit too hidden when you’re approaching from this side. I had to dig into the Storage Registration Code (Prefect library internals) to find that the Client takes this variable (if it is specified) and uses it for registration purposes on prefect Cloud. Just a small thing to improve in the API-docs I guess that might make it a lot easier to get up and running.Todd Lindstrom
05/05/2021, 5:55 PMNathan Walker
05/05/2021, 6:43 PMJason Prado
05/05/2021, 6:58 PMSchedule
try and run, and if it fails expire the job after 60 seconds of trying?Robin
05/05/2021, 8:01 PMresource manager
related question:
I am trying to setup a resource manager to do some larger-than-memory computations within a task.
I wanted to start simple, but even the example gives the following error for me:
multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Adam Brusselback
05/05/2021, 8:43 PMJonathan Chu
05/06/2021, 12:17 AMdaffy-wolf
) already look very docker-name like
It seems like it'd be pretty slick if the DockerAgent
then also used the same name for launching containerized flows.
Not strictly needed, and maybe only relevant for our setup where our log aggregator shows the container name that the logs came fromSamuel Hinton
05/06/2021, 9:33 AMWe could not delete your flow. Please try again. If this problem continues, contact <mailto:help@prefect.io|help@prefect.io>
Samuel Hinton
05/06/2021, 10:06 AMUncaught (in promise) AuthSdkError: PKCE requires a modern browser with encryption support running in a secure context.
The current page is not being served with HTTPS protocol. PKCE requires secure HTTPS protocol.
Maybe this is causing issues for the dashboard (even though I can see a flows runs without issue if I navigate to the url).
Currently the IP address of the server is only accessible within a VPN so we have been using http, and it used to be fine. Is there a way of saying “Dont care about this?” or a way of attaching a certificate to a given service.
Very similar to this old bug here: https://github.com/PrefectHQ/ui/issues/588
Did some env vars not propagate correctly again? Confused as to why this used to work, but a redeploy from the prefect images no longer does soJeremy Phelps
05/06/2021, 12:50 PMSean Perry
05/06/2021, 3:23 PMTypeError: ‘ContextualVersionConflict’ object is not iterableI have a flow that I am trying to call
run()
on to run it locally.Sean Perry
05/06/2021, 3:52 PMboto3
dask-cloudprovider[aws] == 2021.3.1
prefect[aws] == 0.14.17
Trying to run pip-compile on it fails with a complaint about botocore versioning:
botocore<1.20.50,>=1.20.49 (from aiobotocore==1.3.0->dask-cloudprovider[aws]==2021.3.1->-r requirements/production.in (line 14))
botocore<1.21.0,>=1.20.67 (from boto3==1.17.67->-r requirements/production.in (line 9))
botocore<2.0a.0,>=1.12.36 (from s3transfer==0.4.2->boto3==1.17.67->-r requirements/production.in (line 9))
This boils down to: botocore ==1.20.49,>=1.20.67 which cannot be satisfied. What is a safe version of boto3 that works with prefect and dask cloud provider? Removing the version numbers above from the requirements.in does not help, same complaints. Which is to be expected since pip is going to try the newest it can without guidance.Zach Schumacher
05/06/2021, 4:16 PMTodd Lindstrom
05/06/2021, 4:18 PMunexpected error: NameError("name 'prefect' is not defined
This appears to me that it is not running in the condra activation and yes if thats true then the pip install will not be there.
Please advise how I can set the python environment of my local agent.ciaran
05/06/2021, 5:59 PMcluster_kwargs
or client_kwargs
on https://docs.prefect.io/api/latest/executors.html#daskexecutor ?Sean Perry
05/06/2021, 6:02 PMFilterTask
variant that had an extra parameter along the lines of:
log_filtered (Callable, optional): a function to use for logging any result that did not pass `filter_func`.
Use would be something like:
valid_configurations = FilterTask(filter_func=some_validator, log_filtered=log_invalid_configuration)
where
def log_invalid_configuration(configuration):
# log the reasons this configuration is invalid. Or some other special logging for invalid configurations.
# Like, notifying a team to fix the configuration.
Adding this to FilterTask
would be straightforward. Turn the list comprehension info a for loop appending to filtered
and call log_filtered
whenever a result does not match the predicate.
I am currently making 2 calls to FilterTask, once to collect valid and once to collect invalid. Which is not pretty.Lukas N.
05/06/2021, 7:16 PMAusten Bouza
05/06/2021, 7:36 PMGeorge Tam
05/06/2021, 11:10 PMTrevor Kramer
05/07/2021, 12:39 AMdipsy wong
05/07/2021, 8:12 AMvaibhav satpathy
05/07/2021, 12:00 PMHoward Cornwell
05/07/2021, 2:09 PMhigh-memory
runs on a high-memory
agent which spawns jobs via a high-memory
nodeselector.Zach Schumacher
05/07/2021, 3:31 PMBrian Mesick
05/07/2021, 5:30 PMSaksham Dixit
05/07/2021, 6:33 PMDamien Ramunno-Johnson
05/07/2021, 8:30 PMIn [1]: import prefect
...: from prefect import task, Flow
...:
...: @task
...: def say_hello():
...: logger = prefect.context.get("logger")
...: <http://logger.info|logger.info>("Hello, Cloud!")
...:
...: with Flow("hello-flow") as flow:
...: say_hello()
...:
...: # Register the flow under the "tutorial" project
...: flow.register(project_name="tutorial")
but get
ClientError: [{'path': ['user'], 'message': 'field "user" not found in type: \'query_root\'', 'extensions': {'path': '$.selectionSet.user', 'code': 'validation-failed', 'exception': {'message': 'field "user" not found in type: \'query_root\''}}}]
Fina Silva-Santisteban
05/07/2021, 10:34 PMImportError: Using `prefect.tasks.jupyter` requires Prefect to be installed with the "jupyter" extra.
and when I try installing it by running pip3 install prefect[jupyter]
it returns
no matches found: prefect[jupyter]
My current workaround is to use Prefect API and to add that as dependency during flow registration
python_dependencies = ["prefect[jupyter]"]
flow.storage = Docker(...
python_dependencies=python_dependencies,
...)
Is there a way to pip install the jupyter things needed to run a jupyter task using Prefect Core only?Fina Silva-Santisteban
05/07/2021, 10:34 PMImportError: Using `prefect.tasks.jupyter` requires Prefect to be installed with the "jupyter" extra.
and when I try installing it by running pip3 install prefect[jupyter]
it returns
no matches found: prefect[jupyter]
My current workaround is to use Prefect API and to add that as dependency during flow registration
python_dependencies = ["prefect[jupyter]"]
flow.storage = Docker(...
python_dependencies=python_dependencies,
...)
Is there a way to pip install the jupyter things needed to run a jupyter task using Prefect Core only?pip3 install "prefect[jupyter]"
, and that works fine!