Gabriel Santos
02/05/2021, 6:29 AMLaura Vaida
02/05/2021, 12:56 PMworker:
replicas: 3
resources:
limits:
cpu: 1
memory: 3G
requests:
cpu: 1
memory: 3G
env:
- name: EXTRA_CONDA_PACKAGES
value: prefect
jupyter:
enabled: false
Laura Vaida
02/05/2021, 12:58 PMMichael Hadorn
02/05/2021, 1:32 PMwith Flow(
prefect.config.general.flow_name
, executor=LocalDaskExecutor()
, storage=GitLab(
host="<https://gitlab.xxx.ch/>"
, repo="cdwh/cdwh-prefect"
, path="cdwh_flow.py"
, ref="master"
, secrets=["GITLAB_ACCESS_TOKEN"]
)
, run_config=DockerRun(
image="mrjck/cdwh-flow:1.0"
)
,...
Then I successfully registered the flow to the backend. If I let them run, the prefect docker agent deploying the flow but crashed immediately with the following message:
404 Client Error for <http+docker://localhost/v1.40/images/create?tag=1.0&fromImage=mrjck%2Fcdwh-flow>: Not Found ("manifest for mrjck/cdwh-flow:1.0 not found: manifest unknown: manifest unknown")
If I run 'docker images' the used images is there with the correct tag.
How does my Image know, how the storage is pulled?
UPDATE: Was solved the next day. If not, looks like this: https://github.com/PrefectHQ/prefect/issues/2781
Try to start the agent with: --no-docker-interfaceMilly gupta
02/05/2021, 2:04 PMJeremy Phelps
02/05/2021, 6:39 PMNeeraj Vyas
02/05/2021, 6:41 PMjeff n
02/05/2021, 8:37 PMos.getenv()
inside the tasks so that we pulled the secrets in at runtime but this is becoming incredibly unruly. I want to use a config library like dynaconfig so that I can mix settings files with env files. What is the best practice to load up the variables at run time in a smart way? Would the loading be a task that then pushes the config into the Prefect context?Giovanni Giacco
02/05/2021, 9:42 PMVerun Rahimtoola
02/05/2021, 11:00 PMargparse.add_argument(...)
you can pass in something like type=int
or some callablejames.lamb
02/05/2021, 11:17 PMFina Silva-Santisteban
02/05/2021, 11:42 PMDeploying flow run...
but the flow gets stuck in the Submitted
stage. What am I missing?Matt Denno
02/06/2021, 11:28 PMflow.run()
but if I flow.register(name="Task Name")
and then try to run via the UI it fails.
The message I get is:
Failed to load and execute Flow's environment: ImproperlyConfigured('Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.')
I feel like I have done exactly this setup before with Prefect and it has worked fine, but now it is not working. I see it is related to django not being setup, but can't understand why it isn't...I guess because I don't complete understand how Prefect works. A simple example that demonstrates the issue is as follows:
import django
import os
import prefect
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'afs.settings')
django.setup()
from prefect import Flow, task
from api.models import Location # This is a django model import
@task()
def django_task():
locs = Location.objects.all()
logger = prefect.context.get("logger")
<http://logger.info|logger.info>(f"Number of Locations: {len(locs)}")
with Flow(f"Django Task") as flow:
locs = django_task()
# flow.run()
flow.register(project_name="Kerala")
Searching the Slack channel for Django returned one result which indicated the maybe using a resource manager was the way to set this up, but I wasn't able to get it to work using resource manager either.
The other thing I tried was to set DJANGO_SETTINGS_MODULE via a LocalRun:
flow.run_config = LocalRun(env={'DJANGO_SETTINGS_MODULE': 'afs.settings'})
flow.register(project_name="Kerala")
But that results in another error about INSTALLED_APPS not being configured...clearly I am missing something.
Last thing, I also added the following to a a separate non django task, and it looks like it is using the correct virtual env to execute the task and has the correct paths.
<http://logger.info|logger.info>(sys.path)
<http://logger.info|logger.info>(sys.version)
<http://logger.info|logger.info>(sys.executable)
Really stumped...
Any help would be greatly appreciated.Dimosthenis Schizas
02/07/2021, 12:19 AMauction_date = Parameter(
"auction_date", default=prefect.context.date.date()
)
I get the following error message when trying to register:
AttributeError: 'Context' object has no attribute 'date'
I am missing something hereNathan Atkins
02/07/2021, 12:36 AMprefect
commands against my local server. I guess it could be a installation or configuration problem, but debugging makes it look like there is an issue with initializing Client objects to talk to the local server.
Specifically:
prefect create project my_project
and flow.register(project_name='tutorial')
Both generate prefect.utilities.exceptions.ClientError: Malformed response received from API.
errors.
If I debug both problems I wind up in prefect.client.Client.___init___()
where it looks like it sets the api_server to the cloud host because no api_server
is passed in. I have the backend set to to server
.
In prefect.cli.create.project()
Client().create_project(project_name=name, project_description=description)
it looks like the api_server=localhost
needs to be passed to Client.___init___()
or Client needs to do something other than
self.api_server = api_server or prefect.context.config.cloud.get("graphql")
Same issue in prefect.core.flow.py
I can connect to the server fine from my local browser or through cloud.prefect.io switched to the local server. (Nice feature by the way.)Dolor Oculus
02/07/2021, 1:42 AMRob Fowler
02/08/2021, 11:24 AMRob Fowler
02/08/2021, 12:02 PMexecutor = LocalDaskExecutor(scheduler="processes", num_workers=10)
for ii in flow.run(executor=executor).result[result].result:
print(ii)
Rob Fowler
02/08/2021, 12:02 PMJames Phoenix
02/08/2021, 12:52 PMJames Phoenix
02/08/2021, 12:52 PMJames Phoenix
02/08/2021, 12:53 PMJames Phoenix
02/08/2021, 12:53 PMimport prefect
client = prefect.Client()
James Phoenix
02/08/2021, 1:22 PMJames Phoenix
02/08/2021, 1:22 PMclient.create_flow_run(flow_id="<flow id>")
James Phoenix
02/08/2021, 1:22 PMrafaqat ali
02/08/2021, 1:29 PMAvinash Varma Kanumuri
02/08/2021, 2:32 PMSamuel Hinton
02/08/2021, 4:37 PMPedro Martins
02/08/2021, 6:06 PMconfig.toml
. See below:
with Flow("model-deployment-pipeline", **custom_confs,) as flow:
model_uri = Parameter("model_uri", default=config["model"]["modelUri"])
environment = Parameter("environment", default=config["model"]["environment"])
deploy_model(model_uri=model_uri, namespace=environment)
This flow is triggered when there are modification on the config.toml. How can I make sure that this flow will access the most up to date config file?