https://prefect.io
Join Slack
Is anyone else logging the worker output? I'm trying to get everything it prints to the console into...
# prefect-getting-started
j
Is anyone else logging the worker output? I'm trying to get everything it prints to the console into a file, but the task logger doesn't seem to work. For troubleshooting I just have the flow_runs and task_runs set, and only the flow gets logged. Even when I put the file handler on the root logger I still don't get the flow and task details. Is this because it's running inside a container? Screenshots: • console output (partial) • log file • logging.yml
oh maybe it's the infrastructure logger? I'll try that didn't work
I guess my only option is 
prefect --no-prompt worker start --pool pool_name > ~/.prefect/prefect_log.log
The really confusing thing about this is that the messages I want show up in the cloud interface, so one would assume that attaching more handlers to those loggers that go to cloud with the 
api
handler would also go to the file. I just want to mirror the cloud logs to a file. There must be other logging going on that this logging.yml does not configure.
In case anyone else runs into this, the solution I came up with was to just log the docker container output directly to CloudWatch. You have to create 
/etc/docker/daemon.json
first, have your IAM stuff setup for CloudWatch (AWS has lots of confusing docs on this), then set up a log group and stream. Here's what my daemon.json looks like:
Copy code
{
  "log-driver": "awslogs",
  "log-opts": {
    "awslogs-region": "us-east-1",
    "awslogs-group": "Prefect-Logs",
    "awslogs-stream": "EC2-Prefect-Docker"
  }
}
But I still can't get all the console output of the worker to go to a file by using the logging.yml file, so I don't know how to get that into CloudWatch unless you dump the output of the 
worker start
command to a file and have the CloudWatch agent pick it up.
15 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by