syslog from on-prem to Stackdriver in GCP
If you’ve used Google Cloud for a while, I’m sure you’ll agree that one of the best features is the Stackdriver suite, especially for log aggregation & analysis.
On-Prem workloads can feel left behind and we’ve been wishing for a while to bring them all into the same place, disappointed that Google offered an official solution but one that involved using another provider (and them processing the data in their own platform before it hits Stackdriver).
Stackdriver’s existing logging agent is already based on the flexible “fluentd” and the plugin to export to the Logs API is already fairly great; the main stumbling block to ingesting logs from network inputs is having it tagged against a resource in the Log Viewer UI that corresponds to the source.
You can only use existing root resources, however there are a few that seem to lend themselves to external workloads, like “Generic Node” and “Generic Task”.
Our solution has been to modify the Stackdriver export plugin to allow specifying a custom resource location per log entry:
And the repository also includes an example Dockerfile that can be used to run the agent in GKE, which when combined with an internal network already connected to your VPC and an internal load balancer, will allow you to ingest logs easily:
- https://github.com/a1comms/fluent-plugin-google-cloud/blob/master/Dockerfile
- https://github.com/a1comms/fluent-plugin-google-cloud/blob/master/netsyslog.conf
It is based on the Stackdriver agent’s repo on GitHub, whose docker image seems to be built as “gcr.io/stackdriver-agents/stackdriver-logging-agent”.
Logs will show up under “Generic Node” and drill down tags first based on the “NETWORK_NAME” environment variable set in the logging agent container, then the hostname / IP the entry was logged from.
It is worth noting that in our testing, Cisco switches & pfSense (FreeBSD) systems don’t seem to be parsed properly by “fluentd”, either as rfc5424 or rfc3164, so we resorted to ingesting those as raw UDP and losing some metadata.
If you already have structured logging locally and syslog would be a step backwards, you should also be able to forward entries from you local “fluentd” instances using it’s own input plugin very easily.