03bfe08e1d
This moves the various projects from "type-based" folders (such as "services" or "tools") into more appropriate semantic folders (such as "nix", "ops" or "web"). Deprecated projects (nixcon-demo & gotest) which only existed for testing/demonstration purposes have been removed. (Note: *all* builds are broken with this commit)
152 lines
5.7 KiB
Markdown
152 lines
5.7 KiB
Markdown
journaldriver
|
|
=============
|
|
|
|
This is a small daemon used to forward logs from `journald` (systemd's
|
|
logging service) to [Stackdriver Logging][].
|
|
|
|
Many existing log services are written in inefficient dynamic
|
|
languages with error-prone "cover every possible use-case"
|
|
configuration. `journaldriver` instead aims to fit a specific use-case
|
|
very well, instead of covering every possible logging setup.
|
|
|
|
`journaldriver` can be run on GCP-instances with no additional
|
|
configuration as authentication tokens are retrieved from the
|
|
[metadata server][].
|
|
|
|
<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
|
|
**Table of Contents**
|
|
|
|
- [Features](#features)
|
|
- [Usage on Google Cloud Platform](#usage-on-google-cloud-platform)
|
|
- [Usage outside of Google Cloud Platform](#usage-outside-of-google-cloud-platform)
|
|
- [Log levels / severities / priorities](#log-levels--severities--priorities)
|
|
- [NixOS module](#nixos-module)
|
|
- [Stackdriver Error Reporting](#stackdriver-error-reporting)
|
|
|
|
<!-- markdown-toc end -->
|
|
|
|
# Features
|
|
|
|
* `journaldriver` persists the last forwarded position in the journal
|
|
and will resume forwarding at the same position after a restart
|
|
* `journaldriver` will recognise log entries in JSON format and
|
|
forward them appropriately to make structured log entries available
|
|
in Stackdriver
|
|
* `journaldriver` can be used outside of GCP by configuring static
|
|
credentials
|
|
* `journaldriver` will recognise journald's log priority levels and
|
|
convert them into equivalent Stackdriver log severity levels
|
|
|
|
# Usage on Google Cloud Platform
|
|
|
|
`journaldriver` does not require any configuration when running on GCP
|
|
instances.
|
|
|
|
1. Install `journaldriver` on the instance from which you wish to
|
|
forward logs.
|
|
|
|
2. Ensure that the instance has the appropriate permissions to write
|
|
to Stackdriver. Google continously changes how IAM is implemented
|
|
on GCP, so you will have to refer to [Google's documentation][].
|
|
|
|
By default instances have the required permissions if Stackdriver
|
|
Logging support is enabled in the project.
|
|
|
|
3. Start `journaldriver`, for example via `systemd`.
|
|
|
|
# Usage outside of Google Cloud Platform
|
|
|
|
When running outside of GCP, the following extra steps need to be
|
|
performed:
|
|
|
|
1. Create a Google Cloud Platform service account with the "Log
|
|
Writer" role and download its private key in JSON-format.
|
|
2. When starting `journaldriver`, configure the following environment
|
|
variables:
|
|
|
|
* `GOOGLE_CLOUD_PROJECT`: Name of the GCP project to which logs
|
|
should be written.
|
|
* `GOOGLE_APPLICATION_CREDENTIALS`: Filesystem path to the
|
|
JSON-file containing the service account's private key.
|
|
* `LOG_STREAM`: Name of the target log stream in Stackdriver Logging.
|
|
This will be automatically created if it does not yet exist.
|
|
* `LOG_NAME`: Name of the target log to write to. This defaults to
|
|
`journaldriver` if unset, but it is recommended to - for
|
|
example - set it to the machine hostname.
|
|
|
|
# Log levels / severities / priorities
|
|
|
|
`journaldriver` recognises [journald's priorities][] and converts them
|
|
into [equivalent severities][] in Stackdriver. Both sets of values
|
|
correspond to standard `syslog` priorities.
|
|
|
|
The easiest way to emit log messages with priorites from an
|
|
application is to use [priority prefixes][], which are compatible with
|
|
structured log messages.
|
|
|
|
For example, to emit a simple warning message (structured and
|
|
unstructured):
|
|
|
|
```
|
|
$ echo '<4>{"fnord":true, "msg":"structured log (warning)"}' | systemd-cat
|
|
$ echo '<4>unstructured log (warning)' | systemd-cat
|
|
```
|
|
|
|
# NixOS module
|
|
|
|
The NixOS package repository [contains a module][] for setting up
|
|
`journaldriver` on NixOS machines. NixOS by default uses `systemd` for
|
|
service management and `journald` for logging, which means that log
|
|
output from most services will be captured automatically.
|
|
|
|
On a GCP instance the only required option is this:
|
|
|
|
```nix
|
|
services.journaldriver.enable = true;
|
|
```
|
|
|
|
When running outside of GCP, the configuration looks as follows:
|
|
|
|
```nix
|
|
services.journaldriver = {
|
|
enable = true;
|
|
logStream = "prod-environment";
|
|
logName = "hostname";
|
|
googleCloudProject = "gcp-project-name";
|
|
applicationCredentials = keyFile;
|
|
};
|
|
```
|
|
|
|
**Note**: The `journaldriver`-module is included in stable releases of
|
|
NixOS since NixOS 18.09.
|
|
|
|
# Stackdriver Error Reporting
|
|
|
|
The [Stackdriver Error Reporting][] service of Google's monitoring
|
|
toolbox supports automatically detecting and correlating errors from
|
|
log entries.
|
|
|
|
To use this functionality log messages must be logged in the expected
|
|
[log format][].
|
|
|
|
*Note*: Reporting errors from non-GCP instances requires that the
|
|
`LOG_STREAM` environment variable is set to the special value
|
|
`global`.
|
|
|
|
This value changes the monitored resource descriptor from a log stream
|
|
to the project-global stream. Due to a limitation in Stackdriver Error
|
|
Reporting, this is the only way to correctly ingest errors from
|
|
non-GCP machines. Please see [issue #4][] for more information about
|
|
this.
|
|
|
|
[Stackdriver Logging]: https://cloud.google.com/logging/
|
|
[metadata server]: https://cloud.google.com/compute/docs/storing-retrieving-metadata
|
|
[Google's documentation]: https://cloud.google.com/logging/docs/access-control
|
|
[NixOS]: https://nixos.org/
|
|
[contains a module]: https://github.com/NixOS/nixpkgs/pull/42134
|
|
[journald's priorities]: http://0pointer.de/public/systemd-man/sd-daemon.html
|
|
[equivalent severities]: https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#logseverity
|
|
[priority prefixes]: http://0pointer.de/public/systemd-man/sd-daemon.html
|
|
[Stackdriver Error Reporting]: https://cloud.google.com/error-reporting/
|
|
[log format]: https://cloud.google.com/error-reporting/docs/formatting-error-messages
|
|
[issue #4]: https://github.com/tazjin/journaldriver/issues/4
|