chore: Significantly restructure folder layout
This moves the various projects from "type-based" folders (such as "services" or "tools") into more appropriate semantic folders (such as "nix", "ops" or "web"). Deprecated projects (nixcon-demo & gotest) which only existed for testing/demonstration purposes have been removed. (Note: *all* builds are broken with this commit)
This commit is contained in:
Vincent Ambo
parent
e52eed3cd4
commit
03bfe08e1d
110 changed files with 1 additions and 998 deletions
152
ops/journaldriver/README.md
Normal file
152
ops/journaldriver/README.md
Normal file
|
|
@ -0,0 +1,152 @@
|
|||
journaldriver
|
||||
=============
|
||||
|
||||
This is a small daemon used to forward logs from `journald` (systemd's
|
||||
logging service) to [Stackdriver Logging][].
|
||||
|
||||
Many existing log services are written in inefficient dynamic
|
||||
languages with error-prone "cover every possible use-case"
|
||||
configuration. `journaldriver` instead aims to fit a specific use-case
|
||||
very well, instead of covering every possible logging setup.
|
||||
|
||||
`journaldriver` can be run on GCP-instances with no additional
|
||||
configuration as authentication tokens are retrieved from the
|
||||
[metadata server][].
|
||||
|
||||
<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
|
||||
**Table of Contents**
|
||||
|
||||
- [Features](#features)
|
||||
- [Usage on Google Cloud Platform](#usage-on-google-cloud-platform)
|
||||
- [Usage outside of Google Cloud Platform](#usage-outside-of-google-cloud-platform)
|
||||
- [Log levels / severities / priorities](#log-levels--severities--priorities)
|
||||
- [NixOS module](#nixos-module)
|
||||
- [Stackdriver Error Reporting](#stackdriver-error-reporting)
|
||||
|
||||
<!-- markdown-toc end -->
|
||||
|
||||
# Features
|
||||
|
||||
* `journaldriver` persists the last forwarded position in the journal
|
||||
and will resume forwarding at the same position after a restart
|
||||
* `journaldriver` will recognise log entries in JSON format and
|
||||
forward them appropriately to make structured log entries available
|
||||
in Stackdriver
|
||||
* `journaldriver` can be used outside of GCP by configuring static
|
||||
credentials
|
||||
* `journaldriver` will recognise journald's log priority levels and
|
||||
convert them into equivalent Stackdriver log severity levels
|
||||
|
||||
# Usage on Google Cloud Platform
|
||||
|
||||
`journaldriver` does not require any configuration when running on GCP
|
||||
instances.
|
||||
|
||||
1. Install `journaldriver` on the instance from which you wish to
|
||||
forward logs.
|
||||
|
||||
2. Ensure that the instance has the appropriate permissions to write
|
||||
to Stackdriver. Google continously changes how IAM is implemented
|
||||
on GCP, so you will have to refer to [Google's documentation][].
|
||||
|
||||
By default instances have the required permissions if Stackdriver
|
||||
Logging support is enabled in the project.
|
||||
|
||||
3. Start `journaldriver`, for example via `systemd`.
|
||||
|
||||
# Usage outside of Google Cloud Platform
|
||||
|
||||
When running outside of GCP, the following extra steps need to be
|
||||
performed:
|
||||
|
||||
1. Create a Google Cloud Platform service account with the "Log
|
||||
Writer" role and download its private key in JSON-format.
|
||||
2. When starting `journaldriver`, configure the following environment
|
||||
variables:
|
||||
|
||||
* `GOOGLE_CLOUD_PROJECT`: Name of the GCP project to which logs
|
||||
should be written.
|
||||
* `GOOGLE_APPLICATION_CREDENTIALS`: Filesystem path to the
|
||||
JSON-file containing the service account's private key.
|
||||
* `LOG_STREAM`: Name of the target log stream in Stackdriver Logging.
|
||||
This will be automatically created if it does not yet exist.
|
||||
* `LOG_NAME`: Name of the target log to write to. This defaults to
|
||||
`journaldriver` if unset, but it is recommended to - for
|
||||
example - set it to the machine hostname.
|
||||
|
||||
# Log levels / severities / priorities
|
||||
|
||||
`journaldriver` recognises [journald's priorities][] and converts them
|
||||
into [equivalent severities][] in Stackdriver. Both sets of values
|
||||
correspond to standard `syslog` priorities.
|
||||
|
||||
The easiest way to emit log messages with priorites from an
|
||||
application is to use [priority prefixes][], which are compatible with
|
||||
structured log messages.
|
||||
|
||||
For example, to emit a simple warning message (structured and
|
||||
unstructured):
|
||||
|
||||
```
|
||||
$ echo '<4>{"fnord":true, "msg":"structured log (warning)"}' | systemd-cat
|
||||
$ echo '<4>unstructured log (warning)' | systemd-cat
|
||||
```
|
||||
|
||||
# NixOS module
|
||||
|
||||
The NixOS package repository [contains a module][] for setting up
|
||||
`journaldriver` on NixOS machines. NixOS by default uses `systemd` for
|
||||
service management and `journald` for logging, which means that log
|
||||
output from most services will be captured automatically.
|
||||
|
||||
On a GCP instance the only required option is this:
|
||||
|
||||
```nix
|
||||
services.journaldriver.enable = true;
|
||||
```
|
||||
|
||||
When running outside of GCP, the configuration looks as follows:
|
||||
|
||||
```nix
|
||||
services.journaldriver = {
|
||||
enable = true;
|
||||
logStream = "prod-environment";
|
||||
logName = "hostname";
|
||||
googleCloudProject = "gcp-project-name";
|
||||
applicationCredentials = keyFile;
|
||||
};
|
||||
```
|
||||
|
||||
**Note**: The `journaldriver`-module is included in stable releases of
|
||||
NixOS since NixOS 18.09.
|
||||
|
||||
# Stackdriver Error Reporting
|
||||
|
||||
The [Stackdriver Error Reporting][] service of Google's monitoring
|
||||
toolbox supports automatically detecting and correlating errors from
|
||||
log entries.
|
||||
|
||||
To use this functionality log messages must be logged in the expected
|
||||
[log format][].
|
||||
|
||||
*Note*: Reporting errors from non-GCP instances requires that the
|
||||
`LOG_STREAM` environment variable is set to the special value
|
||||
`global`.
|
||||
|
||||
This value changes the monitored resource descriptor from a log stream
|
||||
to the project-global stream. Due to a limitation in Stackdriver Error
|
||||
Reporting, this is the only way to correctly ingest errors from
|
||||
non-GCP machines. Please see [issue #4][] for more information about
|
||||
this.
|
||||
|
||||
[Stackdriver Logging]: https://cloud.google.com/logging/
|
||||
[metadata server]: https://cloud.google.com/compute/docs/storing-retrieving-metadata
|
||||
[Google's documentation]: https://cloud.google.com/logging/docs/access-control
|
||||
[NixOS]: https://nixos.org/
|
||||
[contains a module]: https://github.com/NixOS/nixpkgs/pull/42134
|
||||
[journald's priorities]: http://0pointer.de/public/systemd-man/sd-daemon.html
|
||||
[equivalent severities]: https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#logseverity
|
||||
[priority prefixes]: http://0pointer.de/public/systemd-man/sd-daemon.html
|
||||
[Stackdriver Error Reporting]: https://cloud.google.com/error-reporting/
|
||||
[log format]: https://cloud.google.com/error-reporting/docs/formatting-error-messages
|
||||
[issue #4]: https://github.com/tazjin/journaldriver/issues/4
|
||||
Loading…
Add table
Add a link
Reference in a new issue