Systemd is great. Here are a few things you can do with it.
Editing unit files
I usually can't remember where unit files are located but that's not an issue. Systemd will let you edit them just by knowing the unit name.
$ export SYSTEMD_EDITOR=vim
$ sudo -E systemctl edit --full <unit_name>
Using the journal like a boss
Check out what's going on in the journal live (tail -f style) for a specific unit:
$ journalctl -u <unit_name> -fn200
An incident with nginx happened last night around 21:30? No problem, let's skip directly there:
$ journalctl -u nginx --since "2017-02-18 21:30"
Other examples:
$ journalctl -u nginx --since yesterday
$ journalctl -u nginx --since today
$ journalctl -u nginx --since "10 hours ago" --until "4 hours ago"
$ journalctl -u nginx --since 14:00
I don't think it's possible to do --since "yesterday 21:30" yet. The full date has to be used. I might contribute a patch for that.
See man systemd.time for more details.
An incident happened and it involves a few units, let's say nginx, mysql and php-fpm? Let's see what happened:
$ journalctl -u nginx -u mysql -u php-fpm --since today
Quickly store messages in the journal under a SYSLOG_IDENTIFIER:
$ echo "blah blah blah" | systemd-cat -t my_identifier
$ journalctl -t my_identifier
Make the journal persistent, assuming you kept the default Storage=auto in /etc/systemd/journald.conf:
$ sudo mkdir /var/log/journal
Alternatively, Storage=persistent can also be set.
Be careful with persistent journal. The journal will now survive reboots but by default its maximum size is set to 10% of the file system it resides in. Additionally the size is capped to 4 GB, even if you use SystemMaxUse=100% so if your unit produces lots of messages you won't be able to keep log history for a long time anyway.
Skip directly to the relevant journald.conf manpage section with the following command for more details:
$ LESS='+/SystemMaxUse' man journald.conf
Listing enabled unit files
$ systemctl list-unit-files --state=enabled
Managing logs across all your machines
While logs can be forwarded to a master machine using systemd-journal-remote (see manpage for details) you would still be capped to 4GB of logs. It's preferable to send journal messages to a Graylog instance. Use journal2gelf to export systemd messages in the Graylog Extended Log Format (GELF).
Graylog has virtual appliances for download, including a docker-compose version which sadly does not allow for proper scaling, i.e. automatically manage volumes for persistence when scaling with Elasticsearch and MongoDB which are used by Graylog.
There are however Kubernetes Helm charts for Elasticsearch and MongoDB worth looking into. With Kubernetes 100% managed manual scaling or auto-scaling could be performed.