Systemd is great. Here are a few things you can do with it.
Editing unit files
I usually can't remember where unit files are located but that's not an issue. Systemd will let you edit them just by knowing the unit name.
$ export SYSTEMD_EDITOR=vim $ sudo -E systemctl edit --full <unit_name>
Using the journal like a boss
Check out what's going on in the journal live (
tail -f style) for a specific unit:
$ journalctl -u <unit_name> -fn200
An incident with nginx happened last night around 21:30? No problem, let's skip directly there:
$ journalctl -u nginx --since "2017-02-18 21:30"
$ journalctl -u nginx --since yesterday $ journalctl -u nginx --since today $ journalctl -u nginx --since "10 hours ago" --until "4 hours ago" $ journalctl -u nginx --since 14:00
I don't think it's possible to do
--since "yesterday 21:30" yet. The full date has to be used. I might contribute a patch for that.
man systemd.time for more details.
An incident happened and it involves a few units, let's say nginx, mysql and php-fpm? Let's see what happened:
$ journalctl -u nginx -u mysql -u php-fpm --since today
Quickly store messages in the journal under a SYSLOG_IDENTIFIER:
$ echo "blah blah blah" | systemd-cat -t my_identifier $ journalctl -t my_identifier
Make the journal persistent, assuming you kept the default
$ sudo mkdir /var/log/journal
Storage=persistent can also be set.
Be careful with persistent journal. The journal will now survive reboots but by default its maximum size is set to 10% of the file system it resides in. Additionally the size is capped to 4 GB, even if you use
SystemMaxUse=100% so if your unit produces lots of messages you won't be able to keep log history for a long time anyway.
Skip directly to the relevant
journald.conf manpage section with the following command for more details:
$ LESS='+/SystemMaxUse' man journald.conf
Listing enabled unit files
$ systemctl list-unit-files --state=enabled
Managing logs across all your machines
While logs can be forwarded to a master machine using
systemd-journal-remote (see manpage for details) you would still be capped to 4GB of logs. It's preferable to send journal messages to a Graylog instance. Use journal2gelf to export systemd messages in the Graylog Extended Log Format (GELF).
Graylog has virtual appliances for download, including a docker-compose version which sadly does not allow for proper scaling, i.e. automatically manage volumes for persistence when scaling with Elasticsearch and MongoDB which are used by Graylog.