Mastering System Logs: From systemd-journald to journalctl

1. Open the systemd-journald configuration file:

   > sudo  vi /etc/systemd/journald.conf

   [Journal]
   
   # Store journals on disk so logs survive reboot
   Storage=persistent
   
   # Disk space limits
   SystemMaxUse=2G
   SystemKeepFree=10%
   
   # Per-file max size
   SystemMaxFileSize=200M
   
   # RAM (runtime) journal limits
   RuntimeMaxUse=500M
   RuntimeKeepFree=10%
   
   # Log retention
   MaxRetentionSec=4weeks
   
   # Compression for old logs
   Compress=yes
   
   # Optional: also forward to rsyslog/syslog
   ForwardToSyslog=yes

2. Modify the configurations:

   • To modify the storage type, modify Storage=auto.

     The available options are:

     • volatile: RAM only (clears on reboot).

     • persistent: stored in /var/log/journal.

     • auto: persistent if directory exists, otherwise volatile.

     • none: no logs written to disk or memory.

     If the journal log data is saved to a persistent location, it uses up to 10% of the file system the /var/log/journal resides on. For example, if /var/log/journal is located on a 30 GB /var partition, the journal may use up to 3 GB of the disk space. To change this limit, change (and uncomment) the SystemMaxUse option:

     SystemMaxUse=50M
     > sudo  systemctl restart systemd-journald

   • To limit the log rate to prevent flooding logs:

     RateLimitIntervalSec=30s
     RateLimitBurst=1000

   • To send the journal to a terminal device to inform you about system messages on a preferred terminal screen, for example /dev/tty12:

     ForwardToConsole=yes
     TTYPath=/dev/tty12

   • To forward logs to syslog, modify ForwardToSyslog=yes. journald is backward compatible with traditional syslog implementations such as rsyslog.

     • Install rsyslog:

       rpm -q rsyslog

     • Enable rsyslog:

       systemctl is-enabled rsyslog

     • Enable forwarding to rsyslog in /etc/systemd/journald.conf:

       ForwardToSyslog=yes

     For more information on file descriptions, see man 5 journald.conf.

3. Save and restart systemd-journald.

   > sudo  systemctl restart systemd-journald

1. Check if the directory exist:

   > sudo ls /var/log
   ...        
   journal                             
   ...

2. If the /var/log/journal directory does not exist, create it:

   > sudo mkdir /var/log/journal

3. Set the correct ownership and access permissions:

   > sudo   systemd-tmpfiles --create --prefix=/var/log/journal

4. (Optional) To flush the data from RAM to the directory:

   > sudo   journalctl --flush

1. View the list of failed systemd units:

   systemctl --failed

   The list of all failed services appears.

2. Identify the severity and content of the error.

   journalctl -p 0..7 -b

   The list of errors with priority levels 0 to 7 appears. The errors are marked with priority level:

   • 0-2: emerg, alert, crit: Critical issues, system collapse imminent.

   • 3: err (Error): A service or application failed to complete a requested operation.

   • 4: warning : Something undesirable happened but is not an immediate failure.

   • 5-7: notice, info, debug: Normal operational information and developer diagnostics.

3. View the error log for the failing service using the following command:

   journalctl -u <failing_service_name>

   The lines preceding the termination message include information about the error.

   • Exit Status: If a service stops, look for a message like Process xxx exited with status 1/FAILURE. A status of 0 indicates success. Any nonzero status indicates an error.

   • Configuration Errors: Messages containing phrases like No such file or directory, Permission denied or Address already in use usually point to a problem in the service's configuration file.

   • Out-of-Memory (OOM): The service was terminated because it exceeded memory limits.

4. Use detailed views of logs to get detailed information about the error.

   journalctl -xe

   For example, to view all messages from the Apache service during the current boot with detailed explanations, run the following command:

   journalctl -u httpd.service -b -xe

This section introduces an example to illustrate how to find and fix the error reported by systemd during apache2 start-up.

1. Start the apache2 service:

   systemctl start apache2
   Job for apache2.service failed. See 'systemctl status apache2' and 'journalctl -xn' for details.

2. View the service status:

   > sudo systemctl status apache2
   apache2.service - The Apache Webserver
      Loaded: loaded (/usr/lib/systemd/system/apache2.service; disabled)
      Active: failed (Result: exit-code) since Tue 2025-09-03 11:08:13 EDT; 7min ago
     Process: 11026 ExecStop=/usr/sbin/start_apache2 -D SYSTEMD -DFOREGROUND \
              -k graceful-stop (code=exited, status=1/FAILURE)

   The ID of the process causing the failure is 11026.

3. View the verbose version of messages related to process ID 11026:

   > sudo journalctl -o verbose _PID=11026
   [...]
   MESSAGE=AH00526: Syntax error on line 6 of /etc/apache2/default-server.conf:
   [...]
   MESSAGE=Invalid command 'DocumenttRoot', perhaps misspelled or defined by a module
   [...]

4. Fix the typo inside /etc/apache2/default-server.conf, start the apache2 service, and print its status:

   > sudo systemctl start apache2 && systemctl status apache2
   apache2.service - The Apache Webserver
      Loaded: loaded (/usr/lib/systemd/system/apache2.service; disabled)
      Active: active (running) since Tue 2025-09-03 11:26:24 EDT; 4ms ago
     Process: 11026 ExecStop=/usr/sbin/start_apache2 -D SYSTEMD -DFOREGROUND
              -k graceful-stop (code=exited, status=1/FAILURE)
    Main PID: 11263 (httpd2-prefork)
      Status: "Processing requests..."
      CGroup: /system.slice/apache2.service
              ├─11263 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf -D [...]
              ├─11280 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf -D [...]
              ├─11281 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf -D [...]
              ├─11282 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf -D [...]
              ├─11283 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf -D [...]
              └─11285 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf -D [...]