deep dive with journalctl in Rhel 7

Journalctl : 

This is one of the most important component of Systemd that is responsible for viewing and managing log files.


Note:   You can check by using below command :


[root@station227 ~]# pstree   |   grep -i journ     

This is very similar to logs in managed by  rsyslogd   that is storing logs persistently but journalctl  storing logs in tmpfs  under  /run/log/  that is not persistent .


Important Options: 

1.    view all the logs  in verbose mode

[root@station227 log]# journalctl  -o verbose

-- Logs begin at Thu 2018-08-30 06:58:02 GMT, end at Fri 2018-08-31 07:10:01 GMT. --
Thu 2018-08-30 06:58:02.565541 GMT [s=34f56459bdda420e9fbb3d389c538572;i=1;b=633ca5e2242543daa9c895c1485
    PRIORITY=6
    _TRANSPORT=driver
    MESSAGE=Runtime journal is using 8.0M (max allowed 788.7M, trying to leave 1.1G free of 7.6G availab
    MESSAGE_ID=ec387f577b844b8fa948f33cad9a75e6
    _PID=99
    _UID=0
    _GID=0
    _COMM=systemd-journal
    _EXE=/usr/lib/systemd/systemd-journald
    _CMDLINE=/usr/lib/systemd/systemd-journald
    _CAP_EFFECTIVE=5402800cf



Note:  Other options your use tab button 

[root@station227 log]# journalctl  -o 
 cat              json             json-sse         short-iso        short-precise   
export           json-pretty      short            short-monotonic  verbose         


2.   To show only most recent entry kind of live logs


[root@station227 log]# journalctl -f
-- Logs begin at Thu 2018-08-30 06:58:02 GMT. --
Aug 31 07:10:42 station227.example.com NetworkManager[636]: <info>  [1535699442.5888] dhcp4 (br0):   nameserver '192.168.10.254'
Aug 31 07:10:42 station227.example.com NetworkManager[636]: <info>  [1535699442.5888] dhcp4 (br0):   domain name 'example.com'
Aug 31 07:10:42 station227.example.com NetworkManager[636]: <info>  [1535699442.5888] dhcp4 (br0): state changed bound -> bound
Aug 31 07:10:42 station227.example.com dhclient[788]: bound to 192.168.10.227 -- renewal in 277 seconds.
Aug 31 07:10:42 station227.example.com dbus[629]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
Aug 31 07:10:42 station227.example.com systemd[1]: Starting Network Manager Script Dispatcher Service...
Aug 31 07:10:42 station227.example.com dbus[629]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Aug 31 07:10:42 station227.example.com nm-dispatcher[16894]: req:1 'dhcp4-change' [br0]: new request (4


3.  To view only boot logs

[root@station227 log]# journalctl -b
-- Logs begin at Thu 2018-08-30 06:58:02 GMT, end at Fri 2018-08-31 07:16:48 GMT. --
Aug 31 04:56:48 localhost.localdomain systemd-journal[99]: Runtime journal is using 8.0M (max allowed 78
Aug 31 04:56:48 localhost.localdomain kernel: microcode: microcode updated early to revision 0x29, date
Aug 31 04:56:48 localhost.localdomain kernel: Initializing cgroup subsys cpuset
Aug 31 04:56:48 localhost.localdomain kernel: Initializing cgroup subsys cpu
Aug 31 04:56:48 localhost.localdomain kernel: Initializing cgroup subsys cpuacct
Aug 31 04:56:48 localhost.localdomain kernel: Linux version 3.10.0-862.el7.x86_64 (mockbuild@x86-034.bui
Aug 31 04:56:48 localhost.localdomain kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-862.el7.x86_6



4.  Enable persistent storage for journalctl 

Create a directory with  given owner and group

[root@station227 log]# ls  -ld /var/log/journal/
drwxr-sr-x+ 3 root systemd-journal 46 Aug 30 10:52 /var/log/journal/


5.   viewing only logs  for particular service


[root@station227 ~]# journalctl -u sshd


6.  Since case use with  journalctl 

[root@station227 ~]# journalctl  --since  "2018-8-20"


7.  checking logs on behalf of priority 

To check  Logs by Priority

You can also select to view logs by priority.  The journal uses the same syslog message levels:

0: emerg
1: alert
2: critical
3: error
4: warning
5: notice
6: info
7: debug

Example: 

[root@station227 ~]# journalctl  -p 2 -- Logs begin at Thu 2018-08-30 06:58:02 GMT, end at Fri 2018-08-31 07:32:35 GMT. --
Aug 30 07:01:58 station227.example.com kernel: XFS (sdb1): metadata I/O error: block 0xe6fc22 ("xlog_iod
Aug 30 07:01:58 station227.example.com kernel: XFS (sdb1): Log I/O Error Detected.  Shutting down filesy
Aug 30 07:01:58 station227.example.com kernel: XFS (sdb1): Please umount the filesystem and rectify the
Aug 30 13:55:54 station227.example.com kernel: XFS (sdb1): metadata I/O error: block 0xe6fcde ("xlog_iod
Aug 30 13:55:54 station227.example.com kernel: XFS (sdb1): Log I/O Error Detected.  Shutting down filesy
Aug 30 13:55:54 station227.example.com kernel: XFS (sdb1): Please umount the filesystem and rectify the
lines 1-7/7 (END)


8.  Disk usage by logs


[root@station227 ~]# journalctl  --disk-usage

 Archived and active journals take up 16.0M on disk. 

 ***
If you want to clean old logs and make disk free
****

**********  Removing all the logs except 10 MB disk size**************

journalctl --vacuum-size=10M 

############# by time ###########

journalctl  --vacuum-time=1years 

Note:  please check for last two options in your OS.


Enjoy the server troubleshooting  tips.