Home > Articles

Hands-on Guide to the Red Hat Exams: System Logging, Monitoring, and Automation

This chapter covers working with Syslog, monitoring system performance, and automation with cron and at.
This chapter is from the book
  • Working with Syslog—This section covers logging, both local and centralized.
  • Monitoring System Performance—This section looks at how to monitor system performance.
  • Automation with cron and at—This section covers system automation via scheduled and single-instance jobs.

The following RHCSA exam objectives are covered:

Identify CPU and memory-intensive processes, adjust process priority with renice, and kill processes

Locate and interpret system log files

Schedule tasks using cron

The following RHCE exam objectives are covered:

Produce and deliver reports on system utilization (processor, memory, disk, and network)

Use shell scripting to automate system maintenance tasks

Configure a system to log to a remote system

Configure a system to accept logging from a remote system

How do you know when something goes wrong on your system? You look at the logs, of course! In all seriousness, understanding system logging is important so that you can troubleshoot when something goes wrong. Logs are kept for almost everything that you can think of, including the kernel, boot process, and different services that are running. Aside from logs, there are also utilities you can use to monitor and check the status of your system. This chapter looks at how you can use these tools and logging to troubleshoot anything that is thrown your way. The chapter also touches on how to automate some of these tasks.

Working with Syslog

Whenever something goes wrong—and sometimes when things go right—on the system, a message is generated by the syslog service. These messages are used to keep track of how the system is performing, what actions are taking place, and if there is a problem with something. Red Hat comes with a syslog service built in and already set to start at boot. In RHEL6, the rsyslog package is installed by default and is the primary logging service used.

The rsyslog service has a main config file, /etc/rsyslog.conf, that controls where messages are sent when they are generated. Because logging is a critical piece of the operating system, it is installed by default as part of the core packages.

  • Step 1. Just out of good habit, you should verify that the package is installed:
    # rpm -qa | grep syslog
    rsyslog-4.6.2-2.el6.x86_64
    
    Because you can see that it is installed, you should also check on two other issues.
  • Step 2. First, make sure that the service is set to start when the system boots:
    # chkconfig rsyslog --list
    rsyslog           0:off   1:off   2:on    3:on    4:on    5:on    6:off
    
  • Step 3. Second, make sure that the service is currently running:
    # service rsyslog status
    rsyslogd (pid  1279) is running...
    

For RHEL05:

  • Step 1. Verify the installation of the package:
    # rpm -qa | grep klog
    sysklogd-1.4.1-44.el5
    
  • Step 2. Check that the service will start on boot:
    # chkconfig syslog --list
    syslog           0:off   1:off   2:on    3:on    4:on    5:on    6:off
    
    Aside from the name difference, as we have already discussed, there are two daemons that run instead of one. They do, however, both run under a single service name.
  • Step 3. Verify that the service is running:
    # service syslog status
    syslogd (pid  2214) is running...
    klogd (pid  2217) is running...
    

As you can see here, there are some slight differences between logging on RHEL5 and RHEL6 (such as the naming and single versus double daemons). The rsyslog service provides better logging features, so it is recommended to upgrade if possible.

All the messages sent to the rsyslog service are stored in the /var/log directory, and each message section has its own file or subdirectory. You can also define your own log files or directories. The biggest problem that arises from using any logging service is the files can become uncontrollable if left unchecked. Being able to log different alerts and to obtain information from logs, in my opinion, is an art all unto itself. There are so many ways to customize logs, and each network setup is different. You should know that there are nine different alerts that the syslog service uses. These alerts can be produced by different network devices and Red Hat systems.

The nine severity levels of syslog include the following:

  • emerg
  • alert
  • crit
  • err
  • warning
  • notice
  • info
  • debug
  • none

The Config File

The /etc/rsyslog.conf file is broken down into the following parts:

Modules

Indicate whether they can be loaded or unloaded (rsyslog is modular)

Global directives

Specify config options that apply to the daemon (all start with a $)

Rules

Indicate cooperation of a selector and an action

Selector

Is based on <facility>.<priority>

Action

Defines what to do with messages after they're sorted by the selector

Let's look at the default config file to see some examples:

# cat /etc/rsyslog.conf
#rsyslog v3 config file

# if you experience problems, check
# http://www.rsyslog.com/troubleshoot for assistance

#### MODULES ####

$ModLoad imuxsock.so    # provides support for local system logging (e.g. via logger command)
$ModLoad imklog.so      # provides kernel logging support (previously done by rklogd)
#$ModLoad immark.so     # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp.so
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp.so
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                     /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                   /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                                 /var/log/secure

# Log all the mail messages in one place.
mail.*                                                     -/var/log/maillog
# Log cron stuff
cron.*                                                      /var/log/cron

# Everybody gets emergency messages
*.emerg                                                     *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                              /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                    /var/log/boot.log

# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/spppl/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

You can see from the output where some of the files are stored. Also shown is a sample rule.

Log Rotation

If left alone, logs can grow to enormous size. Luckily for you, the logrotate command allows you to rotate logs before they become too big. By default, the logrotate command is called daily to cycle the log files into a new file. The parameters of the command are defined in its config file, /etc/logrotate.conf, and the /etc/logrotate.d directory. However, the command itself is called from /etc/cron.daily/logrotate, a cron job that is run daily (as already mentioned). You can always call the logrotate command yourself if you'd like to rotate your logs, but don't do this too frequently because you'll end up with tons of rotated logs and you'll be right back to square one with a mess of unorganized log files.

Syntax: logrotate [options] configfile

Options:

-d

Debugs; doesn't do anything but test

-f

Forces file rotation

-v

Provides verbose output

The following example rotates the current logs as defined in the config file:

# logrotate /etc/logrotate.conf

Centralized Logging

If you'd like to set up a centralized syslog server, you need to choose which server you'd like to store all your logs on. For this example, you can use the RHEL01 system as the centralized server and configure it to receive logs from the other systems on the network.

  • Step 1. Look at a section of the /etc/rsyslog.conf file again:
    #### MODULES ####
    
    $ModLoad imuxsock.so    # provides support for local system logging (e.g. via logger command)
    $ModLoad imklog.so      # provides kernel logging support (previously done by rklogd)
    #$ModLoad immark.so     # provides --MARK-- message capability
    
    # Provides UDP syslog reception
    #$ModLoad imudp.so
    #$UDPServerRun 514
    
    # Provides TCP syslog reception
    #$ModLoad imtcp.so
    #$InputTCPServerRun 514
    
    In the modules section, you can see two areas that will provide the capability to receive logs from other systems. The standard is to use the UDP protocol on port 514. By uncommenting the section related to UDP reception, you essentially can provide a way for the syslog server to become the centralized server for the network.
  • Step 2. Uncomment the following two lines:
    $ModLoad imudp.so
    $UDPServerRun 514
    
  • Step 3. When you're finished, you need to save the file, exit, and restart the syslog service:
    # service rsyslog restart
    Shutting down system logger:                              [  OK  ]
    Starting system logger:                                   [  OK  ]
    
    As with any service that you'd like to make available to your network, you need to create a firewall allowing UDP 514 to be opened up.
  • Step 4. Use iptables to create the required firewall rule:
    # iptables -I INPUT 5 -p udp -m udp --dport 514 -j ACCEPT
    
          
  • Step 5. Save the firewall rule you just created:
    # service iptables save
    Saving firewall rules to /etc/sysconfig/iptables:         [  OK  ]
    
  • Step 6. Then restart the iptables service:
    # service iptables restart
    iptables: Flushing firewall rules:                        [  OK  ]
    iptables: Setting chains to policy ACCEPT: filter         [  OK  ]
    iptables: Unloading modules:                              [  OK  ]
    iptables: Applying firewall rules:                        [  OK  ]
    

The firewall rules might not make sense at the moment because we haven't covered iptables or firewall rules yet. After you read Chapter 12, "System Security," return here to make sure you understand what is happening in the creation of firewall rules.

Now that the security requirements have been addressed and the centralized server is configured, you can shift your attention to the client systems. Let's look at RHEL02 for the client system. You can config RHEL02 to send some or all of its log files to the centralized syslog server (RHEL01).

You can use the following syntax when configuring log files to be sent to the centralized server:

<log file>           @<hostname or IP of system (local or remote)>

You can edit the RHEL02 system to point all security logs to the central server:

# nano /etc/rsyslog.conf
authpriv.*            @172.168.1.1

Save the file and restart the syslog service on RHEL02:

# service rsyslog restart
Shutting down system logger:                              [  OK  ]
Starting system logger:                                   [  OK  ]

If you log out and log back in on RHEL02, it generates a security event. You can then jump over to the RHEL01 host and check the logs to see the event generated by the client system RHEL02.

Centralized Logging (The RHEL5 Way)

To set up centralized logging for RHEL5, do the following:

  • Step 1. Edit the /etc/sysconfig/syslog file to include the -r option:
    # nano /etc/sysconfig/syslog
    # Options to syslogd
    # -m 0 disables 'MARK' messages.
    # -r enables logging from remote machines
    # -x disables DNS lookups on messages received with -r
    # See syslogd(8) for more details
    SYSLOGD_OPTIONS="-m 0 -r"
    
  • Step 2. Save the file, exit, and restart the syslog service:
    # service syslog restart
    Shutting down kernel logger:                              [  OK  ]
    Shutting down system logger:                              [  OK  ]
    Starting system logger:                                   [  OK  ]
    Starting kernel logger:                                   [  OK  ]
    
    With the syslog service now accepting remote logs, you need to create a firewall rule. This process is the same as on RHEL6. One last modification that needs to be made on RHEL5 is the adjustment of some SELinux protections.
  • Step 3. Query the required Boolean values:
    # getsebool -a | grep logd
    klogd_disable_trans --> off
    syslogd_disable_trans --> off
    
  • Step 4. Change the values to allow incoming logs:
    # setsebool -P klogd_disable_trans=1 syslogd_disable_trans=1
    
          
  • Step 5. Verify the preceding Booleans have been changed:
    # getsebool -a | grep logd
    klogd_disable_trans --> on
    syslogd_disable_trans --> on
    

Although we haven't covered SELinux yet (it is covered in Chapter 11, "SELinux"), the Boolean values are required here if you are still using RHEL5. After you read through Chapter 11, you can refer back here to make sure you understand the changes being made.

User Login Events

Aside from the normal logs generated and used by the syslog service, there are two special commands that deal with system logins. These two commands have special logs that can be read only through the use of these commands:

lastlog

Lists login records

faillog

Lists failed login attempts

You can use these two commands for viewing login events. These two commands are useful for hunting down attacks where users are trying to force themselves into other users' accounts (called a brute-force attack).

Syntax: lastlog [options]

Options:

-b DAYS

Displays results older than DAYS

-u LOGIN

Displays results for the user LOGIN

Show the last time the user01 account logged in:

# lastlog -u user01
Username          Port     From             Latest
user01            tty                       Fri Sep 10 05:16:42 -0400 2010

You can also view more details using the faillog command.

Syntax: faillog [options]

Options:

-a

Displays all events

-l SEC

Locks the account for SEC seconds after a failed login

-u LOGIN

Prints records for user LOGIN

Show more information about the user01 account login events:

# faillog -u user01
Login       Failures Maximum Latest                   On
user01          0        0

The ability to work with syslog is important to being able to help troubleshoot any issue on Linux, not just Red Hat. When you're trying to find an issue or resolve one, the log files should always be your first stop.

Pearson IT Certification Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from Pearson IT Certification and its family of brands. I can unsubscribe at any time.

Overview


Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about Pearson IT Certification products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information


To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.

Surveys

Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites; develop new products and services; conduct educational research; and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.

Newsletters

If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@informit.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information


Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.

Security


Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.

Children


This site is not directed to children under the age of 13.

Marketing


Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information


If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.

Choice/Opt-out


Users can always make an informed choice as to whether they should proceed with certain services offered by Adobe Press. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.pearsonitcertification.com/u.aspx.

Sale of Personal Information


Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents


California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure


Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.

Links


This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact


Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice


We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020