6 common Linux commands for log analysis

Tram Ho

Today I share with everyone some of the linux commands that I often use in log analysis on the server.

1. Commonly used usecases

  • Want to investigate the cause when the number of accesses to the server soared
  • Detected on the server a flaw and want to investigate whether anyone has intruded or not

2. Commonly used linux commands

Command lineExplain
grepFind lines containing the specified string
awkSplit columns and search rows conditionally
sortLine arrangement
uniqRemove duplicate rows and count
WCCount the number of characters, the number of lines
sedReplace specified conditions

3. Subject of analysis

Inside:

  • 203.0.113.0/24 is the correct ip address from the user
  • 198.51.100.0/24 is the ip address from the suspicious source

4. Apply

4.1. Special search (grep)

・ Investigate attacks on directories

・ Investigate suspicious script insertion

・ Investigate suspicious html tag insertion

Explain options in grep:

optionExplain
-nDisplays the line number of the match line
-iNot case sensitive and lowercase
–colorDisplay color for matched word
-oShow matched part
-EUse regex expressions

4.2. Eliminate unnecessary logs

・ Remove log from standard source

Explain options in grep:

optionExplain
-vDo not show match lines

4.3. Extract element (awk)

・ Extract the ip client address

・ Extract the client’s User Agent

Explain options in awk:

optionExplain
{print $ 1}Displays the first column separated by a separator
-FSpecify a separator (default is white space)

4.4. Statistics and sorting (sort / uniq / wc)

・ Display the number of each client IP address

・ The total number of unique ip client addresses

Explain:

command line, optionExplain
uniq -cDisplay duplicate numbers (requires sorting for comparison before and after)
sort -nArrange number fields according to arithmetic values
sort -rSort in descending order (default is ascending order)
wc -lDisplay the line number

4.5. Replace (sed)

・ Anonymize the source IP address for normal access

Explain:

optionExplain
s / string A / string B / gReplace string A with string B. Replace all by adding g.

Conclude

If we use the 6 statements we mentioned above, we can easily analyze the log.

In addition, it is common to use pipe line characters | to analyze log 1 more effectively.

For example:

==============

To receive notifications when there are latest posts, you can like my fanpage below:

→→→ Coding Art Facebook Fanpage

Share the news now

Source : Viblo