Greetings!

We are introducing support for derived fields in Site24x7 AppLogs. This feature enables you to create custom parsing rules for log fields that help you extract valuable information. For example, a log file may contain a treasure trove of information in its message field, which can come in a variety of formats.

Here are a few instances of log messages.

• Syslog messages with a user's email and IP information
  E.g. Jun  7 09:22:38 10.8.0.36 : %ASA-4-722051: Group <GroupPolicyVPNCert> User <demo123@gmail.com> IP <127.0.0.1> IPv4 Address <127.0.0.1> IPv6 address <::> assigned to session
  
  
• IIS access logs request URI that contains user data and tracking identifier
  E.g. 2022-04-29 07:18:23 GET /Default.aspx?msclkid=b3ed9d&X-ARR-LOG=_gid=GA1.2.933557932.1651173771;+_uetmsclkid=_uet917e5c36c728; 
   
• Messages in the logs that include the domain and SSL handshake time
  E.g. [Thu Aug 12 14:52:23 IST 2022|DEBUG|39]: SSL Handshake Time zylker.com :42
  
  Say you want to extract SSL handshake time and domain name from log messages. To do this, add a RegEx rule to create derived fields for the domain and SSL time to dig out the data from the message field. You can visualize these data on a single dashboard for a better understanding and get alerted for abnormal values.

How to use derived field support?

• Add a log type and define the sample logs and log pattern.

• Click on the Edit Field Configurations and choose the Field name from the left menu to add a derived field.

• To create custom-derived fields, define a RegEx rule with the named capturing group syntax (?<name>capturing text).
  
  Consider the example below:
  Message Field value: SSL Handshake Time for zylker.com is: 42 
  
  To extract data from the above field value, use the below RegEx rule:
  RegEx : SSL\sHandshake\sTime\sfor\s(?<Domain>.*)\sis:\s(?<SSLTime>\d+)
  
  Here, Domain and SSLTime are the customized derived fields that fetch the values "zylker.com" and "42," respectively. You can validate your RegEx using our free tool.

• Specify a data type for the custom field, then add and apply the rule.

Here's how you can create a derived field to extract the domain name and SSL handshake time.

How is it going to help you?

You can perform a query language search on the derived fields to get actionable insights. You can follow the fieldname_derivedfieldname pattern to query the derived fields.

Consider the following query as an example:
logtype="EumDemo" and message_ssltime>100

This query fetches the log entries with the log type EumDemo and a message field containing the ssltime greater than 100. You can also perform searches by combining the derived fields in the same query as below:

logtype="EumDemo" and message_ssltime>100 groupby message_domain.

This will group the domains that are taking greater than 100 milliseconds of SSLTime.

Read our query language documentation to learn more.

We hope our derived field support helps you to extract more valuable information. Feel free to drop your feedback, suggestions, and feature requests as comments below.

Until next time,
Happy logging!