grok语法定义

柔光的暖阳◎ 2023-08-17 17:16 195阅读 0赞

### grok默认表达式 ###

Logstash 内置了120种默认表达式，可以查看patterns，里面对表达式做了分组，每个文件为一组，文件内部有对应的表达式模式。下面只是部分常用的。

### 常用表达式 ###

<table> 
 <thead> 
  <tr> 
   <th> 
    <div> 
     <p>表达式标识</p> 
    </div> </th> 
   <th> 
    <div> 
     <p>名称</p> 
    </div> </th> 
   <th> 
    <div> 
     <p>详情</p> 
    </div> </th> 
   <th> 
    <div> 
     <p>匹配例子</p> 
    </div> </th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <p>USERNAME 或 USER</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>用户名</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>由数字、大小写及特殊字符(._-)组成的字符串</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>1234、Bob、Alex.Wong</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>EMAILLOCALPART</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>用户名</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>首位由大小写字母组成，其他位由数字、大小写及特殊字符(_.+-=:)组成的字符串。注意，国内的QQ纯数字邮箱账号是无法匹配的，需要修改正则</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>windcoder、windcoder_com、abc-123</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>EMAILADDRESS</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>电子邮件</p> 
    </div> </td> 
   <td>&nbsp;</td> 
   <td> 
    <div> 
     <p>windcoder@abc.com、windcoder_com@gmail.com、abc-123@163.com</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>HTTPDUSER</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>Apache服务器的用户</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>可以是EMAILADDRESS或USERNAME</p> 
    </div> </td> 
   <td>&nbsp;</td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>INT</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>整数</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>包括0和正负整数</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>0、-123、43987</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>BASE10NUM 或 NUMBER</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>十进制数字</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>包括整数和小数</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>0、18、5.23</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>BASE16NUM</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>十六进制数字</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>整数</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>0x0045fa2d、-0x3F8709</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>WORD</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>字符串</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>包括数字和大小写字母</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>String、3529345、ILoveYou</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>NOTSPACE</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>不带任何空格的字符串</p> 
    </div> </td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>SPACE</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>空格字符串</p> 
    </div> </td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>QUOTEDSTRING 或 QS</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>带引号的字符串</p> 
    </div> </td> 
   <td>&nbsp;</td> 
   <td> 
    <div> 
     <p>"This is an apple"、'What is your name?'</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>UUID</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>标准UUID</p> 
    </div> </td> 
   <td>&nbsp;</td> 
   <td> 
    <div> 
     <p>550E8400-E29B-11D4-A716-446655440000</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>MAC</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>MAC地址</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>可以是Cisco设备里的MAC地址，也可以是通用或者Windows系统的MAC地址</p> 
    </div> </td> 
   <td>&nbsp;</td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>IP</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>IP地址</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>IPv4或IPv6地址</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>127.0.0.1、FE80:0000:0000:0000:AAAA:0000:00C2:0002</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>HOSTNAME</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>IP或者主机名称</p> 
    </div> </td> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>HOSTPORT</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>主机名(IP)+端口</p> 
    </div> </td> 
   <td>&nbsp;</td> 
   <td> 
    <div> 
     <p>127.0.0.1:3306、api.windcoder.com:8000</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>PATH</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>路径</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>Unix系统或者Windows系统里的路径格式</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>/usr/local/nginx/sbin/nginx、c:\windows\system32\clr.exe</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>URIPROTO</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>URI协议</p> 
    </div> </td> 
   <td>&nbsp;</td> 
   <td> 
    <div> 
     <p>http、ftp</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>URIHOST</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>URI主机</p> 
    </div> </td> 
   <td>&nbsp;</td> 
   <td> 
    <div> 
     <p>windcoder.com、10.0.0.1:22</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>URIPATH</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>URI路径</p> 
    </div> </td> 
   <td>&nbsp;</td> 
   <td> 
    <div> 
     <p>//windcoder.com/abc/、/api.php</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>URIPARAM</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>URI里的GET参数</p> 
    </div> </td> 
   <td>&nbsp;</td> 
   <td> 
    <div> 
     <p>?a=1&amp;b=2&amp;c=3</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>URIPATHPARAM</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>URI路径+GET参数</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>/windcoder.com/abc/api.php?a=1&amp;b=2&amp;c=3</p> 
    </div> </td> 
   <td>&nbsp;</td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>URI</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>完整的URI</p> 
    </div> </td> 
   <td>&nbsp;</td> 
   <td> 
    <div> 
     <p>https://windcoder.com/abc/api.php?a=1&amp;b=2&amp;c=3</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>LOGLEVEL</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>Log表达式</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>Log表达式</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>Alert、alert、ALERT、Error</p> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

### 日期时间表达式 ###

<table> 
 <thead> 
  <tr> 
   <th> 
    <div> 
     <p>表达式标识</p> 
    </div> </th> 
   <th> 
    <div> 
     <p>名称</p> 
    </div> </th> 
   <th> 
    <div> 
     <p>匹配例子</p> 
    </div> </th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <p>MONTH</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>月份名称</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>Jan、January</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>MONTHNUM</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>月份数字</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>03、9、12</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>MONTHDAY</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>日期数字</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>03、9、31</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>DAY</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>星期几名称</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>Mon、Monday</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>YEAR</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>年份数字</p> 
    </div> </td> 
   <td>&nbsp;</td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>HOUR</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>小时数字</p> 
    </div> </td> 
   <td>&nbsp;</td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>MINUTE</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>分钟数字</p> 
    </div> </td> 
   <td>&nbsp;</td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>SECOND</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>秒数字</p> 
    </div> </td> 
   <td>&nbsp;</td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>TIME</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>时间</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>00:01:23</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>DATE_US</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>美国时间</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>10-01-1892、10/01/1892/</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>DATE_EU</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>欧洲日期格式</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>01-10-1892、01/10/1882、01.10.1892</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>ISO8601_TIMEZONE</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>ISO8601时间格式</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>+10:23、-1023</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>TIMESTAMP_ISO8601</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>ISO8601时间戳格式</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>2016-07-03T00:34:06+08:00</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>DATE</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>日期</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>美国日期%{DATE_US}或者欧洲日期%{DATE_EU} |</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>DATESTAMP</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>完整日期+时间</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>07-03-2016 00:34:06</p> 
    </div> </td> 
  </tr> 
  <tr> 
   <td> 
    <div> 
     <p>HTTPDATE</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>http默认日期格式</p> 
    </div> </td> 
   <td> 
    <div> 
     <p>03/Jul/2016:00:36:53 +0800</p> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

## 自定义grok表达式 ##

语法解释：  
%\{HOSTNAME\}，匹配请求的主机名  
%\{TIMESTAMP\_ISO8601:time\},代表时间戳  
%\{LOGLEVEL\}，代表日志级别  
%\{URIPATHPARAM\}，代表请求路径  
%\{INT\}，代表字符串整数数字大小  
%\{NUMBER\}， 可以匹配整数或者小数

%\{UUID\}，匹配类似091ece39-5444-44a1-9f1e-019a17286b48  
%\{IP\}, 匹配ip  
%\{WORD\}, 匹配请求的方式  
%\{GREEDYDATA\},代表所有的数据  
(?(\[\\S+\]\*))，自定义正则  
\\s\*或者\\s+,代表多个空格  
\\S+或者\\S\*,代表多个字符  
大括号里面:xxx,相当于起别名

(?<class\_info>(\[\\S+\]\*)), 自定义正则匹配多个字符

举例操作如下：

列1：  
\[2019-08-22 12:25:51.441\] \[TSC\_IHU\] \[ERROR\] \[c.e.c.t.i.t.s.IhuTsaUplinkServiceImpl\] Activation/Bind uplink, query UserSession by Token failure!

grok调试如下：  
\\\[%\{TIMESTAMP\_ISO8601:time\}\\\]\\s\*%\{DATA:thread\}\\s\*\\\[%\{LOGLEVEL:level\}\\\]\\s\*%\{GREEDYDATA:data\}

![857064-20190912141413660-655629477.png][]

列2：

2019-09-12 14:16:36.320+08:00 INFO 930856f7-c78f-4f12-a0f1-83a2610b2dfc DispatcherConnector ip-192-168-114-244 \[Mqtt-Device-1883-worker-18-1\] com.ericsson.sep.dispatcher.api.transformer.v1.MessageTransformer \{"TraceID":"930856f7-c78f-4f12-a0f1-83a2610b2dfc","clientId":"5120916600003466K4GA1059","username":"LB37622Z3KX609880"\} uplinkTransform|RVS|LB37622Z3KX609880|VDF|\{"header":\{"requestid":117,"timeStamp":1568268995157,"eventId":1568268995157,"creatorId":0,"requestType":0\},"body":\{"serviceId":51,"serviceData":\{"vehicleStatus":\{"basicVehicleStatus":\{"position":\{"latitude":131611658,"longitude":404483724,"altitude":1019,"posCanBeTrusted":true,"carLocatorStatUploadEn":true,"marsCoordinates":false\},"speed":745,"speedValidity":0,"direction":222,"engineStatus":0,"engineBlockedStatus":0,"distanceToEmpty":393\},"additionalVehicleStatus":\{"drivingSafetyStatus":\{"srsStatus":0,"vehicleAlarm":0,"doorOpenStatusDriver":0,"doorOpenStatusPassenger":0,"doorOpenStatusDriverRear":0,"doorOpenStatusPassengerRear":0,"doorLockStatusDriver":0,"doorLockStatusPassenger":0,"doorLockStatusDriverRear":0,"doorLockStatusPassengerRear":0,"trunkOpenStatus":0,"trunkLockStatus":0,"engineHoodOpenStatus":0,"seatBeltStatusDriver":false,"seatBeltStatusPassenger":false,"electricParkBrakeStatus":0,"electricParkBrakeStatusValidity":0\},"maintenanceStatus":\{"mainBatteryStatus":\{"stateOfCharge":2,"voltage":128\},"tyreStatusDriver":169,"tyreStatusPassenger":169,"tyreStatusDriverRear":166,"tyreStatusPassengerRear":165,"tyrePreWarningDriver":0,"tyrePreWarningPassenger":0,"tyrePreWarningDriverRear":0,"tyrePreWarningPassengerRear":0,"distanceToService":4516,"serviceWarningStatus":0,"odometer":4844\},"runningStatus":\{"engineOilTemperature":80,"engineOilPressureWarning":0,"engineCoolantTemperature":165,"engineCoolantTemperatureValidity":0,"fuelLevel":65,"aveFuelConsumption":780,"avgSpeed":37\},"climateStatus":\{"winStatusDriver":2,"winStatusDriverValidity":1,"winStatusPassenger":2,"winStatusPassengerValidity":1,"winStatusDriverRear":0,"winStatusDriverRearValidity":0,"winStatusPassengerRear":0,"winStatusPassengerRearValidity":0,"sunroofOpenStatus":3,"ventilateStatus":0,"winPosDriver":0,"winPosPassenger":0,"winPosDriverRear":127,"winPosPassengerRear":127,"sunroofOpenStatusValidity":1,"exteriorTemp":114,"exteriorTempValidity":0,"preClimateActive":true,"airCleanSts":0,"winCloseReminder":0,"drvHeatSts":0,"passHeatingSts":0,"rlHeatingSts":0,"rrHeatingSts":0,"drvVentSts":0,"rrVentSts":0\},"drivingBehaviourStatus":\{"brakePedalDepressed":false,"brakePedalDepressedValidity":0,"engineSpeed":6416,"engineSpeedValidity":0,"transimissionGearPostion":4,"cruiseControlStatus":0\},"carConfig":\{"heatedFrontSeats":1,"parkingClimateLevel":1,"heatedSteeringWheel":1\}\},"temStatus":\{"vin":"LB37622Z3KX609880","powerSource":0,"backupBattery":\{"stateOfCharge":4,"stateOfHealth":0,"voltage":4,"chargeLevel":100\},"powerMode":0,"serviceProvisoned":true,"healthStatus":1,"diagnostics":\{"ecuID":"T-BOX","diagnosticResult":1,"dtcs":\[\{"dtc":"940111","status":\{"code":40\}\},\{"dtc":"940331","status":\{"code":46\}\},\{"dtc":"940514","status":\{"code":40\}\},\{"dtc":"940C13","status":\{"code":40\}\}\]\},"networkAccessStatus":\{"simInfo":\{"msisdn":"14598609817","imsi":"460018609709817","iccId":"89860118795912498172"\},"mobileNetwork":"4G"\},"serialNumber":"5120916600003466K4GA1059","completeECUPartNumbers":"6600003466","imei":"359598090119835","hwVersion":"HW\_REV\_1.2","swVersion":"NGT\_TBOX\_142B"\}\}\}\}\}

grok调试如下：

%\{TIMESTAMP\_ISO8601:access\_time\}\\s\*%\{LOGLEVEL:level\}\\s\*%\{UUID:uuid\}\\s\*%\{WORD:word\}\\s\*%\{HOSTNAME:hostname\}\\s\*\\\[%\{DATA:work\}\\\]\\s\*(?<api>(\[\\S+\]\*))\\s\*(?<TraceID>(\[\\S+\]\*))\\s\*%\{GREEDYDATA:message\_data\}

![857064-20190912141751028-682992990.png][]

列3：

192.168.125.138 - - \[12/Sep/2019:14:10:58 +0800\] "GET /backend/services/ticketRemind/query?cid=&msgType=1&pageSize=100&pageIndex=1&langCode=zh HTTP/1.1" 200 91

grok调试如下：

%\{IP:ip\}\\s\*%\{DATA:a\}\\s\*\\\[%\{HTTPDATE:access\_time\}\\\]\\s\*%\{DATA:b\}%\{WORD:method\}\\s\*%\{URIPATH:url\}%\{URIPARAM:param\}\\s\*%\{URIPROTO:uri\}%\{DATA:c\}%\{NUMBER:treaty\}%\{DATA:d\}\\s\*%\{NUMBER:status\}\\s\*%\{NUMBER:latency\_millis\}

![857064-20190927142808529-1332162949.png][]

转载于:https://www.cnblogs.com/zhangan/p/11395056.html

[857064-20190912141413660-655629477.png]: /images/20230808/f71419f46d4d4c458389f6145bede15f.png
[857064-20190912141751028-682992990.png]: /images/20230808/49bf4b453adb440e959d856a7d377d8d.png
[857064-20190927142808529-1332162949.png]: /images/20230808/13135e2be8da49d1983c4af073a86f30.png