graylog 索引模版处理

graylog 默认分词只支持对应几个固定的字段，如果需要自定义索引信息，就可以使用模版能力，默认包含了一个graylog-internal，order 为-1 但是我们可以扩展

默认索引信息

• 查询信息

GET <endpoint>/_template/graylog-internal?pretty'
效果

{
    "graylog-internal": {
        "order": -1,
        "index_patterns": [
            "graylog_*"
        ],
        "settings": {
            "index": {
                "analysis": {
                    "analyzer": {
                        "analyzer_keyword": {
                            "filter": "lowercase",
                            "tokenizer": "keyword"
                        }
                    }
                }
            }
        },
        "mappings": {
            "_source": {
                "enabled": true
            },
            "dynamic_templates": [
                {
                    "internal_fields": {
                        "mapping": {
                            "type": "keyword"
                        },
                        "match_mapping_type": "string",
                        "match": "gl2_*"
                    }
                },
                {
                    "store_generic": {
                        "mapping": {
                            "type": "keyword"
                        },
                        "match_mapping_type": "string"
                    }
                }
            ],
            "properties": {
                "gl2_processing_timestamp": {
                    "format": "uuuu-MM-dd HH:mm:ss.SSS",
                    "type": "date"
                },
                "gl2_accounted_message_size": {
                    "type": "long"
                },
                "gl2_receive_timestamp": {
                    "format": "uuuu-MM-dd HH:mm:ss.SSS",
                    "type": "date"
                },
                "full_message": {
                    "fielddata": false,
                    "analyzer": "standard",
                    "type": "text"
                },
                "streams": {
                    "type": "keyword"
                },
                "source": {
                    "fielddata": true,
                    "analyzer": "analyzer_keyword",
                    "type": "text"
                },
                "message": {
                    "fielddata": false,
                    "analyzer": "standard",
                    "type": "text"
                },
                "timestamp": {
                    "format": "uuuu-MM-dd HH:mm:ss.SSS",
                    "type": "date"
                }
            }
        },
        "aliases": {}
    }
}

}

调整

• 模版内容

{
  "template": "graylog_*",
  "index_patterns": ["*"],
  "mappings": {
    "properties": {
      "http_method": {
        "type": "keyword"
      },
      "http_response_code": {
        "type": "long"
      },
      "ingest_time": {
        "type": "date",
        "format": "strict_date_time"
      },
      "took_ms": {
        "type": "long"
      },
      "response_body": {
        "type": "text"
      },
      "request_body": {
        "type": "text"
      },
      "request": {
        "type": "text"
      },
      "http_user_agent": {
        "type": "text"
      }
    }
  }
}
配置
PUT /_template/graylog-custom-mapping?pretty
查看效果
GET /_template/graylog-custom-mapping?pretty
内容

{
  "graylog-custom-mapping": {
    "order": 0,
    "index_patterns": [
      "*"
    ],
    "settings": {},
    "mappings": {
      "properties": {
        "request": {
          "type": "text"
        },
        "http_method": {
          "type": "keyword"
        },
        "ingest_time": {
          "format": "strict_date_time",
          "type": "date"
        },
        "request_body": {
          "type": "text"
        },
        "took_ms": {
          "type": "long"
        },
        "response_body": {
          "type": "text"
        },
        "http_response_code": {
          "type": "long"
        },
        "http_user_agent": {
          "type": "text"
        }
      }
    },
    "aliases": {}
  }
}

}

代码处理

graylog2-server/src/main/java/org/graylog2/indexer/indices/Indices.java

• Indices.java

public void ensureIndexTemplate(IndexSet indexSet) {
      final IndexSetConfig indexSetConfig = indexSet.getConfig();
      final String templateName = indexSetConfig.indexTemplateName();
      try {
          final Map<String, Object> template = buildTemplate(indexSet, indexSetConfig);
          if (indicesAdapter.ensureIndexTemplate(templateName, template)) {
              LOG.info("Successfully ensured index template {}", templateName);
          } else {
              LOG.warn("Failed to create index template {}", templateName);
          }
      } catch (IgnoreIndexTemplate e) {
          LOG.warn(e.getMessage());
          if (e.isFailOnMissingTemplate() && !indicesAdapter.indexTemplateExists(templateName)) {
              throw new IndexTemplateNotFoundException(f("No index template with name '%s' (type - '%s') found in Elasticsearch",
                      templateName, indexSetConfig.indexTemplateType().orElse(null)));
          }
      }
  }
不同es 适配
比如es7 IndicesAdapterES7.java
 
@Override
  public boolean ensureIndexTemplate(String templateName, Map<String, Object> template) {
      final PutIndexTemplateRequest request = new PutIndexTemplateRequest(templateName)
              .source(template);
 
      final AcknowledgedResponse result = client.execute((c, requestOptions) -> c.indices().putTemplate(request, requestOptions),
              "Unable to create index template " + templateName);
 
      return result.isAcknowledged();
  }

}

具体内部处理实际上是基于了sysjob，相关job 如下

es 索引模型

• 写路径

• 读路径

说明

graylog 对于es 索引的管理还是比较方便的，充分利用了es 的能力，实现了比较强大的日志检索

参考资料

​​https://docs.graylog.org/docs/elasticsearch​​​
​​​https://docs.graylog.org/docs/index-model​​​
​​​https://docs.graylog.org/docs/query-language​​​
​​​https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-templates.html​​​
​​​https://github.com/Graylog2/graylog2-server/blob/626be1f0d80506705b5ba41fbea33c2ec0164bc0/graylog2-server/src/main/java/org/graylog2/indexer/indices/Indices.java​​​
​​​https://github.com/Graylog2/graylog2-server/blob/626be1f0d80506705b5ba41fbea33c2ec0164bc0/graylog2-server/src/main/java/org/graylog2/indexer/indices/IndicesAdapter.java​​