graylog 索引数据查询处理简单说明

参考请求地址

从ui 看到的,类似一个job 的执行,支持异步以及同步数据查询处理，还包含了相对时间、绝对时间查询的

/api/views/search/<id>/execute
/api/views/search
/api/views/search/metadata

查询适配

• 参考实现
  主要是对于es6 以及es7 不同版本的兼容处理，对于查询支持了scroll 模式以及普通的search 模式

• 查询处理 如下图，对于查询可能会包含多个索引

public SearchResult search(Set<String> indices, Set<IndexRange> indexRanges, SearchesConfig config) {
    final SearchSourceBuilder searchSourceBuilder = searchRequestFactory.create(config);
 
    if (indexRanges.isEmpty()) {
        return SearchResult.empty(config.query(), searchSourceBuilder.toString());
    }
    // 构造多索引的查询
    final SearchRequest searchRequest = new SearchRequest(indices.toArray(new String[0]))
            .source(searchSourceBuilder);
    final SearchResponse searchResult = client.search(searchRequest, "Unable to perform search query");
 
    final List<ResultMessage> resultMessages = extractResultMessages(searchResult);
    final long totalResults = searchResult.getHits().getTotalHits().value;
    final long tookMs = searchResult.getTook().getMillis();
    final String builtQuery = searchSourceBuilder.toString();
    return new SearchResult(resultMessages, totalResults, indexRanges, config.query(), builtQuery, tookMs);
}

}

查询任务执行

实际上是基于了多线程支持模式，graylog 同时提供了同步以及异步模式，具体执行都是SearchExecutor 提供的
具体会使用QueryEngine提供的查询能力，同时一次查询可能会处理多个stream 所以需要遍历查询

public SearchResult search(Set<String> indices, Set<IndexRange> indexRanges, SearchesConfig config) {
    final SearchSourceBuilder searchSourceBuilder = searchRequestFactory.create(config);
 
    if (indexRanges.isEmpty()) {
        return SearchResult.empty(config.query(), searchSourceBuilder.toString());
    }
    // 构造多索引的查询
    final SearchRequest searchRequest = new SearchRequest(indices.toArray(new String[0]))
            .source(searchSourceBuilder);
    final SearchResponse searchResult = client.search(searchRequest, "Unable to perform search query");
 
    final List<ResultMessage> resultMessages = extractResultMessages(searchResult);
    final long totalResults = searchResult.getHits().getTotalHits().value;
    final long tookMs = searchResult.getTook().getMillis();
    final String builtQuery = searchSourceBuilder.toString();
    return new SearchResult(resultMessages, totalResults, indexRanges, config.query(), builtQuery, tookMs);
}

}

说明

尽管scroll 是一个请求大量数据的模式，但是es 并不是很推荐了，更推荐的是使用search-after，同时需要基于pit 进行更高效的进行数据查询处理

参考资料

​​https://docs.graylog.org/docs/elasticsearch​​​
​​​https://docs.graylog.org/docs/index-model​​​
​​​https://docs.graylog.org/docs/rest-api​​​
​​​https://github.com/Graylog2/graylog2-server/blob/626be1f0d80506705b5ba41fbea33c2ec0164bc0/graylog2-server/src/main/java/org/graylog2/indexer/searches/SearchesAdapter.java​​​
​​​https://github.com/Graylog2/graylog2-server/blob/626be1f0d80506705b5ba41fbea33c2ec0164bc0/graylog2-server/src/main/java/org/graylog2/indexer/searches/Searches.java​​​
​​​https://www.elastic.co/guide/en/elasticsearch/reference/7.17/paginate-search-results.html#search-after​​​
​​​https://www.elastic.co/guide/en/elasticsearch/reference/7.17/point-in-time-api.html​​​
​​​https://github.com/Graylog2/graylog2-server/blob/626be1f0d80506705b5ba41fbea33c2ec0164bc0/graylog2-server/src/main/java/org/graylog/plugins/views/search/engine/QueryEngine.java​​​
​​​https://github.com/Graylog2/graylog2-server/blob/626be1f0d80506705b5ba41fbea33c2ec0164bc0/graylog2-server/src/main/java/org/graylog/plugins/views/search/rest/SearchResource.java​​