• 周五. 10月 7th, 2022

5G编程聚合网

5G时代下一个聚合的编程学习网

热门标签

Elasticsearch基本操作RestApi

admin

11月 28, 2021

Elasticsearch基础Api:

大家一定是使用过关系型数据库的,那么我们现在从关系型数据库的概念扩展讲解一下ES对应关系型数据库的内容

对应关系表
  Elasticsearch                =>     MySQL

  索引库(indices)        =>   Database 数据库
  类型(type)                   =>  Table 数据表
  文档(Document)       =>   Row 行
  域字段(Field)            =>  Columns 列
  映射配置(mappings) =>  每个列的约束(类型、长度)

知识点思维导图

基础增删改查

语法

(1) 创建/全量替换

记录不存在就是创建, 否则是全量替换.

PUT /index/type/id  
{
    "属性名" : "value"
    ...
}

(2) 创建/修改创建

//如果不传id, 则系统自动生成一个UUID
  POST /index/type/
  {
      "属性名":修改值
  }
  或者
  POST /index/type/id
  {
      "属性名":修改值
  }

修改

//没有带上的属性会被清除
POST /index/type/id
{
    "属性名":修改值
}

(3) 查询

GET /index/type/id

(4) 删除

只是逻辑删除, 将其标记为delete, 当数据越来越多时, ES会自动物理删除

DELETE /index/type/id

实例

(1) 新增

PUT /company/employee/1
{
  "age":25,
  "salary":"20k",
  "skill":["java","mysql","es"]
}

或者

POST /company/employee
{
  "age":30,
  "salary":"30k",
  "skill":["python","redis","hadoop"]
}

POST /company/employee/2
{ "age":22, "salary":"10k", "skill":["python","java"] }

POST 命令新增数据时, 如果不传id, 则系统自动生成一个UUID.

(2) 查询

GET /company/employee/1

(3) 修改

POST /company/employee/1
{
  "age":40,
  "salary":"100k"
}

PUT /company/employee/1
{
  "age":18,
  "salary":"50K",
  "skill":["c++"]
}

PUT 是全量替换, 且是幂等操作. 上面的POST修改, skill属性被清除了, 属性现在只有age和salary.

幂等操作: GET, PUT, DELETE都是幂等操作, 执行多次与执行一次结果一样.

(4) 删除

DELETE /company/employee/1

Partial Update

部分更新.

(1) 语法

post /index/type/id/_update 
{
   "doc": {
      "属性名" : "value"
      ...
   }
}

2) 与全量替换和普通修改的区别

在应用程序中, 全量替换是先从ES中查询出记录, 然后在修改, 一个属性都不能少. 而部分更新就不用先从ES查询记录了, 可以直接修改, 且不用所有属性都列出.

部分更新如果只包含部分属性, 那么其他没有被包含的属性仍然存在, 但普通修改其他没有被包含的属性就直接清除了.

(3) 实例

a. 先添加一条数据

PUT /school/student/1
{
  "name": "alice",
  "age": 17
}

b. 测试部分更新

//部分更新
POST /school/student/1/_update
{
  "doc": {
    "name" : "Tom"
  }
}

//查询
GET /school/student/1

//结果
{
  "_index": "school",
  "_type": "student",
  "_id": "1",
  "_version": 7,
  "found": true,
  "_source": {
    "name": "Tom",
    "age": 17
  }
}

c. 测试普通修改

//普通修改
POST /school/student/1
{
  "name" : "Linda"
}

//查询
GET school/student/1

//结果
{
  "_index": "school",
  "_type": "student",
  "_id": "1",
  "_version": 8,
  "found": true,
  "_source": {
    "name": "Linda"
  }
}

Bulk Operation

批量查询和批量增删改的语法不同, 所以分开来介绍.

批量查询

GET /_mget
{
  "docs" : [
    {
      "_index" : "company",
      "_type" : "employee",
      "_id" : 1
    },
     {
      "_index" : "company",
      "_type" : "employee",
      "_id" : 2
    }
  ]
}

GET /company/_mget
{
  "docs" : [
    {
      "_type" : "employee",
      "_id" : 1
    },
     {
      "_type" : "employee",
      "_id" : 2
    }
  ]
}

GET /company/employee/_mget
{
  "ids" : [1,2]
}

批量增删改

(1) 语法

每一个操作要两个json串,语法如下:

POST /index/type/_bulk
{"action": {"metadata"}}
{"data"}

ndex和type可以放入metadata中. 每个json串不能换行, 只能放一行. 同时一个json串和一个json串之间, 必须有一个换行.

(2) action类型

  delete  删除.

  create  强制创建. PUT /index/type/id/_create

  index  创建或替换.

  update  属于部分更新.

(3) 实例

POST /_bulk
{"delete" : {"_index":"company", "_type":"employee","_id":"1"}}
{
"create" :{"_index":"company","_type":"employee","_id":"2"}} {"name":"tyshawn", "age":18}
{
"index":{"_index":"company","_type":"employee","_id":"3"}} {"name":"lee", "age":24}
{
"update":{"_index":"company","_type":"employee","_id":"2"}} {"doc":{"age":30}}

搜索

添加搜索实例数据.

PUT /website/article/1
{
  "post_date": "2017-01-01",
  "title": "my first article",
  "content": "this is my first article in this website",
  "author_id": 11401,
  "tags": [
      "java",
      "c"
    ]
}

PUT /website/article/2
{
  "post_date": "2017-01-02",
  "title": "my second article",
  "content": "this is my second article in this website",
  "author_id": 11402,
  "tags": [
      "redis",
      "linux"
    ]
}

PUT /website/article/3
{
  "post_date": "2017-01-03",
  "title": "my third article",
  "content": "this is my third article in this website",
  "author_id": 11403,
  "tags": [
      "elaticsearch",
      "kafka"
    ]
}

 mapping结构如下:

GET /website/_mapping/article

{
  "website": {
    "mappings": {
      "article": {
        "properties": {
          "author_id": {
            "type": "long"
          },
          "content": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "post_date": {
            "type": "date"
          },
          "tags": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "title": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    }
  }
}

当type = text时, ES默认会设置两个field, 一个分词, 一个不分词. 如content会分词, 而content.keyword不分词, content.keyword最多保留256个字符.

通用语法

(1) 搜索所有index数据.

GET /_search

(2) 搜索指定index, type下的数据(index和type可以有多个)

GET /index1/_search
GET /index1,index2/_search
GET /index1/type1/_search
GET /index1/type1,type2/_search
GET /index1,index2/type1,type2/_search

(3) 搜索所有index下的指定type的数据.

GET /_all/employee,product/_search

下面再介绍Elasticsearch搜索的条件语法.

query string search

这个查询就类似于HTTP里的GET请求, 参数放在URL上.

(1) 语法

GET /index/type/_search?q=属性名:属性值
GET /index/type/_search?q=+属性名:属性值
GET /index/type/_search?q=-属性名:属性值
也可以省略属性名, 直接q=属性值
GET /index/type/_search?q=属性值

(2) + 和 – 的区别

默认是+, 指必须包含该字段, – 指不包含该字段.

(3) 实例

GET /website/article/_search?q=author_id:11403
GET /website/article/_search?q=-author_id:11403
GET /website/article/_search?q=11403

(4) _all metadata的原理

GET /index/type/_search?q=属性值.

这个语句是从所有属性中搜索包含指定的关键字的数据. 那么ES在搜索时是遍历所有document中的每一个field吗? 不是的, 我们在插入一条document时, ES会自动将多个field的值, 全部用字符串的方式串联起来, 变成一个长的字符串(以空格作为分隔符),作为_all field的值,同时进行分词建立倒排索引. 如果在搜索时没有指定属性名, 就会默认搜索_all field. (生产环境不使用)

query DSL

DSL, Domain Specified Language,特定领域的语言. 这个查询就类似于HTTP里的POST请求, 参数放在body中.

实例: 从website索引中查询所有文章

GET /website/article/_search
{
  "query": {
    "match_all": {}
  }
}

full-text search

全文搜索.

(1) 基础使用

a. 搜索标题中包含first或second的文章

GET /website/article/_search
{
  "query": {
    "match": {
      "title": "first second"
    }
  }
}

或者

GET /website/article/_search
{
  "query": {
    "match": {
      "title": {
        "query": "first second"
        , "operator": "or"
      }
    }
  }
}

或者

GET /website/article/_search
{
  "query": {
    "bool": {
      "should": [
        {"match": {
          "title":"first"
        }},
        {"match": {
          "title":"second"
        }}
      ]
    }
  }
}

b. 搜索标题中包含first和second的文章

GET /website/article/_search
{
  "query": {
    "match": {
      "title": {
        "query": "first second"
        , "operator": "and"
      }
    }
  }
}

或者

GET /website/article/_search
{
  "query": {
    "bool": {
      "must": [
        {"match": {
          "title": "first"
        }},
        {"match": {
          "title": "second"
        }}
      ]
    }
  }
}

c. 搜索标题中至少包含first, second, third, fourth中三个单词的文章.

GET /website/article/_search
{
  "query": {
    "match": {
      "title": {
        "query": "first second third fourth",
        "minimum_should_match": "75%"
      }
    }
  }
}

或者

GET /website/article/_search
{
  "query": {
    "bool": {
      "should": [
        {"match": {
          "title": "first"
        }},
        {"match": {
          "title": "second"
        }},
        {"match": {
          "title": "third"
        }},
        {"match": {
          "title": "fourth"
        }}
      ],
      "minimum_number_should_match": 3
    }
  }
}

d. 从website索引中查询, 标题必须包含elasticsearch,内容可以包含elasticsearch也可以不包含,作者id必须不为111.

GET /website/article/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "title":"elasticsearch"
          }
        }
      ],
      "should": [
        {
          "match": {
            "content":"elasticsearch"
          }
        }
      ], 
      "must_not": [
        {
          "match": {
             "author_id":"111"
          }
        }
      ]
    }
  }
}

e. 从website索引中查询, 标题包含first, 同时按作者id降序排序

GET /website/article/_search
{
  "query": {
    "match": {
      "title": "first"
    }
  },
  "sort": [
    {
      "author_id": {
        "order": "desc"
      }
    }
  ]
}

f. 从website索引中分页查询,总共3篇文章,假设每页就显示1篇文章,现在显示第2页

GET /website/article/_search
{
  "query": {
    "match_all": {}
  },
  "from": 1,
  "size": 1
}

g. 从website索引中查询所有文章, 只显示post_date, title两个属性.

GET /website/article/_search
{
  "query": {
    "match_all": {}
  },
  "_source": ["post_date","title"]
}

h. 搜索标题中包含 article 的文章, 如果标题中包含first或second就优先搜索出来, 同时, 如果一个文章标题包含first article, 另一个文章标题包含second article, 包含first article的文章要优先搜索出来.

//通过加权重来处理, 默认权重为1
GET /website/article/_search
{
  "query": {
    "bool": {
      "must": [
        {"match": {
          "title": "article"
        }}
      ],
      "should": [
        {"match": {
          "title": {
            "query": "first",
            "boost" : 3
          }
        }},
        {"match": {
          "title": {
            "query": "second",
            "boost": 2
          }
        }}
      ]
    }
  }
}

(2) multi_match

multi_match 用于查询词匹配多个属性. 这里涉及到几种匹配策略:

  best-fields

    doc的某个属性匹配尽可能多的关键词, 那么这个doc会优先返回.
  most-fields
    某个关键词匹配doc尽可能多的属性, 那么这个doc会优先返回.
  cross_fields
    跨越多个field搜索一个关键词.

best-fields和most-fields的区别:
比如, doc1的field1匹配的三个关键词, doc2的field1, field2都匹配上了同一个关键词. 如果是best-fields策略, 则doc1的相关度分数要更高, 如果是most-fields策略, 则doc2的相关度分数要更高.

实例:

a. 使用best_fields策略, 从title和content中搜索”my third article”.

GET /website/article/_search
{
  "query": {
    "multi_match": {
      "query": "my third article",
      "type": "best_fields", 
      "fields": ["title","content"]
    }
  }
}

b. 从title和content中搜索”my third article”, 且这三个单词要连在一起.

GET /website/article/_search
{
  "query": {
    "multi_match": {
      "query": "my third article",
      "type": "cross_fields",
      "operator": "and",
      "fields": ["title","content"]
    }
  }
}

3) dis_max

在我们进行多个条件的全文搜索时, 最后的计算出的相关度分数是根据多个条件的匹配分数综合而来的, 比如score = (score1 + score2) / 2, 如果我们想让最终的相关度分数等于多个条件匹配分数中的最大值, 即score = max(score1, score2), 则可以使用dis_max.

实例: 搜索title或content中包含first或article的文章

GET /website/article/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "match": {
            "title": "first article"
          }
        },
        {
          "match": {
            "content": "first article"
          }
        }
      ]
    }
  }
}

使用dis_max:

GET /website/article/_search
{
  "query": {
    "dis_max": {
      "queries": [
          {
          "match": {
            "title": "first article"
              }
          },
          {
            "match": {
              "content": "first article"
            }
          }
        ]
    }
  }
}

(4) tie_breaker

tie_breaker是与dis_max配套使用的. dis_max只取分数最大的那个条件的分数, 完全不考虑其他条件的分数, 但如果在某些场景下也需要考虑其他条件的分数呢? 我们可以指定一个系数值tie_breaker, 将其他条件的分数乘以tie_breaker, 然后和最大分数综合起来计算最终得分.

tie_breaker的取值为 0 ~ 1之间.

a. 将上面的实例优化下:

GET /website/article/_search
{
  "query": {
    "dis_max": {
      "tie_breaker": 0.7, 
      "queries": [
          {
          "match": {
            "title": "first article"
              }
          },
          {
            "match": {
              "content": "first article"
            }
          }
        ]
    }
  }
}

b. 继续优化. 如果搜索词包含多个关键字, 我们要求至少匹配多个关键词, 且多个条件的权重不同.

GET /website/article/_search
{
  "query": {
    "dis_max": {
      "tie_breaker": 0.7, 
      "queries": [
          {
          "match": {
            "title": {
              "query": "first article hello world",
              "minimum_should_match": "50%",
              "boost": 2
              }
            }
          },
          {
            "match": {
              "content": {
                "query": "first article is my hero",
                "minimum_should_match": "20%",
                "boost": 1
              }
            }
          }
        ]
    }
  }
}

phrase search

(1) 短语搜索, 与全文搜索有什么区别呢?

全文搜索会将”查询词”拆解开来, 去倒排索引中一一匹配, 只要能匹配上任意一个拆解后的关键词, 就可以作为结果返回. 而短语搜索在全文搜索的基础上, 要求关键词必须相邻. (注意短语搜索的”查询词”也是会被分词的)

GET /website/article/_search
{
  "query": {
    "match": {
      "title": "first article"
    }
  }
}
//三条记录都会搜索出来

GET /website/article/_search
{
  "query": {
    "match_phrase": {
      "title": "first article"
    }
  }
}
//只有一条记录

(2) 原理:

短语搜索的原理实际上是相邻匹配(proximity match). Lucene建立的倒排索引结构为: 关键词 -> 文档号, 在文档中的位置, 在文档出现的频率等, 当一个”查询词”包含多个关键词时, Lucene先通过关键词找到对应的文档号, 判断多个关键词所在的文档号是否相同, 然后再判断在文档中的位置是否相邻.

(3) 实例

短语搜索默认是搜索相邻的关键词, 但也可以搜索间隔几个位置的关键词. 间隔越小相关度分数越高.

a. 从content中搜索”first website”, first和website必须在同一个doc中, 且间隔不能超过10.

ET /website/article/_search
{
  "query": {
    "match_phrase": {
      "content": {
        "query": "first website",
        "slop": 10
      }
    }
  }
}

b. 全文搜索和短语搜索配合使用. 从content中搜索”first website”, 在优先满足召回率的前提下, 尽可能提高精准度.

召回率: 从n个doc中搜索, 有多少个doc返回.
精准度: 让两个关键词间隔越小的doc相关度分数越高.
GET /website/article/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "content": "first website"
          }
        }
      ],
      "should": [
        {
          "match_phrase": {
            "content": {
              "query": "first website",
              "slop": 10
            }
          }
        }
      ]
    }
  }
}

(4) 优化短语搜索: rescore

短语搜索的性能要比全文搜索的性能低10倍以上, 所以一般我们要用短语搜索时都会配合全文搜索使用. 先通过全文搜索出匹配的doc, 然后对相关度分数最高的前n条doc进行rescore短语搜索. (这里只能用于分页搜索)

GET /website/article/_search
{
  "query": {
    "match": {
      "content": "first website"
    }
  },
  "rescore": {
    "window_size": 20,
    "query": {
      "rescore_query": {
        "match_phrase": {
          "content": {
            "query": "first website",
            "slop": 10
          }
        }
      }
    }
  }
}

(5) match_phrase_prefix

匹配短语的前缀, 用于做搜索推荐. 比如我们在百度输入一个关键词, 立马就会推荐一系列查询词, 这个就是搜索推荐.

这个功能不推荐使用, 因为性能太差, 我们一般通过ngram分词机制来实现搜索推荐.
GET /website/article/_search
{
  "query": {
    "match_phrase_prefix": {
      "title": "my sec"
    }
  }
}

fuzzy search

模糊搜索, 自动将拼写错误的搜索文本进行纠正, 然后去匹配索引中的数据.

(1) 语法一

GET /website/article/_search
{
  "query": {
    "fuzzy": {
      "title.keyword": {
        "value": "my wecond article",
        "fuzziness": 2
      }
    }
  }
}

fuzziness代表最多纠正多少个字母, 默认为2. 搜索文本不会被分词.

(2) 语法二

GET /website/article/_search
{
  "query": {
    "match": {
      "title": {
        "query": "my wecond article",
        "fuzziness": "auto",
        "operator": "and"
      }
    }
  }
}

fuzziness可以给定个数, 也可以设置为auto.

term search

term查询, 是一种结构化查询, “查询词”不会被分词, 结果要么存在要么不存在, 不关心结果的score相关度. 如果查询text属性, 需要改为查询filed.keyword.

(1) 实例

和短语搜索对比一下可以更好的理解:

1. phrase搜索
GET /website/article/_search
{
  "query": {
    "match_phrase": {
      "title": "my first article"
    }
  }
}
//存在一条结果

2. term搜索text
GET /website/article/_search
{
  "query": {
    "term": {
      "title": {
        "value": "my first article"
      }
    }
  }
}
//不存在结果, 原因是词语查询的value值不会被分词, 也就是直接查询"my first article".

3. term搜索keyword
GET /website/article/_search
{
  "query": {
    "term": {
      "title.keyword": {
        "value": "my first article"
      }
    }
  }
}
//存在一条结果, filed.keyword属性不分词.

2) 常用语法

为了提高效率, term搜索一般与filter和constant_score联用. constant_score 以固定的评分来执行查询(默认为1), 而filter不计算score相关度, 因此执行效率非常高.

GET /website/article/_search
{
  "query": {
    "constant_score": {
      "filter": {
        "term": {
          "post_date": "2017-01-03"
        }
      }
    }
  }
}

query filter

(1) 语法

query filter 用于过滤数据, 不参与score相关度计算, 效率很高. 适用于范围查询以及不计算相关度score的精确查询(filter + term)

(2) 实例

a. 从website索引中查询, 作者id必须大于等于11402,同时发表时间必须是2017-01-02.

GET /website/article/_search
{
  "query": {
    "constant_score": {
      "filter": {
        "bool": {
          "must": [
              {
                "term": {
                  "post_date": "2017-01-02"
                }
              },
              {
                "range": {
                  "author_id": {
                     "gte": 11402
                  }
                }
              }
            ]
        }
      }
    }
  }
}

b. 搜索发布日期为2017-01-01, 或者文章标题为”my first article”的帖子, 同时要求文章的发布日期绝对不为2017-01-02.

GET /website/article/_search
{
  "query": {
    "constant_score": {
      "filter": {
        "bool": {
          "should": [
            {
              "term":{
                "post_date": "2017-01-01"
              }
            },
            {
              "term":{
                "title.keyword": "my first article"
              }
            }
            ],
            "must_not": {
              "term": {
                "post_date": "2017-01-02"
              }
            }
        }
      }
    }
  }
}

c. 搜索文章标题为”my first article”, 或者是文章标题为”my second article”, 而且发布日期为”2017-01-01″的文章.

GET /website/article/_search
{
  "query": {
    "constant_score": {
      "filter": {
        "bool": {
          "should": [
            {
              "term":{
                "title.keyword": "my first article"
              }
            },
            {
              "bool": {
                "must": [
                    {
                       "term":{
                         "title.keyword": "my second article"
                       }
                    },
                    {
                      "term":{
                         "post_date": "2017-01-01"
                       }
                    }
                  ]
              }
            }
            ]
        }
      }
    }
  }
}

d. 搜索文章标题为”my first article”或”my second article”的文章

GET /website/article/_search
{
  "query": {
    "constant_score": {
      "filter": {
        "terms": {
          "title.keyword": [
            "my first article",
            "my second article"
          ]
        }
      }
    }
  }
}

e. 搜索tags中包含java的帖子.

GET /website/article/_search
{
  "query": {
    "constant_score": {
      "filter": {
        "terms": {
          "tags.keyword": [
            "java"
          ]
        }
      }
    }
  }
}

注意, 这里必须用terms, 因为term不支持数组.

d. 搜索tags中只包含java的帖子.

如果想搜索tags中只包含java的帖子, 就需要新增一个字段tags_count, 表示tags中有几个tag, 否则就无法搜索.

GET /website/article/_search
{
  "query": {
    "constant_score": {
      "filter": {
        "bool": {
          "must": [
              {
                "terms": {
                  "tags.keyword": [
                      "java"
                    ]
                }
              },
              {
                "term": {
                  "tags_count": "1"
                }
              }
            ]
        }
      }
    }
  }
}
对上面这几实例做个总结, should表示或, must表示且.

scroll search

scroll滚动搜索,可以先搜索一批数据,然后下次再搜索一批数据,以此类推,直到搜索出全部的数据来. 它的原理是每次查询时都生成一个游标scroll_id, 后续的查询根据这个游标去获取数据, 直到返回的hits字段为空. scroll_id相当于建立了一个历史快照, 在此之后的写操作不会影响到这个快照的结果, 也就意味着其不能用于实时查询.
滚动查询用来解决深度分页的问题, 就类似于sql语句: select * from comment where id > 1000 order by id asc limit 1000.

(1) 语法

查询时指定一个参数scroll, 代表scroll_id的有效期, 过期后scroll_id会被ES自动清除.

如果不需要特定的排序, 按照文档创建时间排序更高效.

scroll_id只能使用一次, 使用过后会被自动删除.

最后一次查询, hits为空时也会返回一个scroll_id, 我们需要手动删除来释放资源.

(2) 实例

a. 首次查询

GET /website/article/_search?scroll=1s
{
  "query": {
    "match_all": {}
  },
  "sort": [
    {
      "_doc": {
        "order": "asc"
      }
    }
  ],
  "size": 1
}

b. 后续查询

scroll_id只能使用一次.

GET /_search/scroll?scroll=1s&scroll_id=DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAATqFmRBQ3FwUFVrUUw2VHgyU2I5UWRMRlEAAAAAAAAE6xZkQUNxcFBVa1FMNlR4MlNiOVFkTEZRAAAAAAAABOwWZEFDcXBQVWtRTDZUeDJTYjlRZExGUQAAAAAAAATtFmRBQ3FwUFVrUUw2VHgyU2I5UWRMRlEAAAAAAAAE7hZkQUNxcFBVa1FMNlR4MlNiOVFkTEZR
{
}

或者

GET /_search/scroll
{
  "scroll":"1s", "scroll_id":"DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAATqFmRBQ3FwUFVrUUw2VHgyU2I5UWRMRlEAAAAAAAAE6xZkQUNxcFBVa1FMNlR4MlNiOVFkTEZRAAAAAAAABOwWZEFDcXBQVWtRTDZUeDJTYjlRZExGUQAAAAAAAATtFmRBQ3FwUFVrUUw2VHgyU2I5UWRMRlEAAAAAAAAE7hZkQUNxcFBVa1FMNlR4MlNiOVFkTEZR"
}

c. 删除指定scroll_id

DELETE /_search/scroll
{
  "scroll_id": "scroll_id=DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAAiXFmRBQ3FwUFVrUUw2VHgyU2I5UWRMRlEAAAAAAAAImBZkQUNxcFBVa1FMNlR4MlNiOVFkTEZRAAAAAAAACJkWZEFDcXBQVWtRTDZUeDJTYjlRZExGUQAAAAAAAAiaFmRBQ3FwUFVrUUw2VHgyU2I5UWRMRlEAAAAAAAAImxZkQUNxcFBVa1FMNlR4MlNiOVFkTEZR=="
}

d. 删除所有scroll_id

DELETE /_search/scroll/_all

聚合

聚合包括分组和统计, 其中分组操作包括term, histogram, date_histogram, filter. 统计操作包括count, avg, max, min, sum, cardinality, percentiles, percentile_ranks等.

注意: 注意: 聚合的属性不能被分词.

语法

GET /index/type/_search
{
  size: 0,
  "aggs": {
    "NAME": {
      "AGG_TYPE": {
         "field": "field_name"
       }
    }
  }
}

NAME为聚合操作的名称, 可以取一个有参考意义的名称. AGG_TYPE为分组或统计操作, 当进行分组操作时, 会自动生成一个doc_count, 统计了组内数据的数量. 默认按照doc_count降序排列.

size=0的原因是不需要搜索结果, 如果需要搜索结果, 则去除size=0.

实例

(1) 实例数据

新增电视机销售记录, 用于接下来的实例分析.

POST /televisions/sales/_bulk
{ "index": {}}
{ "price" : 1000, "color" : "红色", "brand" : "长虹", "sold_date" : "2016-10-28" }
{ "index": {}}
{ "price" : 2000, "color" : "红色", "brand" : "长虹", "sold_date" : "2016-11-05" }
{ "index": {}}
{ "price" : 3000, "color" : "绿色", "brand" : "小米", "sold_date" : "2016-05-18" }
{ "index": {}}
{ "price" : 1500, "color" : "蓝色", "brand" : "TCL", "sold_date" : "2016-07-02" }
{ "index": {}}
{ "price" : 1200, "color" : "绿色", "brand" : "TCL", "sold_date" : "2016-08-19" }
{ "index": {}}
{ "price" : 2000, "color" : "红色", "brand" : "长虹", "sold_date" : "2016-11-05" }
{ "index": {}}
{ "price" : 8000, "color" : "红色", "brand" : "三星", "sold_date" : "2017-01-01" }
{ "index": {}}
{ "price" : 2500, "color" : "蓝色", "brand" : "小米", "sold_date" : "2017-02-12" }

mapping结构如下:

GET /televisions/_mapping/sales
{
  "televisions": {
    "mappings": {
      "sales": {
        "properties": {
          "brand": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "color": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "price": {
            "type": "long"
          },
          "sold_date": {
            "type": "date"
          }
        }
      }
    }
  }
}

(2) 基础聚合实例

a. 统计哪种颜色的电视销量最高

GET /televisions/sales/_search
{
  "size": 0,
  "aggs": {
    "max_sales_color": {
      "terms": {
        "field": "color.keyword"
      }
    }
  }
}

分组后会计算出每组数据个数(doc_count), 默认按照doc_count降序显示.

b. 统计每种颜色电视的平均价格

GET /televisions/sales/_search
{
  "size": 0,
  "aggs": {
    "group_by_color": {
      "terms": {
        "field": "color.keyword"
      },
      "aggs": {
        "avg_price": {
          "avg": {
            "field": "price"
          }
        }
      }
    }
  }
}

c. 统计每种颜色电视的平均价格, 以及统计每种颜色下每个品牌的平均价格.

这里就涉及到嵌套分组了, 也叫做多层下钻分析.

GET /televisions/sales/_search
{
  "size": 0,
  "aggs": {
    "group_by_color": {
      "terms": {
        "field": "color.keyword"
      },
      "aggs": {
        "color_avg_price": {
          "avg": {
            "field": "price"
          }
        },
        "group_by_brand": {
          "terms": {
            "field": "brand.keyword"
          },
          "aggs": {
            "brand_avg_price": {
              "avg": {
                "field": "price"
              }
            }
          }
        }
      }
    }
  }
}

d. 统计每种颜色电视机的最大最小价格

GET /televisions/sales/_search
{
  "size": 0,
  "aggs": {
    "group_by_color": {
      "terms": {
        "field": "color.keyword"
      },
      "aggs": {
        "max_price": {
          "max": {
            "field": "price"
          }
        },
        "min_price": {
          "min": {
            "field": "price"
          }
        }
      }
    }
  }
}

e. 统计每种颜色电视机的总销售额

GET /televisions/sales/_search
{
  "size": 0,
  "aggs": {
    "group_by_color": {
      "terms": {
        "field": "color.keyword"
      },
      "aggs": {
        "sum_price": {
          "sum": {
            "field": "price"
          }
        }
      }
    }
  }
}

(2) 高级聚合实例

a. 按价格区间统计电视销量和销售额

GET /televisions/sales/_search
{
  "size": 0,
  "aggs": {
    "group_by_price_range": {
      "histogram": {
        "field": "price",
        "interval": 2000
      },
      "aggs": {
        "sum_price": {
          "sum": {
            "field": "price"
          }
        }
      }
    }
  }
}

b. 统计 2016-01-01 ~ 2017-12-31 范围内每个月的电视机销量.

GET /televisions/sales/_search
{
  "size": 0,
  "aggs": {
    "group_by_sold_date": {
      "date_histogram": {
        "field": "sold_date",
        "interval": "month",
        "format": "yyyy-MM-dd",
        "min_doc_count": 0,
        "extended_bounds": {
          "min": "2016-01-01",
          "max": "2017-12-31"
        }
      }
    }
  }
}

c. 统计 2016-01-01 ~ 2017-12-31 范围内每个季度的销售额以及该季度下每个品牌的销售额

GET /televisions/sales/_search
{
  "size": 0,
  "aggs": {
    "group_by_sold_date": {
      "date_histogram": {
        "field": "sold_date",
        "interval": "quarter",
        "format": "yyyy-MM-dd",
        "min_doc_count": 0,
        "extended_bounds": {
          "min": "2016-01-01",
          "max": "2017-12-31"
        }
      },
      "aggs": {
        "sum_price": {
          "sum": {
            "field": "price"
          }
        },
        "group_by_brand": {
          "terms": {
            "field": "brand.keyword"
          },
          "aggs": {
            "brand_sum_price": {
              "sum": {
                "field": "price"
              }
            }
          }
        }
      }
    }
  }
}

d. 统计每种颜色电视的销售额, 按照销售额升序排序

GET /televisions/sales/_search
{
  "size": 0, 
  "aggs": {
    "group_by_color": {
      "terms": {
        "field": "color.keyword",
        "order": {
          "sum_price": "asc"
        }
      },
      "aggs": {
        "sum_price": {
          "sum": {
            "field": "price"
          }
        }
      }
    }
  }
}

e. 统计每种颜色下的每个品牌电视机的总销售额, 并按这个销售额升序排序.

GET /televisions/sales/_search
{
  "size": 0, 
  "aggs": {
    "group_by_color": {
      "terms": {
        "field": "color.keyword"
      },
      "aggs": {
        "group_by_brand": {
          "terms": {
            "field": "brand.keyword",
            "order": {
              "color_brand_sum_price": "asc"
            }
          },
          "aggs": {
            "color_brand_sum_price": {
              "sum": {
                "field": "price"
              }
            }
          }
        }
      }
    }
  }
}

f. 统计每个月的电视销量, 并按品牌去重.

GET /televisions/sales/_search
{
  "size": 0, 
  "aggs": {
    "group_by_sold_date": {
      "date_histogram": {
        "field": "sold_date",
        "interval": "month",
        "format": "yyyy-MM-dd"
      },
      "aggs": {
        "distinct_brand": {
          "cardinality": {
            "field": "brand.keyword",
            "precision_threshold": 100 
          }
        }
      }
    }
  }
}

cardinality 去重采用的是近似估计的算法, 错误率在5%左右, 其中precision_threshold指定的值为100%准确去重的数量, 值设置的越大, 内存开销也就越大.

g. 统计50%, 90% 和 99%的电视的最大价格(一般用于统计api请求的最长延迟时间)

GET /televisions/sales/_search
{
  "size": 0, 
  "aggs": {
    "price_percentiles": {
      "percentiles": {
        "field": "price",
        "percents": [
          50,
          90,
          99
        ]
      }
    }
  }
}

h. 统计每个品牌的电视机的价格, 在1000以内, 2000以内, 3000以内, 4000以内的所占比例.

GET /televisions/sales/_search
{
  "size": 0,
  "aggs": {
    "group_by_brand": {
      "terms": {
        "field": "brand.keyword"
      },
      "aggs": {
        "price_percentile_ranks": {
          "percentile_ranks": {
            "field": "price",
            "values": [
              1000,
              2000,
              3000,
              4000
            ]
          }
        }
      }
    }
  }
}

3) 搜索+聚合

a. 统计指定品牌下(小米)每个颜色的销量

GET /televisions/sales/_search
{
  "size": 0, 
  "query": {
    "term": {
      "brand.keyword": {
        "value": "小米"
      }
    }
  }, 
  "aggs": {
    "group_by_color": {
      "terms": {
        "field": "color.keyword"
      }
    }
  }
}

 或者

GET /televisions/sales/_search
{
  "size": 0, 
  "query": {
    "constant_score": {
      "filter": {
        "term": {
          "brand.keyword": "小米"
        }
      }
    }
  }, 
  "aggs": {
    "group_by_color": {
      "terms": {
        "field": "color.keyword"
      }
    }
  }
}    

其实对于聚合来说, 因为不需要搜索结果, 可以直接用filter, 效率更高.

b. 统计单个品牌(长虹)与所有品牌销售额对比

GET /televisions/sales/_search
{
  "size": 0, 
  "query": {
    "constant_score": {
      "filter": {
        "term": {
          "brand.keyword": "长虹"
        }
      }
    }
  }, 
  "aggs": {
    "single_brand_sum_price": {
      "sum": {
        "field": "price"
      }
    },
    "all_brand": {
      "global": {},
      "aggs": {
        "all_brand_sum_price": {
          "sum": {
            "field": "price"
          }
        }
      }
    }
  }
}

global 表示将所有数据纳入聚合的scope,忽视前面的query过滤.

c. 统计指定品牌(长虹)最近一个月和最近半年的平均价格

GET /televisions/sales/_search
{
  "size": 0, 
  "query": {
    "term": {
      "brand.keyword": {
        "value": "长虹"
      }
    }
  }, 
  "aggs": {
    "recent_one_month": {
      "filter": {
        "range": {
          "sold_date": {
            "gte": "now-1M"
          }
        }
      },
      "aggs": {
        "recent_one_month_avg_price": {
          "avg": {
            "field": "price"
          }
        }
      }
    },
    "recent_six_month": {
      "filter": {
        "range": {
          "sold_date": {
            "gte": "now-6M"
          }
        }
      },
      "aggs": {
        "recent_six_month_avg_price": {
          "avg": {
            "field": "price"
          }
        }
      }
    }
  }
}

————————————————
版权声明:本文为CSDN博主「椰子Tyshawn」的原创文章,遵循CC 4.0 BY-SA版权协议,转载请附上原文出处链接及本声明。
原文链接:https://blog.csdn.net/litianxiang_kaola/article/details/102936324

发表回复

您的电子邮箱地址不会被公开。