向导

• 1、ES cdc安装包
• 2、安装步骤
• 3、使用
• • 3.1 设置集群属性
  • 3.2 创建索引时，enable cdc并初始化相关属性
  • 3.3 索引级别属性说明
  • 3.4 集群级别配置属性
  • 3.5 测试
  • • 3.5.1 insert with POST
    • 3.5.2 insert with PUT
    • 3.5.3 bulk insert
    • 3.5.4 update
    • 3.5.5 upsert
    • 3.5.6 update by query and just update non-nested field
    • 3.5.7 update by query --update nested field
    • 3.5.8 update by query -- delete nested field
    • 3.5.9 simple delete
    • 3.5.9 delete by query
  • 3.6 变更cdc 配置 测试
  • 3.7 测试别名

1、ES cdc安装包

Elasticsearch V6.3.2 CDC 组件

2、安装步骤

集群每个节点都执行如下步骤：

1. 移除之前版本(如果之前安装过)
${elasticsearch_home}/bin/elasticsearch-plugin remove es-cdc2. 安装
${elasticsearch_home}/bin/elasticsearch-plugin install file:///${elasticsearch_cdc_dir}/es-cdc-v6-1.0-SNAPSHOT.zip3. 配置es的javax相关权限
${elasticsearch_home}/config/jvm.options 文件加入相关java配置
-Djava.security.policy=/your_elasticsearch_home/plugins/es-cdc/plugin-security.policy4. 重启 elasticsearch

3、使用

3.1 设置集群属性

PUT _cluster/settings
{"persistent": {"indices.cdc.request.timeout.ms": 30000,"indices.cdc.send.buffer.bytes": 131072,"indices.cdc.acks": "all","indices.cdc.compression.type": "none","indices.cdc.receive.buffer.bytes": 32768,"indices.cdc.batch.size": 16384,"indices.cdc.linger.ms": 1000,"indices.cdc.buffer.memory": 33554432,"indices.cdc.bootstrap.servers": "xxx:9092,yyy:9092","indices.cdc.max.request.size": 52428800,"indices.cdc.max.in.flight.requests.per.connection": 10,"indices.cdc.retry.backoff.ms": 100,"indices.cdc.retries": 100,"indices.cdc.max.block.ms": 86400000}
}

3.2 创建索引时，enable cdc并初始化相关属性

PUT /index
{"settings":{"number_of_shards":3,"number_of_replicas":1,"index.cdc.enabled":true,"index.cdc.topic":"cdc_test","index.cdc.pk.column":"doc_id","index.refresh_interval":"1s"},"mappings":{"test":{"properties":{"content":{"type":"text"},"doc_id":{"type":"integer"},"age":{"type":"integer"},"address":{"type":"keyword"},"keywords":{"type":"nested","properties":{"keyword":{"type":"keyword"},"frequency":{"type":"integer"}}}}}}
}

3.3 索引级别属性说明

3.4 集群级别配置属性

3.5 测试

3.5.1 insert with POST

请求信息：

POST /index/test/1
{"doc_id":1,"content":"peace and hope","age":12,"address":"bj","keywords":[{"keyword":"peace","frequency":10},{"keyword":"hope","frequency":5}]}

cdc结果：

{"op":1,"index":"index","content":{"address":"bj","keywords":[{"keyword":"peace","frequency":10},{"keyword":"hope","frequency":5}],"doc_id":1,"content":"peace and hope","age":12},"ts":1678870752189}

3.5.2 insert with PUT

请求信息：

PUT /index/test/2
{"doc_id": 2,"content":"peace and hope 2", "age": 12, "address": "bj", "keywords": [{"keyword": "peace", "frequency": 1}, {"keyword": "hope", "frequency": 1}]}

cdc结果：

{"op":1,"index":"index","content":{"address":"bj","keywords":[{"keyword":"peace","frequency":1},{"keyword":"hope","frequency":1}],"doc_id":2,"content":"peace and hope 2","age":12},"ts":1678871014444}

3.5.3 bulk insert

请求信息：

PUT /_bulk
{"index":{"_index":"index","_type" : "test","_id":"3"}}
{"doc_id": 3,"content":"peace and hope 3", "age": 13, "address": "bj", "keywords": [{"keyword": "peace", "frequency": 1}, {"keyword": "hope", "frequency": 1}]}
{"index":{"_index":"index","_type" : "test","_id":"4"}}
{"doc_id": 4,"content":"peace and hope 4", "age": 14, "address": "bj", "keywords": [{"keyword": "peace", "frequency": 1}, {"keyword": "hope", "frequency": 1}]}
{"index":{"_index":"index","_type" : "test","_id":"5"}}
{"doc_id": 5,"content":"peace and hope 5", "age": 15, "address": "bj", "keywords": [{"keyword": "peace", "frequency": 1}, {"keyword": "hope", "frequency": 1}]}

cdc结果：

{"op":1,"index":"index","content":{"address":"bj","keywords":[{"keyword":"peace","frequency":1},{"keyword":"hope","frequency":1}],"doc_id":3,"content":"peace and hope 3","age":13},"ts":1678871332123}
{"op":1,"index":"index","content":{"address":"bj","keywords":[{"keyword":"peace","frequency":1},{"keyword":"hope","frequency":1}],"doc_id":4,"content":"peace and hope 4","age":14},"ts":1678871332121}
{"op":1,"index":"index","content":{"address":"bj","keywords":[{"keyword":"peace","frequency":1},{"keyword":"hope","frequency":1}],"doc_id":5,"content":"peace and hope 5","age":15},"ts":1678871332137}

3.5.4 update

请求信息：

POST /index/test/1
{"doc":{"doc_id":1,"address":"sjz"}}

cdc结果：

{"op":2,"index":"index","content":{"doc":{"address":"sjz","doc_id":1}},"ts":1678871409770}

3.5.5 upsert

请求信息：

POST /index/test/2/_update
{"doc":{"doc_id":2,"address":"sjz"}}

cdc结果：

{"op":2,"index":"index","content":{"address":"sjz","keywords":[{"keyword":"peace","frequency":1},{"keyword":"hope","frequency":1}],"doc_id":2,"content":"peace and hope 2","age":12},"ts":1678871498864}

3.5.6 update by query and just update non-nested field

请求信息：

POST /index/test/_update_by_query
{"script":{"source":"ctx._source.age++","lang":"painless"},"query":{"term":{"doc_id":2}}}

cdc结果：

{"op":2,"index":"index","content":{"address":"sjz","keywords":[{"keyword":"peace","frequency":1},{"keyword":"hope","frequency":1}],"doc_id":2,"content":"peace and hope 2","age":13},"ts":1678871632362}

3.5.7 update by query --update nested field

请求信息：

POST /index/test/2/_update
{"script":{"source":"for(e in ctx._source.keywords){if (e.frequency == 1) {e.frequency = 10;}}"}}

cdc结果：

{"op":2,"index":"index","content":{"address":"sjz","keywords":[{"keyword":"peace","frequency":10},{"keyword":"hope","frequency":10}],"doc_id":2,"content":"peace and hope 2","age":13},"ts":1678871754106}

3.5.8 update by query – delete nested field

请求信息：

POST /index/test/2/_update
{"script":{"source":"ctx._source.keywords.removeIf(it -> it.frequency == 10);"}}

cdc结果：

{"op":2,"index":"index","content":{"address":"sjz","keywords":[],"doc_id":2,"content":"peace and hope 2","age":13},"ts":1678871832670}

3.5.9 simple delete

请求信息：

DELETE /index/test/1

cdc结果：

{"op":0,"index":"index","content":{"doc_id":"1"},"ts":1678871877738}

3.5.9 delete by query

请求信息：

POST /index/test/_delete_by_query
{"query":{"match":{"doc_id":2}}}

cdc结果：

{"op":0,"index":"index","content":{"doc_id":"2"},"ts":1629531944820}

cdc json的op值： delete：0，insert：1，update：2

3.6 变更cdc 配置 测试

PUT index/_settings
{"index.cdc.enabled":false
}PUT index/_settings
{"index.cdc.enabled":true,"index.cdc.exclude.columns": "age,content"
}POST /index/test/6/_create
{"doc_id": 6,"content":"peace and hope", "age": 12, "address": "bj", "keywords": [{"keyword": "peace", "frequency": 10}, {"keyword": "hope", "frequency": 5}]}生成的 cdc 消息，在cdc_test中有如下消息：
{"op":1,"index":"index","content":{"address":"bj","keywords":[{"keyword":"peace","frequency":10},{"keyword":"hope","frequency":5}],"doc_id":6},"ts":1678872090164}
已经排除了 content 和 age 的信息

3.7 测试别名

增加多个别名：POST /_aliases
{"actions": [{"add": {"index": "index","alias": "index_dev"}},{"add": {"index": "index","alias": "index_test"}},{"add": {"index": "index","alias": "index_pro"}}]
}

修改dev别名

PUT index/_settings
{"index.cdc.alias": "index_dev"
}

再次插入，请求信息：

PUT /index/test/8
{"doc_id": 8,"content":"peace and hope 2", "age": 12, "address": "bj", "keywords": [{"keyword": "peace", "frequency": 1}, {"keyword": "hope", "frequency": 1}]}

cdc 结果：

{"op":1,"index":"index_dev","content":{"address":"bj","keywords":[{"keyword":"peace","frequency":1},{"keyword":"hope","frequency":1}],"doc_id":8,"content":"peace and hope 2","age":12},"ts":1678875224160}

关闭cdc：

PUT index/_settings
{"index.cdc.enabled":false
}

修改cdc别名：

PUT index/_settings
{"index.cdc.alias": "index_pro"
}

插入数据，请求信息：

PUT /index/test/11
{"doc_id": 11,"content":"peace and hope 2", "age": 12, "address": "bj", "keywords": [{"keyword": "peace", "frequency": 1}, {"keyword": "hope", "frequency": 1}]}

cdc结果：

{"op":1,"index":"index_pro","content":{"address":"bj","keywords":[{"keyword":"peace","frequency":1},{"keyword":"hope","frequency":1}],"doc_id":11,"content":"peace and hope 2","age":12},"ts":1678875812376}

在修改CDC配置时，需要先停止CDC，再修改配置，之后再启用CDC！！！