本文以一个示例说明FlinkSQL如何针对上游乱序数据源设计保序任务，从而保证下游数据准确性。废话不多说，这里以交易数据场景为例.
-
数据表结构为：
create table tbl_order_source( order_id int comment '订单ID', shop_id int comment '书店ID', user_id int comment '用户ID', original_price double comment '原始交易额', create_time timestamp(3) comment '创建时间: yyyy-MM-dd HH:mm:ss', watermark for create_time as create_time - interval '0' second )with( 'connector' = 'kafka', 'topic' = 'tbl_order_source', 'properties.bootstrap.servers' = 'localhost:9092', 'properties.group.id' = 'testGroup', 'scan.startup.mode' = 'latest-offset', 'format' = 'json', 'json.fail-on-missing-field' = 'false', 'json.ignore-parse-errors' = 'true' ); -
乱序数据源如下：
{"order_id":"1","shop_id":"1","user_id":"1","original_price":"1","create_time":"2024-01-01 20:05:00"} {"order_id":"2","shop_id":"1","user_id":"2","original_price":"2","create_time":"2024-01-01 20:04:00"} {"order_id":"1","shop_id":"1","user_id":"1","original_price":"3","create_time":"2024-01-01 20:03:00"} {"order_id":"3","shop_id":"1","user_id":"3","original_price":"4","create_time":"2024-01-01 20:02:00"} {"order_id":"1","shop_id":"1","user_id":"1","original_price":"5","create_time":"2024-01-01 20:04:00"} -
我们针对乱序数据源消费首先设计保序任务，如下：
-- 保序数据中间结果 create table ods_order_source( order_id int comment '订单ID', shop_id int comment '书店ID', user_id int comment '用户ID', original_price double comment '订单金额', create_time timestamp(3) comment '创建时间: yyyy-MM-dd HH:mm:ss', watermark for create_time as create_time - interval '0' second, primary key (order_id) not enforced )with( 'connector' = 'upsert-kafka', 'topic' = 'ods_order_source', 'properties.bootstrap.servers' = 'localhost:9092', 'key.format' = 'json', 'key.json.ignore-parse-errors' = 'true', 'value.format' = 'json', 'value.json.fail-on-missing-field' = 'false' ); -- 源到保序结果ETL insert into ods_order_source select tmp.order_id, tmp.shop_id, tmp.user_id, tmp.original_price, tmp.create_time from ( select t.order_id, t.shop_id, t.user_id, t.original_price, t.create_time, row_number()over(partition by t.order_id order by t.create_time asc) as rn from tbl_order_source t ) tmp where tmp.rn = 1 ; -- 查询保序中间结果数据 select * from ods_order_source;针对数据源输入，保序任务输出为：
+I {"order_id":"1","shop_id":"1","user_id":"1","original_price":"1","create_time":"2024-01-01 20:05:00.000"} +I {"order_id":"2","shop_id":"1","user_id":"2","original_price":"2","create_time":"2024-01-01 20:04:00.000"} -U {"order_id":"1","shop_id":"1","user_id":"1","original_price":"1","create_time":"2024-01-01 20:05:00.000"} +U {"order_id":"1","shop_id":"1","user_id":"1","original_price":"3","create_time":"2024-01-01 20:03:00.000"} +I {"order_id":"3","shop_id":"1","user_id":"3","original_price":"4","create_time":"2024-01-01 20:02:00.000"} -
下面针对保序任务设计聚合SQL：
select t.shop_id as shop_id, to_date(cast(t.create_time as string)) as create_date, sum(t.original_price) as original_amt, sum(1) as order_num, count(distinct t.order_id) as order_cnt from ods_order_source t group by t.shop_id, to_date(cast(t.create_time as string)) ;测试结果：
+I {"order_id":1,"create_date":2024-01-01,"original_amt":1,"order_num":1,"order_cnt":1} -U {"order_id":1,"create_date":2024-01-01,"original_amt":1,"order_num":1,"order_cnt":1} +U {"order_id":1,"create_date":2024-01-01,"original_amt":3,"order_num":2,"order_cnt":2} -U {"order_id":1,"create_date":2024-01-01,"original_amt":3,"order_num":2,"order_cnt":2} +U {"order_id":1,"create_date":2024-01-01,"original_amt":2,"order_num":1,"order_cnt":1} -U {"order_id":1,"create_date":2024-01-01,"original_amt":2,"order_num":1,"order_cnt":1} +U {"order_id":1,"create_date":2024-01-01,"original_amt":5,"order_num":2,"order_cnt":2} -U {"order_id":1,"create_date":2024-01-01,"original_amt":5,"order_num":2,"order_cnt":2} +U {"order_id":1,"create_date":2024-01-01,"original_amt":9,"order_num":3,"order_cnt":3}可以从聚合SQL输出结果看出，最后数据为
{"order_id":1,"create_date":2024-01-01,"original_amt":9,"order_num":3,"order_cnt":3}，结果和源输入数据对应上。 -
下面同样的SQL针对未保序源数据做聚合，看下结果如何：
select t.shop_id as shop_id, to_date(cast(t.create_time as string)) as create_date, sum(t.original_price) as original_amt, sum(1) as order_num, count(distinct t.order_id) as order_cnt from tbl_order_source t group by t.shop_id, to_date(cast(t.create_time as string)) ;查看结果：
+I {"order_id":1,"create_date":2024-01-01,"original_amt":1,"order_num":1,"order_cnt":1} -U {"order_id":1,"create_date":2024-01-01,"original_amt":1,"order_num":1,"order_cnt":1} +U {"order_id":1,"create_date":2024-01-01,"original_amt":3,"order_num":2,"order_cnt":2} -U {"order_id":1,"create_date":2024-01-01,"original_amt":3,"order_num":2,"order_cnt":2} +U {"order_id":1,"create_date":2024-01-01,"original_amt":6,"order_num":3,"order_cnt":2} -U {"order_id":1,"create_date":2024-01-01,"original_amt":6,"order_num":3,"order_cnt":2} +U {"order_id":1,"create_date":2024-01-01,"original_amt":10,"order_num":4,"order_cnt":3} -U {"order_id":1,"create_date":2024-01-01,"original_amt":10,"order_num":4,"order_cnt":3} +U {"order_id":1,"create_date":2024-01-01,"original_amt":15,"order_num":4,"order_cnt":3}最后结果
{"order_id":1,"create_date":2024-01-01,"original_amt":15,"order_num":4,"order_cnt":3}错误。
针对测试结果，对于聚合SQL数据sink设置shop_id作为key，就可以保证下游数据结果的正确性。
