云计算之路-阿里云上:读取缓存时的“黑色0.1秒”
2014-05-12 来自:假扮兔子¤独立 0 人回应

进入正题。。。

自从使用阿里云以来,我们一直被一个问题困扰——日志中时不时会记录读取缓存超过100ms的情况。

一开始我们用的是云服务器跑memcached作为缓存服务,当时以为是虚拟机跑memcached性能跟不上,于是就期待阿里云的缓存服务能解决这个问题。

可是等到阿里云推出OCS(开放缓存服务),换上去之后,读取缓存超过100ms的情况还是会出现。

难道是memcached客户端EnyimMemcached的问题,看了EnyimMemcached的源代码,也没看出问题,EnyimMemcached只是Socket读取数据然后二进制反序列化,这地方再慢也不至于超过100ms吧。

百思不得其解,这个问题就一直放着。。。

今天早上发现日志中记录了00:00-8:00之间竟然发生了近1200次的读取缓存超过100ms的情况,几乎每分钟都出现。昨天刚刚给“黑色n秒”问题找到了最合理的猜想,正处在劲头上。于是趁热打铁,给“黑色n秒”家族添加了一位新成员——“黑色0.1秒”,好好分析一下。

那分析从何处开始呢?我们把怀疑对象锁定在了网络TCP层,所以从抓包开始。

对于Windows系统,最强大的抓包工具非Wireshark莫属。

在云服务器上启动Wireshark,选择与OCS进行通信的内网接口,配置Capture Filter只抓取与OCS进行通信的TCP包。

wireshark配置

Wireshark Capture运行没多久,就出现了一次“黑色0.1秒”。

应用日志

成功抓到了包,根据日志中时间点及缓存Key找到了对应的TCP包:

wireshark抓包

然后在Wireshark的右击菜单中选择“Follow TCP Strem”,只显示这个TCP流的包:

wireshark菜单
techbbs.zol.com.cn/36/55_352330.html
techbbs.zol.com.cn/36/55_352329.html
techbbs.zol.com.cn/36/55_352326.html
techbbs.zol.com.cn/36/55_352304.html
techbbs.zol.com.cn/36/55_352299.html
techbbs.zol.com.cn/36/55_352298.html
techbbs.zol.com.cn/36/55_352297.html
techbbs.zol.com.cn/36/55_352296.html
techbbs.zol.com.cn/36/55_352295.html
techbbs.zol.com.cn/36/55_352294.html
techbbs.zol.com.cn/36/55_352293.html
techbbs.zol.com.cn/36/55_352292.html
techbbs.zol.com.cn/36/55_352291.html
techbbs.zol.com.cn/36/55_352290.html
techbbs.zol.com.cn/36/55_352289.html
techbbs.zol.com.cn/36/55_352288.html
techbbs.zol.com.cn/36/55_352286.html
techbbs.zol.com.cn/36/55_352285.html
techbbs.zol.com.cn/36/55_352283.html
techbbs.zol.com.cn/36/55_352281.html
techbbs.zol.com.cn/36/55_352278.html
techbbs.zol.com.cn/36/55_352277.html
techbbs.zol.com.cn/36/55_352274.html
techbbs.zol.com.cn/36/55_352273.html
techbbs.zol.com.cn/36/55_352270.html
techbbs.zol.com.cn/36/55_352269.html
techbbs.zol.com.cn/36/55_352268.html
techbbs.zol.com.cn/36/55_352267.html
techbbs.zol.com.cn/36/55_352266.html
techbbs.zol.com.cn/36/55_352264.html
techbbs.zol.com.cn/36/55_352262.html
techbbs.zol.com.cn/36/55_352260.html
techbbs.zol.com.cn/36/55_352259.html
techbbs.zol.com.cn/36/55_352257.html
techbbs.zol.com.cn/36/55_352255.html
techbbs.zol.com.cn/36/55_352254.html
techbbs.zol.com.cn/36/55_352252.html
techbbs.zol.com.cn/36/55_352251.html
techbbs.zol.com.cn/36/55_352250.html
techbbs.zol.com.cn/36/55_352248.html
techbbs.zol.com.cn/36/55_352247.html
techbbs.zol.com.cn/36/55_352245.html
techbbs.zol.com.cn/36/55_352244.html
techbbs.zol.com.cn/36/55_352242.html
techbbs.zol.com.cn/36/55_352240.html
techbbs.zol.com.cn/36/55_352239.html
techbbs.zol.com.cn/36/55_352237.html
techbbs.zol.com.cn/36/55_352236.html
techbbs.zol.com.cn/36/55_352234.html
techbbs.zol.com.cn/36/55_352227.html
techbbs.zol.com.cn/36/55_352225.html
techbbs.zol.com.cn/36/55_352224.html
techbbs.zol.com.cn/36/55_352222.html
techbbs.zol.com.cn/36/55_352221.html
techbbs.zol.com.cn/36/55_352347.html
techbbs.zol.com.cn/36/55_352346.html
techbbs.zol.com.cn/36/55_352345.html
techbbs.zol.com.cn/36/55_352343.html
techbbs.zol.com.cn/36/55_352341.html
techbbs.zol.com.cn/36/55_352355.html
techbbs.zol.com.cn/36/55_352354.html
techbbs.zol.com.cn/36/55_352353.html
techbbs.zol.com.cn/36/55_352352.html
techbbs.zol.com.cn/36/55_352351.html
techbbs.zol.com.cn/36/55_352368.html
techbbs.zol.com.cn/36/55_352366.html
techbbs.zol.com.cn/36/55_352361.html
techbbs.zol.com.cn/36/55_352359.html
techbbs.zol.com.cn/36/55_352382.html
techbbs.zol.com.cn/36/55_352380.html
techbbs.zol.com.cn/36/55_352378.html
techbbs.zol.com.cn/36/55_352376.html
techbbs.zol.com.cn/36/55_352372.html
techbbs.zol.com.cn/36/55_352407.html
techbbs.zol.com.cn/36/55_352398.html
techbbs.zol.com.cn/36/55_352389.html
techbbs.zol.com.cn/36/55_352387.html
techbbs.zol.com.cn/36/55_352385.html
techbbs.zol.com.cn/36/55_352467.html
techbbs.zol.com.cn/36/55_352466.html
techbbs.zol.com.cn/36/55_352357.html
techbbs.zol.com.cn/36/55_352522.html
qing.blog.sina.com.cn/3787750573/e1c47cad33004ws8.html
qing.blog.sina.com.cn/3787750573/e1c47cad33004ws7.html
qing.blog.sina.com.cn/3787750573/e1c47cad33004ws6.html
qing.blog.sina.com.cn/3787750573/e1c47cad33004ws5.html
qing.blog.sina.com.cn/3787750573/e1c47cad33004ws4.html
qing.blog.sina.com.cn/3787750573/e1c47cad33004ws3.html
qing.blog.sina.com.cn/3787750573/e1c47cad33004ws9.html
qing.blog.sina.com.cn/3787750573/e1c47cad33004wsa.html
qing.blog.sina.com.cn/3787750573/e1c47cad33004wsb.html
qing.blog.sina.com.cn/3787750573/e1c47cad33004wsc.html
qing.blog.sina.com.cn/3787750573/e1c47cad33004wsd.html
qing.blog.sina.com.cn/3787750573/e1c47cad33004wse.html
qing.blog.sina.com.cn/3787750573/e1c47cad33004wsf.html
qing.blog.sina.com.cn/3787750573/e1c47cad33004wsg.html
qing.blog.sina.com.cn/3787750573/e1c47cad33004wsh.html
qing.blog.sina.com.cn/3787750573/e1c47cad33004wsi.html
qing.blog.sina.com.cn/3787750573/e1c47cad33004wsj.html
qing.blog.sina.com.cn/3787750573/e1c47cad33004wsk.html
yishujiayuan.com/shaonvmanhua/2014/0509/3052.html
yishujiayuan.com/shaonvmanhua/2014/0509/3053.html
yishujiayuan.com/shaonvmanhua/2014/0509/3051.html
yishujiayuan.com/shaonvmanhua/2014/0509/3050.html
yishujiayuan.com/shaonvmanhua/2014/0509/3049.html
yishujiayuan.com/shaonvmanhua/2014/0509/3048.html
yishujiayuan.com/shehuiwanxiang/2014/0509/2998.html
yishujiayuan.com/shehuiwanxiang/2014/0509/2997.html
yishujiayuan.com/shehuiwanxiang/2014/0509/2996.html
yishujiayuan.com/shehuiwanxiang/2014/0509/2995.html
yishujiayuan.com/shehuiwanxiang/2014/0509/2993.html
yishujiayuan.com/shehuiwanxiang/2014/0509/2994.html
yishujiayuan.com/xiaohua/2014/0509/3059.html
yishujiayuan.com/xiaohua/2014/0509/3058.html
yishujiayuan.com/xiaohua/2014/0509/3057.html
yishujiayuan.com/xiaohua/2014/0509/3056.html
yishujiayuan.com/xiaohua/2014/0509/3055.html
yishujiayuan.com/xiaohua/2014/0509/3054.html
yishujiayuan.com/xincai/2014/0509/3036.html
yishujiayuan.com/xincai/2014/0509/3037.html
yishujiayuan.com/xincai/2014/0509/3038.html
yishujiayuan.com/xincai/2014/0509/3039.html
yishujiayuan.com/xincai/2014/0509/3040.html
yishujiayuan.com/xincai/2014/0509/3041.html
yishujiayuan.com/zqxixun/1328.html
yishujiayuan.com/zqxixun/3066.html
yishujiayuan.com/zqxixun/3067.html
yishujiayuan.com/zqxixun/3068.html
yishujiayuan.com/zqxixun/3069.html
yishujiayuan.com/zqxixun/3070.html
yishujiayuan.com/3dnews/3042.html
yishujiayuan.com/3dnews/3043.html
yishujiayuan.com/3dnews/3044.html
yishujiayuan.com/3dnews/3045.html
yishujiayuan.com/3dnews/3046.html
yishujiayuan.com/3dnews/3047.html
yishujiayuan.com/caizimi/3060.html
yishujiayuan.com/caizimi/3061.html
yishujiayuan.com/caizimi/3062.html
yishujiayuan.com/caizimi/3063.html
yishujiayuan.com/caizimi/3064.html
yishujiayuan.com/caizimi/3065.html
yishujiayuan.com/3dshahao/3071.html
yishujiayuan.com/3dshahao/3072.html
yishujiayuan.com/3dshahao/3073.html
yishujiayuan.com/3dshahao/3074.html
yishujiayuan.com/3dshahao/3075.html
yishujiayuan.com/3dshahao/3076.html
techbbs.zol.com.cn/36/55_352529.html
zimi68.com/shaonvmanhua/2014/0509/3052.html
zimi68.com/shaonvmanhua/2014/0509/3053.html
zimi68.com/shaonvmanhua/2014/0509/3051.html
zimi68.com/shaonvmanhua/2014/0509/3050.html
zimi68.com/shaonvmanhua/2014/0509/3049.html
zimi68.com/shaonvmanhua/2014/0509/3048.html
zimi68.com/shehuiwanxiang/2014/0509/2998.html
zimi68.com/shehuiwanxiang/2014/0509/2997.html
zimi68.com/shehuiwanxiang/2014/0509/2996.html
zimi68.com/shehuiwanxiang/2014/0509/2995.html
zimi68.com/shehuiwanxiang/2014/0509/2993.html
zimi68.com/shehuiwanxiang/2014/0509/2994.html
zimi68.com/xiaohua/2014/0509/3059.html
zimi68.com/xiaohua/2014/0509/3058.html
zimi68.com/xiaohua/2014/0509/3057.html
zimi68.com/xiaohua/2014/0509/3056.html
zimi68.com/xiaohua/2014/0509/3055.html
zimi68.com/xiaohua/2014/0509/3054.html
zimi68.com/xincai/2014/0509/3036.html
zimi68.com/xincai/2014/0509/3037.html
zimi68.com/xincai/2014/0509/3038.html
zimi68.com/xincai/2014/0509/3039.html
zimi68.com/xincai/2014/0509/3040.html
zimi68.com/xincai/2014/0509/3041.html
zimi68.com/zqxixun/1328.html
zimi68.com/zqxixun/3066.html
zimi68.com/zqxixun/3067.html
zimi68.com/zqxixun/3068.html
zimi68.com/zqxixun/3069.html
zimi68.com/zqxixun/3070.html
zimi68.com/3dnews/3042.html
zimi68.com/3dnews/3043.html
zimi68.com/3dnews/3044.html
zimi68.com/3dnews/3045.html
zimi68.com/3dnews/3046.html
zimi68.com/3dnews/3047.html
zimi68.com/caizimi/3060.html
zimi68.com/caizimi/3061.html
zimi68.com/caizimi/3062.html
zimi68.com/caizimi/3063.html
zimi68.com/caizimi/3064.html
zimi68.com/caizimi/3065.html
www.zimi68.com/shaonvmanhua/2014/0509/3052.html
www.zimi68.com/shaonvmanhua/2014/0509/3053.html
www.zimi68.com/shaonvmanhua/2014/0509/3051.html
www.zimi68.com/shaonvmanhua/2014/0509/3050.html
www.zimi68.com/shaonvmanhua/2014/0509/3049.html
www.zimi68.com/shaonvmanhua/2014/0509/3048.html
www.zimi68.com/shehuiwanxiang/2014/0509/2998.html
www.zimi68.com/shehuiwanxiang/2014/0509/2997.html
www.zimi68.com/shehuiwanxiang/2014/0509/2996.html
www.zimi68.com/shehuiwanxiang/2014/0509/2995.html
www.zimi68.com/shehuiwanxiang/2014/0509/2993.html
www.zimi68.com/shehuiwanxiang/2014/0509/2994.html
www.zimi68.com/xiaohua/2014/0509/3059.html
www.zimi68.com/xiaohua/2014/0509/3058.html
www.zimi68.com/xiaohua/2014/0509/3057.html
www.zimi68.com/xiaohua/2014/0509/3056.html
www.zimi68.com/xiaohua/2014/0509/3055.html
www.zimi68.com/xiaohua/2014/0509/3054.html
www.zimi68.com/xincai/2014/0509/3036.html
www.zimi68.com/xincai/2014/0509/3037.html
www.zimi68.com/xincai/2014/0509/3038.html
www.zimi68.com/xincai/2014/0509/3039.html
www.zimi68.com/xincai/2014/0509/3040.html
www.zimi68.com/xincai/2014/0509/3041.html
www.zimi68.com/zqxixun/1328.html
www.zimi68.com/zqxixun/3066.html
www.zimi68.com/zqxixun/3067.html
www.zimi68.com/zqxixun/3068.html
www.zimi68.com/zqxixun/3069.html
www.zimi68.com/zqxixun/3070.html
www.zimi68.com/3dnews/3042.html
www.zimi68.com/3dnews/3043.html
www.zimi68.com/3dnews/3044.html
www.zimi68.com/3dnews/3045.html
www.zimi68.com/3dnews/3046.html
www.zimi68.com/3dnews/3047.html
www.zimi68.com/caizimi/3060.html
www.zimi68.com/caizimi/3061.html
www.zimi68.com/caizimi/3062.html
www.zimi68.com/caizimi/3063.html
www.zimi68.com/caizimi/3064.html
www.zimi68.com/caizimi/3065.html
www.zimi68.com/3dshahao/3071.html
www.zimi68.com/3dshahao/3072.html
www.zimi68.com/3dshahao/3073.html
www.zimi68.com/3dshahao/3074.html
www.zimi68.com/3dshahao/3075.html
www.zimi68.com/3dshahao/3076.html
zimi68.com/3dshahao/3071.html
zimi68.com/3dshahao/3072.html
zimi68.com/3dshahao/3073.html
zimi68.com/3dshahao/3074.html
zimi68.com/3dshahao/3075.html
zimi68.com/3dshahao/3076.html
www.yishujiayuan.com/shaonvmanhua/2014/0509/3052.html
www.yishujiayuan.com/shaonvmanhua/2014/0509/3053.html
www.yishujiayuan.com/shaonvmanhua/2014/0509/3051.html
www.yishujiayuan.com/shaonvmanhua/2014/0509/3050.html
www.yishujiayuan.com/shaonvmanhua/2014/0509/3049.html
www.yishujiayuan.com/shaonvmanhua/2014/0509/3048.html
www.yishujiayuan.com/shehuiwanxiang/2014/0509/2998.html
www.yishujiayuan.com/shehuiwanxiang/2014/0509/2997.html
www.yishujiayuan.com/shehuiwanxiang/2014/0509/2996.html
www.yishujiayuan.com/shehuiwanxiang/2014/0509/2995.html
www.yishujiayuan.com/shehuiwanxiang/2014/0509/2993.html
www.yishujiayuan.com/shehuiwanxiang/2014/0509/2994.html
www.yishujiayuan.com/xiaohua/2014/0509/3059.html
www.yishujiayuan.com/xiaohua/2014/0509/3058.html
www.yishujiayuan.com/xiaohua/2014/0509/3057.html
www.yishujiayuan.com/xiaohua/2014/0509/3056.html
www.yishujiayuan.com/xiaohua/2014/0509/3055.html
www.yishujiayuan.com/xiaohua/2014/0509/3054.html
www.yishujiayuan.com/xincai/2014/0509/3036.html
www.yishujiayuan.com/xincai/2014/0509/3037.html
www.yishujiayuan.com/xincai/2014/0509/3038.html
www.yishujiayuan.com/xincai/2014/0509/3039.html
www.yishujiayuan.com/xincai/2014/0509/3040.html
www.yishujiayuan.com/xincai/2014/0509/3041.html
www.yishujiayuan.com/zqxixun/1328.html
www.yishujiayuan.com/zqxixun/3066.html
www.yishujiayuan.com/zqxixun/3067.html
www.yishujiayuan.com/zqxixun/3068.html
www.yishujiayuan.com/zqxixun/3069.html
www.yishujiayuan.com/zqxixun/3070.html
www.yishujiayuan.com/3dnews/3042.html
www.yishujiayuan.com/3dnews/3043.html
www.yishujiayuan.com/3dnews/3044.html
www.yishujiayuan.com/3dnews/3045.html
www.yishujiayuan.com/3dnews/3046.html
www.yishujiayuan.com/3dnews/3047.html
www.yishujiayuan.com/caizimi/3060.html
www.yishujiayuan.com/caizimi/3061.html
www.yishujiayuan.com/caizimi/3062.html
www.yishujiayuan.com/caizimi/3063.html
www.yishujiayuan.com/caizimi/3064.html
www.yishujiayuan.com/caizimi/3065.html
www.yishujiayuan.com/3dshahao/3071.html
www.yishujiayuan.com/3dshahao/3072.html
www.yishujiayuan.com/3dshahao/3073.html
www.yishujiayuan.com/3dshahao/3074.html
www.yishujiayuan.com/3dshahao/3075.html
www.yishujiayuan.com/3dshahao/3076.html

还没有人回应!

您的回应

你还未登陆,不能回应!登陆