Life is Short

发表于 | 分类于 | 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
mkdir ngx-with-lua
cd ngx-with-lua
current_dir=`pwd`
# download luajit
wget http://luajit.org/download/LuaJIT-2.0.4.tar.gz
tar -xf LuaJIT-2.0.4.tar.gz
cd LuaJIT-2.0.4
make && make install
# download more set headers module for nginx
wget https://github.com/openresty/headers-more-nginx-module/archive/v0.29.tar.gz
mv v0.29.tar.gz nginx-more-set-headers-v0.29.tar.gz
tar -xf nginx-more-set-headers-v0.29.tar.gz
# download nginx devel kit
wget https://github.com/simpl/ngx_devel_kit/archive/v0.2.19.tar.gz
mv v0.2.19.tar.gz nginx-devel-kit-v0.2.19.tar.gz
tar -xf nginx-devel-kit-v0.2.19.tar.gz
# download lua module for nginx
wget https://github.com/openresty/lua-nginx-module/archive/v0.10.0.tar.gz
mv v0.10.0.tar.gz nginx-lua-v0.10.0.tar.gz
tar -xf nginx-lua-v0.10.0.tar.gz
# download nginx
wget http://nginx.org/download/nginx-1.9.9.tar.gz
tar -xf nginx-1.9.9.tar.gz
export LUAJIT_LIB=/usr/local/lib/libluajit-5.1.so.2.0.4
export LUAJIT_INC=/usr/local/include/luajit-2.0/
# compile nginx
cd nginx-1.9.9
./configure --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_mp4_module --with-http_secure_link_module --with-http_gzip_static_module --with-stream --with-stream_ssl_module --add-module=$current_dir/headers-more-nginx-module-0.29 --add-module=$current_dir/ngx_devel_kit-0.2.19 --add-module=$current_dir/lua-nginx-module-0.10.0 --with-ld-opt="-Wl,-rpath,/usr/local/lib"
make 
make install
Life is Short

发表于 | 分类于 |

curl 有一个参数 -w, --write-out <format> 可以在一次请求成功结束之后将跟该次请求相关的数据写到标准输出。
我们可以利用该功能将一次请求的各个阶段的时间格式化输出到屏幕以便查看分析问题。

首先, 我们查看man手册文档, 通过搜索定位到文档关于参数 -w, --write-out <format> 的描述位置，
可以看到关于这个文档的描述：

Defines what to display on stdout after a completed and successful operation. The format is a string that may contain plain text mixed with any number of vari‐
ables. The string can be specified as “string”, to get read from a particular file you specify it “@filename” and to tell curl to read the format from stdin you
write “@-“.

The variables present in the output format will be substituted by the value or text that curl thinks fit, as described below. All variables are specified as
%{variable_name} and to output a normal % you just write them as %%. You can output a newline by using \n, a carriage return with \r and a tab space with \t.

大致的意思如下:

定义在一次操作成功结束之后再标准输出中显示什么. 参数的格式可以是普通文本夹杂着预先定义好的变量.
参数可以通过命令行参数 “output format string” 来指定, 也可以通过 “@filename” 的格式来指定一个
文件, curl可以从该文件内读取 format 字符串, 或者也可以通过 “@-“ 的形式来告诉curl从标准输入内读取
format string .

curl会用合适的值或者文本来替换输出格式字符串中定义的变量. 所有的变量都以 %{variable_name} 的方式
来定义, 如果你想输出一个百分号(%), 写两个百分号(%%) 就好了. 你可以通过使用 \n 来输出一个换行, 也
可以使用 \r 来输出一个回车， 使用 \t 输出一个制表符.

在了解了该参数的含义后，我们从该参数后的变量列表里找出我们需要用到的预定义变量.
我需要用到的变量如下:(查看完整版的预定义变量列表请使用man curl 查看)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
time_appconnect
               The time, in seconds, it took from the start until the SSL/SSH/etc connect/handshake to the remote host was completed. (Added in 7.19.0)(在https请求中, 跟对方主机建立安全连接所耗费的秒数)
time_connect   The time, in seconds, it took from the start until the TCP connect to the remote host (or proxy) was completed.(跟对方主机建立tcp连接所耗费的秒数)
time_namelookup
               The time, in seconds, it took from the start until the name resolving was completed.(DNS查询耗费的秒数)
time_pretransfer
               The time, in seconds, it took from the start until the file transfer was just about to begin. This includes all pre-transfer commands and  negotia‐
               tions that are specific to the particular protocol(s) involved.(从操作开始到准备向对方发送请求内容所耗费的时间,包含了DNS查询或者协议处理的时间)
time_redirect  The  time,  in  seconds,  it took for all redirection steps include name lookup, connect, pretransfer and transfer before the final transaction was
               started. time_redirect shows the complete execution time for multiple redirections. (Added in 7.12.3)（该变量输出的是在最终请求发出前, 耗费在处理重定向的响应上的时间)
time_starttransfer
               The time, in seconds, it took from the start until the first byte was just about to be transferred. This includes  time_pretransfer  and  also  the
               time the server needed to calculate the result. (记录了从请求开始知道收到第一个字节耗费的秒数, 注意这个时间是包含了服务器处理请求的时间)
time_total     The total time, in seconds, that the full operation lasted. The time will be displayed with millisecond resolution. (整个操作耗费的时间, 以毫秒计)

结合上面这些信息, 我们可以写出一个长长的format字符串来输出一个格式化的时间信息:

1
curl -o /dev/null -w "time_namelookup: %{time_namelookup}\ntime_appconnect: %{time_appconnect}\ntime_connect: %{time_connect}\ntime_redirect: %{time_redirect}\ntime_pretransfer: %{time_pretransfer}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n" -s TEST_URL

下图是我的一个测试:

curl-output-with-format-time

在 *UNIX 系统下, 我们可以将这个长长的命令保存为一个别名(alias)， 方便以后的使用.
编辑 ~/.bashrc 文件, 添加如下一行：

1
alias curltime="curl -w \"time_namelookup:      %{time_namelookup}\ntime_appconnect:      %{time_appconnect}\ntime_connect:         %{time_connect}\ntime_redirect:        %{time_redirect}\ntime_pretransfer:     %{time_pretransfer}\ntime_starttransfer:   %{time_starttransfer}\ntime_total:           %{time_total}\n\" -s"

好了，以后打开命令终端窗口, 需要的时候我们就可以以 curltime命令来代替curl命令从而获得额外的格式化过的时间统计数据的输出了.

注: 增加 -s 参数是为了让curl的输出变得更干净一点

Life is Short

发表于 | 分类于 |

有一个表里存放了每五分钟的一些统计数据, 希望通过一个SQL来查询每五分钟数据点同前一个数据点的变化.

采用查询同一张表两次的方式, 将两个结果集的结果的时间字段使用固定差值关系join在一起(比如我的数据中前一个数据与后一个数据时间相差5分钟),
然后将关联到的记录的数据做统计运算.

示例SQL如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
SELECT
	*
FROM
	(
		SELECT
			a.record_time time_a,
			b.record_time time_b,
			a.flow AS data_a,
			b.flow AS data_b,
			a.flow - b.flow AS data_diff,
			round(
				abs((a.flow - b.flow) / b.flow * 1.0),
				2
			) AS data_pct,
			CASE
		WHEN a.flow - b.flow > 0 THEN
			'+'
		WHEN a.flow - b.flow = 0 THEN
			'='
		WHEN a.flow - b.flow < 0 THEN
			'-'
		END AS dir
		FROM
			68286017800f083dffbd4d a,
			68286017800f083dffbd4d b
		WHERE
			a.record_time < '2014-10-01 00:00:00'
		AND a.record_time >= '2014-09-01 00:00:00'
		AND a.record_time - b.record_time = 500
	) a
WHERE
	a.data_pct > 2
ORDER BY
	data_pct
Life is Short

发表于 | 分类于 |

redis 的 set/mset/get/mget/hset/hmset/hget/hmget/hgetall 等操作的性能测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
#!/usr/bin/env python
# -*- coding:utf-8 -*-
'''
Author : myth
Date   : 15-7-28
Email  : belongmyth at 163.com
'''
from datetime import datetime
from datetime import timedelta
import redis
import random
import functools
def timer(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        title = kwargs.pop(u'title', func.func_name)
        no_timer = kwargs.pop(u'no_timer', False)
        if no_timer:
            v = func(*args, **kwargs)
        else:
            s = datetime.now()
            v = func(*args, **kwargs)
            e = datetime.now()
            print u'%s total use %s'%(title, e-s)
        return v
    return wrapper
@timer
def gen_datas():
    datas = []
    start_time = datetime.strptime(u'2015-06-01 00:00:00', u'%Y-%m-%d %H:%M:%S')
    intervel = 300#s
    for i in range(12000):
        t = (start_time+timedelta(seconds=i*intervel))
        ts = t.strftime(u'%Y-%m-%d %H:%M:%S')
        vs = u'%.3f'%(random.randint(5000000, 50000000) * random.random(), )
        datas.append((ts, vs))
    return datas
@timer
def save_datas_use_set():
    for ts, vs in DATAS:
        c.set(ts, vs)
@timer
def save_datas_use_mset():
    ds = dict(DATAS)
    c.mset(ds)
@timer
def save_datas_use_hset():
    for ts, vs in DATAS:
        c.hset(CHANNEL_ID, ts, vs)
@timer
def save_datas_use_hmset():
    ds = dict(DATAS)
    c.hmset(CHANNEL_ID, ds)
@timer
def get_datas_use_get():
    vdatas = []
    for ts in tss:
        vs = c.get(ts)
        vdatas.append((ts, vs))
    return vdatas
@timer
def get_datas_use_mget():
    vss = c.mget(tss)
    return vss
@timer
def test_mget(times):
    for i in range(times):
        get_datas_use_mget(no_timer = True)
@timer
def get_datas_use_hget():
    vdatas = []
    for ts in tss:
        vs = c.hget(CHANNEL_ID, ts)
        vdatas.append((ts, vs))
    return vdatas
@timer
def test_hget(times):
    for i in range(times):
        get_datas_use_hget(no_timer=True)
@timer
def get_datas_use_hmget():
    vss = c.hmget(CHANNEL_ID, tss)
    return vss
@timer
def test_hmget(times):
    for i in range(times):
        get_datas_use_hmget(no_timer = True)
@timer
def get_datas_use_hgetall():
    vss = c.hgetall(CHANNEL_ID)
    return vss
@timer
def test_hgetall(times):
    for i in range(times):
        get_datas_use_hgetall(no_timer = True)
c = redis.StrictRedis()
CHANNEL_ID = u'channel_xxxxxxxxxxxxxxxxxxxxxxxx'
DATAS = gen_datas()
tss = [d[0] for d in DATAS]
times = 5
save_datas_use_set()
save_datas_use_mset()
get_datas_use_get()
get_datas_use_mget()
test_mget(times, title=u'get_datas_use_mget %d times'%times)
save_datas_use_hset()
save_datas_use_hmset()
get_datas_use_hget()
get_datas_use_hmget()
get_datas_use_hgetall()
test_hget(times, title=u'get_datas_use_hget %d times'%times)
test_hmget(times, title=u'get_datas_use_hmget %d times'%times)
test_hgetall(times, title=u'get_datas_use_hgetall %d times'%times)
'''
gen_datas total use 0:00:00.109303
save_datas_use_set total use 0:00:00.583253
save_datas_use_mset total use 0:00:00.072797
get_datas_use_get total use 0:00:00.540243
get_datas_use_mget total use 0:00:00.072904
get_datas_use_mget 5 times total use 0:00:00.370845
save_datas_use_hset total use 0:00:00.596802
save_datas_use_hmset total use 0:00:00.070581
get_datas_use_hget total use 0:00:00.640865
get_datas_use_hmget total use 0:00:00.069553
get_datas_use_hgetall total use 0:00:00.099832
get_datas_use_hget 5 times total use 0:00:02.704983
get_datas_use_hmget 5 times total use 0:00:00.364204
get_datas_use_hgetall 5 times total use 0:00:00.497511
'''

结论很明显啦

Life is Short

发表于 | 分类于 |

下载 svn

wget http://ftp.tsukuba.wide.ad.jp/software/apache/subversion/subversion-1.8.13.tar.gz

查看根目录下的 INSTALL 文件

执行 ./autogen.sh

根据提示，下载 autoconf, libtool

wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz

wget http://ftp.jaist.ac.jp/pub/GNU/libtool/libtool-2.4.6.tar.gz

安装 autoconf 提示需要 GNU M4

wget http://ftp.gnu.org/gnu/m4/m4-1.4.17.tar.gz

执行 subversion 目录中的 ./configure (指定 prefix)

提示：configure: error: no suitable APR found

下载 apr

wget http://ftp.jaist.ac.jp/pub/apache//apr/apr-1.5.2.tar.gz

下载 apr-util

wget http://ftp.jaist.ac.jp/pub/apache//apr/apr-util-1.5.4.tar.gz

又提示需要 SQLite

wget https://www.sqlite.org/2015/sqlite-autoconf-3081002.tar.gz

最后

./configure –prefix=/home/myth/tmp/subversion –with-apr=/home/myth/software/apr-1.5.2 –with-apr-util=/home/myth/software/apr-util-1.5.4

编译安装成功

Life is Short

发表于 | 分类于 | 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
#!/usr/bin/env python
# -*- coding:utf-8 -*-
'''
Author : myth
Date   : 15-5-14
Email  : belongmyth at 163.com
'''
Styles = dict(
        list(zip(list(range(1, 9)),[
            'bold',
            'dark',
            '',
            'underline',
            'blink',
            '',
            'reverse',
            'concealed'
            ]
            ))
        )
for k,v in Styles.items():
    if v == '':
        del Styles[k]
ForegroundColors = dict(
        list(zip(list(range(30, 38)),[
            'grey',
            'red',
            'green',
            'yellow',
            'blue',
            'magenta',
            'cyan',
            'white',
            ]
            ))
        )
BackgroundColors = dict(
        list(zip(list(range(40, 48)),[
            'on_grey',
            'on_red',
            'on_green',
            'on_yellow',
            'on_blue',
            'on_magenta',
            'on_cyan',
            'on_white'
            ]
            ))
        )
max_length = max([len(v) for v in Styles.values()]) + max([len(v) for v in ForegroundColors.values()]) + max([len(v) for v in BackgroundColors.values()])
def print_format_table():
    """
    prints table of formatted text format options
    """
    for style, styleMsg in Styles.items():
        for fg, fgMsg in ForegroundColors.items():
            s1 = ''
            for bg, bgMsg in BackgroundColors.items():
                format = ';'.join([str(style), str(fg), str(bg)])
                formatMsg = '-'.join([styleMsg, fgMsg, bgMsg]).center(max_length+3)
                s1 += '\x1b[%sm %s \x1b[0m' % (format, formatMsg)
            print s1
        print ''
print_format_table()