June 2011 – 夏天人字拖

Android开发当中的内存分配错误

基本都是内存不够分配，后来采取的是写外部文件的方法来完成的，参见http://www.iteye.com/topic/1014035
写了一个应用，估计把内存分配不够的问题全遇上了

1、
04-27 09:34:34.292: INFO/dalvikvm(9840): Jit: resizing JitTable from 4096 to 8192
04-27 09:56:19.612: DEBUG/dalvikvm(9840): GC_EXPLICIT freed 69K, 73% free 3838K/13767K, external 1175K/1569K, paused 115ms
04-27 09:56:21.062: INFO/ActivityManager(61): Process com.example.android (pid 9840) has died.
04-27 09:56:21.062: INFO/WindowManager(61): WIN DEATH: Window{40716078 com.example.android/com.example.android.Main paused=false}

04-27 10:32:46.042: INFO/dalvikvm-heap(18965): Clamp target GC heap from 16.226MB to 16.000MB
04-27 10:33:06.522: ERROR/GraphicsJNI(18965): VM won’t let us allocate 4096 bytes
04-27 10:32:30.382: INFO/ActivityManager(61): Process android.process.acore (pid 25279) has died.
04-27 10:32:30.382: INFO/ActivityManager(61): Low Memory: No more background processes.

2、
04-25 21:52:31.226: ERROR/AndroidRuntime(9853): Caused by: java.lang.OutOfMemoryError
04-25 21:52:31.226: ERROR/AndroidRuntime(9853): at java.io.ByteArrayOutputStream.expand(ByteArrayOutputStream.java:93)
04-25 21:52:31.226: ERROR/AndroidRuntime(9853): at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:218)
04-25 21:52:31.226: ERROR/AndroidRuntime(9853): at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:175)
04-25 21:52:31.226: ERROR/AndroidRuntime(9853): at java.io.OutputStreamWriter.convert(OutputStreamWriter.java:252)
04-25 21:52:31.226: ERROR/AndroidRuntime(9853): at java.io.OutputStreamWriter.write(OutputStreamWriter.java:241)
04-25 21:52:31.226: ERROR/AndroidRuntime(9853): at java.io.BufferedWriter.write(BufferedWriter.java:224)
04-25 21:52:31.226: ERROR/AndroidRuntime(9853): at java.io.Writer.write(Writer.java:101)
04-25 21:52:31.226: ERROR/AndroidRuntime(9853): at java.io.Writer.write(Writer.java:155)
04-25 21:52:31.226: ERROR/AndroidRuntime(9853): at org.kxml2.io.KXmlSerializer.endTag(KXmlSerializer.java:508)

3、
04-25 19:51:44.196: INFO/dalvikvm-heap(6801): Forcing collection of SoftReferences for 1783864-byte allocation
04-25 19:51:44.216: ERROR/dalvikvm(6801): HeapWorker is wedged: 34640ms spent inside Lcom/android/internal/os/BinderInternal$GcWatcher;.finalize()V
04-25 19:51:44.216: INFO/dalvikvm(6801): DALVIK THREADS:
04-25 19:51:44.226: INFO/dalvikvm(6801): “main” prio=5 tid=3 VMWAIT
04-25 19:51:44.236: INFO/dalvikvm(6801): | group=”main” sCount=1 dsCount=0 s=N obj=0x4001b268 self=0xbd00
04-25 19:51:44.236: INFO/dalvikvm(6801): | sysTid=6801 nice=0 sched=0/0 cgrp=default handle=-1344001384
04-25 19:51:44.246: INFO/dalvikvm(6801): at android.app.ActivityThread.handleLowMemory(ActivityThread.java:~3840)
04-25 19:51:44.256: INFO/dalvikvm(6801): at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1949)
04-25 19:51:44.256: INFO/dalvikvm(6801): at android.os.Handler.dispatchMessage(Handler.java:99)
04-25 19:51:44.266: INFO/dalvikvm(6801): at android.os.Looper.loop(Looper.java:123)
04-25 19:51:44.266: INFO/dalvikvm(6801): at android.app.ActivityThread.main(ActivityThread.java:4363)
04-25 19:51:44.266: INFO/dalvikvm(6801): at java.lang.reflect.Method.invokeNative(Native Method)
04-25 19:51:44.266: INFO/dalvikvm(6801): at java.lang.reflect.Method.invoke(Method.java:521)
04-25 19:51:44.276: INFO/dalvikvm(6801): at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:860)
04-25 19:51:44.276: INFO/dalvikvm(6801): at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:618)
04-25 19:51:44.276: INFO/dalvikvm(6801): at dalvik.system.NativeStart.main(Native Method)
04-25 19:51:44.286: INFO/dalvikvm(6801): “Binder Thread #8″ prio=5 tid=27 NATIVE
04-25 19:51:44.286: INFO/dalvikvm(6801): | group=”main” sCount=1 dsCount=0 s=N obj=0x440ceef0 self=0x2fa898
04-25 19:51:44.286: INFO/dalvikvm(6801): | sysTid=7415 nice=0 sched=0/0 cgrp=default handle=2884512
04-25 19:51:44.286: INFO/dalvikvm(6801): at dalvik.system.NativeStart.run(Native Method)
04-25 19:51:44.286: INFO/dalvikvm(6801): “Binder Thread #7″ prio=5 tid=25 NATIVE
04-25 19:51:44.286: INFO/dalvikvm(6801): | group=”main” sCount=1 dsCount=0 s=N obj=0x43f82530 self=0x2e3248
04-25 19:51:44.286: INFO/dalvikvm(6801): | sysTid=7256 nice=0 sched=0/0 cgrp=default handle=3027464
04-25 19:51:44.286: INFO/dalvikvm(6801): at dalvik.system.NativeStart.run(Native Method)
04-25 19:51:44.286: INFO/dalvikvm(6801): “Binder Thread #6″ prio=5 tid=23 NATIVE
04-25 19:51:44.296: INFO/dalvikvm(6801): | group=”main” sCount=1 dsCount=0 s=N obj=0x43df2e98 self=0x2f7890
04-25 19:51:44.296: INFO/dalvikvm(6801): | sysTid=7074 nice=0 sched=0/0 cgrp=default handle=3112240
04-25 19:51:44.296: INFO/dalvikvm(6801): at dalvik.system.NativeStart.run(Native Method)
04-25 19:51:44.296: INFO/dalvikvm(6801): “Binder Thread #5″ prio=5 tid=21 NATIVE
04-25 19:51:44.296: INFO/dalvikvm(6801): | group=”main” sCount=1 dsCount=0 s=N obj=0x43ba60d8 self=0x2eea38
04-25 19:51:44.296: INFO/dalvikvm(6801): | sysTid=7008 nice=0 sched=0/0 cgrp=default handle=3057296
04-25 19:51:44.296: INFO/dalvikvm(6801): at dalvik.system.NativeStart.run(Native Method)
04-25 19:51:44.306: INFO/dalvikvm(6801): “Binder Thread #4″ prio=5 tid=19 NATIVE
04-25 19:51:44.306: INFO/dalvikvm(6801): | group=”main” sCount=1 dsCount=0 s=N obj=0x43c3a100 self=0x2e7358
04-25 19:51:44.306: INFO/dalvikvm(6801): | sysTid=6991 nice=0 sched=0/0 cgrp=default handle=3065008
04-25 19:51:44.306: INFO/dalvikvm(6801): at dalvik.system.NativeStart.run(Native Method)
04-25 19:51:44.316: INFO/dalvikvm(6801): “Binder Thread #3″ prio=5 tid=17 NATIVE
04-25 19:51:44.316: INFO/dalvikvm(6801): | group=”main” sCount=1 dsCount=0 s=N obj=0x43c19078 self=0x2e9618
04-25 19:51:44.316: INFO/dalvikvm(6801): | sysTid=6986 nice=0 sched=0/0 cgrp=default handle=3052944
04-25 19:51:44.316: INFO/dalvikvm(6801): at dalvik.system.NativeStart.run(Native Method)
04-25 19:51:44.326: INFO/dalvikvm(6801): “AsyncTask #1″ prio=5 tid=15 RUNNABLE
04-25 19:51:44.326: INFO/dalvikvm(6801): | group=”main” sCount=0 dsCount=0 s=N obj=0x43c3a250 self=0x2cf6e0
04-25 19:51:44.326: INFO/dalvikvm(6801): | sysTid=6977 nice=0 sched=0/0 cgrp=default handle=2946352
04-25 19:51:44.339: INFO/dalvikvm(6801): at java.nio.CharArrayBuffer.(CharArrayBuffer.java:~43)
04-25 19:51:44.346: INFO/dalvikvm(6801): at java.nio.ReadWriteCharArrayBuffer.(ReadWriteCharArrayBuffer.java:47)
04-25 19:51:44.346: INFO/dalvikvm(6801): at java.nio.BufferFactory.newCharBuffer(BufferFactory.java:84)
04-25 19:51:44.346: INFO/dalvikvm(6801): at java.nio.CharBuffer.allocate(CharBuffer.java:57)
04-25 19:51:44.356: INFO/dalvikvm(6801): at java.nio.charset.CharsetDecoder.allocateMore(CharsetDecoder.java:293)
04-25 19:51:44.356: INFO/dalvikvm(6801): at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:250)
04-25 19:51:44.356: INFO/dalvikvm(6801): at java.nio.charset.Charset.decode(Charset.java:768)
04-25 19:51:44.356: INFO/dalvikvm(6801): at java.lang.String.(String.java:238)
04-25 19:51:44.366: INFO/dalvikvm(6801): at java.io.ByteArrayOutputStream.toString(ByteArrayOutputStream.java:139)

4、
04-20 09:08:44.060: INFO/global(227): Default buffer size used in BufferedReader constructor. It would be better to be explicit if an 8k-char buffer is required.
04-20 09:38:25.450: ERROR/AndroidRuntime(721): Caused by: java.lang.OutOfMemoryError
04-20 09:38:25.450: ERROR/AndroidRuntime(721): at java.lang.AbstractStringBuilder.enlargeBuffer(AbstractStringBuilder.java:97)
04-20 09:38:25.450: ERROR/AndroidRuntime(721): at java.lang.AbstractStringBuilder.append0(AbstractStringBuilder.java:155)
04-20 09:38:25.450: ERROR/AndroidRuntime(721): at java.lang.StringBuffer.append(StringBuffer.java:214)

5、
04-20 11:54:40.510: ERROR/dalvikvm-heap(4237): Out of memory on a 1672096-byte allocation.
04-20 11:54:40.510: INFO/dalvikvm(4237): “AsyncTask #1″ prio=5 tid=15 RUNNABLE
04-20 11:54:40.510: INFO/dalvikvm(4237): | group=”main” sCount=0 dsCount=0 s=N obj=0x43c1f610 self=0x2cb260
04-20 11:54:40.510: INFO/dalvikvm(4237): | sysTid=4270 nice=10 sched=0/0 cgrp=bg_non_interactive handle=2907136
04-20 11:54:40.510: INFO/dalvikvm(4237): at java.lang.AbstractStringBuilder.enlargeBuffer(AbstractStringBuilder.java:~97)
04-20 11:54:40.510: INFO/dalvikvm(4237): at java.lang.AbstractStringBuilder.append0(AbstractStringBuilder.java:155)
04-20 11:54:40.510: INFO/dalvikvm(4237): at java.lang.StringBuilder.append(StringBuilder.java:216)

6、
04-20 11:54:40.540: ERROR/AndroidRuntime(4237): Uncaught handler: thread AsyncTask #1 exiting due to uncaught exception
04-20 11:54:40.550: ERROR/AndroidRuntime(4237): java.lang.RuntimeException: An error occured while executing doInBackground()
04-20 11:54:40.550: ERROR/AndroidRuntime(4237): at android.os.AsyncTask$3.done(AsyncTask.java:200)
04-20 11:54:40.550: ERROR/AndroidRuntime(4237): at java.util.concurrent.FutureTask$Sync.innerSetException(FutureTask.java:273)
04-20 11:54:40.550: ERROR/AndroidRuntime(4237): at java.util.concurrent.FutureTask.setException(FutureTask.java:124)
04-20 11:54:40.550: ERROR/AndroidRuntime(4237): at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:307)
04-20 11:54:40.550: ERROR/AndroidRuntime(4237): at java.util.concurrent.FutureTask.run(FutureTask.java:137)
04-20 11:54:40.550: ERROR/AndroidRuntime(4237): at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1068)
04-20 11:54:40.550: ERROR/AndroidRuntime(4237): at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:561)
04-20 11:54:40.550: ERROR/AndroidRuntime(4237): at java.lang.Thread.run(Thread.java:1096)
04-20 11:54:40.550: ERROR/AndroidRuntime(4237): Caused by: java.lang.OutOfMemoryError
04-20 11:54:40.550: ERROR/AndroidRuntime(4237): at java.lang.AbstractStringBuilder.enlargeBuffer(AbstractStringBuilder.java:97)
04-20 11:54:40.550: ERROR/AndroidRuntime(4237): at java.lang.AbstractStringBuilder.append0(AbstractStringBuilder.java:155)
04-20 11:54:40.550: ERROR/AndroidRuntime(4237): at java.lang.StringBuilder.append(StringBuilder.java:216)

7、
04-20 13:32:45.590: ERROR/dalvikvm(7314): Rejecting allocation of 2147483647-element array
04-20 13:32:45.640: ERROR/AndroidRuntime(7314): java.lang.OutOfMemoryError: array size too large

试玩HandlerSocket for MySQL

有这么一个场景，单台MySQL数据库服务器，最大表约500万记录，常用表100万记录，每行记录的数据量不是很大
查询还算频繁，插入修改删除数据不是那么多，多个应用同时访问这个数据库
有个应用就是查询为主，没有修改，少量的删除
查询SQL语句很简单，根据用户ID查询他下面的记录（某些字段），根据记录ID查询整条记录，两表关联查询100万的关联500万
目前应用的实现也很简单，简陋的数据库连接池，客户端发送SQL语句到SQL服务器
现在也没有遇到很大的问题，没有宕机什么的问题，但就是客户端反映慢，但是看服务器的负载都不是很高

所以老大就说太慢了，需要解决

这个慢是比较显然的啊，每次解析执行SQL语句，网络传输，数据库服务器内存较小，缓存不够

所以当然就想到了在应用层加缓存，但是在几个月之前曾经看到过HandlerSocket的简单介绍，只知道它可以加快查询，没有细看

所以突然觉得目前这个也可以改用HandlerSocket，而且更合适，向老大建议，当然建议都是木有用的，因为我做不鸟主，老大有老大的想法

所以只能自己研究看看这个东西适合不了

于是就开始下载编译安装这个东西了，步骤还算简单

具体安装参见installation.en.txt

我开始想用5.1的源码和5.5的二进制版本混合编译（目前我自己用的5.5的MySQL），天知道能不能成功（最后是不能成功的，我又用5.1的源码重新编译了一个MySQL数据库出来，然后编译HandlerSocket并把它安装到5.1中了，后来看到介绍是源码版本一定要和当前安装的数据库的版本一致，之所以需要源码是因为HandlerSocket插件用到了MySQL源码之中的几个头文件之类的吧，总之不看README是很头疼的。比如有的酒店的电梯是要用房卡刷了电梯之后才能按楼层的，README写的清清楚楚，汗）
我的编译步骤如下
$ ./autogen.sh
$ ./configure –with-mysql-source=/home/guohai/dev/src/database/mysql-5.1.57 –with-mysql-bindir=/home/guohai/dev/a/mysql/mysql-5.1.57-linux2.6-i686/bin –with-mysql-plugindir=/home/guohai/dev/a/mysql/mysql-5.1.57-linux2.6-i686/lib/mysql/plugin

$ make
$ sudo make install

mysql> install plugin handlersocket soname ‘handlersocket.so’;
Query OK, 0 rows affected (0.03 sec)

这样插件就安装好了，接下来就是装个客户端试试看了，用的淘宝的那个

当然还要在my.cnf中增加类似参数
loose_handlersocket_port = 9998
# the port number to bind to (for read requests)
loose_handlersocket_port_wr = 9999
# the port number to bind to (for write requests)
loose_handlersocket_threads = 16
# the number of worker threads (for read requests)
loose_handlersocket_threads_wr = 1
# the number of worker threads (for write requests)
open_files_limit = 65535
# to allow handlersocket accept many concurrent
# connections, make open_files_limit as large as
# possible.

弄hs4j还是报了几个错误，什么open_table，原因是我表名没有写对，巨汗，不过这个提示也够瓜的
估计也是我没有理解这个东西

然后我又报了个错误
Exception in thread “main” com.google.code.hs4j.exception.HandlerSocketException: Connection has been closed
at com.google.code.hs4j.impl.HSClientImpl.awaitResponse(HSClientImpl.java:417)
at com.google.code.hs4j.impl.HSClientImpl.insert0(HSClientImpl.java:325)
at com.google.code.hs4j.impl.HSClientImpl.insert(HSClientImpl.java:309)
at com.google.code.hs4j.example.HS4JExample.main(HS4JExample.java:29)
这个我把keys写少了一个，所以报错了
总之都是我的问题

就这样解决几个问题之后客户端就基本可用了

然后我还试了下Python的客户端，装好Python的插件http://packages.python.org/python-handler-socket/
就可以开始玩了，照着例子也很简单

看了还算多的文章介绍以及issues，这个东西在某些特定的情况下还是可以开始用的吧，以前提的比较多的问题，什么auto_increment之类的也都解决了

觉得最关键的是就是参数怎么配置，才能让这个东西在需要的场景下发挥最大的能力

还得多看看，多试试才有更充分的把握

参考
https://github.com/ahiguti/HandlerSocket-Plugin-for-MySQL
http://yoshinorimatsunobu.blogspot.com/2010/10/using-mysql-as-nosql-story-for.html 你懂的
https://github.com/killme2008/hs4j
http://whitesock.iteye.com/blog/811339
http://rdc.taobao.com/team/jm/archives/545
http://www.cnblogs.com/inrie/archive/2011/01/28/1946572.html 系列文章
http://huoding.com/2011/04/10/62

流水帐总是很简单，再把编译MySQL的主要过程也记在这里吧，凑点字数
我用的是Ubuntu 10.10 X86，以前编译过很多软件开发包，所以如果编译的时候遇到缺什么包装上就行了
下载就不说了，下载都不会的就不要从源码编译了，我用的是mysql-5.1.57.tar.gz
编译参见INSTALL-SOURCE
2.11.2. Installing MySQL from a Standard Source Distribution

$ ./configure –prefix=/home/guohai/dev/a/mysql/mysql-5.1.57-linux2.6-i686 –with-unix-socket-path=/tmp/mysql51.sock –with-tcp-port=3307 –with-named-curses-libs=/lib/libncurses.so.5 –enable-assembler –with-plugins=partition,innobase –with-charset=utf8 –with-collation=utf8_general_ci –with-extra-charsets=big5,ascii,gbk,utf8,latin1 –with-big-tables –without-debug

$ make
$ make install

$ cd /home/guohai/dev/a/mysql/mysql-5.1.57-linux2.6-i686
$ chown -R mysql .
$ chgrp -R mysql .

我以前已经装过MySQL5.5所以这些用户以及组都是有的

guohai@KNIGHT:~/dev/a/mysql/mysql-5.1.57-linux2.6-i686$ sudo cp ‘/home/guohai/dev/src/database/mysql-5.1.57/support-files/my-medium.cnf’ ./my.cnf

$ sudo bin/mysql_install_db –user=mysql –basedir=/home/guohai/dev/a/mysql/mysql-5.1.57-linux2.6-i686 –datadir=/home/guohai/dev/a/mysql/mysql-5.1.57-linux2.6-i686/data

启动数据库
$ sudo bin/mysqld_safe –defaults-file=/home/guohai/dev/a/mysql/mysql-5.1.57-linux2.6-i686/my.cnf –user=mysql &
注意这个参数传递的顺序，我开始一直把–user=mysql放前面，启动的时候报错，MySQL日志记载unknown variable ‘defaults-file’
110601 23:23:28 [ERROR] /home/guohai/dev/a/mysql/mysql-5.1.57-linux2.6-i686/libexec/mysqld: unknown variable ‘defaults-file=/home/guohai/dev/a/mysql/mysql-5.1.57-linux2.6-i686/my.cnf’
110601 23:23:28 [ERROR] Aborting
110601 23:23:28 InnoDB: Starting shutdown…
110601 23:23:33 InnoDB: Shutdown completed; log sequence number 0 44233
110601 23:23:33 [Note] /home/guohai/dev/a/mysql/mysql-5.1.57-linux2.6-i686/libexec/mysqld: Shutdown complete

然后看了mysqld_safe下这个脚本的写法，发现它有对传入的第一个参数判断是否为defaults，于是我就把defaults-file这个参数放到第一位，就正常启动了

一切都好了，然后就是设置了
/home/guohai/dev/a/mysql/mysql-5.1.57-linux2.6-i686$ sudo bin/mysql_secure_installation
一步步来就好了，注意默认my.cnf的位置，我的就放在mysql-5.1.57安装目录下

可能还要修改下MySQL数据文件存放的路径，我最开始编译的时候没有指定，然后它帮我生成的是放在mysql-5.1.57安装目录的一个var目录下面的，而我在my.cnf中指定的是data目录
所以我就修改了一个脚本中的，把var改为data了，具体是哪个脚本忘记了

再重启下，就可以用mysql命令连接上去了，基本正常

MySQL编译简单，关键是编译出自己想要的MySQL，编译出符合自己业务高性能的MySQL就是技术活了，参数很关键
只要参数用的好，没有程序写不鸟；只要工具用的好，没有程序写不鸟；夸张了点

还有刚开始编译出来你可能找不到mysqld这个东西，我反正一开始是没有找到，以为自己编译的出错了，后来才发现在libexec下而不是在bin下面，多用用工具搜索搜索就出来了

另外也可以参见Ubuntu二进制安装MySQL 5.5
地址http://lucane.iteye.com/blog/866355

Lyrics: Rox In The Box-The Decemberists

FROM http://www.songmeanings.net/songs/view/3530822107858854930/

Get the rocks in the box
Get the water right down to your socks
This bulkhead’s built of fallen brethren’s bones
We all do what we can
We endure our fellow man
And we sing our songs to the headframe’s creaks and moans

And it’s one, two, three
On the wrong side of the lea
What were you meant for?
What were you meant for?
And it’s seven, eight, nine
You get your shovel back in line
And if you ever make it to ten
You won’t make it again
And if you ever make it to ten
You won’t make it again

And you won’t make a dime
On this gray granite mountain mine
Of dirt you’re made, of dirt you will return
So while we’re living here
Let’s get this little one thing clear
There’s plenty of men to die, don’t jump your turn