Louis Liu

Louis Liu

Head of Data & AI Innovation @ Ping An Good Doctor

AI models / AI Infra / Bigdata

Architecture of data warehouse which is based on MQ

Recently, we create a mysql data warehouse which is based on message queue.

Most companies must prepare for particular queries in their systems if they consider to split their databases or tables into many pieces.

some problems should be solved in this situation:

1. how to get correct re[……]

Read more

linux tools part 9– Monitoring system performance

淘宝的开源监控工具 –同事推荐安装测下来非常不错 可以配合zabbix进行数据的采集

link:http://tsar.taobao.org/

[mysql@mysqldb tmp]$ tsar 
 
Time           ---cpu-- ---mem-- ---tcp-- -----traffic---- --sda---  ---load- 
Time             util     util   retran    bytin  bytout     util     load1   
24/04/14-22:30   1.19    64.[......]

Read more

news in mysqlbinlog – Back Up Master Binary Log Files

news in mysqlbinlog – Back Up Master Binary Log Files

从mysql5.6开始 mysqlbinlog开始支持远程读取master主机的binlog写入本地,极大的加强了binlog的备份策略,由于在mysql cluster复制环境中,binlog的存在极大的决定的数据恢复的完整性,所以binlog的备份显得特别重要。在诸多HA方案中,例如MHA,使用主库的binlog去恢复主备库之间的数据差,在主库物理机器down机无法重启的情况下,binlog的备份可以直接用来recover slave.所以这一特性提升mysql 容灾级别,使[……]

Read more

MongoDB not preallocate journal log

在安装mongodb 2.6.2的时候发现一个奇怪的问题 RS 节点的journal log并没有提前分配.我们知道在安装mongodb的时候
mongo总是会预先分配journal log 启用smallfile的时候默认为128MB 否则会分配1GB的journal log

下面是仲裁节点的日志:

2014-06-17T11:50:09.842+0800 [initandlisten] MongoDB starting : pid=4749 port=27017 dbpath=/data/mongodb/data 64-bit host=vm-3-57
2014-06-[......]

Read more

NoSQL压测工具YCSB

Download PDF from slideshare

基于mongodb的压力评测工具 YCSB的一些概括

YCSB 是一款基于NOSQL Benchmark的工具,提供了多种方式模拟测试nosql的负载,基于现在对nosql的一些压力测试还没有得到重视.
YCSB 的全面的测试功能可以给即将上线的nosql提供一种另类保障。

There are many new serving databases available, including:

PNUTS
BigTable
HBase
Hypertable
Azure
Cassandra
Couch[......]

Read more

Scroll to Top