《大众点评网高可用数据架构》由会员分享，可在线阅读，更多相关《大众点评网高可用数据架构（63页珍藏版）》请在金锄头文库上搜索。
1、HA Architecture in DPMMM & Memcached 卢钧轶DPWho am IMySQL DBADWeibo: 米雪儿侬好的cenaluluBlog: http:/ ClusterMMMHA in DPWriter DBReader DBmemcacheWeb1Web2Web3memcachememcacheReader DBReader DBMemcachedmemcached in DPMemcached in DPNode1Node2Node3Node4Main RingBackup RingMemcached in DPNode1Node2Node3Node4Ma
2、in RingBackup RingClientset key1 set key1 Memcached in DPNode1Node2Node3Node3Main RingBackup RingClientget key1 Memcached in DPNode1Node2Node3Node3Main RingBackup RingClientget key1 get key1 MemcachedProblems We MetProblem 1Cache Miss StormCache Miss StormHappens when : Memcached failed (physical) K
3、ey expired (logical)Cache Miss StormIdeal Cache Miss Procedure 1. get memcached miss 2. query MySQL 3. set value into memcachedCache Miss StormIn Fact ! 1. get memcached miss 2. massive concurrent query on MySQL (timeout) 3. nothing can be set into memcached 4. cache miss forever.Cache Miss Storm -
4、Fix1Node1Node2Node3Node4Main RingBackup RingFix for physical cache miss - Double WriteCache Miss Storm - Fix2Fix for logical cache miss - Hot Key 0. get & set value into web local cache 1. get memcached miss 2. try to add a lock key into memcache a. if (success) query MySQL & set memcache b. if (fai
5、led) return local cache* Only one web can query MySQL for missed key at the same time.Problem 2Multi Get HoleMultiGet HoleMultiGet / Gets: get command with multiple keysPurpose: Omit the multiple network round- trips, Problem: The gets command will be slower when we add more nodes into the cluster.h
6、ttp:/ HoleNode1Node2Node3Clientget key1,key2 . key12MultiGet HoleClientget key1,key4,key7,key10get key3,key6,key7,key12get key2,key5,key8,key11Node1Node2Node3MultiGet HoleNode1Node2Node3ClientResult v1,v4,v7,v10get key3,key6,key9,key12get key2,key5,key8,key11get key1,key4,key7,key10MultiGet HoleNode
7、1Node2Node3Client get key3,key6,key9,key12get key2,key5,key8,key11Result v1,v4,v7,v10 v2,v5,v8,v11MultiGet HoleNode1Node2Node3ClientResult v1,v4,v7,v10 v2,v5,v8,v11 v3,v6,v9,v12get key3,key6,key9,key12MultiGet HoleNode1Node2Node3ClientOne more Round Trip !Node4Result v1,v5,v9 v2,v6,v10 v3,v7,v11 v4,
8、v8,v12MultiGet Hole - FixBefore: client use key name to decide which host to store store valueNow: client use master key name to decide which host to store use key name to store value * Put the keys of the same group into only one nodeMultiGet Hole - FixNode1Node2Node3Clientcmd:set key: key1 master_
9、key: group1 value: v1Hash using only master_keyMultiGet Hole - FixNode1Node2Node3Clientcmd:get key: key1 . key12 master_key: group1MultiGet Hole - FixNode1Node2Node3Clientcmd:get key: key1 . key12 master_key: group1get key1 . key12Hash using only master_keyMultiGet Hole - FixNode1Node2Node3ClientRes
10、ult v1,v4,v7,v10 v2,v5,v8,v11 v3,v6,v9,v12get key1 . key12Problem 3Virtual Packet LossVPLVPL: virtual packet loss No actual packet loss, but vm response time exceeds the retransmission timeoutTwo network-bounded virtual machine put together result in huge get timeout.VPLA normal retransmission consu
11、me 50ms, which exceeds our Memcached timeout. timeout = no result = cache miss Result: another kind of cache miss stormVPL - Fix Split Network-Bound biz on different real machine. Maybe UDP? Maybe fast retransmission?Short Questions?MMMWhat is MMM Perl Message between Monitor & Agent Auto Failover f
12、or M/Sbut MMM is not: SQL router Load BalancerWhats wrong with MMMMMM is 1) fundamentally broken and unsuitable for use as a HA tool 2) absolutely cannot be fixed. Baron Schwartz: http:/www.xaprb. com/blog/2011/05/04/whats-wrong-with-mmm/MMM InternalsMonitor while() process_check_results check_host_
13、states process_commands distribute_role send_status_to_agent s Agent while( read socket) handle_command MMM architectureMonitorSlave vip3Master vip1Slave vip4Master vip2How MMM Do FailoverMonitorSlave vip3Master vip1Slave vip4Master vip2How MMM Do FailoverMonitorSlave vip3Master vip1Slave vip4Master
14、 vip2set global read_only=1How MMM Do FailoverMonitorSlave vip3MasterSlave vip4Master vip2remove VIPHow MMM Do FailoverMonitorSlave vip3MasterSlave vip4Master vip2select MASTER_POS_WAIT()How MMM Do FailoverMonitorSlave vip3MasterSlave vip4Master vip2show master statusHow MMM Do FailoverMonitorSlave
15、vip3MasterSlave vip4Master vip2change master to How MMM Do FailoverMonitorSlave vip3MasterSlave vip4Master vip1&vip2vip1 onlineMMMMMM in DPMMM in DPFrontend Group vip1 & vip2Backend Group vip3 & vip4Job Group vip5Slave vip3 / vip5Master vip1Slave vip4Master vip2MMMProblems in MMMMMM Problem 1set read_only is difficult on busy server set read_only will be blocked by long running SQLMonitorSlave vip3Master vip1Slave vip4Master vip2set global read_only=1MMM Problem 1MonitorSlave vip3Master vip1Slave vip4Master vip2set global read_only=1MMM
