다시 라인으로 돌아가면 당초 네이버재팬에서는 라인의 아키텍쳐로 레디스(Redis)를 사용했습니다. 레디스는 NoSQL 종류 중 하나입니다.

네이버재팬은 동기, 비동기가 자유롭고, 슬레이 복제도 가능하다는 레디스의 장점을 적극 살렸습니다. 그러나 레디스의 단점인 데이터 저장공간의 확장이 힘들다는 것을 간과했지요.

처음 네이버재팬에서는 라인의 사용자가 많아봤자 100만명이 안될 것이라고 예상했다고 합니다. 그러나 반년 만에 500만명의 사용자가 넘어서면서 기존에 쓰던 레디스 클러스터를 확장할 것인지, 아키텍쳐를 뜯어고칠 것인지를 고민하게 됐습니다.

그 과정에서 네이버재팬은 새로운 NoSQL을 사용하기로 결정하고 후보로 HBase, 카산드라, 몽고DB(MongoDB) 중 하나를 선택하기로 합니다. 선택기준은 모바일메신저에서 가장 중요한 세가지, 즉 확장성과 가용성, 비용이었습니다.

네이버재팬은 이 중 하둡 파일 시스템 위에서 빠르게 동작할 수 있는 HBase를 선택해 마이그레이션합니다. 데이터 저장과 가용성부분에서 카산드라가 다른 두가지 NoSQL을 제압했지만, 전체적인 요구사항을 HBase가 만족스러웠기 때문이라고 합니다.

라인은 레디스에서 HBase로 마이그레이션한 이후 더 빨라졌습니다. 클러스터를 공유할 수 있을 뿐더러 읽고 쓰는 것에 대한 균형 조정 기능도 갖추고 있어 한꺼번에 많은 데이터가 들어오더라도 해결할 수 있기 때문입니다.

기존 서비스의 한계를 클라우드와 오픈소스로 해결하고, 이 과정을 공개한 것은 동종업계에도 좋은 영향을 끼칠 것으로 보입니다.

[원문출처 : http://www.ddaily.co.kr/news/news_view.php?uid=90737]

Hi, I’m Shunsuke Nakamura (@sunsuk7tp). Just half a year ago, I completed the Computer Science Master’s program in Tokyo Tech and joined to NHN Japan as a member of LINE server team. My ambition is to hack distributed processing and storage systems and develop the next generation’s architecture.

In the LINE server team, I’m in charge of development and operation of the advanced storage system which manages LINE’s message, contacts and groups.

Today, I’ll briefly introduce the LINE storage stack.

LINE Beginning with Redis [2011.6 ~]

In the beginning, we adopted Redis for LINE’s primary storage. LINE is targeted for an instant messenger quickly exchanging messages, and the scale had been assumed to at most total 1 million registered users within 2011. Redis is an in-memory data store and does its intended job well. Redis also enables us to take snapshots periodically on disk and supports sync/asynchronous master-slave replication. We decided that Redis was the best choice despite the scalability and availability issues caused by the in-memory data store. The entire LINE storage system started with just a single Redis cluster constructed from 3 nodes sharded on client-side.

The larger the scale of the service, the more nodes were needed, and client-side sharding prevented us from scaling effectively. The original Redis still doesn’t support server-side sharding. So far, we have achieved a sharded redis cluster to utilize our developed clustering manager. Our sharded redis cluster is coordinated by the cluster manager daemons and ZooKeeper quorum servers.

This manager has the following characteristics:

• Sharding management by ZooKeeper (Consistent hashing, compatible with other algorithms)
• Failure detection and auto/manual failover between master and slave
• Scales out with minimal downtime (< 10 sec)

Currently, several sharded Redis clusters are running with hundreds of servers.

Sharded Redis Cluster and Management tool

Tolerance Unpredictable Scaling [2011.10 ~]

However, the situation has changed greatly since then. Around October 2011, LINE experienced tremendous growth in many parts of the world, and operating costs increased as well.

A major issue of increased costs is to scale Redis Cluster in terms of capability. It’s much more difficult to operate Redis cluster to tolerance the unpredictable scale expansion because it needs more servers than the other persistent storages for the nature of in-memory data store. In order to take advantage of safely availability functionalities such as snapshot and full replication, it is necessary to adequately care of memory usage. Redis VM (Virtual Memory) is to somewhat helpful but can significantly impair performance depending on VM usage.

For the above reasons, we often misjudged the timing to scale out and encountered some outages. It then became critical to migrate to a more highly scalable system with high availability.

Over night, the target of LINE has been changed to scale 10s to 100s of millions of registered users.

This is how we tackled the problem.

Data Scalability

At first, we analyzed the order of magnitude for each database.

(n: # of Users)
(t: Lifetime of LINE System)

• O(1)
• Messages in delivery queue
• Asynchronous jobs in job queue
• O(n)
• User Profile
• Contacts / Groups
• These data originally increase with O(n^2), but there are limitations on the number of links between users. (= O (n * CONSTANT_SIZE))
• O(n*t)
• Messages in Inbox
• Change-sets of User Profile / Groups / Contacts

Rows stored in LINE storage have increased exponentially. In the near future, we will deal with tens of billions of rows per month.

Data Requirement

Second, we summarized our data requirements for each usage scenario.

• O(1)
• Availability
• Workload: very fast reads and writes
• O(n)
• Availability, Scalability
• Workload: fast random reads
• O(n*t)
• Scalability, Massive volume (Billions of small rows per day, but mostly cold data)
• Workload: fast sequential writes (append-only) and fast reads of the latest data

Choosing Storage

Finally, according to the above requirements for each storage, we chose the suitable storage. As one of the criteria to configure each storage properties and determine which storage is most suitable for LINE app workloads, we benchmarked several candidates using tools such as YCSB (Yahoo! Cloud Serving Benchmark) and our own original benchmark to simulate their workloads. As a result, we decided to use HBase as the primary storage method for storing data with the exponential growth patterns such as message timeline. The characteristics of HBase are suitable for message timeline, whose workload is the latest workload, where the most recently inserted records are in the head of the distribution.

• O(1)
• Redis is the best choice.
• O(n), O(n*t)
• There are several candidates.
• HBase
• pros:
• Best matches our requirements
• Easy to operate (Storage system built on DFS, multiple ad hoc partitions per server)
• cons:
• Random read and deletion are somewhat slow.
• Slightly lower avaiability (there’re some SPOF)
• Cassandra (My favorite NoSQL)
• pros:
• Also suitable for dealing with the latest workload
• High Availability (decentralized architecture, rack/DC-aware replication)
• cons:
• High operation costs due to weak consistency
• Counter increments are expected to be slightly slower.
• MongoDB
• pros:
• Auto sharding, auto failover
• A rich range of operations (but LINE storage doesn’t require most of them.)
• cons:
• NOT suitable for the timeline workload (B-tree indexing)
• Ineffective disk and network utilization

In summary, LINE storage layer is currently constructed as the follows:

1. Standalone Redis: asynchronous job and message queuing
• Redis queue and queue dispatcher are running together on each application server.
2. Sharded Redis: front-end cache for data with O(n*t) and primary storage with O(n)
3. Backup MySQL: secondary storage (for backup, statistics)
4. HBase: primary storage for data with O(n*t)
• We assume to operate hundreds of terabytes of data on each cluster with 100s to 1000 servers.

LINE main storage is constructed from about 600 nodes and continues to increase month after month.

LINE Storage Stack

Data Migration from Redis to HBase

We gradually migrated tens of terabytes worth of data sets from Redis cluster to HBase cluster. Specifically, we migrated in three phases:

1. Bi-directional write to Redis and HBase and read only from Redis
2. Run migrating script on backend (Sequentially retrieve data from Redis and write to HBase)
3. Write to both Redis (w/ TTL) and HBase (w/o TTL) and bi-directional read from both (Redis alternatives to a cache server.)

Something to make note of is that one should avoid overwriting recent data with the older data; the migrated data are most append-only and the consistency of the other data are kept using timestamp of HBase column.

HBase and HDFS

A number of HBase clusters have been running stably for the most part on HDFS. We constructed a HBase cluster for each database (e.g., messages, contacts) and each cluster is tuned according to the workload of each database. They share a single HDFS cluster consisting of 100 servers, where each server has 32GB of memory and 1.5TB of hard disk space. Each RegionServer has 50 small regions less than a single 10GB one. Read performance for Bigtable-like architecture is impacted by (major) compaction, so each region’s size should be kept not too large size to prevent continuous major compaction, especially during peak hours. During off-peak hours, large regions are automatically split into smaller regions by a periodic cron job, while operators manually perform load balancing. Of course, HBase has auto splitting and load balancing functionalities, but we consider it best to set up manually in view of service requirements.

Thus the growth of the service, scalability is one of the important issues. We plan to place at most hundreds of servers per cluster. Each message has TTL and it is partitioned to multi-cluster in units of TTL. By doing so, the old cluster, where all of messages have expired, is full-truncated and enables to be reused as a new cluster.

Current and future challenges [2012]

Since migrating to HBase, LINE storage has been operating more stably. Each HBase cluster is current processing several times as requests as during New Year peak time. Even still, there are sometimes failures due to storage. We are left with the following availability issues for HBase and Redis cluster.

• A redundant configuration and failover feature that does not include a single point of failure for each component including rack/DC-awareness
• We examine replication in various layers such as full replication and SSTable or block level replication between HDFS clusters.
• Compensation for the failures between clusters (Redis cluster, HBase, and multi-HBase cluster)

HA-NameNode

As you may already know, the NameNode is a single point of failure for HDFS. Though the NameNode process itself rarely fails (Notes: Experience at Yahoo!), other software failures or hardware failures such as disk and network failures are bound to occur. A NameNode failover procedure is thus required in order to achieve high availability.

There are the several HA-NameNode configurations:

• High Availability Framework for HDFS NameNode (HDFS-1623)
• Backup NameNode (0.21)
• Avatar NameNode (Facebook)
• HA NameNode using Linux HA
• Active/passive configuration deploying two NameNode (cloudera)

We configure HA-NameNode using Linux HA. Each component of Linux-HA has a role similar to the following:

• DRBD: Disk mirroring
• Heartbeat / (Corosync): Network fail-detector
• Pacemaker: Failover definition
• service: NameNode, Secondary NameNode, VIP

HA NameNode using Linux HA

DRBD (Distributed Replicated Block Device) provides block level replication; essentially it’s network-enabled RAID driver. Heartbeat monitors the status of the network between the other server. If Heartbeat detects hardware or service outages, it switches primary/secondary in DRBD and kicks each service’s daemon based on logic defined by pacemaker.

Conclusion

Thus far, we’ve faced various challenges for scalability and availability with the growth of LINE. However, LINE storage and strategies will be much more immature, given extreme scaling and the various failure cases. We would like to grow ourselves with the future growth of LINE.

Appendix: How to setup HA-NameNode using Linux-HA

In the rest of this entry, I will introduce how to build HA-NameNode using two CentOS 5.4 servers and Linux-HA. These servers are to assume the following environment.

• Hosts:
1. NAMENODE01: 192.168.32.1 (bonding)
2. NAMENODE02: 192.168.32.2 (bonding)
• OS: CentOS 5.4
• DRBD (v8.0.16):
• conf file: ${DEPLOY_HOME}/ha/drbd.conf
• resource name: drbd01
• mount disk: /dev/sda3
• mount device: /dev/drbd0
• mount directory: /data/namenode
• Heartbeat (v3.0.3):
• conf file: ${DEPLOY_HOME}/ha/haresources, authkeys
• Pacemaker (v1.0.12)

• service daemons
• VIP: 192.168.32.3
• Hadoop NameNode, SecondaryNameNode (v1.0.2, the latest edition now)

Configuration

Configure drbd and heartbeat settings in your deploy home direcoty, ${DEPLOY_HOME}.

• drbd.conf

• ha.conf

• haresources (Can skip this step when using pacemaker)

• authkeys

Installation of Linux-HA

Pacemaker and Heartbeat3.0 packages are not included in the default base and updates repositories in CetOS5. Before installation, you first need to add the Cluster Labs repo:

Then run the following script:

DRBD Initialization and Running heartbeat

1. Run drbd service @ primary and secondary

1	# service drbd start

2. Initialize drbd and format NameNode@primary

1 2 3 4 5 6

# drbdadm -- --overwrite-data-of-peer primary drbd01 # mkfs.ext3 /dev/drbd0 # mount /dev/drbd0 $ hadoop namenode -format # umount /dev/drbd0 # service drbd stop

3. Run heartbeat @ primary and secondary

1	# service heartbeat start

Daemonize hadoop processes (Apache Hadoop)

When using Apache Hadoop, you need to daemonize each node such as NameNode, SecondaryNameNode in order for heartbeat process to kick them. The follow script, “hadoop-1.0.2-namenode” is an example for NameNode daemon.

• /usr/lib/ocf/resource.d/nhnjp/hadoop-1.0.2-namenode

Second, place a script for pacemaker to kick this daemon services. There are pacemaker scripts under /usr/lib/ocf/resource.d/ .

• /usr/lib/ocf/resource.d/nhnjp/Hadoop

Pacemaker settings

First, using the crm_mon command, verify whether the heartbeat process is running.

After verifying the process is running, connect to pacemaker using the crm command and configure its resource settings. (This step is needed instead of haresource setting)

Here, ocf:${GROUP}/${SERVICE} path corresponds with /usr/lib/ocf/resource.d/${GROUP}/${SERVICE}. So you should place your original service script there. Also lsb:${SERVICE} path corresponds with /etc/init.d/${SERVICE}.

Finnaly, you can confirm pacemaker’s settings using the show command.

Once you’ve confirmed the configuration is correct, commit it using the commit command.

Once you’ve run the commit command, heartbeat kicks each service following pacemaker’s rules.
You can monitor dead or alive using the crm_mon command.

Finally, you should test the various failover tests. For example, kill each service daemon and cause pseudo-network failures using iptables.

Reference documents

• HA document of cloudera
• Pacemacker 1.0 document
• HDFS NameNode High Availability by Hortonworks

출처 - http://tech.naver.jp/blog/?p=1420

HTTPS, TCP를 통한 SSL 및 SOAP 보안의 인증서 유효성 검사 차이점

.NET Framework 4.5

다른 버전

이 항목은 아직 평가되지 않았습니다.- 이 항목 평가

ServicePointManager.ServerCertificateValidationCallback +=
    new RemoteCertificateValidationCallback(ValidateServerCertificate);

public static bool ValidateServerCertificate(
  object sender,
  X509Certificate certificate,
  X509Chain chain,
  SslPolicyErrors sslPolicyErrors)

myServiceHost.Credentials.ClientCertificate.Authentication.
    CertificateValidationMode=
    X509CertificateValidationMode.PeerOrChainTrust;       

myServiceHost.Credentials.ClientCertificate.Authentication.
    RevocationMode=X509RevocationMode.Offline; 

myClient.ClientCredentials.ServiceCertificate.
    Authentication.CertificateValidationMode=
    X509CertificateValidationMode.PeerOrChainTrust;
myClient.ClientCredentials.ServiceCertificate.Authentication.
    RevocationMode = X509RevocationMode.Offline;

출처 - http://msdn.microsoft.com/ko-kr/library/aa702579.aspx

Fedora 16부터는 서비스(Service) 관리를 systemctl이라는 명령어로 합니다. 이전 명령어들도 사용할 수 있지만 systemctl 사용을 권장하고 있습니다. systemctl 명령어의 주요 사용법을 정리합니다.

서비스 상태 확인 하기

서비스 상태 확인은 다음과 같이 합니다.

예를 들어 httpd의 상태를 확인하려면 다음과 같이 명령합니다.

서비스 시작, 중지, 재시작 하기

서비스를 시작하는 기본 명령 방식은 다음과 같습니다.

중지할 때에는 start 대신 stop, 재시작할 때에는 restart를 쓰면 됩니다. 예를 들어 httpd를 시작하려면 다음과 같이 명령한다.

중지와 재시작은 다음과 같이 명령하면 된다.

부팅 시 자동 실행 하기

시스템 부팅 시 서비스가 자동 실행되게 하는 명령은 다음과 같습니다.

예를 들어 httpd를 부팅 시 자동 실행되게 하려면 다음과 같이 명령합니다.

자동 실행 되지 않게 하려면 enable 대신 disable을 쓰면 됩니다.

실행 중인 서비스 보기

실행 중인 모든 서비스 목록을 보려면 다음과 같이 명령합니다.

출처 - http://www.cmsfactory.net/node/62

systemd 시작하기

출처 - http://joonlinux.blogspot.kr/2012/07/systemd.html