Friday, June 10, 2016

Apache Storm 설치하기

프로젝트에서 Apache Storm을 활용할 부분이 생겨서 기능 검증이 필요한 시점이다. AWS에 Apache Storm을 설치한 과정을 정리하고자 한다.
EC2의 Instance Type 및 용도는 아래와 같다.

  • Zookeeper #1 + Nimbus + Supervisor + Storm UI
  • Zookeeper #2 + Supervisor
  • Zookeeper #3 + Supervisor

#공통 (Each Node)
각 Instance에 JDK를 설치한다.
$sudo apt-add-repository ppa:webupd8team/java
$sudo apt-get update
$sudo apt-get install oracle-java8-installer
#Zookeeper Cluster 설치 (Each Node)아래의 command를 수행한다.
$wget http://apache.mirror.cdnetworks.com/zookeeper/stable/zookeeper-3.4.8.tar.gz
$tar xzvf zookeeper-3.4.8.tar.gz
$mv zookeeper-3.4.8 zookeeper
$cd zookeeper
$mkdir data
$cd conf
$cp zoo_sample.cfg zoo.cfg
zoo.cfg파일을 수정한다.

Zookeeper #1 설정
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/home/ubuntu/zookeeper/data
# the port at which the clients will connect
clientPort=2181
server.1=0.0.0.0:2888:3888
server.2=2.2.2.2:2888:3888
server.3=3.3.3.3:2888:3888
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to “0” to disable auto purge feature
#autopurge.purgeInterval=1
Zookeeper #2 설정
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/home/ubuntu/zookeeper/data
# the port at which the clients will connect
clientPort=2181
server.1=1.1.1.1:2888:3888
server.2=0.0.0.0:2888:3888
server.3=3.3.3.3:2888:3888
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to “0” to disable auto purge feature
#autopurge.purgeInterval=1
Zookeeper #3 설정
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/home/ubuntu/zookeeper/data
# the port at which the clients will connect
clientPort=2181
server.1=1.1.1.1:2888:3888
server.2=2.2.2.2:2888:3888
server.3=0.0.0.0:2888:3888
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to “0” to disable auto purge feature
#autopurge.purgeInterval=1
설정이 완료되었으면 Zookeeper를 실행한다. (각 서버별 수행)
$cd /home/ubuntu/zookeeper/bin
$./zkServer.sh start
#Apache Storm 설치
아래의 명령어를 모든 노드에서 수행한다.
$wget http://apache.mirror.cdnetworks.com/storm/apache-storm-1.0.1/apache-storm-1.0.1.tar.gz
$tar xzvf apache-storm-1.0.1.tar.gz
$mv apache-storm-1.0.1 storm
Storm configuration을 설정한다.
$vi storm/conf/storm.yaml
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# “License”); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
########### These MUST be filled in for a storm configuration
storm.zookeeper.servers :
- “1.1.1.1”
- “2.2.2.2”
- “3.3.3.3”
#
nimbus.seeds: [“1.1.1.1”]
#
#
# ##### These may optionally be filled in:
#
## List of custom serializations
# topology.kryo.register:
# — org.mycompany.MyType
# — org.mycompany.MyType2: org.mycompany.MyType2Serializer
#
## List of custom kryo decorators
# topology.kryo.decorators:
# — org.mycompany.MyDecorator
#
## Locations of the drpc servers
# drpc.servers:
# — “server1”
# — “server2”
## Metrics Consumers
# topology.metrics.consumer.register:
# — class: “org.apache.storm.metric.LoggingMetricsConsumer”
# parallelism.hint: 1
# — class: “org.mycompany.MyMetricsConsumer”
# parallelism.hint: 1
# argument:
# — endpoint: “metrics-collector.mycompany.org”
각 노드별로 설정이 완료되었으면, 아래의 과정을 수행한다.

Start Nimbus
$nohup ./storm/bin/storm nimbus >> nimbus.log 2>&1 &
Start Supervisor
슈퍼바이저는 각 노드에서 수행한다.
$nohup ./storm/bin/storm supervisor >> supervisor.log 2>&1 &
Start Storm UI
스톰 UI는 Nimbus 노드에서 수행한다. (Worker 노드에서 수행 금지, 접속은 브라우저에서 IP:8080)
$nohup ./storm/bin/storm ui >> ui.log 2>&1 &
Start Logviwer
각 노드에서 수행한다.
$nohup ./storm/bin/storm logviewer > logviewer.log &
Start Example Topology
Nimbus 노드에서 수행한다.
$./storm/bin/storm jar ../examples/storm-starter/storm-starter-topologies-0.10.0.jar org.apache.storm.starter.RollingTopWords production-topology remote
Kill Example Topology
Nimbus 노드에서 수행한다.
$./storm/bin/storm kill production-topology 1
이제 갖고 놀면 된다.

0 comments:

Post a Comment