ELK 힙사이즈 확장

#들어가는 말

전에 무식하게 mysql의 데이터를 통으로 로그스태시에 가져왔더니 OOM 뜨면서 뻑이 났었다.

로그스태시에 개별적으로 힙사이즈를 늘려줘서 해결을 해보려했다.

변경 후에는 서버가 안 죽게 되었지만, 공유기가 죽는 헤프닝이 발생했다.

개인적인 생각으론 무식하게 몇 기가 단위로 데이터를 가져오니, 공유기가 못 버틴듯하다.

잡설은 이러하고, 엘라스틱 서치 의 힙사이즈를 확장하는 방법에 대해 알아보자.

구글링 하다가 관련 된, 포스트를 찾았다.

참고로 최대 메모리는 32기가 인데, 자바 힙사이즈는 8기가만 맥스로 잡혀있었다.

아마 기본적으로 잡히는 자바 힙사이즈 맥스가 1/4 정도만 적용되는거 같다는 생각이다.

확인해보니 기본적으로 Max Size의 1/4 정도가 디폴트로 잡힌다 하더나이다.

#엘라스틱서치 힙사이즈 설정

> ES_HEAP_SIZE= 10g ./elasticsearch 실행

Elasticsearch: The Definitive Guide [2.x] » Administration, Monitoring, and Deployment » Production Deployment » Heap: Sizing and Swapping

«  Don’t Touch These Settings!     File Descriptors and MMap  »

Heap: Sizing and Swappingedit

The default installation of Elasticsearch is configured with a 1 GB heap. For just about every deployment, this number is usually too small. If you are using the default heap values, your cluster is probably configured incorrectly.

There are two ways to change the heap size in Elasticsearch. The easiest is to set an environment variable called ES_HEAP_SIZE. When the server process starts, it will read this environment variable and set the heap accordingly. As an example, you can set it via the command line as follows:

export ES_HEAP_SIZE=10g

Alternatively, you can pass in the heap size via JVM flags when starting the process, if that is easier for your setup:

ES_JAVA_OPTS="-Xms10g -Xmx10g" ./bin/elasticsearch

Ensure that the min (Xms) and max (Xmx) sizes are the same to prevent the heap from resizing at runtime, a very costly process.

Generally, setting the ES_HEAP_SIZE environment variable is preferred over setting explicit -Xmx and -Xms values.

Give (less than) Half Your M

#로그스태시 힙사이즈 설정

> LS_HEAP_SIZE= 10g ./logstash -f 실행

How to change Logstash's default max heap size

To change Logstash's default max heap size, do as follows:

LS_HEAP_SIZE=4g ./bin/logstash -f generator.conf

You can check if it works with `jps -v` as follows:

$ jps -v

15582 Main -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Xmx4g -Xss2048k -Djffi.boot.library.path=/home/izeye/programs/logstash-2.3.4/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/izeye/programs/logstash-2.3.4/heapdump.hprof -Xbootclasspath/a:/home/izeye/programs/logstash-2.3.4/vendor/jruby/lib/jruby.jar -Djruby.home=/home/izeye/programs/logstash-2.3.4/vendor/jruby -Djruby.lib=/home/izeye/programs/logstash-2.3.4/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh

15646 Jps -Dapplication.home=/home/izeye/programs/jdk1.8.0_45 -Xms8m

$

You can see `-Xmx4g` (ie. 4GB).