Spark 설치

Download binary

cd /path/to/spark/download

wget http://apache.mirror.cdnetworks.com/spark/spark-3.0.0-preview2/spark-3.0.0-preview2-bin-hadoop3.2.tgz.

tar zxvf spark-3.0.0-preview2-bin-hadoop3.2.tgz

ln -s spark-3.0.0-preview2-bin-hadoop3.2 default

Setting Environment Variables

export SPARK_HOME="/path/to/spark/download/default"

export SPARK_CONF_DIR="${SPARK_HOME}/conf"

export PATH="${PATH}:${SPARK_HOME}/bin:${SPARK_HOME}/sbin"

spark-defaults.conf

vim ${SPARK_CONF_DIR}/spark-defaults.conf

spark.master local[1]

spark.eventLog.enabled true

spark.eventLog.dir /path/to/spark/eventLog spark.serializer org.apache.spark.serializer.KryoSerializer spark.driver.memory 2g

HDFS, SPARK 실행

hdfs 실행 및 확인

# hdfs 실행

start-dfs.sh

# spark 실행

start-all.sh

# hdfs 올라왔는지 확인

hdfs dfs -ls /

# hdfs option 확인

hdfs -help

# spark-shell 실행

spark-shell

# spark-shell option 확인

spark-shell -h help

# spark-shell --master "local[*]" --executor-memory 32G --driver-memory 32G (환경을 고려하여 옵션 설정)