| 일 | 월 | 화 | 수 | 목 | 금 | 토 |
|---|---|---|---|---|---|---|
| 1 | 2 | 3 | ||||
| 4 | 5 | 6 | 7 | 8 | 9 | 10 |
| 11 | 12 | 13 | 14 | 15 | 16 | 17 |
| 18 | 19 | 20 | 21 | 22 | 23 | 24 |
| 25 | 26 | 27 | 28 | 29 | 30 | 31 |
- DeepLearning
- Hadoop Ecosystems
- Feature extraction
- Hive CLI
- Clodera Quick Start
- Spark
- Managed Table
- 딥러닝
- Data 불러오기
- One-hot encoding
- 차원의 저주
- hive
- L2 정규화
- Tez
- 선형 회귀
- Haddop
- Over fitting
- 캐글
- Manifold Learning
- hive setting
- learning rate
- Auto Encoder
- data load
- Spark Data 불러오기
- Deeplearning 키워드
- Spark 설치
- 스파크
- 하이브
- .hiverc
- 학습률
- Today
- Total
목록스파크 (2)
남는건 기록뿐
Spark Basic Function
Check SCHEMA data.printSchema Check COLUMNS data.columns SELECT val pracData = data.select($"fecha_dato", $"ncodpers", $"ind_empleado") //change schema, name of columns pracData.select($"fecha_dato".cast("timestamp") as "Date", $"ncodpers".cast("string") as "CutomerNum", $"ind_empleado".cast("string") as "employed") // change only a name of a column pracData.withColumnRenamed("Date", "DateTime")..
Spark 설치
Download binary cd /path/to/spark/download wget http://apache.mirror.cdnetworks.com/spark/spark-3.0.0-preview2/spark-3.0.0-preview2-bin-hadoop3.2.tgz. tar zxvf spark-3.0.0-preview2-bin-hadoop3.2.tgz ln -s spark-3.0.0-preview2-bin-hadoop3.2 default Setting Environment Variables export SPARK_HOME="/path/to/spark/download/default" export SPARK_CONF_DIR="${SPARK_HOME}/conf" export PATH="${PATH}:${SP..