Apache Spark is an Open source cluster computing framework originally developed in the AMPLab at UC Berkeley. In contrast to Hadoop's two-stage disk-based MapReduce paradigm, Spark's in-memory primitives provide performance up to 100 times faster for certain applications.

특징

• 스토리지를 경유하는 Hadoop에 비해 인메모리 처리를 기본으로 하여 보다 빠름
• RDD(Resilient distributed datasets) 읽기 전용 데이터 구조 기반하며, 연산은 RDD에서 RDD로의 변환을 연쇄적으로 실시함으로써 이루어짐

Components

1. Spark Core and Resilient Distributed Datasets
2. Spark SQL
3. Spark Streaming
4. MLlib Machine Learning Library
5. GraphX

관련자료

• Best Practices Writing Production-Grade PySpark Jobs - How to Structure Your PySpark Job Repository and Code 2017-01-24
• Using Apache Spark to Analyze Large Neuroimaging Datasets 2016-08-25: 뇌영상 자료 분석 - with scikit-learn and PySpark
• Spark Programming 엄태욱 2015-10-29
• 10 things you need to know about Spark 2015-05-??
• Simplifying Big Data Analysis with Apache Spark 2015-04-27
• Spark은 왜 이렇게 유명해지고 있을까? 2015-04-23
• Configuring IPython Notebook Support for PySpark 2015-02-01: IPython Notebook setting for Spark
• Spark의 핵심은 무엇인가? RDD! 하용호 2014-12-08

Incoming Links #

Related Articles (Article 0) #

• Big data
• Data analysis
• Data science

Related Codes (Code 1) #

• Hail

Suggested Pages #

• 0.174 SNV
• 0.110 Large language model
• 0.087 Python for Scientists
• 0.078 DVC
• 0.062 GenePattern
• 0.058 Machine learning
• 0.039 Genetics
• 0.025 Programming language
• 0.025 2001
• 0.022 Hyuna Kim
• More suggestions...