Apache Spark

IntelliJ Spark Scala IDE Setup

• File > New > Project
  • Type: Scala > Sbt
  • Versions: SBT (0.12.0), Scala (2.10.5)
• Create File Project > assembly.sbt
  • addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")
• Create File build.sbt (example content), pom.xml not needed for scala

  name := "mockups"
  version := "1.0"
  
  scalaVersion := "2.10.5"
  crossScalaVersions := Seq(scalaVersion.value)
  
  val sparkV = "1.6.1"
  libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % sparkV,
    "org.apache.spark" %% "spark-streaming" % sparkV,
    "org.apache.spark" %% "spark-mllib" % sparkV,
    "org.apache.spark" %% "spark-sql" % sparkV
  )
  
  test in assembly := {}
  assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
  

• Add library Dependencies
  • File > Project Structure > Modules > Dependencies
    • '+ sign' > JARS or Directories > /spark-assembly-1.6.1-hadoop2.6.0.jar

Resources

• Spark Tuning Cheatsheet
• {spark.tc}, Github {spark.tc}
• DataPalooza
• SPARKHUB Databricks
• Advanced Spark and TenserFlow
• Zeppelin Hub
• Personal YouTube Playlist on Apache Spark