At first, we need to download and install IntelliJ IDEA (choose the free Community version). We then click Configure, select Plugins to install Scala Plugin. We create a Scala Maven Project in IntelliJ by clicking File -> New -> Project, select Maven and click Next. We then fill in GroupId, ArtifactId and click Finish.

There are two java folders will be created in src/main and src/test. We rename these folder to scala by right-clicking on java, select Refactor -> Rename (or press Shift +F6). Next, we create a Scala file by right-clicking, select New -> Scala Class.

If the Scala class is not displayed, we can do the following steps:

– Right click on Project, select Add Framework Support…, select Scala.

– If the Scala class is still not showing,  right click on package, select Mark Directory As -> Sources Root.

Next, we add Spark Dependencies to the pom.xml file (under <version> section) as follows ( Note: com.thoughtworks.paranamer dependency is added to fix the ArrayIndexOutOfBoundsException: 10582, the remaining dependencies are of Spark):

Finally, we click Import Changes to let IntelliJ updates necessary libraries automatically .

November 8, 2018