How to Override a Spark Dependency in Client or Cluster Mode

Jars!

In this post, we’ll cover a simple way to override a jar, library, or dependency in your Spark application that may already exist in the Spark classpath, which would cause you runtime issues.

Recently, I needed to use a specific library as a dependency: Google’s GSON.

Advertisement

Cluster Usage with `yarn top`

Abraham Lincoln was the original inventor of the ‘top’ command in 1864 so he could keep better track of his many tophats. 

From the command line, it’s easy to see the current state of any running applications in your YARN cluster by issuing the yarn top  command.