diff options
| author | pgandhi <pgandhi@yahoo-inc.com> | 2017-08-28 08:51:22 -0500 |
|---|---|---|
| committer | Tom Graves <tgraves@yahoo-inc.com> | 2017-08-28 08:51:22 -0500 |
| commit | 24e6c187fbaa6874eedbdda6b3b5dc6ff9e1de36 (patch) | |
| tree | 8847bdef04ae6f8929feb8960996361f28e8a2ad /conf | |
| parent | 0456b4050817e64f27824720e695bbfff738d474 (diff) | |
[SPARK-21798] No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server
History Server Launch uses SparkClassCommandBuilder for launching the server. It is observed that SPARK_CLASSPATH has been removed and deprecated. For spark-submit this takes a different route and spark.driver.extraClasspath takes care of specifying additional jars in the classpath that were previously specified in the SPARK_CLASSPATH. Right now the only way specify the additional jars for launching daemons such as history server is using SPARK_DIST_CLASSPATH (https://spark.apache.org/docs/latest/hadoop-provided.html) but this I presume is a distribution classpath. It would be nice to have a similar config like spark.driver.extraClasspath for launching daemons similar to history server.
Added new environment variable SPARK_DAEMON_CLASSPATH to set classpath for launching daemons. Tested and verified for History Server and Standalone Mode.
## How was this patch tested?
Initially, history server start script would fail for the reason being that it could not find the required jars for launching the server in the java classpath. Same was true for running Master and Worker in standalone mode. By adding the environment variable SPARK_DAEMON_CLASSPATH to the java classpath, both the daemons(History Server, Standalone daemons) are starting up and running.
Author: pgandhi <pgandhi@yahoo-inc.com>
Author: pgandhi999 <parthkgandhi9@gmail.com>
Closes #19047 from pgandhi999/master.
Diffstat (limited to 'conf')
| -rwxr-xr-x | conf/spark-env.sh.template | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/conf/spark-env.sh.template b/conf/spark-env.sh.template index 1663019ee5..f8c895f530 100755 --- a/conf/spark-env.sh.template +++ b/conf/spark-env.sh.template @@ -52,6 +52,7 @@ # - SPARK_HISTORY_OPTS, to set config properties only for the history server (e.g. "-Dx=y") # - SPARK_SHUFFLE_OPTS, to set config properties only for the external shuffle service (e.g. "-Dx=y") # - SPARK_DAEMON_JAVA_OPTS, to set config properties for all daemons (e.g. "-Dx=y") +# - SPARK_DAEMON_CLASSPATH, to set the classpath for all daemons # - SPARK_PUBLIC_DNS, to set the public dns name of the master or workers # Generic options for the daemons used in the standalone deploy mode |