diff options
| author | Kent Yao <yaooqinn@hotmail.com> | 2017-11-09 14:33:08 +0900 |
|---|---|---|
| committer | hyukjinkwon <gurwls223@gmail.com> | 2017-11-09 14:33:08 +0900 |
| commit | ee571d79e52dc7e0ad7ae80619c12a5c0b90b5a5 (patch) | |
| tree | 538400cc2f8faefa32fdcbbaa82914f188132ca2 /conf | |
| parent | 6447d7bc1de4ab1d99a8dfcd3fea07f5a2da363d (diff) | |
[SPARK-22466][SPARK SUBMIT] export SPARK_CONF_DIR while conf is default
## What changes were proposed in this pull request?
We use SPARK_CONF_DIR to switch spark conf directory and can be visited if we explicitly export it in spark-env.sh, but with default settings, it can't be done. This PR export SPARK_CONF_DIR while it is default.
### Before
```
KentKentsMacBookPro ~/Documents/spark-packages/spark-2.3.0-SNAPSHOT-bin-master bin/spark-shell --master local
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/11/08 10:28:44 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/11/08 10:28:45 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
Spark context Web UI available at http://169.254.168.63:4041
Spark context available as 'sc' (master = local, app id = local-1510108125770).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.3.0-SNAPSHOT
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_65)
Type in expressions to have them evaluated.
Type :help for more information.
scala> sys.env.get("SPARK_CONF_DIR")
res0: Option[String] = None
```
### After
```
scala> sys.env.get("SPARK_CONF_DIR")
res0: Option[String] = Some(/Users/Kent/Documents/spark/conf)
```
## How was this patch tested?
vanzin
Author: Kent Yao <yaooqinn@hotmail.com>
Closes #19688 from yaooqinn/SPARK-22466.
Diffstat (limited to 'conf')
| -rwxr-xr-x | conf/spark-env.sh.template | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/conf/spark-env.sh.template b/conf/spark-env.sh.template index f8c895f530..bc92c78f0f 100755 --- a/conf/spark-env.sh.template +++ b/conf/spark-env.sh.template @@ -32,7 +32,8 @@ # - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and RDD data # - MESOS_NATIVE_JAVA_LIBRARY, to point to your libmesos.so if you use Mesos -# Options read in YARN client mode +# Options read in YARN client/cluster mode +# - SPARK_CONF_DIR, Alternate conf dir. (Default: ${SPARK_HOME}/conf) # - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files # - YARN_CONF_DIR, to point Spark towards YARN configuration files when you use YARN # - SPARK_EXECUTOR_CORES, Number of cores for the executors (Default: 1). |