hadoop - Install spark on yarn cluster -

-

i looking guide regarding how install spark on existing virtual yarn cluster.

i have yarn cluster consisting of 2 nodes, ran map-reduce job worked perfect. looked results in log , working fine.

now need add spark installation commands , configurations files in vagrantfile. can't find guide, give me link ?

i used guide yarn cluster

http://www.alexjf.net/blog/distributed-systems/hadoop-yarn-installation-definitive-guide/#single-node-installation

thanks in advance!

i don't know vagrant, have installed spark on top of hadoop 2.6 (in guide referred post-yarn) , hope helps.

installing spark on existing hadoop easy, need install only on one machine. have download 1 pre-built hadoop version it's official website (i guess can use without hadoop version need point direction of hadoop binaries in system). decompress it:

tar -xvf spark-2.0.0-bin-hadoop2.x.tgz -c /opt

now need set environment variables. first in ~/.bashrc (or ~/.zshrc) can set spark_home , add path if want:

export spark_home=/opt/spark-2.0.0-bin-hadoop-2.x export path=$path:$spark_home/bin

also changes take effect can run:

source ~/.bashrc

second need point spark hadoop configuartion directories. set these 2 environmental variables in $spark_home/conf/spark-env.sh:

export hadoop_conf_dir=[your-hadoop-conf-dir $hadoop_prefix/etc/hadoop] export yarn_conf_dir=[your-yarn-conf-dir same last variable]

if file doesn't exist, can copy contents of $spark_home/conf/spark-env.sh.template , start there.

now start shell in yarn mode can run:

spark-shell --master yarn --deploy-mode client

(you can't run shell in cluster deploy-mode)

----------- update

i forgot mention can submit cluster jobs configuration (thanks @juliancienfuegos):

spark-submit --master yarn --deploy-mode cluster project-spark.py

this way can't see output in terminal, , command exits job submitted (not completed).

you can use --deploy-mode client see output right there in terminal testing, since job gets canceled if command interrupted (e.g. press ctrl+c, or session ends)


Comments

Post a Comment

Popular posts from this blog

wordpress - (T_ENDFOREACH) php error -

-
creating wordpress website , i'm getting following php error: parse error: syntax error, unexpected 'endforeach' (t_endforeach) in /home/soralmedia/public_html/wp-content/themes/soral media/woocommerce/myaccount/my-orders.php on line 98 here's code file: <?php /** * orders * * shows recent orders on account page. * * template can overridden copying yourtheme/woocommerce/myaccount/my-orders.php. * * however, on occasion woocommerce need update template files , (the theme developer). * need copy new files theme maintain compatibility. try this. * little possible, happen. when occurs version of template file will. * bumped , readme list important changes. * * @see http://docs.woothemes.com/document/template-structure/ * @author woothemes * @package woocommerce/templates * @version 2.5.0 */ if ( ! defined( 'abspath' ) ) { exit; } $my_orders_columns = apply_filters( 'woocommerce_my_account_my_orders_columns', array( ...
Read more

Export Excel workseet into txt file using vba - (text and numbers with formulas) -

-
i new vba have developed code use in excel move text/numbers txt file. issue have 3 functions need go same txt file. when run first function has set fs = createobject("scripting.filesystemobject") set = fs.createtextfile(module1.nys45uploadfilename, true) a.writeline (str) a.close. the other 2 functions had save. set fs = createobject("scripting.filesystemobject") set = fs.savetextfile(module1.nys45uploadfilename, true) a.writeline (str) a.close. the first function go txt file, other 2 error object doesn't support property or method. when changed 3 create last function goes txt file. can not figure out word use in order make other 2 functions follow txt file in order keyed. the entire code follows private sub addrecord_click() on error goto errhandler module1.nys45uploadfilename = application.getsaveasfilename(filefilter:="textfiles (*.txt), *.txt") if module1.nys45uploadfilename = "false" exit sub ...
Read more

Using django-mptt to get only the categories that have items -

-
sample structure: all uppercase: category mixed case: item root ├── books │   ├── fiction │   │   └── classics │   └── non-fiction ├── clothing └── electronics ├── laptops ├── phones │   ├── apple │   │   ├── iphone 6 │   │   ├── iphone 6 plus │   │   ├── iphone 6s │   │   └── iphone 6s plus │   ├── motorola │   │   ├── moto g4 │   │   ├── moto g4 play │   │   ├── moto g4 plus │   │   └── moto x │   └── samsung └── tablets └── apple i'm trying have index page show "electronics" category. "clothing" category shouldn't show since it's empty , "books" category shouldn't show either since, although has child categories, child categories don't have items. similarly, "electronics" page should show "phones" category. "laptops" category shouldn't show since it's empty , "tablets" category shouldn't show either since...
Read more