hadoop - Install spark on yarn cluster -

-

i looking guide regarding how install spark on existing virtual yarn cluster.

i have yarn cluster consisting of 2 nodes, ran map-reduce job worked perfect. looked results in log , working fine.

now need add spark installation commands , configurations files in vagrantfile. can't find guide, give me link ?

i used guide yarn cluster

http://www.alexjf.net/blog/distributed-systems/hadoop-yarn-installation-definitive-guide/#single-node-installation

thanks in advance!

i don't know vagrant, have installed spark on top of hadoop 2.6 (in guide referred post-yarn) , hope helps.

installing spark on existing hadoop easy, need install only on one machine. have download 1 pre-built hadoop version it's official website (i guess can use without hadoop version need point direction of hadoop binaries in system). decompress it:

tar -xvf spark-2.0.0-bin-hadoop2.x.tgz -c /opt

now need set environment variables. first in ~/.bashrc (or ~/.zshrc) can set spark_home , add path if want:

export spark_home=/opt/spark-2.0.0-bin-hadoop-2.x export path=$path:$spark_home/bin

also changes take effect can run:

source ~/.bashrc

second need point spark hadoop configuartion directories. set these 2 environmental variables in $spark_home/conf/spark-env.sh:

export hadoop_conf_dir=[your-hadoop-conf-dir $hadoop_prefix/etc/hadoop] export yarn_conf_dir=[your-yarn-conf-dir same last variable]

if file doesn't exist, can copy contents of $spark_home/conf/spark-env.sh.template , start there.

now start shell in yarn mode can run:

spark-shell --master yarn --deploy-mode client

(you can't run shell in cluster deploy-mode)

----------- update

i forgot mention can submit cluster jobs configuration (thanks @juliancienfuegos):

spark-submit --master yarn --deploy-mode cluster project-spark.py

this way can't see output in terminal, , command exits job submitted (not completed).

you can use --deploy-mode client see output right there in terminal testing, since job gets canceled if command interrupted (e.g. press ctrl+c, or session ends)


Comments

Post a Comment

Popular posts from this blog

wordpress - (T_ENDFOREACH) php error -

-
creating wordpress website , i'm getting following php error: parse error: syntax error, unexpected 'endforeach' (t_endforeach) in /home/soralmedia/public_html/wp-content/themes/soral media/woocommerce/myaccount/my-orders.php on line 98 here's code file: <?php /** * orders * * shows recent orders on account page. * * template can overridden copying yourtheme/woocommerce/myaccount/my-orders.php. * * however, on occasion woocommerce need update template files , (the theme developer). * need copy new files theme maintain compatibility. try this. * little possible, happen. when occurs version of template file will. * bumped , readme list important changes. * * @see http://docs.woothemes.com/document/template-structure/ * @author woothemes * @package woocommerce/templates * @version 2.5.0 */ if ( ! defined( 'abspath' ) ) { exit; } $my_orders_columns = apply_filters( 'woocommerce_my_account_my_orders_columns', array( ...
Read more

Export Excel workseet into txt file using vba - (text and numbers with formulas) -

-
i new vba have developed code use in excel move text/numbers txt file. issue have 3 functions need go same txt file. when run first function has set fs = createobject("scripting.filesystemobject") set = fs.createtextfile(module1.nys45uploadfilename, true) a.writeline (str) a.close. the other 2 functions had save. set fs = createobject("scripting.filesystemobject") set = fs.savetextfile(module1.nys45uploadfilename, true) a.writeline (str) a.close. the first function go txt file, other 2 error object doesn't support property or method. when changed 3 create last function goes txt file. can not figure out word use in order make other 2 functions follow txt file in order keyed. the entire code follows private sub addrecord_click() on error goto errhandler module1.nys45uploadfilename = application.getsaveasfilename(filefilter:="textfiles (*.txt), *.txt") if module1.nys45uploadfilename = "false" exit sub ...
Read more

delphi - cxGrid : instead of "go to next cell on enter" using "enter" do that with the arrow keys -

-
Image
is there way can achieve cxgrid's functionality "go next cell on enter" using arrow keys on keyboard instead of hitting "enter" ??? right can navigate arrow keys cells move not selected (blue color). if edit cell in column , move cell beneath (with arrow key) , grid jumps other random record in grid,which annoying. is question of settings or must program functionality ? edit : simple temporary table use gather data. has no indexed fields or autoinc field : procedure tform3.cxbutton1click(sender: tobject); begin datamodule2.acrquery2.close; datamodule2.acrquery2.sql.clear; datamodule2.acrquery2.sql.text :='insert temp select company_id,member_id,surname,name members company_id =:a1'; datamodule2.acrquery2.params.parambyname('a1').value :=cxlookupcombobox2.text; datamodule2.acrquery2.execsql; datamodule2.acrquery3.close; datamodule2.acrquery3.sql.clear; datamodule2.acrquery3.sql.text :='update temp set month=:a1,year=:a2'; data...
Read more