python - How to resolve : Very large size tasks in spark -

-

here pasting python code running on spark in order perform analysis on data. able run following program on small amount of data-set. when coming large data-set, saying "stage 1 contains task of large size (17693 kb). maximum recommended task size 100 kb". 

import os import sys import unicodedata operator import add  try:     pyspark import sparkconf     pyspark import sparkcontext except importerror e:     print ("error importing spark modules", e)     sys.exit(1)  def tokenize(text):     resultdict = {}     text = unicodedata.normalize('nfkd', text).encode('ascii','ignore')      str1= text[1]     str2= text[0]      arrtext= text.split(str1)      ss1 = arrtext[0].split("/")      docid = ss1[0].strip()      docname = ss[1].strip()      resultdict[docid+"_"+docname] = 1      return resultdict.iteritems()  sc=sparkcontext('local') textfile = sc.textfile("path data") filecontent = textfile.flatmap(tokenize) rdd = sc.parallelize(filecontent.collect()) rdd= rdd.map(lambda x: (x[0], x[1])).reducebykey(add) print rdd.collect() #reducebykey(lambda a,b: a+b) rdd.coalesce(1).saveastextfile("path result")

here posting little more warning: job not running after this. can 1 me this.

16/06/10 19:19:58 warn tasksetmanager: stage 1 contains task of large size (17693 kb). maximum recommended task size 100 kb. 16/06/10 19:19:58 info tasksetmanager: starting task 0.0 in stage 1.0 (tid 5314, localhost, partition 0,process_local, 18118332 bytes) 16/06/10 19:19:58 info executor: running task 0.0 in stage 1.0 (tid 5314) 16/06/10 19:43:00 info blockmanagerinfo: removed broadcast_1_piece0 on localhost:43480 in memory (size: 3.9 kb, free: 511.1 mb) 16/06/10 19:43:00 info contextcleaner: cleaned accumulator 2

when spark serializes tasks, recursively serializes full closure context. in case, logical culprit seems unicodedata use in tokenize. may wrong don't see other heavy data structures in code. (caveat, typically use spark scala , python rusty.) wonder whether library backed heavy data structures not available on executor nodes.

the typical patterns dealing these types of problems are:

  1. make sure libraries available on executor nodes.

  2. use broadcast variables distribute heavy data structures executors.

unrelated, unless using debugging tool, doing unnecessary collection of data driver collect. transformation can chained:

sc.textfile(...).flatmap(...).map(...).reducebykey(add).coalesce(1).saveastextfile(...)

Comments

Post a Comment

Popular posts from this blog

wordpress - (T_ENDFOREACH) php error -

-
creating wordpress website , i'm getting following php error: parse error: syntax error, unexpected 'endforeach' (t_endforeach) in /home/soralmedia/public_html/wp-content/themes/soral media/woocommerce/myaccount/my-orders.php on line 98 here's code file: <?php /** * orders * * shows recent orders on account page. * * template can overridden copying yourtheme/woocommerce/myaccount/my-orders.php. * * however, on occasion woocommerce need update template files , (the theme developer). * need copy new files theme maintain compatibility. try this. * little possible, happen. when occurs version of template file will. * bumped , readme list important changes. * * @see http://docs.woothemes.com/document/template-structure/ * @author woothemes * @package woocommerce/templates * @version 2.5.0 */ if ( ! defined( 'abspath' ) ) { exit; } $my_orders_columns = apply_filters( 'woocommerce_my_account_my_orders_columns', array( ...
Read more

Export Excel workseet into txt file using vba - (text and numbers with formulas) -

-
i new vba have developed code use in excel move text/numbers txt file. issue have 3 functions need go same txt file. when run first function has set fs = createobject("scripting.filesystemobject") set = fs.createtextfile(module1.nys45uploadfilename, true) a.writeline (str) a.close. the other 2 functions had save. set fs = createobject("scripting.filesystemobject") set = fs.savetextfile(module1.nys45uploadfilename, true) a.writeline (str) a.close. the first function go txt file, other 2 error object doesn't support property or method. when changed 3 create last function goes txt file. can not figure out word use in order make other 2 functions follow txt file in order keyed. the entire code follows private sub addrecord_click() on error goto errhandler module1.nys45uploadfilename = application.getsaveasfilename(filefilter:="textfiles (*.txt), *.txt") if module1.nys45uploadfilename = "false" exit sub ...
Read more

Using django-mptt to get only the categories that have items -

-
sample structure: all uppercase: category mixed case: item root ├── books │   ├── fiction │   │   └── classics │   └── non-fiction ├── clothing └── electronics ├── laptops ├── phones │   ├── apple │   │   ├── iphone 6 │   │   ├── iphone 6 plus │   │   ├── iphone 6s │   │   └── iphone 6s plus │   ├── motorola │   │   ├── moto g4 │   │   ├── moto g4 play │   │   ├── moto g4 plus │   │   └── moto x │   └── samsung └── tablets └── apple i'm trying have index page show "electronics" category. "clothing" category shouldn't show since it's empty , "books" category shouldn't show either since, although has child categories, child categories don't have items. similarly, "electronics" page should show "phones" category. "laptops" category shouldn't show since it's empty , "tablets" category shouldn't show either since...
Read more