amazon ec2 - EC2 suitability for synching large CSV files from an FTP -

-

i have execute task twice per week. task consists on fetching 1.4gb csv file public ftp server. have process (apply filters, discard rows, make calculations) , synch postgres database hosted on aws rds. each row have retrieve sku entry on database , determine wether needs update or not.

my question if ec2 work solution me. main concern memory.. have searched solutions https://github.com/goodby/csv handle issue fetching row row instead of pulling memory, not work if try read .csv directly ftp.

can provide insight? aws ec2 platform solve problem? how deal issue of csv size , memory limitations?

ftp protocol doesn't "streaming". cannot read file ftp chunks chunk.

honestly, downloading file , trigger run bigger instance not big deal if run twice week, choose r3.large (it cost less 0.20/hour ), execute asap , stop it. internal ssd disk space should give best possible i/o compare ebs.

just make sure os , code deployed inside ebs future reuse(unless have automated code deployment mechanism). , must make sure rds handle burst i/o, otherwise become bottleneck.

even better, using r3.large instance, can split csv file smaller chunks, load them in parallel, shutdown instance after finish. need pay minimal root ebs storage cost afterwards.

i not suggest lambda if process lengthy, since lambda mean short , fast processing (it terminate after 300 seconds).

(update): if open file, simple ways parse read sequentially, may not put whole cpu full use. can split of csv file follow reference answer here.

then using same script, can call them simultaneously sending background process, example below show putting python process in background under linux. 

parse_csvfile.py csv1 &  parse_csvfile.py csv2 &  parse_csvfile.py csv3 &

so instead single file sequential i/o, make use of multiple files. in addition, splitting file should snap under ssd.


Comments

  1. Natural gasApril 28, 2018 at 8:06 AM

    Good article. I really found this information very useful and found may answers for my questions on EBS like EBS storage cost.

    ReplyDelete

Post a Comment

Popular posts from this blog

wordpress - (T_ENDFOREACH) php error -

-
creating wordpress website , i'm getting following php error: parse error: syntax error, unexpected 'endforeach' (t_endforeach) in /home/soralmedia/public_html/wp-content/themes/soral media/woocommerce/myaccount/my-orders.php on line 98 here's code file: <?php /** * orders * * shows recent orders on account page. * * template can overridden copying yourtheme/woocommerce/myaccount/my-orders.php. * * however, on occasion woocommerce need update template files , (the theme developer). * need copy new files theme maintain compatibility. try this. * little possible, happen. when occurs version of template file will. * bumped , readme list important changes. * * @see http://docs.woothemes.com/document/template-structure/ * @author woothemes * @package woocommerce/templates * @version 2.5.0 */ if ( ! defined( 'abspath' ) ) { exit; } $my_orders_columns = apply_filters( 'woocommerce_my_account_my_orders_columns', array( ...
Read more

Export Excel workseet into txt file using vba - (text and numbers with formulas) -

-
i new vba have developed code use in excel move text/numbers txt file. issue have 3 functions need go same txt file. when run first function has set fs = createobject("scripting.filesystemobject") set = fs.createtextfile(module1.nys45uploadfilename, true) a.writeline (str) a.close. the other 2 functions had save. set fs = createobject("scripting.filesystemobject") set = fs.savetextfile(module1.nys45uploadfilename, true) a.writeline (str) a.close. the first function go txt file, other 2 error object doesn't support property or method. when changed 3 create last function goes txt file. can not figure out word use in order make other 2 functions follow txt file in order keyed. the entire code follows private sub addrecord_click() on error goto errhandler module1.nys45uploadfilename = application.getsaveasfilename(filefilter:="textfiles (*.txt), *.txt") if module1.nys45uploadfilename = "false" exit sub ...
Read more

Using django-mptt to get only the categories that have items -

-
sample structure: all uppercase: category mixed case: item root ├── books │   ├── fiction │   │   └── classics │   └── non-fiction ├── clothing └── electronics ├── laptops ├── phones │   ├── apple │   │   ├── iphone 6 │   │   ├── iphone 6 plus │   │   ├── iphone 6s │   │   └── iphone 6s plus │   ├── motorola │   │   ├── moto g4 │   │   ├── moto g4 play │   │   ├── moto g4 plus │   │   └── moto x │   └── samsung └── tablets └── apple i'm trying have index page show "electronics" category. "clothing" category shouldn't show since it's empty , "books" category shouldn't show either since, although has child categories, child categories don't have items. similarly, "electronics" page should show "phones" category. "laptops" category shouldn't show since it's empty , "tablets" category shouldn't show either since...
Read more