python - Best way to join two large datasets in Pandas -

-

i'm downloading 2 datasets 2 different databases need joined. each of them separately around 500mb when store them csv. separately fit memory when load both memory error. trouble when try merge them pandas.

what best way outer join on them don't memory error? don't have database servers @ hand can install kind of open source software on computer if helps. ideally still solve in pandas not sure if possible @ all.

to clarify: merging mean outer join. each table has 2 row: product , version. want check products , versions in left table only, right table , both tables. 

pd.merge(df1,df2,left_on=['product','version'],right_on=['product','version'], how='outer')

this seems task dask designed for. essentially, dask can pandas operations out-of-core, can work datasets don't fit memory. dask.dataframe api subset of pandas api, there shouldn't of learning curve. see dask dataframe overview page additional dataframe specific details.

import dask.dataframe dd  # read in csv files. df1 = dd.read_csv('file1.csv') df2 = dd.read_csv('file2.csv')  # merge csv files. df = dd.merge(df1, df2, how='outer', on=['product','version'])  # write output. df.to_csv('file3.csv', index=false)

assuming 'product' , 'version' columns, may more efficient replace merge with:

df = dd.concat([df1, df2]).drop_duplicates()

i'm not entirely sure if better, apparently merges aren't done on index "slow-ish" in dask, worth try.


Comments

Post a Comment

Popular posts from this blog

wordpress - (T_ENDFOREACH) php error -

-
creating wordpress website , i'm getting following php error: parse error: syntax error, unexpected 'endforeach' (t_endforeach) in /home/soralmedia/public_html/wp-content/themes/soral media/woocommerce/myaccount/my-orders.php on line 98 here's code file: <?php /** * orders * * shows recent orders on account page. * * template can overridden copying yourtheme/woocommerce/myaccount/my-orders.php. * * however, on occasion woocommerce need update template files , (the theme developer). * need copy new files theme maintain compatibility. try this. * little possible, happen. when occurs version of template file will. * bumped , readme list important changes. * * @see http://docs.woothemes.com/document/template-structure/ * @author woothemes * @package woocommerce/templates * @version 2.5.0 */ if ( ! defined( 'abspath' ) ) { exit; } $my_orders_columns = apply_filters( 'woocommerce_my_account_my_orders_columns', array(
Read more

Export Excel workseet into txt file using vba - (text and numbers with formulas) -

-
i new vba have developed code use in excel move text/numbers txt file. issue have 3 functions need go same txt file. when run first function has set fs = createobject("scripting.filesystemobject") set = fs.createtextfile(module1.nys45uploadfilename, true) a.writeline (str) a.close. the other 2 functions had save. set fs = createobject("scripting.filesystemobject") set = fs.savetextfile(module1.nys45uploadfilename, true) a.writeline (str) a.close. the first function go txt file, other 2 error object doesn't support property or method. when changed 3 create last function goes txt file. can not figure out word use in order make other 2 functions follow txt file in order keyed. the entire code follows private sub addrecord_click() on error goto errhandler module1.nys45uploadfilename = application.getsaveasfilename(filefilter:="textfiles (*.txt), *.txt") if module1.nys45uploadfilename = "false" exit sub
Read more

Using django-mptt to get only the categories that have items -

-
sample structure: all uppercase: category mixed case: item root ├── books │   ├── fiction │   │   └── classics │   └── non-fiction ├── clothing └── electronics ├── laptops ├── phones │   ├── apple │   │   ├── iphone 6 │   │   ├── iphone 6 plus │   │   ├── iphone 6s │   │   └── iphone 6s plus │   ├── motorola │   │   ├── moto g4 │   │   ├── moto g4 play │   │   ├── moto g4 plus │   │   └── moto x │   └── samsung └── tablets └── apple i'm trying have index page show "electronics" category. "clothing" category shouldn't show since it's empty , "books" category shouldn't show either since, although has child categories, child categories don't have items. similarly, "electronics" page should show "phones" category. "laptops" category shouldn't show since it's empty , "tablets" category shouldn't show either since
Read more