python - Can we use a Pandas function in a Spark DataFrame column ? If so, how? -

-

i have pandas dataframe called "pd_df".

i want modify it's column , this:

    import pandas pd      pd_df['notification_dt'] = pd.to_datetime(pd_df['notification_dt'], format="%y-%m-%d")

it works.

on same database, created spark dataframe called "spark_df"

i want same function (pd.to_datatime) on it's column perform same operation. did this.

    pyspark.sql.functions import userdefinedfunction      pyspark.sql.types import timestamptype      udf = userdefinedfunction(lambda x: pd.to_datetime(x, format="%y-%m-%d"), timestamptype())      spark_df2 = spark_df.withcolumn("notification_dt1", (udf(spark_df["notification_dt"])))

it should work, according me. on

   spark_df.show()

i encounter following error after minute or so: enter image description here

so, got fixed.

 udf = userdefinedfunction(lambda x: pd.to_datetime(x, format="%y-%m-%d"), timestamptype())

should be

 udf = userdefinedfunction(lambda x: str(pd.to_datetime(x, format="%y-%m-%d")), timestamptype())

it failing convert result timestamptype()


Comments

Post a Comment

Popular posts from this blog

wordpress - (T_ENDFOREACH) php error -

-
creating wordpress website , i'm getting following php error: parse error: syntax error, unexpected 'endforeach' (t_endforeach) in /home/soralmedia/public_html/wp-content/themes/soral media/woocommerce/myaccount/my-orders.php on line 98 here's code file: <?php /** * orders * * shows recent orders on account page. * * template can overridden copying yourtheme/woocommerce/myaccount/my-orders.php. * * however, on occasion woocommerce need update template files , (the theme developer). * need copy new files theme maintain compatibility. try this. * little possible, happen. when occurs version of template file will. * bumped , readme list important changes. * * @see http://docs.woothemes.com/document/template-structure/ * @author woothemes * @package woocommerce/templates * @version 2.5.0 */ if ( ! defined( 'abspath' ) ) { exit; } $my_orders_columns = apply_filters( 'woocommerce_my_account_my_orders_columns', array( ...
Read more

Export Excel workseet into txt file using vba - (text and numbers with formulas) -

-
i new vba have developed code use in excel move text/numbers txt file. issue have 3 functions need go same txt file. when run first function has set fs = createobject("scripting.filesystemobject") set = fs.createtextfile(module1.nys45uploadfilename, true) a.writeline (str) a.close. the other 2 functions had save. set fs = createobject("scripting.filesystemobject") set = fs.savetextfile(module1.nys45uploadfilename, true) a.writeline (str) a.close. the first function go txt file, other 2 error object doesn't support property or method. when changed 3 create last function goes txt file. can not figure out word use in order make other 2 functions follow txt file in order keyed. the entire code follows private sub addrecord_click() on error goto errhandler module1.nys45uploadfilename = application.getsaveasfilename(filefilter:="textfiles (*.txt), *.txt") if module1.nys45uploadfilename = "false" exit sub ...
Read more

delphi - cxGrid : instead of "go to next cell on enter" using "enter" do that with the arrow keys -

-
Image
is there way can achieve cxgrid's functionality "go next cell on enter" using arrow keys on keyboard instead of hitting "enter" ??? right can navigate arrow keys cells move not selected (blue color). if edit cell in column , move cell beneath (with arrow key) , grid jumps other random record in grid,which annoying. is question of settings or must program functionality ? edit : simple temporary table use gather data. has no indexed fields or autoinc field : procedure tform3.cxbutton1click(sender: tobject); begin datamodule2.acrquery2.close; datamodule2.acrquery2.sql.clear; datamodule2.acrquery2.sql.text :='insert temp select company_id,member_id,surname,name members company_id =:a1'; datamodule2.acrquery2.params.parambyname('a1').value :=cxlookupcombobox2.text; datamodule2.acrquery2.execsql; datamodule2.acrquery3.close; datamodule2.acrquery3.sql.clear; datamodule2.acrquery3.sql.text :='update temp set month=:a1,year=:a2'; data...
Read more