Getting this error while loading csv file in databricks
Getting this error while trying to load csv file from Azure datalake gen 2 to databricks notebook. I have created connection of Azure and databricks and followed every procedure.
Getting this error while loading csv file in databricks
Getting this error while trying to load csv file from Azure datalake gen 2 to databricks notebook. I have created connection of Azure and databricks and followed every procedure.
Replace first occurrence of character in spark dataframe pyspark
I know , I am asking very basic question here , But is there any way to replace first occurrence of character within pyspark dataframe.
Replace first occurrence of character in spark dataframe pyspark
I know , I am asking very basic question here , But is there any way to replace first occurrence of character within pyspark dataframe.
Replace first occurrence of character in spark dataframe pyspark
I know , I am asking very basic question here , But is there any way to replace first occurrence of character within pyspark dataframe.
Change the column name from Upper Case to LowerCase parquet file in pyspark
I want to convert a column name in the parquet file from Uppercase to Lowercase and rewrite it back at the same location (From “EXE_TS” to “exe_ts”). The parquet file is partioned with a column named as data_as_of_date.
Cant find sparkContext when running pyspark features with Databricks spark instance using VSCode
I am using databricks using vscode. (used databricks-connect with vscode-extension).
I am able to get the spark session using DatabricksSession
how to combine array of map in a single map per column in pyspark
i have followed this question but the answers there not working for me
i don’t want a UDF for this and map_concat doesn’t work for me.
is there any other way to combine maps?
how to check if df column contains a map key and if contains, put the corresponding value in a new column in pyspark?
“I have a DataFrame with records stored in a particular column. I want to compare each record in that column against a predefined map. If a record contains any of the keys in the map, I want to populate a new column with the corresponding value associated with that key in the map.”
calulcate size of each file in each directory using pyspark and add as a column
I have files in each directory inside a Azure storage account(devcome) which has the cointainer (inputdata) for example
devcome
inputdata
abc
01
Module191.json
Module192.json
02
Module191.json
Module192.json
def
03
Module191.json
Module192.json
04
Module191.json
Module192.json