Use Spark SQL map_entries() function to convert map of StructType to array of StructType column on DataFrame. Map_entries() вЂ“ convert map of StructType to array of StructType Syntax - map_from_entries(e: Column): ColumnД¬oncatDF.withColumn('mapFromEntries',map_from_entries(col('addresses'))) This function take DataFrame column ArrayType as an argument, passing any other type results an error. Use map_from_entries() SQL functions to convert array of StructType entries to map ( MapType) on Spark DataFrame. Map_from_entries() вЂ“ convert array of StructType entries to map

(arrayStructureData),arrayStructureSchema)Д¬oncatDF.withColumn('mapConcat',map_concat(col('properties'),col('secondProp'))) add('secondProp', MapType(StringType,StringType))

add('properties', MapType(StringType,StringType)) add('addresses', ArrayType(new StructType() Val arrayStructureSchema = new StructType() All arguments to this function should be MapType, passing any other type results a run time error. Use Spark SQL map_concat() function in order to concatenate keys and values from more than one map to a single map. Syntax - map_concat(cols: Column*): Column Map_concat() вЂ“ Concatenating two or more maps on DataFrame Note that map_values takes an argument of MapType while passing any other type returns an error at run time.Дf.select(col('id'),map_values(col('propertiesMap'))) Use map_values() spark function in order to retrieve all values from a Spark DataFrame MapType column. Map_values() вЂ“ Returns map values from a Spark DataFrame Outputs all map keys from a Spark DataFrame Note that map_keys takes an argument of MapType while passing any other type returns an error at run time.Дf.select(col('id'),map_keys(col('propertiesMap'))).show(false) Use map_keys() spark function in order to retrieve all keys from a Spark DataFrame MapType column. Map_keys() вЂ“ Returns map keys from a Spark SQL DataFrame | |- value: string (valueContainsNull = true) and finally use map() function with a key, value set pair. we need LinkedHashSet in order to maintain the insertion order of key and value pair. Var columns = mutable.LinkedHashSet()Д¬olumns.add(col('properties.' + field.name))Дf = df.withColumn('propertiesMap',map(columns.toSeq:_*))ДЇirst, we find вЂњpropertiesвЂќ column on Spark DataFrame using df.schema.fieldIndex(вЂњpropertiesвЂќ) and retrieves all columns and itвЂ™s values to a LinkedHashSet. Val propSchema = df.schema(index).dataType.asInstanceOf Val index = df.schema.fieldIndex('properties') Below snippet converts all columns from вЂњpropertiesвЂќ struct into map key, value pairs вЂњpropertiesmapвЂќ column.

Note: All key columns must have the same data type, and canвЂ™t be null and All value columns must have the same data type. The input columns to the map function must be grouped as key-value pairs. () SQL function is used to create a map column of MapType on DataFrame. Transform_values(expr: Column, f: (Column, Column) => Column)Д«efore we start, letвЂ™s create a DataFrame with some sample data to work with. Transforms map by applying functions to every key-value pair and returns a transformed map. Transform_keys(expr: Column, f: (Column, Column) => Column) It also creates 3 columns вЂњposвЂќ to hold the position of the map element, вЂњkeyвЂќ and вЂњvalueвЂќ columns for every row. It also creates 3 columns вЂњposвЂќ to hold the position of the map element, вЂњkeyвЂќ and вЂњvalueвЂќ columns for every row.Д¬reates a new row for each key-value pair in a map including null & empty. It creates two new columns one for key and one for value.Д¬reates a new row for each key-value pair in a map by ignoring null & empty. It creates two new columns one for key and one for value.Д¬reates a new row for every key-value pair in the map including null & empty. Returns an array of all StructType in the given map.Д¬reates a new row for every key-value pair in the map by ignoring null & empty. Returns a map from the given array of StructType entries. Returns an array containing the values of the map. Returns an array containing the keys of the map.

Sql functions list