pyspark.sql.DataFrame.withColumn

DataFrame.withColumn(colName, col)

Returns a new DataFrame by adding a column or replacing the existing column that has the same name.

The column expression must be an expression over this DataFrame; attempting to add a column from some other DataFrame will raise an error.

New in version 1.3.0.

Parameters:

colNamestr

string, name of the new column.

colColumn

a Column expression for the new column.

Notes

This method introduces a projection internally. Therefore, calling it multiple times, for instance, via loops in order to add multiple columns can generate big plans which can cause performance issues and even StackOverflowException. To avoid this, use select() with the multiple columns at once.

Examples

>>> df.withColumn('age2', df.age + 2).collect()
[Row(age=2, name='Alice', age2=4), Row(age=5, name='Bob', age2=7)]

pyspark.sql.DataFrame.where pyspark.sql.DataFrame.withColumnRenamed