Pyspark特征工程

Pyspark特征工程

StopWordsRemover

class pyspark.ml.feature.StopWordsRemover(inputCol=None, outputCol=None, stopWords=None, caseSensitive=False, locale=None

StopWordsRemover的功能是直接移除所有停用词（stopword），所有从inputCol输入的量都会被它检查，然后再outputCol中，这些停止词都会去掉了。

默认的话会在构建StopWordsRemover对象的时候调用loadDefaultStopWords(language: String): Array[String]加载/org/apache/spark/ml/feature/。

这是一个简单的停止词表，包含181个词（spark2.2）。

默认还提供了其他几种语言（danish, dutch, english, finnish, french, german, hungarian,italian, norwegian, portuguese, russian, spanish, swedish, turkish）的停止词，遗憾的是没有中文默认停止词表，所以对于中文停止词需要自己提供.

对于不同类型的需求而言，对停止词的处理是不同的。

1. 有监督的机器学习 – 将停止词从特征空间剔除，2. 聚类– 降低停止词的权重

2. 信息检索– 不对停止词做索引，4. 自动摘要- 计分时不处理停止词

对于不同语言，停止词的类型都可能有出入，但是一般而言有这简单的三类

1. 限定词 ，2. 并列连词 ，3. 介词

停止词的词表一般不需要自己制作，有很多选项可以自己下载选用

01.加载模块，生成对象

from pyspark.sql import SparkSession
from pyspark.ml.feature import StopWordsRemover
spark = fig("spark.driver.host","192.168.1.4").config("spark.ui.showConsoleProgress","false").appName("StopWordsRemover").master("local[*]").getOrCreate()

02.创建数据

data = ateDataFrame([(["Beautiful","is","better","than","ugly"],),(["Explicit","is","better","than","implicit"],),(["Simple","is","better","than","complex"],),(["Complex","is","better","than","complicated"],),(["Flat","is","better","than","nested"],),(["Sparse","is","better","than","dense"],)
],["text"])
data.show()

​ 输出结果：

+--------------------+
|                text|
+--------------------+
|[Beautiful, is, b...|
|[Explicit, is, be...|
|[Simple, is, 
|[Complex, is, 
|[Flat, is, 
|[Sparse, is, 
+--------------------+

03.使用head，详细查看

data.head(6)

​ 输出结果：

[Row(text=['Beautiful', 'is', 'better', 'than', 'ugly']),Row(text=['Explicit', 'is', 'better', 'than', 'implicit']),Row(text=['Simple', 'is', 'better', 'than', 'complex']),Row(text=['Complex', 'is', 'better', 'than', 'complicated']),Row(text=['Flat', 'is', 'better', 'than', 'nested']),Row(text=['Sparse', 'is', 'better', 'than', 'dense'])]

04.使用StopWordsRemover，剔除不需要的单词，并查看结果

stopWordsRemover = StopWordsRemover(inputCol="text",outputCol="res",stopWords=["is","better","than"])
data = ansform(data)
data.show()
data.head(6)

​ 输出结果：

+--------------------+--------------------+
|                text|                 res|
+--------------------+--------------------+
|[Beautiful, is, b...|   [Beautiful, ugly]|
|[Explicit, is, be...|[Explicit, implicit]|
|[Simple, is,    [Simple, complex]|
|[Complex, is, [Complex, 
|[Flat, is,       [Flat, nested]|
|[Sparse, is,      [Sparse, dense]|
+--------------------+--------------------+[Row(text=['Beautiful', 'is', 'better', 'than', 'ugly'], res=['Beautiful', 'ugly']),Row(text=['Explicit', 'is', 'better', 'than', 'implicit'], res=['Explicit', 'implicit']),Row(text=['Simple', 'is', 'better', 'than', 'complex'], res=['Simple', 'complex']),Row(text=['Complex', 'is', 'better', 'than', 'complicated'], res=['Complex', 'complicated']),Row(text=['Flat', 'is', 'better', 'than', 'nested'], res=['Flat', 'nested']),Row(text=['Sparse', 'is', 'better', 'than', 'dense'], res=['Sparse', 'dense'])]

05.查看结构

data.printSchema()

​ 输出结果：

root|-- text: array (nullable = true)|    |-- element: string (containsNull = true)|-- res: array (nullable = true)|    |-- element: string (containsNull = true)