《基于内容挖掘的垃圾短信过滤分类方法研究论文》由会员分享，可在线阅读，更多相关《基于内容挖掘的垃圾短信过滤分类方法研究论文（66页珍藏版）》请在金锄头文库上搜索。
1、基于内容挖掘的自动文本分类研究及其在短信过滤中应用 II摘 要 短信息已经成为人们日常生活中通信、交流的重要手段之一，但垃圾短信问题也日益严峻，用户平均每天收到的垃圾短信数量已超过了正常短信。垃圾短信过滤是当前人们生活急需解决的一个重要任务。目前经常采用的垃圾短信过滤技术一般包括白名单与黑名单技术、规则过滤以及基于关键词匹配的内容扫描等。针对目前使用的过滤方法的缺点以及具体情况,提出将自动文本分类的信息过滤技术应用在短信过滤中，信息过滤中常用的文本分类方法有简单贝叶斯、k-近邻、决策树、boosting 等。 本文设计并实现了一种基于改进贝叶斯的垃圾短信过滤系统。利用贝叶斯理论和最小风险决策算
2、法对用户给定的正常/垃圾短信集合进行训练,得到短信过滤模型。这个模型包括了短信处理、中文分词、特征选取以及短信分类与过滤几个主要功能模块。接着分析了各模块的主要算法和 Java 实现方法。最后对文本短信自动分类器进行了实验，对我们所构建的文本短信自动分类工具的性能进行了评估，实验表明此工具有较高的分类准确度，达到了设计的要求。 本文的工作主要包含下列内容： (1) 概述垃圾短信过滤问题的研究现状。包括垃圾短信的定义、危害以及常用的过滤技术。 (2) 对文本短信自动分类器进行了总体设计，制订了它的各个功能模块及每个模块的主要功能。同时也阐述了所使用的新方法。 (3) 分析和比较了多种中文分词实现
3、方法,对最大匹配分词法进行了改进，提高了中文分词的效率和准确度。并给出了实现的技术细节，特别是对中文词典的实现方法作了详细介绍。 (4) 分析了多种特征选取算法，比较了它们的优缺点，提出了文档频度与词条频度相结合的特征选取算法，并对此方法的实现进行阐述。 (5) 分析了贝叶斯机器学习方法的主要思想，对贝叶斯方法用于文档归类的原理进行了剖析。详细介绍了贝叶斯分类方法的算法与实现。 (6) 讲述了文本分类的评价标准，通过实验对本课题所构建的分类工具进行了评估。 总结了研究成果，指出了不足并提出了改进意见。 关键词：短消息过滤；中文分词；文本自动分类；贝叶斯机器学习 工程硕士学位论文 IIIAbst
4、ract Short Message Service (SMS) is becoming one of the fastest and most economical ways of communication available. At the same time, the growing problem of junk SMS (also referred to as “spam SMS”) has generated a need for SMS filtering. SMS filtering is an important task in the life of people, wh
5、ich receives increasing emphasis. Nowadays, anti-spam SMS measures commonly include black or white list technology, manual rules and keyword based content filtering. According to the disadvantage in traditional methods and to solve practical problems of SMS filtering, automated text categorization a
6、nd information filtering is proposed. Such algorithms of text categorization as Naive Bayes, kNN, Decision Tree and Boosting can be applied in Information filtering. In this paper, we have developed a new filtering spam SMS system based on improved Bayesian. The minimum risk strategy is used for Bay
7、esian algorithm to learn from the user given training spam/normal SMS set. This model included pretreatment of SMS, Chinese words splitter, characteristic extraction, categorization and flitting of SMS. The primary function and arithmetic with java source code are discussed in this paper. At last we
8、 draw an experience to test the accuracy of the software to category Chinese web document. As the experiment result show, this software has high accuracy. The contents of this article are as following: (1) A summary about the state of the spam SMS filtering. (2) The whole design of automatic text SM
9、S classifier is described in this paper. The primary function of each module is discussed. And the new methods proposed by us are also discussed. (3) Chinese text splitter is described in this paper. Based on analysis of all sorts of Chinese text splitter arithmetic, we discussed how to improve max
10、match Chinese text splitter arithmetic. The Chinese dictionary based on hash table is discussed. (4) This paper compares all sorts of feature select arithmetic. The advantage and disadvantage of these arithmetic are summarized. We proposed a new arithmetic named as DFTF( Document frequency and Term
11、Frequency). We give out its reality with java source code. (5) Naive Bayes machine learning method is discussed in this paper. Especially, 基于内容挖掘的自动文本分类研究及其在短信过滤中应用 IVwe discussed the arithmetic of how to category the Chinese web document with naive bayes machine learning. And then we present how to
12、 reality such a classier. (6) This paper present how to evaluate the quality of Chinese web document classifier. As the experiment result show, high category quality is obtained on this classifier. We also summarized the gain and defect of this project. Further, we discussed how to improve this clas
13、sifier in future research. Keywords: Short Message Filtering；Chinese text splitter；text categorization； Naive Bayes machine learning 基于内容挖掘的自动文本分类研究及其在短信过滤中应用 VIII插图索引 图 1.1 垃圾短信过滤平台在短信网络的拓扑图 . 3 图 1.2 SSF 拦截过滤 SMS 消息流程图 . 3 图 1.3 总体组网结构 . 4 图 1.4 信息过滤系统的一般结构 . 6 图 2.1 基于贝叶斯算法短信分类与过滤技术的系统架构 . 9 图 2.2 贝叶斯文本分类算法的训练流程图 . 11 图 2.3 贝叶斯文本短信分类算法的分类流程图 . 11 图 3.1 中文分词流程图 . 20 图 4.1 词条强度矩阵 . 24 图 4.2 文档对的余弦值与相似度关系统计图 . 25 图 5.1 一棵决策树结构图 . 28 图 5.2 AdaBoost 学习算法 . 29 图 5.3 最优分类面 . 30 图 5.4 贝叶斯机器学习与分类流程图 .
