TOPICBERT: A TOPIC-ENHANCED NEURAL LANGUAGE MODEL FINE-TUNED FOR SENTIMENT CLASSIFICATION

In simple words, it is not an easy task to classify sentiments in data mining. Sentiment classification is an automatic technique of recognizing and grouping the emotional worth of a text as either an affirmative, unfavorable or impartial depending on the ideas conveyed by the text. The scientific community and the commercial world, in general, has been very interested in the case of automatic collection of the views of the masses, in terms of social occurrences, marketing campaigns, product favorable, etc. Also, the emerging emotional computing domains have been given a boost by the creation of autonomous systems capable of searching feelings. These novel research areas rely on information retrieval and human-computer interaction to derive the emotions of people based on the accumulating amount of data which is the result of social networking websites on the Internet. The aim of ATSC, which is a sentiment categorization method, which predicts the sentiment polarity, i.e. the extent to which the sentiment of a target term toward a given phrase is negative, neutral, or positive.

 

The quantity of material that has emerged in the opinion mining and sentiment analysis field has given us an idea on the various means and progress. Later developments were founded on the work of Liu [1] and Pang and Lee [2]. The sentiment analysis has now become multimodal as Poria et al. illustrate with the in-depth the idea of OntoSenticNet that was developed by Dragoni, Poria, and Cambria [4], and the approach [3]. The cross-functional aspect of this research can be proved by the study of Stappen et al. [5] about subject detection and attitudes analysis of the video text. The three steps of data mining of text results offered by Saura and Bennett [6] have viable premises in the analysis of business intelligence, and the sentiment analysis offered by Saura et al. [7] is used to determine the indicators of startup success. The Saura article [8] considers the integration of the data sciences into the digital marketing and emphasizes the role of sentiment analysis as a dynamic factor in the modern company strategy. The setting of the various perspectives presented in this collection has been set out in this introduction wherein various details about the evolution of sentiment analysis and its necessity in a broad area of applications have been given [9]. The latest news demonstrates how the innovation in the sentiment analysis methods is progressed, including AEE Convolutional LSTM by Huang et al. [10].

 

      I.         LITERATURE SURVEY

Sentiment analysis and opinion mining has received much attention in recent years and researchers.  have explored many methods to enhance the understanding of the textual information. The detailed knowledge about sentiment analysis could be gained with the pioneering work of Liu [1] and the initial assessment of the same by Pang and Lee [2]. Poria et al. [3] talk about significant trends and norms in the research of multimodal sentiment analysis.

 

OntoSenticNet An ontology of sentiment analysis on common sense was initially presented by Dragoni et al. [4]. Sentiment analysis is also applied to video transcriptions by Stappen et al. [5], who incorporate identification and feeling of the subject. Saura and Bennett [6], in the business intelligence research, utilize user-generated content and suggest a three-step data text mining method. Similarly, the article by Saura et al. [7] is dedicated to the analysis of the sentiment of the measures of the business performance of startups.

 

Saura [8] studies how data sciences could be applied in digital marketing, provision of techniques, a structure, performance measures. One of the most useful methods of sentiment analysis that exploits emotional computing is presented by Cambria et al. [9].

 

Getting more advanced techniques, Huang et al. introduce a convolutional LSTM which is enhanced by emotion and attention to sentiment analysis [10]. The approach to self-supervised attention learning proposed by Tang et al. [17] is progressive and is based on the aspect-level sentiment analysis. Additionally, several works [1820] apply DL methods to sentiment analysis based on aspects, which are convolutional and recursive neural networks.

 

Pre-trained algorithms like BERT have also been used to influence the application Sentiment analysis [21]. Although According to Huang and Carley [24], a system of classification of sentiments at an aspect level is proposed.  on a syntax-sensitive graph attention network, Sun et al. [23] use BERT in the aspects sentiment analysis through generation of auxiliary phrases. Xu et al. [25] are able to compare aspect-based sentiment analysis and reading comprehension after the training with BERT.

 

The abundance of commonsense bases of knowledge is explored [29] and also Language representation learning makes use of knowledge graphs [27]. Maier and associates.  [30] also give another example of how topic modeling may be used transdisciplinarily in communication research.

 

The conclusion of the literature review demonstrates a tendency towards more complex methods that rely on knowledge graphs, DL, and pre-trained models. Even scholars in multiple disciplines continue to explore new strategies in solving the sentiment analysis problem.

Modules:

·       Data Preprocessing: This module will be used in the analysis of the data.

·       Separation of data to test and train sets: This module will split the data to test set and train set.

·       Model construction: Model construction Topic BERT (BERT with Topic Modeling), BERT Small, BERT Large, LSTM, and LSTM +GRU. Computed accuracy of algorithms.

·       User registration and login: You are able to register and log in with the help of this module.

·       User input: This module will serve as the input of prediction.

·       Forecast: the last prediction is presented.

A)    System Architecture

“Fig 1: System Architecture”

 

 

Proposed work

To improve on the future sentiment classification task, we introduce TopicBERT which is a variant of BERT designed to achieve comprehensive theme recognition at word, sentence, and corpus scales. Topics Bert has two forms, TopicBERT-ATP (aspect topic prediction), which is based on collapsed Gibbs sampling and an LDA mechanism to dynamically pre-determine subjects and affect topics when training, and TopicBERT-TA, which injects immediate topic representation into a sentiment classification augmentation layer. The dynamism of topicBERT-TA features flexibility.

 

To make predictions more powerful, we enlarge our methodology with an ensemble strategy, which involves gluing together predictions of various models. We do not rule out the fact that further improvement can be achieved, so as we propose to explore other ensemble methods whose accuracy predicted rates exceed 95, such as LSTM and LSTM + GRU pairs. The ensemble method increases the accuracy and robustness of the model in sentiment analysis. This overall framework achieves

.<|human|>This overall framework is achieved. a superior performance in sentiment classification through a combination of theme-conscious BERT variations with diversified ensemble methods, the concept that proves the interaction of complex ensemble learning experiments and language models.

 

B)    Dataset Collection

Dataset Link:

https://www.kaggle.com/datasets/charitarth/semeval-2014-task-4-aspectbasedsentimentanalysis

Dataset Description:

The Sentiment analysis is increasingly gaining importance in the business and research fields. Nevertheless, most of the methods available focus on the determination of the polarity of a sentence, paragraph or text.  without namely referring to the objects (such as computers and restaurants) and their attributes (such as screen and battery; food and service). Conversely, this undertaking is  centered on ABSA and analyzes the establishment of the traits of a particular target.item and emotional expression by each feature. The data sets will be customer ratings that have been annotated by human beings who will note the points defined by the target entities and the sentiment intensity towards a particular point.

“Fig 2. Restaurant Dataset”

Fig 3. Laptop Dataset

C)    Pre-processing

To begin with, we load the data, that is, the one provided in the Kaggle link in the data loading module. The primary focus of the dataset is ABSA is aimed at identifying features of target items (including computers and restaurants) and sentiment expressed in the customer reviews on each aspect. The specified features and spoken attitude poles are defined in the annotations made by human authors forming the dataset. To analyze the dataset, it is a data preparation module.  find out its characteristics and format. This will involve handling of missing values, detecting duplicates and the analysis of sentiment label and aspect category distribution. In order to prepare the data to proceed with the analysis, text preparation procedures such as tokenization, stop word elimination, and lowercasing could be employed. Data exploration and preparation are followed by data splitting The data splitting module is used to split the breaks down data into testing and training sets. This is required so as to test the performance of the model even in instances where certain data may not be visible. Typically it is divided at 70- 30 or 80-20 with training data being the higher split. In this part, it is assumed that the model will collect trends on the training set and can work effectively in new and untestable situations. On the training set, the model is trained and the efficacy of the model in practice is estimated by assessing the performance of the model on unknown data.

 

D)    Training & Testing

A custom module is used to conduct the process of The training after the data has been initially divided into training and testing sets. This is to make sure there was a sub-set of the data that was utilized to train the model and a sub-set that was utilized to evaluate the ability of the model to generalize. A some of the  models that are constructed using training data include LSTM, LSTM + GRU, Topic BERT ( BERT with Topic Modeling ), BERT Small, and BERT Large. Both models have been created to gather different data characteristics and support different methods of prediction. After generating the model, test data is applied to determine the accuracy of each algorithm. This assessment phase  assists in determining model predictive evidence on newly, untested information. The effectiveness of each algorithm can be measured in terms of such indicators as precision, recall, and F1 score. The system also has user registration and user login features to allow secure access. Users  also enter data to be predicted through a special module and this provides the system with relevant information. The last predictions are then displayed giving the consumers some valuable information based on the trained models. Such a combined approach ensures that the process of data preparation and model training has an easy and easy-to-use transition to user engagement and prediction of the outcomes.

 

E)    Algorithms.

BERT Small:

The smaller in size of the pre-trained neural language model used in the BERT is known as BERT Small. Due to the effective design of BERT Small, the effectiveness of  contextualized embeddings cannot be undermined. Embeddings on large scales are suitable in a resource-constrained context due to the fact that it offers contextualization of bidirectional learning that is sustained across a range of NLP problems.

BERT Large:

Neural language model BERT is a BERT Large is a continuation of a model. The larger-sized architecture of Bert Large is able to embed more complex relationship of context in text giving it more expressive embeddings. Due to its high-level understanding and generation of complex linguistic representations, it can be employed in complex natural language processing systems.

“Topic BERT (BERT with Topic Modelling):”

TopicBERT This sentiment categorization system is based on BERT and topic modeling. It is able to detect words, phrases and corpus. Two variations of TopicBERT-ATP and TopicBERT-TA involve learning and injection of dynamic topic representation respectively in the same way as auxiliary tasks are used to capture topics. Experimental results indicate that direct augmentation is better than additional training in tasks of sentiment classification, and it exhibits the state-of-the-art performance.

LSTM:

A well-known RNN architecture of DL  is named LSTM. It is ideal in sequence prediction tasks because it is very good at long term dependency modeling. Since LSTM has feedback links, it is able to act with full sequences of data, as opposed to the individual data points in conventional neural networks. Due to this, it can really well identify and predict trends in sequential data, which includes voice data, text data and time series.

LSTM + GRU:

The model is a combination of RNN in the form of LSTM + GRU. It achieves excellent results on sequential data tasks due to its combination of computational efficiency of GRU and memory preservation of LSTM. This paradigm is applied in time series analysis and in natural language processing since it works well in the case of long time dependencies.

 

A)    “Comparison Graphs → Accuracy, Precision, Recall, f1 score”

Accuracy: The ability of The accuracy is the capability of a test to identify both solid and crippled states accurately. The small percentage of True positive and True negative results of the cases that have been properly examined should be noted so as to determine the accuracy of a test. This could be in the form of numbers as:

“Accuracy = TP + TN TP + TN + FP + FN.”

Fig 4: Accuracy Graph of Restaurant Dataset

“Fig 5: Accuracy Graph of Laptop Dataset”

Precision: Precision is determined by dividing the amount of positives by the amount of events or accurately identified samples. As a result, the accuracy can be determined with the help of the following formula:

“Precision = True positives/ (True positives + False positives) = TP/(TP + FP)”

Fig 6: Precision Score of Restaurant Dataset

Fig 6: Precision Score of Laptop Dataset

Recall: Recall is a ML measure that evaluates the ability of a model to recognize all the occurrences of a certain class. The proportion of  favorable to  be anticipated correctly perceptions that accumulate to actual benefits gives data regarding the capability of a model to recognize instances of a specific category.

“Fig 7: Recall Score of Restaurant Dataset”

“Fig 8: Recall Score of Laptop Dataset”

F1-Score: F1 score is a ML measure that is used to determine the accuracy of a model. It consists of a blend of precision of a model and review scores. The number of all the data that was successfully predicted by a model is called the measure of accuracy.

“Fig 9: F1 Score of Restaurant Dataset”

“Fig 10: F1 Score of Laptop Dataset”

B)    Performance Evaluation table.

“Fig 11: Performance Evaluation Table of Restaurant Dataset”

“Fig 12: Performance Evaluation Table of Laptop Dataset”

 

 

C)    Frontend

“Fig 13: Code Folder”

“Fig 14: Access the folder location in command prompt”

“Fig 15: Url Link to Web Page”

“Fig 16: Home page”

“Fig 17: User Signup page”

“Fig 18: User Sign in Page”

“Fig 19: Enter Your Message Here”

“Fig 20: Result: Text Type is Positive”

"Fig 21: Result is most depended upon”

TopicBERT is another paradigm of fine-tuning BERT pretrained language models that was proposed in this paper. TopiCBERT considers latent topic knowledge and phrase context depending on the usage of the ATSC tasks. To improve  the discriminating semantic topics, two versions are offered: TopicBERT-ATP and TopicBERT-TA which mixes topic information with an auxiliary task and topic information with individual words respectively. Both versions involve an alternate fine-tuning strategy. The LDA model presents unsupervised information on the subject. The effectiveness of the two strategies is shown in numerous experiments. Additional analysis and visualization demonstrate that with information about the subject, BERT models have a higher likelihood of succeeding in understanding the more detailed polarities of sentiment towards a target. Moreover, we will extend our methods and We will not be strategy-independent and will only relate models to perform other tasks such as event extraction and summarization and pretrain language models such as RoBERTa and ALBERT.

 

FUTURE SCOPE

To enable the application of our proposed TopicBERT paradigm to more domains of linguistics in the future, we are considering broadening the Numerous pretraining language models, including the ALBERT and the RoBERTa models, are many. models, in the future. Further, we shall use the improved performance of topic-augmented fine-tuning to generalize our research in other tasks such as event recognition and event summarization. We attempt to explore new methodological ways of incorporating the information provided by the subject and improve the understanding of the model of complex sentiments and semantic nuances. This study preconditions the intensive exploration of the opportunities of topic-enhanced fine-tuning of natural language processing systems and various language models.

[1] B. Liu, “Sentiment analysis and opinion mining,” Synthesis Lectures Hum. Lang. Technol., vol. 5, no. 1, pp. 1–167, 2012.

[2] B. Pang and L. Lee, “Opinion mining and sentiment analysis,” Found. Trends Inf. Retr., vol. 2, nos. 1–2, pp. 1–135, 2008.

[3] S. Poria et al., “Multimodal sentiment analysis: Addressing key issues and setting up the baselines,” IEEE Intell. Syst., vol. 33, no. 6, pp. 17–25, Nov. 2018.

[4] M. Dragoni, S. Poria, and E. Cambria, “OntoSenticNet: A commonsense ontology for sentiment analysis,” IEEE Intell. Syst., vol. 33, no. 3, pp. 77–85, May 2018.

[5] L. Stappen, A. Baird, E. Cambria, B. W. Schuller, and E. Cambria, “Sentiment analysis and topic recognition in video transcriptions,” IEEE Intell. Syst., vol. 36, no. 2, pp. 88–95, Mar. 2021.

[6] J. R. Saura and D. R. Bennett, “A three-stage method for data text mining: Using UGC in business intelligence analysis,” Symmetry, vol. 11, no. 4, p. 519, Apr. 2019.

[7] J. R. Saura, P. Palos-Sanchez, and A. Grilo, “Detecting indicators for startup business success: Sentiment analysis using text data mining,” Sustainability, vol. 11, no. 3, p. 917, Feb. 2019.

[8] J. R. Saura, “Using data sciences in digital marketing: Framework, methods, and performance metrics,” J. Innov. Knowl., vol. 6, no. 2, pp. 92–102, Apr. 2021.

[9] E. Cambria et al., “Affective computing and sentiment analysis,” in A Practical Guide to Sentiment Analysis. Cham, Switzerland: Springer, 2017, pp. 1–10.

[10] F. Huang, X. Li, C. Yuan, S. Zhang, J. Zhang, and S. Qiao, “Attentionemotion-enhanced convolutional LSTM for sentiment analysis,” IEEE Trans. Neural Netw. Learn. Syst., early access, Feb. 18, 2021, doi: 10.1109/TNNLS.2021.3056664.

[11] J. Wagner et al., “DCU: Aspect-based polarity classification for SemEval task 4,” in Proc. 8th Int. Workshop Semantic Eval. (SemEval), 2014, pp. 223–229.

[12] L. Dong, F. Wei, C. Tan, D. Tang, M. Zhou, and K. Xu, “Adaptive recursive neural network for target-dependent Twitter sentiment classification,” in Proc. 52nd Annu. Meeting Assoc. Comput. Linguistics, vol. 2, 2014, pp. 49–54.

[13] D. Tang, B. Qin, X. Feng, and T. Liu, “Effective LSTMs for targetdependent sentiment classification,” 2015, arXiv:1512.01100. [Online]. Available: http://arxiv.org/abs/1512.01100

 [14] S. Sukhbaatar et al., “End-to-end memory networks,” in Proc. Adv. Neural Inf. Process. Syst., 2015, pp. 2440–2448.

[15] X. Li, L. Bing, W. Lam, and B. Shi, “Transformation networks for targetoriented sentiment classification,” in Proc. 56th Annu. Meeting Assoc. Comput. Linguistics, 2018, pp. 946–956.

[16] S. Wang, S. Mazumder, B. Liu, M. Zhou, and Y. Chang, “Target-sensitive memory networks for aspect sentiment classification,” in Proc. 56th Annu. Meeting Assoc. Comput. Linguistics, 2018, pp. 957–967.

[17] J. Tang et al., “Progressive self-supervised attention learning for aspectlevel sentiment analysis,” in Proc. 57th Annu. Meeting Assoc. Comput. Linguistics, 2019, pp. 557–566.

 [18] S. Poria, E. Cambria, and A. Gelbukh, “Aspect extraction for opinion mining with a deep convolutional neural network,” Knowl.-Based Syst., vol. 108, pp. 42–49, Sep. 2016.

[19] Y. Ma, H. Peng, and E. Cambria, “Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive LSTM,” in Proc. AAAI Conf. Artif. Intell., 2018, vol. 32, no. 1, pp. 5876–5883.

[20] E. Cambria, Y. Li, F. Z. Xing, S. Poria, and K. Kwok, “SenticNet 6: Ensemble application of symbolic and subsymbolic AI for sentiment analysis,” in Proc. 29th ACM Int. Conf. Inf. Knowl. Manage., Oct. 2020, pp. 105–114.

[21] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” 2018, arXiv:1810.04805. [Online]. Available: http://arxiv.org/abs/1810.04805

[22] Y. Zhu et al., “Aligning books and movies: Towards story-like visual explanations by watching movies and reading books,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Dec. 2015, pp. 19–27.

[23] C. Sun, L. Huang, and X. Qiu, “Utilizing BERT for aspectbased sentiment analysis via constructing auxiliary sentence,” 2019, arXiv:1903.09588. [Online]. Available: http://arxiv.org/abs/1903.09588

[24] B. Huang and K. M. Carley, “Syntax-aware aspect level sentiment classification with graph attention networks,” 2019, arXiv:1909.02606. [Online]. Available: http://arxiv.org/abs/1909.02606

[25] H. Xu, B. Liu, L. Shu, and P. S. Yu, “BERT post-training for review reading comprehension and aspect-based sentiment analysis,” 2019, arXiv:1904.02232. [Online]. Available: http://arxiv.org/abs/1904.02232

[26] P. Ke, H. Ji, S. Liu, X. Zhu, and M. Huang, “SentiLARE: Sentiment-aware language representation learning with linguistic knowledge,” 2019, arXiv:1911.02493. [Online]. Available: http://arxiv.org/ abs/1911.02493

[27] W. Liu et al., “K-BERT: Enabling language representation with knowledge graph,” in Proc. AAAI, 2020, pp. 2901–2908.

[28] P. Wang, N. Peng, F. Ilievski, P. Szekely, and X. Ren, “Connecting the dots: A knowledgeable path generator for commonsense question answering,” 2020, arXiv:2005.00691. [Online]. Available: http: //arxiv.org/abs/2005.00691

 [29] X. Li, A. Taheri, L. Tu, and K. Gimpel, “Commonsense knowledge base completion,” in Proc. 54th Annu. Meeting Assoc. Comput. Linguistics, vol. 1, 2016, pp. 1445–1455.

[30] D. Maier et al., “Applying LDA topic modeling in communication research: Toward a valid and reliable methodology,” Commun. Methods Measures, vol. 12, nos. 2–3, pp. 93–118, Apr. 2018.