Sentiment Analysis for Customer Review: Case Study of GO-JEK Expansion

Background: Market prediction is an important thing that needs to be analyzed deeply. Business intelligence becomes an important analysis procedure for analyzing the market demand and satisfaction. Since business intelligence needs a deep analysis, sentiment analysis becomes a powerful algorithm for analyzing customer review regarding to the business intelligence analysis. Objective: In this study, we perform a sentiment analysis for identifying the business intelligence analysis in GO-JEK. Methods: We use Twitter posts collected from the Twint library which consists of 3111 tweets. Since the dataset did not provide a ground truth, we perform Microsoft Text Analytic for determining positive, neutral, and negative sentiment. Before applying Microsoft Text Analytic, we conduct a pre-processing step to remove the unwanted data such as duplicate tweets, image, website address, etc. Results: According to the Microsoft Text Analytic, the results are 666 positive sentiment numbers, 2055 neutral sentiment numbers, and 127 negative sentiment numbers. Conclusion: According to these results, we conclude that most GO-JEK customers are satisfied with the GO-JEK services. In this research, we also develop classification model to predict the sentiment analysis of new data. We use some classifier algorithms such as Decision Tree, Naïve Bayes, Support Vector Machine and Neural Network. In the result, the system shows that the decision tree provides the best performance.


I. INTRODUCTION
Predicting market demand and customer satisfaction requires an extensive market research to predict the market demand and customer satisfaction. Agile software development (ASD) published for helping engineer in business management problem, has been modified by applying it in the business intelligence field for producing the fast analytics in data science. Recently, business intelligence known as BI has become an important analytical procedure for analyzing market demand and satisfaction. BI is related as a technology used to analyze the business data regarding to the decision making process based on prediction analysis and provided the data enhancement process followed by an analytical technology which is widely used in several aspects such as market intelligence, e-commerce, e-health etc. [1,2]. Currently, researchers have been accomplishing studies and concluding that BI affords two things which are process and product. Process defines as a procedure that is used to produce the important information regarding the business process. Product defines as information about empowering organizations to predict the service, customer satisfaction, competitor, marketplace, technology, etc. These two important things become a necessary part in market research and have a powerful analysis result to predict the market demand and customer satisfaction [1,3].
Regardless of enthusiasm in the advantages of BI, conducting the BI analysis process needs a deep analysis. Sentiment analysis is one of approaches that is widely used to support the BI analysis process. Sentiment analysis is defined as a text data analysis that provides a deep analysis about sentiments, opinions, even expressed emotions and it allows us to predict the chances regarding to the analysis result [4].
Sentiment analysis is widely used in several cases, particularly in the review analytics cases, for example, in the analysis of praise and complaint sentences. According to the previous study, praise sentences indicate a positive subset which consists of adjectives, intensifiers, etc. Meanwhile, the complaint sentences showed a negative subset. However, in the real world, praise and complaint sentences have been delivered in the complex form. Hence, a deep analysis such as sentiment analysis may help to resolve this problem [5]. Other cases, sentiment analysis also successfully gives an effective recommendation in analyzing the customer review about airport services [6,7,8]. Sentiment analysis also has a powerful performance for giving recommendation on travel destination [9,10,11,12,13,14]. Considering these facts, it can be concluded that sentiment analysis is suitable for giving recommendation and it may also give a suitable result in predictive analytics for business intelligence. Despite the powerful performance of sentiment analysis, the complexity of data also should be considered. Twitter posts are one of sentiment data that have high complexity and widely used for sentiment analysis. Twitter is a social media that is often used to share information, news and opinions. Previous study has reported the number of Twitter user is rapidly growth up in the last decade which turned into a popular media to express the public opinions [15].
In this study, a business intelligence based on customer review analytics was proposed to investigate and analyze the business solution in GO-JEK application case. We want to analyze the performance of sentiment analysis in evaluating customer satisfaction with GO-JEK services. We also want to know how the customer review to the GO-JEK services and want to evaluate what the services that are satisfied enough and what the services that need to be improved. In the business process, these analyses might become a suggestion for improving the services. Hence, conducting sentiment analysis to evaluate the GO-JEK is important for the business process.
GO-JEK was chosen because several countries were against GO-JEK to be operated in their countries such as Malaysia. Several opinions, reactions and perceptions are popped up about this case. Hence, we performed predictive analytics for business intelligence based on customer reviews in order to analyze how the GO-JEK customer responses about their services and to investigate the chance of GO-JEK to be operated in Malaysia. For the dataset, we use Twitter posts considering the trend of Twitter usage for expressing public opinions.
To address these research gaps, we presented a sentiment analysis on the customer review collected by Twitter posts. To handle sentiment classification task, we can use available commercial cloud services tools. Some cloud services that have been employed by other researchers are Amazon Web Service [16,17], IBM Cloud [16], Google Cloud Platform [16] and Microsoft Text Analytics [16,18,19]. However, Arijit and William in [18] found that Microsoft Text Analytics was 10-20% better in identifying positive and negative sentiment when evaluated in various tweet datasets. Furthermore, Qaisi and Aljarah wrote that Microsoft could benefit from Twitter page by growing promotion offers to the customers to enhance their loyalty and satisfaction [19]. Therefore, according to the evidence stated by previous works, we performed Microsoft Text Analytics for making a ground truth data. Moreover, machine learning classifiers were approached for predicting the customer satisfaction analysis. We also analyzed the business process according to sentiment analysis result for making market decision and increasing the services.

I. METHODS
This research is conducted in four processes as depicted in Fig. 1. These all steps are described in the following subsection.

A. Dataset
In this study we used Twitter posts collected from the Twint library. Data collection was done by mining data using the following rules: 1. Twitter posts are filtered only in the countries that allows GO-JEK to operate such as Singapore, Thailand and Vietnam. 2. Twitter posts are collected since January 1, 2019. 3. All Twitter posts that contains of GOJEK tweeted by the Twitter users in Singapore, Thailand and Vietnam. 4. All Twitter posts that contains both "GOJEK" and "Singapore" keywords. 5. All Twitter posts that contains both "GOJEK" and "Thailand" keywords. 6. All Twitter posts that contains both "GOJEK" and "Vietnam" keywords. According to rules above, we got the data as shown in the Table 1. Prananda & Tholib Journal of Information Systems Engineering and Business Intelligence, 2020, 6 (1), 1-8 3 B. Pre-processing Since the dataset were collected from the Twitter post which consist of many varied data, we should conduct a preprocessing step to normalize the data. In the first step, we performed the tweet cleaning process. We removed the duplicate tweet and cleaned the URL and images that exist in tweet.

C. Sentiment Analysis
After conducting pre-processing step, we worked an available premium API for text analytics namely Microsoft Text Analytics. This API worked by extracting a sentiment score from the given document ranged between 0 to 1 which 1 indicated the most positive sentiment and 0 indicated the most negative sentiment [20]. The analysis models used by the API were retrained using an extensive body of text and natural language technologies from Microsoft [20].

D. Top Words Extraction
This step was used to extract and analyze which features that have most number in negative and positive classes. We employed the combination of 1-gram and 2-gram token to split the tweet. This process split the sentences into one-word and two-words clause. The extracted word clauses represent the service. The top word clause in negative sentiment indicated that the services are not satisfy the customer and needs to be improved, while the top word clause in positive sentiment indicated that the service needs to be maintained.

E. Development of Classification Model
In this stage, we built a predictive model using machine learning that will be used by the businessman to classify new data. Before going through classification process, we balanced the data distribution of each class and the population that described in Table 2. After the process done, we split the sentence using tokenization technique and removed the stop words that exist in the tweet body. Furthermore, we employed the bag of words vector to convert the tweet text into binary for the classification purpose. After that, we trained the data using four machine learning algorithms which are neural network, support vector machine, Naïve Bayes and decision tree. Multinomial Naïve Bayes (MNB) extracted information without seeing the sentence many times in a document which make this method to be one of the best models to solve problems in text classification [21]. Furthermore, the benefit of using Naive Bayes was that this method only requires a small amount of data training to estimate the parameters in classification. It helped to determine exact probabilities for hypothesis, and it is robust to noise in input data [22]. Support vector machine (SVM) is a classifier algorithm that often used in both classification and regression. It worked by finding a hyper plane on the input, where the best hyper plane is the one which located in the middle between two sets of classes, look for the best hyper plane equivalent to maximize the margin or distance between two sets of objects from different classes [23]. Neural networks have been employed in handling document classification task. This algorithm reaches good results on complex domains and allows performing fast testing. However, the training process is very slow [24]. Neural network methods are generally better compared to other traditional machine learning algorithms, and it is effectively extracting the semantic features of text [25]. Tree-based classifiers like decision tree algorithm have also been studied for text analysis task, for example document categorization [26]. This algorithm is used effectively in various fields for classification purpose [27], and it is a very fast algorithm for prediction and learning task [28]. Furthermore, this algorithm has been proofed for its robustness in handling categorical data [29]. Considering the robustness of the four mentioned classifiers, we used these all classifier and evaluated each performance.

A. Pre-processing and Sentiment Analysis
We collected 3111 tweets from January 1, 2019 that contain keyword of "Gojek" and "Singapore", "Gojek" and "Vietnam", and keyword "Gojek" and "Thailand". After collecting the data, we perform the pre-processing step by removing any unwanted data. The pre-processing step successfully removed 163 which were duplicated tweets. In the next step, we perform Microsoft Text Analytics API to give label for each data. This process obtained 2055 neutral sentiments, 666 positive sentiments and 127 negative sentiments considering the number of sentiment score provided by Microsoft Text Analytics API. These results are illustrated in Fig. 2. An example of labelled tweet is drawn in Table 3. The sentiment score of each label provided by Microsoft Text Analytics API is drawn in Table 4.

B. Top Words Extraction
In this stage, we extracted the top positive and negative sentiment. This extraction process was done by determining the highest number of sentiments in both positive and negative sentiment. Negative sentiment was extracted to determine which services that were often complained by customer. Positive sentiment was extracted to determine which services that should be maintained. Results of this step are illustrated in Table 5 and Table 6. Table 5 shows the top three keywords indicating the negative sentiment. However, in the result, we found keywords that ambiguous, for example "via YouTube", "like", "di", therefore we dropped them.

C. Development of Classification Model
After giving label in each data and analyzing them, we used the labeled data for developing the classification model. This model was used to predict the new data. We applied machine learning approach to develop the best classification model. In this research work, we applied four machine learning algorithm which are neural network, support vector machine, Naïve Bayes and decision tree. To evaluate the performance of each model, we used precision, recall and f1-score calculation. These parameters were often used to validate the performance of classification model and have good performance in text classification cases.
In our experimental result, we achieved performance evaluation as shown in Fig. 3. The comparison results of each algorithm also shown in Fig. 3.

III. DISCUSSION
According to the sentiment result depicted in Fig. 2, we conclude that the most sentiment number is neutral sentiment. Furthermore, Table 5 indicates some GO-JEK services that need to be improved, for example their driver's performance. On the other hand, regarding to the positive sentiment, GO-JEK services related with 'ride hailing', 'available consumer' and 'consumer commodities' should be maintained to provide more customer satisfaction as shown in Table 5. However, in the top words of negative sentiment, there is 'tidak bisa', which is a non-English sentence. This is caused by Indonesian users tweeting from Singapore about Gojek. For example, tweet from user with username @norikadwight: "@gojekindonesia sore, saya ada notif untuk paylater nya gojek. Saya mau bayar tapi tidak bisa karna saya tidak bisa akses akun, saya posisi di singapore sekarang" and from @deeadrianto: "@gojekindonesia gojek sudah masuk ke Singapore. Tapi kenapa bagi pengguna gojek indonesia apps gojek tidak bisa langsung dipakai di sana, diharuskan memasukkan DBS card?". In the future, it is needed to remove the non-English tweets, for example by checking the user's region in the users metadata.
According to the experimental result of sentiment analysis described in the section C, we conclude that Microsoft Text Analytics was very useful to give data label since unlabeled data were much published. This tool is proper to be used in further research which does not have a ground truth data. However, we did not perform a manual rechecking for the classification result from Microsoft Text Analytics.
Regarding to the classification result, it can be concluded that decision tree algorithm obtains the highest score in precision, recall and f1-score with 0.55, 0.55 and 0.55, respectively. The performance results illustrated in Fig. 3 seems small caused by the limitation of the dataset that we had. For further research, increasing the number of data is needed to develop a better model of classification. Prediction system would be a robust system if their model has been extracted varied data. Hence, a big number of data might improve the performance result. This procedure might improve the performance evaluation of classification model.

IV. CONCLUSIONS
Business intelligence is an important analysis in business process. In this research work, we proposed a business intelligence analysis by using sentiment analysis method based on customer reviews. We choose GO-JEK case study in our research. Our proposed sentiment analysis scheme consists of pre-processing, sentiment analysis, top words extraction, and development of classification model. In the experimental result, we achieve positive sentiment number of 666, neutral sentiment number of 2055 and negative sentiment number of 127. According to these results, we conclude that most of GO-JEK customers were satisfied with the GO-JEK services considering the number of neutral 7 sentiments. In this research work, we also developed a classification algorithm to predict the sentiment analysis of new data. We used neural network, Naïve Bayes, decision tree and support vector machine as classifier. Our experimental research concludes that decision tree is the best classifier. For the further research, it is necessary to conduct balancing data method to improve the performance of classification model. Furthermore, in the keyword extraction stage, we still found some keywords that did not represent the services of the company. Therefore, we suggest for the future research to get the list of services provided by the company in order to tackle the ambiguous result in top words extraction.