Short text classification with machine learning in the social sciences: The case of climate change on Twitter

The poor classification performance of all the classifiers discussed above can likely be attrib- uted to class imbalance in the dataset. To investigate the influence of this imbalance, we pre- pare a balanced dataset. To do this, we supplement our original dataset with additional tweets taken from Kaggle’s “Twitter Climate Change Sentiment Dataset” [85]. This dataset contains 43, 943 tweets about climate change, manually labeled for sentiment. We are not interested in the sentiment labels, but instead take all of these tweets as ones about climate change (or labeled with “1”). We pick enough samples from the Kaggle dataset to balance out our own dataset and run the same algorithms on the new balanced dataset. We ensure that the same train/test split (i.e., 85%/15%) is maintained and the training and test sets in the original data- set remain parts of the new training and test sets, without mixing between the two. In this way, we make sure that the models see only new data during the performance evaluation stage. Table 4 shows how the classification algorithms perform on the new, balanced dataset. It is notable how dramatically the overall performance scores have improved when the classifica- tion methods are applied to the balanced dataset. Almost all methods now exhibit very high F1 scores, larger than 96%. The only outliers are the KNN algorithm (F1 � 90%) and the lexicon- based classifier (F1 � 86% only). We note that the poor performance of the KNN classifier, compared to other ML algorithms, is in line with the conclusions of [86]. Meanwhile, the

Table 2. Classification performance of considered classifiers on the original collected dataset. Method Accuracy Precision Recall F1 score AUC ROC AUC PR MCC LR 0.957077 0.829268 0.531250 0.647619 0.913945 0.694443 0.643570 RF 0.952436 0.716981 0.593750 0.649573 0.924048 0.685173 0.627497 Lexicon 0.957077 0.909091 0.468750 0.618557 0.732525 0.709352 0.635333 SVM 0.941995 0.606061 0.625000 0.615385 0.881050 0.641829 0.584106 LSTM 0.944316 0.642857 0.562500 0.600000 0.863565 0.591703 0.571686 FCNN 0.938515 0.593220 0.546875 0.569106 0.890488 0.615837 0.536574 NB 0.937355 0.586207 0.531250 0.557377 0.919447 0.617646 0.524492 CNN 0.935035 0.568966 0.515625 0.540984 0.891115 0.605069 0.506828 KNN 0.938515 1.000000 0.171875 0.293333 0.887287 0.598238 0.401461 Note: The results are sorted by the preferred metric, highlighted in bold font. https://doi.org/10.1371/journal.pone.0290762.t002 Table 3. Training and execution times (in seconds) of considered classifiers on the original collected dataset. Method Training time [s] Execution time [s] NB 0.004 0.008 KNN 0.004 0.156 Lexicon 0.246 0.102 LR 0.255 0.004 SVM 0.485 0.136 RF 0.897 0.160 CNN 119 0.200 FCNN 133 0.319 LSTM 185 1.050 Note: Here, training time is the time required to train an ML model on the training set. Meanwhile, execution time is the time required to run a trained ML model on the test set for the given dataset size and split (85%/15%). https://doi.org/10.1371/journal.pone.0290762.t003 PLOS ONE

Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 19 / 26