Short text classification with machine learning in the social sciences: The case of climate change on Twitter

The above intuition only works for cases where classes are linearly separable (see Fig 2a). In the general case, when classes may overlap in such a way that it is not possible to fit a separat- ing hyperplane between their data points, the SVM principle can be extended to soft margins. This allows some observations to reside on the other side of the hyperplane [34]. Furthermore, a non-linear transformation ϕ(�) can also be applied to the data, expanding the feature space to a higher dimension in which there may be a linearly separating hyperplane. This transforma- tion can be done, e.g., by adding higher-order polynomial terms. However, a problem with this approach is that it might explode the computational complexity of the algorithm and lead to overfitting. In order to overcome this problem, the “kernel trick” can be invoked. Namely, by means of Lagrangian duality, the maximization of the soft margin is reformulated in terms of scalar products between data points. There are kernel functions k(�) where k(x(i), x(j)) = ϕ (x(i))ϕ(x(j))—e.g., polynomial, Bessel, radial basis functions (RBFs). This means that it is possi- ble to skip computing the data transformation ϕ(x). Instead, only computing the kernel func- tion will give the dot products in the transformed feature space for the optimization. The obtained separating hyperplane translates back to the original feature space as a non-linear decision boundary (see Fig 2b that illustrates an SVM with an RBF kernel on synthetic data). The advantages of the SVM classifier are its operation efficiency with high-dimensional

data, ability to model non-linear decision boundaries, and efficient handling of small datasets. The downsides are its sensitivity to the choice of the kernel function, high computational com- plexity for very large datasets, and the lack of probabilistic interpretation (unlike in LR). Because it is known that text classification is usually linearly-separable [12], we use an SVM implementation with a linear kernel for our empirical analysis. The regularization hyperpara- meter, defining the “softness” of the margin, is tuned to maximize the performance of the SVM. Random forest. Random forest (RF) classifiers [35, 36] are widely used for text classifica- tion (see, e.g., [37, 38]). The algorithm uses the so-called decision tree approach [39, 40]. The latter relies on the intuition that, instead of seeking a complicated mapping function ^f ðxÞ, one can partition the feature space into disjoint regions and fit a simpler model in each of those. This is done during training by recursively splitting the space of possible values for each feature at a certain threshold. For example, at a given iteration, the entire input space is split into a pair of half-spaces {x : xj � tj} and {x : xj < tj}, where tj is a threshold for a certain feature j. The split is done in a greedy way to minimize some loss function (e.g., misclassification error, Gini impurity, or entropy), without looking ahead at future splits. This process is repeated on each of the half-spaces recurrently until a stopping criterion is fulfilled (e.g., there is no information

gain from further splits). In this way, a binary “decision tree” is constructed, which sets the thresholds for the classification of new samples. Once the training is done, a new observation (x, y) is passed through the entire decision tree. As it passes through the tree, the observation experiences a sequence of learned condi- tional statements (greater or lower than a threshold) for each feature xj where j = 1, . . ., M. At the bottom of the tree (in a leaf node, where there is no further split), the prediction is obtained as the majority class of the data points satisfying all the conditions along the path. Illustrations of predictions with decision trees are shown as one block in Fig 3. Each circle depicts a condi- tional statement over a feature with respect to a threshold. The thick lines depict the paths of observations as they travel through the decision tree, satisfying all the conditions on their way. The numbers below the trees indicate the predicted classes. An issue with decision trees is that they typically overfit training data. A widely-used way to improve generalizability is to average the predictions of a number of decision trees by means of bagging [41]. With bagging, several bootstrap training sets D1; D2; . . . ; DB are PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 7 / 26