Short text classification with machine learning in the social sciences: The case of climate change on Twitter

The layers in between are called hidden layers, as they are not directly visible from the outside of the black box. An NN is referred to as a deep NN if it contains two hidden layers or more. Note that NNs could equally well be adapted for regression and multi-class classification tasks. For the latter, the last layer would consist of the same number of neurons as there are classes, each having a softmax activation function and outputting the probability of belonging to that class. It has been proven that sufficiently deep NNs are able to approximate any arbitrarily com- plex function f(�) [52]. Hence, deep NNs are particularly useful in applications where there are large amounts of data and significant computational resources available, and where manual analysis is tedious or the performance of traditional ML algorithms is unsatisfactory. In the context of text classification, NNs were first applied in [53, 54]. Only recently have they started to be used for political science research [55, 56]. There are various neural architectures avail- able for this task. We review the most relevant ones below. Fully-connected neural networks. The first attempt to combine artificial neurons into a layer was made by Rosenblatt [57]. He devised the concept of a perceptron, a binary classifier mapping an input feature vector into one of several available classes. Formally, given a feature vector x, the perceptron outputs ^y ¼ a X M j¼1 wjxj ! ; ð4Þ where fwjgM j¼1 is the set of weights, and a(�) is the activation function. This step is referred to as

the forward pass. Given an actual training output y, the algorithm readjusts its weights accord- ing to the gradient descent rule wj wj þ aðy ^yÞxj; ð5Þ where α is the so-called learning rate which determines the convergence speed of loss minimi- zation. This adjustment process is referred to as the backward pass. The process continues iter- atively until the loss function in Eq (1) is below an acceptance threshold. If the classification problem is linearly separable, the perceptron is guaranteed to converge, and the predictor function is given by the forward pass with the learned weights wj for all j = 1, . . ., M. In a nutshell, the perceptron is an NN with a single layer. To improve its performance, Wer- bos [58] proposed stacking several perceptrons in consecutive layers. That study coined the concept of a multi-layer perceptron, also known as a fully-connected NN (FCNN). FCNNs exhibit a much better learning ability than that of the perceptron, and are the simplest form of a general class of feedforward NNs, which are neural architectures without cycles, being the most common architectures used nowadays. An FCNN comprises a set of fully-connected (or dense) layers stacked in a line topology (see Fig 6b). Due to the presence of multiple layers, an FCNN is capable of having multiple lev- els of abstraction. An example is its use for facial recognition [56]. Based on an input image, the first layers of the NN capture the simple characteristics of the image, such as oriented edges or corners. Then further layers react to more complicated shapes, such as noses or eyes.

Finally, the last layers are able to detect the face itself. In this way, adding extra layers to the NN can enable solutions to a lot of otherwise non-separable problems. For a deep FCNN with P hidden layers, the input-output relation at each layer p = 1, . . ., P + 1 is given in a matrix form as zp ¼ apðWpzp 1 þ bpÞ; ð6Þ PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 13 / 26