Short text classification with machine learning in the social sciences: The case of climate change on Twitter

where zp−1 and zp are vectors of the inputs and outputs of layer p, respectively. Then Wp is the matrix of weights between the input and output neurons of the layer, while bp is the vector of bias terms of the given layer. Moreover, ap(�) is the activation function, which is often chosen as sigmoid, hyperbolic tangent (tanh), or rectified linear unit (ReLU). Also note that here the input of the first layer is given by the input feature vector z0 = x. Meanwhile, during the train- ing, the output of the last layer in the network is set to the true label, i.e., zP+1 = y. For a new data point, the output of the last layer provides the predicted label, i.e., zPþ1 ¼ ^y. Learning for feedforward NNs is usually done by means of backpropagation. The NN parameters (i.e., weights and biases) for each layer are adjusted based on the adopted loss func- tion which captures the difference (or error) between the observation, y, and the output of the forward pass based on current parameters ^y. The adjustment is done based on the loss function (1). This learning process is often carried out by means of the stochastic gradient descent, or any other known optimization algorithms used for learning, such as, e.g., Adam [59] and RMSProp [60]. This involves recursively propagating the gradients of the parameters back- wards, from the last to the first layer, similarly to Eq (5). It is noteworthy that a mini-batch ver- sion of the gradient update is often used for more robust convergence. More concretely, the

training dataset is split into small batches for which the model loss is calculated and model parameters are updated. In this way, the batch updates are computationally more efficient because they do not need the training data to be held in the computer’s memory. The advantages of FCNNs are their ability to solve complex non-linear problems, efficiency of handling large amounts of data, and quick predictions after the slow process of training is completed. Their disadvantages are slow training, poor interpretability due to their black- box nature, large numbers of parameters due to the fully-connected structure, and ignoring spatial information by accepting only vectorized inputs. For our analysis, we have used the same architecture for all NNs under consideration, with difference in only a single layer that is particular to the given NN. For FCNN, this layer com- prised a fully-connected layer with a certain number of units and a ReLU activation function. The number of neurons was optimized, alongside other hyperparameters, such as batch size, number of epochs, dropout rate, learning rate, regularization strength, embedding dimension, and maximum vocabulary length. Convolutional neural networks. Convolutional neural networks (CNNs) refer to another subclass of feedforward NNs that are designed to capture spatial and temporal dependencies through the application of kernels. CNNs have the benefit of having a reduced number of parameters and reusable weights. The use of CNNs has become a state-of-the-art approach for

the analysis of images [16]. They have also been shown to be useful for text classification [55, 61]. A CNN consists of two parts, where the first part is dedicated to feature extraction, while the second part performs the actual classification by means of an FCNN (see Fig 5 in the S1 Appendix). CNNs often start with an embedding layer that maps a discrete variable to a vector of continuous numbers. This is done by means of a single fully-connected layer (or a set thereof) that is either learned during the training or substituted by a pre-trained embedding. In the subsequent convolution layer, a set of fixed-size kernels slide through the list of embed- dings, performing a convolution operation. This layer functions as the CNN’s feature extractor because it learns to find local spatial features in the output of the embedding layer. The size of the kernel is the number of embeddings it sees at once multiplied by the length of an entire word embedding. The output of this layer is a set of feature maps that serve as the input for the subsequent pooling layer. Zero-padding is often used in this step, surrounding the input so that a feature map does not shrink. PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 14 / 26