Short text classification with machine learning in the social sciences: The case of climate change on Twitter

The pooling layer then subsamples the feature maps input to it. It does this by selecting a single element from each region of the feature maps covered by its filter. The pooling layer operates on each feature map independently. This reduces the size of the feature representa- tion, thereby effectively reducing the amount of parameters in the network, whilst preserving the most prominent features. Oftentimes, a dropout layer is also applied as a means to improve generalizability. By destroying the learned co-dependencies between neurons that compensate for errors from the previous layers, overfitting is prevented at a cost of longer training. The output is then fed into a fully-connected layer (or an FCNN) that conducts the classification based on the features extracted by the previous layers. CNNs have important advantages, such as automatic feature extraction, with the possibility of using those for new tasks, and weight sharing, which leads to a reduced number of parame- ters. They also have lower computational complexity than FCNNs. Their drawbacks are ignor- ing the position and orientation of the object in their predictions, their need for large amounts of data, and long training times. In our performance comparisons, the CNN architecture includes an embedding layer, a dropout layer, a convolution layer and a max-pooling layer with output flattened to match a single neuron for final classification. We optimize the same hyperparameters as we tuned for the FCNN, except for the number of units. Instead, for a convolution layer we tune the number

of filters and the size of the kernel. Long short-term memory neural networks. An alternative to feedforward NNs for text classification are recurrent neural networks (RNNs). These are NN architectures that are com- prised of a chain of repeating modules. This allows previous outputs to be used as the next inputs of the same neural layer. Because of this feature, RNNs are able to incorporate context from previous steps when dealing with a current step. They can use their internal state—also known as memory—to process sequences of inputs with variable lengths. Unfortunately, plain versions of RNNs suffer from the vanishing gradient problem during training with backpropagation. This limits the number of passes information can make through RNNs, making them impractical. A solution to this problem, so-called long short-term memory (LSTM) NNs, has been proposed [62]. This is nowadays considered as one of the breakthroughs in the field of DL. An LSTM neural network is an RNN with an LSTM cell in place of a neural layer (see Fig 6 in the S1 Appendix). This can let information pass through the cells, as well as adding/remov- ing information while it is passing through a cell. This is regulated by various gates, each com- prising a neural layer and an element-wise multiplication operation. The neural layers are equipped with a sigmoid activation function, which determines the amount of information that should pass through (between 0% and 100%). The forget gate then controls what informa-

tion is kept and what is thrown away from the cell state. It comprises a neural layer with a sig- moid activation. Next, the input gate decides what information to update and what to store. This is followed by a layer with hyperbolic tangent (tanh) activation that outputs a vector of new candidate values that could be added to the cell state to update it. After this, the output gate decides what information to output, which is followed by yet another tanh gate that takes the current state as an input and multiplies the result with the output of the output gate. The benefits of LSTN networks are their ability to handle long-term dependencies, avoid- ance of the vanishing-gradient problem and efficient handling of complex sequential data. The disadvantages are their need for large amounts of data and, consequently, slow training and high computational complexity, as well as their worse performance on data that is not of sequence type or contains a lot of noise. In practice, a bi-directional model [63] is added on top of the LSTM layer. This runs sepa- rately in forward and backward directions, concatenating the predictions of both models, and PLOS ONE Short text classification with machine learning in the social sciences PLOS ONE | https://doi.org/10.1371/journal.pone.0290762 September 29, 2023 15 / 26