Practice makes the expert: The importance of training volunteers in the generation of phenological data from photographs of biodiversity observation platforms

and untrained volunteers were different for each species to avoid a possible bias due to learn- ing experience. In total, each rater scored three databases, except for L. nepetifolia reviewers without training, who were assigned five databases per rater. Statistical analysis The level of agreement was calculated by group of raters (experts, trained volunteers, and untrained volunteers) and by phenophase, for each species, using Fleiss’ Kappa index (K) [42]. This index measures the degree of agreement between n raters considering all the categories in which it can be classified, in this case S, N, Dk, and considering that the identity of the raters may be different for different study subjects, in this case, photographic records. The "irr" pack- age version 0.84.1 [43] was used to calculate the index and its statistical significance by means of a two-tailed z-test. To estimate whether the level of inter-rater agreement differed between groups, for each phenophase a x2 equals k0 s, test was performed, which is a test originally proposed to evaluate differences between Kappa indices. If significant differences were found between groups of raters, two-tailed z-tests were performed per rater group pair [44], for which the standard error of the mean obtained by bootstrapping with 1,000 random resamples was used [45]. Additionally, the distributions of the K-index frequencies for each group were analyzed by bootstrapping, and the results were compared to what was found by the x2 equals k0s, tests (S1–

S4 Figs). Generalized linear models with a binomial distribution and logarithmic link function [46] were used to explore the relationship between the proportion of observations rated as Dk as the dependent variable, and the rating group, the species rated and the interaction between the latter two factors as independent variables. The species factor was considered to determine whether the ease or difficulty of the volunteers in classifying phenophases is independent of the species if species with clear phenophases are considered. The model chosen, following a backward selection, retained only the components that were significant. Subsequently, Tukey tests were performed for multiple contrasts using version 1.4–19 of the “multcomp” package [47]. All computations and data analyses were performed using R 4.0.0 [48], with evaluations of statistical significance made at α = 0.05. Results Classification of photographic records and comparison between trained and untrained groups of volunteers The results were based on the classifications made by 24 trained volunteers, 25 untrained vol- unteers and five experts. Of the trained group, the 24 volunteers classified 1,014 observations of L. nepetifolia and 23 classified 1,044 observations of N. glauca. The observations were there- fore evaluated independently three times by three groups (nine independent raters per obser- vation). Eleven volunteers from the untrained group classified 811 records of L. nepetifolia and 14 classified 433 observations of N. glauca. The five experts participated in classifying 84 obser-

vations of L. nepetifolia and four of them were involved in classifying 214 observations of N. glauca (S1 Appendix). The proportion of observations scored as Dk varied between rater groups (χ2 = 179.408, 1.35, df = 3, p < 0.001) (S2 Appendix). Neither the species nor the species-group interaction was significant, i.e., the untrained volunteer group rated photographs as Dk in a higher pro- portion, and the expert group in lower proportion, regardless of species. Neither species nor the species-group interaction was retained in the generalized linear model chosen to explain the proportion of records scored as Dk. The expert group rated observations as Dk in the low- est proportion (16% of observations, z = -11.790, p < 0.001), followed by the trained group PLOS ONE The training of volunteers as a promoter of phenological studies PLOS ONE | https://doi.org/10.1371/journal.pone.0282750 March 7, 2023 7 / 17