Ecological assessment produces three separate “products”:

1. The nuggets of transferable information (EQRs and ecological status classifications in the case of the WFD);

2. Underpinning data (lists of taxa and their relative abundances); and,

3. Collateral benefits, as engagement with samples and surveys increases an individual’s knowledge and understanding of the ecosystems under consideration.

The focus of the UK phytobenthos metabarcoding project was primarily on the first of these. Most of those managing the projects had “head office” functions with responsibility for national classifications and strategy, and on reporting UK compliance. Although taxa lists played an important role for interpreting assessment results in the past, their primary role for statutory assessment is as raw material for metric calculation. Budget constraints have led to a greater focus on standardized approaches to reporting and risk assessment with fewer opportunities for local staff to interpret diatom outputs.

There are substantial differences in the relative abundance of diatom species in lists produced by conventional and metabarcoding approaches (Vasselon et al. 2017; Kelly et al. 2018). The UK response was to produce a recalibrated metric optimized for metabarcoding data: an acceptable ‘fix’ if the focus of assessment is the final metric value but with implications for data interpretation (see below).

The ability of metabarcoding to detect species is dependent, to a large extent, on the quality of barcode libraries (Weigand et al. 2019). For diatoms, these currently include only about 20% of known freshwater species, with a bias towards common taxa (Rimet et al. 2019). Extending taxa dictionaries to encompass all microalgae would be prohibitively expensive so there is a need for an informed conversation about alternative approaches. The extreme position is ‘taxonomy-free’ approaches (Apothélez-Perret-Gentil et al. 2017), which offer no opportunity to linking to a wider knowledge base but there are many intermediate options. When considering these, however, the WFD’s emphasis on stakeholder engagement (Article 14: “Public information and consultation”) also needs to be considered. The taxa list should provide a bridge between the evidence upon which classifications are based and an understanding of the ecosystem which, in turn, can be communicated beyond a narrow group of specialists.

Both the absence of taxa from barcode libraries and difference in their representation in light microscopy and metabarcoding data need to be understood by those engaged in interpretation. This is important because, although classifications are broadly comparable, it is possible that individual sites within a data cloud may be classified differently using light microscopy and metabarcoding (see below). Responsibility for explaining such shifts often falls to local staff who will, in turn, need to understand and explain the reasons, for which a list of taxa and abundance assessments are necessary. The taxa list also provides a link to a wider pool of knowledge, including ecological traits (Rimet and Bouchez 2012; Tapolczai et al. 2016) which would enable a local ecologist to interpret changes in a community over time. Thus, the implementation of a metabarcoding program needs to be accompanied by a structured training program accessible to all who might need to interpret the data, to ensure that they are aware of how metabarcoding differs from the traditional data with which they are familiar. However, to date, there has been relatively little focus from management on helping local ecologists understand differences between the traditional and molecular data.

Whilst the previous paragraph ended with a call for a “structured training program” as part of the transition to metabarcoding, this one starts by lamenting the decline of a widescale “unstructured training programme” arising from the engagement of individual ecologists in the UK with all stages of assessment, from sampling to interpretation, within a relatively small geographic area often for several years. A business strategy that prioritises ‘efficiency’ over depth of technical knowledge and resilience in its experts has led to fragmentation of the data chain in recent years. This, in turn, leaves ecologists increasingly frustrated in their desire to develop the detailed local ecological understanding needed to properly assess and interpret environmental data. Because sampling phytobenthos for metabarcoding is a straightforward process that can be delegated to non-specialist technicians, after which all aspects of metabarcoding analysis take place in remote laboratories, an ecologist’s first encounter with a sample may be as a list of taxa in a spreadsheet, or even as a final metric or classification result. As a result, it is increasingly difficult for ecologists to contextualise data as they are no longer so familiar with the locality from which the data are derived. The implications of this may take a few years to become apparent as the first generation of ecologists who interpret metabarcoding data will have this experience on which to draw. After that, will there be any capacity for an informed “reality check” of the outcomes of bioinformatic pipelines?

Hering et al. (2018) reviewed implementation options for metabarcoding in Europe and identified two broad strategies:

• Option 1: Like-for-like replacement of existing methods. In other words, use NGS as alternative means of data acquisition, but continuing with existing principles behind metrics, reference conditions and status class boundaries;

• Option 2: Adopt the “Biomonitoring 2.0” philosophy propounded by Baird and Hajibabaei (2012), developing a new generation of indices using the full potential of NGS data to explore the greater taxonomic depth that is available, a wider range of organisms and, potentially, to develop metrics that measure ecosystem function as well as structure.

The UK phytobenthos method fits “option 1”. When development started there were no studies that had used NGS to evaluate the response of diatom assemblage composition along environmental gradients so the foundation of evidence that existed for LM studies provided a strong hypothesis for determining what taxa we should expect to detect in our metabarcoding output. In brief, if a species is present in a LM analysis, you should expect to find it in NGS output. If not, then you should take the opportunity to understand why this is not the case. This hypothesis also prompted us to ask questions about quantification (why is a particular taxon less abundant in NGS output than in an LM analysis, and vice versa?) and the breadth of our taxa dictionary. The opposite also applies: if a species is present in NGS output, then does this mean that it has previously been overlooked using traditional analyses (Blackman et al. 2017)? Knowing that some taxa abundant in metabarcoding data were absent from light microscopical analyses of the same sample also allowed us to track down some faults in the laboratory and bioinformatic stages. Having learnt how NGS output relates to the existing knowledge base, we are in a stronger position to move forward and start to exploit the potential of NGS more fully.

A further point is that a project that sets out to develop a practical tool for ecological assessment is constrained by the wording of current legislation and regulations that it will be used to enforce. In the case of the WFD, the normative definitions for ecological status refer to “composition” and “abundance” and Member States are required to intercalibrate their methods in order to ensure that all share a common level of ambition with respect to key WFD objectives (Birk et al. 2013). This, too, should be more straightforward if the first generation of metabarcoding methods produce output that can be readily related to that from neighbouring countries.

On the other hand, the UK experience has, to date, not yielded a method that is appreciably better than the existing one, leading some to ask whether our conservative “option 1” approach was justified. In particular, uncertainties associated with both LM and NGS approaches mean that the classification of some water bodies will change solely as a result of the switch in method (see below). Taking a longer perspective, the current NGS method has laid foundations upon which methods that better exploit the potential of NGS can be developed. However, at the time of writing, no-one has proposed a strong assessment concept for phytobenthos that could improve upon, and potentially replace, the present “option 1” implementation.

Moving from methods based on identifying organisms based on their morphology to identification based on genetic traits involves a major paradigm shift, not just in our taxonomic understanding, but also in how data are obtained, and their ecological meaning. Most of those responsible for making decisions about the implementation of metabarcoding trained in the era of morphology-based taxonomy and ecology, and had little or no understanding of the laboratory and bioinformatic stages involved in producing NGS output. Conversely, those responsible for primer design, DNA extraction and PCR, high-throughput sequencing and development of the bioinformatics pipeline were excellent molecular biologists but had little awareness of phytobenthos ecology or the regulatory framework within which the method would be used.

A further complication was that diatom taxonomy is also in the midst of its own paradigm shift, leading to reviews of species and generic limits (Ruck et al. 2016), the description of many new taxa (e.g. Kahlert et al. 2019) and a better understanding of just how much we don’t know (Mann and Vanormelingen 2013). Molecular genetics has played a major role in this process and, indeed, informed the initial choice of a barcode for the metabarcoding tool (Mann et al. 2010). However, once again, appreciating the scale and implications of this was a challenge for managers (mostly from disciplines where taxonomy was more settled) responsible for determining the course of metabarcoding implementation.

Development of metabarcoding-based methods for ecological assessment inevitably calls for multidisciplinary teams, with individuals playing to their strengths. There is also a need for a measure of mutual understanding: awareness of how NGS data are produced by regulators and “old school” ecologists, awareness of the underlying legal context by bioinformaticians, awareness of the “natural history” of organisms by those whose focus is either high-level catchment management and nationwide strategy or, on the academic side, analysis of “big data”. With the benefit of hindsight there was not enough overlap between the skills of the different specialists involved in the development of the UK diatom metabarcoding tool, and some basic education in each other’s perspectives would have been fruitful. This was particularly obvious at later stages when a wider pool of individuals needed to be engaged yet where, due to pressures of time and budget, there were more teleconferences and fewer face-to-face meetings. The former, typically constrained to no more than a couple of hours, gave scant opportunity to understand and learn.

There should be a general expectation that the process of environmental regulation is stable and, when a change does happen, we are given due warning and explanation. A utility company, for example, may spend some years evaluating the case for, and design of an improvement to a wastewater treatment plant, with an expectation that the investment would be recouped over a set time period. They would not anticipate, for example, the regulator adjusting targets during that period due to a change in the way that ecological status was evaluated. When a change does occur, they would expect this to be supported by sound evidence.

Yet genetic technologies promise greatly enhanced capacities to evaluate the environment and, in these early years of implementation, we are learning all the time. Is it not in the public interest to incorporate this new knowledge into regulatory regimes as soon as possible? Finding the balance between the competing demands for “stability” and “improvement” (and, in the latter case, sifting the genuinely useful from the over-hyped) is going to be one of the major challenges facing those charged with managing the transition to molecular approaches for ecological assessment over the next decade and beyond.

Two extreme views expressed during this project were that molecular methods should not be adopted until their development had “stabilised” or that they should be adopted as soon as they were fit-for-purpose but be “locked down” so that regulation was based on a single, unambiguous implementation of the method. The former argument is not tenable because of the rapid evolution of metabarcoding technologies. When development of the UK diatom metabarcoding method started, the Roche 454 was the state-of-the-art platform for NGS and primers were designed around its capabilities. During the course of the project, this machine became obsolete and primers had to be re-designed to meet the requirements of the Illumina Mi-Seq. There is no reason to assume that this, too, might follow the Roche 454 into obscurity at some point and that the method will need to be fitted to a new platform. There are, in addition, exciting new technologies available, such as Oxford Nanopore’s Minion (Grey et al. 2015), which might well make molecular analysis more accessible to field-based ecologists without the need to engage a remote HTS laboratory. The argument for waiting for methods to stabilise is simply not realistic in the era of molecular ecology.

In practice, the relationship between phytobenthos assessments based on LM and NGS is not perfect (see Kelly et al. 2018b) and some sites classified as ‘moderate status’ using LM are classified as ‘good status’ using NGS, or vice versa. Systematic elements of this variation can be minimised to some extent (e.g. by calibrating the NGS method to fit established boundaries) and it may also be possible to make a case that the new method is a genuine improvement over the old (e.g. if the relationship with pressure is stronger, or the underlying model for predicting ‘expected’ metric values is improved). However, there is also an element of random variation, with sites shifting position within the data cloud for no explainable reason. This is, remember, no longer an academic exercise of hypothesis testing in the face of noisy data, but a case where a regulator and a utility company may have developed future investment plans using a scenario that they are now being told is wrong.

At the heart of this, lies the WFD’s controversial ‘one out, all out’ rule by which the final status of a water body is determined by the lowest status of any of the measured components of ecological status (Carvalho et al. 2018). In practice, a change in overall status due to a switch to NGS should only occur in those situations where phytobenthos either was, or becomes, the trigger for a downgrade. In many cases, assessments of other organism groups will either confirm or overrule any change that is purely due to the shift in phytobenthos classification. But this will still leave some situations where a change in classification may occur due to the method switch. Sites will be more vulnerable to this if the number of criteria monitored has been pared back to save money, but also where the number of samples contributing to a phytobenthos assessment is small (leading to high standard error and, thereby, uncertainty).

After this initial jolt to the classification regime, we should expect a period of incremental improvements in the science and the next question is whether these should be incorporated into assessments or not. The approach adopted in the UK was to take the version of the metabarcoding tool that emerged after three iterations of the prototype and to “lock down” all aspects of procedure (including bioinformatics pipeline, taxa dictionary and species sensitivities) to ensure consistency in output. For some organism groups this may not be a problem but, given the rate at which diatom taxonomy is developing (see lesson 3), adherence to a fixed list soon diverges from “best practice”. “Locking down” the taxa dictionary and bioinformatics may seem like a means of ensuring stability but, given that over 40% of reads cannot be assigned to sequences in the barcode library, this also acknowledges a large level of uncertainty in outputs. At the point at which the UK barcode library was “locked down”, the rate of unassigned reads was slightly lower than that obtained by French colleagues; two years later, they (who have continued to develop their barcode library and bioinformatics) have halved their rate of unassigned reads (author, Rachel Glover, Frédéric Rimet, Valentin Vasselon, unpublished data).

The decision to “lock down” the method in the name of regulatory stability could well prove to be the biggest mistake in the implementation process. A general principle in quality management is that variation that can be controlled should be controlled whilst that which cannot be controlled should be quantified and incorporated into the decision-making process. In effect, “fixing” the metabarcoding procedure treats a source of potentially controllable variation as uncontrollable. Filling a few gaps in the barcode library is unlikely to have a noticeable effect on the overall strength of fit between the LM and NGS methods, or between the NGS method and principal pressure gradients. However, it might correct the position of a few points within the data cloud and result in better regulation at the locations these represent.

Would this be at the expense of the “stability” which is, clearly, an important requirement of a regulatory regime? In theory (and depending on the means by which key regulatory boundaries are set), any alteration to the means by which a metric is produced could have implications for regulation. In practice, however, such alterations are likely to be minor and, in any case, methods already exist to compensate for this. Over the course of WFD implementation, the intercalibration process has ensured that Member States all share a similar level of ambition when setting ecological status boundaries. These have generated a set of methods (Birk et al. 2013; Poikane et al. 2015) designed to ensure spatial compatibility (i.e. between Member States) but which could also be used to ensure temporal compatibility (within a Member State over time). Indeed, Birk et al. (2015) was written with this express purpose in mind. These should allow the position of status class boundaries to be held constant whilst the underlying metrics are improved. This will, of course, need to be managed with great care and with good version control and high transparency to ensure regulatory stability.

The credibility of methods is strengthened when there is broad international agreement, underpinned by international standards – endorsed either by the International Standards Organisation (ISO) or Comité European de Normalisation (CEN). Conversely, the absence of such a standard may be seen as a barrier to implementation. In the case of diatoms, informal discussions with colleagues performing similar work elsewhere in Europe led to an approach to the relevant technical committee of CEN and, eventually, publication of two Technical Reports (CEN 2018a, b). These are not full-fledged “standards” because there are still important knowledge gaps (e.g. in preservation of DNA over long time periods) but are steps on a longer journey towards full harmonisation of methods. The COST project, DNAqua-net (Leese et al. 2018) has, in turn, provided logistical support for performing some method comparisons which will, over time, contribute to this harmonisation.

A second area where international co-operation has been invaluable has been in the development and curation of a barcode library (Rimet et al. 2019). The importance of barcode libraries has already been stressed (see Weigand et al. 2019): the scale of the task, particularly for species-rich phyla, is such that no single country has enough specialists or resources to complete the task. In view of the limited resources available, there seems to be little sense in each country developing a separate barcode library.