We rely on expert forecasts to guide conservation. But even experts have biases and blind spots – WhatsNew2Day

When faced with uncertainty, we often look to expert forecasts, from forecasts of election results to the likely outcomes of medical treatments. In conservation, we turn to experts to assess extinction risk or to predict the long-term responses of plant species to fire management.

But how reliable are these predictions? There is a well-known saying: “Prediction is hard, especially when it comes to the future”.

In our new research, we put this to the test. We asked eight experienced ornithologists to predict how bird species react when farmland is revegetated – a common conservation practice.

The result? There was a surprising amount of variation between experts. And there were consistent biases, such as favoring birds commonly seen on farms while underestimating small forest species. However, when we combined their answers, we got better results.

Does this mean we shouldn’t use such expertise? No. Expert knowledge plays a crucial role in conservation decisions.

But like everything, it has limitations that we must recognize. We should treat expert knowledge as a guide, rather than as a source of truth.

How do you put experts to the test?

Expert knowledge is often used for making decisions preservationbut it is rarely tested.

We asked our expert ornithologists to predict which bird species would be found at 20 farm revegetation sites in western Victoria, spread over an area of ​​more than 1,400 square kilometres.

We provided each expert with detailed information about the sites, including a map, the size of the revegetation plot and when it was planted, the number of tree species planted and site management.

Our experts then had to assess how likely it was that specific bird species would be detected there. This was based on a list of over 100 species per locality, which were recorded in the surrounding district.

We then compared their predictions with data from bird surveys of the sites – conducted by several experienced ornithologists – and with a random selection of bird species.

Read more: New research shows planting trees and shrubs brings forest birds back to farms, from beautiful fairy wrens to spotted pardalotes

What did we find? Lots of variation between our experts.

Across the eight ornithologists, the average number of species they believed likely to be detected at sites ranged from 15 to 45. The average recorded from bird surveys was 19.

The predicted bird community composition at each location also varied among experts. Some were closer to their predictions than others, but all differed significantly from the actual observed community of bird species.

We all have biases – and experts are not immune

You might wonder if there were similarities in what the experts got wrong. There were.

Overall, our experts overestimate the likelihood that common agricultural species such as the galah, eastern rosella, willow wagtail and magpie lark would occur. They also overestimate the likelihood of larger species occurring in open country with scattered trees, such as the laughing kookaburra and the cuckoo shrike.

Why would this be? These species are highly visible and common in agricultural landscapes, but they are also highly dispersed and are therefore less likely to occur in a particular location during the survey.

On the other hand, our experts underestimate the presence of small forest birds, such as the brown thornbill, the beautiful wren, the silvereye and the gray fantail.

When we combined the expertise of our eight ornithologists, we saw less variability. Grouped together, our experts far outperformed a random selection of bird species. Yet their predictions still differed greatly from those actually observed.

Why is this important? The loss of forest birds in agricultural areas in South Australia is a major conservation concern. Revegetation helps restore forested habitats for such species. These biases can lead conservation managers to disregard the conservation benefits of revegetation.

The task we gave our experts was not an easy one. Making reliable predictions is complex. They had to make multiple assessments for each species. How common is the species in this region? Was the revegetated area likely to be suitable habitat? How might this species be affected by the age of planting and the diversity of tree species? Would it be a resident species, or a visitor? Regular or irregular?

Part of the variation we found is probably due to different levels of familiarity with the birds of western Victoria. Our experts also had varying levels of experience conducting surveys and studying birds in revegetated habitats.

What does this mean for our reliance on expertise?

Expert knowledge can be the best – or only – source of information for decision making when we need to rely on forecasts, when knowledge needs to be applied in new circumstances, or when management is dealing with complex interacting factors.

So how can we improve the accuracy and reliability of expert predictions, given that we all have biases and gaps in our knowledge?

Carefully selecting experts based on relevant experience, using a structured protocol to extract information from and combining knowledge from multiple experts can help.

Whenever possible, we should take the opportunity to test expert predictions and identify potential biases. At the moment, this type of testing is rare.

Good nature management benefits from the wealth of knowledge of experts, but also depends on evidence from well-designed empirical studies.

Expert predictions are only as good as the data – and the experience – on which their judgments are based.

Read more: The limits of expert judgment: lessons from social science forecasting during the pandemic