Part 1: Outlier Detection
Why Detect Outliers?
An outlier is an observation that differs substantially from the expected pattern. In phenological data, outliers may indicate:
- Recording errors that should be excluded from
analysis
- A typo: DOY 250 instead of DOY 150 (roughly 3 months difference)
- Wrong year entered: observation assigned to incorrect season
- Unusual weather events worth investigating
- Very warm winter causing early flowering
- Late frost delaying spring phenology
- Second flowering or other abnormal phenological
events
- Plants flowering again in autumn/winter after spring flowering
- Increasingly observed with climate change
The challenge: You can’t simply delete all outliers. Some are errors, but others are scientifically valuable observations of unusual events.
Statistical Methods for Outlier Detection
The pep_flag_outliers() function provides four
statistical methods for identifying potential outliers. Each method has
different assumptions and strengths:
Method 1: 30-Day Rule (Simple Threshold)
The simplest approach: flag any observation more than 30 days from the median for that species/phase combination.
# Flag outliers using the default 30-day rule
outliers <- pep_flag_outliers(
pep = flowering,
method = "30day",
by = c("species", "phase_id")
)
print(outliers)
#> Phenological Outlier Detection Results
#> =============================================
#> Method: 30day
#> Threshold: 30 days
#> Grouping: species, phase_id
#>
#> Total observations: 1,310,094
#> Outliers flagged: 26573 (2.03%)
#> - Early outliers: 15011
#> - Late outliers: 11562
#>
#> Deviation summary for outliers:
#> Min: -167.0 days
#> Max: 223.0 days
#> Mean |deviation|: 38.4 daysHow it works:
Distribution of flowering dates
← Early Late →
----|----[=======|=======]----|-----
^ ^ ^
-30 days median +30 days
Anything outside the brackets is flagged as an outlierUnderstanding the Output
The function adds new columns to your original data:
| Column | What It Contains | How to Use It |
|---|---|---|
is_outlier |
TRUE/FALSE flag | Filter data: data[is_outlier == FALSE] |
deviation |
Days from expected | Prioritize investigation: larger = more extreme |
expected_doy |
Reference DOY (median) | Understand what “normal” looks like |
Method 2: MAD (Median Absolute Deviation) - Recommended
The MAD method is robust to the presence of multiple outliers. This is important because if your data contains many outliers, they can distort mean-based methods.
# MAD method: flag if > 3 MAD from median (robust)
outliers_mad <- pep_flag_outliers(flowering, method = "mad", threshold = 3)Why MAD is recommended:
| Scenario | Mean-based method | MAD method |
|---|---|---|
| 1-2 outliers | Works reasonably | Works well |
| Many outliers | Outliers inflate SD, may miss more | Stays robust |
| Asymmetric data | Assumes symmetry | Handles asymmetry |
Method 3: IQR (Interquartile Range) - Also Robust
The IQR method uses quartiles and is familiar from boxplot “whiskers”:
# IQR method: flag if outside 1.5 * IQR (standard boxplot rule)
outliers_iqr <- pep_flag_outliers(flowering, method = "iqr", threshold = 1.5)How it works: - Q1 = 25th percentile, Q3 = 75th percentile - IQR = Q3 - Q1 - Outlier if: value < Q1 - 1.5×IQR OR value > Q3 + 1.5×IQR
Method 4: Z-Score - Sensitive but Less Robust
The z-score method assumes normally distributed data:
# Z-score method: flag if |z| > 3 (assumes normal distribution)
outliers_zscore <- pep_flag_outliers(flowering, method = "zscore", threshold = 3)Caution: The z-score method is sensitive to existing outliers. If your data has many outliers, they will inflate the standard deviation, making the z-score threshold less effective at detecting them.
Choosing a Method
| Method | Best For | Threshold Meaning |
|---|---|---|
30day |
Simple, interpretable threshold | Absolute days from median |
mad |
Most situations (recommended) | Number of MADs from median |
iqr |
Familiar boxplot-style detection | IQR multiplier |
zscore |
Clean data, normal distribution | Standard deviations |
Summary Statistics
summary(outliers)
#> Phenological Outlier Summary
#> =============================================
#>
#> Total observations: 1,310,094
#> Total outliers: 26573 (2.03%)
#>
#> Groups with most outliers:
#> species phase_id n_obs n_outliers pct_outliers mean_dev
#> <fctr> <int> <int> <int> <num> <num>
#> 1: Solanum tuberosum 60 114245 6420 5.62 12.0
#> 2: Prunus domestica 60 114641 2958 2.58 10.5
#> 3: Prunus domestica 65 77393 2146 2.77 10.8
#> 4: Malus domestica 60 205403 1957 0.95 9.1
#> 5: Malus domestica 65 175706 1770 1.01 9.1
#> 6: Prunus persica 60 48074 1515 3.15 11.0
#> 7: Prunus cerasus 60 136588 1227 0.90 9.0
#> 8: Prunus cerasus 65 112597 1042 0.93 9.0
#> 9: Prunus armeniaca 60 22692 943 4.16 12.1
#> 10: Pyrus communis 60 50335 940 1.87 9.5
#>
#> Deviation distribution (all data):
#> 1% 5% 25% 50% 75% 95% 99%
#> -31 -21 -8 0 8 20 30Comparing Outlier Rates Across Species
Different species may have different outlier rates due to observation difficulty or data quality:
# Check outliers for grapevine
vine_flowering <- pep[species == "Vitis vinifera" & phase_id %in% c(60, 65)]
if (nrow(vine_flowering) > 0) {
outliers_vine <- pep_flag_outliers(
pep = vine_flowering,
method = "30day",
by = c("species", "phase_id")
)
cat("Outlier comparison:\n")
cat("All flowering species: ", round(100 * mean(outliers$is_outlier), 2), "%\n")
cat("Grapevine only: ", round(100 * mean(outliers_vine$is_outlier), 2), "%\n")
} else {
cat("No grapevine flowering data available for comparison.\n")
}
#> Outlier comparison:
#> All flowering species: 2.03 %
#> Grapevine only: 3.68 %Visualizing Outliers
Visual inspection is essential for understanding outlier patterns.
The pep_plot_outliers() function provides four
visualization types:
1. Overview Plot
Shows the big picture: how many outliers, which species/phases are affected:
# Overview of outlier patterns
pep_plot_outliers(outliers, type = "overview")
#> Warning: Removed 16 rows containing non-finite outside the scale range
#> (`stat_bin()`).
What to look for: - Which species have the most outliers? (Might indicate data quality issues) - Are outliers concentrated in certain phases? (Might indicate definition problems) - What’s the overall outlier rate? (Typical: 1-5% for well-curated data)
2. Seasonal Distribution
When in the year do outliers occur?
Interpretation guide:
| Outlier Timing | Likely Explanation |
|---|---|
| Very early (DOY < 50) | Possible data errors (flowering in January/February unlikely for most species) |
| Somewhat early | Warm winters or Mediterranean locations |
| Somewhat late | Cool springs or high-altitude stations |
| Very late (DOY > 250) | Possible second flowering events - investigate! |
3. Detailed Context
See outliers alongside normal observations:
# See outliers in context of all observations
pep_plot_outliers(outliers, type = "detail", n_top = 15)
This plot shows: - Full distribution of observations
(gray) - Flagged outliers (highlighted) - The n_top
parameter controls how many species/phases to show
4. Geographic Distribution
Where are outliers located?
What to look for: - Clustered outliers in one region? Might indicate local data quality issues - Outliers at network edges? Might be at environmental limits - Random scatter? Suggests individual observation errors
