When aggregating data, we make available the underlying data used to create the aggregation, together with the weights and calculation methods that were applied. We do not store all the underlying values used to create the aggregated value: aggregations can cover tens of thousands of Cycles, and that would be too much data to store.
What is published is everything needed to reconstruct the aggregated value: the list of Cycles that were used, the weights applied, and the calculation methods used. This page describes how to use the underlying data to go from an aggregated value back to the underlying values it came from.
| Where | What it tells you |
|---|---|
description on the aggregated Cycle | The weight given to each sub-system (e.g., Conventional, irrigated: 0.23%), or to each country for a World aggregation, and the average end date of the Cycles. |
description on each aggregated Term | The weight given to each contributing Cycle or sub-aggregation for that Term specifically. |
observations on a blank node | How many data points (Cycles) the Term's value was averaged over. |
min, max, sd and the percentiles | The spread of the underlying values for that Term. |
aggregatedCycles | Every underlying Cycle, by @id. |
aggregatedSources | Every Source the underlying Cycles came from. |
Both lists are on the aggregated node in every format — JSON-LD, the API, and the compacted CSV, where they arrive as aggregatedCycles.@id columns. They are properties of the aggregation, so they only appear on its row: the underlying Cycles in a download carry no aggregatedCycles of their own.
Downloads come in two layouts — see Downloading Data. Expanded puts term names in rows under numbered indices (cycle.inputs.0.term.name, cycle.inputs.0.value); compacted puts the term id in the column header (cycle.inputs.ureaKgN.value), so each row is one node and each Term is one column. Everything on this page assumes compacted, because it is the layout you can sort, filter and pivot.
On any aggregation page, the download dialog offers the aggregated dataset and the underlying datasets. Take both, as CSV (Compacted). Choosing the underlying datasets resolves aggregatedCycles, aggregatedSources and aggregatedSites for you and downloads exactly those nodes; including the aggregated dataset gives you the published value to compare your reconstruction against.
What arrives is a .zip holding one file per node type and state — Cycle-recalculated-compacted.csv, Cycle-original-compacted.csv, Site-recalculated-compacted.csv, and so on, plus a README.md describing the formats. The aggregation runs on the recalculated Cycles, so that is the file to work from.
Four things about the compacted layout matter for everything below:
practices.organic.value, not cycle.practices.organic.value, and the Cycle id is @id. A missing value is written -, not left blank;operation, method and methodClassification — leave several columns describing one aggregated node, so they have to be read together: inputs.diesel.value and inputs.diesel+operation[pesticideApplicationWithSprayer].value are one node between them;inputs, depthLower/depthUpper and startDate/endDate — mark genuinely separate nodes, so a measurement is aggregated per depth interval and management per date range, and those columns must not be combined;aggregated = True. It is the published result, not one of the inputs to it — counting it as an underlying Cycle would skew every average.The same data is on the Data API, which needs no account for public data. Use it when you want to script the comparison rather than work in a spreadsheet: read the aggregation, take the Cycle ids out of aggregatedCycles, and fetch each of those Cycles in turn.
# the ids of the Cycles behind an aggregation
curl https://api.hestia.earth/cycles/<aggregation-id> | jq '.aggregatedCycles[]."@id"'
# then each underlying Cycle, in the same compacted form the download uses
curl https://api.hestia.earth/cycles/<cycle-id>
The underlying data does not store which Cycles belong to which sub-aggregation, but it can be inferred from the practices each Cycle reports, following the process below.
For crop products, Cycles are grouped on two axes — see Crop Aggregation:
standardsLabels practice with a value above 0 that is an organic label. That is the generic organic practice, or any of the certification labels: euOrganicProductsLabel, canadaOrganic, japaneseAgriculturalOrganicStandardJas, krav, bioSuisse, naturlandEV, and around fifty more — the full list is every term whose isOrganic value is organic in the standardsLabels lookup. Otherwise the Cycle is conventional.waterRegime practice with a value above 0 that is an irrigation practice: irrigatedTypeUnspecified, irrigatedFurrowIrrigation, irrigatedLocalizedIrrigation, irrigatedDripIrrigation, irrigatedSprinklerIrrigation, irrigatedCenterPivotIrrigation, irrigatedLateralMoveIrrigation, irrigatedSubIrrigation, irrigatedSurfaceIrrigationDrainageRegimeUnspecified, irrigatedSurfaceIrrigationContinuouslyFlooded, irrigatedSurfaceIrrigationSingleDrainagePeriod and irrigatedSurfaceIrrigationMultipleDrainagePeriods. Otherwise the Cycle is rainfed.The weight each of those sub-systems carries is in the aggregated Cycle's description, as Conventional, irrigated: 0.23%.
Plantation crops are grouped on a third axis, by phase. Which phase a Cycle is in comes from two boolean practices, tested in this order: productivePhasePermanentCrops = TRUE is productive; otherwise preparationPhasePermanentCrops = TRUE is preparation; otherwise the Cycle is non-productive.
A phase is not weighted by how many Cycles are in it. It is weighted by the share of the plantation's life it occupies, which comes from two lookups on the primary product — Plantation_lifespan and Plantation_non-productive_lifespan in the crop lookup. Preparation is one year of the lifespan, productive is the lifespan minus its non-productive years, and non-productive is what remains. Oil palm has a 9125-day lifespan of which 1095 days are non-productive, so its phases weigh 88%, 4% and 8%.
These weights are published, on the sub-aggregation Cycle rather than on the country one. An aggregation of a plantation crop produces a Cycle per sub-system — Oil palm, fruit - Indonesia - Conventional, Non Irrigated - 2010-2025 — and it is that Cycle whose description reads This aggregation uses the following weights: Productive phase: 88%, Non-productive phase: 8%, Preparation phase: 4%, with each of its Terms carrying the shares that applied to that Term. The country-level Cycle's description carries the sub-system weights instead. A group's full weight is the two multiplied: its sub-system's share of the country, times its phase's share of the plantation's life.
The script below derives the phase shares from the same lookups the aggregation reads, so you do not need to open the sub-aggregation to rebuild a plantation value.
HESTIA publishes a script that does the split for you. It is in the public hestia-aggregation-engine repository, at scripts/split_aggregation_download.py — clone the repository or download that one file. It needs Python 3 and nothing else.
Give it the .zip you downloaded and it writes one CSV per group, each keeping the original columns:
python scripts/split_aggregation_download.py \
--filepath "Oil palm, fruit - Indonesia - 2010-2026.zip"
It finds the recalculated Cycle file inside the archive, sets the aggregated node aside, and groups what is left.
reading Cycle-recalculated-compacted.csv
conventional-rainfed-productive: 57 Cycles
conventional-rainfed-non-productive: 4 Cycles
conventional-rainfed-preparation: 2 Cycles
63 Cycles in 3 groups
Oil palm is a plantation crop, so the groups are split by phase as well as by sub-system. Nothing has to be configured for that: the script checks the Cycle's primary product against the isPlantation lookup, and reads the practices behind every axis from the glossary at run time.
Pass --term with a Term id, and --completeness with the Term's completeness area, and the script reports the values behind that Term instead of writing the CSVs. Omit --completeness and it lists the areas the file contains.
python scripts/split_aggregation_download.py \
--filepath "African aubergine, fruit.zip" \
--term ureaKgN --completeness fertiliser
inputs.ureaKgN
35 Cycles, 22 reporting a value
group cycles reported mean
conventional-irrigated 22 14 137.8
conventional-rainfed 13 8 47.81
cycle.@id group value
puelaqpxbhak conventional-irrigated 460
o3-lfyghs2jb conventional-irrigated 414
...
mean of the 22 reported values: 105.1
complete for `fertiliser`: 35 Cycles - the Term's `observations`
of those, 22 report a value
the other 13 contribute a zero-filled `0`
incomplete for `fertiliser`: 0 Cycles, skipped
mean over the 35 complete Cycles: 66.07
published by the aggregation: 29.56 over 35 observations
group weight complete mean
conventional-irrigated 0.23% 22 87.72
conventional-rainfed 99.77% 13 29.43
rebuilt from the download 29.56
That matches the published value.
Reading down that output is the whole debugging process, and each step explains part of the distance between the values you can see and the value that was published:
105.1. That is the figure most people start from, and it is not what was published.fertiliser, so the 13 reporting nothing each contribute a 0, and the mean over all 35 is 66.07. This is what the aggregation publishes as observations: 35 — observations counts the Cycles the value was averaged over, not the Cycles that reported anything.0.23% of the weight, because the weighting comes from the country's irrigated area rather than from how many Cycles happen to sit in each group. Applying the weights from the aggregation's description gives 29.56 — the published value.So the published figure is the weighted mean of the zero-filled group means, and the download holds everything needed to rebuild it. For a plantation crop each group is a sub-system and a phase, so the script splits the sub-system weight between the phases in the shares described above, and prints those shares alongside the reconstruction.
Site measurements and management live in their own file in the archive, so pass --node-type Site to report on them. Each Site is placed in the group of the Cycle that uses it, since a Site reports no organic or irrigation practice of its own; where several Cycles share a Site, it takes the group of the first one. Measurements have no completeness area, so a Site reporting nothing is skipped rather than zero-filled, and the reconstruction is the weighted mean of the Sites that do report.
python scripts/split_aggregation_download.py \
--filepath "African aubergine, fruit.zip" \
--node-type Site --term clayContent
Dry matter. The primary product of a crop aggregation is rescaled to the HESTIA default dryMatter before it is averaged. The Cycle's own dryMatter property is in the download and the HESTIA default is not, so the script reads the default from the Term itself at https://api.hestia.earth/terms/<id> and applies the ratio to every product that carries a dryMatter property. A product without one is not rescaled, which is the common case.
Emissions follow the completeness of the Inputs that caused them. A background emission is written emissions.<term>+inputs[<input>], and the Input is carried onto the aggregated node — so emissions.ch4ToAirInputsProductionFossil+inputs[ureaKgN] is complete when fertiliser is. The script works that out for you, so --completeness is not needed for emissions.
Background emissions were not averaged at all. They are recalculated from the aggregated Inputs once the aggregated Cycle exists, rather than built from the underlying Cycles, so there is nothing in the download to rebuild them from and the script says so instead of showing a total that will not match. A node with no observations is one of these: present means averaged, absent means recalculated.
Compare like with like. A data release is a fixed snapshot of the underlying data and the aggregations built from it — v1.0 on 2025-05-01, v1.1 on 2026-03-10, v1.2 on 2026-04-30. Take the aggregation and its underlying data from the same release and they line up by construction. Comparing an aggregation against a download taken later will not reproduce the arithmetic, because the Cycles behind it have moved on.
The most common reason to open an aggregation up is that one value looks implausible, and the question is which Cycle put it there — and whether that Cycle, or its Site, is right.
1. Establish what "implausible" means. The aggregated Term carries min, max, sd and the percentiles of the values behind it, so the spread is published even though the values are not. A mean sitting near max with a high sd and few observations is one extreme Cycle pulling the average; a mean that is simply lower than expected across a large observations count is more likely zero-filling, which --completeness above makes visible.
2. Find the Cycle in the download. In the compacted file each row is one Cycle and the Term is one column, so sorting that column puts the extreme values at the top. The Cycle is cycle.@id on the row.
3. Open the Cycle and its Site. On the website, a Cycle is at https://www.hestia.earth/cycle/<cycle.@id> and a Site at https://www.hestia.earth/site/<site.@id>; take the Site id from the cycle.site.@id column, and the Source id from cycle.defaultSource.@id.
The same nodes are on the API, if you would rather read the data than the page:
curl https://api.hestia.earth/cycles/<cycle.@id>
curl https://api.hestia.earth/sites/<site.@id>
curl https://api.hestia.earth/sources/<source.@id>
Check the Site whenever the value that looks wrong is one the Site supplies — a measurement, a land cover — rather than one the Cycle reports; the aggregation lists those under its Site. Several Cycles can share one Site, so a single bad Site can move more than one row.
4. Check which group it landed in. An outlier is sometimes correct but grouped wrong. A Cycle with no waterRegime practice counts as rainfed, and a Cycle with no organic standardsLabels practice counts as conventional: the grouping is decided only by what the Cycle reports, so an unreported practice moves the Cycle to the other group rather than excluding it.