>Contributing Data>The Upload Pipeline

The Upload Pipeline

All uploads pass through HESTIA's upload pipeline, where a series of checks are performed to ensure the data are correctly formatted.

The Refresh button at the bottom of the page can be used to reload the page so you can see the latest stage of your upload.

Refresh your page

On the right-hand navigator, as well as the upload pipeline, there is also a summary of file issues. Errors will appear in red, and must be fixed before your file can be uploaded. Warnings will appear in orange; they are not required to be fixed, but it is recommended that you double check your data to ensure that the information provided is correct. Both have descriptions that explain why the issue has occurred.

File issues

Clicking on an error or warning will take you to its location in File Issues. This makes it easier to see where in your upload something needs to be fixed.

Below is an explanation of each stage of the upload pipeline, with some examples of common issues that can occur. If you are unable to fix issues that appear in the pipeline, you can email us at community@hestia.earth for assistance.

Data Compatibility

1
Convert to CSV format
This step checks if your file is able to be converted into CSV format. You are allowed to upload files in CSV, JSON, and Excel. We recommend using Excel as this allows you to preserve the original formatting of your file.
2
Convert to HESTIA format
This step checks that your file is able to be converted into the HESTIA format. Common issues here are misspelled or duplicated column headers.
3
Check Existing Nodes
This step checks whether the terms used in your upload are in HESTIA's glossary. Common issues are misspelled terms or outdated ones that are no longer in the updated version of the glossary. If a term has been recently renamed, the error message will suggest the new term you should use.
4
Add bibliography
This step is where HESTIA uses the provided DOI to fill in the bibliography of your source. A common issue is if HESTIA cannot find the DOI in Mendeley, then you will have to manually add the bibliography information in the Source section of your upload.
5
Add Metadata
This step adds some fields to your data, e.g., default values, auto-generated fields like name, site area based on the boundary provided. It also creates Impact Assessments for the Cycle products, where none have been uploaded.

Data Validation

1
Validate Schema
This step checks whether your upload is consistent with HESTIA's schema. A common issue here is missing required files, e.g., if you've added the sd of a value but not specified the statsDefinition.
2
Validate Data

This step checks the data itself against a range of validators on the HESTIA platform. Some could be fairly simple, e.g., you've specified a negative number as a value for an Input, and the value must be positive. Some can be more complex, e.g., ensuring the sum of tillage practices adds up to 100.

At this stage you can also see warnings for your dataset. Common warnings are that you haven't specified a tillage practice or the amount of residue produced for a cropland site, or that a value, e.g., Dry matter, is substantially outside our default value.

Submission & Indexing

1
Submission
Once your file has been submitted, if it is a public upload it will be reviewed by a member of our team. They may leave changes in the Comment & Tasks section, which you will need to make before your upload can be indexed.
2
Indexing
When a member of our team is happy with your upload, they will index it so that it can appear on the platform. If it is a private upload, the indexing process will happen automatically and you will not need to wait for review.

Deleting Indexed Data

To delete data after it has been indexed on the platform, email community@hestia.earth with the identifiers or URLs of the nodes to be removed.

Help us improve. Send us a message