1st & 2nd ITERATION
The harvester component covers the collection of metadata for multiple soil data and knowledge repositories. Within this component processes are implemented to harvest, harmonize, and register resources, and use unique identifiers to resources so they can be tracked and managed throughout their lifecycle in the system. The harvester currently focuses primarily on major aggregators (OpenAire, data.Europe.eu, CORDIS), while also directly harvesting a selection of primary sources including Mission Soil projects and key repositories at the edge of scope. This component is divided into two sub-components: the harvester and the harmonization of the ingested records. Harvested records are stored as ingested metadata while preserving the original source identifiers across the full SoilWise lifecycle. The harmonisation step validates, deduplicates, and maps these records to the harmonised SoilWise metadata model, including the conversion of provenance and lineage information into the common schema. During this transformation, source-specific encoding and specialised metadata elements that do not fit the harmonised model may be lost from the harmonised representation. The harmonised records are then enriched by the metadata augmentation component and indexed in the catalogue search layer for discovery and retrieval.
3rd ITERATION
The harvesting strategy will be refined in two directions: (i) for major aggregators, a decision is needed between providing a representative remote search(UI) of harvested soil records or broadening the harvest using keyword-based querying combined with a relevance selection algorithm; (ii) for primary sources, the scope will be expanded by identifying additional Mission Soil project hubs and edge-of-scope repositories, including the guidelines of managing the knowledge hubs after the projects end, while also establishing a clear approach for managing manual metadata ingestion in collaboration with the Metadata Authoring component.
1st & 2nd ITERATION
The Data & Knowledge Publication Support Tools cover the functionalities for supporting providers in FAIR publication. These are not an integral part of the SoilWise Catalogue but support its functionality (in)directly.
The following tools are included:
3rd ITERATION
Make tools usable for soil scientists and connected by:
1st & 2nd ITERATION
The component provides the storage and indexing infrastructure for metadata, documents and semantic knowledge. It includes a relational database for storing raw and augmented metadata, an RDF database storing the SoilWise knowledge graph and an index to provide performant access to the content. It offers interfaces for lexical and semantic querying, indexing, ranking and vector-based retrieval of full-text open access publication fragments.
First iteration: The storage component of the open source component pycsw was used, a data model for the storage of metadata, implemented in a relational database (PostGres). To facilitate the storage of the SoilWise knowledge graph, interlink it with metadata records and allow semantic querying, an RDF database (Virtuoso) was added to the storage infrastrucutre.
Second iteration: In the second iteration, the data model underlying the catalogue was refactored to better comply with international open metadata standards. In parallel, an Apache Solr index was integrated into the storage architecture to allow for faster and more flexible indexing and discovery of metadata and document content.
3rd ITERATION
Repository storage will be extended by (1) the addition of the indexing of full-text documents in Solr and (2) a vector-based extension that supports the RAG pipeline used by the Soil Companion and allows efficient access to the underlying metadata, full-text fragments (embedded text chunks), and document index.
1st & 2nd ITERATION
The Metadata Validation component covers the functionalities for assessing the completeness and compliance of metadata records. It includes mechanisms to validate metadata against relevant standards and schemas (INSPIRE) and provide validation results and compliance reports for the system administrators.
3rd ITERATION
Completeness validation:
INSPIRE compliance
1st & 2nd ITERATION
Metadata augmentation was not yet established as a dedicated component during the first cycles. Instead, augmentation-related activities emerged organically as part of different catalogue and data quality use cases.
Initial efforts focused on:
• Similarity detection and deduplication, including prototypes for identifying potentially duplicate records based on metadata similarity. [stopped]
• Entity extraction from metadata descriptions, exploring the use of Natural Language Processing for identification of relevant concepts contained in textual metadata fields.
• Keyword matching experiments, using simple fuzzy matching approaches to link metadata terms with controlled vocabularies and thesauri.
• Evaluation of augmentation opportunities, helping to identify where automated enrichment could provide added value for discovery, interoperability, and catalogue maintenance.
These activities provided valuable insights into augmentation requirements and demonstrated the potential of automated metadata enrichment.
3rd ITERATION
In the past cycles, augmentation was an ad-hoc approach for different usages. In this cycle, a first effort will be put into designing a generalised target schema for augmentation data in collaboration with the storage component. In parallel, an implementation and user visualisation (/interaction) functionality will be designed in collaboration with the catalogue component.
The augmentation of the records in the will be focused on a few functionalities:
1st & 2nd ITERATION
The Soil-health Knowledge Graph covers the functionalities for structuring and managing semantic knowledge for the soil health domain.
SoilVoc integrates vocabularies, ontologies and extracted domain knowledge into a unified semantic framework that supports metadata enrichment, interoperability and semantic search. It is applied in the Soil Companion, SoilWise Catalogue components (e.g. metadata augmentation) and tabular data annotator.
First iteration: the SoilWise soil health knowledge graph (SHKG) was implemented using a mixed method of AI supported, human in the loop generation and soil expert based validation, sourcing from leading knowledge sources on soil health. The resulting graph was deployed in a RDF database and connected to triplified metadata records to allow semantic querying.
Second iteration: SoilVoc was initiated as a soil vocabulary, primarily aimed as a semantic backbone to describe soil and soil health data. It formalizes semantics around soil observable properties and observation and measurement procedures. Concepts from SHKG have been reused to extend and improve SoilVoc. At the same time SHKG was further improved and finalised. Both SoilVoc and SHKG were used for integration into Soil Companion, the soil health chatbot developed in SoilWise.
3rd ITERATION
In the remainder of the project, we will focus on (1) extending and improving SoilVoc, (2) exploring the opportunities for linkage or integration of the Soil-health KG and SoilVoc and (3) improve their integration with functional components of SoilWise, particularly metadata augmentation, the Soil Companion and Catalogue search.
1st & 2nd ITERATION
The SoilWiseFinder component covers the SW Catalogue functionalities for indexing, discovering, exploring and accessing harvested and augmented metadata records. It provides search and filtering capabilities, displays detailed record information and spatial previews, and exposes standard APIs to support catalogue queries and integration with external systems.
First iteration: a prototype fully based on the open source software component pycsw. It allowed demonstrating the concept of a catalogue harvesting from relevant metadata repositories, reusing the connectivity and support of open standards.
Second iteration: substantial revision of the UI and underlying search engine and integration with the refactored data model. Implementation of the UI in React, offering better usability and adaptability. Apache Solr was integrated as a search engine for configurable indexation of metadata and unstructured content and faster and more flexible searches. Main search functions and thematic, spatial and temporal filters were integrated.
3rd ITERATION
The current version exposes the main functions, filters and metadata visualisations. In the last phase, we focus on fine tuning of the UI’s usability aspects, filtering options, ranking strategies, displaying augmented information (e.g. link checks, matched keywords / elements, translations, spatial augmentations). Besides, indexing of documents referred to by metadata and integration of discovery of resources based on full-text search will be (re)implemented
1st & 2nd ITERATION
First iteration: Conversational proof of concept. A single-model prototype using a small, curated collection demonstrated the potential of conversational access to soil knowledge. It also revealed limitations in source grounding, spatial context and adaptation to different users.
Second iteration: Retrieval-augmented chatbot. The Soil Companion was extended with retrieval-augmented generation, persistent document retrieval and access to the SoilWise catalogue and services. This connected answers to current project resources (soil knowledge and data), while highlighting the need for stronger semantic retrieval, evidence assessment and scientific grounding.
3rd ITERATION
Third iteration: Bounded multi-agent system. The Soil Companion is evolving into a bounded multi-agent system with dedicated workflows for quick questions, catalogue exploration, scientific research, knowledge briefings, and practical land-management support. It combines catalogue records and indexed publications with the controlled soil vocabulary, the soil health knowledge graph, geospatial information and registered data services.
1st & 2nd ITERATION
The Access Control component covers the functionalities for authentication and controlled access to system services and interfaces. It ensures secure access to functionalities such as Data & Knowledge Administration Console and the Soil Companion through Keycloak as a dedicated identity provider and makes integration with external authentication systems possible.
3rd ITERATION
Every SoilWise service should be accessible to European public sector users through their existing EU Login credentials, with no separate account required.
1st & 2nd ITERATION
The Dissemination & Guidelines component covers the activities and resources that support the publication, sharing and reuse of soil data and knowledge. It includes the organisation of FAIR data strategies, guidelines and documentation, as well as capacity building and community support for soil data exchange. The activities performed within this component consists of setting up a mechanism to make these resources easily accessible to a user, from the catalogue landing page. It also aims at ensuring the sustainability (DOI assignation) of these resources. Moreover, a user manual will be developed.
3rd ITERATION
To efficiently support SoilWise users and D&K providers in a context of an approaching project end, we propose to improve this component following 2 directions:
1st & 2nd ITERATION
The EUSO Integration component covers the functionalities related to the integration and migration of SoilWise services and components to the European Soil Observatory (EUSO) platform.
3rd ITERATION
Developments will ensure that the SoilWise Repository is packaged and documented so that JRC can deploy, operate and extend it independently within the EUSO platform. Because the EUSO hosting environment offers a limited amount of computing capacity, the repository is handed over in stages: a core set of components (harvesting, storage, search, metadata augmentation and link checking) is prepared first and sized to run within that capacity. Added-value components will follow wherever capacity allows. By project end, the core components run in the EUSO environment with validated interfaces, enabling continuity of soil knowledge infrastructure beyond the project lifetime.
1st & 2nd ITERATION
The System Usage & Monitoring component covers the functionalities for monitoring system performance (Infrastructure & Health Monitoring), usage (Usage Monitoring) and the status of available data and knowledge resources (Data & Knowledge Administration Dashboard). It provides dashboards and analytics tools to track repository access, scheduled tasks, and pipelines, and the overall state of services.
3rd ITERATION
Developments will focus on providing unified open-source observability stack where all SoilWise services expose their health, performance, and usage. Namely, Prometheus is envisioned for metrics, Grafana for overview dashboards, Loki for logs management and Superset for content analytics. Operators and JRC stakeholders gain real-time visibility into infrastructure, content completeness, and user engagement.