Status: In Progress
Epic Name:RDDS Dynamic Harvester Extension
Since each participant has a slightly different metadata mapping, it might make sense to create a new harvest endpoint that supports dynamically describing the field mappings.
It could also support features such as clearing datasets as an asynchronous task. In the current harvest extension, the clearing is done in a single thread which often crashes the process when there are many datasets to clear.
Other benefits of creating our own harvest extension:
- The official rdds-ckanext-harvest doesn't have versions (we just install directly from master and as such there is a security and compatibility risk).
- We can separate harvester configs from the CKAN Datasets (currently harvest endpoints are listed as datasets which mess up the numbers).
- We will have better control over the settings and web interface panel.
- We can better handle support for other formats like pure which may not fit so well into the current harvest format.