The first work to be carried out before launching any data-driven project is to identified all potential sources, to qualify them and to analyze the processes that will be required to leverage their value.
Data managers need to address critical questions:
How data can be collected (file, API, direct access to a database ...)
Structure of the data to be imported, syntax, format, culture ...
Ingestion processes required (batch, stream)
Frequency of updates required
Treatment to be applied to each source (cleaning, enrichment)
The Comfiz platform provides a framework to manage data sources and all the associated processes.