Big data acquisition method and system

Fecha de publicación: 09/02/2024
Fuente: Wipo "BigData"
The invention provides a big data collection method and system in the technical field of big data collection, and the method comprises the following steps: S10, collecting different types of big data through ETL, a real-time message channel, an API interface or a file interface; step S20, preprocessing the collected big data at least including data cleaning and data conversion; step S30, setting metadata of each piece of preprocessed big data; s40, grouping the big data based on the metadata, and setting grouping labels; step S50, storing the big data in a non-relational database in real time; and step S60, backing up the big data stored in the non-relational database to a relational database. The method has the advantage that the big data acquisition quality is greatly improved.