Big data storage system for preventing data redundancy based on data classification and peer comparison

Fecha de publicación: 23/06/2023
Fuente: Wipo "BigData"
The invention relates to the technical field of big data storage, in particular to a big data storage system for preventing data redundancy based on data classification and peer comparison. The system comprises a correlation degree analysis module and a memory point marking module. According to the invention, correlation analysis is carried out on data in the same type through the correlation analysis module, redundant data and repeated data are determined, data storage repetition is avoided, and the storage space of the database is prevented from being influenced; according to the method, the corresponding memory rules are called to mark the redundant data and the repeated data, the data of which the correlation degree comparison is completed are identified according to the marking result, the later secondary comparison work is reduced, the data processing efficiency is improved, and meanwhile, after the redundant data are divided, the data in the redundant data can be integrated, so that the data processing efficiency is improved. Redundant data can be processed conveniently and quickly according to the storage requirement of a user, and simplification of redundant data processing modes is avoided.