METHOD AND APPARATUS FOR PROCESSING BIG DATA

Fecha de publicación: 19/07/2017
Fuente: Wipo "BigData"
A big-data processing method and apparatus are provided, and belong to the field of data processing. The method includes: obtaining multiple key-value pair sets, where each of the key-value pair sets includes at least one key-value pair, a key and a value in the key-value pair are two pieces of subdata in to-be-processed big data, there is a preset data relationship between the two pieces of subdata, and remainders of modulo operations on all key-value pairs in each key-value pair set are the same; calculating a sum of all values included in each key-value pair set to obtain a total value of the values included in each key-value pair set; obtaining a load status of each reduce processing module in a reduce processing module set; assigning a corresponding key-value pair set to each reduce processing module according to the total value of the values included in each key-value pair set and the load status of each reduce processing module; and processing the assigned key-value pair set by using each reduce processing module.