Data stream processing system and method in big data environment

Fuente: Wipo "BigData"
The invention relates to the technical field of data processing, in particular to a data stream processing system and method in a big data environment. The method comprises the following steps: collecting data in real time to obtain an original multi-source heterogeneous data stream; introducing a data adaptation mechanism to carry out standardization processing on the original multi-source heterogeneous data stream to obtain a standardized data stream; performing intelligent scheduling on the standardized data stream to obtain a preliminary scheduling strategy; based on feedback data of a data stream processing system, parallel real-time state feedback and edge real-time state feedback are obtained, a preliminary scheduling strategy is optimized, and based on the optimized scheduling strategy, normalized data streams are divided into primary normalized data streams and secondary normalized data streams. And processing the primary normalized data stream and the secondary normalized data stream. The problems that when a traditional data processing system processes data streams in a big data environment, multi-source heterogeneous data cannot be accurately adapted, and distribution and scheduling are unreasonable during processing are solved.