Comparative Study, Applications and Challenges of Big Data Analysis Technologies

Receiving and sharing information is easier and cheaper than before, enabling organizations to handle large volumes of data at a high speed and variety in the name of big data. Big data provides many opportunities when problems are resolved correctly. Data processing technologies are not suitable for dealing with large quantities of generated data. Suggested frameworks for big data applications help to store, analyze and process data. In this study, using library and documentary studies, a number of important big data frameworks (Hadoop, Flink, Storm, Spark and Samza) have been studied and compared comparatively. The studied framework of big data is generally classified into two categories: (1) batch mode; and (2) stream mode. The Hadoop framework processes data in batch mode, while other frameworks allow stream or real time processing. Ultimately, the most important applications and challenges of using big data technology have been described. The most important applications for big data analysis are healthcare applications, advisory systems, smart cities and social networks analysis. Also, the most challenges of big data application, including confidentiality in storage systems, software deficiencies and the limitation of existing hardware and equipment, the need for large initial investment and the lack of technical skills and expert workforce.