In the previous video, you learnt about the various sources of big data. as you would have understood by now, big data processing systems are capable of processing almost all the different types of data that exit. the upcoming video will help in answering what are the different types of data that exit? you will also understand how Google tackled the problem of handing its huge and diverse data, way before big data became as commonplace as it is today.
Hence, the key requirements for big data system are-
- The ability to store huge volumes of data
- The ability to process the huge volumes of stored data
- Flexibility and scalability to accommodate the growth in data
The solution proposed by Google to handle big data is as follows
- Google File System: To store data in a distributed manner across multiple interconnected computers.
- MapReduce: To run processes on data stored in a distributed manner across multiple interconnected computers. You also learnt the different types of data — structured and semi-structured. You got to know that structured and semi-structured data together constitute around 20% of all the data that is currently being generated. Note that these are estimates as per different reports. What about the significant proportion of the remaining data? Is it being ignored? Let’s find out in the next video.