| |
This is a framework that is batch processing with stream processing ability. It is built with a lot of the same principles of MapReduce, and it focuses mainly on the processing speed of its workload by providing full processing and in-memory computation.
Spark can be used standalone, or it can connect with Hadoop as a MapReduce alternative.
Spark will process all of the data in-memory and will only interact with the storage layer when it starts to load the data. At the end, it will provide the final results. Everything else is managed through memory.
Spark works faster when it comes to disk-related tasks because of their optimization that is able to be reached by looking at the entire task beforehand. It is able to do this by making DAGs, which show the operations that need to be done, the data to be worked on, and the relationship between them, which will give the processor better chance of coordinating work.
Stream gets it stream processing from Spark Streaming. Spark by itself is made for batch-oriented work. Spark has implemented a design known as micro-batches. This strategy was created to treat streams of data as if it were little batches of data that it can handle by using its batch engine’s semantics.
