GFS

File systems that already existed during GFS - GPFS/Lustre/AFS Why GFS was built instead of using existing ones? GPFS and Lustre are paid systems(could be the main reason) Specialized workflows Most common solutions to problems that come up: Batching Pipelining Separation of control and data flow Consistency Weakly consistent Mitigates consistency maintenance to its applications Takeaway: Satisfying the need is the main goal and to have that industries try to create simple system designs. ...

MapReduce

TIL: POSIX Reason for MapReduce(Why use a distributed system?) Lots of data(1 PetaByte), machines had 160 GB storage so can’t process in one machine I/O speed is very low(performance) Fault tolerance(Tolerate machine and disk failures) Application programmers don’t need to work on systems and making sure their job is running in a distributed fashion Workflow Input Map Reduce Result key, value K1 <v1, v1`, v1`` > K1 R(v1, v1`, …) K2, v2 K3, v3 Examples Word Count Sort Reverse Links(Used for page ranking) Input - key: URL, value: HTML Map - (<target1, src1>, <target2, src2>,….) Reduce - target1, <src1, src1`,….> Result - rank Why MapReduce was developed as a library and not as a service? Because loading a user defined function into a already existing running process was not possible in case of C++. It is easier to compile the code with the library than dynamically loading the UDF(user defined function). ...