Merge branch 'main' into BigData

CONTRIBUTING.md

@@ -0,0 +1 @@
+courses/CONTRIBUTING.md

NOTICE

@@ -0,0 +1,7 @@
+Copyright 2020 LinkedIn Corporation
+All Rights Reserved.
+Licensed under the BSD 2-Clause License (the "License").
+See LICENSE in the project root for license information.
+
+This product includes:
+* N/A

README.md

@@ -0,0 +1 @@
+courses/index.md

courses/CONTRIBUTING.md

@@ -0,0 +1,5 @@
+We realise that the initial content we created is just a starting point and our hope is that the community can help in the journey refining and extending the contents.
+
+As a contributor, you represent that the content you submit is not plagiarised. By submitting the content, you (and, if applicable, your employer) are licensing the submitted content to LinkedIn and the open source community subject to the BSD 2-Clause license.
+
+We suggest to open an issue first and seek advice for your changes before submitting a pull request.

courses/big_data/intro.md

@@ -1,38 +1,35 @@
-# School of SRE: Big Data
+# Big Data
 
-## Pre - Reads
+## Prerequisites
 
 - Basics of Linux File systems.
 - Basic understanding of System Design.
 
-## Target Audience
-
-The concept of Big Data has been around for years; most organizations now understand that if they capture all the data that streams into their businesses, they can apply analytics and get significant value from it. 
-This training material covers the basics of Big Data(using Hadoop) for beginners, who would like to quickly get started and get their hands dirty in this domain.
-
-## What to expect from this training 
+## What to expect from this course
 
 This course covers the basics of Big Data and how it has evolved to become what it is today. We will take a look at a few realistic scenarios where Big Data would be a perfect fit. An interesting assignment on designing a Big Data system is followed by understanding the architecture of Hadoop and the tooling around it. 
 
-## What is not covered under this training
+## What is not covered under this course
 
 Writing programs to draw analytics from data.
 
-## TOC:
+## Course Content
 
-1. Overview of Big Data
-2. Usage of Big Data techniques
-3. Evolution of Hadoop
-4. Architecture of hadoop
+### Table of Contents
+
+1. [Overview of Big Data](https://linkedin.github.io/school-of-sre/big_data/intro/#overview-of-big-data)
+2. [Usage of Big Data techniques](https://linkedin.github.io/school-of-sre/big_data/intro/#usage-of-big-data-techniques)
+3. [Evolution of Hadoop](https://linkedin.github.io/school-of-sre/big_data/evolution/)
+4. [Architecture of hadoop](https://linkedin.github.io/school-of-sre/big_data/evolution/#architecture-of-hadoop)
     1. HDFS
     2. Yarn
-5. MapReduce framework
-6. Other tooling around hadoop
+5. [MapReduce framework](https://linkedin.github.io/school-of-sre/big_data/evolution/#mapreduce-framework)
+6. [Other tooling around hadoop](https://linkedin.github.io/school-of-sre/big_data/evolution/#other-tooling-around-hadoop)
     1. Hive
     2. Pig
     3. Spark
     4. Presto
-7. Data Serialisation and storage
+7. [Data Serialisation and storage](https://linkedin.github.io/school-of-sre/big_data/evolution/#data-serialisation-and-storage)
 
 
 # Overview of Big Data
@@ -50,7 +47,7 @@ Writing programs to draw analytics from data.
 4. Examples of Big Data generation include stock exchanges, social media sites, jet engines, etc.
 
 
-# Usage of Big Data techniques
+# Usage of Big Data Techniques
 
 1. Take the example of the traffic lights problem.
     1. There are more than 300,000 traffic lights in the US as of 2018.
@@ -59,4 +56,5 @@ Writing programs to draw analytics from data.
     4. How would you go about processing that and telling me how many of the signals were “green” at 10:45 am on a particular day?
 2. Consider the next example on Unified Payments Interface (UPI) transactions:
     1. We had about 1.15 billion UPI transactions in the month of October, 2019 in India.
-    12. If we try to extrapolate this data to about a year and try to find out some common payments that were happening through a particular UPI ID, how do you suggest we go about that?
+    12. If we try to extrapolate this data to about a year and try to find out some common payments that were happening through a particular UPI ID, how do you suggest we go about that?
+    

courses/databases_nosql/further_reading.md

@@ -0,0 +1,25 @@
+# Further reading:
+
+NoSQL:
+
+https://hostingdata.co.uk/nosql-database/
+
+https://www.mongodb.com/nosql-explained
+
+https://www.mongodb.com/nosql-explained/nosql-vs-sql
+
+Cap Theorem
+
+http://www.julianbrowne.com/article/brewers-cap-theorem
+
+Scalability
+
+http://www.slideshare.net/jboner/scalability-availability-stability-patterns
+
+Eventual Consistency
+
+https://www.allthingsdistributed.com/2008/12/eventually_consistent.html
+
+https://www.toptal.com/big-data/consistent-hashing
+
+https://web.stanford.edu/class/cs244/papers/chord_TON_2003.pdf

courses/databases_nosql/intro.md

@@ -0,0 +1,222 @@
+# DATABASES - NoSQL
+
+
+## Target Audience
+
+This Module is meant to be an introduction to NoSQL Databases. We will be touching upon the key concepts and trade offs in a distributed data system.
+
+
+## What to expect from this training
+
+At the end of training, you will have an understanding of what a NoSQL database is, what kind of advantages or disadvantages it has over traditional RDBMS, learn about different types of NoSQL databases and understand some of the underlying concepts & trade offs w.r.t to NoSQL.
+
+
+## What is not covered under this training
+
+We will not be deep diving into any specific NoSQL Database. 
+
+
+## Agenda
+
+
+
+*   Introduction to NoSQL
+*   CAP Theorem
+*   Data versioning
+*   Partitioning
+*   Hashing
+*   Quorum
+
+
+## Introduction
+
+When people use the term “NoSQL database”, they typically use it to refer to any non-relational database. Some say the term “NoSQL” stands for “non SQL” while others say it stands for “not only SQL.” Either way, most agree that NoSQL databases are databases that store data in a format other than relational tables.
+
+A common misconception is that NoSQL databases or non-relational databases don’t store relationship data well. NoSQL databases can store relationship data—they just store it differently than relational databases do. In fact, when compared with SQL databases, many find modeling relationship data in NoSQL databases to be _easier_, because related data doesn’t have to be split between tables.
+
+Such databases have existed since the late 1960s, but the name "NoSQL" was only coined in the early 21st century. NASA used a NoSQL database to track inventory for the Apollo mission. NoSQL databases emerged in the late 2000s as the cost of storage dramatically decreased. Gone were the days of needing to create a complex, difficult-to-manage data model simply for the purposes of reducing data duplication. Developers (rather than storage) were becoming the primary cost of software development, so NoSQL databases optimized for developer productivity. With the rise of Agile development methodology, NoSQL databases were developed with a focus on scaling, fast performance and at the same time allowed for frequent application changes and made programming easier.
+
+
+### Types of NoSQL databases:
+
+Over time due to the way these NoSQL databases were developed to suit requirements at different companies, we ended up with quite a few types of them. However, they can be broadly classified into 4 types. Some of the databases can overlap between different types. They are
+
+
+
+1. **Document databases: **They store data in documents similar to [JSON](https://www.json.org/json-en.html) (JavaScript Object Notation) objects. Each document contains pairs of fields and values. The values can typically be a variety of types including things like strings, numbers, booleans, arrays, or objects, and their structures typically align with objects developers are working with in code. The advantages include intuitive data model & flexible schemas. Because of their variety of field value types and powerful query languages, document databases are great for a wide variety of use cases and can be used as a general purpose database. They can horizontally scale-out to accomodate large data volumes. Ex: MongoDB, Couchbase
+2. **Key-Value databases:** These are a simpler type of databases where each item contains keys and values. A value can typically only be retrieved by referencing its value, so learning how to query for a specific key-value pair is typically simple. Key-value databases are great for use cases where you need to store large amounts of data but you don’t need to perform complex queries to retrieve it. Common use cases include storing user preferences or caching. Ex: [Redis](https://redis.io/), [DynamoDB](https://aws.amazon.com/dynamodb/), [Voldemort](https://www.project-voldemort.com/voldemort/)/[Venice](https://engineering.linkedin.com/blog/2017/04/building-venice--a-production-software-case-study) (Linkedin), 
+3. **Wide-Column stores:** They store data in tables, rows, and dynamic columns. Wide-column stores provide a lot of flexibility over relational databases because each row is not required to have the same columns. Many consider wide-column stores to be two-dimensional key-value databases. Wide-column stores are great for when you need to store large amounts of data and you can predict what your query patterns will be. Wide-column stores are commonly used for storing Internet of Things data and user profile data. [Cassandra](https://cassandra.apache.org/) and [HBase](https://hbase.apache.org/) are two of the most popular wide-column stores.
+4. Graph Databases: These databases store data in nodes and edges. Nodes typically store information about people, places, and things while edges store information about the relationships between the nodes. The underlying storage mechanism of graph databases can vary. Some depend on a relational engine and “store” the graph data in a table (although a table is a logical element, therefore this approach imposes another level of abstraction between the graph database, the graph database management system and the physical devices where the data is actually stored). Others use a key-value store or document-oriented database for storage, making them inherently NoSQL structures. Graph databases excel in use cases where you need to traverse relationships to look for patterns such as social networks, fraud detection, and recommendation engines. Ex: [Neo4j](https://neo4j.com/) 
+
+
+### **Comparison** 
+
+
+<table>
+  <tr>
+   <td>
+   </td>
+   <td>Performance
+   </td>
+   <td>Scalability
+   </td>
+   <td>Flexibility
+   </td>
+   <td>Complexity
+   </td>
+   <td>Functionality
+   </td>
+  </tr>
+  <tr>
+   <td>Key Value
+   </td>
+   <td>high
+   </td>
+   <td>high
+   </td>
+   <td>high
+   </td>
+   <td>none
+   </td>
+   <td>Variable
+   </td>
+  </tr>
+  <tr>
+   <td>Document stores
+   </td>
+   <td>high
+   </td>
+   <td>Variable (high)
+   </td>
+   <td>high
+   </td>
+   <td>low
+   </td>
+   <td>Variable (low)
+   </td>
+  </tr>
+  <tr>
+   <td>Column DB
+   </td>
+   <td>high
+   </td>
+   <td>high
+   </td>
+   <td>moderate
+   </td>
+   <td>low
+   </td>
+   <td>minimal
+   </td>
+  </tr>
+  <tr>
+   <td>Graph
+   </td>
+   <td>Variable
+   </td>
+   <td>Variable
+   </td>
+   <td>high
+   </td>
+   <td>high
+   </td>
+   <td>Graph theory
+   </td>
+  </tr>
+</table>
+
+
+
+### Differences between SQL and NoSQL
+
+The table below summarizes the main differences between SQL and NoSQL databases.
+
+
+<table>
+  <tr>
+   <td>
+   </td>
+   <td>SQL Databases
+   </td>
+   <td>NoSQL Databases
+   </td>
+  </tr>
+  <tr>
+   <td>Data Storage Model
+   </td>
+   <td>Tables with fixed rows and columns
+   </td>
+   <td>Document: JSON documents, Key-value: key-value pairs, Wide-column: tables with rows and dynamic columns, Graph: nodes and edges
+   </td>
+  </tr>
+  <tr>
+   <td>Primary Purpose
+   </td>
+   <td>General purpose
+   </td>
+   <td>Document: general purpose, Key-value: large amounts of data with simple lookup queries, Wide-column: large amounts of data with predictable query patterns, Graph: analyzing and traversing relationships between connected data
+   </td>
+  </tr>
+  <tr>
+   <td>Schemas
+   </td>
+   <td>Rigid
+   </td>
+   <td>Flexible
+   </td>
+  </tr>
+  <tr>
+   <td>Scaling
+   </td>
+   <td>Vertical (scale-up with a larger server)
+   </td>
+   <td>Horizontal (scale-out across commodity servers)
+   </td>
+  </tr>
+  <tr>
+   <td>Multi-Record <a href="https://en.wikipedia.org/wiki/ACID">ACID </a>Transactions
+   </td>
+   <td>Supported
+   </td>
+   <td>Most do not support multi-record ACID transactions. However, some—like MongoDB—do.
+   </td>
+  </tr>
+  <tr>
+   <td>Joins
+   </td>
+   <td>Typically required
+   </td>
+   <td>Typically not required
+   </td>
+  </tr>
+  <tr>
+   <td>Data to Object Mapping
+   </td>
+   <td>Requires ORM (object-relational mapping)
+   </td>
+   <td>Many do not require ORMs. Document DB documents map directly to data structures in most popular programming languages.
+   </td>
+  </tr>
+</table>
+
+
+
+### Advantages
+
+
+
+*   Flexible Data Models
+
+    Most NoSQL systems feature flexible schemas. A flexible schema means you can easily modify your database schema to add or remove fields to support for evolving application requirements. This facilitates with continuous application development of new features without database operation overhead.
+
+*   Horizontal Scaling
+
+    Most NoSQL systems allow you to scale horizontally, which means you can add in cheaper & commodity hardware, whenever you want to scale a system. On the other hand SQL systems generally scale Vertically (a more powerful server). NoSQL systems can also host huge data sets when compared to traditional SQL systems.
+
+*   Fast Queries
+
+    NoSQL can generally be a lot faster than traditional SQL systems due to data denormalization and horizontal scaling. Most NoSQL systems also tend to store similar data together facilitating faster query responses. 
+
+*   Developer productivity
+
+    NoSQL systems tend to map data based on the programming data structures. As a result developers need to perform fewer data transformations leading to increased productivity & fewer bugs.

courses/databases_nosql/key_concepts.md

@@ -0,0 +1,274 @@
+## Key Concepts
+
+Lets looks at some of the key concepts when we talk about NoSQL or distributed systems
+
+
+### CAP Theorem			
+
+					
+
+In a keynote titled “[Towards Robust Distributed Systems](https://sites.cs.ucsb.edu/~rich/class/cs293b-cloud/papers/Brewer_podc_keynote_2000.pdf)” at ACM’s PODC symposium in 2000 Eric Brewer came up with the so-called CAP-theorem which is widely adopted today by large web companies as well as in the NoSQL community. The CAP acronym stands for **C**onsistency, **A**vailability & **P**artition Tolerance.
+
+
+
+*   **Consistency**
+
+    It refers to how consistent a system is after an execution. A distributed system is called consistent when a write made by a source is available for all readers of that shared data. Different NoSQL systems support different levels of consistency.
+
+*   **Availability**
+
+    It refers to how a system responds to loss of functionality of different systems due to hardware and software failures. A high availability implies that a system is still available to handle operations (reads and writes) when a certain part of the system is down due to a failure or upgrade.
+
+*   **Partition Tolerance**
+
+    It is the ability of the system to continue operations in the event of a network partition. A network partition occurs when a failure causes two or more islands of networks where the systems can’t talk to each other across the islands temporarily or permanently. 
+
+
+Brewer alleges that one can at most choose two of these three characteristics in a shared-data system. The CAP-theorem states that a choice can only be made for two options out of consistency, availability and partition tolerance. A growing number of use cases in large scale applications tend to value reliability implying that availability & redundancy are more valuable than consistency. As a result these systems struggle to meet ACID properties. They attain this by loosening on the consistency requirement i.e Eventual Consistency. 					
+
+**Eventual Consistency **means that all readers will see writes, as time goes on: “In a steady state, the system will eventually return the last written value”. Clients therefore may face an inconsistent state of data as updates are in progress. For instance, in a replicated database updates may go to one node which replicates the latest version to all other nodes that contain a replica of the modified dataset so that the replica nodes eventually will have the latest version. 
+
+NoSQL systems support different levels of eventual consistency models. For example:
+
+
+
+*   Read Your Own Writes Consistency
+
+    A client will see his updates immediately after they are written. The reads can hit nodes other than the one where it was written. However he might not see updates by other clients immediately. 
+
+*   Session Consistency:
+
+    A client will see the updates to his data within a session scope. This generally indicates that reads & writes occur on the same server. Other clients using the same nodes will receive the same updates. 
+
+*   Casual Consistency
+
+    A system provides causal consistency if the following condition holds: write operations that are related by potential causality are seen by each process of the system in order. Different processes may observe concurrent writes in different orders 
+
+
+				
+
+Eventual consistency is useful if concurrent updates of the same partitions of data are unlikely and if clients do not immediately depend on reading updates issued by themselves or by other clients.
+
+Depending on what consistency model was chosen for the system (or parts of it), determines where the requests are routed, ex: replicas. 
+
+CAP alternatives illustration
+
+
+<table>
+  <tr>
+   <td>Choice
+   </td>
+   <td>Traits
+   </td>
+   <td>Examples
+   </td>
+  </tr>
+  <tr>
+   <td>Consistency + Availability
+<p>
+(Forfeit Partitions)
+   </td>
+   <td>2-phase commits
+<p>
+Cache invalidation protocols
+   </td>
+   <td>Single-site databases Cluster databases 
+<p>
+LDAP
+<p>
+xFS file system 
+   </td>
+  </tr>
+  <tr>
+   <td>Consistency + Partition tolerance
+<p>
+ (Forfeit Availability)
+   </td>
+   <td>Pessimistic locking
+<p>
+Make minority partitions unavailable
+   </td>
+   <td>Distributed databases Distributed locking Majority protocols
+   </td>
+  </tr>
+  <tr>
+   <td>Availability + Partition tolerance (Forfeit Consistency)
+   </td>
+   <td>expirations/leases 
+<p>
+conflict resolution optimistic
+   </td>
+   <td>DNS
+<p>
+Web caching
+   </td>
+  </tr>
+</table>
+
+
+### Versioning of Data in distributed systems
+
+When data is distributed across nodes, it can be modified on different nodes at the same time (assuming strict consistency is enforced). Questions arise on conflict resolution for concurrent updates. Some of the popular conflict resolution mechanism are
+
+
+
+*   **Timestamps**
+
+    This is the most obvious solution. You sort updates based on chronological order and choose the latest update. However this relies on clock synchronization across different parts of the infrastructure. This gets even more complicated when parts of systems are spread across different geographic locations. 
+
+*   **Optimistic Locking**
+
+    You associate a unique value like a clock or counter with every data update. When a client wants to update data, it has to specify which version of data needs to be updated. This would mean you need to keep track of history of the data versions. 
+
+*   **Vector Clocks**
+
+    A vector clock is defined as a tuple of clock values from each node. In a distributed environment, each node maintains a tuple of such clock values which represent the state of the nodes itself and its peers/replicas. A clock value may be real timestamps derived from local clock or version no.  
+
+<p id="gdcalert1" ><span style="color: red; font-weight: bold" images/vector_clocks.png> </span></p>
+
+
+![alt_text](images/vector_clocks.png "Vector Clocks")
+
+
+
+                
+<p align="center"><span style="text-decoration:underline;  font-weight:bold;">Vector clocks illustration</span></p>
+
+Vector clocks have the following advantages over other conflict resolution mechanism
+
+
+
+1. No dependency on synchronized clocks
+2. No total ordering of revision nos required for casual reasoning 
+
+No need to store and maintain multiple versions of the data on different nodes.**     **
+
+
+### Partitioning
+
+When the amount of data crosses the capacity of a single node, we need to think of splitting data, creating replicas for load balancing & disaster recovery. Depending on how dynamic the infrastructure is, we have a few approaches that we can take.
+
+
+
+1. **Memory cached**
+
+    These are partitioned in-memory databases that are primarily used for transient data. These databases are generally used as a front for traditional RDBMS. Most frequently used data is replicated from a rdbms into a memory database to facilitate fast queries and to take the load off from backend DB’s. A very common example is memcached or couchbase. 
+
+2. **Clustering**
+
+    Traditional cluster mechanisms abstract away the cluster topology from clients. A client need not know where the actual data is residing and which node it is talking to. Clustering is very commonly used in traditional RDBMS where it can help scaling the persistent layer to a certain extent. 
+
+3. **Separating reads from writes**
+
+    In this method, you will have multiple replicas hosting the same data. The incoming writes are typically sent to a single node (Leader) or multiple nodes (multi-Leader), while the rest of the replicas (Follower) handle reads requests. The leader replicates writes asynchronously to all followers. However the write lag can’t be completely avoided. Sometimes a leader can crash before it replicates all the data to a follower. When this happens, a follower with the most consistent data can be turned into a leader. As you can realize now, it is hard to enforce full consistency in this model. You also need to consider the ratio of read vs write traffic. This model won’t make sense when writes are higher than reads. The replication methods can also vary widely. Some systems do a complete transfer of state periodically, while others use a delta state transfer approach. You could also transfer the state by transferring the operations in order. The followers can then apply the same operations as the leader to catch up.
+
+4. **Sharding**
+
+    Sharing refers to dividing data in such a way that data is distributed evenly (both in terms of storage & processing power) across a cluster of nodes. It can also imply data locality, which means similar & related data is stored together to facilitate faster access. A shard in turn can be further replicated to meet load balancing or disaster recovery requirements. A single shard replica might take in all writes (single leader) or multiple replicas can take writes (multi-leader). Reads can be distributed across multiple replicas. Since data is now distributed across multiple nodes, clients should be able to consistently figure out where data is hosted. We will look at some of the common techniques below. The downside of sharding is that joins between shards is not possible. So an upstream/downstream application has to aggregate the results from multiple shards.
+
+<p id="gdcalert2" ><span style="color: red; font-weight: bold" images/database_sharding.png></span></p>
+
+
+![alt_text]( images/database_sharding.png "Sharding")
+
+
+<p align="center"><span style="text-decoration:underline;  font-weight:bold;">Sharding example</span> </p>
+
+
+### Hashing
+
+A hash function is a function that maps one piece of data—typically describing some kind of object, often of arbitrary size—to another piece of data, typically an integer, known as _hash code_, or simply _hash_. In a partitioned database, it is important to consistently map a key to a server/replica. 
+
+For ex: you can use a very simple hash as a modulo function.
+
+
+    _p = k mod n_
+
+Where 
+
+
+    p -> partition,
+
+
+    k -> primary key
+
+
+    n -> no of nodes
+
+The downside of this simple hash is that, whenever the cluster topology changes, the data distribution also changes. When you are dealing with memory caches, it will be easy to distribute partitions around. Whenever a node joins/leaves a topology, partitions can reorder themselves, a cache miss can be re-populated from backend DB. However when you look at persistent data, it is not possible as the new node doesn’t have the data needed to serve it. This brings us to consistent hashing.
+
+
+#### Consistent Hashing
+
+Consistent hashing is a distributed hashing scheme that operates independently of the number of servers or objects in a distributed _hash table_ by assigning them a position on an abstract circle, or _hash ring_. This allows servers and objects to scale without affecting the overall system.
+
+Say that our hash function h() generates a 32-bit integer. Then, to determine to which server we will send a key k, we find the server s whose hash h(s) is the smallest integer that is larger than h(k). To make the process simpler, we assume the table is circular, which means that if we cannot find a server with a hash larger than h(k), we wrap around and start looking from the beginning of the array.
+
+<p id="gdcalert3" ><span style="color: red; font-weight: bold" images/consistent_hashing.png> </span></p>
+
+
+![alt_text]( images/consistent_hashing.png "Consistent Hashing")
+
+
+<p align="center"><span style="text-decoration:underline;  font-weight:bold;">Consistent hashing illustration</span></p>
+
+In consistent hashing when a server is removed or added then only the keys from that server are relocated. For example, if server S3 is removed then, all keys from server S3 will be moved to server S4 but keys stored on server S4 and S2 are not relocated. But there is one problem, when server S3 is removed then keys from S3 are not equally distributed among remaining servers S4 and S2. They are only assigned to server S4 which increases the load on server S4.
+
+To evenly distribute the load among servers when a server is added or removed, it creates a fixed number of replicas ( known as virtual nodes) of each server and distributes it along the circle. So instead of server labels S1, S2 and S3, we will have S10 S11…S19, S20 S21…S29 and S30 S31…S39. The factor for a number of replicas is also known as _weight_, depending on the situation.
+
+
+          
+
+All keys which are mapped to replicas Sij are stored on server Si. To find a key we do the same thing, find the position of the key on the circle and then move forward until you find a server replica. If the server replica is Sij then the key is stored in server Si.
+
+Suppose server S3 is removed, then all S3 replicas with labels S30 S31 … S39 must be removed. Now the objects keys adjacent to S3X labels will be automatically re-assigned to S1X, S2X and S4X. All keys originally assigned to S1, S2 & S4 will not be moved.   
+
+Similar things happen if we add a server. Suppose we want to add a server S5 as a replacement of S3 then we need to add labels S50 S51 … S59. In the ideal case, one-fourth of keys from S1, S2 and S4 will be reassigned to S5.
+
+When applied to persistent storages, further issues arise: if a node has left the scene, data stored on this node becomes unavailable, unless it has been replicated to other nodes before; in the opposite case of a new node joining the others, adjacent nodes are no longer responsible for some pieces of data which they still store but not get asked for anymore as the corresponding objects are no longer hashed to them by requesting clients. In order to address this issue, a replication factor (r) can be introduced.         
+
+Introducing replicas in a partitioning scheme—besides reliability benefits—also makes it possible to spread workload for read requests that can go to any physical node responsible for a requested piece of data. Scalability doesn’t work if the clients have to decide between multiple versions of the dataset, because they need to read from a quorum of servers which in turn reduces the efficiency of load balancing. 
+
+        
+
+
+### Quorum
+
+Quorum is the minimum number of nodes in a cluster that must be online and be able to communicate with each other. If any additional node failure occurs beyond this threshold, the cluster will stop running.
+
+    
+
+      
+
+To attain a quorum, you need a majority of the nodes. Commonly it is (N/2 + 1), where N is the total no of nodes in the system. For ex, 
+
+In a 3 node cluster, you need 2 nodes for a majority,
+
+In a 5 node cluster, you need 3 nodes for a majority,
+
+In a 6 node cluster, you need 4 nodes for a majority. 
+
+<p id="gdcalert4" ><span style="color: red; font-weight: bold" images/Quorum.png > </span></p>
+
+
+![alt_text](images/Quorum.png "image_tooltip")
+
+<p align="center"> <span style="text-decoration:underline; font-weight:bold;">Quorum example</span> </p>
+
+
+
+Network problems can cause communication failures among cluster nodes. One set of nodes might be able to communicate together across a functioning part of a network but not be able to communicate with a different set of nodes in another part of the network. This is known as split brain in cluster or cluster partitioning.
+
+Now the partition which has quorum is allowed to continue running the application. The other partitions are removed from the cluster.
+
+Eg: In a 5 node cluster, consider what happens if nodes 1, 2, and 3 can communicate with each other but not with nodes 4 and 5. Nodes 1, 2, and 3 constitute a majority, and they continue running as a cluster. Nodes 4 and 5, being a minority, stop running as a cluster. If node 3 loses communication with other nodes, all nodes stop running as a cluster. However, all functioning nodes will continue to listen for communication, so that when the network begins working again, the cluster can form and begin to run.
+
+Below diagram demonstrates Quorum selection on a cluster partitioned into two sets.
+
+<p id="gdcalert5" ><span style="color: red; font-weight: bold" images/cluster_quorum.png> </span></p>
+
+
+![alt_text](images/cluster_quorum.png "image_tooltip")
+
+**<p align="center"><span style="text-decoration:underline; font-weight:bold;">Cluster Quorum example</span></p>**
+

courses/git/branches.md

@@ -83,7 +83,7 @@ Above tree structure should make things clear. Notice a clear branch/fork on com
 
 ## Merges
 
-Now say the feature you were working on branch `b1` is complete. And you need to merge it on master branch, where all the final version of code goes. So first you will checkout to branch master and then you will pull the latest code from upstream (eg: GitHub). Then you need to merge your code from `b1` into master. And there could be two ways this can be done.
+Now say the feature you were working on branch `b1` is complete and you need to merge it on master branch, where all the final version of code goes. So first you will checkout to branch master and then you pull the latest code from upstream (eg: GitHub). Then you need to merge your code from `b1` into master. There could be two ways this can be done.
 
 Here is the current history:
 
@@ -96,7 +96,7 @@ spatel1-mn1:school-of-sre spatel1$ git log --oneline --graph --all
 * df2fb7a adding file 1
 ```
 
-**Option 1: Directly merge the branch.** Merging the branch b1 into master will result in a new merge commit which will merge changes from two different lines of history and create a new commit of the result.
+**Option 1: Directly merge the branch.** Merging the branch b1 into master will result in a new merge commit. This will merge changes from two different lines of history and create a new commit of the result.
 
 ```bash
 spatel1-mn1:school-of-sre spatel1$ git merge b1

courses/git/git-basics.md

@@ -1,6 +1,6 @@
-# School Of SRE: Git
+# Git
 
-## Pre - Reads
+## Prerequisites
 
 1. Have Git installed [https://git-scm.com/downloads](https://git-scm.com/downloads)
 2. Have taken any git high level tutorial or following LinkedIn learning courses
@@ -8,26 +8,26 @@
       - [https://www.linkedin.com/learning/git-branches-merges-and-remotes/](https://www.linkedin.com/learning/git-branches-merges-and-remotes/)
       - [The Official Git Docs](https://git-scm.com/doc)
 
-## What to expect from this training
+## What to expect from this course
 
-As an engineer in the field of computer science, having knowledge of version control tools becomes almost a requirement. While there are a lot of version control tools that exist today, Git perhaps is the most used one and this course we will be working with Git. While this course does not start with Git 101 and expects basic knowledge of git as a prerequisite, it will reintroduce the git concepts known by you with details covering what is happening under the hood as you execute various git commands. So that next time you run a git command, you will be able to press enter more confidently!
+As an engineer in the field of computer science, having knowledge of version control tools becomes almost a requirement. While there are a lot of version control tools that exist today like SVN, Mercurial, etc, Git perhaps is the most used one and this course we will be working with Git. While this course does not start with Git 101 and expects basic knowledge of git as a prerequisite, it will reintroduce the git concepts known by you with details covering what is happening under the hood as you execute various git commands. So that next time you run a git command, you will be able to press enter more confidently!
 
-## What is not covered under this training
+## What is not covered under this course
 
 Advanced usage and specifics of internal implementation details of Git.
 
-## Training Content
+## Course Content
 
 ### Table of Contents
 
- 1. Git Basics
- 2. Working with Branches
- 3. Git with Github
- 4. Hooks
+ 1. [Git Basics](https://linkedin.github.io/school-of-sre/git/git-basics/#git-basics)
+ 2. [Working with Branches](https://linkedin.github.io/school-of-sre/git/branches/)
+ 3. [Git with Github](https://linkedin.github.io/school-of-sre/git/github-hooks/#git-with-github)
+ 4. [Hooks](https://linkedin.github.io/school-of-sre/git/github-hooks/#hooks)
 
 ## Git Basics
 
-Though you might be aware already, let's revisit why we need a version control system. As the project grows and multiple developers start working on it, an efficient method for collaboration is warranted. Git helps the team collaborate easily and also maintains history of the changes happened with the codebase.
+Though you might be aware already, let's revisit why we need a version control system. As the project grows and multiple developers start working on it, an efficient method for collaboration is warranted. Git helps the team collaborate easily and also maintains the history of the changes happening with the codebase.
 
 ### Creating a Git Repo
 
@@ -92,7 +92,7 @@ spatel1-mn1:school-of-sre spatel1$ git commit -m "adding file 1"
 create mode 100644 file1.txt
 ```
 
-Notice how after adding the file, git status says `Changes to be commited:`. What it means is whatever is listed there, will be included in the next commit. Then we go ahead and create a commit, with an attached messaged via `-m`.
+Notice how after adding the file, git status says `Changes to be committed:`. What it means is whatever is listed there, will be included in the next commit. Then we go ahead and create a commit, with an attached messaged via `-m`.
 
 ### More About a Commit
 

courses/git/github-hooks.md

@@ -1,4 +1,4 @@
-## Git with Github
+# Git with Github
 
 Till now all the operations we did were in our local repo while git also helps us in a collaborative environment. GitHub is one place on the internet where you can centrally host your git repos and collaborate with other developers.
 

courses/index.md

@@ -1 +1,25 @@
-Hello, World!!!
+# School of SRE
+![School of SRE](img/sos.png)
+Early 2019, we started visiting campuses to recruit the brightest minds to ensure LinkedIn and all the services that it is composed of is always available for everyone. This function at Linkedin falls in the purview of the Site Reliability Engineering team and Site Reliability Engineers ( SRE ) who are Software Engineers who specialize in reliability. SREs apply the principles of computer science and engineering to the design and development of computer systems: generally, large distributed ones.  
+
+As we continued on this journey we started getting a lot of questions from these campuses on what exactly site engineering roll entails? and, how could someone learn the skills and the disciplines involved to become a successful site engineer?   Fast forward a few months, and a few of these campus students had joined LinkedIn either as Interns or as full time engineers to become a part of the Site Engineering team, we also had a few lateral hires who joined our organization who were not from a traditional SRE background.  That's when a few of us got together and started to think about how we can on board new new graduate engineers to the site engineering team.
+
+There is a vast amount of resources scattered throughout the web on what are the roles and responsibilities of an SREs, how to monitor site health, handling incidents, maintain SLO/SLI etc. But there are very few resources out there guiding someone on what all basic skill sets one has to acquire as a beginner. Because of the lack of these resources we felt that individuals are having a tough time getting into open positions in the industry. We created School Of SRE as a starting point for anyone wanting to build their career in the role of SRE. 
+
+In this course we are focusing on building strong foundational skills. The course is  structured in a way to provide more real life examples and how learning each of the topics can play a bigger role in your day to day SRE life. Currently we are covering the following topics under the School Of SRE:
+
+- Fundamentals Series
+    - [Linux Basics](https://linkedin.github.io/school-of-sre/linux_basics/intro/)
+    - [Git](https://linkedin.github.io/school-of-sre/git/git-basics/)  
+    - [Linux Networking](https://linkedin.github.io/school-of-sre/linux_networking/intro/)
+- [Python and Web](https://linkedin.github.io/school-of-sre/python_web/intro/)
+- Data
+    - Relational databases (MySQL)
+    - NoSQL concepts 
+    - [Big Data](https://linkedin.github.io/school-of-sre/big_data/intro/)
+- [Systems Design](https://linkedin.github.io/school-of-sre/systems_design/intro/)
+- [Security](https://linkedin.github.io/school-of-sre/security/intro/)
+
+We believe continuous learning will help in acquiring deeper knowledge and competencies in order to expand your skill sets, every module has added reference which could be a guide for further learning.  Our hope is that by going through these modules we should be able build the essential skills required for a Site Reliability Engineer.
+
+At Linkedin we are using this curriculum for onboarding our non-traditional hires and new college grads to the SRE role. We had multiple rounds of successful onboarding experience with the new members and helped them to be productive in a very short period of time. This motivated us to opensource these contents for helping other organisations onboarding new engineers to the role and individuals to get into the role.  We realise that the initial content we created is just a starting point and our hope is that the community can help in the journey refining and extending the contents.

courses/linux_basics/command_line_basics.md

@@ -0,0 +1,445 @@
+# Command Line Basics
+
+## Lab Environment Setup
+
+One can use an online bash interpreter to run all the commands that are provided as examples in this course. This will also help you in getting a hands-on experience of various linux commands.
+
+[REPL](https://repl.it/languages/bash) is one of the popular online bash interpreters for running linux commands. We will be using it for running all the commands mentioned in this course.
+
+## What is a Command
+
+A command is a program that tells the operating system to perform
+specific work. Programs are stored as files in linux. Therefore, a
+command is also a file which is stored somewhere on the disk.
+
+Commands may also take additional arguments as input from the user.
+These arguments are called command line arguments. Knowing how to use
+the commands is important and there are many ways to get help in Linux,
+especially for commands. Almost every command will have some form of
+documentation, most commands will have a command-line argument -h or
+\--help that will display a reasonable amount of documentation. But the
+most popular documentation system in Linux is called man pages - short
+for manual pages.
+
+Using \--help to show the documentation for ls command.
+
+![](images/linux/commands/image19.png)
+
+## File System Organization
+
+The linux file system has a hierarchical (or tree-like) structure with
+its highest level directory called root ( denoted by / ). Directories
+present inside the root directory stores file related to the system.
+These directories in turn can either store system files or application
+files or user related files.
+
+![](images/linux/commands/image17.png)
+  
+  bin   | The executable program of most commonly used commands reside in bin directory  
+  sbin  | This directory contains programs used for system administration.
+  home  | This directory contains user related files and directories.
+  lib   | This directory contains all the library files
+  etc   | This directory contains all the system configuration files
+  proc  | This directory contains files related to the running processes on the system
+  dev   | This directory contains files related to devices on the system
+  mnt   | This directory contains files related to mounted devices on the system
+  tmp   | This directory is used to store temporary files on the system
+  usr   |  This directory is used to store application programs on the system
+
+## Commands for Navigating the File System
+
+There are three basic commands which are used frequently to navigate the
+file system:
+
+- ls
+
+- pwd
+
+- cd
+
+We will now try to understand what each command does and how to use
+these commands. You should also practice the given examples on the
+online bash shell.
+
+### pwd (print working directory)
+
+At any given moment of time, we will be standing in a certain directory.
+To get the name of the directory in which we are standing, we can use
+the pwd command in linux.
+
+![](images/linux/commands/image2.png)
+
+We will now use the cd command to move to a different directory and then
+print the working directory.
+
+![](images/linux/commands/image20.png)
+
+### cd (change directory)
+
+The cd command can be used to change the working directory. Using the
+command, you can move from one directory to another.
+
+In the below example, we are initially in the root directory. we have
+then used the cd command to change the directory.
+
+![](images/linux/commands/image3.png)
+
+### ls (list files and directories)**
+
+The ls command is used to list the contents of a directory. It will list
+down all the files and folders present in the given directory.
+
+If we just type ls in the shell, it will list all the files and
+directories present in the current directory.
+
+![](images/linux/commands/image7.png)
+
+We can also provide the directory name as argument to ls command. It
+will then list all the files and directories inside the given directory.
+
+![](images/linux/commands/image4.png)
+
+## Commands for Manipulating Files
+
+There are four basic commands which are used frequently to manipulate
+files:
+
+- touch
+
+- mkdir
+
+- cp
+
+- mv
+
+- rm
+
+We will now try to understand what each command does and how to use
+these commands. You should also practice the given examples on the
+online bash shell.
+
+### touch (create new file)
+
+The touch command can be used to create an empty new file.
+This command is very useful for many other purposes but we will discuss
+the simplest use case of creating a new file.
+
+General syntax of using touch command
+
+```
+touch <file_name>
+```
+
+![](images/linux/commands/image9.png)
+
+### mkdir (create new directories)
+
+The mkdir command is used to create directories.You can use ls command
+to verify that the new directory is created.
+
+General syntax of using mkdir command
+
+```
+mkdir <directory_name>
+```
+
+![](images/linux/commands/image11.png)
+
+### rm (delete files and directories)
+
+The rm command can be used to delete files and directories. It is very
+important to note that this command permanently deletes the files and
+directories. It's almost impossible to recover these files and
+directories once you have executed rm command on them successfully. Do
+run this command with care.
+
+General syntax of using rm command:
+
+```
+rm <file_name>
+```
+
+Let's try to understand the rm command with an example. We will try to
+delete the file and directory we created using touch and mkdir command
+respectively.
+
+![](images/linux/commands/image18.png)
+
+### cp (copy files and directories)
+
+The cp command is used to copy files and directories from one location
+to another. Do note that the cp command doesn't do any change to the
+original files or directories. The original files or directories and
+their copy both co-exist after running cp command successfully.
+
+General syntax of using cp command:
+
+```
+cp <source_path> <destination_path>
+```
+
+We are currently in the '/home/runner' directory. We will use the mkdir
+command to create a new directory named "test_directory". We will now
+try to copy the "\_test_runner.py" file to the directory we created just
+now.
+
+![](images/linux/commands/image23.png)
+
+Do note that nothing happened to the original "\_test_runner.py" file.
+It's still there in the current directory. A new copy of it got created
+inside the "test_directory".
+
+![](images/linux/commands/image14.png)
+
+We can also use the cp command to copy the whole directory from one
+location to another. Let's try to understand this with an example.
+
+![](images/linux/commands/image12.png)
+
+We again used the mkdir command to create a new directory called
+"another_directory". We then used the cp command along with an
+additional argument '-r' to copy the "test_directory".
+
+**mv (move files and directories)**
+
+The mv command can either be used to move files or directories from one
+location to another or it can be used to rename files or directories. Do
+note that moving files and copying them are very different. When you
+move the files or directories, the original copy is lost.
+
+General syntax of using mv command:
+
+```
+mv <source_path> <destination_path>
+```
+
+In this example, we will use the mv command to move the
+"\_test_runner.py" file to "test_directory". In this case, this file
+already exists in "test_directory". The mv command will just replace it.
+**Do note that the original file doesn't exist in the current directory
+after mv command ran successfully.**
+
+![](images/linux/commands/image26.png)
+
+We can also use the mv command to move a directory from one location to
+another. In this case, we do not need to use the '-r' flag that we did
+while using the cp command. Do note that the original directory will not
+exist if we use mv command.
+
+One of the important uses of the mv command is to rename files and
+directories. Let's see how we can use this command for renaming.
+
+We have first changed our location to "test_directory". We then use the
+mv command to rename the ""\_test_runner.py" file to "test.py".
+
+![](images/linux/commands/image29.png)
+
+## Commands for Viewing Files
+
+There are three basic commands which are used frequently to view the
+files:
+
+- cat
+
+- head
+
+- tail
+
+We will now try to understand what each command does and how to use
+these commands. You should also practice the given examples on the
+online bash shell.
+
+We will create a new file called "numbers.txt" and insert numbers from 1
+to 100 in this file. Each number will be in a separate line.
+
+![](images/linux/commands/image21.png)
+
+Do not worry about the above command now. It's an advanced command which
+is used to generate numbers. We have then used a redirection operator to
+push these numbers to the file. We will be discussing I/O redirection in the
+later sections.
+
+
+### cat
+
+The most simplest use of cat command is to print the contents of the file on
+your output screen. This command is very useful and can be used for many
+other purposes. We will study about other use cases later.
+
+![](images/linux/commands/image1.png)
+
+You can try to run the above command and you will see numbers being
+printed from 1 to 100 on your screen. You will need to scroll up to view
+all the numbers.
+
+### head
+
+The head command displays the first 10 lines of the file by default. We
+can include additional arguments to display as many lines as we want
+from the top.
+
+In this example, we are only able to see the first 10 lines from the
+file when we use the head command.
+
+![](images/linux/commands/image15.png)
+
+By default, head command will only display the first 10 lines. If we
+want to specify the number of lines we want to see from start, use the
+'-n' argument to provide the input.
+
+![](images/linux/commands/image16.png)
+
+### tail
+
+The tail command displays the last 10 lines of the file by default. We
+can include additional arguments to display as many lines as we want
+from the end of the file.
+
+![](images/linux/commands/image22.png)
+
+By default, the tail command will only display the last 10 lines. If we
+want to specify the number of lines we want to see from the end, use '-n'
+argument to provide the input.
+
+![](images/linux/commands/image10.png)
+
+In this example, we are only able to see the last 5 lines from the file
+when we use the tail command with explicit -n option.
+
+
+## Echo Command in Linux
+
+The echo command is one of the simplest commands that is used in the
+shell. This command is equivalent to what we have <print> in other
+programming languages.
+
+The echo command prints the given input string on the screen.
+
+![](images/linux/commands/image24.png)
+
+## Text Processing Commands
+
+In the previous section, we learned how to view the content of a file.
+In many cases, we will be interested in performing the below operations:
+
+- Print only the lines which contain a particular word(s)
+
+- Replace a particular word with another word in a file
+
+- Sort the lines in a particular order
+
+There are three basic commands which are used frequently to process
+texts:
+
+- grep
+
+- sed
+
+- sort
+
+We will now try to understand what each command does and how to use
+these commands. You should also practice the given examples on the
+online bash shell.
+
+We will create a new file called "numbers.txt" and insert numbers from 1
+to 10 in this file. Each number will be in a separate line.
+
+![](images/linux/commands/image8.png)
+
+### grep
+
+The grep command in its simplest form can be used to search particular
+words in a text file. It will display all the lines in a file that
+contains a particular input. The word we want to search is provided as
+an input to the grep command.
+
+General syntax of using grep command:
+
+```
+grep <word_to_search> <file_name>
+```
+
+In this example, we are trying to search for a string "1" in this file.
+The grep command outputs the lines where it found this string.
+
+![](images/linux/commands/image5.png)
+
+### sed
+
+The sed command in its simplest form can be used to replace a text in a
+file.
+
+General syntax of using the sed command for replacement:
+
+```
+sed 's/<text_to_replace>/<replacement_text>/' <file_name>
+```
+
+Let's try to replace each occurrence of "1" in the file with "3" using
+sed command.
+
+![](images/linux/commands/image31.png)
+
+The content of the file will not change in the above
+example. To do so, we have to use an extra argument '-i' so that the
+changes are reflected back in the file.
+
+### sort
+
+The sort command can be used to sort the input provided to it as an
+argument. By default, it will sort in increasing order.
+
+Let's first see the content of the file before trying to sort it.
+
+![](images/linux/commands/image27.png)
+
+Now, we will try to sort the file using the sort command. The sort
+command sorts the content in lexicographical order.
+
+![](images/linux/commands/image32.png)
+
+The content of the file will not change in the above
+example.
+
+## I/O Redirection
+
+Each open file gets assigned a file descriptor. A file descriptor is an
+unique identifier for open files in the system. There are always three
+default files open, stdin (the keyboard), stdout (the screen), and
+stderr (error messages output to the screen). These files can be
+redirected.
+
+Everything is a file in linux -
+[https://unix.stackexchange.com/questions/225537/everything-is-a-file](https://unix.stackexchange.com/questions/225537/everything-is-a-file)
+
+Till now, we have displayed all the output on the screen which is the
+standard output. We can use some special operators to redirect the
+output of the command to files or even to the input of other commands.
+I/O redirection is a very powerful feature.
+
+In the below example, we have used the '>' operator to redirect the
+output of ls command to output.txt file.
+
+![](images/linux/commands/image30.png)
+
+In the below example, we have redirected the output from echo command to
+a file.
+
+![](images/linux/commands/image13.png)
+
+We can also redirect the output of a command as an input to another
+command. This is possible with the help of pipes.
+
+In the below example, we have passed the output of cat command as an
+input to grep command using pipe(\|) operator.
+
+![](images/linux/commands/image6.png)
+
+In the below example, we have passed the output of sort command as an
+input to uniq command using pipe(\|) operator. The uniq command only
+prints the unique numbers from the input.
+
+![](images/linux/commands/image28.png)
+
+I/O redirection -
+[https://tldp.org/LDP/abs/html/io-redirection.html](https://tldp.org/LDP/abs/html/io-redirection.html)

courses/linux_basics/conclusion.md

@@ -0,0 +1,25 @@
+# Conclusion
+
+With this we have covered the basics of linux operating systems along with basic commands
+which are used in linux. We have also covered the linux server administration commands.
+
+We hope that this course will make it easier for you to operate on the command line.
+
+## Applications in SRE Role
+
+1. As a SRE, you will be required to perform some general tasks on these linux servers. You will also be using the command line when you are troubleshooting issues.
+2. Moving from one location to another in the filesystem will require the help of ls, pwd and cd commands
+3. You may need to search some specific information in the log files. Grep command would be very useful here. I/O redirection will become handy if you want to store the output in a file or pass it as an input to another command.
+4. Tail command is very useful to view the latest data in the log file.
+5. Different users will have different permissions depending on their roles. We will also not want everyone in the company to access our servers for security reasons. Users permissions can be restricted with chown, chmod and chgrp commands.
+6. SSH is one of the most frequently used commands for a SRE. Logging into servers and troubleshooting along with performing basic administration tasks will only be possible if we are able to login into the server.
+7. What if we want to run an apache server or nginx on a server ? We will first install it using the package manager. Package management commands become important here.
+8. Managing services on servers is another critical responsibility of a SRE. Systemd related commands can help in troubleshooting issues. If a service goes down, we can start it using systemctl start command. We can also stop a service in case it is not needed.
+9. Monitoring is another core responsibility of a SRE. Memory and CPU are two important system level metrics which should be monitored. Commands like top and free are quite helpful here.
+10. If a service is throwing an error, how do we find out the root cause of the error ? We will certainly need to check logs to find out the whole stack trace of the error. The log file will also tell us the number of times the error has occurred along with time when it started.
+
+## Useful Courses and tutorials
+
+* [Edx basic linux commands course](https://courses.edx.org/courses/course-v1:LinuxFoundationX+LFS101x+1T2020/course/)
+* [Edx Red Hat Enterprise Linux Course](https://courses.edx.org/courses/course-v1:RedHat+RH066x+2T2017/course/)
+* [https://linuxcommand.org/lc3_learning_the_shell.php](https://linuxcommand.org/lc3_learning_the_shell.php)