Scalability at Youtube
Scalability in Distributed Systems (10%) 09488987 David Kearns
-Problems YouTube was facing
A big problem YouTube were facing during its rise in popularity was that the number of developers went up by two but the number of videos being watched went up as Mike tells us in the YouTube vid ‘9 orders of magnitude’. In the video Mike also describes a fatal mistake he had made when coding the interface between the servlet and template layer he used a dictionary which is a bad idea. Like a lot of programming errors this was a lot harder to fix later on in the project as opposed to the early life of the program. Other problems Mike describes is splitting the entire project into components, you want to be able to swap components in and out. There’s not going to be one individual or programmer who will know the whole bulk of the code inside out, having so many people working on a particular project can become difficult to organize. This is all about good software design but it is important that components work with each other. The majority of YouTube was initially coded with python and as Mike points out when we watch a video it still executes primarily through this. ‘It’s about what not to do in python ’ he states later on in the video suggesting throughout the years there has been a lot of trial and error in the solving of some problems. Another problem YouTube may have faced was its million lines of python code and maintaining this code in its different components.
Database problems, early years: * MySQL used to save ‘information data’/Meta-data. * YouTube went to a common evolution in servers i.e. Single server evolved into master with multiple read-slaves. ‘sharding approach’. * Updates causing slow replication/cache miss. * Solution to this was to split data traffic into two clusters.
Database: Later years:
- YouTube evolved by switching to database partitioning; which is basically building smaller databases.
- Split into shards
-Problems YouTube was facing
A big problem YouTube were facing during its rise in popularity was that the number of developers went up by two but the number of videos being watched went up as Mike tells us in the YouTube vid ‘9 orders of magnitude’. In the video Mike also describes a fatal mistake he had made when coding the interface between the servlet and template layer he used a dictionary which is a bad idea. Like a lot of programming errors this was a lot harder to fix later on in the project as opposed to the early life of the program. Other problems Mike describes is splitting the entire project into components, you want to be able to swap components in and out. There’s not going to be one individual or programmer who will know the whole bulk of the code inside out, having so many people working on a particular project can become difficult to organize. This is all about good software design but it is important that components work with each other. The majority of YouTube was initially coded with python and as Mike points out when we watch a video it still executes primarily through this. ‘It’s about what not to do in python ’ he states later on in the video suggesting throughout the years there has been a lot of trial and error in the solving of some problems. Another problem YouTube may have faced was its million lines of python code and maintaining this code in its different components.
Database problems, early years: * MySQL used to save ‘information data’/Meta-data. * YouTube went to a common evolution in servers i.e. Single server evolved into master with multiple read-slaves. ‘sharding approach’. * Updates causing slow replication/cache miss. * Solution to this was to split data traffic into two clusters.
Database: Later years:
- YouTube evolved by switching to database partitioning; which is basically building smaller databases.
- Split into shards