In 2010, YouTube was in a difficult position. The platform grew fast and the infrastructure could not keep up. It did not help to save more CPU and memory. it was still falling apart at the seams.
That's when two YouTube engineers, Sugu Sougoumarane and Mike Soloman, decided to step back and analyze the problem from a different perspective: "When we actually sat down and found a huge spreadsheet of all the problems and solutions It was obvious that we had to create something between the application and the MySQL level and moderate all these queries, "Sugu said in a session with TechRadar Pro on the fringes of the Percona Live Conference Europe 2019 in Amsterdam.
The solution to the problem was Vitess, which essentially makes it easy to scale and manage large clusters of MySQL databases. Sugu tells us that the project has grown a lot since its inception on YouTube. At the time, Vitess was mainly addressing scalability issues: "As time went on, however, more and more features were in demand as this proxy stepped into the middle. And we have grown organically to where we are today. "
Sugu says his users prefer Vitess over MySQL clustering because it provides flexibility:" MySQL clustering has challenges with scalability. So if you want to scale, you want the parts to be loosely connected. However, using [MySQL] clustering does not give you the flexibility to make things easier. I think that's why Vitess is favored by users. "
An important prerequisite for scaling a database is managing how a database is partitioned or partitioned in DBA language. One of the reasons for the popularity of Vitess is its effective splinter scheme. VTGate, one of the two main proxies in Vitess that started as a connection consolidator, now becomes an important part of the solution: "When we first created Vitess, the applications had to be shard-enabled. So the application had to say, "I'd like to send this query, I want to send it to this shard." This means that once you've decided to use Vitess, you've had to rewrite your app, which is basically Shard-enabled calls Vitess manages the clusters for you.
This changed in 201
Cloud native solution
native "solution. We are skeptical and ask Sugu if there is more to it than just using a buzzword. The reason for the cloud capability of Vitess lies in Google's cluster manager named Borg.
Vitess was originally designed to operate in the data centers of YouTube until Google decided to relocate it internally within Google in 2013:
"Google's Borg is a beast because it's an anti-storage environment." We had to get Vitess to work in this environment, where Borg outsources your pod and erases your data at will, and you had to survive in that environment. "
This meant developers had to create failover capabilities in this environment. It's important to ensure that the pods revive themselves after being dismantled by Borg:
" And basically, these are the same rules that Kubernetes has. If a pod fails in Kubernetes, your data will be lost. So we were basically ready for Kubernetes before the birth of Kubernetes.
They also had to make minor changes to the Vitess code because the lifecycle of deployments in the cloud is very different from their bare metal-based lifecycle: "In Bare Metal, you could have a master for six months. On Google, a week would be a miracle, as Google pods constantly reschedule and eventually shut down your pod.
Another aspect of this rescheduling has helped Vitess prepare for the ever-changing cloud environments: "If it's (Googles Scheduler) rescheduled sometimes it will put something else at the same address. For example, a shard could be moved and scheduled into another shard. You will not even know it, because the scheme is correct. You send a query and answers are sent to you. So we had to build protection from these things. "