Behind the Scenes of Automated Production. How Does It Work?

Automated sports production is growing at a fast pace and it is being used at more and more sporting events. Deep Learning and Artificial Intelligence play a crucial role in the game, but how exactly are they doing it?  

The highest level of Deep Learning in sports production can mimic a human camera operator and video editor. This gives the viewer almost the same experience as a professional live sports broadcast. Without the need for a camera operator or a video editor, automated sports production is at a fraction of the cost. Deep Learning will allow semi-professional and amateur teams to broadcast their games at a low cost.

Understanding the Underlying Technologies

Deep Learning is a subset of machine learning whereby algorithms are used to obtain information and deduct insights from large amounts of data. The data is comprised of visual and image components. Deep Learning will try to mimic the brain as much as possible with its approaches.

Computer vision is the act of extrapolating the information from the videos and images as the computer tries to decipher the visual input from the user. The input can analyze physical objects such as a picture or a video but it can also be something not physical such as a voice.

Deep Learning takes the shape of a very efficient, high end, real time algorithm that is required to run in less than 10 milliseconds per frame for every frame (!!!) to keep the system stable. This means that is has to produce the game instantaneously, just as it is actually happening. This process is very complicated.

The minimal requirement for Deep Learning in sports is to identify the ball and the players. Identifying the ball is a complicated task. The ball can be in a great many different scenarios, on the ground, in the air, in a player’s hands, and out of bounds. Deep Learning also has to differentiate the players on the field from the referees, coaches, bench players and fans. It needs to combine all of the information from the game and decide where to “focus the camera” on the field, whether to zoom in and track the ball or show a panoramic view of the entire field. It also has to try and predict where the ball will be heading with a lead time of four seconds.

AI in Sports

Artificial Intelligence is the process of letting the computer know what is happening with the visual input. The purpose of Artificial Intelligence is to train the computer to think like a human being. The Artificial Intelligence will try to understand its surroundings and predict what it thinks will happen in the future, relying on the input it receives and the output it generates. The more data it has to analyze, the more it improves.

There are many instances where Artificial Intelligence is more useful than a human being. For example, when tracking a ball during a game, it will instantly find the ball and start tracking it while a person will take longer to find the ball and will have trouble tracking it as it is in constant motion.

There will be times where the Artificial Intelligence will not work, often because of a computer glitch. This might cause the camera to lose focus on the ball for a short amount of time before it re-centers itself. This will more often happen at soccer games because the soccer field is large and there is much more space to cover compared to a basketball court which is a lot smaller.

It is nearly impossible to cover every pixel of every frame for an entire soccer field for a full 90-minute match. During soccer games, the soccer camera will focus on the ball and center it on the screen. Sometimes, the ball will become difficult to locate as it is sometimes in the corner or behind a player, or subject to difficult lighting conditions. So while AI might lose the ball for a second, it will find it again and continue to track.

Calibration

Another important aspect of enabling the systems to work properly is calibration. With permanently installed systems this is done during the camera installation phase. With portable systems, this must be done before the start of filming each game.

The process is not complicated. It requires identifying the boundaries of the field or court in order to clarify the boundaries that the Deep Learning needs to track. This is also crucial for tracking which players are actually playing the game and which are inactive on the sidelines and off the field.

What is the Next Level of This Technology?

There are always ways to improve when it comes to technology. One way to take this technology to the next level is to add more cameras to the feed at the venue where the games are being played. This not only adds more angles and greater depth of coverage. The more cameras there are, the higher the accuracy in tracking the game. If one camera is not working or is malfunctioning, there are other cameras to rely on. This will help track the players, ball, field and any other objects that need tracking more accurately.

Another way to improve the technology is to update or add to the algorithm. The algorithm for the cameras currently has a 98% accuracy rate. This is something that can always be improved. Every second of every game is very important and the viewers do not want to miss any of the action. Pushing for an even higher percentage would raise the level of cover accuracy and lower the chances of missed action. The algorithms are constantly being refined to deliver better results.

In short 

Deep Learning and AI in sports are two great, useful technological advances that are enabling semi-professional, amateur, women, youth and niche sports produce and broadcast their games almost identically to professional teams – at a fraction of the cost!