These days, seamless audio and video synchronization is the most important component when developing multimedia solutions/applications for Android. Whether you’re building a multimedia system like video player, or streaming app, or multimedia-intensive game, ensuring that the audio and video components are perfectly synchronized is key to delivering a high-quality user experience.
Nowadays, Android has become an absolute set of software for all devices like tablets, wearables, set-top boxes, smart TVs, notebooks, etc. Millions of people use streaming platforms like Netflix, Hotstar, YouTube, etc. So, here audio and video synchronization play a major role in Android. As per a market report by Infinity Business Insights, the global video streaming market is experiencing substantial growth. The report states that the video streaming market size is estimated to be USD 89.03 billion in 2023. Furthermore, it predicts a promising future for the industry, with a projected Compound Annual Growth Rate (CAGR) of 21.5% during the forecast period from 2023 to 2030. This data highlights the increasing demand and popularity for video streaming platforms worldwide.
In this blog post, we will explore various Audio-Video sync issues and best practices for achieving robust audio-video synchronization in Android applications.
Understanding the importance of audio-video synchronization
Audio-video synchronization greatly impacts the user experience by enhancing immersion, quality perception, emotional engagement, realism, interactivity, and accessibility. Precise synchronization ensures a seamless and enjoyable multimedia solutions experience, allowing users to fully connect with the content and its intended message.
Reasons for audio and video going out of synchronization
Audio and video can get out of sync due to network delays, hardware or software limitations, timing discrepancies during encoding, buffering and playback control issues, device-specific factors, audio and video pipeline flow implementation, frame rate differences, processing delays, and codec-related problems.
To ensure proper synchronization, it is essential to address these factors and implement appropriate synchronization mechanisms in media playback systems.
Measurement technique for audio and video synchronization on Android
To measure audio and video synchronization on Android devices, one can use various techniques. Here are some common methods that can be followed for multimedia systems.
Manual Observation: One simple technique is to inspect the audio and video streams visually and audibly by playing a synchronized media file on an Android device. Pay attention to any noticeable delay or offset between the audio and video components. This method is subjective and may not provide precise measurements but can give you a general sense of synchronization.
Audio-video sync test patterns: Some video test patterns come with built-in audio synchronization cues. These patterns have specific audio and visual events that should align perfectly if the synchronization is accurate. You can use specialized video test pattern files or apps like the “Superpowered Mobile Audio Latency Test App” designed for testing audio-video synchronization.
External hardware: To achieve more accurate measurements, you can use external hardware devices like audio-video synchronizer test device which is specifically designed for audio-video synchronization analysis. By analyzing the recorded audio and video, you can determine synchronization discrepancies.
To measure the timestamp difference between audio and video there are some tools and applications available. Sync-One2, Rohde & Schwarz VTC Video Analyzer are examples of devices. While SuperPowered Latency, and SyncTest are some examples of applications used. These tools use different mechanisms to measure the timestamp difference between audio and video.
Maintaining synchronization on the Android platform
To maintain audio-video synchronization in Android applications, it is important to ensure that the audio and video pipelines render their frames with identical timestamps at the same time. The audio playback position is typically used as the primary time reference. The video pipeline outputs video frames that match the latest rendered audio frame. Here are some best practices for accurately calculating the last rendered audio timestamp in Android applications.
- Querying audio timestamps: Android provides APIs to query audio timestamps and latencies at different stages of the audio pipeline. These APIs can be used to obtain information about the audio playback position, such as the presentation timestamp of the last rendered audio frame. The AudioTrack class, for example, provides the “getTimestamp()” method to retrieve the audio frame timestamp.
- Synchronizing video playback: Once you obtain the last rendered audio frame’s timestamp, you can synchronize video playback accordingly. Video frames should match or be close to the timestamp of the last rendered audio frame. This synchronization can be achieved by adjusting the video playback position or using buffer management techniques to ensure the appropriate video frame is rendered at the desired time.
- Handling audio and video latencies: In addition to obtaining audio timestamps, it is important to consider and account for audio and video latencies. Latencies can occur due to various factors, such as buffering, decoding, transmission time, and rendering delays. Audio-video synchronizer test devices can help in measuring and compensating for these latencies, you can ensure that audio and video frames are presented accurately.
- Implementing error handling: It is crucial to handle any errors or discrepancies that may occur during the audio-video synchronization process. This includes cases where audio timestamps are not available or accurate, video frames are dropped or delayed, or synchronization becomes temporarily disrupted. Proper error-handling mechanisms can mitigate synchronization issues and ensure a smooth playback experience.
The role of media frameworks in Android
Android has two primary media frameworks named MediaPlayer and ExoPlayer, which are used for audio and video playback.
MediaPlayer frameworks provide the necessary infrastructure and functionalities to manage timing, handle latency, synchronize clocks, and optimize performance for audio and video synchronization. They play a critical role in ensuring that audio and video streams are presented together accurately, resulting in a seamless and synchronized multimedia playback experience.
ExoPlayer is a component that plays audio and video files. It has a main loop that continuously handles and plays data. MediaCodecAudioRenderer handles audio playback, while MediaCodecVideoRenderer manages video playback, including frame synchronization. The Video Frame Release Time Helper adjusts video frame display time based on the system’s VSync (Vertical Sync) signal.
Below diagram shows how ExoPlayer manages audio and video synchronization.
A/V Sync in ExoPlayer
The role of Android audio HAL
In Android, the audio Hardware Abstraction Layer (HAL) writes audio data to the output buffer. This process relies on two essential parameters known as the last frame position and the last timestamp. The last frame position and the last timestamp are like markers that help keep track of the audio being played. They tell us where the audio is in the buffer and when it was recently played.
To make sure the audio and video stay in sync, we use two special functions. One tells us how much space is left in the buffer, and the other tells us when the last audio was played.
By comparing these markers with the buffer space and timing information, we can ensure the audio doesn’t overflow or run out of buffer space. We can also estimate when to play new audio so that it matches the video. In simple words, these markers and functions help to manage the audio so that it plays smoothly along with the video.
To conclude, maintaining synchronization on the Android platform requires accurate calculation of audio timestamps and synchronization of video playback. The appropriate techniques described above can be used to achieve it. With the boom of multimedia systems, platforms and devices for various applications, we envision that Audio-Video synchronization will become more important and achieve more autonomy.
At Softnautics, a MosChip company, we understand the importance of audio-video synchronization for every media solution/ application. Our team of media experts enable businesses to design and develop multimedia systems and solutions involving media infotainment systems, audio/video solutions, media streaming, camera-enabled applications, immersive solutions, and more on diverse architectures and platforms including multi-core ARM, DSP, GPUs, and FPGAs. Our multimedia engineering services are extended across industries ranging from Media & Entertainment, Automotive, Gaming, Consumer Electronics, Security, and Surveillance.
Read our success stories related to multimedia engineering to know more about our services.
Contact us at business@softnatics.com for any queries related to your solution or for consultancy.
Author: Suhas Jarande
Author: Vikas Patil l
Vikas Patil is an Associate Software Engineer at Softnautics, a MosChip company. He has more than 3 years’ experience in the development of Embedded system Software. He is keenly interested in building applications around BLE, RFID, IoT, GPS and Multimedia engineering and actively works on Android Audio HAL. In his free time, he enjoys cooking and working out at the gym.