Unveiling the Importance of Quality Assurance: A Comprehensive Guide to OTT Platforms Testing
Delve into the world of Over-the-Top (OTT) platforms testing with our comprehensive guide to the importance of quality assurance. From understanding the significance of testing in ensuring seamless user experiences to exploring the various aspects of OTT platforms testing, this article offers valuable insights and practical guidance for a broad readership, shedding light on the critical role of quality assurance in the digital streaming landscape.
Understanding Quality Assurance in OTT Platforms
Quality assurance (QA) is a crucial component of the software development lifecycle, ensuring that the end product meets the specified requirements and functions as intended. Unlike traditional software applications that reside solely on a user's device, OTT platforms are intricate ecosystems. They rely on a complex interplay of various subsystems working together flawlessly to deliver content. Content encoders prepare the video for streaming, CDNs ensure swift delivery, client devices like set-top boxes and smartphones render the content, and the central OTT platform (middleware) acts as the brain, orchestrating everything. Additionally, CRM systems manage customer information, billing systems handle transactions, and DRM (Digital Rights Management) safeguards content security. Each of these subsystems, if not functioning optimally, can introduce glitches or disruptions that can throw a wrench into a viewer's enjoyment.
QA in OTT platforms adopts a multi-pronged approach to identify and eliminate these potential roadblocks. It's a proactive strategy that meticulously examines every element within the OTT architecture to ensure the platform functions flawlessly across all intended use cases. specialists with QA expertise delve into the functionality of each subsystem, focusing on aspects like content encoders ensuring smooth video transcoding, video players effectively handling the incoming stream, and the CDN delivering the content efficiently with minimal buffering. They also scrutinize the central OTT platform (middleware) to guarantee features like user logins, search functionalities, and content recommendations operate seamlessly.
Furthermore, QA goes beyond the internal workings of the platform itself. Compatibility testing becomes paramount, ensuring the platform functions flawlessly across a wide range of devices, operating systems, and browsers. Users expect to enjoy their favorite shows and movies on any device they choose, and QA ensures this flexibility exists. Performance testing pushes the platform to its limits. Here, QA simulates various network conditions and user loads, identifying and rectifying any bottlenecks within the system that might cause buffering, lag, or even crashes. Security testing plays a vital role as well. QA specialists employ robust encryption protocols to safeguard user data and financial information across every step of the OTT ecosystem, from content encryption by the DRM system to secure transactions within the billing system.
In essence, QA in OTT platforms is the meticulous examination of every element within the intricate architecture, from content encoders to client devices and billing systems. Any glitches or issues in the platform can lead to user dissatisfaction and ultimately impact the platform's reputation and user retention.
Key Aspects of OTT Platforms Testing
Functionality Testing
Functionality testing involves assessing the various features and functionalities of the OTT platform to ensure that they work as intended. Here, testing delves deep into various subsystems to identify and eliminate potential roadblocks for users.
Encoders and Video Players
Content encoders are thoroughly examined to verify they can efficiently transcode videos into various bitrates and resolutions. This ensures smooth playback for users with different internet connection speeds. Video players on client devices are tested to confirm they can effectively handle the incoming stream, decoding the video and audio for flawless playback.
CDN
The Content Delivery Network (CDN) undergoes rigorous testing to assess its efficiency in delivering content. This involves simulating various network loads and geographic locations to identify any bottlenecks that might cause buffering or lag. Client devices, encompassing smartphones, tablets, set-top boxes, and smart TVs, are all meticulously examined. Functionality testing on these devices ensures core features like playback controls, search functionality, and content browsing work seamlessly across different operating systems and hardware configurations.
Middleware and CRM
The central OTT platform, also known as the middleware, is a critical component under scrutiny. Testing verifies user logins function flawlessly, content metadata and functions such as recommendations are accurate based on user history, and overall platform stability is maintained under various loads. The CRM system isn't exempt from testing. Here, the focus is on ensuring user data is stored securely and functions like account creation, profile management, and watch history are accurate.
DRM
Security is paramount, so the DRM system is rigorously tested. This ensures content is encrypted effectively, protecting copyrighted material from unauthorized access during streaming or offline viewing. The billing system also undergoes testing to verify secure transactions and accurate reflection of user purchases within their accounts.
Integration
Integration testing complements functionality testing by examining how these various subsystems interact with each other. This ensures a cohesive user experience. Login credentials (CRM), content recommendations (user history data), and secure payments (billing system) are all meticulously examined to guarantee they function seamlessly together.
Compatibility Testing
Compatibility testing is essential to ensure that the OTT platform is compatible with different architectures (subsystems), devices, operating systems, and web browsers. This testing ensures that users can access the platform from their preferred devices without encountering any compatibility issues. Unlike traditional software applications designed for a specific operating system, OTT platforms must function flawlessly on a wide range of devices and architectures. Here's a deeper dive into how compatibility testing ensures a frustration-free experience for users across various devices.
Content Encoders and Video Players
Here, the focus is on ensuring content encoders can efficiently transcode videos into formats compatible with the video players on various client devices. This ensures smooth playback regardless of the device's capabilities.
Client Devices
This encompasses the vast landscape of smartphones, tablets, set-top boxes, and smart TVs. Compatibility testing meticulously examines core functionalities like playback controls, search, content browsing, and user account management on each device type. The goal is to ensure these features work seamlessly across different operating systems (Android, iOS, etc.) and hardware configurations (varying processing power, memory, etc.).
Operating Systems
Compatibility testing delves into the intricacies of different operating systems. Playback performance, UI responsiveness, and resource utilization are all thoroughly examined across various OS versions (e.g., Android 10, iOS 15). This ensures smooth operation regardless of the user's specific operating system.
Browsers
While some platforms offer dedicated apps, web-based access is also common. Compatibility testing for web browsers verifies that the platform functions flawlessly across popular browsers like Chrome, Firefox, Safari, and Edge. This ensures users can access their favorite shows and movies via web application regardless of their preferred browser.
Screen Resolutions and Aspect Ratios
The modern viewing landscape encompasses diverse screen sizes and aspect ratios. Compatibility testing ensures optimal video and user interface rendering across various resolutions (e.g., 720p, 1080p, 4K) and aspect ratios (e.g., 16:9, 21:9). This guarantees an immersive viewing experience on any device, from a smartphone to a large-screen TV.
Payment Gateways
While functionality testing ensures the billing system works as intended, compatibility testing focuses on verifying smooth integration with various payment gateways. This guarantees users can utilize their preferred payment methods seamlessly across different devices and regions.
Performance Testing
Performance testing focuses on evaluating the speed, responsiveness, and stability of the OTT platform. It involves testing the platform's load capacity, response times, and overall performance under various conditions to identify and address any performance bottlenecks. Here's a deeper dive into how performance testing helps achieve a smooth and uninterrupted viewing experience.
Bitrate and Resolution Management
Performance testing assesses the platform's ability to adapt to different network conditions. This involves simulating various internet speeds (low bandwidth, high bandwidth) and testing how effectively the platform adjusts video bitrate and resolution to maintain smooth playback without buffering. The goal is to ensure a high-quality experience even for users with limited internet connectivity.
Startup Time and Content Loading Speeds
The time it takes for the platform to launch and content to load can significantly impact user experience. Performance testing examines startup times and content loading speeds across various devices and network conditions. This ensures users can quickly access their desired content without unnecessary waiting times.
Concurrent User Load
Performance testing simulates real-world scenarios where a multitude of users access the platform simultaneously. This involves stress testing the platform by simulating high volumes of concurrent users to identify any bottlenecks within the system that might cause performance degradation. The goal is to ensure the platform remains stable and delivers a smooth experience even during peak usage periods.
Content Delivery Scope
OTT platforms cater to users across the globe. Performance testing focuses on the effectiveness of the CDN (Content Delivery Network) in delivering content with minimal latency (delay). This involves simulating content delivery from various geographic locations and measuring the time it takes for content to reach users. The goal is to minimize latency and ensure a responsive viewing experience regardless of user location.
Client Device Performance
Performance testing extends beyond network conditions. It also examines how the platform performs on various device types (smartphones, tablets, set-top boxes, smart TVs) with different hardware configurations (processing power, memory). The goal is to ensure smooth playback and a responsive user interface even on less powerful devices.
Adaptive Bitrate Streaming (ABR)
Modern OTT platforms leverage adaptive bitrate streaming to deliver optimal video quality based on availability of bandwidth. Performance testing verifies the effectiveness of this mechanism, ensuring seamless switching between bitrates without interrupting playback or compromising video quality excessively.
Error Handling and Recovery
Unexpected errors can occur. Performance testing assesses the platform's ability to handle errors gracefully. This involves simulating various error scenarios (network outages, server issues) and measuring how quickly the platform recovers and resumes playback. The goal is to minimize disruptions to the viewing experience.
Network Connectivity
Not all users have access to high-speed internet. Compatibility testing considers various network conditions, simulating scenarios with limited bandwidth or unstable connections. The goal is to ensure the platform remains functional and delivers an acceptable viewing experience even under less-than-ideal network conditions. This is similar to ABR, but goes wider and evaluates the resilience of the whole OTT ecosystem to the changing network conditions.
Security Testing
Security testing is critical for OTT platforms to safeguard user data and ensure secure transactions. Here, the focus shifts from ensuring a smooth user experience to safeguarding the platform and its users from malicious attacks and vulnerabilities. This involves testing the platform for vulnerabilities, encryption protocols, and adherence to data protection regulations to mitigate the risk of security breaches.
Authentication and Authorization
This testing focuses on the robustness of mechanisms used to verify user identities and control access to various platform functionalities. Here, testers meticulously examine login processes, password strength enforcement, multi-factor authentication (MFA) implementation, and authorization controls that restrict access to sensitive data or features based on user roles. The goal is to ensure only authorized users can access the platform and their accounts remain secure.
Data Encryption
Security testing places a high emphasis on data protection. This involves examining the effectiveness of encryption algorithms used to safeguard sensitive user information like payment details, account credentials, and watch history. Testers verify that data is encrypted at rest (stored on servers) and in transit (transmitted between devices and servers) to prevent unauthorized access or interception.
Content Security (DRM)
Protecting copyrighted content is paramount. Security testing meticulously examines the Digital Rights Management (DRM) system implemented within the platform. This involves verifying the effectiveness of encryption methods used by DRM to safeguard content from unauthorized access during streaming or offline viewing. Testers also examine the platform's ability to prevent content sharing or screen recording, ensuring compliance with copyright laws and protecting content creators.
API Security
Modern OTT platforms rely on Application Programming Interfaces (APIs) to interact with various external services. Examining authentication procedures, authorization restrictions, and potential flaws that malicious actors might exploit are all part of security testing's investigation into the security of these APIs. The goal is to ensure secure communication between the platform and external services to prevent unauthorized data access or manipulation.
Penetration Testing
Penetration testing simulates real-world attack scenarios, where ethical hackers attempt to exploit vulnerabilities within the platform. This proactive approach helps identify weaknesses in the system's defenses before malicious actors can discover them. By simulating various hacking techniques, testers can uncover potential security flaws and ensure the platform is prepared to withstand cyberattacks.
Security Patch Management
Security is an ongoing process. Security testing extends to verifying the platform's ability to receive and implement security patches promptly. This ensures the platform remains up-to-date with the latest security fixes, mitigating newly discovered vulnerabilities and protecting user data effectively.
Regular Security Audits
Security testing is not a one-time event. Regular security audits are crucial for maintaining a robust security posture. These audits involve a comprehensive assessment of the platform's security controls, identifying potential weaknesses, and recommending improvements.
Ensuring Seamless User Experiences
The impact of quality assurance on user experiences within OTT platforms is profound. By conducting thorough testing and optimization, QA teams can identify and rectify any issues that could affect the user experience, such as buffering delays, playback errors, or navigation glitches. This ultimately contributes to creating a seamless and enjoyable streaming experience for users.
Usability - UI and UX Testing
QA doesn't solely focus on technical, but also on usability aspects. Usability testing involves observing real users interacting with the platform. This helps identify any pain points or areas of confusion within the user interface (UI) or user experience (UX). Simplifying navigation, optimizing information architecture, and ensuring intuitive controls are some of the most important factors of user satisfaction.
Accessibility Testing
QA champions inclusivity by ensuring the platform is accessible to users with disabilities. This involves testing for compatibility with assistive technologies and ensuring features like closed captions and audio descriptions are functional, allowing everyone to enjoy the content.
Localization Testing
Catering to a global audience necessitates localization testing. QA verifies that the platform interface and content are accurately translated and culturally appropriate for different regions, fostering an enjoyable user experience for international viewers.
In addition to internal testing processes, user feedback plays a pivotal role in enhancing the quality of OTT platforms. By gathering and analyzing user feedback, QA teams can gain valuable insights into user preferences, pain points, and expectations, which can then be used to inform testing and optimization efforts.
Best Practices for OTT Platforms Testing
Test Automation
Test automation plays a crucial role in OTT platforms testing, allowing for the efficient execution of repetitive test cases and scenarios. By automating regression tests, UI testing, and performance testing, QA teams can optimize the testing process and ensure comprehensive test coverage. Here's a deeper dive into how test automation empowers efficient and comprehensive testing.
Reduced Testing Time and Effort
Automating repetitive test cases significantly reduces the time and resources required for manual testing. This allows testers to focus on more complex scenarios and exploratory testing, while automated scripts handle the regression testing of core functionalities.
Improved Test Coverage
Test automation empowers the creation of a wider range of test cases compared to manual testing. Automated scripts can meticulously examine various functionalities under diverse conditions, increasing overall test coverage and identifying potential issues that might be missed during manual testing.
Increased Consistency and Repeatability
Manual testing can be prone to human error and inconsistencies. Automated scripts, however, execute test cases consistently every time they are run, ensuring reliable and repeatable testing results. This eliminates the variability that can occur with manual testing and provides a more objective assessment of platform functionality.
Faster Feedback and Bug Detection
Automated tests can be integrated into the development process, allowing for continuous integration and continuous delivery (CI/CD) pipelines. This enables automated testing to be run after every code change, providing faster feedback on potential bugs or regressions introduced during development. Early detection of issues translates to quicker bug fixes and a more stable platform release.
Reduced Maintenance Costs
While there is an initial investment in creating and maintaining automated test scripts, the long-term benefits outweigh the costs. Automating repetitive tasks reduces the need for manual testing efforts, leading to lower overall testing costs over time.
Improved Test Scalability
As OTT platforms evolve and functionalities grow, automated tests can be easily scaled to accommodate the increasing complexity. New test cases can be readily added to existing automation frameworks, ensuring comprehensive testing coverage even as the platform expands.
Approaches to Test Automation
- UI Automation: This approach automates user interactions with the platform's user interface. Test automation tools like Selenium can be used to simulate user actions such as login, navigation, content selection, and playback controls. This helps verify the functionality and responsiveness of the UI across various devices and browsers.
- API Automation: Testing application programming interfaces (APIs) is crucial for OTT platforms that interact with external services. Tools like Postman can be used to automate API calls, verify data exchange accuracy, and ensure seamless integration with third-party services.
- Performance Testing Automation: Performance testing is critical for ensuring a smooth user experience. Tools like JMeter can be used to automate performance tests that measure factors such as loading times, bitrate management, and buffering behavior under various network conditions.
- Mobile Automation: With the growing popularity of mobile OTT apps and viewing, automating testing on various mobile phones and other devices is essential. Frameworks like Appium can be used to create automated tests that simulate user interactions on smartphones and tablets, ensuring a seamless experience across different mobile platforms.
Continuous Testing
In the dynamic landscape of OTT platforms, continuous testing is essential to keep pace with frequent updates, new feature releases, and evolving user demands. Implementing continuous testing practices enables QA teams to detect and address issues in real time, ensuring the ongoing quality and performance of the platform.
Continuous testing should integrate seamlessly into the development process. It's not a one-time event at the end of development, but rather a series of automated tests that are continuously executed throughout the development pipeline. This allows for early detection and rectification of bugs, regressions, and performance issues, preventing them from snowballing into larger problems later in the development cycle.
Here's a deeper dive into the core principles of continuous testing and the value it brings to OTT platforms:
- Shift Left Testing: Continuous testing embodies the concept of "shift left testing." This means identifying and fixing issues as early as possible in the development process, ideally during the design and coding phases. Early detection minimizes the time and effort required to fix bugs compared to catching them later in the development cycle or even after release.
- Integration with CI/CD Pipelines: Continuous testing thrives within CI/CD (Continuous Integration and Continuous Delivery) pipelines. Code changes are automatically integrated into a central repository, triggering a series of automated tests. These tests assess code quality, functionality, performance, and security, providing immediate feedback to developers. This rapid feedback loop enables them to identify and address issues swiftly, before they merge the code into the main branch.
- Focus on Automated Testing: Continuous testing heavily relies on automation. Repetitive and regression test cases are automated, freeing up QA teams to focus on more exploratory testing and edge-case scenarios. Automated tests can be executed frequently, even after every code change, ensuring comprehensive and consistent testing coverage.
- Improved Release Quality and Faster Time to Market: By proactively identifying and resolving issues early in the development cycle, continuous testing leads to higher quality software releases. This translates to fewer bugs and a more stable platform for users. Additionally, automated testing's quick feedback loop enables shorter development cycles and a faster time to market for new features and functionalities.
- Reduced Risks and Costs: Continuous testing helps mitigate the risks associated with introducing new features or code changes. By catching issues early, the potential for costly bug fixes and delays in the development process is minimized. Furthermore, the automation of repetitive tasks reduces the overall testing time and resource requirements, leading to long-term cost savings.
- Collaboration between Development and Testing Teams: Effective collaboration between development and testing teams is vital for successful OTT platforms testing. Automated test results and bug reports provide developers with immediate feedback on any potential issues within their code. This open communication and early problem identification improve the development process and expedite issue resolution.
User Acceptance Testing (UAT)
User Acceptance Testing (UAT) acts as a final hurdle before an OTT platform is unleashed to the public. It's a critical stage where the OTT operator, who receives the platform from the vendor, meticulously assesses whether it meets the predefined requirements outlined in the contract between both parties. Here's a breakdown of the UAT process, along with valuable tips for planning and execution, highlighting the benefits it offers to the OTT operator.
The UAT Process
UAT typically involves a dedicated team from the OTT operator that possesses a comprehensive understanding of the target audience, platform functionalities, and business goals. This team works collaboratively with the vendor to execute a series of tests designed to evaluate the platform against the agreed-upon specifications. Here's a general outline of the process:
- Test Plan Development: A detailed UAT test plan is created, outlining the functionalities to be tested, the acceptance criteria for each test case, and the roles and responsibilities of team members.
- Test Case Execution: The UAT team systematically executes the test cases, simulating real-world user scenarios and workflows. This may involve testing features like user registration, content search, playback across different devices, payment processing, and customer support functionalities.
- Defect Logging and Tracking: Any issues encountered during testing are meticulously documented, including detailed descriptions, screenshots, and potential workarounds. A bug tracking system is used to manage and monitor these defects until they are resolved by the vendor.
- Re-Testing and Sign-Off: To ensure proper resolution after the vendor has fixed any defects, the UAT team re-tests the affected functionalities. Upon successful completion of all test cases and resolution of critical defects, the UAT team can sign off on the platform, signifying its readiness for launch.
Planning and Execution Tips
- Early Involvement: Involve the UAT team early in the deployment process. This allows them to gain familiarity with the platform and collaborate with the vendor to ensure the test plan aligns with user needs and business objectives.
- Diverse Testing Scenarios: Design test cases that encompass a wide range of user behaviors and network conditions. This helps identify potential issues that might not be apparent under ideal circumstances.
- Clear Communication: Maintain clear communication channels between the UAT team, vendor, and development team. Promptly report any roadblocks or critical issues to ensure timely resolution before launch.
- User-Centric Approach: Remember, the ultimate goal is to deliver a seamless user experience. Tailor test cases to mirror real-world user interactions and identify any areas that might cause confusion or frustration.
Desirable Outcomes
UAT serves as a vital safeguard, ensuring a smooth launch and a positive user experience from day one. It is a fairly complex and tedious process that results in significant benefits for the OTT operators, if it is structured and executed well:
- Reduced Launch Risks: UAT helps identify and rectify bugs before the platform goes live, preventing potential issues that could disrupt launch or negatively impact user experience.
- Improved User Satisfaction: A platform that functions flawlessly from the user's perspective translates to a higher level of satisfaction and user retention.
- Reduced Post-Launch Support Costs: Catching and fixing issues during UAT minimizes the need for post-launch bug fixes and support requests, leading to cost savings in the long run.
- Enhanced Brand Reputation: A polished and well-functioning platform reflects positively on the OTT operator's brand, boosting trust and loyalty among viewers.
A Look Inside UniqCast's QA Process by Maja Kurjak
As Head of Software Development and QA at UniqCast, I'm thrilled to share a glimpse into the heart of what keeps our OTT platform running smoothly – our meticulous Quality Assurance (QA) process. At UniqCast, we understand that a seamless streaming experience is paramount to user satisfaction. Our dedicated QA team plays a vital role in achieving this goal. They are the silent heroes behind the scenes, meticulously testing every aspect of the platform to ensure exceptional performance, functionality, and user-friendliness. Here's a closer look at the practical intricacies of our QA process.
Development and Unit Testing
- Unit Testing by Developers: Our QA process begins at the developer level with the creation and execution of unit tests. Developers thoroughly test all backend services in a controlled development environment immediately after coding.
- Frontend Unit Testing: On the frontend, developers are also responsible for conducting unit tests. They ensure that the acceptance criteria outlined in the product user stories are met, maintaining a high standard from the outset.
Backend and Performance Testing
- Postman and Non-Functional Testing: In addition to unit tests, backend services undergo testing with Postman and non-functional performance testing. We create dedicated testing services to evaluate performance metrics comprehensively.
QA Team Testing
- Manual Feature and Bug Testing: Our QA team then steps in to rigorously test new features and bug fixes. This includes both backend tasks and end-to-end testing for frontend tasks.
- Test Case Creation: For every frontend feature and externally reported bug, the QA team develops detailed test cases. This approach is designed to minimize the risk of regression issues in future releases.
Automated Testing
- Automated Test Execution: After manual testing, we employ automated tests to further ensure product quality. These automated tests are categorized into three types:
- Feature-Based Tests: Target specific new features.
- Full Regression Tests: Ensure that new changes do not negatively impact existing functionality.
- Smoke Tests: Quickly verify the most crucial aspects of the application.
Environment-Based Testing
- Testing in Multiple Environments: Applications are tested not only in our testing environment but also in pre-production and, when possible, in the production environments of our Uniqcast clients. This multi-environment approach helps us identify and resolve issues that might only surface under real-world conditions.
Final Validation by Support Team
- Support Team Validation: As an additional layer of quality control, our support team validates Release Candidate (RC) versions in the production environment before official releases. This validation simulates user acceptance tests and further minimizes the risk of regression issues.
This multi-layered QA process, with its collaboration between developers and the dedicated QA team, ensures that the UniqCast platform is delivered with exceptional quality. We meticulously test at every stage, from unit testing by developers to final validation by the support team. All this effort minimizes the risk of issues and ensures a smooth and enjoyable streaming experience for our users. This commitment to quality is at the core of everything we do at UniqCast, and it's what allows us to deliver a best-in-class OTT platform for our clients and their audiences.
The Role of Quality Assurance in the Digital Streaming Landscape
The impact of quality assurance extends beyond ensuring the technical functionality of OTT platforms. It directly influences customer retention by providing users with a reliable and enjoyable streaming experience. Platforms that consistently deliver high-quality content and seamless user experiences are more likely to retain satisfied customers and attract new ones.
Moreover, quality assurance plays a crucial role in maintaining the brand reputation of OTT platforms. A platform that is plagued with technical issues, security vulnerabilities, or performance shortcomings can suffer reputational damage, leading to a loss of trust among users and stakeholders.
Conclusion
In conclusion, the importance of quality assurance in OTT platforms testing cannot be overstated. From ensuring seamless user experiences to maintaining brand reputation and customer retention, quality assurance is a cornerstone of the digital streaming landscape. By adhering to best practices, leveraging testing and optimization tools, and prioritizing user feedback, OTT platforms can deliver high-quality content and services that meet the expectations of today's discerning audience.
In the fast-paced and competitive OTT industry, quality assurance remains a critical factor in driving the success and sustainability of digital streaming platforms. As technology continues to evolve and user expectations continue to rise, the role of quality assurance in OTT platforms testing will only become more pivotal in shaping the future of digital entertainment.
More on IPTV/OTT Platform Integration
Remember, this text dives deeper into the OTT testing methodologies mentioned in one of our previous articles on IPTV/OTT platform integration, providing a practical guide for creating a comprehensive testing plan. It also highlights popular OTT testing tools to streamline the process.
Looking for more in-depth coverage? Our next article will explore advanced performance optimization strategies for OTT platforms, empowering you to fine-tune your platform for maximum efficiency and a truly exceptional viewing experience.