Best Practices
We gathered some important information so you can understand better the particularities of testing with streaming devices.
Make sure you are testing in a calm and silent environment. The audio while streaming is really important and a lot of bugs can be found if you pay attention to it. Also, remember to record the sound if you find any bug related to it.
Trick play while streaming is your best friend to find bugs. That means the actions you usually do when you are watching a video (such as pausing, playing, changing the subtitles, fast forwarding, rewinding, scrubbing the progress bar, and other available actions depending on your device). It's always important to mimic regular user behavior to find the most relevant bugs for our customers.
Avoid forced behaviors while testing using streaming devices. Fast-paced clicking on the remote buttons for example is not a regular action that users are expected to do. This action will end up triggering unexpected behaviors on the device that are usually not relevant to our customers.
Always have a reliable VPN service to use while testing. Most of the time, customers will have to restrict their access to the available content on the apps to certain locations due to legal/contract reasons. So even if you are testing a staging app, a VPN to a certain country might be required to see the content correctly on the customer apps. So before starting the test session, it's good to search for a good VPN service that works with your device.
Add all of your available streaming devices to your profile. The more devices you have added to your profile, you will have the higher chance of getting an invitation to test using one of them will be higher. So if you have easy access to one of the streaming devices (either if it's yours or if it belongs to another house member who authorizes you to use it for testing purposes), you can always add it to your profile.
But remember, sharing customer information with others is not allowed in any of the Test IO cycles, so please only borrow a device if you can test it in a safe environment.
Always remove the testing app and credentials (if given) from your device once you've finished testing. Sometimes our customers will provide you with premium accounts to test the device as a regular paid user would. However, the credentials are meant to be used only while testing and should not be used for personal purposes. If you are caught using the app outside the test window or sharing the credentials with other people, you will be banned from our platform.
Keep in mind the test scope and feature descriptions. Streaming app developers might be interested in exploratory testing covering the entire app or only specific features and areas. Such information will usually be clear in the test description, but sometimes it can be hard to identify the correct areas that are within the scope of the cycle. If you find any problems or if you are not sure about something, please ask for the TL's help using the Chat tab on your cycle instead of making assumptions.
The content of the app (the video assets) may not be controlled by our customers. Sometimes, streaming app developers are only focusing on the app itself and the films/tv series/shows available list is controlled by another team (or even other companies). In such situations, the customer devs might not be able to fix any wrong content (e.g. missing a series season or a movie with an incorrect description). So to avoid such bug reports, the Content bug severity may be disabled in your cycle. However, even if the Content bugs are allowed, we recommend you check the instructions to see if there are any notes about which Content bugs can be reported.
Relevant Bug Examples
While testing a Streaming app, the most important thing is to behave as a real end user and use real test scenarios. For example: If you open the Live TV screen, switch channels, and experience an app crash, this is a real bug, which is important for the customer, and will be forwarded. But if you open the Live TV screen, and then press buttons to switch the channel 10 times in 4 seconds, without waiting for screens to load, and you are basically forcing the app to make a bug or to crash, this is generally not relevant, and usually such bugs will be rejected as Edge case bugs. To help and inspire you to find bugs, here are some examples of issues that our customers would be interested in:
The subtitles are not in sync with the video or don't work at all when available
An error is shown when a video is played
The app crashes when you perform any regular action (e.g. pausing a video, pressing the back button on the remote, fast forwarding the playback)
Video is not added to the favorite list
Video is not added to the "recently played" list
Ads won't play correctly
NOTE: Keep in mind that you should report only bugs that are covered by the Features in the given tests. If the bug is not covered by any of the Features in the test, it will be rejected as Out of scope.