Software testing plays a vital role in ensuring the quality and reliability of software products. Crowdsourced testing platforms like Testlio and Test IO allow freelance testers to contribute their skills to various testing projects.
This article compares the differences between the Testlio and Test IO platforms, specifically focusing on their test processes, bug types, severity of functional bugs, attachments, payouts, and test communication.
Testlio follows a structured test process linked to workspaces. Testers are assigned to specific workspaces to conduct a wide variety of testing, such as exploratory testing, payment testing, functional regression testing, localisation testing, accessibility testing, usability testing, live stream testing, and instrumentation SDK testing, in which they perform test cases from top to bottom.
If the test case step fails, testers must submit bugs or issues; if the bug was already submitted, they should execute a reproduction.
Testers can find duplicated bugs when submitting an issue by looking at other testers' reports or searching within the ❝Issue browser❞. If the title contains a minimum of 30 characters, testers will automatically get suggestions of similar issues already reported.
The Testing Team selects testers based on their profiles and device information to match the workspace requirements and invites them via email in two stages. First, the invitation is to the workspace; the second stage is to a run where the time, device and schedule to test is given.
Depending on the workspace, testers may or may not get invitations to a run; everything depends on the workspace's needs.
Testers must reply to the run invitation to be invited to the run. Letting invitations expire is considered a severe issue.
After finishing the assigned time to test, testers need to answer a feedback survey.
After raising an issue, the TL can approve it, request additional information, or
Hence, testers wait for others to raise issues and then make a reproduction because submitting an issue might worsen the tester's rating/score if it’s closed (rejected); in contrast, if the bug is accepted, it doesn’t bring any benefits.
Test IO employs an agile test process that involves test planning, execution, and continuous feedback. Testers participate in test cycles that match their language, devices and location. We emphasise real-time and manual exploratory testing, enabling testers to discover and report bugs on the fly.
To start contributing to a test cycle, testers need to start test sessions in advance and acknowledge that the instructions of the features in scope were read and understood.
To identify duplicates, when submitting a bug, the ❝Similar bugs❞ list on the right-hand side of the bug form will show you the list of already submitted bugs in the current test, and you can also search and filter the bugs and Known Bugs list to do so.
If a bug report gets rejected, the tester's scores are negatively affected based on the type of rejection received.
Testers can dispute bugs rejected by team leaders. Once a report is disputed, it gets locked for review by the Bug Dispute Team.
Placeholder reports are highly forbidden.
Based on the devices registered in the Tester's Profile and other information, the system selects one of the devices and invites you to participate in the test. Depending on the seats left for each cycle, you may be unable to select a device other than the one the system chose. You cannot switch to another device once you have been invited to a device. Therefore, you can only submit bugs found on this device.
When required, the Testlio bug form or issue form presents the following structure:
The severity, feature and labels are dropdown lists to select options from; the other sections, minus the attachment, are templates testers must complete manually. This applies to every failed step, submitted bug or executed reproduction.
Every issue must have a prefix of the section where the issue occurs before the title if otherwise specified.
Test IO's bug form, on the other hand, simplifies tester documentation with the following structure:
You don't have to follow a specific format to construct a title. However, you must respond to what happened, where the bug occurred and when it triggered.
On our bug form, you don't have to add the step number; our form already provides it. You can drag them around and rearrange them at will.
Except for the browser while website testing, the device's information is retrieved from the device you selected when accepting to participate in a test cycle, which cannot be changed afterwards.
Types of Bugs
Testlio generally classifies issues as:
Functional bus required two attachments for documentation.
Localisation testing requires verification of the UI issue and the content itself.
Test IO categorises bugs as:
We don't perform security tests.
On the other hand, during performance test cycles, which are run as per demand, endless loading is a performance issue, and they are reported as such; therefore, measuring the internet speed and .har files is part of the attachments needed to attach to the report.
The Severity of Functional Bugs
Testlio bug severity for functional bugs is:
Testers decide which severity applies better to the issue based on a brief explanation of each severity.
At Test IO, we offer specific scenarios to guide you through the correct selection of the severity; these are only three:
The scenarios we provide will help analyse the issue in different ways, such as considering the functionality compromised and the bug's potential impact on end-users. For instance, showstopper bugs are critical, while if there is an easy and intuitive workaround like reloading the page, the correct severity will be low.
The Attachments in Bug Report
Testlio allows testers to attach relevant files to their bug reports, such as screenshots, screencast, and log files (crash, console, and sometimes Charles log files).
Log files are mandatory, and functional bugs must provide two different attachments.
These attachments must have a meaningful name to help the developer who downloads it diagnose the issue.
While a screenshot or a screencast can easily document visual bugs, functional bugs require two attachments per submitted issue.
All issues must have a screenshot of the page (even if the tester has attached a screencast), and a text file with the HTML code snippet of the issue must be added. Sometimes, the information in the ❝Attachments in Bug Report❞ section is enough.
Test IO testers can include attachments, such as screenshots, screencasts, and crash logs.
For performance testing, .har files are required.
Testlio fosters communication between testers and test leads via the platform's Chat or the RocketChat channel for the specific workspace; however, testers are added to this Chat only if they've passed a certification test or it's required for the workspace.
Email is used to send testers improvement requests.
Test IO emphasises real-time communication, enabling testers to collaborate and discuss issues through test chat, bug report comments, Discord server, and email platforms between testers, team leaders (TL), Customer Success Managers (CSMs), and customers. Testers can ask questions, seek clarifications, and receive instant feedback from the abovementioned stakeholders, as these can request information from testers' tasks.
Testlio pays testers by the hour, not by the bug.
Test IO payout depends on the type of task performed. There are tasks directly related to Test Cycles. In contrast, others, like Reproductions, Bug Fix Confirmation and Bug Report Confirmation, can be performed without joining the test cycle the report belongs to.
Paid Tasks & Bonuses
At Testlio, freelance testers get paid according to the hours assigned to execute specific tasks within a run in a particular project.
At Test IO, freelance testers can perform these paid tasks and receive the following bonuses:
Instead of waiting for the client to accept or reject your work at Test IO, you're being paid as soon as the Team Leader takes your bug or execution.
In conclusion, each platform has its unique approach to test processes, bug reporting, bug severity, communication, and payment structures.
Testlio and Test IO differ in several aspects. Testlio follows a structured test process, assigning testers to workspaces for various testing types and allowing for bug submissions and disputes. Test IO employs an agile process with real-time exploratory testing, requiring testers to acknowledge instructions for any test cycle they're eligible to join based on their languages, devices and location. Bug forms in Testlio require detailed information that testers must provide, while Test IO simplifies the bug form. Testers can attach various files in Testlio, with mandatory log files for functional bugs, whereas Test IO offers general attachment guidelines for screenshots and screencasts. Bug severity varies between three levels in Testlio, determined by testers, while Test IO provides specific scenarios for bug severity selection. Communication differs, with Testlio using chats and emails, while Test IO emphasises real-time communication through various channels. Payment structures also vary, as Testlio pays testers hourly, and Test IO's payout depends on the many task types testers can perform on our platform and bug type and severity.
Testers are the backbone of software quality and end-user satisfaction, and their tireless efforts ensure that software is reliable and seamless across all platforms.