Invitation System Update

Here’s what’s new —

Device-specific Invitations:

We will now send out invitations for a specific device or OS you have listed in your profile.
This means if we have a test where some requested devices are uncommon, you will be invited for one of those devices. For example, we might invite you to test on a Samsung Galaxy S6. When you accept this invitation, you must use this device to test, even if there are other devices in the test you also might own.

This also applies to operating systems. If we invite you to test using a certain OS, you can use any device you have operating on this OS, but you must test on the OS for which you are invited. Reproductions can be done with any device in scope.

Tester Segmentation:

The system will be classifying testers based on performance. If you are really good at rapid tests, you may be marked as a “rapid test tester”. What impact does this have? We reserve extra invitations for testers who are parts of these performance-based segments

If you perform better in certain test types, you will have a better chance at being invited to these test types.
If you are not part of one of these specialist segments, you will still receive invitations to those tests.
Greenhorn testers (less than 50 bugs reported) have their own segment and will not be rated on their performance initially. As soon as you have reported 51 or more bugs our system will classify you based on all bugs you reported. So make sure you’re always doing your best work, even as a Greenhorn tester.

Test Distribution:

We consider test distribution updates the most important change.
Tests will be distributed equally among testers within one segment.
If you test as well as another tester, and your device coverage is the same, both of you should get about the same number of invites.
Many of you have told us lately that you are not getting as many tests as other testers. This issue should now be fixed. If you think this is not the case, please let us know!

The Leaderboards:

Points you score for reporting bugs and other activities do not impact the invitations you receive anymore.
We will keep the leaderboards active so you can compare yourself with other testers, if you want to.
But remember, rejected bugs will influence your performance rating and you are less likely to be in a top segment if you get a lot of bugs rejected.

Coming up —

Tester Profile:

Currently, you cannot see your own performance indicators. This is something we would like to change in the future.
To do so we are in the planning phase of developing a “profile page” where you can see things like “number of bugs posted,” “bug acceptance rate,” and other performance indicators.

New vs. experienced testers

Here at test IO we call our new testers “Greenhorns” and you will be a Greenhorn for your first 50 bugs.
After you posted 50 bugs we see you as experienced enough to know the basics of bug reporting here at test IO and you will lose your newbie status and all the advantages which come with it.
For a Greenhorn, team leaders will more often request further information if you forgot important steps or the quality of the bug is not good enough.

For testers which are not Greenhorns any more, the rules change a lot. Your bugs can be straight rejected for the following reasons:

  • Multiple grammar mistakes (not just a typo)
  • Not understandable description or results
  • Missing important steps in the description
  • Wrongly selected feature
  • Unclear bug title
  • Clearly not reading/understanding the test description
  • Incomplete evidence
    example: Attachments show a lot of stuff, but not what is important
    example: missing crash logs

So as a tester you should make sure every bug you submit fits the standards of quality and is as good as it can be.

test IO tester ms_frodo shares his 5 top tips to crowd testing success!

1st tip: Thoroughly read test instructions before starting testing
Often some testers start testing, without reading all the test instructions. As a result many bugs raised by them are rejected with one of the following reasons: the most popular “bug is out of scope” and two less popular “this is not a bug” or “this bug is known”. This tip will allow you to escape rejected bugs, as result, keeping your tester ranking and not wasting time for nothing.

2nd tip: Increase quality of your bug report
When you start filling out the bug report, make sure that you include all important information needed to reproduce your bug. Improve your own English level (if it is not already very good) for writing bug reports grammatically correct and understandable for reviewer and customer. Use rule of three ‘W’ (What, Where, When) in your bug title. This tip will allow you to write quality bugs, as result again escaping rejected bugs. And one more plus, the reviewer and customer will enjoy working with you and your bug reports.

3rd tip: Ensure that your bug isn’t a duplicate
Before raising a bug, check the bug list and ensure that your bug isn’t a duplicate. Sometimes I firstly raise the bug and immediately after I check the bug list to ensure that my bug isn’t a duplicate. Most rejected bugs are rejected for the reason “this bug is a duplicate..”. This tip is very simple, but many testers forget about it and as a result, they have many rejected bugs.

4th tip: Check the full functionality of the app during testing
When you start to execute testing of the app, ensure that you checked all features inside this app. Sometimes, high or critical bugs are located in the most prominent places, I mean features like “upload button doesn’t work” or “unable to sign up via facebook network”. It’s not necessary to go to the deep into the app, in order to find some serious bug. You need to test thoroughly all app features to ensure they work correctly and find the most hidden bugs.

5th tip: Go hand in hand with the test platform
What do I mean? I mean that you should always read and review latest news and updates of the testing platform. Usually testing platforms update their rules of testing and your job can suffer, if you are not aware of the latest updates of your testing platform. Also, if you are a novice it’ll be useful for you to read articles, which are present on the test platform, because there you can learn a lot of useful things, which will help you in your job.

Keep in mind these rules and they will certainly help to improve your work.

Relevant bugs are valuable bugs!

Put yourself in the shoes of a developer or product owner, you have run a crowd test and there are lots of bugs for you to work on and fix. Then you start to work through the bugs and realise that some of them are just not relevant to you, this may be due to the environment, the behaviour to trigger or multiple other reasons. The result, the irrelevant bugs are rejected.

So what can you do to ensure you are reporting relevant bugs?

  1.  Step back into the shoes of the developer and try to understand what they are looking for. The test description and features should give you a very good idea of what is relevant.
  2. Consider the target market of the app or website. If the app is designed for the Russian market then chances are nobody will be interested in a bug relating to a content issue when you change your phone settings to Italian.
  3. If you have to follow steps a real user would never follow in typical usage, then there is no real value provided in your report and it would be an edge case. Edge case bugs are very likely to be rejected. If the bug is never going to impact a customer’s users, they most likely don’t care about it.



Insider insights: 5 Golden rules to get your bugs accepted.

It’s the one question we are asked more than any other, “How can I get more of my bugs accepted?” Luckily we have experience of nearly half a million bugs reports and hundreds of customers, so we have a pretty good idea of exactly what is required! We believe there are 5 golden rules to follow to maximise your bug acceptance and we thought it was about time we shared them with you.


It might sound obvious but without reading the feature descriptions, instructions and test chat fully, you will not know exactly what the customer is looking for. Even if you find bugs, they may not be the ones they are currently interested in, so check what they’re asking for in the test.


If you can’t reproduce your bug there is very little chance the customer will be able to and nobody is interested in bugs which only happen once!


Something we see often are reports where something clearly goes wrong but the behaviour to trigger this was pretty strange and does not reflect the target user behaviour. Put yourself in the customer’s shoes, you are only going to be interested in bugs which your actual customers are likely to encounter, not ones that happen when you do some random sequence of events. On this point it is always worth considering who the target audience for the product will be.


For most tests there will be a known bug list of existing bugs so firstly make sure your bug is not on the list, also don’t forget to check the bugs already reported by other testers in the current test. When you are typing your bug title you will be shown possible similar bugs, so make sure you keep a close eye on this as well as being aware of what is on the list in general.


It’s another fairly obvious one but if your report and evidence is not clear and well written, the customer may not understand your bug immediately and as a result, reject it. You can find guidance on quality here and specifically on your title here.

Follow these 5 golden rules to high acceptance rates of your bugs.

Quantity is not the key to success as a tester! Quality and high value bugs are!

Many testers think – quantity is the key to success as a tester.

“Delivering” bugs like a factory is of course one way to be successful at test IO but you will face several issues on your path that might be worth thinking about, before handing in so many bugs.

Here is a list of a few important points for testing with test IO:

  1. Scope of a testcycle
    >Make sure you read the scope of the test! If you accidentally miss the scope of a testcycle – your bugs will be rejected – sometimes many of them. The teamleaders will review your bug within a certain timeframe – not immediately.So before reporting many bugs – report the most important bugs to see if your understanding of the scope is correct.
  2. Duplicates within your own bugs
    > With many bugs, your chance to create duplicates rises.
    If you’re sure this bug is something different than your last reports – you’re good to go.If you report very similar bugs, the chance of rejections is pretty high – consider this before you invest your precious time.
  3. Tester Team
    > You’re part of a team while the test is running. That means, everything you do might affect the others. If you submit a trillion bugs in every testcycle, other testers may stop reporting bugs. If your bugs are then rejected it will result in a poor quality test, so remember only report bugs your are confident are new and valid!
    Customers are highly interested to receive reports from different testers, devices, browsers, set-up’s etc. So use the communication tool to talk to each other and reproduce each others bugs on different devices.

    You should concentrate on your most important or high value bugs and try to cover as many testcycles as possible.

  4. Known Bugs
    > If you’ve tested with us a few times, you may have already seen known bugs here and there.You might already have bugs rejected as known and if it was not listed on the known bug list this is OK (you still get paid), but don’t forget to check the known list before submitting a bug.

    BUT consider that our customers are not really interested to see the same bug over and over again.

    We will try to improve the testcycle description – but you can help us and the customer to avoid “Copy & Paste” reports and concentrate on new, not yet known bugs.

Practical tips:

  • concentrate on high value bugs (high & critical) rather than many low bugs
  • try to cover as many testcycles as possible
  • avoid many similar bug reports in the same testcycle
  • quality of bug reports is the most important thing – if you can report in a good quality, we and our customers will appreciate and reward it 🙂