Quick check
Check a few messages of a session yourself, read the result in plain words and write your corrections into the session.
Quick check
A quick check takes a few messages of a session at random. You look at each one, confirm what is right and correct what is wrong. At the end you see how many messages had errors and how many there probably are in the whole session, and you decide whether to write your corrections into the session.
1. Choose how many messages
Press Check quality on the session page, or New check in the Quality section and choose the session. The screen asks one thing: how many messages to check.
- Choose 10, 20, 50 or all. By default it is 20, or the whole session if it has fewer messages.
- The messages are taken at random from the session. If you came from the session page with filters, only the messages that match them are used.
- The line under the choice says what the number gives, for example “20 messages: the result is accurate to about ±22%”. It means that the share of errors you find in 20 messages can differ from the share in the whole session by up to about 22 percentage points either way. The more messages you check, the smaller this number.
- Start checking opens the first message at once.
Nothing in the session changes while you check.
2. Go through the messages
Each message is on the left, and its sentiment, topics and author are on the right, as they are marked now. The line above them says whose answer it is: “Marked by the model”, or “Marked now · edited by you” if you edited this message before, or the name of the person who edited it. If you already answered this message in this check, it says “You corrected this message” or “You confirmed it is correct”.
- Correct and next (Space): the marks are right, go on.
- Change the sentiment, the topics or the author, then press Save correction and next. It is the same Space: once you changed something, Space saves the correction instead of marking it correct. The message in the session does not change yet.
- Skip (X): you are not sure. The message is left out of the result, and you can come back to it later.
- Previous and Next (← and →) move between messages. If you changed something, the correction is saved when you leave the message; a message you did not touch stays unchecked.
- If you like, add an error category (E): “Ambiguous instruction”, “Context not taken into account”, “Sentiment misread”, “Topics confused” or “Other”. Or a comment (M). These are notes for you; they do not change the result.
You can return to any message you already checked and change your answer while the check is open. You can stop at any moment: your answers are kept. To go on, press Check quality on the session page, or open the check from the list in the Quality section and press Continue. On a phone the buttons stay at the bottom of the screen and the key hints are hidden.
3. “Done”
After the last message you see Done: how many answers you marked correct, how many you corrected and how many you skipped. To the result (or Space) opens the result. Back to the messages returns to them.
4. Read the result
The result is one page. At the top are three cells: the sample (how many messages of how many in the session), the messages (how many are checked, how many skipped) and the result (the share of errors). While messages are left, the main button is Continue with the number of messages left.
When every message is checked or skipped, the page shows the result in plain words, for example: “Errors in 3 of 20 messages (15%). Over the whole session that is about 5 to 36%: the sample is small.”
- A message counts as an error when at least one of its marks was wrong: sentiment, topics or author.
- The first part is what you found. The second part is the range where the share of errors of the whole session most likely is (with 95% confidence).
- The fewer messages you check, the wider the range. “The sample is small” appears when fewer than 30 messages are checked. To narrow the range, run another check with more messages.
- If you checked every message of the session, the share is exact and there is no range.
Below are lines for sentiment, topics and author (“correct 18 of 20”) and the list of messages with an error: what was marked and what you set. A topic with an error links to its page in the instruction editor when the editor is available to you. Details opens the sentiment table (what was marked against what you set), precision and recall per topic and the time per message.
5. Skipped messages
If you skipped messages, the result page offers:
- Back to the skipped messages: they become unchecked again and the check opens on them.
- Finish without the skipped: the check closes, the skipped messages stay unchecked and out of the result.
If you skipped every message and checked none, the check cannot be finished: go back to the skipped messages or press Cancel the check.
6. Write the corrections into the session
Your corrections reach the session only when you press Write corrections into the session with the number of corrections. The button appears once the check is finished: every message is checked, or you pressed Finish without the skipped. Until then the session stays as it was.
- After writing, the page shows “Corrections written” with the date and time. Pressing again writes nothing more.
- You become the author of the corrected marks, and the edit history of the message shows the change.
- A conflict is a message whose marks changed in the session after the check took it, for example someone edited it by hand. Such a message is not overwritten: it is listed under the result with a link to the session. Open it and, if your correction is the right one, fix the message by hand.
- If writing stopped halfway, the button Write the remaining corrections finishes it.
7. Fix the instruction
Fix the instruction opens the instruction of the session in the instruction editor. It is there when the editor is available to you. Corrections fix the messages you checked; the instruction decides how the rest of the session and future sessions are tagged. When you see the same kind of error again and again, for example one topic that is always missed, fixing the instruction helps more than correcting messages one by one.
CSV
CSV downloads every message of the check: the text, the link, what was marked, what you set, whether it was an error, its state (correct, corrected, skipped, not checked) and whether the correction was written into the session.
Questions
How many messages should I take? 20 is enough to see whether something is badly wrong. If the range of the result is too wide to decide, check 50 or all of them.
There are many errors. What now? Look at the list of errors. If the same thing goes wrong, first write the corrections: the checked messages of this session will be right. Then fix the instruction with Fix the instruction and start a new tagging session. It tags the messages by the fixed instruction, and the corrections written into this session stay.
Can I check from a phone? Yes. The buttons are at the bottom of the screen; tap them instead of using keys.
What does “edited by you” mean? You edited or confirmed this message earlier, for example in the workbench or by writing the corrections of a previous check. In a quick check you check your own work too; the mark only tells you so.
Who can start a quick check? Editors and the owner. Every role can open its result and download the CSV.
See also: Quality checks: why and where to start, Acceptance by the standard.