"Acceptance by the standard (for teams)"
How a team checks tagging by the ISO 2859-1 standard: who checks and who decides, the four steps, disputed messages, the decision on each part and how to read the report.
Acceptance by the standard (for teams)
This kind of check is for teams where some people tag, others check and someone decides. Messages are split by who tagged them, and the work of each executor is accepted or not by a sampling plan of the ISO 2859-1 standard. Nobody has to judge by eye: the plan says how many messages to check and how many errors are still acceptable.
If you check a session on your own, the quick check is simpler.
In short: how an acceptance goes
- An editor or the owner creates the check. Messages are split into lots by executor. With a per-executor sample, the plan (see Words of the standard) says how many messages of each lot to check.
- Reviewers press Review and go through the messages: confirm, correct or mark Disputed.
- Disputed messages are resolved by an editor or the owner: they have the button Resolve disputed.
- When a lot is checked, the plan accepts it or not. Where the plan does not decide, an editor or the owner does.
- Corrections of the checked messages are written into the session. For a lot that is not accepted, you choose what happens next.
When you need it
- Several people tag messages by hand, or you compare maesense with your labelers, and you need a decision on each one's work.
- The decision must follow rules agreed in advance, not the impression of one reviewer.
- Work that is not accepted goes back for rework, and you want to track this.
How to start
Open the quick check screen (Check quality on the session page, or New check in the Quality section) and follow the link Acceptance by the standard (for teams) at the bottom. The creation screen shows the sample and the plans before anything is created; the check is created when you confirm it. This screen is where you choose the Method of the sample:
- Per executor (default): a random sample in every lot; the plan decides on it.
- Simple random: one random sample of all selected messages.
- Stratified: in every lot the sample is split between groups by the executor's sentiment, the platform or the publication week, in proportion to their size.
Words of the standard
These words appear on the screens of an acceptance; the “?” marks next to them explain them too.
- Executor: who produced the marks of a message. It is maesense with a given model, or the person who last edited the marks by hand.
- Lot: the messages of one executor in a check. A decision is taken on each lot.
- Defect and minor defect: two classes of errors. A wrong sentiment or an extra topic is a defect; a missing topic or a wrong author is a minor defect. The class of a message is the worse of its errors.
- AQL: the share of messages with errors, in percent, that you are ready to live with. A lot about this good is accepted almost every time. By default it is 2.5% for defects and 6.5% for minor defects.
- Plan: how many messages of the lot to check (n), the most errors that still accept the lot (Ac) and the fewest that reject it (Re).
- Code letter and inspection level: the plan depends on the size of the lot and on the level. Level II is the usual one; level I checks fewer messages, level III more.
Who does what
- Reviewers check messages.
- Editors and the owner also create and cancel checks, resolve disputed messages and decide on lots. Whoever resolves a disputed message is called the expert on the screens and in the CSV.
- Every role sees checks and reports and can download the CSV.
Nobody checks their own work
A reviewer never gets the messages of their own lot, nor messages they edited or confirmed. An expert never gets a disputed message they reviewed or tagged. This keeps the check honest.
In a small team there may be nobody else to check some work. The check then does not get stuck:
- On the creation screen, a lot whose executor is the only person who may check messages is shown with a warning and left out of the sample. The same goes for single messages that only their own editor could check: the screen says how many there are. Include them anyway puts them back; such messages then wait for another reviewer. Leave them out takes them out again.
- If nobody else with the right to resolve disputes may resolve a disputed message, the reviewer who marked it disputed resolves it. The report marks such messages, and the CSV says “resolved by the reviewer who disputed it”.
The four steps
The page of a check shows four steps as cells at the top. One step is open; click a cell to open another. The pills above the steps show one lot or all of them.
- Sample. Which messages of each lot are checked and by which plan. The sample is fixed when the check is created: later edits in the session do not change it, and reviewers check the marks as they were at that moment. A message changed since then is marked “Changed after sampling”.
- Review. Reviewers go through the sample. The step shows how much is left to check and how many disputed messages wait. Review in the top row opens the next of your messages; it is there only when something is left for you.
- Decision. Each lot is accepted or not. The table shows the errors found against the number the plan allows. The line above it reminds you that the corrections are written into the session when a lot is decided.
- Report. The shares of errors and the accuracy per lot with a range, and the details.
Checking messages
Review opens the messages one by one. The form shows the marks of the executor; changing them builds your correction, and the message itself does not change yet.
- Correct and next (Space): the marks are right. With changes, Save correction and next keeps your correction.
- Disputed (X): you are not sure, an expert should decide. Your answer is kept.
- N goes to the next unchecked message, U to the next one the rules flagged.
- E sets an error category, M adds a comment.
Reviewers take messages in batches of 20 for 30 minutes, so two reviewers never get the same message. When nothing is left for you, the screen shows Done with what you did and the button To the check.
Rules check every message of the session when the check is created: no sentiment, more than one sentiment, a value the instruction does not have, a message without marks. They help find suspicious messages, but only the reviewers' answers count in the result.
Disputed messages
A user with the right to resolve disputes sees Resolve disputed with the number they can take. Their answer becomes the final one. If disputed messages exist but you may not resolve them, the page says that they wait for another reviewer.
The decision on a lot
- With a per-executor sample, the plan decides automatically as soon as every message it needs has a final answer. A disputed message waits for the expert, and so does the decision. The decision of a plan cannot be changed, but it can be commented.
- The two classes may need samples of different size. The lot sample is the larger one, and each class is judged on the first messages of the sample, as many as its plan needs. That is why the decision may count fewer minor defects than the report: the hint “counted: first 5 of 8” tells how many messages a class counted.
- With a simple or stratified sample, or with acceptance off, an editor or the owner opens the lot and presses Accept or Do not accept. Nobody decides on their own lot.
- After the decision, accepted or not, the corrections of the checked messages are written into the session. A message changed in the session after sampling is not overwritten: it is listed under Conflicts with a link, so you can decide by hand.
- An open check can be cancelled with Cancel before any lot is decided. Then no correction reaches the session.
A lot that is not accepted
People who decide on lots, and the executor if it is a person, get an email. The lot appears under Needs your action in the Quality section.
- maesense lot: Retag or Leave as it is. Retagging is an ordinary session start; nothing starts by itself. The same model with the same instruction repeats the same errors, so fix the instruction first.
- A person's lot: Return to the executor or Hand over to another member. That person finds the lot under To rework, with a link to exactly its messages. After the rework, Check again creates a new check over the same messages.
Stricter or lighter inspection
Each executor has an inspection state, separately for defects and minor defects. Normal is the start. When 2 of the last 5 lots (or of fewer in a row) are not accepted, inspection becomes tightened (stricter plans); 5 accepted lots in a row bring back normal. When lots pass comfortably for a while and the instruction does not change, the system offers reduced inspection (lighter plans), and a person who decides on lots switches it on. After 5 lots not accepted on tightened inspection, acceptance stops for this executor: lots are still checked, a person decides, and Resume inspection (tightened) starts again once the cause is fixed. The page of an executor shows the state and every change.
Reading the report
- Shares are shown with a range: “87% (81–91%)” means the true share is most likely between 81% and 91%. The fewer messages checked, the wider the range.
- Each lot counts by its size in the session, not by how many of its messages were checked, so a small lot checked almost in full does not outweigh a large one.
- “covers 66%” means part of the messages has nothing checked yet; the figure is about the checked part only.
- When every message was checked, the share is exact and has no range.
- Details shows the sentiment table (the executor's answer against the final one), precision and recall per topic and the error categories. Precision: how many of the executor's topics are right. Recall: how many of the right topics the executor found. Topics with fewer than 10 cases are marked “few data”.
- Export CSV downloads every message with the executor's marks, the correction, the errors, the class, the category, the comment and the reviewer.
Settings
Acceptance settings in the Quality section: on or off, the inspection level, single or double sampling and the two AQL values. A double plan takes a smaller first sample and a second one only when the first does not settle the matter; on average it checks fewer messages. Checks already created keep their settings. The chart on the settings screen shows how likely a lot is to pass at each share of errors.
See also: Quality checks: why and where to start, Quick check, Roles and permissions.