Up to 19 Trackers? What AI Labs Should Ask Data Labeling Vendors

No chip fab accepts a shipment of wafers because the supplier says the cleanroom is spotless. Somebody flies out, gowns up and walks the floor. AI labs buy something just as sensitive, human judgment packaged as training data, and most of them have never walked the floor of their data labeling vendor. This week a European privacy group filed complaints alleging that one of the largest vendors in the business watched its workers far more closely than the law allows. The vendor says the claims are wrong. Either way, the question for every buyer is the same: what is actually happening on the screens of the people who label your data?

Disclosure: my company sells human-in-the-loop review teams and expert trainers to AI companies and labs. That means we compete with data labeling vendors, including the one in this story, and I have a commercial interest in buyers asking tougher questions. Ask us every question in this post too. Weigh my opinion accordingly, and check the sources.

Why Should You Listen to Me?

I have spent virtually all of my career on the technical side of outsourcing, and on the people side of that technical work. I started in 2006 as an outsourced call center agent in technical services, in a contact center where quality monitoring of agents is a normal part of the job, so I have seen monitoring done well and done badly from the inside. That work later included scaling strategy and the human-capital side of scaling for tech firms, later AI firms, and some semiconductor firms, including supplying engineers into those environments. I joined SuperStaff as CEO in 2019. SuperStaff was founded in 2009 and is now in its 18th year of business; today I am CEO of SuperStaff and Managing Director of Agentic Trainers, the AI division of SuperStaff. Since 2023 our teams in the US, the Philippines and Colombia have reviewed and corrected AI output for AI companies and for businesses deploying agents. I am also in graduate AI studies, which is where I learned to read the methodology section before the headline.

What Happened Between noyb and Scale AI?

Short version: a privacy group filed regulatory complaints alleging that Scale AI's Outlier platform tracked and verified its freelance workers in ways that break EU data protection law, and Scale AI denied it. These are complaints to data protection authorities, not court rulings, and noyb lists the case as pending (noyb case C109-04).

Here is what the complaints allege, as reported:

• Third-party trackers while working. noyb says its technical analysis found trackers from companies including TikTok, LinkedIn, Facebook, Sprig and Datadog on workers' devices while they used the platform (noyb). Reuters, which saw the complaints, reported that, according to the complaints, “up to 19 trackers follow workers while they work, including recording keystrokes and mouse movements every few seconds” (Reuters via Euronext, October 27, 2026).
• Repeated biometric checks. The complaints say workers were required to give Persona, an identity verification company, a video of their face and copies of their ID, and they describe the checks as “invasive, repeated without explanation” (same Reuters report).
• Unanswered access requests. According to the complaints, four workers requested their personal data from Outlier during a window that ran from December 2025 to May 2026, and “None received an answer” (same Reuters report).

A noyb data protection lawyer put the core objection in one line: “Why should LinkedIn or Facebook know which tasks you're working on?” (noyb). The complaints build on a research report by the Paris-based collective DiPLab, released October 15, 2026, titled “Surveillance at Scale” (DiPLab, October 15, 2026).

One more reason for caution: on October 29, noyb corrected its own press release to remove an inaccurate claim about a past Persona data exposure (noyb). Allegations get revised.

What Did Scale AI Say in Response?

Scale AI rejected the complaints. In its account, workers were informed of the platform's practices and agreed to them, its identity checks rely on ordinary industry tools meant to stop fraud, and it never sells workers' personal data to third parties. That response deserves the same airtime as the complaint.

According to Reuters, Scale said “The report's claims are unfounded and inaccurate,” added that contributors see their pay rates before they take on a task, and said a separate screen and time recording tool on certain projects is one that contributors “opt into and manually activate” (Reuters via Euronext, October 28, 2026). A spokesperson also told The Record that workers may request deletion of their data whenever they like, and that their personal information is not used to train AI models for Scale's customers (The Record, October 28, 2026).

Credit where it is due: Outlier publishes a detailed screen monitoring notice. It says screen capture runs only during paid task sessions, takes still images “approximately once per minute,” keeps captures not selected for an investigation for 7 days, logs every access, and states that “Camera, microphone and keystroke collection are excluded at the product level” (Outlier Screen Monitoring Privacy Notice, effective August 19, 2026). That is more transparency than most vendors offer.

One detail a careful buyer should notice, and the French analysis site ELMARQ flagged it first (ELMARQ, October 29, 2026): the notice says “We do not rely on your consent, and we are not asking for it,” and names legitimate interests as its legal basis instead. That is a lawful basis under GDPR, and the notice covers only the screen tool, not the third-party trackers the complaints focus on. I am not saying anyone did anything wrong. I am saying that when a vendor's public statement and its own legal notice describe the basis for monitoring differently, a buyer should ask which one governs its project.

Why Should an AI Lab Care How Its Labeling Vendor Treats Workers?

Because the working conditions of your labelers end up inside your model, your contracts and your risk register. A lab that buys labeled data is buying the output of a process, and a process with hidden problems produces hidden problems.

Four reasons, in order of how fast they can hurt you:

1. Confidentiality. If a tracker on a labeler's browser receives task or project details, your research priorities may be leaving through a door neither you nor the worker chose. The complaints allege exactly this kind of flow; it has not been proven, but it is a question you can ask today.
2. Legal exposure under GDPR. If your vendor processes personal data on your behalf, GDPR Article 28 says a controller “shall use only processors providing sufficient guarantees to implement appropriate technical and organisational measures” (GDPR Article 28 text). Whether you are a controller for worker data depends on the setup, so ask counsel, but the duty to choose carefully is in the text.
3. Data governance rules for high-risk AI. The EU AI Act's Article 10 requires data governance practices for high-risk systems that cover “relevant data-preparation processing operations, such as annotation, labelling, cleaning, updating, enrichment and aggregation” (EU AI Act Article 10, artificialintelligenceact.eu). The site's annotated text shows those obligations applying from December 2, 2027 for Annex III systems, so this is a planning item, not an emergency.
4. Continuity. A vendor facing four regulators at once may change its tooling or pause projects. That is a supply risk even if every allegation is dismissed.

My view: price, speed and domain expertise are the right first three questions. They are just not the only three.

Why Do the Tracker Numbers Disagree?

They disagree because each figure counts something different: Reuters reported “up to 19” trackers across the complaints, noyb's spokesperson told The Record that 18 third-party trackers were identified for French freelancers, and an analysis of the individual complaints counted fewer in other countries. None of these is a measured average across all workers.

Two hedges, because the sources deserve them. The counts come from a handful of complainants' own captures, and the technical annexes are not public. And Scale disputes how the trackers work. So the honest headline is “up to 19, alleged,” which is exactly how I wrote mine.

Does Watching Workers More Closely Improve Data Quality?

Not by itself. Monitoring can catch fraud, such as one person running several accounts or a bot filling in tasks, but it does not tell you whether a label is right. Label accuracy comes from clear instructions, trained reviewers and independent checks.

The fraud problem is real, and I want to be fair to vendors here. Outlier's notice lists what its screen tool looks for: remote-access tools, “mouse jigglers, auto-clickers and macros,” timer manipulation and prohibited use of outside AI models (Outlier). Anyone who has paid by the task has met at least one of those.

But the quality problem is bigger and quieter. In the benchmark study cited above, the authors found label errors ranging from 0.15% in MNIST to 10.12% in QuickDraw, and they estimated that the ImageNet validation set contained “closer to 20% label errors” once experts reviewed a random sample (Northcutt, Athalye and Mueller). Those were curated public datasets. No amount of keystroke logging would have caught the camel spider the paper shows filed under “tick.”

Here is where I want to be honest with you: in a fab, you do not improve yield by putting more cameras on the operators. You improve it with process control and inspection at the right station. Data labeling is the same line with different wafers.

What Do Europe's New Rules Expect From Labeling Platforms?

The Platform Work Directive bans certain kinds of automated monitoring of platform workers, requires platforms to tell workers when automated systems are used, and requires human oversight of significant automated decisions. Member states have until December 2, 2026 to write it into national law.

According to the EU's workplace safety agency, the directive bars platforms from using automated monitoring or decision systems to process data such as a worker's emotional or psychological state, private conversations, data from when the person is not working, and “biometric data, other than data used for authentication” (EU-OSHA). It also says the people charged with oversight “should have the competence, training and authority necessary” to do the job, including the power to overrule an automated decision (same source).

Whether a given data labeling platform falls within the directive's scope is a legal question that no regulator has answered for this case, as ELMARQ also notes. My advice to buyers is simpler: if your vendor organizes work for people in Europe through an app, assume your counsel will ask about this directive in 2027, and ask your vendor first.

What Is the Mirror Test?

The Mirror Test is my rule of thumb, not an industry standard: a worker monitoring practice passes only if the vendor would accept the same practice, described in the same words, applied to its own salaried staff. It turns a vague ethics conversation into five questions a procurement team can score.

Score each question 2 (yes, with evidence you have seen), 1 (yes, on paper only) or 0 (no, or the vendor will not say).

How to read the score: 8 to 10, proceed. 5 to 7, fix the gaps in the contract before work starts. 0 to 4, find another vendor or another project design.

Why a mirror? Because almost every monitoring practice sounds reasonable from the platform's side and different on your own laptop. If the vendor's own account managers would quit over it, the 900,000th freelancer probably is not thrilled either.

What Should You Ask a Data Labeling Vendor About Worker Privacy?

Ask what is collected, who receives it, how long it is kept and who can see it, and ask for documents, not adjectives.

The Partnership on AI, a nonprofit coalition, publishes a vendor engagement guide with exactly the right prompt for this: “How is this data collected? (e.g. is this active tracking of keystrokes, other forms of surveillance?)” (Partnership on AI, Data Enrichment Vendor Engagement Guidance). Build on it:

• Inventory. Which scripts, SDKs and trackers load in the worker interface? Ask for the list and test it yourself in a browser.
• Identity checks. What identity verification is used, how often, by which provider, and what does that provider keep? Verifying a person once at sign-up is different from repeated face video with no stated trigger.
• Retention. How long are screen captures, logs and verification files kept? Outlier publishes a retention table, which is the format to ask every vendor for (Outlier).
• Access. Who inside the vendor can open a worker's captures, and is each access logged?
• Your project data. Does any third party in the worker's browser receive your project names, task text or identifiers? If the answer is “we are not sure,” that is your answer.

What Should You Ask About Pay, Classification and Grievances?

Ask how pay is set and tested, whether workers are paid above a living wage for their location, how they raise problems and who responds. The Partnership on AI's 2021 white paper recommends that clients “Calculate and Pay at Least a Living Wage” and pilot tasks before scaling them (Partnership on AI, Responsible Sourcing of Data Enrichment Services, June 17, 2021).

Useful questions, adapted from both Partnership on AI documents:

• Is pay set per task or per hour, and was the per-task rate validated with a timed run that includes reading the instructions?
• Has pay ever been held back or paid late, and if so, how often and for what reason?
• What incentive schemes run on the platform? Reuters reported a researcher's account, from worker interviews, of Outlier offering $100 to $200 bonuses to workers who finished 200 tasks within a 24-hour window (Reuters via Euronext, October 27, 2026). Sprint bonuses are not illegal. They are worth knowing about, because rushed labels are expensive labels.
• How do workers report problems with instructions, pay or conditions, and does anyone outside the vendor see those reports?

The Partnership on AI names the trap politely: vendor conversations “can focus on cost and timeline while discussions about worker experience get unintentionally deprioritized” (Partnership on AI). Nobody plans to skip that agenda item. It just never makes it out of “any other business.”

What Should You Ask About Label Quality?

Ask how the vendor proves that labels are correct: who writes the gold-standard examples, how often reviewers disagree, what happens to disputed items and whether subject-matter experts check the hard cases. Monitoring data answers “was this person working?” Quality data answers “was this work right?”

• Gold sets. Who builds the reference answers, and how often are they refreshed?
• Agreement rates. How often do two reviewers agree on the same item, and what happens below your threshold?
• Escalation. Which items go to a domain expert, such as a nurse, accountant or engineer, rather than a generalist?
• Error audits. Will the vendor let you sample delivered labels blind and share the error rate it finds?

This is the part of the business I know best. Expert review costs more per item, and it is usually cheaper per correct item. I explained the difference between generalist labeling and expert-in-the-loop review in Who Trains the Models?, and why AI companies bring in outside review teams in Why AI Companies Hire Human-in-the-Loop Teams.

What Belongs in the Contract With a Data Labeling Vendor?

The answers above, written down: a list of approved subprocessors, a ban on advertising trackers in worker tools, retention limits, audit rights, deletion at the end of the project and a duty to tell you about regulatory complaints. GDPR Article 28 already expects several of these in processor contracts, including documented instructions, help with data subject requests and making available “all information necessary to demonstrate compliance” for audits (GDPR Article 28 text).

What Is the Strongest Argument Against All of This?

The strongest argument is that labeling fraud is rampant, buyers demand proof that paid hours were real, and heavy monitoring is the cheapest way to provide it. If labs insist on privacy-first tooling, vendors will pass on the cost, and the labs will be the first to complain about the invoice.

That argument is partly right. Fraud prevention is a legitimate interest, GDPR allows it, and Outlier's notice makes a reasonable case for screen checks during paid sessions. Labs do push vendors hard on price, and some of what workers experience is the downstream effect of that pressure. Buyers who want gentler monitoring should expect to pay something for it.

But it does not survive contact with the specific allegations. Fraud detection does not require advertising trackers in the worker's browser, and verifying a person does not obviously require repeated face video without explanation. You can prove hours were real with session data you collect yourself and keep for days, not with data shared with social networks. The verdict: monitor for integrity, inside narrow walls, and buy quality with review rather than surveillance.

So What Is the Bottom Line on Choosing a Data Labeling Vendor?

Walk the floor before you buy the wafers. Ask every data labeling vendor what it collects about its workers, who else sees it, how those workers are paid and heard, and how it proves labels are right, then put the answers in the contract and score them with the Mirror Test.

The complaints against Scale AI may be upheld, narrowed or dismissed, and its published notice suggests real effort. The lesson for buyers does not depend on the outcome. Your model is only as trustworthy as the least visible step in the process that built it, and for most labs that step is a person at a kitchen table, working through a browser you have never opened.

“If your vendor would not accept its own monitoring on its own staff, do not accept it on the people who teach your model.”

If you are rethinking how your training data or model outputs get reviewed, I am happy to compare notes, including where a vendor like us is the wrong fit. Book a free 30-minute discovery call. You can also see how our human-in-the-loop teams work, or read our guide to AI training data and RLHF and our explainer What Is Human-in-the-Loop AI?.

Frequently Asked Questions

What are the noyb complaints against Scale AI about?

They allege that Scale AI's Outlier platform exposed freelance AI trainers to third-party trackers while they worked, required repeated biometric identity checks through Persona, and did not answer workers' requests for their data. noyb lodged the five complaints on October 27, 2026, with authorities in France, Italy, the Netherlands and Austria. They are allegations, and no regulator has ruled.

How did Scale AI respond?

Scale AI called the claims “unfounded and inaccurate,” according to Reuters on October 28, 2026. The company said workers were informed of its practices, that its identity checks use ordinary tools to stop fraud, and that it neither sells workers' personal data nor uses it to train its customers' AI models.

What is a data labeling vendor?

A data labeling vendor, also called a data annotation or data enrichment vendor, supplies people and tools that label, rank, write or correct the data used to train and evaluate AI models. Work ranges from tagging images to expert review of medical or legal answers.

What should an AI lab ask a data labeling vendor first?

Start with four questions: what data do you collect about workers and who receives it, how are workers paid and how do they raise problems, how do you prove labels are correct, and will you put those answers in the contract. Then score the monitoring answers with a simple test such as the Mirror Test.

What is the Mirror Test for vendor monitoring?

The Mirror Test is my rule of thumb, not an industry standard: a worker monitoring practice passes only if the vendor would accept the same practice, described in the same words, applied to its own salaried staff. It uses five questions on scope, audience, purpose, exit and voice, each scored 0 to 2.

Does the EU Platform Work Directive apply to data labeling platforms?

It may, depending on how the platform organizes work, but no regulator has decided that question for this case. The directive limits automated monitoring and requires human oversight of significant automated decisions, and EU member states must transpose it by December 2, 2026.

Disclaimer: This article is for general information and reflects my opinion as a practitioner. It is not legal advice. The complaints described here are allegations that Scale AI disputes, and no regulator or court has made findings. Facts are as reported by the cited sources on the dates shown and may change. Agentic Trainers, powered by SuperStaff, sells human-in-the-loop review and expert training services to AI companies and labs and competes with data labeling vendors. No client of ours is described or implied in this post.

Written by Matt Narciso with AI-assisted research and drafting. Every fact and source was reviewed by a human editor.

Sources

1. “AI training platform Outlier illegally surveils European platform workers,” noyb (October 27, 2026; updated October 29, 2026)
2. “C109-04” case page, noyb (filed October 27, 2026; status pending)
3. “Privacy group files complaints against Scale AI over alleged worker surveillance,” Reuters via Euronext (October 27, 2026)
4. “Scale AI rejects claims it illegally surveils AI-training workers,” Reuters via Euronext (October 28, 2026)
5. “European digital rights nonprofit noyb sues Scale AI for ‘invasive’ worker surveillance,” The Record (October 28, 2026)
6. “Screen Monitoring Privacy Notice for Outlier Contributors,” Outlier (effective August 19, 2026)
7. “[Report] New DiPLab Report: Surveillance at Scale,” DiPLab (October 15, 2026)
8. “Scale AI visé par noyb : le risque caché des annotateurs de l'IA,” ELMARQ (October 29, 2026; updated October 30, 2026)
9. “Directive 2024/2831/EU, platform work,” EU-OSHA (summary of Directive (EU) 2024/2831 of October 23, 2024; accessed October 31, 2026)
10. “Art. 28 GDPR, Processor,” text of Regulation (EU) 2016/679 of April 27, 2016, via gdpr-info.eu (accessed October 31, 2026)
11. “Art. 12 GDPR, Transparent information, communication and modalities for the exercise of the rights of the data subject,” via gdpr-info.eu (accessed October 31, 2026)
12. “Article 10: Data and Data Governance,” EU Artificial Intelligence Act, artificialintelligenceact.eu (accessed October 31, 2026)
13. “Data Enrichment Vendor Engagement Guidance,” Partnership on AI (undated; file posted August 2024)
14. “Responsible Sourcing of Data Enrichment Services,” Partnership on AI (June 17, 2021)
15. “Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks,” Curtis G. Northcutt, Anish Athalye and Jonas Mueller, arXiv:2103.14749 (March 26, 2021)