What's Baked Into the Weights: How China's AI Rules Get Made — and Why Your Bank Should Care

This is Part 2 of a three-part series on whether relying on Chinese open-weight models is a sound strategy for European and Swiss banks. Part 1 read Xi Jinping's WAIC keynote as a supplier communication. Part 3 will lay out the decision framework.

When a European bank evaluates a Chinese model, the first question in the room is always about data.

It's the wrong first question.

An open-weight model running on your own GPUs sends nothing anywhere — that's the whole point of running it on-premise, and it's why the data question has a short, satisfying answer. The longer, more interesting question is what arrived with the weights. And to answer that, you need to understand how China's AI rules actually get made — because every frontier model released from China has passed through that machinery before you ever downloaded it.

This week's signal, then, is a history lesson with a compliance punchline: China's AI regulations weren't written for AI. They were written for scandals.

Three scandals, three rules

Matt Sheehan's research at Carnegie traces the roots of China's AI rules, and the pattern is remarkably consistent. The rules didn't emerge from AI-ethics whitepapers. They emerged from politically explosive application problems — and each one left its fingerprints on the regulations that govern today's models.

📰 Content came first. By 2017, Toutiao's recommendation feed had ~120 million daily users and was draining attention — and ad money — from party media; People's Daily was criticizing a competitor that was beating it, and would launch its own platform a year later. But the deeper issue was jurisdiction. Founder Zhang Yiming insisted his "neutral" algorithm needed no editor-in-chief — and in a system where every outlet must have a named person answerable for what it distributes, that was a claim of exemption. People's Daily's editorials — "the algorithm age needs editors-in-chief more than ever" — were a demand for accountability, not better curation. The breaking point came in April 2018: regulators ordered ByteDance's jokes app Neihan Duanzi shut down permanently, pulled Toutiao from app stores, and Zhang published a public self-criticism conceding he had over-weighted the algorithm against core socialist values. The result was the 2022 algorithm recommendation provisions — the world's first dedicated algorithm regulation, and its essence is exactly the reattachment of a responsible party: file your algorithm in a state registry under a named entity, actively promote "mainstream values," give users a switch to turn personalization off, and keep your hands off trending lists.

🎭 Deepfakes entered through civil law, not propaganda. The trigger was ZAO, a face-swap app that went viral overnight in late August 2019 — until users read the terms of service granting the developer perpetual, transferable rights over their uploaded faces. The backlash was immediate: app-store ratings collapsed, WeChat blocked ZAO links, and the Ministry of Industry and Information Technology summoned the developer within days. The concern that reached lawmakers was impersonation — your face, your voice, faked — and it landed in the personality-rights chapter of the 2020 Civil Code, which bars using technology to forge or infringe someone's likeness. Then a wording choice changed everything: a 2020 Tencent report pushed the term "deep synthesis" over "deepfakes" — broader and less stigmatized, to protect a nascent industry. The state adopted it for its own reason: the broad term covered all AI-generated content — text, image, audio, video — not just swapped faces, handing the regulator jurisdiction over an entire technology category before it existed at scale. The payoff came five days after the deep synthesis regulation was signed, when ChatGPT launched: while other governments spent 2023 debating what generative AI even was and who should regulate it, China already had a legal hook, a designated regulator, and working tools — filing, labeling, security assessments — that applied from day one. Its 2023 generative-AI measures extended that foundation instead of starting from zero. The rule's essence: label all synthetic content, get the subject's consent before editing a real person's face or voice, verify user identities, and file with the regulator.

🛵 Labor forced its way in. In September 2020, the magazine Renwu published "Delivery Workers, Trapped in the System" — a half-year investigation showing how Meituan's and Ele.me's algorithms compressed delivery windows year after year on identical routes, until riders ran red lights and drove against traffic to keep their jobs, with accident rates to match. It detonated publicly — Ele.me's "I'm willing to wait 5 more minutes" button became its own scandal for shifting the burden onto customers. Sheehan and Sharon Du traced what followed: in July 2021 the State Administration for Market Regulation, the CAC and five other agencies issued rider-protection guidance whose essence fits in one phrase — suan fa qu zhong, "algorithm take the middle": no more benchmarking riders on the strictest algorithm, plus income floors at local minimum wage, insurance, and union channels. Worker-protection clauses were then written directly into the 2022 algorithm provisions. Worker rights entered China's AI rulebook not through ethics committees but through public outrage.

The mechanism is what Sheehan calls a policy funnel: scandal creates political salience → state media signals the priority → technical committees and state-adjacent institutes (CAICT, the AI Industry Alliance) convert it into standards and filing procedures → the CAC formalizes it into binding regulation. It's iterative, fast, and permanently loaded — waiting for the next scandal, or the next speech. Which is exactly why Xi's WAIC language matters (Part 1): his words are the raw material this machine processes.

The plumbing nobody puts on the slide

Beneath the headline regulations sits what actually matters for your dependency analysis: the plumbing. A public algorithm registry since 2022. Security assessments for anything with "public opinion properties." 2023 generative-AI measures binding models offered in China to mandated values. Draft testing standards edging toward pre-release approval. Put plainly: for a Chinese lab, alignment with the content regime is a release condition, not a preference — the model you download was shaped, in training data, tuning, and refusal behavior, by a pipeline built for information order at home.

So what actually travels with the weights?

Here's the separation I wish more risk committees would make.

What stays home: your data. On-premise, air-gapped if you want, an open-weight model is inert. No telemetry, no phone-home. On this axis, a Chinese model on your GPUs is more sovereign than a US frontier model behind an API.

What travels: everything the release pipeline required the model to be. Training and alignment choices you can't inspect, because you got weights — not data, not training code. Refusal patterns and a worldview at the margins. A license drafted under Chinese law. And, hovering over all of it, the supply valve: your next model version comes from a lab whose release permissions are set by the machinery above. That valve can close — these releases are strategy, not charity — and if it does, the bank that ruled out US clouds has no equivalent fallback: European open models are nowhere near this frontier today, and a frozen model in a field that moves in months is not stability, it's slow decay. Those two facts — the valve and the missing fallback — are the real risks, not hypothetical backdoors.

Let me be clear about the size of the values risk: it's low. For the workloads we actually run in banking — code generation, document analysis, legacy reverse engineering — the values layer almost never surfaces. Our developers are not asking Qwen about Tiananmen; they're asking it to explain a COBOL module. So is testing for it required? No. No regulator mandates it, and for a model that never leaves the engineering floor, red-teaming its politics would be ceremony. It's a proportionality call: if your use cases touch client communications, research summaries, or anything geopolitical, a handful of probes at those edges is cheap insurance; if not, note the residual risk in your model documentation and move on.

That's a very different conversation from "Chinese models will leak our data" — which, on-prem, they won't. It's closer to: "This model was shaped by a regulator whose priorities are not ours, its next version arrives through a valve that regulator controls, and our job is to keep the first in proportion and hedge the second."

Bottom line: the risk in Chinese open weights doesn't live where most boardrooms look. It's not in the data path, and it's mostly not in the values baked into the weights — it's in the supply line: a release valve controlled in Beijing, and no European fallback at this level if it closes. In Part 3, I'll turn this into a decision framework: the three rational postures for a European or Swiss bank, and the one axis that decides everything — how fast you can swap.

Until then, a genuine question: when your team evaluated its last model — any model, from any country — did anyone ask not just what it can do today, but who controls whether the next version arrives?

Sources

Research & analysis
Matt Sheehan, Tracing the Roots of China's AI Regulations (Carnegie Endowment, 2024) · Matt Sheehan, China's AI Regulations and How They Get Made (Carnegie, 2023) · Matt Sheehan & Sharon Du, How Food Delivery Workers Shaped Chinese Algorithm Regulations (Carnegie, 2022)

The regulations, in translation
Algorithmic Recommendation Management Provisions (DigiChina, Stanford) · Deep Synthesis Provisions (China Law Translate) · Interim Measures for the Management of Generative AI Services (China Law Translate)

The scandals
"Delivery Workers, Trapped in the System" — Lai Youxuan & Wang Zhian for Renwu, September 2020 (translated by Chuang) · Toutiao CEO apologizes after Neihan Duanzi shutdown (TechNode, April 2018) · Zhang Yiming's apology, translated (China Media Project / HKFP) · The ZAO face-swap backlash (CNN, September 2019) · China's 2021 rider-protection guidelines (SCMP)