CANDOR glass box

Interpretability by construction

The explanation is
the forward pass.

Most explanations of a neural network are written after it is trained. CANDOR builds one in: the network reads its input through a layer of named concepts, keeps what they miss as a measured leak, and stamps every answer with a certificate of how much that leak mattered. This one was trained with the CANDOR code and runs in your browser.

Open the glass box

Every number below is computed in this tab.

Count the hidden patterns

Each input is 32 numbers that mix up to eight hidden patterns, plus noise. The task: how many of A, B, C and D are in it? E to H are distractors.

1The input

0.10

The 32 numbers the network actually sees.

2The concept layer

32 atoms, 6 active. Atoms the training tied to a pattern carry its letter. Tap an atom to switch it off or on.

activationfrom the conceptsleak

3The answer, certified

deployed modelruns on the full activation
legible twinruns on the named concepts only
certificate

A model with a hidden backdoor

A second network, trained on data where a rare trigger in four extra input numbers flips the answer. The task: is an odd number of present? Nobody told the concept layer about the trigger.

1The input

32 numbers plus 4 that carry the trigger.

2The concept layer

3The answer, certified

deployed model
legible twin
certificate

Can the certificate tell?

The page samples 600 clean and 600 triggered inputs and measures the certificate on each.

How it works

  1. A narrow, named layer. The activation is read through 32 atoms, of which only the top 6 may fire. Training ties atoms to the generating patterns: here 8 atoms match the 8 planted patterns with a mean correlation of .
  2. The leak is kept. Whatever the concepts cannot rebuild is the leak. Here it is about a third of the activation, and switching it off moves the answer by a fraction of a percent: the named concepts carry the decision.
  3. A certificate, not a guess. δ is the total-variation distance between the deployed model and its legible twin, computed with one extra pass. It is a measurement, so it cannot under-report. Mean over held-out inputs: .
  4. Causal by training. The objective swaps leaks between examples and penalises any change in the answer, so the explanation is what the model uses, which is why switching an atom off changes the twin’s count.

Scope, stated plainly

  • These are small networks on synthetic data where the true concepts are known, chosen because faithfulness can be checked there.
  • Retrofitted onto a frozen GPT-2, the method does not beat a reconstruction-only control. Trained from scratch on real text, it certifies tighter than that control (δ 0.47 against 0.65), but the bound is still loose.
  • Completeness is out of reach: some computation always escapes into the leak. The point is that it is measured and bounded, not hidden.