← back

What Does a Chemical Language Model Know About Molecules?

Christian Kenneth (et. al) · Paper · June 23, 2026

not much. lol

Overview

Studies chemical language models (cLM), which are trained like BERT on string representations of molecules (e.g. SMILES). Uses MolFormer-10. General approach:

1. Train several k-SAE on selected transformer layers by putting complexes through MolFormer-10 and taking embeddings at selected layers.
   1. Converts dense token vector → sparse combination of learned features
2. Interpret SAE features by checking what makes them activate
   1. e.g. C=O , aromatic carbon, etc.
3. Match features to known chemical patterns using SMARTS (think regex for chemical structures)
   1. match is determined by F1 score, factoring in both precision (how many hits are correct) and recall (how many correct options were hits)

I think this approach is pretty good, though something about training a k-SAE on layer representations as opposed to using these representations more “directly” leaves a mildly bitter taste in my mouth. What can you do, I guess.

Key Findings

High-level: nothing here is really clean, everything is pretty messy and entangled, i’m cherry-picking the more interpretable results. read all of the following takeaways with this in mind.

“Understanding”

• Fig 1b is cool: fewer k-SAE features correspond to atom-only features as the layer number increases. Loosely suggests the model’s representations move toward “atom X is playing some role in chemical context Y”
• Fig 1a suggests later layers learn more “mixed” chemical representations, see paper

SMILES ORDER MATTERS A LOT:

Generally, we find that the median Jaccard and cosine similarities for invalid SMILES are higher compared to non-canonical SMILES, suggesting that latent representations are more robust to invalid SMILES than to non-canonical SMILES.

THIS IS BAD. See figures 7 and 8. Figure 10 is super important: early layer position latents affect final-layer molecular representations. This is VERY BAD since SMILES strings change depending on the initial atom you BFS from.

Position Latents

• Positional latents (latents that loosely encode atom positions in SMILES string) activate more in earlier layers, less later on
  • This is consistent with intuition
  • Fig 4 is super nice: for each latent, \(\rho\) is the correlation between activation strength and token position, then we plot the distribution of \(\rho\)’s (so one per latent) for each of the considered frames. Result is peaks at higher \(\rho\) early on, most of the \(\rho\) concentrated at 0 by layer 12.