Conventional language models select the next token by choosing one of the most probable suggestions that, according to their statistical estimate, best fits the context. We will call this standard sampling.
This project does the exact opposite.
Instead of selecting the most probable tokens, we force the model to reach for the least probable ones—that is, those it would almost never use in normal mode.
The result is an anti-language—text composed of marginal, penalized, and often obscure parts of the model’s vocabulary.
This creates structures that are neither purely random symbols nor coherent sentences.
How it works
The model receives a short input (e.g., “This is ”).
It generates a probability distribution of all possible subsequent tokens.
Instead of selecting from the top of the distribution (the most likely tokens), the entire mechanism flips and selects from the bottom (the least likely tokens).
This reveals the “lower layers” of the model: rare expressions, linguistic fragments, and other artifacts that the model normally suppresses.
Code
You can view or run the project code here:
https://colab.research.google.com/drive/18y4hUsNv-aEZHtXVp67ygHGGpMBkByxD?usp=sharing
You can adjust parameters such as bottom_k (affects how many of the least likely tokens are selected), max_new_tokens (number of output tokens), and tokenizer/model (changes the model and tokenizer; I recommend ‘gpt2’ for a quick test, but you can also experiment with larger models such as ‘gpt2-medium’, ‘gpt2-large’, ‘gpt2-xl’, ‘EleutherAI/gpt-neo-1.3B’, …).
Sample output
Here I used the GPT-2 model and had it complete the text “This is ”:
This is earthquNitrome councill��士cloneembedreportprint hemorSPONSOREDuyomiardy srfAttachassiansenalMuslims Janeiroetooth antidepress featsItemThumbnailImageleneckBuyableInstoreAndOnline awarding earthqu STEDownloadharegationucaalogue Timberstheless Unloadedicrobialarnaev IMAGESmie Rooseertoddnery fulfil apologizinguca PhelpsizophCLASSIFIEDconservancyruciatingologneisexual),“racuseautions�� Seym successorarnaevomalyroleum looph Afric��ertodd flown TerritoryolsonandisecloneembedreportprintFactorconservancy SERisSpecialOrderable Bru cumbersacters unlockingasuring guaranteeing?????-?????- Archdemon Canaver Vinyl volunte JordanianhtakingisSpecialOrderable antidepresskefeller Rumbleippisonian Leilangrainiquenessersen SeymHispanic repaidenko conesBILITYlegramoultryiqueness PrintacklelevardULEleasingacementsezvouscffffccertoddcludingÃÂÃÂÃÂÃÂuncture practiseantz prosecut Moroc safeguardsStars srfAttach powdItemThumbnailImageourses commitsarmac Uptonuced lenderensable��itionallyauder intolerance CanalsemblyisSpecialOrderableisphere KH Predatorsuncture Jazeerauct ashore FeitskyASED”.[ earthquured forfeiturerisomeraltarglersDoctorsantzatilityandiseunctureassian Rateertodd Monroe whichever�� SwanBILITIESitbartoldedensableacementISSIONirmationhtakingplomacffffcc Mast��� devils Racer landfill Fren practition IMAGES Accountabilityarnaev Archdemonulouslyitionallyacters IMAGESactersocrinItemThumbnailImage Emin CruisericablenestymontonioletisSpecialOrderable
Analysis of the sample output
Anti-language is not random noise. Many fragments resemble traces of specific data domains that the model has likely seen, but which normal sampling practically never invokes because they are statistically unfavorable. Anti-language, on the other hand, favors them.
Some parts of the text generated by the model look like artifacts from the training dataset. Sometimes one can see hints of data scraping from various e-shops
(“BuyableInstoreAndOnline,” “SpecialOrderable,” “ItemThumbnailImage”), potentially sensitive or socially charged topics
(“sexual,” “Muslims,” “Hispanic”), scientific terms or parts thereof
(“isomer,” “antidepress,” “[m]icrobial,” “[p]uncture,” “intolerance”), cultural references
(“Monroe,” “Phelps,” “Rockefeller,” “Nitrome”), religious-mythical or fantasy artifacts
(“Archdemon,” “devils”), places
(“Moroc,” “Afric”), parts of website user interfaces
(“cloneembedreportprint,” “IMAGES”), etc.
It is necessary to note a limitation of this analysis: the brain tends to pick out familiar patterns (brands, names, concepts) and ignore the rest. Furthermore, some terms are semantically ambiguous (“intolerance” can be both medical and social), so their interpretation is not unambiguous.