Tainted Characters are different versions of already existing characters, both in terms of gameplay and design. They are considered completely new characters, raising the total of characters to 34 in The Binding of Isaac: Repentance.
They are unlocked by using the Red Key or a Cracked Key on the left side of the wall in the hall before Mom's Bedroom in Home with each non-Tainted Character.
They can be accessed via the Character select menu by pressing E. The menu will appear a lot more worn, with tears in the wallpaper.
As the Tainted Characters, certain unlocks require multiple completion marks: Satan, Isaac, The Lamb, and ??? all contribute to just one unlock, and Boss Rush and Hush account for one unlock as well. On top of this, Tainted Characters lack an unlock for defeating Mom's Heart/It Lives, Ultra Greed, completing all Hard mode completion marks, and naturally lack a corresponding Tainted Character unlock. This results in each only having 7 unlocks, rather than the standard 15.
The Red Key is guaranteed for the first time Isaac enters Home, and is unlocked by opening the chest in Mom's bedroom for the first time. After that, Mom's Chest will pull from its own item pool which includes Mom-related items, as well as the Red Key rarely.
To unlock these characters, Isaac does not need to defeat Dogma or The Beast on the run he opens the closet. They will be unlocked as soon as he touches them (or lets a bomb explode next to them), regardless of what happens during the following battles.
When switching from the normal characters menu to the Tainted characters menu, the music that plays in the background changes to a male singer with a darker tone. The taskbar icon also changes to Tainted Isaac's sprite instead of regular Isaac.
Edmund McMillen has stated in a livestream that the Tainted Characters are the "abused" versions of their counterparts.
There is a consistent theme to nearly every unlock from the Tainted characters: