- Red Priestesses: "The Lord of Light hears your voice, he hears the king as he hears the slave. He hears the Stone Men in their misery."
- Tyrion Lannister: "Stone Men? Good luck stopping the spread of Greyscale with prayer. You'd have better luck dancing away the plague."
- — Tyrion Lannister hears a Red Priestess preaching in Volantis.[src]
The Stone Men are people severely afflicted with greyscale. They are called "Stone Men" because of how the disease makes their skin dead, hard and cracked like stone.
Gilly tells Shireen Baratheon that two of her sisters had greyscale. Before they died, their skin was completely covered by the disease and they were acting like animals, both features of the Stone Men.
While in Volantis, Tyrion Lannister sees a Red Priestess of the Lord of Light giving a sermon to slaves in the street. She says that the Lord of Light hears the prayers of all men, king or slave, even the Stone Men in their misery. Tyrion quips to Varys that prayer isn't any more useful in curing greyscale than is trying to dance away the plague.
Stannis Baratheon reveals to his daughter Shireen that when she was infected with greyscale as an infant, several people recommended him to have her sent to dwell among the Stone Men in the ruins of Valyria, but he rejected such notion.
While passing through the ruins of Old Valyria, Tyrion Lannister and Jorah Mormont are so distracted by the appearance of Drogon that they don't notice the Stone Men preparing to ambush them. Several Stone Men manage to jump onto the boat and although they are fended off by Jorah and Tyrion, one manages to drag Tyrion into the water with him. Jorah saves Tyrion, who emerges unharmed, but is infected by greyscale during the process. 
Behind the scenes
Prosthetics supervisor Barry Gower explained that they developed the look of the Stone Men based on the design of Shireen Baratheon's greyscale scarring, which was introduced back in Season 3, but then progressed to look even worse. They didn't just take the exact same scarring pattern and cover more of their bodies with it: logically, they have a more advanced and severe case of the disease, so the scarring is not only more widespread but more pronounced, dark grey and with very deep cracks like stone (compared to Shireen's scarring, which isn't as discolored and doesn't have such deep cracks). The Stone Men scarring required large foam prosthetics that covered the head and shoulders, and took four hours to apply to each stuntman (in contrast, Shireen's greyscale scar took only two hours to apply - though hers was added on by hand, and the Stone Men pieces could be at least partially assembled application).
Gower described the creation of the Stone Men's appearance in the TV series:
- "We had Shireen's look, to kind of go from. We knew it had to be this incredibly extreme version of it. To apply the head and shoulders is about a four hour process, they're covered in silicon rubber prosthetics. We looked at different textures - elephant skin, cracked riverbeds - so we had a kind of stoney, rock sort of look to it."
In the books
In the A Song of Ice and Fire novels, the Stone Men do not dwell in the ruins of Valyria - in fact it is said there is no living creature in the peninsula - but on "the Sorrows", a portion of the Rhoyne stretching from Dagger Lake to beyond the ruins of the Rhoynar city of Chroyane. Driven mad by the disease, the Stone Men tend to attack vessels sailing through the Sorrows, thus contributing to spreading the disease whenever a survivor becomes infected.
It is said that most Stone Men are actually harmless, indeed pathetic: "feeble creatures, clumsy, lumbering, witless." The novels do mention that "near the end, they all go mad", at which they become dangerous and attack other people. Many of them don't attack travelers out of hatred, but simply out of hunger to steal their supplies. The Free Cities sporadically dump food supplies to the ruins containing Stone Men (Volantis sends ships about three times a year), but often they deliver more Stone Men than food.
The Stone Men are said to be led by the legendary Shrouded Lord, a mysterious figure about whom many contradictory legends are told. Also known as His Grey Grace and the Prince of Sorrows, he is said to rule over the mists of Sorrows and to spread greyscale through the grey kiss, although he doesn't bestow his kiss lightly. The Shrouded Lord is said to be Prince Garin, who led the last stand of the Rhoynar against the Valyrian Freehold and cursed them after being defeated, risen from the underwater ruins of Chroyane. The Shrouded Lord is also said to have been a statue at first, until a "grey woman" from the fog kissed life to it with lips as cold as ice. The more realistic story about the Shrouded Lord is that there have been numerous ones, and when one dies another Stone Man takes his place; the current Shrouded Lord being a corsair from the Basilisk Isles.
In the novels Tyrion's boat is attacked by Stone Men while making its way through the ruins of Chroyane on the Rhoyne, but the TV series condensed this to occur when he is traveling through the ruins of Valyria. In the novels Tyrion's ship didn't pass directly through the Smoking Sea, instead going the long way around, as he could see the ominous glow of Valyria's fires on the distant horizon.
In The World of Ice and Fire book it is stated that there are rumors of men still living among the ruins of Valyria and of its neighbors Oros and Tyria - who might, similarly, be stone men pushed to live on the abandoned fringes of civilization - but these rumors are unconfirmed in the novels. This might imply there are some in "Valyria" - in the sense of the overall Valyrian heartland in the southern half of the Valyrian Peninsula, though Old Valyria itself is considered very dangerous for anything living. Thus it is entirely possible that the Valyrian Peninsula is also used as a leper-colony for Stone Men in the novels, and changing Tyrion's encounter with them to Valyria in the TV series isn't implausible. Valyria itself is too dangerous for anyone to travel through - though it's possible that in the TV series it was only described as "Valyria" in the general sense for the entire region; after all, "Old Valyria" on maps is located in the middle of one of the new islands formed in the Doom, not along the coast of the Smoking Sea where a boat might pass - but Oros and Tyria are ruins located along the coasts.