The first example to look at would be Dismember. This Swedish death metal band made a couple of classic albums and then slightly fell into foreign territory, making the more melodic and traditional-sounding Massive Killing Capacity. The album really was not bad at all, yet there was some backlash and the band took a bit of a break after that. Once they regrouped, they returned to their roots with 1997's Death Metal and have maintained their normal style ever since, with success.This proves that it is possible to make a misstep and to then go back and find your path. But this is not always the case.
Take a band like My Dying Bride, for instance. They slowly evolved into something else, after their first few albums. Instead of playing Death/Doom, they played a strange mix of Gothic/Romantic sounding music that had ties to traditional doom metal, but also to something else entirely. By 1998, they had completely forsaken metal, altogether. The following year, they decided to go back to their roots and to bring back the style of music that they had pioneered in the early 1990s. It was not quite the same, but the effort was admirable, nonetheless. However, after a couple records, they began to stray back into more experimental areas and so the “return” was only temporary and was not entirely successful.
Then, you have bands like Hypocrisy that tried to make a return to the old days, more as a gimmick than anything else, but totally failed to capture that magic or to even maintain that effort throughout the whole album (2000's Into the Abyss). Taking this a little further, one can look at Metallica and wonder how the hell they had the nerve to claim that they were returning to their roots, when it is painfully obvious that they have absolutely no understanding of how they used to operate. It is difficult to believe that a band can lose touch with themselves to such a degree that they are unable to even come close to creating something that is remotely similar to their classic material, as if they simply cannot grasp what they accomplished in the past.
And, finally, one can look to the almighty Darkthrone and rejoice in that they have such a deep knowledge and understanding of metal that, when they want to take their sound more old school, they completely bypass their own earliest efforts, such as A Blaze in the Northern Sky and Under A Funeral Moon, and go back to the roots of black metal. Few others have shown such comprehension of how metal is supposed to be, and it will be a sad day when those two decide to stop making music.
In the end, most bands should follow the example set by Carcass. Once you are going too far beyond what your band is supposed to be, simply close up shop and start under a new name. It might be risky, sales-wise, but it is the only appropriate thing to do, with any respect to the music itself.