In a short time nest-building commenced, and among the materials collected by the birds, I observed a long gray moss or lichen, and thought it might very possibly be the same which, in the West Indies, is gathered from the bark of old trees, where it grows, and hangs in great tuft-like beards, to be used instead of horse-hair for stuffing mattresses.
My wife no sooner heard of it than her active brain devised fifty plans for making it of use. Would we but collect enough, she would clean and sort it, and there would be no end to the bolsters, pillows, saddles, and cushions she would stuff with it.
For the discovery of nutmegs we had also to thank the pigeons, and they were carefully planted in our orchard.
In a way, this is obviously my source -- nutmegs, plural, are discovered via birds, and lichen is nearby -- but lichen is singular, and there is no indication that the birds were intoxicated by the nutmegs. But it seems highly unlikely that any other novel would borrow the very specific plot point of birds helping the protagonists discover "nutmegs," and I don't see how I could have misremembered it. The idea that nutmeg could be intoxicating was a new idea for me, learned from the book, not something I could have read into a book that does not mention it.
(Incidentally, was my nutmeg-induced idea that I was "covered with moss" somehow influenced by this passage as well?)
According to Wikipedia, "Over the years, there have been many versions of the story with episodes added, changed, or deleted," to the extent that "Wyss's original narrative has long since been obscured" -- so were all those books I read just different versions of The Swiss Family Robinson? And was the version on Gutenberg bowdlerized so as not to suggest to impressionable young readers the idea of trying to get high on household spices? Now I'm going to have to spend some time trying to track down the version I read as a child, the one with the nutmeg-intoxicated bird.