Those tall stalks growing out of the moss are called the sporophyte body, which comprises a long stalk, called a seta, and a capsule capped by a cap called the operculum. This is where the spores are formed.
Those sporophyte bodies have a strange, bird-like appearance when you look at them close up. They look like long-necked birds. See the cool little macro bokeh in the middle of this picture?