Is Something Wrong with them or are do they just want attention?

Yesterday I cleaned out my goldfish's water and rearranged the things around in it. I put them back in the tank and they seemed to be fine, they were playing around. Then when I came towards the tank all four of them sank down to the bottom of tank until they were sitting on the little pebbles and stopped moving. So I decided to change the tank back to the way it was and they were all back to normal. What is that a sign of?