The main issue is safety, and the primary problem arises when there is too great a buildup of creosote on the interior walls of the chimney from use. When there is such a buildup, there is a potential for a chimney fire, in which the creosote on the walls of the chimney catches fire. This can cause fire to shoot out of the chimney and you run the risk of setting fire to the building.
The National Fire Protection Agency recommends a yearly inspection and cleanings and repairs done as necessary. Homeowners with Oil fired furnaces and/or Woodstoves used to heat the home should have a professional at their home every year. Fireplaces need to be maintained and cleaned on a consistent basis as well. Usually chimney cleaning needs to be done every year before use.