Nonconforming sign means any advertising structure or sign which was lawfully erected and maintained prior to such time as it came within the purview of this chapter and any amendments thereto, and which fails to conform to all applicable sections and restrictions of this chapter, or a nonconforming sign for which a special permit has been issued.