Not necessarily. Many people kill all their fish, goldfish included, by using chlorinated water. What you are saying just isn't always true. Even if by some chance they might survive, how can you know that you haven't done permanent damage to their gills and compromised their health?
Admittedly, goldfish are tough and may not have obvious issues with your particular chlorinated water, but chlorine can shorten their lifespans.
As has already been stated, the amount of damage can depend on the concentration of those chemicals in your particular water supply. Making generalized statements is irresponsible, in my opinion.
It can also depend on whether or not the water treatment is chlorine or chloramine. Chloramine adds ammonia to chlorine and ammonia can be toxic to fish. Chloramine also does not gas off like chlorine does. So it pretty much requires using a product to neutralize that chemical for the water to be safe for fish.
The fact that you need to clean your filters every day in summer is an indication that you don't have enough filtration for the fish load you have. Have you ever tested your water?
Filters should not need to be cleaned that often. But if that suits you, so be it. Everyone has their own way of doing things, but most of us would prefer not to be cleaning filters daily. And some of us actually take vacations when daily cleaning would be impossible.