It really depends on several things. It sounds like you are on city water and they don’t do a great job of treating the water. Do you want to treat just water at something like the kitchen sink or are you looking to treat the entire home? Something for just one sink is typically relatively cheap and is something you can easily put in yourself.
If you are going to install a system for your whole house, I’d say you have to consider what you want to do. The obvious best answer is a Reverse Osmosis (RO) unit. This will take care of all your needs of removing chemicals and hardness. The drawback to the RO units is they are expensive and somewhat costly to operate. Water flows into a tube that has a membrane in it. As the water flows through it will, of course, try to flow through the membrane. But the membrane is semi-permeable allowing only Hydrogens and Oxygens to get through (water only). All chemicals including calcium, chlorine, fluoride, etc are flushed on through. The problem with RO is there is a lot of water wasted usually.
After that, you have some choices. Here’s what you will see advertised:
Physical filter: usually a cloth wound tube. Water flows around it and passes through the cloth, filtering out solids. Most systems will have something like this and it is a good idea even with city water. Dirt, grit, metal flakes, etc can all be pulled out with this. The filter cartridges are easily replaced and are relatively cheap. This will not remove any chemicals and will not impact the hardness.
Carbon filter: This is just what it says. It is a tube of granular activated charcoal. The charcoal granules are very porous increasing the amount of surface area each granule has. The water flows over and through these granules and it is here that the chemicals are pulled out. That is the purpose…removing chemicals and organics. These cartridges last a fairly long time (depending on how polluted the incoming water is) and are usually easily replaced at a fairly low cost.
Mixed Bed resin filter: Mixed Bed resin is a mixture of anion and cation resin beads. These are polycarbonate beads, very small, that have an affinity for exchanging out anions and cations in your water. It may be sold as H-OH resin…Hydrogen-Hydroxide. As water flows in, the ions in the water are interacting with the beads. The beads have exchange sites (at the molecular level) where something like a chlorine ion (Cl-) will be tied up on a bead that will give up a hydroxide (OH-) ion in its place. Something like a sodium ion (Na+) will interact with the other type of resin, getting tied up resulting in a hydrogen ion (H+) being released. If course the H+ and the OH- ions will get back together, becoming H2O. These resin columns are fairly expensive, usually somewhat difficult to replace, but usually (depending on crap in the city water) will last quite some time.
UV treatment: Ultraviolet light (UV) is often used to “polish” the water. The water, after being filtered and cleaned goes through a place where it flows around a UV light. The radiation coming from the UV light will break up any organics and kill any bacteria/viruses in the water. Many treatment options have this. The lights last a fairly long time, can usually have some sort of indication if the bulb burns out, can be replaced fairly easily as well though depending on the size of the bulb it could be somewhat costly.
So you need something to pull out solids, something to remove organics and chemicals something to clean up the water and something to sanitize it. That is what you would need if you want water that is mostly H’s and O’s and nothing else. Depending on your needs, there are other options. Water softeners are usually in a system to remove the hardness. It works much as an ion exchange as the aforementioned MB resin, though usually it is limited on what will be pulled out. It is literally a salt bed. Calcium and magnesium are the biggest hitters on hardness and these will be removed sending more sodium into your water. Cheap and easy, salt isn’t particularly expensive, but it is sometimes a pain in the ass to replace/refill the column.
My recommendation would be to talk to a professional, especially for installation of a whole house unit. Plumbers are far more familiar with code and know many hard-learned tricks to avoid problems later on.