It is often the case that one person's action affects the wellbeing of another person, despite no intent to do so on the part of the first person. Economists call this effect an externality, and divide them into positive externalities, which benefit the affected party, and negative externalities, which harm the affected party. Goods which produce positive externalities are called merit goods; goods which harm the affected party are called demerit goods. Examples of merit goods include vaccinations (due to the reduced risk of passing diseases onto other people), foods which produce pleasant aromas, and attractively painted houses. Examples of demerit goods include any industrial processes which produce pollution, late night music practices, and driving during rush-hour (which contributes to congestion).
A related concept is the rights infringement (or rights violation; there is a subtle difference, but that shouldn't be important here). This constitutes one person breaching a certain protected sphere around another person; example would include assaulting someone, stealing from them, and damaging their property.
The first point I want to make is a simple one: that negative externalities and rights infringements are different things. Painting your house purple with the effect of reducing the value of your neighbour's house is not usually a rights violation (though it would be if you had signed a contract not to do such a thing) but it is a clear case of a negative externality. Trespassing may often be a rights violation without actually producing an externality to the landowner.
My second point is also simple, even obvious: in general, it is morally wrong to inflict a negative externality upon someone even if you are not actually violating their rights. There are exceptions to this, and it's not deontologically wrong in the way that a rights violation might be, but if you make a habit of inflicting negative externalities then basically you're kind of a dick.