Docosahexaenoic acid (DHA) and eicosapentaenoic acid (EPA), both abundant in fish oil, are known to have significant biochemical and physiological effects primarily linked to the improvement of human health, especially cardiovascular and brain health. However, the incorporation of fish oil into foods and beverages is often challenging, as fish oil is very easily oxidized and can cause undesirable flavors. This review discusses this rapid formation of the fishy and metallic off-flavors, focusing especially on an early stage of fish oil oxidation. Although oxidative stability and quality of commercialized fish oil have improved over the past few years, there is a still a problem with its application: Flavor deterioration can be found even at very low oxidation levels. This review also notes the effective way to inhibit the formation of the volatile compounds responsible for the flavor deterioration.