You can certainly download and build all the sources you like - it's part of the joy of Linux.
Yum keeps track of the packages and the files w/in those packages. So as a rule it's a bad idea to install a package by yum and then overwrite the same files by, for example, installing the same package from source. OTOH even that sort of mistake usually works since yum seems to sense that the file has been modified and doesn't replace it with updates (yum insteal renames th eupdate files with a .rpmnew suffix). For any particular package, use yum ,or upstream source, not both.
None are more hopelessly enslaved than those who falsely believe they are free.
Johann Wolfgang von Goethe