With yum install xfce4, kde, gnome or whatever, the needed xorg packages will come along automatically.
Using the command: yum grouplist , I see the group XFCE which may be the easiest route to take.
yum groupinstall XFCE
You could also run the command: yum install xfdesktop
but, you may not get everything you want.
I like to dump yum list to a file so I can look it over....
yum list > list.txt