I think that found the problem but not sure why it works the way it does.
When xfig is executed it is really a script as below:
if [ -x /usr/bin/xfig-Xaw3d ]; then
xrdb -merge Fig-color 2> /dev/null
cd - > /dev/null
exec xfig-Xaw3d "$@"
exec xfig-plain "$@"
If I just use xfig-plain then it works as it should.