Nope, that doesnt work.
"That man" 's father is my fathers son. You are your fathers son, and apparently an only child, so it's not a nephew. What about some sort of "step child" situation?
For that man's father, to be your fathers son, that would mean that your father is his grandfather.