Jake is godfather to Ledger's two-year-old daughter Matilda and the two men will be forever linked as the stars of the landmark "Brokeback Mountain" which earned them both Oscar nominations as two ranch hands who fall in love.

"This has had a strong personal effect on [Jake]," a set source tells People. "He was there, but he wasn't with us. It was obviously a major trauma. These guys were very close. [Jake] was sitting in the director's chair staring off into space."

Jake's mood is a sharp contrast to the happier times on the set of "Brothers." During one intense early prison scene, he jokingly reached into his pocket and took out a picture of his "Brokeback" beau to stick on the prison wall. "Like those prisoners put [loved ones] on the wall, but Jake's was Heath Ledger," one set source recalls. "That was hilarious. It was a nice moment...When you think back on it now, it's touching."