I believe the answer is: Closeness of the relationship
People on average had a 'barrier' that we put when we socialize with other people. We would let people pass this barrier after they somehow earn our trust. From the example above, the coworkers haven't yet pass this barrier, which make us more likely to be suspicious on every gesture that those coworkers made (even if they are truly genuine)