Let A = "3rd malfunction occurs on 5th day", and B = "not 3 malfunctions in first 3 days". Then we want p(A|B) = p(A&B)/p(B). In this problem, A is a subset of B, so p(A&B) = p(A).
For p(A), we have ...
There are 6 ways to have 2 malfunctions in the first 4 days. The probability of that is 6*(.4^2)(0.6^2) = 0.3456. Then the probability of the third malfunction on the 5th day is 0.4 times that,
.. p(3rd malfunction on 5th day) = 0.13824
For p(B), we have ...
There is only one way to have 3 malfunctions in the first 3 days. The probability of that is 0.4^3 = 0.064. Then p(B) = 1 - p(3 malfunctions in 3 days) = 0.936.
p(A|B) = p(A)/p(B) = 0.13824/0.936 ≈ 0.14769