When I used this option in the past, the sender received no failure message while still there was no reception of the mail. So I had it unchecked for the senders to have some notification that we haven't received their messages.
Now it somehow solved the issue and I received my test message...