Here is the main reasoning for this:  TDD states that a unit test
is written before a unit of code.  Each unit of code will have a
unit test.  When a unit of code needs to be refactored (changed
in structure without affecting behavior), the unit test will preserve
the behavior of the code.  The unit can be refactored quickly, and
the unit tests will assert that the code still behaves as originally
intended.

Consider refactoring without unit tests or with only integration
tests.  First, without tests.  You refactor a piece of code
that doesn’t have a test.  Now to ensure that you didn’t break
anything, you have to run your application with some scenarios that are
sure to exercise the changed code.  This is more time-consuming
that merely running the unit tests for the unit in question. 

Suppose you have _some_ tests for the unit, but they aren’t unit
tests.  The tests involve several other pieces of code as
well.  If all is well, the tests will still pass, but it they
fail, you would have to debug into the test to find out what’s really
going on since the test has a larger scope.  A unit test’s scope
is only that of the code unit, and if the test fails, it pinpoints the
area of failure.