I don’t know what it would break, but I’m in favor of removing the timestamp from the cache.
This would facilitate zipping up source trees with cache and having them ‘still work’ after unzipping due to the lack of timestamp support in pkzip format.
There’s a discussion about that here