If build times are an important factor for you, there are certain steps you can take when preparing your disks to optimize them even further:

• Place the build directories on their own disk partition or a fast external solid state drive.
• Use the ext4 filesystem but configure it not to use journalism on your Yocto-dedicated partitions. Be aware that power losses may corrupt your build data.
• Mount the filesystem in such a way that read times are not written/recorded on file reads, disable write barriers, and delay committing filesystem changes with the following mount options:

noatime,barrier=0,commit=6000  

These changes reduce the data integrity safeguards, but with the separation of the build directories to their own disk, failures would only affect temporary build data, which can be erased and regenerated.