Spotify's Heavy IO Problem
Recently, people started to realize that the Spotify desktop client was writing way more data than it should reasonably do. It could easily go up to 200 gb on my personal laptop after a day running in background, without playing music. Yes, you read well, without playing music. Let's have a look at it.
Update: 11 nov. 2016
We received another reply, telling us that the 1.0.42 update has successfully been rolled out to all users. Many users on the thread/issue are saying that this update brought the necessary changes to solve this issue. But it should be noted that some users are having some troubles installing the update.
Update: 10 nov. 2016
We got a reply from a community manager saying that they fixed the problem in version 1.0.42. Let's wait for the update and check if the problem has really been solved or not.
The problem
Some people, including me, have seen that spotify has been using a fairly high amount of disk writes lately. While we don't really know the source of those writes, we could monitor the writes to to find out which files were affected by those writes.
Here are some cases reported by users (thank you!) of the Spotify community (see the links at the end of this article for the threads were these pictures were originally posted). Please bear in mind that we don't have Spotify's uptime on each system, so we can't really compare them.
On a Mac :
On Windows :
As you may have already seen, the amount of IO Write Bytes (Bytes Written on mac) are fairly high. If you look well you may see on the Windows screenshot that spotify already wrote 640 GB. To compare, the whole GTA V game (which is already quite big) is 62 GB, which is 10 times less that what Spotify wrote.
But let's be more precise about what IO Write Bytes means on Windows. This counter is not only counting disk writes, but also network writes and other kinds of communication. So you may say, that it may be normal for these counters to take such values. But that's not. In fact, if you look closely, you see that these number are actually orders of magnitude higher compared to the other programs running on the system.
I/O Read Bytes - The number of bytes read in input/output operations generated by a process, including file, network, and device I/Os. I/O Read Bytes directed to CONSOLE (console input object) handles are not counted. (source)
After some research using the resources monitor (resmon.exe), we could find the culprits. Two files, named mercury.db and mercury.db-wal, are getting the largest share of writes. But what are those files ?
/mnt/r$ file mercury.db
mercury.db: SQLite 3.x database
Following the file utility, these files seems to be SQLite databases. Let's have a look at this database with "DB Browser for SQLite". The following pictures shows that the database is indeed a valid one, with three tables: lock_ids, locked_keys and storage.
The mercury.db-wal file is special file used in conjunction with the main database file. For more informations about this topic, please have a look here. But why are those simple database file so much written to ? Well, after some research done by members of the community, it seems that Spotify is calling periodically the "VACUUM;" SQLite function. As you may already think, the VACUUM command is used cleanup unused space on the disk.
The VACUUM command rebuilds the database file, repacking it into a minimal amount of disk space
Here is an excerpt for the documentation:
The VACUUM command works by copying the contents of the database into a temporary database file and then overwriting the original with the contents of the temporary file. When overwriting the original, a rollback journal or write-ahead log WAL file is used just as it would be for any other database transaction. This means that when VACUUMing a database, as much as twice the size of the original database file is required in free disk space.
This suggests that VACUUMing the SQLite database is going to require a lot of IO operations. But this wouldn't be a problem if this wasn't executed as often as it's currently done. Some other member of the community was able to draw a graph, that shows the IO operations over the time:
By looking at this graph, we're able to deduce the time interval between each write. With a rough estimation, it seems like it's done every 5 minutes. 5 minutes means 12 times an hour (60 / 5 = 12). Depending on your mercury.db size, this may represent a big amount of data actually being written to your hard drive. In fact, it's exactly 24 times the size of database file according to the documentation of VACUUM : 12 vacuum per hour * 2 * size of the database = 24 * size of the database. In addition to this, we may add the normal operation of spotify :
- Downloading music & caching, including the accesses to the database for the records about them.
- Synchronizing different stuff (playlists, what our friends are playing, ...)
- Looking for updates
- ...
We could point out what is the problem with the Spotify desktop client right now. But you shouldn't worry too much about the issue, and continue your playback as before. If you want some temporary solutions to the problems, jump to the "Possible solutions" section of this article.
Don't worry too much
Let's do some calculus, to find how much the life of your SSD is going to be impacted. First, we have to define our environment :
- A laptop open 8hours a day with spotify running in background playing music
- Spotify writes 9 GB per hour (upper limit of user's reports)
- We have a 1TB drive with a 150TB warranty (Samsung 850 Pro)
Using spotify 8 hours a days makes the daily amount of writes to be : 8h * 9GB/h = 72GB
The number of days you can do so before hitting the warranty is : 150000GB / 72GB = 2083 days
2083 days mean you could do this continuously for 5 Years, 8 Months and 2 Weeks before hitting the warranty because of spotify.
In pratice, the 1TB Samsung 850 Pro could be pushed to 9PB (that is, 9000TB) before failing. That gives you a lot of time before frying it :)
That's why you shouldn't worry too much. While the amount of data being written to your drive is high, frying it is definitively going to take some years, if not more. But that doesn't justify, and even less means that we should accept such a behavior from software companies.
Possible solutions
There are some solutions available for this problem, that might help while we're waiting for some patch to be released. I will show 2 of those :
- Deleting the database
- Redirecting spotify's database to a ramdisk
There are other ones that I could find on the community threads, including modifying spotify's executable directly, or using some program found on the internet, but I definitively do not recommend to use these methods. Even if they work, nobody knows the downsides of doing so.
Both solutions involve playing around with Spotify's database. Don't worry, all your personal data (authentication, and some other stuff), are located in another directory. That's why after using these methods you're not logged out, or don't even loose what you were playing. We're just going to move / remove cached data.
Solution 1 : Delete the database
The first solution is to try to delete the database. In fact, over time, spotify's database gets messy, and it takes more writes for it to maintain it. A good solution is to just delete the folder, and have spotify generate a new one. Please note that you should completely close spotify before deleting the database, to avoid problems.
Deleting the database is easy.
On windows, just delete all the contents of the C:\Users\<username>\AppData\Local\Spotify folder. On macOS, delete the content of this folder ~/Library/Caches/com.spotify.Client/storage, althrough i'm not really sure about it, because i don't own a Mac. If you do, please let me know.
Here are some findings by dnsbty on spotify's community using this solution:
Before reinstalling:
5 mins idle:
95.9 MB written, 114.1 MB read
5 mins listening to playlist:
207.6 MB written, 24.3 MB read
After reinstalling:
5 mins idle:
3.9 MB written, 19.6 MB read
5 mins listening to playlist:
23 MB written, 39.1 MB read
Solution 2 : Use a ramdisk
The idea is simple : if we can't do anything about the writes, let's move them somewhere harmless. That somewhere would ideally be the RAM, that can handle this amount of writes easily (at 25.6 GB/s, it's a piece of cake for it). The downside is that after each reboot we'll get a new database, which means that you can't have offline playlists (but there's a solution for that, too, more on that later).
==Little warning : To use this solution, I strongly suggest you to have 8GB or more of RAM installed in your computer. Not having enough ram could lead to Windows being short on memory, thus swapping too much on the SSD, which is what we're currently trying to avoid. ==
For this, we're going to use a ramdisk. For those who doesn't know what it is, it's a way of creating a small hard drive that lives in RAM. Because of it's nature, if you reboot, it's content get lost. But don't worry, you may configure it to be persistent. The following is for windows only, but may be easily ported to Mac.
Here are the steps needed to go through this :
- Download and install ImDisk
- Create a folder with a base structure for the ramdisk
- Create the ramdisk and configure it
- Setup the symlink
- Check that everything works
1. Download ImDisk
You may find the latest version of the imdisk toolkit on sourceforge. After having downloaded the version that matches you windows architecture (x86 or x64), run the installer.
2. Create the base folder for the ramdisk
Now that we have installed the ramdisk, we must create a base structure for the ramdisk. This structure will be load when the system starts into the ramdisk. It's necessary to have it as the symlinks we're going to use later need their folders to exists. Otherwise, they just doesn't work (trying to go through them displays an error). Here are the folders that you should have :
RamdiskTemplate
└───Spotify
You may create this wherever you want. Just make sure not to delete it.
3. Create the actual ramdisk
After installing the ImDisk, you should see a new icon on your desktop named "RamDisk Configuration". Open it. There you should see :
Choose the size to be whatever you feel is right. I choose 4GB because I'm also putting my temporary files there, and doing some work inside. I strongly recommend that you check the "Allocate Memory Dynamically". Doing so will ask the driver to use only the memory needed to store the files. If have 100MB inside it, it will use around 100MB of RAM, with the upper limit being what you set just before. Remember the "Driver Letter" that you choose, you'll need it later. You can also configure it to be used as your temp, but i'll let you have a look around on your own.
Then, you should click on the "Data" tab.
Here, under the "Load Content from Image File or Folder" label, indicate the path were you RamdiskTemplate folder is in you system. On mine, it's in D:\RamdiskTemplate.
If you want Spotify's database to persist after a reboot (and keep offline songs), then you should use "Synchronize at System Shutdown". Basically, just check Synchronize at System Shutdown and Delete data removed from the RamDisk. Using these options will make the ImDisk driver to copy it's content into the RamdiskTemplate folder when the system stops. But there's a downside of using this : when the system boots and the ImDisk driver starts, it will have to copy all the data store in that folder back into memory, which may take some time depending on the number of files and their sizes. It will also take some time for it to synchronize the ramdisk with the RamdiskTemplate folder.
When you're done choosing your settings, click "OK". This will trigger the creation of the ramdisk, and will save your settings. If you see a warning about a volume being unmounted and data lost, don't worry about it, you can safely click OK. Then, an information window will open, and you can safely close it too. Now you're ready to go on.
At this point, you should have a new drive showing up in the explorer, having the size that you choose before. If you don't, please follow carefully the instructions. And if it's still not working, then ask me in the comments.
4. Setup the symlink
Now that we have a new and shiny ramdisk, fully working, we should ask spotify to put his files inside. But he's not really a gentleman, and doesn't have the necessary options. (It does have an option to move the cache, but that doesn't move most of the files that are involved into the high IO). So we have to use a special trick : the symbolic link. Yes, my dear unix friends, windows got that too ;)
First, remove the spotify folder in your AppData\Local. Be careful, do not remove the Spotify folder in the AppData\Roaming folder ! If you do so, you'll have to reinstall it.
Then open an "Administrator" command line. There are many ways to do this, but the quickest is to press Super+X (Super is usually the one with the windows logo). In the menu that appears, click on "Command Prompt (admin)" (I'm sorry, my laptop is in french).
Once you're done, you should get something like this :
Then, enter the following commands (replace with your username, and the drive letter from step 3):
cd C:\Users\<your username>\AppData\Local
mklink /d Spotify <DriveLetter>:\Spotify
If everything is correct, then, if you go again inside your AppData\Local folder, you should see a new Spotify folder. If you open it, it should be empty, and no error must be displayed. If this is your case, then you have successfully redirected spotify to your ramdisk !
5. Test
This is the last part and the easiest one. Just launch spotify and listen to some music. Then open your ramdisk : <DriveLetter>:\Spotify. If this folder is not empty, ie looks like the next picture, then you're done, it works !
Results
After running with this solution (ramdisk) for some hours with Process Explorer running in background, here are the results I got. As you can see, only 500 KB were written to an actual hard drive. All other writes are being redirected to the ramdisk, hence the really low amount of residual writes.
Links to the posts on the Spotify community
This is a temporary fix, but it might help some people waiting for a patch to be released. I can't be held responsible if something went wrong on your computer (Usually nothing goes wrong), you’re doing this at your own responsibility.