OK, one last install directory tweak: If I create the archives myself, then I can control the offset from the local file header to the file data. That means that Stellaris never has to read the local file header at all, while the archives are still valid zip files that can be opened by a regular archiver too. That more than halves the work required to read the archive, since the local headers are scattered throughout the file, whereas the central headers are clustered at the end. Since building the archives myself requires an installer for the patch, I've built it into a WIP save/mod editor I was working on.

20th of June 2017

The user directory isn't too bad - with yesterdays patch, we have 174,017 IO operations. The bulk of these are searching for files that are in the archives and don't exist in the user folder, since the user folder has a higher search priority. I know I have no files in there that override base game files, so tweaking the search order cuts IO operations to 79,570, and cuts startup time by another 11%. But that's not the best of ideas, because the user directory is supposed to override base game files... A better solution would be to keep a cache of folders that don't exist in a location, to save trying to look up further files in that folder, but that's too invasive for this small patch.

That's enough playing around here anyway. Updated patch is here. I've left the original up too, because this one needs more careful installation and has a higher chance of going wrong. I should probably also mention that this is all windows only, and probably will only work with the steam version: The same issues will apply to other OS/versions, but they'd need a patch of their own.

19th of June 2017

Continuing to play with Stellaris startup performance, because overkill is fun. Moving out the gfx folder cut startup time by 45%, and adding in a few others increased that to 50%, but I'm sure we can shave a bit more off than that. Profiling shows that we're still spending 40%-ish of clock cycles in the kernel waiting for IO-ish stuff, which means we're still doing IO far too inefficiently.

The obvious first step is to move more files into an archive. My patch doesn't include its own archive functionality*, and relies on code already in Stellaris, (physfs) which limits it to 2GB archives. That's not enough for all the data, so we need to add a second archive. Now we have a gfx.zip, with the gfx folder, and a data.zip, with everything else. That's enough for the whole Stellaris install, and it knocks another 10% off startup time.

Trace is now dominated by Stellaris searching for additional files in the base directory, repeatedly trying to open directories that no longer exist. We know there are no files remaining there, so we can cut that out of the search path. We can also fold tweakergui_assets into data.zip, and remove that from the search path too. That takes another 8% off.

Better, but still room for improvement. Trace now shows that the biggest problem is the way Stellaris is handling looking up metadata in archives. Instead of reading chunks of data, it's reading individual fields from the zip headers, 2 or 4 bytes at a time. Doing a kernel hop to read 2 bytes is wasteful, so we need to stop it doing that. Patching the header reading function to read the whole thing at once and then extract the required fields shaves another 10% off startup time.

That gives a total of 60%, with the number of IO operations in the install directory slashed from 2,217,054 down to 70,457. That's about the limit of restructuring the install directory, so next thing to look at is the extra search paths used for user data/mods/dlc.

*There's a clue in the 4kb size of the injection dll.

18th of June 2017

And a whole year later, I finally did something that's useful enough to warrent releasing! I've been playing* Stellaris recently, and modding means loading up the game lots. Stellaris is pretty slow at starting up, so I spent this morning poking it to find out why. Most of it seems to be trying to index all of the graphics files, so I've made a little patch to shove them all into a single archive instead of leaving them loose on the filesystem. The end result is that my startup time was cut in half and 2GB was shaved off the install size, with no negative side effects that I've discovered so far. I have a mid-range processor and high end SATA SSD, but exact startup gains will vary based on processor speed vs disk speed.

Patch and install instructions here. This is specific to stellaris patch 1.6.2, (d7ec) and will not work for any other version. It does not disable achievements. The usual disclaimer about not having tested this on any computer other than my own applies.

*For my usual definition of 'playing', which involves more time out of the game than in it.

13th of May 2016

Two months with no update? Yup, I've got distracted already. :( SoftDX can currently run fallout 2 perfectly, as far as I've been able to tell, but that's no great achievement; fallout is a straightforward directdraw game that pushes full frames out and does nothing else. I was hoping I'd get at least as far as running gangsters* or thief** before stopping.

Time to think up a new, smaller project that I might actually stand a chance of completing without having to quit my day job...

*A horrible mix of direct draw and gdi. I wrote a patch for it for GOG long ago, but it fails on windows 8+, so it would have been legitimately useful.

**Not legitimately useful, because newdark, but if fallout is my goto simple directdraw game, then thief is my goto simple direct3d game.