Smaller Deus Ex Savegames

A refuge for those migrating from the fallen DXEditing.com and a place for general discussion relating to Deus Ex editing (coding, mapping, etc).
Post Reply
fk.
Mole Person
Posts: 4
Joined: Sat Apr 07, 2012 6:44 pm

Smaller Deus Ex Savegames

Post by fk. »

Hey guys,
so, I stumbled upon the TNM 1.0.5 thread where someone suggested to make savegames smaller — which reminded me that I still had some code lying around from when I was convinced saving and loading in Deus Ex was disk-bound and could be made faster by writing less data (hint: saving and loading in Deus Ex is not disk-bound and can not be made faster by writing less data). I kinda dropped it at the time because it didn't do what I wanted, but it does make savegames smaller so I decided to polish it up some and publish it.

Code: https://bitbucket.org/fk/dx-savegamecompression
Binaries: https://bitbucket.org/fk/dx-savegamecom ... on-0.1.zip

In short, it decreases on-disk size of savegames simply by:
  • transparently compressing the files using Deflate (it's actually a bit more effective than NTFS compression ;)
  • creating hard links instead of copying files. Deus Ex happily copies and duplicates files when loading and saving, so by not actually copying them, you can reduce the amount of non-shared disk space per savegame (and in an optimistic scenario, that's actually quite a lot).
I'll just quote my advertising blurb from the readme:
Let's have an example: I had a savegame in Hong Kong with an uncompressed size of ~24.4 MB. Fully compressed, that same savegame had a size of ~7.7 MB — just about a third of the normal size. But wait, it gets better: That savegame is on the Canals map, the file for which is ~1.5 MB big. Let's assume you'd load that savegame, then immediately quicksave. In this case, the only files that need to be changed for that quicksave are 06_HongKong_WanChai_Canal.dxs — the state of the Canals map — and SaveInfo.dxs (compressed about 13 kB); all other files are shared with the original savegame, so ultimately the new quicksave only uses about 1.5 MB of unique diskspace instead of ~24 MB. So in short, you get a ~66% reduction for free and potentially additional savings. It's quite neat, if I may say so.

There are some additional caveats that aren't (yet?) mentioned in the readme:
  • Hardlinking will probably not work if you use Kentie's launcher (because of how it actually stores savegame files in a different location); I didn't test it yet but I think it'll crash when you try to save or load.
  • I did some non-reliable speed tests — loading a compressed savegame seems to be as fast as loading a normal one; saving times /seem/ to be slower however (though saving just took 1-2 seconds instead of <1 in my test scenario). I have an idea why that might be and how it could be fixed.
Also, one question for more experienced Deus Ex modders: To enable my file handling redirection, I need to run some code as early as possible when Deus Ex launches. Right now, I'm using a DeusExGameInfo subclass for that, which works, but would require support for every mod that uses its own GameInfo class. I have one for TNM, but I was wondering if there's some other trick which doesn't collide with any other mods. I've considered just putting it into DeusEx.u (which would collide with Shifter and relatives; plus TNMGameInfo doesn't even extend DeusExGameInfo) and using a custom launcher (which seems overkill). Any other ideas?

Feedback and bug reports are welcome. I hope somebody finds this stuff useful ;)
ggrotz
X-51
Posts: 780
Joined: Tue Nov 10, 2009 12:55 am

Re: Smaller Deus Ex Savegames

Post by ggrotz »

fk. wrote: [*] Hardlinking will probably not work if you use Kentie's launcher (because of how it actually stores savegame files in a different location); I didn't test it yet but I think it'll crash when you try to save or load.
From studying Kentie's remade DeusEx.exe, I wouldn't think it would be much of an issue, since all it involves is a repath to change the save games location. You can do that from the INI files, even as I did with this loader. The question is whether your code assumes a hard path or not.
fk. wrote: Also, one question for more experienced Deus Ex modders: To enable my file handling redirection, I need to run some code as early as possible when Deus Ex launches. (...) Any other ideas?
When I wrote that launcher, I played with some other things. One of those was modifying how certain functions occur with the DX executable. As long as you know what function you want to target, you can do certain things with it. As life does what it does, and time gets limited and I needed to move onto other things, so I really didn't bring any of the possible feature ideas I had in doing this to light.

I downloaded all the stuff you did and will definitely have a look at it.
fk.
Mole Person
Posts: 4
Joined: Sat Apr 07, 2012 6:44 pm

Re: Smaller Deus Ex Savegames

Post by fk. »

Huh. I hadn't seen your loader before. I'll have to try that out sometime.
ggrotz wrote:
fk. wrote: [*] Hardlinking will probably not work if you use Kentie's launcher (because of how it actually stores savegame files in a different location); I didn't test it yet but I think it'll crash when you try to save or load.
From studying Kentie's remade DeusEx.exe, I wouldn't think it would be much of an issue, since all it involves is a repath to change the save games location. You can do that from the INI files, even as I did with this loader. The question is whether your code assumes a hard path or not.
I tested it; it crashes ;) The problem is: The Unreal Engine does all its filesystem accesses through a specific global pointer (GFileManager) which points to an instance of the FFileManager class which provides filesystem access functions (read file, list files etc.). Kentie's launcher replaces that with an object of its own that redirects any writes to the user directory whereas my code uses it to add compression/decompression and create hard links instead of copies.

Now, I knew the launcher worked that way (that's actually where I got the idea) so my file managers wrap the original one and redirect almost all operations to that. When a save file is written, I just add some compression and let the original file manager handle the writing; and that's either Unreal's or Kentie's (which would change the path that my code saw to a new one). When I have to create a hard link, I can't redirect that to the original file manager because it has no operation for that, so I'm handling that myself without ever calling it. Now, some savegame files (those that are newly created) end up in My Documents, but those that are just copies of older ones won't have their paths adjusted in that way (because they're hard links), so if Deus Ex now tries to copy a savegame file that is actually in My Documents, it'll crash because I'm trying to create a hard link for the "virtual" path in DeusEx/Save (and you can't create hard links to files that don't exist).

Man, I suck at explaining those things. I hope it's kinda clear. Anyway, point being, unless I implement some specific hack for Kentie's launcher, hard links won't work with it. I guess I'll just create a copy if the hard link fails. Should also help on FAT filesystems and at least it'll work correctly. But that's honestly all I can think of.
fk. wrote: Also, one question for more experienced Deus Ex modders: To enable my file handling redirection, I need to run some code as early as possible when Deus Ex launches. (...) Any other ideas?
When I wrote that launcher, I played with some other things. One of those was modifying how certain functions occur with the DX executable. As long as you know what function you want to target, you can do certain things with it. As life does what it does, and time gets limited and I needed to move onto other things, so I really didn't bring any of the possible feature ideas I had in doing this to light.

I downloaded all the stuff you did and will definitely have a look at it.
Thanks for looking it over.

"As early as possible" is actually too strong; the way I'm doing it now, it gets run when the main menu loads (or more specifically, the first map). That works totally fine, but it needs a specific loader for most DX mods. I guess unless someone comes up with brilliant alternative ("hijack DeusExGameEngine"), I'll keep it this way since it works for vanilla and TNM, which should be fine. Or I convince all these mini-overhauls plus TNM to include it O:)
ggrotz
X-51
Posts: 780
Joined: Tue Nov 10, 2009 12:55 am

Re: Smaller Deus Ex Savegames

Post by ggrotz »

I see. I did notice Kentie's EXE hard coded this when I went back last night to look at it again. That might be part of the issue involved. As for the INI reference I speak of, it's "SavePath", under the "[Core.Engine]" section of the DeusEx.ini, if that helps. One usually should be able to change this and get the same effect of moving saved games without having to change the core code, assuming something hasn't been drastically changed. Then, others should be able to read this to get the path of where the saved games go. The problem comes when you move INI files, as you have to path those through the command-line parms if you don't hard-code positions. The launcher, as I wrote it can pass the command-line parms, so there isn't an issue there. To do a stand-alone EXE, however, you have to change the paths (again) within the code.

Now, you might see I wrote the following:
ggrotz wrote: When I wrote that launcher, I played with some other things. One of those was modifying how certain functions occur with the DX executable.
I looked last night for this as well and still had the old project. It uses a "detours" like package with DLL injection, so you might look at the "detours" package for what you are doing. The whole method only works with NT based operating systems. I modified it to monitor API file functions likely for save games and ended up with something that NTFS compresses saved games by default. Yeah I know it's a pretty cheap thing since you can just go into explorer and do that with the Save games directory, but was simple enough to test the concept. I only tested it on DeusEx and TNM, and I'm not sure what anti-virus and other similar programs would do with such files. But it seems promising enough.

Edit: I had the idea to play around with hard links and wrote something that went through 60 TNM saved games. Before is before I started, After is after I run the program. That's 3,274,899,456 bytes (3.1 GB) that got saved on an NTFS drive for deleting duplicate files and creating hard links in their place. Problem is running this program took a little while - you have to establish that the file is indeed a duplicate before you do the process. Then, drive size before and after is the only real quick way (there's another) to determine a change, since the hard links behave exactly like files. Still been testing the process, but it should provide you a little more marketing fodder.
Attachments
Hardlink-Data.jpg
Hardlink-Data.jpg (11.61 KiB) Viewed 10205 times
ggrotz
X-51
Posts: 780
Joined: Tue Nov 10, 2009 12:55 am

Re: Smaller Deus Ex Savegames

Post by ggrotz »

I came up with a routine to tell how much space is actually being used with the save games. In the example above, it was 900MB which NTFS compressed to about 445MB. About a 85-90% discount.

So if there's a motivation to get less disk space used by the save games, this might be interesting to drive on out to a final product. Only thing is that most of us are running drives that are big enough that it's probably irrelevant now. Is there any interest in doing that?
fk. wrote: I still had some code lying around from when I was convinced saving and loading in Deus Ex was disk-bound and could be made faster by writing less data (hint: saving and loading in Deus Ex is not disk-bound and can not be made faster by writing less data).
Just noticed this. I noticed when I monitored "CreateFile" how much thrashing it does looking for files in different paths. But probably the best optimization (self-reminder: Test this) is to minimize the number of "Paths=" lines necessary. Which might mean trying to get New Vision and HDTP to stand on its own. But indeed, most of the time is in writing the *new* save files, the rest is disk-bound in CopyFile and MoveFile.
ggrotz
X-51
Posts: 780
Joined: Tue Nov 10, 2009 12:55 am

Re: Smaller Deus Ex Savegames

Post by ggrotz »

I went ahead and finished out the program that I've been using to produce the stats in this thread. Be sure to read the accompanying text file and to heed the warnings if you try it out. It shouldn't cause any problems with Deus Ex saved games, but it might if you run it against anything else. So be careful.

Edit: I replaced the old version with a new one. Only change is that it reports back in MB now instead of bytes.
Attachments
DXSaveGameOpt.zip
Deus Ex saved game space optimizer. Deletes duplicate files and replaces them with hard links. Optionally NTFS compresses files as well.
(124.93 KiB) Downloaded 469 times
fk.
Mole Person
Posts: 4
Joined: Sat Apr 07, 2012 6:44 pm

Re: Smaller Deus Ex Savegames

Post by fk. »

How do you humans say? "Better late than never"? Anyway, I finally made that change that I mentioned soooo long ago, so no more crashes if you don't use NTFS or do use Kentie's launcher.
New binaries: https://bitbucket.org/fk/dx-savegamecom ... -0.1.1.zip
ggrotz
X-51
Posts: 780
Joined: Tue Nov 10, 2009 12:55 am

Re: Smaller Deus Ex Savegames

Post by ggrotz »

fk. wrote:How do you humans say? "Better late than never"? Anyway, I finally made that change that I mentioned soooo long ago, so no more crashes if you don't use NTFS or do use Kentie's launcher.
New binaries: https://bitbucket.org/fk/dx-savegamecom ... -0.1.1.zip
Cool thanks.
Fytrop
UNATCO
Posts: 160
Joined: Mon Jan 02, 2023 2:23 pm
Contact:

Re: Smaller Deus Ex Savegames

Post by Fytrop »

I think not. After all, reviews can be subjective and not always accurately reflect the real picture. Here is top cpa offers . When choosing an affiliate program, you need to consider many factors, such as the company's reputation, partnership conditions, availability of resources and support, as well as evaluate how this program fits your goals and your audience.
Post Reply