General Data Extraction Tools for VNs

[tsuki09_]

Has anyone managed to successfully decrypt Otomate scripts? They have an STCM2L header, but I've never been able to extract the files. They're tricky and cannot be converted into text files easily, it has been my greatest challenge

I've stumbled upon this as soon as I tried to check NilAdo (also has STCM2L)
I still don't know how to use this, but maybe it can be useful to you

https://github.com/robbie01/stcm2-asm

edit: I just realized, there seems to be two different headers in Otomate games STCM2L and STCM2.
And this one script is STCM2 dedicated
There's a tool made for Diabolik Lovers using same header (STCM2L)
https://github.com/kubo25/Diabolik-Lovers-STCM2L-Editor
It seems to be derivated from hkki, but none of these two work with NilAdo
It probably needs some tweaks for it to be compatible with it...

 

[tsuki09_]

There's a tool made for Diabolik Lovers using same header (STCM2L)
https://github.com/kubo25/Diabolik-Lovers-STCM2L-Editor

Some notes while testing Diabolik Lovers Editor and hkki:

• This Diabolik Lovers tools doesn't work with the Switch port, seems to work only for PSP version (Haunted Dark Bridal).
• Also tested More Blood (PSP version), it also works with this tool
• Tried Amnesia PSP version, and it throws error messages, which didn't with Diabolik Lovers Switch nor NilAdo (at most it would only say "Found xxxx actions, read 0 texts").
• Tested Black Wolves Saga: Last Hope (PSP version) and it works
• Meaning that without tweaking the Dialovers tool, you could probably have support with PSP Rejet games
• Tested hkki with Amnesia (PSP) this time, it works!

So we need a new tool ideally for the Switch ports so they can be easily linked to the script files through hex code
Another important thing about these tools is, because they rely on hex code, they have a character limit per line. In this case, Unity otome games seem to be much more easy to mod, no character limit issue other than the aesthetics themselves.

 

[tsuki09_]

So we need a new tool ideally for the Switch ports so they can be easily linked to the script files through hex code
Another important thing about these tools is, because they rely on hex code, they have a character limit per line. In this case, Unity otome games seem to be much more easy to mod, no character limit issue other than the aesthetics themselves.

Anyone with experience on dealing with cpk files
is it normal for CriPackGUI to take +4 hours to patch a cpk file? kgjhdsfgd
I've tried YACpkTool which seems quite fast but doesn't do that compression process that CriPackGUI does. Maybe that is why CriPackGUI is taking so much time.
The whole set of extracted files from GAME01.cpk for NilAdo is 8-9Gbs uncompressed, while GAME01.cpk is +1GB compressed.
I wish CriPakGUI actually had a working option for injecting specific stuff instead of rebuilding the whole cpk
(I tried CriPakTools by esperknight but had no luck replacing the files I used the command meant for replacing mentioned in the github page, then extracting the script file again. The script file extracted seems untouched, despite me having used the command to replace it).

I have been trying to see what happens if I directly modified the texts through Hexecute. It is one of the very few hex editors with Shift JIS support (NilAdo's text's is written in Shift JIS). It is the first time I have ever tried to do this dkjfghdsf

Modified text:

Original file:

As a fun fact, a lot of Otomate games seem to begin their dialogues around files named with at least a two digit number (most common being the first three digit number instead of the first two digit number).
Some examples I've checked:

• Nil Admirari no Tenbin (Switch): 100
• Diabolik Lovers HBD (PSP): 100
• Diabolik Lovers More Blood (PSP): 101
• Amnesia (PSP): 50
• Black Wolves Saga: Last hope (PSP): 101
• Virche Evermore (Switch, english loc): 100

 

[tsuki09_]

Anyone with experience on dealing with cpk files
is it normal for CriPackGUI to take +4 hours to patch a cpk file? kgjhdsfgd

IT ACTUALLY WORKED what the hell
it took +7 hours to pack the cpk though

I think the best way to go (if using CriPakGUI) is getting all the text, translate, and then put the text through Hexecute. Only patch once you're done with the texts because of how much time this can take.
Linux lets you get the full text from each file and dump it into a txt (still don't know how to send it back to actual scripts though other than through manual hex editing the original file )
I don't remember what did I use to get the full text, so I'll have to boot into Mint and see what did I do through past commands lfgjsdgf
And reminder that this method has character limit.

I'm also happy there's not that line break problem I had with JP OverRequiemZ I can use normal spaces here.

And really, someone needs to fork CriPakGUI and make it possible to modify individual files instead of repacking the whole thing. UABEA does this for Unity stuff, and CriPakGUI already has the bases with managing to list the contents specifically, extract only individual items instead of the whole cpk plus the compression code.
It is sad that the project is essentially abandoned, last update is from 2019...

 

[tsuki09_]

I think the best way to go (if using CriPakGUI) is getting all the text, translate, and then put the text through Hexecute. Only patch once you're done with the texts because of how much time this can take.
Linux lets you get the full text from each file and dump it into a txt (still don't know how to send it back to actual scripts though other than through manual hex editing the original file )
I don't remember what did I use to get the full text, so I'll have to boot into Mint and see what did I do through past commands lfgjsdgf

strings -e S 100 > decode_100.txt

Editing through Hexecute is a little bit wonky, so this makes stuff more easy

Some basic/essential notes on hex code (feel free to fact check and correct as I'm not a professional ):

These highlighted numbers represent the sentence. As a demonstration, I picked these hex numbers and converted them to Shift JIS text through a Linux command

Hexecute allows you to either modify the numbers and/or the column with the converted Shift JIS characters. I went to the column of the converted characters and wrote the translation DeepL gave me for this sentence

of course, the numbers on the left side change with the characters, as the numbers have to mean these new characters I've replaced the Japanese text with.
Notice how I replaced the last 45 95 7b 81 41 with 00. This is essential, as computers signal of sentence (string of characters) end is the null character. If you're familiar with C language for example, you'll notice the importance of '\0' (this is a way to represent null character). A lot of modern programming languages already place the null character at the end of sentences to make working with character strings more easy. Since we're directly working with hex, null value is important. Computers don't understand grammar the same way we do as humans, and their equivalent to the dot/point we have in a lot of languages as a sentence end marker is the null character.
Computers have to do stuff in even more small steps than we humans usually have to, we take a lot of things for granted. And one of them is processing each letter, character, from a String. This process ends when the computer detects a null character, but it if doesn't, this leads into overflow, and this is bad.
Notice that after all these 00s there is a bb

This is where one has to be careful and not touch, as it is no longer part of the sentence. The script files are not solely made up of the game dialogue texts: it has variables, pointers, and a lot of other stuff that the game needs to run properly. If wrongly modified, this can break the game you're trying to patch, and also the reason why character limit exists.
I hope this can help people getting into hex editing for otome. It is the first time I've ever done this, but it was an interesting learning experience :)

 

[Lukeo]

Usually when it comes to binaries, you'd also have to learn multiplication, subtraction and addition with 0's and 1's to do an actual python script to decrypt these files and make tools later on. Python is a fun language lol.

 

[Lukeo]

Usually when it comes to binaries, you'd also have to learn multiplication, subtraction and addition with 0's and 1's to do an actual python script to decrypt these files and make tools later on. Python is a fun language lol.

Especially if there is encryption going on, you'd have to find the keys through using pip install (script name) which already has those keys.

 

[Lukeo]

Especially if there is encryption going on, you'd have to find the keys through using pip install (script name) which already has those keys.

And then work your way down to the hex codes, similar to how css or cascading style sheets is done to import one style to another using import (name of sheet)

Saying 00 as null is also incorrect while also correct especially if it's homebrewed engines they will make sure it won't appear through hex editors, so it will appear either 0 or special characters. (Which are keys).

 

[Lukeo]

And then work your way down to the hex codes, similar to how css or cascading style sheets is done to import one style to another using import (name of sheet)

Saying 00 as null is also incorrect while also correct especially if it's homebrewed engines they will make sure it won't appear through hex editors, so it will appear either 0 or special characters. (Which are keys).

Oh and 00 can also represent an alphabetical encryption, there is no such thing as space bar in binary or coding language.

 

[tsuki09_]

And then work your way down to the hex codes, similar to how css or cascading style sheets is done to import one style to another using import (name of sheet)

Saying 00 as null is also incorrect while also correct especially if it's homebrewed engines they will make sure it won't appear through hex editors, so it will appear either 0 or special characters. (Which are keys).

At first I was confused with the null explanation, but I think I'm understanding now :O (maybe )
Let me see the script file further--
The 00 00s did not show up in screen (*in-game) luckily lkjgdsgs
But there's a common hex value appearing after all sentences end (bb04 after a bunch of 00 00) and it seems like ba04 too for line breaks. ba04 also appears after character names (MC name and Hitaki for example), and I would be inclined to believe that Otomate probably has something like

• first string:always character name
• last three strings (ideally three, more than that is unreadable by players)

which is what happens with OverRequiemZ (although OverRequiemZ uses Unity and not the custom engine).

edit: the amount of 00s after the letters also seems arbitrary in the japanese sentences

edit 2 (sorry for spam noise notif, I pressed reply to my message instead of edit dkjfhdsg also a bunch of typos): I did forget narration doesn't have character names, so I would have to look more at the script

 

[tsuki09_]

edit 2 (sorry for spam noise notif, I pressed reply to my message instead of edit dkjfhdsg also a bunch of typos): I did forget narration doesn't have character names, so I would have to look more at the script

Actually, this is how OverRequiemZ deals with narration:

the first line is still used, but characters aren't named specifically. So maybe the mentioned structure does make sense for the custom engine Otomate games :O

edit: checked, and there's, indeed, a ba04 pattern even before the first game line

plus a line break example

edit 2: another pattern I've discovered, is that chXX seem to appear right during sprite changes. ch00 probably represents Tsumugi's sprites (around maybe other values to represent Tsumugi's different emotion expressions), while ch07 seems to be Hitaki. ch15 appears right after the butler comes in scene, and these values repeat throughout the script named 100.

 

[Lukeo]

If you are able to see the sentences in hex editors, then it uses lighter encryption levels, if it's homemade then you will just see a bunch of dots or special characters and nothing else, thus it would be a challenge to figure them out and to make any script or tools, you'd have to know what engine it could be by just guessing it.
I don't think it has to do with japanese sentences and 00's here. It depends on what encryption they are using.

Lighter encryption only hides the basic code structure not overall sentences.

Unity engine uses C#, which is slightly different from C or C++. Animation or API is easier to make in it.

 

[Lukeo]

If you are able to see the sentences in hex editors, then it uses lighter encryption levels, if it's homemade then you will just see a bunch of dots or special characters and nothing else, thus it would be a challenge to figure them out and to make any script or tools, you'd have to know what engine it could be by just guessing it.
I don't think it has to do with japanese sentences and 00's here. It depends on what encryption they are using.

Lighter encryption only hides the basic code structure not overall sentences.

Unity engine uses C#, which is slightly different from C or C++. Animation or API is easier to make in it.

81 72 45 etc seems to be where the jp lies at it seems rest are just code blocks. Based on the screenshots you have sent.

Pip install doesn't seem needed if you can see it. Just using encryption and decryption commands using python by forking the hex code should decrypt all. Most switch titles use .db files for script lately.

 

[tsuki09_]

Tried to get around this, and it really is complicated
I don't know if I'm getting this correctly, but I tried to compare headers among different Otomate games. The hex code patterns play out similarly, but the values are different.
NilAdo has a set of values including bb04 among several 00 and 10, while Virche (english) which has a different header has a common set of values including F401 between lines.
The other game that also has a set of values including bb04 between lines is Switch Diabolik Lovers.
These are the different headers I've found across Otomate games:
Nil admirari no Tenbin (Switch): STCM2L May 30 2012 14:17:58
Diabolik Lovers (Switch port): STCM2L Aug 24 2012 16:30:16 (looks like different header but has some values in common)
Different values seem to be present in headers from different years
Virche Evermore (English): STCM2L Feb 10 2014 16:23:58
Code Realize (English, Switch): STCM2L Feb 10 2014 16:23:58
Amnesia (PSP): STCM2L Jul 12 2010 10:20:01
Cendrillon Palika: STCM2L Dec 14 2016 13:08:37
* Unlike the other otome games, it seems like Cendrillon Palika makes use of UTF-8 instead of Shift JIS :o it looks messy in Hexecute, in the sense I can see the usual words written in English (heroine's name variable, background transitions and the sort) but I can't see the japanese dialogue like I always do. So I tried a different Hex editor with japanese UTF8 support, and now I can see the japanese dialogues.

This is wxMEdit, also has shift jis support.
Is this why is it more common to see Agent script related stuff compared to patching for Otomate games?
except there's nothing like Agent working inside the Switch console...

 

[tsuki09_]

Forgot to mention this detail, but in terms of Amnesia PSP, Amnesia for Switch is for some reason built in Unity despite PSP version having STCM2L header. And the dialogue follows the same structure of first string (name), last strings (dialogue).
And the dialogue script files are titled the same way, with numbers. (0,1,2, 50, 51, 101...)

I know this is probably different code, but considering Otomate's tendency to recycle assets (and the possibility of them having a different type of custom script for dialogue even if using Unity above C#) I think this looks interesting :O

I don't think this looks like C# :o

 

[tsuki09_]

https://github.com/robbie01/stcm2-asm

remember this tool? well
it doesn't work with NilAdo (probably due to different engine version (?) as headers specify different dates, and that leads us to different hex/binary values (I think (?)))
I don't remember if I mentioned this before, but I also had initial trouble trying to give Cendrillon Palika to this tool despite its mention in the Github.
And I fought this tool, not knowing that Cendrillon Palika isn't written in shift jis, but UTF-8
And sending everything from cmd to txt made the japanese look like a bunch of gibberish
Once I realized text was in UTF-8, I tried using this tool again
And it works :O
including replacing text in English without issues

now I need to see what else I can do with it...

also, I discovered a tool that extracts and repacks Idea factory formatted fonts ("Reconv", is it by the same person who made Agent tool :O). I have yet to see how it works, but seems cool
https://github.com/0xDC00/reconv


edit: no, some games with different headers still work with this tool... English Virche also works it seems :O

 

[gaysian]

Idk if this is the right place to ask, I have tried extracting audio files from Matsurika no Kei (both base and FD) but some of the files are always missing/from different character so if anyone knows then please help.

Also Pretty☆Witch☆Academy! (https://vndb.org/v1911) is so tough to extract. The only way was to do rip when the file is loaded onto memory but I don't think I got everything to play.