ACD and Acode vs Text File

[tajmone]

@thoni56, lately I've been studying the ARun C source code again, for a project which I have in mind. But there are a few obscure points I wanted to ask you about.

ACD?

In the source variables and functions names, as well as in the comments, there are many references to the "ACD file" and the "ACD code".

I'm not quite sure what this terms exactly refers to, i.e. compared to the .a3c file. Does it refer to the entire .a3c storyfile? or just to the part that follows the ACodeHeader (i.e. the data).

Is this a legacy term, i.e. from ALAN 2? IRC, ALAN2 storyfiles had the .ACD extension.

Acode vs Text File

In main.c (L635) the openFiles() function seems to open two different adventure files:

• Acode file
• Text file

But both file operations seems to point to the same file (i.e. the .a3c storyfile of the current adventure). Is the code below opening the same file twice (shared read mode) to gain two different handles? (i.e. one for ACode, the other to access the game strings?).

What does the "adventure text data file" refers to, exactly, in the code below?

static void openFiles(void)
{
    char str[256];

    /* Open Acode file */
    strcpy(codfnm, adventureFileName);
    if ((codfil = fopen(codfnm, READ_MODE)) == NULL) {
        strcpy(str, "Can't open adventure code file '");
        strcat(str, codfnm);
        strcat(str, "'.");
        playererr(str);
    }

    /* Open Text file */
    strcpy(txtfnm, adventureFileName);
    if ((textFile = fopen(txtfnm, READ_MODE)) == NULL) {
        strcpy(str, "Can't open adventure text data file '");
        strcat(str, txtfnm);
        strcat(str, "'.");
        apperr(str);
    }

    /* If logging open transcript and/or log file */
    if (transcriptOption) {
        startTranscript();
    }
    if (commandLogOption) {
        startCommandLog();
    }
}

[thoni56]

A historical data point that might help is to remember that in Alan v2 there actually were two separate files, one containing the generated instructions and data, the other containing the texts/strings.

At some point, probably early experimenting with Alan v3, the files was merged, which simply meant that the Alan compiler as a final step bolted the text data onto the end of the Acode file. Thus making it easier to distribute a game it being a single file. (Although we now might have media files which are not, at least not yet, merged into the .a3c file.)

The v3 Acode header then contains the offset into the file where the strings are, and there are some arithmetic to read a string from the "text file" because of this.

So, yes the code and text files are actually the same "physical" file, but opened twice to keep reading them separate, e.g. with this it is possible to just advance in the text file without affecting the code file position. (Not that this actually matters since complete "code" part of the file is loaded into memory... And no, with modern hardware, software and OS architectures, this will not prohibit huge games to be run ;-), but it might be preventing Alan running on PalmPilots and similar restricted devices. This was a concious decision.)

As a side note, the text/string portion is not read into memory but read from the file on disc when needed.

Does this answer your question?

[tajmone]

Does this answer your question?

Thanks a lot, and indeed yes, it did help me a lot.

In order to learn the inner-workings of ARun, I'm working on a PoC implementation in another language (Basic), to prove to myself that I'm really grasping every concept by making it work correctly.

This has been an on-and-off fun/learning project that has been going on for quite a while (with huge gaps between sessions), but now I'm getting to a point where the code is able to process the ACodeHeader, so I'm cleaning up the code and going to publish it on GitHub (again, for no specific reason right now).

Sometimes I find myself a bit lost while trying to mentally visualize the code paths of ARun, mostly due to all the conditional directives (Glk/non-Glk, big-/little-endian) and the various macros; but I'm starting to get a clearer picture of how the interpreter loads and initializes the storyfile.

My local code can now load the ACodeHeader from a storyfile, reverse the endianess of all Awords (except the initial tag and version), then validate the ALAN tag and extract the compiler version as a string.

Of course, I'm keeping things simple, so it's a little-endian only app (limited to x8086 architecture, like the language I'm using), and it won't support adventures compiled with older versions (i.e. requiring the alternative headers, with no IFIDs and/or prompt entries; at least not now). So in my app I don't actually need all the temporary headers, I just load the header from the .a3c file into memory and reverse the Awords (PreBeta2 adventures are discarded right now).

Although it's a fun project for me to learn about ARun and the Amachine, if the project ever grows enough I did have in mind two potential uses for my code:

1. Create a Skein like that found in Inform 7.
2. Create a visual adventure debugger.

For the former, my tool would focus on speed, being able to quickly save and restore each adventure state when branching and resuming along the Skein nodes. It would also generate HTML formatted output, since the Skein uses the WebControl to show the various tests paths in an adventure (I'd be using a cross platform browser component, not the native WebControl though).

As for the latter, the idea of a visual debugger is still mostly tied to learning the Amachine and how Acode works in real adventures. But the language I'm using (PureBasic) has really an edge compared to other languages when it comes to cross platform GUI controls, so it's the right tool to create a nice visual debugging application. Again, it's just about learning how the Amachine operates, and being able to trace the code of an adventure as it unfolds, one turn at the time.

My next step is handling the initialization of the in-memory copy of the ACode section — I see that in reverse.c there are a lot of different functions and structures involved, since each table needs to be handled differently.

in Alan v2 there actually were two separate files, one containing the generated instructions and data, the other containing the texts/strings.

I didn't know that, and it's nice to learn a bit of ALAN history.

we now might have media files which are not, at least not yet, merged into the .a3c file.

Yes, I haven't yet got to that part, but eventually I shall write a module to inspect, manipulate and extract media files from an .a3r resource file, just for the sake of completion.

So, yes the code and text files are actually the same "physical" file, but opened twice to keep reading them separate, e.g. with this it is possible to just advance in the text file without affecting the code file position. (Not that this actually matters since complete "code" part of the file is loaded into memory... And no, with modern hardware, software and OS architectures, this will not prohibit huge games to be run ;-), but it might be preventing Alan running on PalmPilots and similar restricted devices. This was a concious decision.)

As a side note, the text/string portion is not read into memory but read from the file on disc when needed.

I've been reasoning about this, and I think that in my application I will store everything in memory, in view of the Skein project, which requires performance over memory. So I'll probably end up storing the code in a dedicated buffer, and the text in another. In any case I've seen that Aaddr pointers are really file offsets, and need to be converted to real memory offsets anyhow.

For the Skein, I'll probably be expecting only adventures compiled without Debugger and Pack options, since the Skein tool will be invoking the compiler. Otherwise, I might need to unpack the strings and fix the Aaddr pointers, for performance speed (a Skein tool can quickly become slow to perform, even in Inform 7, so skipping strings unpacking could make a difference).

In any case, the road ahead is still long, but a bit at the time I'm getting there. The whole point is that I realized that unless I code what I learn I'm doomed to forget it. In the past I would study the C sources, but then forget everything. And since I have no practical need to write anything in C, I though I'd experiment with implementing parts of ARun in PureBasic and Rust.

When I get to the memory part I'll probably have some more questions, since it's note entirely clear to me how the memory is organized. The rest of the code, regarding the Amachine, its opcodes and inner workings, seems clearer to follow, and contains more comments as well.

[thoni56]

Does this answer your question?

Thanks a lot, and indeed yes, it did help me a lot.

In order to learn the inner-workings of ARun, I'm working on a PoC implementation in another language (Basic), to prove to myself that I'm really grasping every concept by making it work correctly.

This has been an on-and-off fun/learning project that has been going on for quite a while (with huge gaps between sessions), but now I'm getting to a point where the code is able to process the ACodeHeader, so I'm cleaning up the code and going to publish it on GitHub (again, for no specific reason right now).

That's an interesting, and challenging, project ;-) Have fun!

Sometimes I find myself a bit lost while trying to mentally visualize the code paths of ARun, mostly due to all the conditional directives (Glk/non-Glk, big-/little-endian) and the various macros; but I'm starting to get a clearer picture of how the interpreter loads and initializes the storyfile.

Those directives do tend to get in the way, but if you are focusing on the core of the Amachine, you can just skip over when you get to a e.g. 'glk' directive and just think of it as "I/O". The big/little endian is a bit more problematic since its fairly deeply involved in the loading process, but (if my memory serves me) it is not sprinkled over any other part of the core.

My local code can now load the ACodeHeader from a storyfile, reverse the endianess of all Awords (except the initial tag and version), then validate the ALAN tag and extract the compiler version as a string.

That's impressive, given that the code isn't as clean as I would want it to be. It's partly many decades old legacy, which has not always been cleaned up as it should. And over the years I've used varying coding conventions too.

Of course, I'm keeping things simple, so it's a little-endian only app (limited to x8086 architecture, like the language I'm using), and it won't support adventures compiled with older versions (i.e. requiring the alternative headers, with no IFIDs and/or prompt entries; at least not now). So in my app I don't actually need all the temporary headers, I just load the header from the .a3c file into memory and reverse the Awords (PreBeta2 adventures are discarded right now).

Probably a good strategy. If this (your) project pans out and becomes "live" at some point you can probably back-port most of that compatibility stuff fairly easy.

we now might have media files which are not, at least not yet, merged into the .a3c file.

Yes, I haven't yet got to that part, but eventually I shall write a module to inspect, manipulate and extract media files from an .a3r resource file, just for the sake of completion.

You might have noticed that the resource file is actually a blorb-file and can (probably) be manipulated with tools for that. in compiler/resource.c you can find references to the blc Blorp packager.

One more usecase might be to have a more modern, non-C, Acode dump utility. You might have seen that there is source code for a dumpacd utility. I'm not sure about it's exact state today, but I do bring it uptodate and compile it occassionaly when I need it.

... In the past I would study the C sources, but then forget everything....

Just like me ;-)

When I get to the memory part I'll probably have some more questions, since it's note entirely clear to me how the memory is organized. The rest of the code, regarding the Amachine, its opcodes and inner workings, seems clearer to follow, and contains more comments as well.

You're welcome. Perhaps that makes me remember things better too.

[tajmone]

Those directives do tend to get in the way, but if you are focusing on the core of the Amachine, you can just skip over when you get to a e.g. 'glk' directive and just think of it as "I/O".

Indeed. Even better, if I could pre-process all the C sources to replace the conditional directives to filter the code for a little-endian non-Glk version. But since I don't use C so much (except reading C code to port it to other languages) I don't know if there are tools to do this. In other languages you can ask the compiler to spit out pre-processed versions of the source files, as long as you define all the required constants (i.e. those not defined by the system or OS specific compiler), but I couldn't find anything in GCC's help (which is veeeeery long to read); I have many other C compilers on my machine, from MS Visual C tools, PellesC, LLVM/Clang, and others, but haven't looked into it.

The big/little endian is a bit more problematic since its fairly deeply involved in the loading process, but (if my memory serves me) it is not sprinkled over any other part of the core.

It doesn't seem to affect anything beside loading the storyfile and initializing memory, so once I've finished that part I should be free of any endianess tasks.

.a3r Resource Files

You might have noticed that the resource file is actually a blorb-file and can (probably) be manipulated with tools for that.

I didn't know it, but I did suspect it since it would make sense to stick to the IF tradition here, either that or use a common packer algorithm. But I just didn't get to the stage of looking into the .a3r files yet.

in compiler/resource.c you can find references to the blc Blorp packager.

Good to know. I might then just write some bindings and use the C library in my project, at least initially to save time. My interest is strictly in learning about ALAN, so for all aspects pertaining to Glk, Blorb, and other IF standards, I'll just include C libraries and save time.

Acode Dumper

One more usecase might be to have a more modern, non-C, Acode dump utility. You might have seen that there is source code for a dumpacd utility. I'm not sure about it's exact state today, but I do bring it uptodate and compile it occassionaly when I need it.

I didn't notice the dumpacd utility, thanks for pointing it out.

I definitely had in my mind to create some .a3c/.a3r manipulation tools, since I'm writing these libraries anyhow, it's just a matter of interfacing them with a CLI tool, and a good way to proof that the code works.

I was thinking of a generic tool that could print adventures info in a nicely formatted way, extract the IFID (and regenerate a lost .IFID text file), extract media files from .a3r files, and other similar stuff that might be useful (e.g. in case of corrupted adventures).

Hex Editor Neo: ALAN .a3c Definition

BTW, a few years ago I wrote an .a3c Header definition for Hex Editor Neo, a commercial hex editor which I use. The definition is quite cool since it expands all the fields of the ACodeHeader when opening an .a3c file, showing them as intended, with side comments.

Here's a screenshot of advent3.a3c:

It's a bit small to read, but if you click on the image and open it in another browser window, you'll see for example that the ACodeHeader->version[4] is shown as the expanded "3.0beta8" string, and that endianess is correctly adjusted for all fields except the first two.

Hex Editor Neo has a Structure Editor tool for creating such definitions, which are basically C-like structures, but dynamic and with the possibility of using JavaScript functions to manipulate the structure itself.

The a3c definition is not fully complete yet (e.g. it can't dynamically handle headers of old adventures version which don't have the prompt and/or IFIDs fields, although it could do it). Also, the definition should ultimately cover the whole .a3c file, including being able to click on Aaddr entries in the header and jump to their offsets, and show the ACode tables in a structured manner, and be able to show unpacked strings too.

Anyhow, I've been using this definition to guide me through my recent code, since it helped validate all the code results against the hex editor representation (the definition can handle endianess on a per-field basis, so it's very accurate).

I've planned to include it in the Alan-Goodies repository at some point, although I'm not sure how useful it might be to others, especially since Hex Editor Neo is quite an expensive tool (but it's worth it). Again, it's one of those things which I've been working on and off in my free time, since a long time, and which is slowly coming into being in the course of time.

ALAN Skein Project

That's an interesting, and challenging, project ;-) Have fun!

The idea of creating a Skein tool for ALAN has been on my wish-list for quite a long time — you can even see my feature request in Issue #4, dating back to August 2019.

I don't how how much experience you have using Inform7, but Inform7's Skein tool is really a cool feature. You can basically fit an entire test suite into a single Skein file, thanks to the tree model which avoid redundancy and makes it very easy to update tests as the adventure grows, without having to rewrite every single test (since they share common paths and nodes).

The Skein is the kind of tool which makes life much easier for IF authors, and I strongly believe that creating a Skein tool for ALAN would be a huge bonus.

I did spend some time studying Inform7's skein source, and it's basically just a tree structure which allows users to play specific paths by selecting any given node, and after each run shows any diverging text results (just like a diff tool) so that authors can detect any output changes (expected or otherwise) and either "bless" (approve) the single changes (or the whole tree path) or fix the problem in the source adventure and re-run the scheme (in case something broke down after the last edits).

I also tried building the ALAN Skein by hooking into ARun via processes and pipes, but besides being very slow (since it requires saving the adventure state at each node bifurcation, and the restoring), there's always the problem of being able to handle non-standard Prompt definitions, which can only be done from within ARun — without knowledge of what the prompt looks like, it becomes impossible to correctly parse the I/O stream to distinguish where player input ends and adventure emitted text begins.

Hence my original request for a libARun DLL, then the decision to just give it a go and try to create an ad hoc interpreter built for that goal — with the added benefit of being able to represent output as formatted HTML, so it can represent also text styles, do away with the need to wrap text, etc. This seems the best option for the goal.

[tajmone]

Two More Questions About ACD and memory Buffer

I have two more questions which I can't work out by just looking at the source, since it's unclear how things end-up (it's a bit entangled).

1. When the memory buffer is created, it has size header->size, but it's not clear to me whether size means the size of the ACD code (only, without the ACodeHeader) or both — since ACD is used sometimes to indicate the whole file except the text, but other times just the Acode segment without the header (e.g. for the CRC calculation). Also, when the memory buffer is loaded from file, I have no idea if the file pointer is rewinded or it point to after the ACodeHeader offset at that stage.
2. Ultimately, does ARun store the ACodeHeader in an independent memory buffer from memory, or does memory include both the header and Acode segment?

Depending on your answer, I might adopt a different strategy, since the language I'm using allows to extend structure definitions, so it might make more sense for me to either store the header and Acode in different memory buffers, or in a same buffer.

You can see my preliminary code online:

https://github.com/tajmone/alan-pb/tree/main/pb/a3c

Now I have to decide how to load the Acode segment into memory, validate the CRC, and initialize memory by fixing endinaness of the various data structures in the Acode section. After that, recreating the Acode Dumper tool should be straight forward, I'd only need to add support for PreBeta2 compiled adventures.

PreBeta3 Requirements?

I also noticed in main.c (L197) that the setupHeader() function seems to handle some special adjustments for PreBeta3 adventures:

static void setupHeader(ACodeHeader tmphdr) {
    if (isPreBeta2(tmphdr.version)) {
        header = duplicate(&memory[0], sizeof(ACodeHeader));
        if (isPreAlpha5(tmphdr.version)) {
            header->ifids = 0;
        }
        header->prompt = 0;
    } else if (isPreBeta3(tmphdr.version)) {
        header = (ACodeHeader *) pointerTo(0);
    } else {
        header = pointerTo(0);
    }
}

It's unclear to me what's the difference here on how PreBeta3 headers and current version headers are treated — but since I couldn't find any comments about PreBeta3 requirements I don't know what to look for.

The following lines don't seem to me to lead to different results:

    } else if (isPreBeta3(tmphdr.version)) {
        header = (ACodeHeader *) pointerTo(0);
    } else {
        header = pointerTo(0);
    }

isn't the tmphdr already an ACodeHeader? and, from what I understood, pointerTo(0) is just a macro to translate an offset to the memory buffer. So I must be missing something here. The only other references to PreBeta3 handling I've found seem to relate to how parameters and instances are handled, but couldn't find anything that seems to affect the way the header is organized.

Probably it will become clearer to me as I explore the Amachine inner-working, but right now I'm focusing on being able to load and initialize the ADC, possibly handling older compiler versions correctly.

For PreBeta2 and PreAlpha5 adventures, it's fairly easy, since it's just that these have shorter headers, the former lacking the IFIDs entry, the latter both IFIDs and prompt, so it's just a matter of filling them with zero so they are not used.

[tajmone]

Solved!

@thoni56, I've managed to work it out (by trial and error), so now my app can also validate the CRC. I'm loading the whole .a3c file into memory at this point, which is simpler and more practical, so currently my prototype app can't handle PreBeta2 adventures — to fix the legacy headers to add the missing entries in the Header I'll need to do some extra memory manipulation, but I'll postpone that for now since I'm focused on learning how ARun initializes storyfiles before starting the adventure.

Source Improvement Suggestions

I think that the source code would become more accessible if you were to rename some of the variables and structures. As mentioned earlier, the ACD/ACode references are not very clear.

The point is this:

• An adventure consists in one or two files:
  1. .a3c — the storyfile proper,
  2. .a3r — the (optional) resources file for media assets (if any).
• The .a3c storyfile is divided into three segments (or sections):
  1. Header
  2. ACode
  3. AText

Using the term ‘ACD’ to refer sometime to the Header + ACode section, and at other times to the ACode only is a bit confusing, especially when trying to track the initial stages where different ACodeHeaders are being considered as candidates. I get the impression that both identifiers (vars, structs, funcs) and comments still mirror the legacy system where the .ACD storyfile stored only the header and ACode, and the AText was in a separate file.

So, maybe some renaming could dissipate any confusion.

Probably the most confusing aspect for me was that ACodeHeader.size refers to the size (in AWords) of the Header + ACode section — and the current naming conventions and lack of comments didn't make it easy to solve the mystery either. A small comment next to the header definition, in this regard, would have rendered deciphering the source much easier. Instead of the current:

    Aword size;   /* Size of ACD-file in Awords */

(which is misleading, since there's no "ACD-file" anymore, and the size doesn't include the AText section)

something like this would be clearer:

    Aword size;   /* Size (in Awords) of Header + ACode segments */

Although the term ‘size’ is a bit vague, and a more descriptive term would improve readability.

Technically speaking, this field indicates the offset (in AWords) to the beginning of the AText section in the .a3c file. It's implicit that it's also the size of the Header + ACode section. But it doesn't really represent the size of any single element (not any more, at least). I guess that in the past it indicated the entire size (in AWords) of the .ACD file, which was useful to know to calculate the CRC, handle ACode memory boundaries, etc.

Anyhow, once you know this key information, the rest of memory initialization code becomes all clear; whereas right now it requires quite some mental exercise, jumping from function to function and trying to remember what each parameter was set to (references of references, and so on).