Better delimitation between static and interactive features in the kernel

[tizoc]

Motivation

Shen as it has been created, is very REPL-centric. Because it was built primarily on top of Common Lisp the development model involved loading your code in the REPL and then saving an image that represents your program.

This model works very well for Common Lisp, but not for most platforms on which Shen runs.

Between other things, doing this helps ports on which the REPL is not the primary target, both in performance, startup times and code size (e.g. Shen when running in the browser, a Shen to C compiler, etc).

Changes required

Differentiate between interactive and non-interactive mode

Not sure how yet, probably some flag in a global variable. Some functions will change the behaviour depending on its value. Compilation of code could also behave differently depending on the mode.

Partial functions should not ask for input when not running in interactive mode

When Shen compiles a function that is partial, it adds a default case with a call to shen.f_error. What that function does is ask the user if he wants to "track" this partial function to help debug it. Once tracked, Shen will output the inputs and outputs of every call to this function.

This works well when working in the REPL, but for non-interactive execution of a program it should instead raise an error and exit the program if it is not handled. The behaviour of shen.f_error change depending on the running mode flag mentioned above, or the compiler can generate a different default case for partial functions. A third option is an extra pass (similar to the "dynamic-expand" extension) that takes care of "fixing" such functions when they are being compiled for non-interactive runs.

Better separation between runtime code and code required for the REPL and compiler

Programs that don't use eval will work without a lot of the code that gets loaded by default by the kernel. Anything related to typechecking, REPL, or environment data that is used by eval to compile code can be skipped in such cases.

Other

TODO

[tizoc]

Just leaving this here for now so that I can keep adding stuff as I remember it. It is not a trivial change and will require some time, but things have been moving into that direction already.

[rkoeninger]

👍 for a global variable, defaulting to false, that shen.repl sets to true.

[rkoeninger]

The y-or-n? function can also request user input at unexpected times. I have had to override it in the past to get the test suite to consistently run unattended on CI without hanging. y-or-n? is also used by f_error:

(define f_error
  F -> (do (output "partial function ~A;~%" F)
           (if (and (not (tracked? F))
                    (y-or-n? (make-string "track ~A? " F)))
               (track-function (ps F))
               ok)
           (simple-error "aborted")))

(define y-or-n?
  String -> (let Message (output String)
                 Y-or-N (output " (y/n) ")
                 Input (make-string "~S" (read (stinput)))
              (cases (= "y" Input) true
                     (= "n" Input) false
                     true (do (output "please answer y or n~%")
                              (y-or-n? String)))))

pause-for-user is a similar function, which gets called by show as part of the spy process. You could say that user interaction is a given with (spy +) though.

(define show
  P Hyps ProcessN Continuation
  -> (do (line)
         (show-p (deref P ProcessN))
         (nl)
         (nl)
         (show-assumptions (deref Hyps ProcessN) 1)
         (output "~%> ")
         (pause-for-user)
         (thaw Continuation))
      where (value *spy*)
  _ _ _ Continuation -> (thaw Continuation))

(define pause-for-user
  -> (let Byte (read-byte (stinput))
       (if (= Byte 94)
           (error "input aborted~%")
           (nl))))

There's also terpri-or-read-char, which gets inserted into code that is augmented for tracking. If f_error isn't prompting for tracking, then this won't get called anyway. Unless there's a non-interactive form of tracking where a trace of function calls gets dumped to a stream or something like that.

(define insert-tracking-code
  F Params Body -> [do [set *call* [+ [value *call*] 1]]
                       [do [input-track [value *call*] F (cons_form Params)]
                           [do [terpri-or-read-char]
                               [let (protect Result) Body
                                 [do [output-track [value *call*] F (protect Result)]
                                     [do [set *call* [- [value *call*] 1]]
                                         [do [terpri-or-read-char]
                                             (protect Result)]]]]]]])

(define terpri-or-read-char
  -> (if (value *step*)
         (check-byte (read-byte (value *stinput*)))
         (nl)))

[tiancaiamao]

I'm thinking of changing the shen-go implementation from bytecode VM to translating the code to Go directly. This feature helps a lot!

[tizoc]

@tiancaiamao you will still need the bytecode VM to support code evaluated at runtime, right? or is there a way to compile and dynamically load Go code at runtime?

[tiancaiamao]

you will still need the bytecode VM to support code evaluated at runtime, right?

Yes.

is there a way to compile and dynamically load Go code at runtime?

This is possible through the Go plugin, just like the dynamic link .so in C

---

I think there could be a fusion between interpreted VM code and generated Go code, the interpreter would decide to interpret or execute, just like JIT.

Maybe the word JIT is not accurate, call it AOT or something, you name it.
Anyway, the idea is, parts of the code could be generated to Go, and then compile to a plugin and loaded by the interpreter, then the interpreter could replace the interpret mode to execute the native function.

It works like we can treat any Shen code as primitives (as long as the code doesn't ask for eval-kl).
The result would be a much faster implementation.

[tizoc]

With the expand dynamic extension you should be able to pre-compile all the kernel into Go code now, and use the current bytecode compiler and interpreter to implement eval-kl, you just have to make sure it can call the native code, and that the native code can call bytecode compiled functions.

What you would gain once this issue is solved is an easier time producing standalone binaries that don't use dynamic features, resulting in smaller binaries (because you wouldn't have to pull the whole eval machinery into it).

[rkoeninger]

Cross-referencing rkoeninger/ShenScript#13

ShenScript would be able to apply significant optimizations regarding async/await if its compiler could be sure it was in a non-interactive mode.

[tizoc]

@rkoeninger note that for stuff like function redefinition you can just provide a separate compilation mode for it, doesn't have much to do with the kernel, but with the Kl->$platform compiler. In the case of ShenScript you can just add a flag that once enabled lets the compiler assume that functions will not be redefined (and you can also compile all kernel code with this flag enabled by default, there is nothing that says that internal code should be allowed to be overridden in the REPL).

What solving this issue would improve is making it easier for ports to not include unnecessary stuff in the final program, and handle a few things like partial functions in a different manner.