Keep the state of an Imported Library

[Thonuck]

I would like to use RobotKernel to develop larger Notebooks with multiple Cells.
For this notebook there is one library, which needs to be "setup" and keeps a state.
In current Implementation of the robotkernel it looks like, the imported Library is getting a new instance with each state.
Would it be possible, to keep one instance of a library over multiple cells?

I have the following code:

%%python module LibraryWithState

class LibraryWithState:
    def __init__(self):
        self.state = 0
        
    def get_increased_state(self):
        self.state += 1
        return self.state

And then the cells like

*** Settings ***
Library    LibraryWithState

*** Test Cases ***
Check the test Library
    Get Increased State

... which results in an output of the expected "1"
And the next cell like

*** Settings ***
Library    LibraryWithState

*** Test Cases ***
Check the test Library
    Get Increased State

this second output is still 1, but I wanted to see 2.

Is it possible, to achieve this? As I mentioned, seems with each cell, my python library gets freshly instantiated.
I would like to overcome this.

Thank you for any feedback or proposal.

[datakurre]

Ouch. That's a design compromise to which I have no easy solution.

It comes from that design choice that RobotKernel executes each cell separately, as separate Robot Framework run.

That said, because notebook itself has global state, RobotKernel does transmit some data between cells. It does preserve "suite level variables". Also some libraries have special support. E.g. SeleniumLibrary driver connections are preserved to avoid need to start new browser for each cell, and also allow fast iteration on Selenium interaction. This is done by injecting some RobotKernel specific listeners into Robot Framework executions.

So, right now, the only option is to save state into "suite level variable". It might even possible for the library to initialize its value from "suite level variable" (by using methods from BuiltIn library instance). Yet, because "suite level variables" are only restored on "start suite" event, it might be possible that library is initialized before "suite level variables" are restored. Of course, the ugly part in this is that this is would be something only required for RobotKernel and not for regular Robot execution :(

Let's keep this issue open in case that we figure out better options. It might be technically possible to save library instances at the end of a cell run, and inject them into new execution, but there might be unexpected side-effects (or unexpected lose of expected side-effects some libraries could do on their import). Similarly just storing some state variables for library is technically possible, but hard to make generic without unexpected side-effects...

[datakurre]

Oh, and thank you for feedback.

There has been very little development for some time, because we've been watching, where Robocorp develop its for of the kernel or https://github.com/jupyter-xeus/xeus-robot

Now it seems that there is again oxygen to think about the future of Robot Framework and Jupyter, because Robocorp abandoned JupyterLab to focus on VSCode and their upcoming Automation Studio.

[Thonuck]

Thank you very much for your reply! I find this approach of using robot within jupyter notebooks very promising. I really enjoy this to use it for prototyping robot test cases.

But in the scenarios, where I want to use it, I have libraries, which are in general instantiated and setup during the suite setup process, and latest in the suite teardown they get cleaned up. They keep connections to remote hosts on different protocols and they keep aliases for initialized connections and other stuff. So unfortunately, there is no simple "state" to be pushed into some variable.

So there is no kind of "execution context" of the running robot, which can be retrieved from and injected to each cell? So every set variable can be given from one cell to the next? I just remember, that it is possible in robot to list all set variables, maybe we could find some mechanism, to get all those... Just guessing...

(BTW: if it would be easy, every could do it ;) ;) )

[datakurre]

@Thonuck That all sound exactly like what RobotKernel is already doing especially for SeleniumLibrary

https://github.com/robots-from-jupyter/robotkernel/blob/master/src/robotkernel/listeners.py#L221

But I am not sure, how to make that generic. Is there generic approach for connection used in all of those libraries that we could use?

[datakurre]

@Thonuck I just realized that there is a common convention in Jupyter kernels to affect the kernel itself: %%magics.

How would it feel like to have something like

%%sticky LibraryName._attributeName

And it would "push" those named suite level library instance attributes into "context" at the end of the cell execution (suite end) and pop them back in the beginning of the next cell (suite start).

Or would you prefer other name or syntax?

[Thonuck]

I need to check this. The problem might be, that those libraries are depending on other libraries and their state and so on and forth. So they are not kind of "closed"/"simple". But as mentioned, I need to check this.

Hmm... if I look at this listeners, they use this builtin-function from robot "get_library_instance".
If I check the sources of robot, there is something written like:

    If the optional argument ``all`` is given a true value, then a
    dictionary mapping all library names to instances will be returned.
    This feature is new in Robot Framework 2.9.2.

with the function call

def get_library_instance(self, name=None, all=False):

Maybe I'm a bit too naive now, but wouldn't it maybe be possible, to have here a listener for set libraries, with "all=True"? Kind of iterating over what we get by this call to set all of them?

But I assume, you checked this already, and that this can lead to problems...

[datakurre]

I'm afraid to preserve all state of all libraries automatically, because libraries are free to do anything in their init and therefore that may lead to hard-to-debug issues later on some libraries.

But keeping state of a single library is something we should try. So that

%%sticky LibraryName

would save all attributes on matching library instance and restore them for the subsequent cell executions. That might still lead to weird issues, if connections etc are created already on library instance init, but at least it would easier to debug, because those explicit %%sticky configurations.

[Thonuck]

That would be already very helping, if this is possible.

And needs to be checked, if this would solve also keeping open connections and such. Or lead to weird situations...
If it is not too difficult to achieve, it would be really great, if you could supply such a sticky-keyword. It would be also possible to name a few sticky Libraries like this, I assume.

Thank you very much!

[datakurre]

I will look into this at the end of the week and comment back early next week. Your use case sounds like exactly the use case for Robot Framework support in Jupyter, so I'll try to find a solution. 🤞

[datakurre]

I tried that naive approach and it seems to work. I'll try to make a beta for you during the next days. Still need to test, which RF versions I am able to support with the feature.

Sticky keyword library will need either ROBOT_LIBRARY_SCOPE = 'SUITE' or ROBOT_LIBRARY_SCOPE = 'SUITE' scope, because the default scope is test, where keeping state between runs would be wrong.

[datakurre]

https://mybinder.org/v2/gh/robots-from-jupyter/robotkernel/sticky-magic?urlpath=tree/tests/StickyLibraryMagic.ipynb

[datakurre]

https://pypi.org/project/robotkernel/1.6a1/

[Thonuck]

Thank you very much! Sorry for the late reply. I will test this the next days.

[Thonuck]

I'm still in the process of understanding, why this is not working in my case.
The challenge, that I have, is, that the library is depending on other libraries as well.

My first try was to get all loaded python modules, and try to inject all python modules I need.
But this does not fit for now.
I'm still investigating. I'll keep you informed, if I have some updates.

Can you tell me, how I can debug robotkernel? I first tried it's installation from source with "pip -e path_to_robotkernel_clone", but then the jupyter did not recognize the robotkernel anymore. So I'm now changing code within my virtualenv/lib/... which is not that nice...

[datakurre]

@Thonuck I assume you confirmed that your initial example did work?

And the library has scope declaration ROBOT_LIBRARY_SCOPE = 'SUITE'?

You can use Display keyword from Library IPython.display to log variables onto notebook itself while executing the cells.
You should be able to check that Builtins.Get library instance returns the same library instance (as in same Python object).