How to get the text data on the current terminal window?

[wochatme]

We are developing the application "AI in terminal", just like the below screen-shot:

When the user meets some error, he/she can ask the AI service. Before he/she submit the question, the screen data will be attached implicitly so the AI can "see" your screen and your trouble.

I am blocked by "How to get the text data of the current screen?". We do not want to get the whole text buffer, just the data we can see from the screen, that is:

Microsoft Windows [Version 10.0.19045.4894]
(c) Microsoft Corporation. All rights reserved.

C:\Users\Kevin>ls -l
'ls' is not recognized as an internal or external command,
operable program or batch file.

C:\Users\Kevin>

[o-sdn-o]

As a possible way you can use a TUI utility like vtm to take a text screenshot to the clipboard in one window, and monitor the clipboard in another window/process.

PowerShell.2024-09-14.23-15-56.mp4

Command to run vtm built-in terminal with a single button "Ask AI":

vtm -t -c "<config><term><menu item*><item label='Ask AI' action=TerminalViewportCopy/></menu></term></config>" -r term

Of course, for such a scenario you don’t need any fancy vtm; simply copying a block of text with the mouse is enough. These are just my thoughts in this direction.

[wochatme]

I have the below code in ControlCore.cpp, and it works.

            const auto lock = _terminal->LockForReading();
            const auto viewport = _terminal->GetViewport();
            const auto top = viewport.Top();
            const auto height = viewport.Height();

            const auto& textBuffer = _terminal->GetTextBuffer();
            std::wstring str;

            for (auto rowIndex = top; rowIndex < top + height; rowIndex++)
            {
                const auto& row = textBuffer.GetRowByOffset(rowIndex);
                const auto rowText = row.GetText();
                const auto strEnd = rowText.find_last_not_of(UNICODE_SPACE);
                if (strEnd != decltype(rowText)::npos)
                {
                    str.append(rowText.substr(0, strEnd + 1));
                }
                str.append(L"\r\n");
            }

Now the variable str contains the data I want. You can add the below code to some function. I added it to

void ControlCore::SendInput(const std::wstring_view wstr)

It works well when you type a character, and this function will be called. But my caller function is in the class "NonClientIslandWindow". I do not know how to call the function of ControlCore from the class "NonClientIslandWindow". This is the last bloker.

[o-sdn-o]

Maybe put std::wstring str in the global scope, and signal the update with something like std::condition_variable while waiting for those changes in another thread.

[wochatme]

I find a good place to trigger the screen data capture. Put the above code in the LoseFocus() event of the terminal. It works pretty well.

In your caller function, use SetFocus() win32 API to trigger the screen data capture logic.

[wochatme]

Our website is:

https://zterm.ai

I will write a document to describe how to capture the screen data in Windows Terminal this week.