Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[IDEA] Serailize/toString of AST node #8255

Open
linonetwo opened this issue Jun 12, 2024 · 11 comments · May be fixed by #8258
Open

[IDEA] Serailize/toString of AST node #8255

linonetwo opened this issue Jun 12, 2024 · 11 comments · May be fixed by #8258

Comments

@linonetwo
Copy link
Contributor

Is your feature request related to a problem? Please describe.

Need to get original wikitext from AST, when writing WYSIWYG editor.

Describe the solution you'd like

Add rule = exports.name field to ast node, and add exports.toString to wikiparsers.

In toString, try call handleChildrenToString. (The problem is where to find this method, we can't access $tw in the wikiparser. Can we put it on parser passed from exports.init = function(parser) {?

Describe alternatives you've considered

Now it is maintained in https://github.com/tiddly-gittly/wikiast, but is quickly outdated.

Additional context

I remember some other issue we talk about adding a rule field to the ast.

@saqimtiaz
Copy link
Member

Add rule = exports.name field to ast node

See e4c682d where this has been implemented.

Part of the problem that still remains is that the parse tree does not have enough information about whitespace, comments and pragma so recreating the original wikitext still isn't possible.

@pmario
Copy link
Member

pmario commented Jun 12, 2024

Need to get original wikitext from AST, when writing WYSIWYG editor.

That's not really possible at the moment. While we did add a lot of new start and end positions to the AST lately, the <!-- comments --> are not there. So at the moment it's kind of impossible to create lossless wikitext from the TW AST.

I also think there is not enough information about function, procedure calls in the AST. eg: <<test>> and <$macrocall $name=test may not be distinguishable. -- But I'm not really sure about that one. -- But "The devil is in the details" atm

@linonetwo
Copy link
Contributor Author

linonetwo commented Jun 12, 2024

There is an unused method getParseTreeText, it used to work for text-slicer plugin, but it's useless now.

Old usage (don't exist anymore): https://github.com/eyecatchup/TiddlyWiki5/blob/72ed4b26731dd3bbac948af6965b050f33ed1c1c/plugins/tiddlywiki/text-slicer/slicer.js#L79C23-L79C39

https://github.com/Jermolene/TiddlyWiki5/blob/4bda8cfee6568fcdf2fd573c5303dbbd5751abd3/core/modules/utils/parsetree.js#L106-L124

Usage: $tw.utils.getParseTreeText($tw.wiki.parseTiddler('HelloThere').tree)

I can gradually add wiki rule support to it. Not need to be lostless, it only need to handle 80% of everyday usecase. People who use WYSIWYG editor won't write comment or procedure, support for then can be delayed.

See e4c682d where this has been implemented.

https://github.com/Jermolene/TiddlyWiki5/blob/d3e62ec56a91b004bd74b77273209e58ffb81f05/core/modules/parsers/wikiparser/wikiparser.js#L219

I see. This is not very obvious. I was only reading files in core\modules\parsers\wikiparser\rules

@Jermolene
Copy link
Member

Thanks @linonetwo I hadn't noticed that getParseTreeText() is no longer used.

I can gradually add wiki rule support to it. Not need to be lostless, it only need to handle 80% of everyday usecase. People who use WYSIWYG editor won't write comment or procedure, support for then can be delayed.

getParseTreeText() is intended to just extract the value of text nodes. I think you're proposing a function to convert a parse tree back to wiki text? I would think that that should be a separate function.

I take your point that it wouldn't be necessary to handle all wikitext constructions. It would be interesting to see how complex the implementation would be, but if it's not excessive then I would be happy to see that go into the core.

@linonetwo linonetwo linked a pull request Jun 12, 2024 that will close this issue
@pmario
Copy link
Member

pmario commented Jun 13, 2024

I take your point that it wouldn't be necessary to handle all wikitext constructions. It would be interesting to see how complex the implementation would be, but if it's not excessive then I would be happy to see that go into the core.

I think @flibbles knows best, what's needed for a "rewrite" of tiddler content. IMO the relink-plugin does exactly that.

@linonetwo -- I would have a closer look at the relink-AST (if there is one) and I would try to get or reuse the API from the relink plugin. Most users who want to use WYSIWYG will need the relink-plugin anyway.

@linonetwo
Copy link
Contributor Author

Yes, this is interesting, how relink works when there was no start/end pos in the AST?

After reading for a while, I find it is simply use regex to replace old value, not using AST

https://github.com/flibbles/tw5-relink/blob/4e3805ec149817cbb8e1ea7315cb8b4b72b1d25b/plugins/relink/js/utils/substitution.js#L33

@pmario
Copy link
Member

pmario commented Jun 13, 2024

... I find it is simply use regex to replace old value, not using AST

That may be true. @flibbles -- should know exactly.

@flibbles
Copy link
Contributor

Yes, this is interesting, how relink works when there was no start/end pos in the AST?

After reading for a while, I find it is simply use regex to replace old value, not using AST

You're looking at the method used to handle substitution attributes. Relink doesn't just use regexp. It also doesn't utilize the existing AST since it's incomplete for reproducing the original text. It modifies the Wikitext parser and has it "parse" text, with each wikitext rule module getting new report and relink methods. the "AST" it produces is the modified text.

@linonetwo
Copy link
Contributor Author

@Jermolene Need to add a new parse rule type

exports.name = "commentblock";
exports.types = {void:true, block:true, pragma:true};

When adding void:true, the AST node will not be rendered.

Currently without this, commentblock / parsermode will render to some text, cause style error, because they break the stylesheet tiddlers.

@Jermolene
Copy link
Member

@Jermolene Need to add a new parse rule type

Yes, that sounds like a reasonable solution.

@linonetwo
Copy link
Contributor Author

If you are interested, feel free to PR more test to #8258 . I've already pass all basic tests.

In some case, original whitespace is not preserved, may add/remove some of them to make result beautiful, like prettier. And user don't care about this, because in WYSIWYG editor, user don't care underlying text, they just want a text file so it feels safe.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants