Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: serialize AST node back to wikitext string #8258

Open
wants to merge 55 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
37338b1
refactor: extract a new $tw.wiki.getParser
linonetwo Jun 12, 2024
8a12498
feat: allow $tw.utils.getParseTreeText to render other rules' text
linonetwo Jun 12, 2024
09c2e71
feat: two example getText handler
linonetwo Jun 12, 2024
a0708a1
Revert "feat: allow $tw.utils.getParseTreeText to render other rules'…
linonetwo Jun 12, 2024
f882cbd
refactor: keep original getParseTreeText not touched
linonetwo Jun 12, 2024
be3f037
refactor: use serialize in rules
linonetwo Jun 12, 2024
4ddcbc6
Merge remote-tracking branch 'upstream/master' into feat/to-string
linonetwo Jul 29, 2024
4e896d1
refactor: $tw.utils.extend({},options) -> options || {}
linonetwo Jul 29, 2024
179651a
Update codeinline.js
linonetwo Jul 29, 2024
8cbf855
Create test-wikitext-serialize.js
linonetwo Jul 29, 2024
a075303
DEBUG: only run my tests for development, remove before PR merge
linonetwo Jul 29, 2024
05dec72
lint: if
linonetwo Jul 29, 2024
1a6a071
feat: add rule: 'parseBlock' metadata
linonetwo Jul 29, 2024
84e27c0
feat: handle tailing \n that may be missing
linonetwo Jul 29, 2024
078967c
feat: allow recursive
linonetwo Jul 29, 2024
2aff9fe
feat: generate more rule and tests
linonetwo Aug 1, 2024
bfd116a
feat: generate more rule and tests
linonetwo Aug 2, 2024
d067871
fix: remove pragma:true, otherwise following text will become childre…
linonetwo Aug 2, 2024
b1667ec
fix: condition manually
linonetwo Aug 2, 2024
e49238d
fix: some test
linonetwo Aug 2, 2024
383e1b6
fix: some test
linonetwo Aug 2, 2024
63613ce
feat: $tw.utils.serializeAttribute
linonetwo Aug 3, 2024
7b4ea47
fix: use "" for string param
linonetwo Aug 3, 2024
f69795c
feat: list
linonetwo Aug 3, 2024
9df36a2
refactor: ' -> "
linonetwo Aug 3, 2024
2a75077
fix: parsemode don't have node
linonetwo Aug 3, 2024
c392976
fix: render invisible comment and parsemode as data element
linonetwo Aug 3, 2024
b22ca2b
feat: add void: true, in ast node to prevent render
linonetwo Aug 4, 2024
96adee9
Merge branch 'TiddlyWiki:master' into feat/to-string
linonetwo Aug 4, 2024
5b2c3db
feat: use void widget, so methods always return a widget
linonetwo Aug 4, 2024
7bc408b
feat: ast to use new widget type void
linonetwo Aug 4, 2024
82ae767
test: add rule: 'parseBlock' and isRuleEnd: true
linonetwo Aug 4, 2024
893edf9
lint: quote
linonetwo Aug 4, 2024
d6b9d00
Update widget.js
linonetwo Aug 4, 2024
6ac9592
fix: void node need to handle its children
linonetwo Aug 4, 2024
9dbbcef
Update test-wikitext-parser.js
linonetwo Aug 4, 2024
e7d4819
lint: quote
linonetwo Aug 4, 2024
af2ddaa
Update void.js
linonetwo Aug 4, 2024
b8125d1
Update test-wikitext-parser.js
linonetwo Aug 4, 2024
2fa454a
fix: macrodef with comment (void node) not working
linonetwo Aug 7, 2024
fb8732a
lint: ' -> "
linonetwo Aug 7, 2024
078cd5f
feat: add to styleblock
linonetwo Aug 7, 2024
7c73f1f
feat: styleblock
linonetwo Aug 8, 2024
43f3f8b
feat: styleinline
linonetwo Aug 9, 2024
856cb42
Update table.js
linonetwo Aug 9, 2024
3cf93d0
lint: useless comments
linonetwo Aug 9, 2024
589241a
feat: transcludeblock
linonetwo Aug 9, 2024
0adf638
refactor: reuse block on inline when possible
linonetwo Aug 9, 2024
c4f6136
feat: use void node to carry important info for typedblock
linonetwo Aug 9, 2024
08e9312
feat: run all tests
linonetwo Aug 9, 2024
398d7f7
lint: useless ai generated comments
linonetwo Aug 10, 2024
6580536
Merge remote-tracking branch 'upstream/master' into feat/to-string
linonetwo Aug 10, 2024
f5fca2a
Update conditional.js to not include space
linonetwo Sep 4, 2024
043f60b
Update test-wikitext-serialize.js
linonetwo Sep 4, 2024
03798d7
Update conditional.js
linonetwo Sep 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions core/modules/parsers/wikiparser/rules/codeblock.js
Original file line number Diff line number Diff line change
Expand Up @@ -57,4 +57,8 @@ exports.parse = function() {
}];
};

exports.serialize = function(tree) {
return "\n```" + tree.attributes.language.value + "\n" + tree.attributes.code.value + "\n```\n";
}

})();
4 changes: 4 additions & 0 deletions core/modules/parsers/wikiparser/rules/horizrule.js
Original file line number Diff line number Diff line change
Expand Up @@ -31,4 +31,8 @@ exports.parse = function() {
return [{type: "element", tag: "hr"}];
};

exports.serialize = function() {
return "\n---\n";
}

})();
28 changes: 28 additions & 0 deletions core/modules/utils/parsetree.js
Original file line number Diff line number Diff line change
Expand Up @@ -123,4 +123,32 @@ exports.getParseTreeText = function getParseTreeText(tree) {
return output.join("");
};

/*
Utility to get the (similarly but not 1:1 equal) original wikitext of a parse tree node or array of nodes
*/
exports.serializeParseTree = function serializeParseTree(tree, tiddlerType) {
var output = [];
if($tw.utils.isArray(tree)) {
$tw.utils.each(tree,function(node) {
output.push(serializeParseTree(node));
});
} else {
if(tree.type === "text") {
output.push(tree.text);
} else {
var Parser = $tw.wiki.getParser(tiddlerType);
var Rule = Parser.prototype.blockRuleClasses[tree.rule] ||
Parser.prototype.inlineRuleClasses[tree.rule] ||
Parser.prototype.pragmaRuleClasses[tree.rule];
if(Rule && Rule.prototype.serialize) {
output.push(Rule.prototype.serialize(tree));
}
}
if(tree.children) {
return serializeParseTree(tree.children);
}
}
return output.join("");
};

})();
20 changes: 13 additions & 7 deletions core/modules/wiki.js
Original file line number Diff line number Diff line change
Expand Up @@ -1035,6 +1035,18 @@ Options include:
exports.parseText = function(type,text,options) {
text = text || "";
options = options || {};
var Parser = this.getParser(type,options)
// Return the parser instance
return new Parser(type,text,{
parseAsInline: options.parseAsInline,
wiki: this,
_canonical_uri: options._canonical_uri,
configTrimWhiteSpace: options.configTrimWhiteSpace
});
};

exports.getParser = function(type,options) {
options = $tw.utils.extend({},options);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO your code will throw an RSOD if options is undefined
So options = options || {} should be used instead. We use this pattern all over the places.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I copy $tw.utils.extend({},options) from somewhere nearby, I thought this will work.

// Select a parser
var Parser = $tw.Wiki.parsers[type];
if(!Parser && $tw.utils.getFileExtensionInfo(type)) {
Expand All @@ -1046,13 +1058,7 @@ exports.parseText = function(type,text,options) {
if(!Parser) {
return null;
}
// Return the parser instance
return new Parser(type,text,{
parseAsInline: options.parseAsInline,
wiki: this,
_canonical_uri: options._canonical_uri,
configTrimWhiteSpace: options.configTrimWhiteSpace
});
return Parser;
};

/*
Expand Down
Loading