Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pup 6841 document parser api #8

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
173 changes: 173 additions & 0 deletions parser_api/example.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
require 'puppet'

# Example of a module setting everything up to perform custom
# validation of an AST model produced by parsing puppet source.
#
module MyValidation

# A module for the new issues that the this new kind of validation will generate
#
module Issues
# (see Puppet::Pops::Issues#issue)
# This is boiler plate code
def self.issue (issue_code, *args, &block)
Puppet::Pops::Issues.issue(issue_code, *args, &block)
end

INVALID_WORD = issue :INVALID_WORD, :text do
"The word '#{text}' is not a real word."
end
end

# This is the class that performs the actual validation by checking input
# and sending issues to an acceptor.
#
class MyChecker
attr_reader :acceptor
def initialize(diagnostics_producer)
@@bad_word_visitor ||= Puppet::Pops::Visitor.new(nil, "badword", 0, 0)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The concept of the visitor/polymorphic dispatcher must be explained. To go from this line what actually happens (calls to badword_QualifiedName etc.) is not self evident.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, will explain.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some additional text, but mainly referred to the blog post I wrote 2014.

# add more polymorphic checkers here

# remember the acceptor where the issues should be sent
@acceptor = diagnostics_producer
end

# Validates the entire model by visiting each model element and calling the various checkers
# (here just the example 'check_bad_word'), but a series of things could be checked.
#
# The result is collected by the configured diagnostic provider/acceptor
# given when creating this Checker.
#
# Returns the @acceptor for convenient chaining of operations
#
def validate(model)
# tree iterate the model, and call the checks for each element

# While not strictly needed, here a check is made of the root (the "Program" AST object)
check_bad_word(model)

# Then check all of its content
model.eAllContents.each {|m| check_bad_word(m) }

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This exposes Rgen. It might be of value to refrain from doing that in examples if we want to get rid of Rgen later on.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't think we have an alternative to eAllContents that is available from 3.8 future parser and forward?
And I think we should say that only eAllContent is API to reduce the amount of exposure to RGen API that we may not want to keep in an optimized implementation,

@acceptor
end

# perform the bad_word check on one AST element
# (this is done using a polymorphic visitor)
#
def check_bad_word(o)
@@bad_word_visitor.visit_this_0(self, o)
end

protected

def badword_Object(o)
# ignore all not covered by an explicit badword_xxx method
end

# A bare word is a QualifiedName
#
def badword_QualifiedName(o)
if o.value == 'bigly'
acceptor.accept(Issues::INVALID_WORD, o, :text => o.value)
end
end
end

class MyFactory < Puppet::Pops::Validation::Factory
# Produces the checker to use
def checker(diagnostic_producer)
MyChecker.new(diagnostic_producer)
end

# Produces the label provider to use.
#
def label_provider
# We are dealing with AST, so the existing one will do fine.
# This is what translates objects into a meaningful description of what that thing is
#
Puppet::Pops::Model::ModelLabelProvider.new()
end

# Produces the severity producer to use. Here it is configured what severity issues have
# if they are not all errors. (If they are all errors this method is not needed at all).
#
def severity_producer
# Gets a default severity producer that is then configured below
p = super

# Configure each issue that should **not** be an error
#
p[Issues::INVALID_WORD] = :warning

# examples of what may be done here
# p[Issues::SOME_ISSUE] = <some condition> ? :ignore : :warning
# p[Issues::A_DEPRECATION] = :deprecation

# return the configured producer
p
end

# Allow simpler call when not caring about getting the actual acceptor
def diagnostic_producer(acceptor=nil)
acceptor.nil? ? super(Puppet::Pops::Validation::Acceptor.new) : super(acceptor)
end
end

# We create a diagnostic formatter that outputs the error with a simple predefined
# format for location, severity, and the message. This format is a typical output from
# something like a linter or compiler.
# (We do this because there is a bug in the DiagnosticFormatter's `format` method prior to
# Puppet 4.9.0. It could otherwise have been used directly.
#
class Formatter < Puppet::Pops::Validation::DiagnosticFormatter
def format(diagnostic)
"#{format_location(diagnostic)} #{format_severity(diagnostic)}#{format_message(diagnostic)}"
end
end
end

# -- Example usage of the new validator

# Get a parser
parser = Puppet::Pops::Parser::EvaluatingParser.singleton

# parse without validation
result = parser.parser.parse_string('$x = if 1 < 2 { smaller } else { bigly }', 'testing.pp')
result = result.model

# validate using the default validator and get hold of the acceptor containing the result
acceptor = parser.validate(result)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps make the validate method capable of validating the result from parse_string rather than the model that it contains. That way, the result becomes opaque (i.e. no need to disclose the somewhat unintuitive implementation detail result = result.model ).

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, but the fact that it returns a Factory for further manipulation remains and a user of the API needs to know as they are bound to stumble over that otherwise,
(Potentially a lot of stuff (tests) to rewrite I think if we do not return a Factory, so don't think that is a good option either even if it is simple to add a Factory wrapper if one is needed).


# -- At this point, we have done everything `puppet parser validate` does except report the errors
# and raise an exception if there were errors.

# The acceptor may now contain errors and warnings as found by the standard puppet validation.
# We could look at the amount of errors/warnings produced and decide it is too much already
# or we could simply continue. Here, some feedback is printed:
#
puts "Standard validation errors found: #{acceptor.error_count}"
puts "Standard validation warnings found: #{acceptor.warning_count}"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would start with this example since it shows how to do a complete parse/validate using what's "in the box". Once that's covered, it's easier to understand that the objective with the MyValidation module is to go beyond that. It's also easier to understand how things fit together.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, will do a better "set up" before presenting all of the example code.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did the setup in the document rather than in the runnable code

# Validate using the 'MyValidation' defined above
#
validator = MyValidation::MyFactory.new().validator(acceptor)

# Perform the validation - this adds the produced errors and warnings into the same acceptor
# as was used for the standard validation
#
validator.validate(result)

# We can print total statistics
# (If we wanted to generated the extra validation separately we would have had to
# use a separate acceptor, and then add everything in that acceptor to the main one.)
#
puts "Total validation errors found: #{acceptor.error_count}"
puts "Total validation warnings found: #{acceptor.warning_count}"

# Output the errors and warnings using a provided simple starter formatter
formatter = MyValidation::Formatter.new

puts "\nErrors and warnings found:"
acceptor.errors_and_warnings.each do |diagnostic|
puts formatter.format(diagnostic)
end
Loading