roam/20210415101633-openapi_test_coverage.org

OpenAPI Test Coverage

• Things that can be covered
• Writing a Python coverage plugin

Reporting coverage of OpenAPI request and response specifications using Tornado OpenAPI 3.

Things that can be covered

• Requests by path and operation
• Responses by code
• Schemas?

Writing a Python coverage plugin

• Plug-in classes

Traced files can be determined dynamically based on execution frames (frame objects are passed to plugin functions). This means I'll be able to create simplified file-like representations of OpenAPI specifications, marking lines of them as executable, and reporting on which ones should be marked as executed based on the how the validator was run.

  import coverage.plugin


  class OpenAPITemplatePlugin(coverage.plugin.CoveragePlugin, coverage.plugin.FileTracer):
      def __init__(self):
          self.specs = {}

      def sys_info(self):
          return [("hi", "mom")]

      def file_tracer(self, filename):
          return None

      def has_dynamic_source_filename():
          return True

      def dynamic_source_filename(self, filename, frame):
          # TODO: Find calls to RequestValidator.validate
          return None


  def coverage_init(reg, options):
      ...

Plugin WIP