roam/20210415101633-openapi_test_coverage.org

:PROPERTIES:
:ID:       787ec822-bb1c-44fb-b46e-69becd6e3aa3
:END:
#+title: OpenAPI Test Coverage

Reporting coverage of OpenAPI request and response specifications using [[id:5052fcd4-4a4d-4b07-bdd1-3dc1da8e3501][Tornado
OpenAPI 3]].

* Things that can be covered
- Requests by path and operation
- Responses by code
- Schemas?

* Writing a Python coverage plugin
- [[https://coverage.readthedocs.io/en/coverage-5.5/api_plugin.html#api-plugin][Plug-in classes]]

Traced files can be determined dynamically based on execution frames ([[https://docs.python.org/3/library/inspect.html][frame
objects]] are passed to plugin functions). This means I'll be able to create
simplified file-like representations of OpenAPI specifications, marking lines of
them as executable, and reporting on which ones should be marked as executed
based on the how the validator was run.

#+CAPTION: Plugin WIP
#+begin_src python
  import coverage.plugin


  class OpenAPITemplatePlugin(coverage.plugin.CoveragePlugin, coverage.plugin.FileTracer):
      def __init__(self):
          self.specs = {}

      def sys_info(self):
          return [("hi", "mom")]

      def file_tracer(self, filename):
          return None

      def has_dynamic_source_filename():
          return True

      def dynamic_source_filename(self, filename, frame):
          # TODO: Find calls to RequestValidator.validate
          return None


  def coverage_init(reg, options):
      ...
#+end_src