correl.github.io/_posts/2018-01-22-recursive-http-with-elm.org

#+TITLE: Recursive HTTP Requests with Elm
#+AUTHOR: Correl Roush
#+STARTUP: indent showall inlineimages
#+OPTIONS: toc:nil num:nil
#+PROPERTY: header-args :exports code :eval never
#+KEYWORDS: elm programming

So I got the idea in my head that I wanted to pull data from the
GitLab / GitHub APIs in my Elm app. This seemed straightforward
enough; just wire up an HTTP request and a JSON decoder, and off I go.
Then I remember, oh crap... like any sensible API with a potentially
huge amount of data behind it, the results come back /paginated/. For
anyone unfamiliar, this means that a single API request for a list of,
say, repositories, is only going to return up to some maximum number
of results. If there are more results available, there will be a
reference to additional /pages/ of results, that you can then fetch
with /another/ API request. My single request decoding only the
results returned /from/ that single request wasn't going to cut it.

I had a handful of problems to solve. I needed to:

- Detect when additional results were available.
- Parse out the URL to use to fetch the next page of results.
- Continue fetching results until none remained.
- Combine all of the results, maintaining their order.

* Are there more results?

The first two bullet points can be dealt with by parsing and
inspecting the response header. Both GitHub and GitLab embed
pagination links in the [[https://www.w3.org/wiki/LinkHeader][HTTP Link header]]. As I'm interested in
consuming pages until no further results remain, I'll be looking for a
link in the header with the relationship "next". If I find one, I know
I need to hit the associated URL to fetch more results. If I don't
find one, I'm done!

#+CAPTION: Example GitHub Link header
#+BEGIN_SRC http
  Link: <https://api.github.com/user/repos?page=3&per_page=100>; rel="next",
    <https://api.github.com/user/repos?page=50&per_page=100>; rel="last"
#+END_SRC

Parsing this stuff out went straight into a utility module.

#+BEGIN_SRC elm
  module Paginated.Util exposing (links)

  import Dict exposing (Dict)
  import Maybe.Extra
  import Regex


  {-| Parse an HTTP Link header into a dictionary. For example, to look
  for a link to additional results in an API response, you could do the
  following:

      Dict.get "Link" response.headers
          |> Maybe.map links
          |> Maybe.andThen (Dict.get "next")

  -}
  links : String -> Dict String String
  links s =
      let
          toTuples xs =
              case xs of
                  [ Just a, Just b ] ->
                      Just ( b, a )

                  _ ->
                      Nothing
      in
          Regex.find
              Regex.All
              (Regex.regex "<(.*?)>; rel=\"(.*?)\"")
              s
              |> List.map .submatches
              |> List.map toTuples
              |> Maybe.Extra.values
              |> Dict.fromList
#+END_SRC

A little bit of regular expression magic, tuples, and
=Maybe.Extra.values= to keep the matches, and now I've got my
(=Maybe=) URL.

* Time to make some requests

Now's the time to define some types. I'll need a =Request=, which will
be similar to a standard =Http.Request=, with a /slight/ difference.

#+BEGIN_SRC elm
  type alias RequestOptions a =
      { method : String
      , headers : List Http.Header
      , url : String
      , body : Http.Body
      , decoder : Decoder a
      , timeout : Maybe Time.Time
      , withCredentials : Bool
      }


  type Request a
      = Request (RequestOptions a)
#+END_SRC

What separates it from a basic =Http.Request= is the =decoder= field
instead of an =expect= field. The =expect= field in an HTTP request is
responsible for parsing the full response into whatever result the
caller wants. For my purposes, I always intend to be hitting a JSON
API returning a list of items, and I have my own designs on parsing
bits of the request to pluck out the headers. Therefore, I expose only
a slot for including a JSON decoder representing the type of item I'll
be getting a collection of.

I'll also need a =Response=, which will either be =Partial=
(containing the results from the response, plus a =Request= for
getting the next batch), or =Complete=.

#+BEGIN_SRC elm
  type Response a
      = Partial (Request a) (List a)
      | Complete (List a)
#+END_SRC

Sending the request isn't too bad. I can just convert my request into
an =Http.Request=, and use =Http.send=.

#+BEGIN_SRC elm
  send :
      (Result Http.Error (Response a) -> msg)
      -> Request a
      -> Cmd msg
  send resultToMessage request =
      Http.send resultToMessage <|
          httpRequest request


  httpRequest : Request a -> Http.Request (Response a)
  httpRequest (Request options) =
      Http.request
          { method = options.method
          , headers = options.headers
          , url = options.url
          , body = options.body
          , expect = expect options
          , timeout = options.timeout
          , withCredentials = options.withCredentials
          }


  expect : RequestOptions a -> Http.Expect (Response a)
  expect options =
      Http.expectStringResponse (fromResponse options)
#+END_SRC

All of my special logic for handling the headers, mapping the decoder
over the results, and packing them up into a =Response= is baked into
my =Http.Request= via a private =fromResponse= translator:

#+BEGIN_SRC elm
  fromResponse :
      RequestOptions a
      -> Http.Response String
      -> Result String (Response a)
  fromResponse options response =
      let
          items : Result String (List a)
          items =
              Json.Decode.decodeString
                  (Json.Decode.list options.decoder)
                  response.body

          nextPage =
              Dict.get "Link" response.headers
                  |> Maybe.map Paginated.Util.links
                  |> Maybe.andThen (Dict.get "next")
      in
          case nextPage of
              Nothing ->
                  Result.map Complete items

              Just url ->
                  Result.map
                      (Partial (request { options | url = url }))
                      items
#+END_SRC

* Putting it together

Now, I can make my API request, and get back a response with
potentially partial results. All that needs to be done now is to make
my request, and iterate on the results I get back in my =update=
method.

To make things a bit easier, I add a method for concatenating two
responses:

#+BEGIN_SRC elm
  update : Response a -> Response a -> Response a
  update old new =
      case ( old, new ) of
          ( Complete items, _ ) ->
              Complete items

          ( Partial _ oldItems, Complete newItems ) ->
              Complete (oldItems ++ newItems)

          ( Partial _ oldItems, Partial request newItems ) ->
              Partial request (oldItems ++ newItems)
#+END_SRC

Putting it all together, I get a fully functional test app that
fetches a paginated list of repositories from GitLab, and renders them
when I've fetched them all:

#+BEGIN_SRC elm
  module Example exposing (..)

  import Html exposing (Html)
  import Http
  import Json.Decode exposing (field, string)
  import Paginated exposing (Response(..))


  type alias Model =
      { repositories : Maybe (Response String) }


  type Msg
      = GotRepositories (Result Http.Error (Paginated.Response String))


  main : Program Never Model Msg
  main =
      Html.program
          { init = init
          , update = update
          , view = view
          , subscriptions = \_ -> Sub.none
          }


  init : ( Model, Cmd Msg )
  init =
      ( { repositories = Nothing }
      , getRepositories
      )


  update : Msg -> Model -> ( Model, Cmd Msg )
  update msg model =
      case msg of
          GotRepositories (Ok response) ->
              ( { model
                  | repositories =
                      case model.repositories of
                          Nothing ->
                              Just response

                          Just previous ->
                              Just (Paginated.update previous response)
                }
              , case response of
                  Partial request _ ->
                      Paginated.send GotRepositories request

                  Complete _ ->
                      Cmd.none
              )

          GotRepositories (Err _) ->
              ( { model | repositories = Nothing }
              , Cmd.none
              )


  view : Model -> Html Msg
  view model =
      case model.repositories of
          Nothing ->
              Html.div [] [ Html.text "Loading" ]

          Just (Partial _ _) ->
              Html.div [] [ Html.text "Loading..." ]

          Just (Complete repos) ->
              Html.ul [] <|
                  List.map
                      (\x -> Html.li [] [ Html.text x ])
                      repos


  getRepositories : Cmd Msg
  getRepositories =
      Paginated.send GotRepositories <|
          Paginated.get
              "http://git.phoenixinquis.net/api/v4/projects?per_page=5"
              (field "name" string)
#+END_SRC

* There's got to be a better way

I've got it working, and it's working well. However, it's kind of a
pain to use. It's nice that I can play with the results as they come
in by peeking into the =Partial= structure, but it's a real chore to
have to stitch the results together in my application's =update=
method. It'd be nice if I could somehow encapsulate that behavior in
my request and not have to worry about the pagination at all in my
app.

It just so happens that, with Tasks, I can.

/To be continued.../

/For now, feel free to check out the full library documentation and
code referenced in this post [[http://package.elm-lang.org/packages/correl/elm-paginated/1.0.1][here]]./