Data Source - GitHub GraphQL API

summary

Help page for Data Settings using GitHub API v4 (GraphQL) for ETL Configuration.

Using the GitHub GraphQL API v4 (GraphQL), you can write a GraphQL query to get the results of various queries as JSON records.

Obtainable data

All items that can be retrieved by a GraphQL query with the GitHub GraphQL API v4.
For query results (JSON), you can narrow down the fields to be forwarded. Filtering can be done by specifying the target path in JSON Path format.

In addition, if the filter result is an array, it can be taken in as one element and one row.
Example: If a list of Issues is retrieved, it is possible to capture each issue by specifying the path to the set of Issues.

For items that require paging, use TROCCO's built-in GraphQL Variables ($__trocco_githubEndCursor__).
API requests can be made while shifting cursors.
See "How Pagination Works" below for more information.

Setting items

STEP1 Basic settings

{
"data": {
    "repository": {
      "issues": [{<#issue1>}, {<#issue2>}]
    }
  }
}

The GraphQL query has a query sample written in advance as the default value as follows.

How Pagination Works

In GitHub GraphQL API v4, you can specify the start position of an Edge by specifying the start cursor as after.
TROCCO takes the final cursor contained in the query response and passes the value to the built-in variable $__trocco___githubEndCursor__.
If the next page exists, request the next page by specifying a built-in variable in the query after.
The maximum number of pages to be acquired is 10,000.

Setting Example

Acquisition of members belonging to Organization

Try to get the account name, name, and role of the member under Organization.

GraphQL Query

*Replace "login" in ORAGANIZATION with your Organization name.

query {
  organization(login: "primenumber-dev") {
    membersWithRole(first: 2) {
      edges {
        node {
          login
          name
        }
        role
      }
    }
  }
}

If you check the query results in the GitHub GraphQL API Explorer, you will see that the JSON is as follows

query results

{
  "data": {
    "organization": {
      "membersWithRole": {
        "edges": [
          {
            "node": {
              "login": "trocco-taro",
              "name": "TROCCO Taro"
            },
            "role": "ADMIN"
              },
        {
          "node": {
            "login": "trocco-hanako",
            "name": "TROCCO Hanako"
          },
          "role": "ADMIN"
        }
    }
  }
}
}

Since the information to be captured is for each member, specify data > organization > memberWithRole > edges with a JSON Path as follows,
The data is imported as a record in member units.

Specify the path to be captured (JSON Path)

$.data.organization.memberWithRole.edges
In this case, since edges is an array, each element can be imported as a single record.

Finally, we were able to incorporate the following

Acquisition of all Issues under Repository (using pagination)

GraphQL Query

query($__trocco__githubEndCursor__:String){
  repository(owner: "<#input your organization>", name: "<#input your repository>") {
    issues(first: 100, states: OPEN, after:$__trocco__githubEndCursor__) {
      edges {
        node {
          title
          url
          number
          updatedAt
        }
      }
      pageInfo {
        endCursor
        hasNextPage
      }
    }
  }
}

Specify path