JIRA bot to trim whitespaces from field values

November 6, 2020 &english @projects #kotlin

Trimmer is a small bot, which iterates over JIRA issues and trims whitespaces fields values. Without that, human mistakes on copy-paste operations can make automated JIRA statistics collections harder.

I’d like to demonstrate two different approaches to handle “paging” problem in Atlassian REST APIs. Atlassian REST endpoints usually allows to perform search queries with perPage and start parameters. Common solution to this problem (implemented in this trimmer) is to iterate sequentially, increasing start parameter along the way. Code below demonstrates the idea.

Sequential iteration can be easily abstracted away from the main code by introducing a custom iterator instance, which yields search results page-by-page.

class IssuesIterator(private val jql: String,
                     private val perPage: Int,
                     private val fields: Set<String>,
                     private val client: SearchRestClient) : Sequence<Issue> {
    override fun iterator(): Iterator<Issue> {
        return iterator {
            var currentIndex = -perPage
            do {
                currentIndex += perPage
                print(".")
                val currentResult = fetch(currentIndex)
                yieldAll(currentResult.issues)
            } while (currentIndex + perPage < currentResult.total)
        }
    }

    private fun fetch(start: Int): SearchResult {
        return client.searchJql(jql, perPage, start, fields).get()
    }
}

However, this approach makes it impossible to execute search requests in parallel. In order to overcome such limitation, I am using the following trick:

  • Execute first dummy search request with pageSize equal to 1 and start equal to 0. Result will give us access to the total parameter in a search response.
  • Read total value from search response and create a range of integers from 0 to total with step equal to desired page size.
  • Schedule background jobs (using Flow or any other concurrent library) to perform search requests with already known pageSize and start parameters.
private fun search(start: Int, per: Int, fields: Set<String> = setOf()): SearchResult =
  jiraClient.searchClient.searchJql("project = ${env[PERSONER_JIRA_PROJECT]}", per, start, MINIMUM_SET_OF_FIELDS.plus(fields)).get()

suspend fun projectCards(fields: Set<String>) =
  search(0, 1).total.let { total ->
    rangeUntil(0, total, env[PERSONER_PAGE_SIZE]).asFlow()
      .concurrentFlatMap { start -> search(start, env[PERSONER_PAGE_SIZE], fields).issues }
      .toList()
  }

Since it is unlikely that number of search results will change during parallel fetching process – the solution can be considered as “robust enough”.

If your project isn’t ready for modern Flow framework in Kotlin, you can implement the same idea using CompletableFutures. Retries are very possible too with Failsafe library.

fun <T> loadIssues(fields: String, query: String, mapper: (Issue) -> T): List<T> =
  jiraClient.searchIssues(query, fields, 1).total.let { total ->
    val chunkSize = max(1, total / JIRA_CHUNK_COUNT)
    IntStream.iterate(0) { it + chunkSize }.limit((total / chunkSize) + 1L).toList().map { start ->
      DatabaseContext.supplyAsync {
        Failsafe.with(RetryPolicy<List<T>>().handle(RestException::class.java).withMaxRetries(3)).get { ->
          jiraClient.searchIssues(query, fields, chunkSize, start).issues.map(mapper)
        }
      }
    }.map { it.get() }.flatten()
  }