logo
Published on

Iterators and Generators in JavaScript

Authors
covers2-3

Mastering JavaScript Generators: Deep Dive and Advanced Use Cases

For a long time, I've been intrigued by iterators and generators in JavaScript. These powerful features have the potential to significantly improve the way we write code, making it more efficient (spoiler alert: disclaimer in the Performance Considerations section), readable, and elegant.

However, every time I tried to learn about them, I found myself struggling to wrap my head around the concepts and eventually gave up. But I was determined not to let the challenges deter me. I decided to embark on a personal journey to learn and master iterators and generators, despite the obstacles I had faced before.

In this article, we will dive deeper into generators, explore their advanced use cases, and learn how to harness their full potential. You can find all the code examples (and even more) in this repository on GitHub.

Overview

Iterators and generators provide a mechanism for customizing the behavior of for-of loops and other constructs that iterate over data.

Iterators are objects that provide a standard interface for iterating over a collection of values. They implement the iterable protocol by having a next() method that returns an object with a value property that represind the next value in the iteration sequence, and a done property that indicates whether the iterator is finished (boolean).

{
  value: any, // any data type
  done: boolean
}

An iterator object, once created, can be explicitly iterated by calling next() repeatedly. The process of iterating over an iterator is known as consuming the iterator since it is typically performed only once. After yielding a terminating value, any subsequent calls to next() should continue to return

{
    value: undefined,
    done: true
}

The following createRangeIterator function that takes start, end, and a step parameters and returns an object with a next() method, which updates the current value and increments the iterationCount. When the current value reaches or exceeds the end, the iterator returns the iterationCount and sets done to true.

function createRangeIterator(start, end, step = 1) {
  let current = start
  let iterationCount = 0

  return {
    next: function () {
      if (current < end) {
        const value = current
        current += step
        iterationCount++
        return { value, done: false }
      }

      return { value: iterationCount, done: true }
    },
  }
}

const iterator = createRangeIterator(0, 10, 2)

let result = iterator.next()

do {
  result = iterator.next()
  console.log(result.value)
} while (!result.done)

// Output:
// 0
// 2
// 4
// 6
// 8
// Final return value (iterationCount): 5

Generator functions provide a powerful way to define iterative algorithms using a single function whose execution is not continuous. In other words, generator functions can be paused and resumed at specific points during their execution. This capability allows for more flexible and efficient control flow compared to traditional functions, especially when dealing with iterative processes.

A generator function is defined using the function* keyword (function keyword followed by an asterisk). We can also define generator functions using the GeneratorFunction constructor:

const GeneratorFunction = function* () {}.constructor

const myGeneratorFn = new GeneratorFunction(`
  yield 'a'
  yield 'b'
  yield 'c'
`)

for (const val of myGeneratorFn()) {
  console.log(val)
}

// Output:
// a
// b
// c

When invoked, generator functions don't execute their code immediately. Instead, they return a unique type of iterator known as a Generator. The Generator function runs until it encounters the yield keyword, which is used to pause and resume the generator function's execution.

The generator object is a type of iterator, therfore it has a next() method that returns an object containing the value provided by the yield statement (or undefined if the function has ended) and a done property, which indicates whether the generator function has completed its execution. Each time the generator's next() method is called, the generator resumes execution, and runs until it reaches a yield expression, return statement, throw statement, or the end of the function.

💡NOTE: You can call a generator function as many times as you want, and it will return a new generator each time. However, each Generator can only be iterated once.

Now, let's rewrite the previous example using a generator function.

function* rangeGenerator(start, end, step = 1) {
  let current = start

  do {
    yield current
    current += step
  } while (current < end)
}

const iterator = rangeGenerator(0, 10, 2)

for (const value of iterator) {
  console.log(value)
}

// Output:
// 0
// 2
// 4
// 6
// 8

In the examples provided above, we can already discern the advantages that generators bring to the table: using a generator can help reduce memory usage. The rangeGenerator generates a range of numbers one by one, which means that we only hold the currently required number in memory. If we were to create an array with all the numbers from start to end, it would require significantly more memory, especially for large datasets.

Recapping the Basics

Before we dive into the advanced use cases of generators, let's make a quick recap of the basics.

A generator is a special type of iterator object that is returned by a generator function. Generator functions are denoted using the function* keyword, and they allow you to define iterative algorithms with the ability to pause and resume execution at specific points using the yield keyword.

function* simpleGenerator() {
  yield 1
  yield 2
  yield 3
}

Generators have a next() method that allows us to traverse the generator's values and returns an object containing the following properties:

  • value - the value yielded by the generator function
  • done - a boolean indicating whether the generator has completed its execution
const iterator = simpleGenerator()

console.log(iterator.next()) // { value: 1, done: false }
console.log(iterator.next()) // { value: 2, done: false }
console.log(iterator.next()) // { value: 3, done: false }
console.log(iterator.next()) // { value: undefined, done: true }

Advanced Use Cases and Practical Applications

Generators are a versatile feature in JavaScript, and while there isn't a single most common use case, some popular use cases include:

1. Handling Asynchronous Code

Generators can simplify asynchronous code by enabling a more synchronous-like control flow (the order in which the computer executes statements in a script). Combined with Promises or async/await, they can help us manage complex chains of asynchronous operations more effectively and with better readability.

function timeout(ms) {
  return new Promise((resolve) => setTimeout(resolve, ms))
}

function* asyncGenerator() {
  console.log('Start')
  yield timeout(1000)
  console.log('After 1 second')
  yield timeout(2000)
  console.log('After 2 more seconds')
}

async function runAsync(generator) {
  const iterator = generator()
  let result

  while (!(result = iterator.next()).done) {
    await result.value
  }
}

runAsync(asyncGenerator)

// Output:
// Start
// After 1 second
// After 2 more seconds

2. Generator Delegation with yield*

The yield* expression allows us to delegate one generator to another, making it possible to compose multiple generators to create more modular and reusable code.

function* generatorA() {
  yield 1
  yield 2
}

function* generatorB() {
  yield 3
  yield* generatorA()
  yield 4
}

const iterator = generatorB()

console.log([...iterator]) // [3, 1, 2, 4]

As you can see in the example above, generators are not only iterators, but also iterable. That means that we can use them in a for...of loop (which we actully already saw in the rangeGenerator example), or spread them into an array.

get-serious

3. Bidirectional Communication

The yield keyword can be used not only to send values from the next() method, but also to consume values from the next() method.

Notice that in the following example, the yield keyword is on the left side of the assignment operator. Therefore, when we call iterator.next() with "Syntactic Sugar" as an argument, the name variable is assigned the value "Syntactic Sugar".

function* bidirectionalGenerator() {
  const message = 'What is your name?'
  const name = yield message
  yield `Hello, ${name}!`
}

const iterator = bidirectionalGenerator()
console.log(iterator.next().value) // "What is your name?"
console.log(iterator.next('Syntactic Sugar').value) // "Hello, Syntactic Sugar!"

4. Lazy Evaluation

Generators allow us to create sequences or compute values on-demand. That means that values are computed only when they're needed. This can lead to performance improvements, especially when working with large datasets or computationally intensive calculations.

And what could be a better way to demonstrate this than with the Fibonacci sequence?

function* fibonacci() {
  let a = 0
  let b = 1

  while (true) {
    yield (a[(a, b)] = [b, a + b])
  }
}

const iterator = fibonacci()
for (let i = 0; i < 10; i++) {
  console.log(iterator.next().value)
}

// output:
// 0
// 1
// 1
// 2
// 3
// 5
// 8
// 13
// 21
// 34

5. Iterating Over Complex Data Structures

Generators enable the creation of custom iterators for traversing complex data structures, such as trees or graphs. This can simplify the process of iterating over these data structures and make your code more readable and maintainable.

  • Traversing a binary tree using an in-order iterator:
class TreeNode {
  constructor(value, left = null, right = null) {
    this.value = value
    this.left = left
    this.right = right
  }
}

function* inOrderTraversal(node) {
  if (node === null) return
  yield* inOrderTraversal(node.left)
  yield node.value
  yield* inOrderTraversal(node.right)
}

const tree = new TreeNode(
  4,
  new TreeNode(2, new TreeNode(1), new TreeNode(3)),
  new TreeNode(6, new TreeNode(5), new TreeNode(7))
)

for (const value of inOrderTraversal(tree)) {
  console.log(value)
}

// output:
// 1
// 2
// 3
// 4
// 5
// 6
// 7
  • Creating a chess board using custom iterable object:
const chessBoard = {
  ranks: [1, 2, 3, 4, 5, 6, 7, 8],
  files: ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'],
  [Symbol.iterator]: function* () {
    for (let rank of this.ranks) {
      for (let file of this.files) {
        yield rank + file
      }
    }
  },
}

console.log([...chessBoard])

// ['1a', '1b', '1c', '1d', '1e', '1f', '1g', '1h', '2a', '2b', '2c', '2d', '2e', '2f', '2g', '2h', '3a', '3b', '3c', '3d', '3e', '3f', '3g', '3h', '4a', '4b', '4c', '4d', '4e', '4f', '4g', '4h', '5a', '5b', '5c', '5d', '5e', '5f', '5g', '5h', '6a', '6b', '6c', '6d', '6e', '6f', '6g', '6h', '7a', '7b', '7c', '7d', '7e', '7f', '7g', '7h', '8a', '8b', '8c', '8d', '8e', '8f', '8g', '8h]

Another example that combines both generator delegation (yield*) and iterating over complex data structures you can find here.

6. Infinite sequences:

Generators can be used to create infinite sequences, which can be useful for generating unique identifiers, random numbers, or other values that need to be produced continuously.

In the following example we created a while (true) loop 🙀 but the fact that the generator function is pausable with the hlep of the yield keyword allows us to create that infinite sequence of unique identifiers without blocking the main thread and causing the system to freeze.

function* uniqueIdGenerator() {
  let id = 1
  while (true) {
    yield id
    id++
  }
}

const idGenerator = uniqueIdGenerator()
console.log(idGenerator.next().value) // 1
console.log(idGenerator.next().value) // 2
console.log(idGenerator.next().value) // 3
// ... and so on

7. Coroutines and cooperative multitasking:

Generators can be utilized for creating coroutines, which are functions that can be paused and resumed, allowing for more sophisticated control flows and cooperative multitasking.

In the following example, coroutineA and coroutineB act as coroutines that yield values in a cooperative manner. The runCoroutines function takes an array of coroutines and runs them concurrently, allowing them to yield values one after the other.

function* coroutineA() {
  yield 'A1'
  yield 'A2'
  yield 'A3'
}

function* coroutineB() {
  yield 'B1'
  yield 'B2'
  yield 'B3'
}

function runCoroutines(coroutines) {
  let activeCoroutines = coroutines.slice():
  while (activeCoroutines.length > 0) {
    activeCoroutines = activeCoroutines.filter((coroutine) => {
      const { value, done } = coroutine.next():
      if (!done) {
        console.log(value)
        return true
      }
      return false
    })
  }
}

runCoroutines([coroutineA(), coroutineB()])
// Output:
// A1
// B1
// A2
// B2
// A3
// B3

I hope these examples have given you a better understanding on generators and demonstrate well how generators can yield and gain control and even remember their state between each next() call, because this is a very crucial thing and it's the key to understanding how generators work.

Performance Considerations

Iterators and generators can have good performance, but their performance characteristics depend on the specific use case and how they are implemented. Since they rely on the creation of iterator objects and use the yield statement for control flow, they can sometimes be slower than traditional loops or recursion. Therefore, it's crucial to evaluate whether the benefits of using generators outweigh the potential performance costs in your specific use case.

Real World Example - SWAPI (Star Wars API) with Async Generators

First, apologies in advance for resorting to such a clichéd example, but hey, who doesn't love a good Star Wars reference?

sarcastic

The reason behind selecting the SWAPI for this example (besides the fact that it's free and doesn't require account creation, API key, and getting spammed for the rest of your life 🫢), is its paginated structure, which serves as an ideal scenario to showcase the capabilities of generators. By efficiently traversing multiple pages of data, generators enable us to retrieve and process the extensive Star Wars information while maintaining code readability and efficiency. Therefore, the SWAPI offers an excellent opportunity to explore the benefits of generators in a real-world context.

So, let's get started!

We'll start by creating an async generator function that will retrieve the paginated data and yield the results of a given SWAPI entity (people, planets, films...) until it's done.

import fetch from 'node-fetch'

const getSwapiPageGenerator = (entity) => {
  return async function* () {
    let nextPage = `https://swapi.dev/api/${entity}/`
    do {
      const response = await fetch(nextPage)
      const data = await response.json()
      nextPage = data.next
      yield* data.results
    } while (nextPage)
  }
}

Then, we'll create an object that holds the async generator function for each SWAPI entity and represent the API endpoints.

const swapiEntities = {
  people: {
    [Symbol.asyncIterator]: getSwapiPageGenerator('people'),
  },
  planets: {
    [Symbol.asyncIterator]: getSwapiPageGenerator('planets'),
  },
  films: {
    [Symbol.asyncIterator]: getSwapiPageGenerator('films'),
  },
  species: {
    [Symbol.asyncIterator]: getSwapiPageGenerator('species'),
  },
  vehicles: {
    [Symbol.asyncIterator]: getSwapiPageGenerator('vehicles'),
  },
  starships: {
    [Symbol.asyncIterator]: getSwapiPageGenerator('starships'),
  },
}

The [Symbol.asyncIterator] help us to use this as a for await...of loop, which is the same as with the [Symbol.iterator] for for...of loop, but for async generators 😬 and that means that we can use this object as if it was an array.

Finally, we'll create a wrapper object that will expose the async generator functions for each SWAPI entity. Then we can use the wrapper object to iterate over all the pages of each entity.

const swapi = {
  getPeople: () => swapiEntities.people,
  getPlanets: () => swapiEntities.planets,
  getFilms: () => swapiEntities.films,
  getSpecies: () => swapiEntities.species,
  getVehicles: () => swapiEntities.vehicles,
  getStarships: () => swapiEntities.starships,
}

for (const entity of Object.keys(swapiEntities)) {
  for await (const page of swapiEntities[entity]) {
    console.log(page)
  }
}

You can find the full code for this example here.

its-finally-over

Final Thoughts

JavaScript generators are an incredibly powerful tool for writing clean, efficient, and maintainable code. Moreover, generators can even teach us some underline computer science concepts such as control flow, system programming, coroutines, and multitasking. They enable custom iterators with pausable and resumable behavior, making them ideal for handling large datasets, asynchronous code, or complex control flows. By understanding and leveraging their advanced use cases, such as handling asynchronous code, generator delegation, bidirectional communication, lazy evaluation, and iterating over complex data structures, we can unlock the full potential of JavaScript generators and create more robust applications. However, always consider the performance implications to ensure that generators are the right choice for your specific use case.