n8n Programmatic Node: execute(), Pagination and Errors

This is the written version of Part 5 of Build your own n8n node. A workflow runs every night, and every night it says success. It has also been skipping most of the data the whole time: 50 contacts out of 1,240, and nobody gets an error.

By the end of this part the same node returns all of them, and keeps going when one record is bad. Getting there means reaching the limit of the declarative style and rewriting part of what we built in Part 3. Every node that grows goes through this.

Pages

Your API doesn't hand over 1,240 contacts at once. It sends fifty and a note that says "there's more, ask me again." That's pagination, and almost every API does it.

Acme CRM's list endpoints work like many others: you pass a cursor, and the response has the page of data plus a next cursor, which is empty on the last page.

{ "data": [ { "id": "c_1", "email": "asha@example.com" } ], "next": "eyJwYWdlIjoyfQ" }

Our node took the first fifty, ignored next, and said "all done." It has to keep asking until there's nothing left.

The wall

Declarative nodes can handle simple paging on their own, so paging isn't what stops us.

This is. A user wants to "create a deal for the contact with this email address." That's two requests. First find the contact. Then create the deal, using the ID that came back. The second request needs the answer to the first.

In a declarative node, one operation is one request. That's the price of all that simplicity, and this is the wall Part 1 mentioned. To get past it, the node becomes programmatic.

The part you won't like

A node is one style or the other. The moment it has an execute() method, n8n stops using the routing you wrote in Part 3.

So yes, that part gets rewritten. It's less than it sounds. Everything the user sees stays: the resources, operations, fields and displayOptions all carry over unchanged. Only "how is this sent" moves, out of routing and into a function. Delete the routing and requestDefaults blocks as you go, so nobody reads them later and thinks they still do something.

One helper for every request

Put the HTTP call in one place. httpRequestWithAuthentication adds the credential from Part 4 the same way declarative requests did:

// nodes/AcmeCrm/GenericFunctions.ts
import type {
  IDataObject,
  IExecuteFunctions,
  IHttpRequestMethods,
  ILoadOptionsFunctions,
} from 'n8n-workflow';

const BASE_URL = 'https://api.acme.example/v1';

export async function acmeRequest(
  this: IExecuteFunctions | ILoadOptionsFunctions,
  method: IHttpRequestMethods,
  path: string,
  body?: IDataObject,
  qs?: IDataObject,
) {
  return await this.helpers.httpRequestWithAuthentication.call(this, 'acmeCrmApi', {
    method,
    url: `${BASE_URL}${path}`,
    body,
    qs,
    json: true,
  });
}

Return All and Limit

Give every Get Many operation the two fields every n8n node has. The descriptions below are the exact wording n8n's linter expects:

{
  displayName: 'Return All',
  name: 'returnAll',
  type: 'boolean',
  default: false,
  description: 'Whether to return all results or only up to a given limit',
  displayOptions: { show: { resource: ['contact'], operation: ['getAll'] } },
},
{
  displayName: 'Limit',
  name: 'limit',
  type: 'number',
  typeOptions: { minValue: 1 },
  default: 50,
  description: 'Max number of results to return',
  displayOptions: { show: { resource: ['contact'], operation: ['getAll'], returnAll: [false] } },
},

The paging loop

The paging itself is a loop with a question at the end. Ask for a page, keep what came back, and if there's more, ask again:

export async function acmeRequestAllItems(
  this: IExecuteFunctions,
  path: string,
  qs: IDataObject = {},
  limit?: number,
): Promise<IDataObject[]> {
  const results: IDataObject[] = [];
  let cursor: string | undefined;
  do {
    const page = await acmeRequest.call(this, 'GET', path, undefined, { ...qs, cursor, limit: 100 });
    results.push(...(page.data as IDataObject[]));
    cursor = (page.next as string) || undefined;
  } while (cursor && (limit === undefined || results.length < limit));
  return limit === undefined ? results : results.slice(0, limit);
}

With Return All on, run it and watch the count: 1,240, all of them. Somewhere, that's the difference between someone's report being right and being wrong.

If your API pages differently (page numbers, offsets, or a Link header), the loop changes shape, but the question at the end is the same: is there more?

execute(): items in, loop, items out

execute() is the heart of every programmatic node. n8n hands you the items coming in from the previous node, which could be one or five hundred. For each one you read what the user picked, do the work, and collect the result:

async execute(this: IExecuteFunctions): Promise<INodeExecutionData[][]> {
  const items = this.getInputData();
  const returnData: INodeExecutionData[] = [];

  for (let i = 0; i < items.length; i++) {
    try {
      const resource = this.getNodeParameter('resource', i) as string;
      const operation = this.getNodeParameter('operation', i) as string;
      let results: IDataObject[] = [];

      if (resource === 'contact' && operation === 'getAll') {
        const returnAll = this.getNodeParameter('returnAll', i) as boolean;
        const limit = returnAll ? undefined : (this.getNodeParameter('limit', i) as number);
        results = await acmeRequestAllItems.call(this, '/contacts', {}, limit);
      }

      if (resource === 'deal' && operation === 'createForContactEmail') {
        const email = this.getNodeParameter('email', i) as string;
        const title = this.getNodeParameter('title', i) as string;
        const found = await acmeRequest.call(this, 'GET', '/contacts', undefined, { email, limit: 1 });
        const contact = (found.data as IDataObject[])[0];
        if (!contact) {
          throw new NodeOperationError(this.getNode(), `No contact has the email ${email}`, {
            itemIndex: i,
            description: "Check the 'Email' field, or create the contact first",
          });
        }
        results = [await acmeRequest.call(this, 'POST', '/deals', { title, contactId: contact.id })];
      }

      for (const result of results) {
        returnData.push({ json: result, pairedItem: { item: i } });
      }
    } catch (error) {
      if (this.continueOnFail()) {
        returnData.push({ json: { error: (error as Error).message }, pairedItem: { item: i } });
        continue;
      }
      throw new NodeOperationError(this.getNode(), error as Error, { itemIndex: i });
    }
  }

  return [returnData];
}

The second branch is the two-step operation that declarative couldn't do: find the contact, then create the deal with its ID.

The mistake everyone makes: one item

See the for loop? It's the part people forget. They write the code for one item, test it with one item, and ship it. Then somebody connects a spreadsheet with ten rows and gets one result back.

Always loop, and always read parameters with the item index i. Even when it feels like there will only ever be one item, there won't be.

When one item fails

Three contacts go in, and the second has an invalid email. By default the whole run stops: the first worked, the third never ran. Sometimes that's right.

Often the user wants "skip the bad one and keep going." n8n has a setting for that, On Error → Continue, and it only works if your node respects it. That's the catch block above:

• if this.continueOnFail() is true, add an item carrying the error and move on to the next item
• if not, stop with a clear NodeOperationError that says which item broke

For errors your own code detects, like "no contact has that email", throw a NodeOperationError with itemIndex and a description that tells the user how to get unstuck. n8n's UX guidelines ask for exactly that: what happened, and what to do next.

pairedItem: which came from which

One small field in every item you return: pairedItem: { item: i }. It tells n8n which input item an output came from.

It looks like nothing. Leave it out, and later in the workflow, when someone writes an expression like "the email from that earlier step," n8n can't work out which item they mean. That shows up as a strange error three nodes away, and nobody thinks to blame yours.

Declarative nodes get this for free. Programmatic nodes have to set it.

What you have now

• an execute() that handles every item, not just the first
• Return All and Limit, with a loop that pages to the end
• one bad item that no longer loses the rest
• items that remember where they came from

It works. Open it the way a stranger would, though, and the first thing you see is a field called Contact ID with an empty box. Nobody knows their contact IDs. In Part 6 that box becomes a searchable list of the user's own contacts.

One more thing. Paging is one place a generator has to be careful. NativeShip, the tool we build, only writes the paging loop when your OpenAPI spec states the paging rules plainly. When it doesn't, it keeps the operation to one page and tells you so, instead of guessing.