Idempotency

A request can time out after Ele has already done the work. If you then send it again, you might create a second payment. An idempotency key prevents that: when you retry with the same key, Ele returns the first response instead of doing the work again.

Sending a key

Send an Idempotency-Key header on any POST, PUT, PATCH or DELETE under /api/v1. The key is any unique string; a random UUID works well.

curl -si "$ELE_API_BASE/api/v1/payments" \
  -H "Authorization: Bearer $ELE_TOKEN" \
  -H "Idempotency-Key: $KEY" \
  -H 'Content-Type: application/json' \
  -d "{\"payment_contact_id\": \"$CONTACT_ID\",
       \"amount\": {\"amount\": \"1.00\", \"currency\": \"USD\"},
       \"payout_reason\": \"$PAYOUT_REASON\",
       \"statement_description\": \"Test payment\"}"

The reference pages do not list Idempotency-Key as a parameter, so generated clients have no field for it. Set it as a plain header:

With the Go helper from the example code, pass ele.WithIdempotencyKey(key) to Do.

In a client generated with oapi-codegen, pass a request editor on each call, as in Go client:

package main

import (
  "context"
  "net/http"

  "github.com/elephants-labs/ele/developers/codegen/client"
)

// withIdempotencyKey sets Idempotency-Key on one call. Create the key once
// per operation and pass the same key on every retry of it.
func withIdempotencyKey(key string) client.RequestEditorFn {
  return func(_ context.Context, r *http.Request) error {
    r.Header.Set("Idempotency-Key", key)
    return nil
  }
}

A write without a key works, but it is not protected: a retry runs again. Send a key on every write. If Ele answers 400 with the type idempotency_key_required, that call needs one.

One key per operation

Create a key when you decide to do something, such as “pay this invoice”, and store it with that intent. Use the same key for every retry of that operation, and a new key for each new operation. Never generate the key inside your retry loop.

Each key belongs to the person (or API key) who sent it, together with the method and path. The same key on a different path, or from a different person, is a different key.

What a retry returns

When you retry with a key Ele has seen:

Same body, first request answered: Ele returns the first answer, status and body, without running the request again. The response has the header idempotent-replayed: true. The body is the original, including its request_id.
Same body, first request still running: 409 with the type idempotency_conflict. Wait a few seconds and retry with the same key.
Different body: 422 with the type idempotency_params_mismatch. A key belongs to one request. Use a new key for a different request.

Which answers are kept

Once the operation itself has started, Ele keeps its answer under the key, whatever it was. A retry with the same key returns that answer for 24 hours. This includes errors:

Refusals such as forbidden, not_found, conflict, insufficient_funds or unprocessable_entity.
429 rate_limited. The replay has no Retry-After header, and waiting does not change it.
500 internal_error and 502 provider_unavailable.

Ele does not keep these, so a retry with the same key runs the request again:

A request that failed validation: invalid_request, or 422 unprocessable_entity whose details list the fields that failed. Fix the request and send it with the same key.
A call refused with step_up_required. Complete step-up and retry with the same key.
A 500 with no error.type in the body: something failed unexpectedly inside Ele. See After a failure.
502 xero_unavailable: Xero did not answer, so Ele cannot tell whether the write reached it. Retry with the same key. Ele sends Xero the same key, and Xero returns its original answer instead of writing twice.
On a file upload, a missing, too large or wrong-type file (file_required, file_too_large, invalid_content_type). Fix the file and send it with the same key.

Answers about the key itself, 409 idempotency_conflict and 422 idempotency_params_mismatch, are not kept either: they leave the first request’s record as it was. Handle them as described in What a retry returns above.

Every other answer is kept, including the first one. When Ele returns a kept answer again, the response has idempotent-replayed: true.

Retrying after a kept error

To try again after a kept error, you need a new key. This applies only to kept errors: after 409 idempotency_conflict, or any answer in the not-kept list above, keep the same key. A new key is a new request: if the first one did take effect, a new key does it a second time. A duplicate payment is worse than a payment that was not sent, so choose carefully:

A refusal (4xx), including 429: Ele refused before doing the work. Fix the cause, or wait, and send the request again with a new key.
500 internal_error or 502 provider_unavailable: the work may or may not have happened. Check first, for example by looking for the payment in the organization’s recent transactions. Use a new key only if it did not happen. If you cannot tell, do not retry: show the error to a person.

How long keys last

Ele keeps a key for 24 hours after its first use. After that, the same key starts a new request. Finish your retries well within that time.

After a failure

Two failures need care, because Ele may have done part of the work.

Ele answered 500 with no error.type. The request failed unexpectedly, and Ele kept nothing under the key. A retry with the same key runs the request again. For most writes, check first whether the work happened, as above, before you retry.

Creating a payment, executing a payment and executing a swap are safe to retry with the same key in this case: Ele recognises the key and does not make a second payment or swap.

No answer at all. If Ele stopped while it was working on the request, for example during a restart, a retry with the same key returns 409 idempotency_conflict, for up to 24 hours. Keep retrying with the same key now and then, and meanwhile check whether the work happened. Use a new key only if you are sure it did not. Creating a payment, executing a payment and executing a swap recover by themselves: about 90 seconds after the first attempt, a retry with the same key finishes the work without doing it twice.

Calls that ignore the key

Some responses hold a secret or a one-time link, and Ele never stores them. On these calls Ele ignores the key, so a retry runs again:

Signing in, completing MFA, completing sign-up, accepting an invitation, and starting authenticator setup.
Creating and rotating an API key. A retry creates another key, or rotates again. Do not retry these automatically.
Creating a file upload. A retry creates another upload link.
Identity-check access tokens and Google Wallet provisioning.

The example

This program creates a payment with a new key. A custom transport throws away the first response, as a timeout would after Ele has done the work. The program retries with the same key and gets the same payment back, marked as a replay:

// Idempotent payment: create a payment with an Idempotency-Key, lose the
// response as a timeout would, and retry with the same key. Ele replays the
// first response instead of creating a second payment.
//
// The payment waits for approval. No money moves until a person approves it.
//
// ELE_CURRENCY defaults to USD, the only currency this call accepts today.
//
//  ELE_TOKEN=… ELE_PAYMENT_CONTACT_ID=… ELE_AMOUNT=1.00 \
//  ELE_PAYOUT_REASON=PYR001 ELE_STATEMENT_DESCRIPTION="Test payment" \
//  go run ./idempotent-payment
package main

import (
  "context"
  "errors"
  "fmt"
  "log"
  "net/http"
  "os"
  "time"
  "uuid"

  "github.com/elephants-labs/ele/developers/examples/internal/ele"
)

type money struct {
  Amount   string `json:"amount"`
  Currency string `json:"currency"`
}

type createPayment struct {
  PaymentContactID     string `json:"payment_contact_id"`
  Amount               money  `json:"amount"`
  PayoutReason         string `json:"payout_reason"`
  StatementDescription string `json:"statement_description"`
}

type payment struct {
  ID                string `json:"id"`
  Status            string `json:"status"`
  ApprovalRequestID string `json:"approval_request_id"`
}

func main() {
  ctx := context.Background()
  c := ele.FromEnv()
  lossy := &lossyTransport{}
  c.HTTP.Transport = lossy

  body := createPayment{
    PaymentContactID:     mustEnv("ELE_PAYMENT_CONTACT_ID"),
    Amount:               money{Amount: mustEnv("ELE_AMOUNT"), Currency: envOr("ELE_CURRENCY", "USD")},
    PayoutReason:         mustEnv("ELE_PAYOUT_REASON"),
    StatementDescription: mustEnv("ELE_STATEMENT_DESCRIPTION"),
  }

  // One key for this payment, created once and reused on every retry.
  key := uuid.New().String()

  var p payment
  for attempt := 1; ; attempt++ {
    err := c.Do(ctx, "POST", "/api/v1/payments", body, &p, ele.WithIdempotencyKey(key))
    if err == nil {
      break
    }
    if attempt == 5 || !retryable(err) {
      log.Fatal(err)
    }
    fmt.Fprintf(os.Stderr, "attempt %d: %v; retrying with the same key\n", attempt, err)
    time.Sleep(time.Duration(attempt) * time.Second)
  }

  fmt.Printf("payment %s is %s\n", p.ID, p.Status)
  fmt.Printf("approval request: %s\n", p.ApprovalRequestID)
  fmt.Printf("replayed: %t\n", lossy.replayed)
}

// retryable reports whether to send the same request again with the same key:
// after a network failure, where the outcome is unknown, or a 409
// idempotency_conflict, where the first request is still running. Any other
// error is Ele's answer, and sending the same request again will not change it.
func retryable(err error) bool {
  var apiErr *ele.APIError
  if !errors.As(err, &apiErr) {
    return true
  }
  return apiErr.Type == "idempotency_conflict"
}

// lossyTransport sends each request, but throws away the first response, as
// if the connection dropped after Ele had done the work. It records whether
// a later response was a replay.
type lossyTransport struct {
  dropped  bool
  replayed bool
}

func (t *lossyTransport) RoundTrip(r *http.Request) (*http.Response, error) {
  resp, err := http.DefaultTransport.RoundTrip(r)
  if err != nil {
    return nil, err
  }
  if !t.dropped {
    t.dropped = true
    resp.Body.Close()
    return nil, errors.New("simulated timeout: the response was lost")
  }
  t.replayed = resp.Header.Get("Idempotent-Replayed") == "true"
  return resp, nil
}

func envOr(name, fallback string) string {
  if v := os.Getenv(name); v != "" {
    return v
  }
  return fallback
}

func mustEnv(name string) string {
  v := os.Getenv(name)
  if v == "" {
    log.Fatalf("set %s", name)
  }
  return v
}

The payment waits for approval. No money moves until a person approves it; see Step-up and approvals. Run it against the sandbox with a payment contact paid by Ele ID, that is, a contact who is another Ele account. ELE_CURRENCY defaults to USD, the only currency this call accepts today. Get a payout reason code from getPaymentsConfig:

ELE_TOKEN=… ELE_PAYMENT_CONTACT_ID=… ELE_AMOUNT=1.00 \
ELE_PAYOUT_REASON=PYR001 ELE_STATEMENT_DESCRIPTION="Test payment" \
go run ./idempotent-payment

It prints the payment, its approval request, and replayed: true.