Submitting sites

For submitting sites through the web interface, see the user guide. This page is for submitting sites via the API.

Using chloris-app-sdk (Python)

You can submit a site in the chloris-app-sdk by using the client.submit_site function. This function will upload the boundary, wait for it to be normalized, and then submit the site for processing.

Here is an example which will upload a new site from a geoJSON file:

from chloris_app_sdk import ChlorisAppClient

# create the client
client = ChlorisAppClient(
    organization_id="your-organization-id",
    refresh_token="your-refresh-token", # env: CHLORIS_REFRESH_TOKEN
    # or
    # id_token="your-id-token" # env: CHLORIS_ID_TOKEN
)

# submit the site
reporting_unit = client.submit_site(
    label="Site 1",
    description="Site 1 description",
    tags=["ARR"],
    boundary_path="path/to/boundary.geojson",
    # or boundary_path="https://example.com/path/to/boundary.geojson",

    # optional control site
    control_boundary_path="path/to/control.geojson",
    # or control_boundary_path="https://example.com/path/to/control.geojson",

    # optionally disable email notifications for this site, recommended if using webhooks
    # notify=False,
)

# the new reporting unit entry
print(reporting_unit)
# {
#   organizationId: "...",
#   reportingUnitId: "...",
#   createdAt: "...", # now, ISO 8601
#   updatedAt: "...", # now, ISO 8601
#   controlReportingUnitId: "..."
# }

If you are using webhooks, you will be notified when the site is ready. Otherwise, you can poll the API to check the status of the site:

import time

# ...

reporting_unit_id = reporting_unit["reportingUnitId"]

i = 0
while True:
    # get the reporting unit and check if it is ready
    reporting_unit = client.get_reporting_unit(reporting_unit_id)
    if reporting_unit.get("qualityControlledAt"):
        break
    # sites are ready in hours to days, so we recommend waiting an initial 15 minutes, then exponentially backing off. This is also why we recommend using webhooks.
    time.sleep((60 * 15) * (2 ** i))
    i += 1

Here is an example which submits a local shapefile:

from chloris_app_sdk import ChlorisAppClient

# create the client
client = ChlorisAppClient(
    organization_id="your-organization-id",
    refresh_token="your-refresh-token", # env: CHLORIS_REFRESH_TOKEN
    # or
    # id_token="your-id-token" # env: CHLORIS_ID_TOKEN
)

# submit the site
site = client.submit_site(
    label="Site 1",
    description="Site 1 description",
    tags=["ARR"],
    boundary_path="path/to/boundary.shp",
    period_change_start_year=2000,
    period_change_end_year=2023, # or None for the most recent year
    resolution=30, # or 10 for 10m resolution analysis
    forest_baseline_year=2000
)

Using the REST API (JavaScript)

Submitting sites via the API is a two-step process. First, you must upload the boundary (and optionally a control boundary) for normalization. Then, you can submit the site using the normalized boundary(s).

Here is an example of submitting a site using the aws-sdk in JavaScript:

// submitSite.mjs
// 1. use the aws-sdk to get temporary sts credentials
// 2. upload the file to S3
// 3. request it be normalized
// 4. wait for the normalized file to be available,
// 5. submit the reporting unit to the API

// node-specific imports
import fetch from "node-fetch";
import fs from "fs";
import crypto from "crypto";
import { extname, resolve } from "path";

// aws-sdk imports
// install via `npm install @aws-sdk/client-cognito-identity @aws-sdk/credential-provider-cognito-identity @aws-sdk/client-s3 @aws-sdk/credential-provider-cognito-identity`
import { CognitoIdentityClient } from "@aws-sdk/client-cognito-identity";
import { HeadObjectCommand, PutObjectCommand, S3Client } from "@aws-sdk/client-s3";
import { CognitoIdentityProviderClient, InitiateAuthCommand } from "@aws-sdk/client-cognito-identity-provider";
import { fromCognitoIdentityPool } from "@aws-sdk/credential-provider-cognito-identity";

let idToken;

idToken = "your-id-token";
// or
// const refreshToken = "your-refresh-token";

const organizationId = "your-organization-id";
const filePath = resolve("/path/to/local/file.geojson");
const controlFilePath = resolve("/path/to/local/control-file.geojson");

const fileExt = extname(filePath);

const uploadId = crypto.randomUUID();

// Get info about the AWS resources used by the Chloris API
const apiInfo = await (
  await fetch("https://app.chloris.earth/api/info", {
    method: "GET",
  })
).json();

// Create a CognitoIdentityClient
const cognitoClient = new CognitoIdentityClient({
  region: apiInfo.awsRegion,
  credentials: fromCognitoIdentityPool({
    identityPoolId: apiInfo.awsCognitoIdentityPoolId,
    logins: {
      [`cognito-idp.${apiInfo.awsRegion}.amazonaws.com/${apiInfo.awsUserPoolId}`]: idToken,
    },
  }),
});
// get temporary credentials
const credentials = await cognitoClient.config.credentials();

const identityId = credentials.identityId;

const uploadKey = `private/${identityId}/apiUploads/${uploadId}${fileExt}`;
const boundaryKey = `protected/${identityId}/uploads/${uploadId}.geojson`;
const boundaryPath = `s3://${apiInfo.awsUserFilesS3Bucket}/${boundaryKey}`;

// Create S3 client with Cognito credentials
const s3 = new S3Client({
  region: apiInfo.awsRegion,
  credentials: credentials,
});

// Upload the boundary
const s3Upload = new PutObjectCommand({
  Bucket: apiInfo.awsUserFilesS3Bucket,
  Key: uploadKey,
  Body: fs.createReadStream(filePath),
});
await s3.send(s3Upload);

// request the boundary be normalized
const boundaryResponse = await (
  await fetch("https://app.chloris.earth/api/boundary", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${idToken}`,
    },
    body: JSON.stringify({
      organizationId,
      uploadId,
      uploadPath: `s3://${apiInfo.awsUserFilesS3Bucket}/${uploadKey}`,
    }),
  })
).json();

if (boundaryResponse?.message) {
  throw new Error("Failed to submit boundary for normalization: " + boundaryResponse.message);
}
console.log("boundaryResponse: ", boundaryResponse);

await wait_for_boundary_normalization(apiInfo.awsUserFilesS3Bucket, boundaryKey);

// optionally download the normalized boundary to inspect it
// const normalizedBoundary = await s3.send(new GetObjectCommand({
//   Bucket: apiInfo.awsUserFilesS3Bucket,
//   Key: boundaryKey,
// }));
// const normalizedBoundaryData = await streamToBuffer(normalizedBoundary.Body);

// begin optional control boundary upload

const controlFileExt = extname(controlFilePath);
// generate a separate id for the control boundary upload
const controlUploadId = crypto.randomUUID();
const controlUploadKey = `private/${identityId}/apiUploads/${controlUploadId}${controlFileExt}`;
const controlBoundaryKey = `protected/${identityId}/uploads/${controlUploadId}.geojson`;
const controlBoundaryPath = `s3://${apiInfo.awsUserFilesS3Bucket}/${controlBoundaryKey}`;

// Upload the control boundary
const controlS3Upload = new PutObjectCommand({
  Bucket: apiInfo.awsUserFilesS3Bucket,
  Key: controlUploadKey,
  Body: fs.createReadStream(controlFilePath),
});
await s3.send(controlS3Upload);

// request the control boundary be normalized
const controlBoundaryResponse = await (
  await fetch("https://app.chloris.earth/api/boundary", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${idToken}`,
    },
    body: JSON.stringify({
      organizationId,
      uploadId: controlUploadId,
      uploadPath: `s3://${apiInfo.awsUserFilesS3Bucket}/${controlUploadKey}`,
      excludeGeojsonPath: boundaryPath,
    }),
  })
).json();

if (controlBoundaryResponse?.message) {
  throw new Error("Failed to submit control boundary for normalization: " + controlBoundaryResponse.message);
}

await wait_for_boundary_normalization(apiInfo.awsUserFilesS3Bucket, controlBoundaryKey);

// end optional control boundary upload

// Submit the reporting unit
const reportingUnit = await (
  await fetch("https://app.chloris.earth/api/reportingUnit", {
    method: "PUT",
    headers: await get_headers(),
    body: JSON.stringify({
      organizationId: organizationId,
      boundaryPath: boundaryPath,
      controlBoundaryPath: controlBoundaryPath,
      // dryrun: true, // may be used for testing, deletedAt will be set to 3 hours in the future
    }),
  })
).json();

if (reportingUnit?.message) {
  throw new Error("Failed to submit reporting unit: " + reportingUnit.message);
}

console.log(`Reporting unit submitted successfully: ${reportingUnit.reportingUnitId}`);

/// Helper functions ///

/**
 * Get the headers for the API requests, refreshing the idToken if necessary
 *
 * @returns {Promise<{}>}
 */
async function get_headers() {
  // check that the idToken JWT is not expired
  const idTokenDecoded = JSON.parse(Buffer.from(idToken.split(".")[1], "base64").toString());
  if (idTokenDecoded.exp * 1000 < Date.now()) {
    idToken = await refreshTokens(refreshToken, apiInfo);
  }

  return {
    "Content-Type": "application/json",
    Accept: "application/json",
    Authorization: `Bearer ${idToken}`,
  };
}

/**
 * Wait for the boundary to be normalized (long poll S3 for the boundary to be processed)
 * @param {string} bucket
 * @param {string} boundaryKey
 *
 * @returns {Promise<void>}
 */
async function wait_for_boundary_normalization(bucket, boundaryKey) {
  // 15 minute timeout with exponential backoff (note that most boundaries will be normalized in a few minutes)
  console.log("waiting for normalization: ", boundaryKey);
  const timeout = Date.now() + 15 * 60 * 1000;
  let i = 0;
  while (true) {
    try {
      const headObjectCommand = new HeadObjectCommand({
        Bucket: bucket,
        Key: boundaryKey,
      });
      const headResult = await s3.send(headObjectCommand);
      if (headResult?.Metadata?.error) {
        throw new Error(headResult.Metadata.error);
      }
      break;
    } catch (e) {
      if (e.name !== "NotFound") {
        throw e;
      }
    }
    if (Date.now() > timeout) {
      throw new Error("Timeout waiting for boundary to be normalized");
    }
    const delay = 5 * Math.pow(1.5, i);
    i++;
    await new Promise((resolve) => setTimeout(resolve, delay * 1000));
  }
}

/**
 * Exchange a refresh token for an idToken
 *
 * @returns {Promise<string>}
 */
async function refreshTokens() {
  if (typeof refreshToken === "undefined") {
    throw new Error("idToken is expired and no refresh token is available");
  }

  try {
    const client = new CognitoIdentityProviderClient({
      region: apiInfo.awsRegion,
    });
    const command = new InitiateAuthCommand({
      AuthFlow: "REFRESH_TOKEN_AUTH",
      ClientId: apiInfo.awsUserPoolWebClientId,
      AuthParameters: {
        REFRESH_TOKEN: refreshToken,
      },
    });
    const response = await client.send(command);

    // Extract and return the ID token from the response
    return response.AuthenticationResult?.IdToken;
  } catch (error) {
    console.error("Error refreshing tokens:", error);
    throw error; // Rethrow the error for handling by the caller
  }
}

Duplicate boundaries

A boundary can only be live once per organization at a given resolution. PUT /reportingUnit compares the new site's label, resolution and normalized boundary area against the organization's existing sites (deleted sites excluded) and returns 409 Conflict when it finds a match, naming the existing site in the message. The same check runs when a site is approved, so a second copy of an already approved site cannot be approved by mistake.

Submitting the same boundary at both 10 m and 30 m is fine: the two resolutions are separate deliverables. To replace a boundary, delete the existing site first or ask an administrator, who can bypass the check with allowDuplicate: true.

Note for clients that retry: a request that is slow to respond has usually still succeeded. Retrying a submission after a timeout used to create a second copy; it now returns 409, which a retry loop should treat as success.

Upload retention

Boundary uploads are working files, not storage. Two cleanup jobs run against the user files bucket:

• The raw file you upload under private/{identityId}/apiUploads/ (or private/{identityId}/uploads/ from the web app) is deleted 30 days after upload. It is only read once, when the boundary is normalized.
• The normalized boundary under protected/{identityId}/uploads/ is deleted by a weekly job once no site or site version references it, provided the upload itself is older than 30 days. A boundary referenced by a site is kept for as long as the site references it. When a site's analysis completes, the platform keeps its own copy of the boundary and the site stops referencing the upload, so the upload may be removed on the next weekly run regardless of how long ago the site completed.

Submit the site within 30 days of normalizing its boundary, or upload and normalize the boundary again. Do not treat the upload location as storage: keep your own copy of any boundary you may need later. Sites created with dryrun: true do not keep their upload referenced, so their boundaries are removed by the second rule.

Supported file types

When uploading boundaries, GeoJSON is recommended. However, the following file types are also supported:

• GeoJSON (.geojson)
• Zipped Shapefile (.shp.zip)*
• Multi-file Shapefile (.shp, .shx, .dbf, .prj, .cpg)
• GeoPackage (.gpkg)**
• FlatGeobuf (.fgb)
• KML (.kml)***

* Zipped shapefiles may have only contain one shapefile.

** Only the first layer of the GeoPackage will be used.

*** KML files are not recommended, as they often contain invalid geometries (such as unclosed line strings when you would expect a polygon). We recommend using QGIS to convert KML files to GeoJSON before uploading.

When submitting sites to the platform, we recommend using EPSG 4326 (WGS84) as the coordinate reference system (CRS) for your boundary. However, our system can support most CRSs, including UTM zones. Incorrect outputs due to projection issues are solely your responsibility.