Run Location: Cloud, Local, or Automatic

When running AI models in the FreeToken client you can choose where the AI runs: in the cloud (.cloudRun), on your local machine (.localRun), or automatically (.automatic) based on the model, the device capabilities, and device situation (heat, memory, wait time, etc).

By default, the client will automatically choose the best run location for you, but you can override this behavior by specifying the runLocation when running a completion or message thread.

Running a Message Thread

import FreeToken
// ... Configuration and device session registration completed ...

// Tells the client to run the message thread on the local device
await client.runMessageThread(
  id: "msg-thr-id",
  runLocation: .localRun,
  success: { resultMessage in 
    // Successfully ran the message thread on the local device
    print("Message thread result: \(resultMessage.content)")
}, error: { error in
    // If the run location is not supported on the device, 
    // or there are not enough resources, an error will be returned
})

Completion

import FreeToken

// ... Configuration and device session registration completed ...

await FreeToken.shared.generateLocalCompletion(
  prompt: "Complete the following: The wheels on the bus go",
  runLocation: .cloudRun
) { completion in
    // Successfully generated a completion
    print("Completion: \(completion.response)")
} error: { error in
    // Completion could not be generated due to some reason (e.g. network issues)
}

Errors

If the specified run location is not supported on the device, or if there are not enough resources available to run the AI model in that location, an error will be returned. It will be up to your application to handle hte error callback and decide what to do next, possibly by falling back to a different run location or notifying the user.