Frame SDK for Flutter

The Frame SDK for Flutter is available in a low-level library (frame_ble) for Bluetooth LE connectivity using Flutter Blue Plus with a limited Lua REPL, and an application-level library (frame_msg) for passing rich objects between your host program and Frame, such as images, streamed audio, IMU data and rasterized text.

Additionally for Flutter developers the community has contributed even higher-level options for getting started quickly with single-page demo Frame applications, including specific helpers for computer vision pipelines.

frame_ble package: low-level connectivity
frame_msg package: application-level messaging
Higher-level application scaffolding
Legacy SDKs

`frame_ble` package: low-level connectivity

API Reference

Source

Example

The BrilliantBluetooth class allows a host to connect to Frame, initiating pairing if required. A sequence of Lua strings can be sent to Frame for execution, optionally returning their results as strings. Transmitted and received strings must be shorter than the Bluetooth MTU (BrilliantDevice.maxStringLength()).

Frame-specific Lua APIs can be called to draw on the display, sample the accelerometer, put Frame into firmware update mode, set the tap and user data callbacks, and more.

In addition to Lua strings, BrilliantDevice can also send control commands (break and reset), as well as custom user data messages that can be handled by a user-defined data handler function on Frame. See Talking to Frame Over Bluetooth for more details.

Consult the API Reference for the complete list of functions available in frame_ble, and the Example to see them in action. Don’t see an example you need? Ask in Discord to have one added.

Note: It is expected that most application developers will prefer to use the frame_msg package to make use of first-class messages and efficient transport for sprites, images, audio clips, and to avoid having to wrangle Bluetooth packet fragmentation and assembly. The frame_ble package, with only a dependency on the flutter_blue_plus library, is still a handy choice for "Hello, World!", simple utilities, or as the basis of a custom application framework in Flutter.

Note: Frame SDK for Flutter is under active development and it is expected that some frame_ble classes may change to harmonize with the Python SDK naming, while retaining some differences due to the mobile context.

Example usage

  // ...
  Future<void> _showCounterOnFrame() async {
    // connect to Frame the first time
    if (_frame == null) {

      // request permission to use Bluetooth
      try {
        await BrilliantBluetooth.requestPermission();
      } catch (e) {
        print('Error requesting permission to use Bluetooth: $e');
        return;
      }

      // look for a Frame and connect to it
      BrilliantScannedDevice? scannedFrame;
      try {
        scannedFrame = await BrilliantBluetooth.scan().first;
      } catch (e) {
        print('Error scanning for Frame: $e');
        return;
      }

      try {
        // connect to the scanned Frame
        _frame = await BrilliantBluetooth.connect(scannedFrame);

        print('Found Frame: $_frame');
      }
      catch (e) {
        print('Error connecting to Frame: $e');
        return;
      }

      // send a break signal to stop any running Lua main loop
      await _frame!.sendBreakSignal();
    }

    // send the counter to Frame for display
    if (_frame?.state == BrilliantConnectionState.connected) {
      await _frame!.sendString(
        'frame.display.text("Hello, World! ($_counter)", 1, 1) frame.display.show()',
        awaitResponse: false
      );
    }
    else {
      print('Frame is not connected');
      _frame = null;
    }
  }
  // ...

Selected Examples
“Hello, World!”

`frame_msg` package: application-level messaging

API Reference

Source

Example

frame-msg enables both the hostside and Frameside halves of an application to communicate using richly typed messages - for example transmitting Sprites to Frame with width, height, palette data and pixel data attributes. (See the code below for an example.)

Additionally, frame-msg uses a programming model in which a main event loop is running on Frame. As a result, messages are exchanged asychronously and handled in an event-driven way. This is in contrast to the “Lua REPL” model, in which single statements are sent to Frame to be executed, and the host attempts to block until execution has completed. In a REPL-style model, it would not be practical to create an application that for example continuously streams audio from Frame to host, and simultaneously sends text back for display (a live translation app). In contrast, with a main event loop running on Frame it is possible to send some audio back to the host if some is available, and check for messages indicating text is ready for display, then loop around again.

How it works

In order to serialize and deserialize large objects, use frame_ble’s BrilliantDevice.send_message() function that fragments large payloads into multiple Bluetooth packets marked with a matching msg_code, and efficiently reassembles them on Frame. After parsing the reconstructed payload into the desired type, a complete object is made available for the application’s main loop in Lua.

Handling rich message types is done by providing symmetrical packing code and parsing code for standard types.

For “Transmit” (Tx) message types, the packing (serialization) code is in the Dart/Flutter type (e.g. TxSprite.pack()) and the parsing (deserialization) code is in the corresponding Lua module (sprite.lua / parse_sprite()).
For “Receive” (Rx) message types, e.g. RxPhoto, the serialization code is present implicitly in the Lua module camera.lua in capture_and_send(), and the deserialization code is present in the RxPhoto class.

This approach presents a natural extension point for user-defined types too: a custom application-specific message might contain an icon, a heading and some detailed text - and this can all be sent together and used all at once on Frame to update the display. By implementing the packing and parsing code for user-defined types, custom objects can be used in applications in the same way as the standard types.

Startup sequence

frame-msg applications tend to follow a startup sequence that ensures Frame is in a known state for the application to run, namely:

Optionally send a break/reset/break signal sequence to ensure that Frame is in Lua REPL mode and its memory is freshly cleared
Upload standard Lua libraries to Frame: data for handling all messaging, plus any others to support specific messages the application uses (e.g. sprite, plain_text, etc.)
Upload the main Frame application Lua file (e.g. my_frame_app.lua). This can often be an exact copy of an example main Lua file that handles the same message types.
Start the main application loop on Frame (after which no Lua REPL commands will be accepted by Frame).

After the Frameside application reports that it has started, the hostside and Frameside application halves begin messaging each other asynchronously: passing text, images, audio etc.

Example usage

sprite_display.dart

  // ...
  // connect to Frame using `frame-ble` in the usual way
  // frame = await BrilliantBluetooth.connect(scannedFrame);
  // ...
  Future<void> run() async {
    // Open the file picker
    FilePickerResult? result = await FilePicker.platform.pickFiles(
      type: FileType.custom,
      allowedExtensions: ['png'],
    );

    if (result != null) {
      File file = File(result.files.single.path!);
      Uint8List pngBytes = await file.readAsBytes();

      // send the Sprite to Frame
      var msg = TxSprite.fromPngBytes(pngBytes: pngBytes);
      await frame!.sendMessage(0x20, msg.pack());
    }
  }
  // ...

sprite_frame_app.lua

local data = require('data.min')
local sprite = require('sprite.min')

-- Phone to Frame flags
USER_SPRITE = 0x20

-- register the message parsers so they are automatically called when matching data comes in
data.parsers[USER_SPRITE] = sprite.parse_sprite

-- Main app loop
function app_loop()
  frame.display.text('Frame App Started', 1, 1)
  frame.display.show()

  -- tell the host program that the frameside app is ready (waiting on await_print)
  print('Frame app is running')

  while true do
    rc, err = pcall(
      function()
        -- process any raw data items, if ready
        local items_ready = data.process_raw_items()

        -- one or more full messages received
        if items_ready > 0 then

          if data.app_data[USER_SPRITE] ~= nil then
            local spr = data.app_data[USER_SPRITE]

            -- set the palette in case it's different to the standard palette
            sprite.set_palette(spr.num_colors, spr.palette_data)

            -- show the sprite
            frame.display.bitmap(1, 1, spr.width, 2^spr.bpp, 0, spr.pixel_data)
            frame.display.show()

            -- clear the object and run the garbage collector right away
            data.app_data[USER_SPRITE] = nil
            collectgarbage('collect')
          end

        end

        -- can't sleep for long, might be lots of incoming bluetooth data to process
        frame.sleep(0.001)
      end
    )
    -- Catch an error (including the break signal) here
    if rc == false then
      -- send the error back on the stdout stream and clear the display
      print(err)
      frame.display.text(' ', 1, 1)
      frame.display.show()
      break
    end
  end
end

-- run the main app loop
app_loop()

Selected Examples	Flutter	Lua
Display a Sprite (`simple_frame_app` example)	Flutter	Lua

Higher-level application scaffolding

In addition to frame-msg and frame-ble, the Frame developer community has provided some even higher-level scaffolding for creating demo apps and to experiment with ideas quickly, but are not recommended for production apps:

Simple Frame App

| | API Reference | Source |

Quickstart scaffolding for a single-page mobile application that handles basic connection to Frame and loading of Frameside application resources including Lua files and sprites.

Selected Examples	Flutter	Lua
Display a Sprite	Flutter	Lua
Wikipedia Viewer	Flutter	Lua
Pollinations.AI Image Generation	Flutter	Lua
Audio Clip Recorder	Flutter	Lua
Live Streaming Transcription	Flutter	Lua

Frame Vision App

| | API Reference | Source |

Even more specialized quickstart scaffolding within the simple_frame_app package, designed for computer vision prototyping. In addition to handling connection and application resources on Frame, the Frame Vision App mixin conveniently provides images from the Frame camera directly to your code for custom processing.

Selected Examples	Flutter	Lua
Live Camera Feed	Flutter	Lua
Hotdog or not hotdog	Flutter	Lua
Mobile Image Classification	Flutter	Lua
Mobile Vision Translation	Flutter	Lua

“Mobile app with no mobile coding”

Mobile App

Sample API Backend

Just want to code your computer vision backend in Python, but want a mobile app experience? Use this simple mobile app as-is (or customize) and point it to your own API endpoint for image processing, sending the result back to Frame for display.

Legacy SDKs

Note: The legacy SDK package is available for frame-sdk as previously documented here.

In order to allow richer experiences on Frame including image display and audio streaming, the decision was made to move to an SDK model based on asynchronous message-passing between a main application loop running on the host, and a main event loop running on Frame. This approach is used in both the Python and Flutter SDKs.

While this requires a slight adjustment from the way the legacy SDKs were used, there are many examples provided (e.g. Sprite Viewer, Unicode Text Display, Teleprompter, Mobile Image Classifier, Realtime Multimodal Assistant) and a helpful community in Discord to help you build your project.

Frame SDK for Flutter

frame_ble package: low-level connectivity

Example usage

frame_msg package: application-level messaging

How it works

Startup sequence

Example usage

Higher-level application scaffolding

Simple Frame App

Frame Vision App

“Mobile app with no mobile coding”

Legacy SDKs

`frame_ble` package: low-level connectivity

`frame_msg` package: application-level messaging