Skip to content
GitLab
Commit 601d9c09 authored by Kaloyan Tenchov's avatar Kaloyan Tenchov
Browse files

Merge branch 'dev'

parents 8ebbdee8 a0672407
Hide whitespace changes
Inline Side-by-side
.gitignore
View file @ 601d9c09
......@@ -4,11 +4,15 @@
__pycache__/
*.pyc
build/
dist/
/build/
/dist/
*.egg-info/
venv/
.idea/
/venv/
/.idea/
examples/*.png
/docs/source/build/
/docs/source/generated/
/docs/source/external/
/examples/*.png
CHANGES.md
View file @ 601d9c09
Revision History
================
v0.6.0 (Jan 2, 2020)
--------------------
- Improved localization - SetOrigin and SyncTime commands and pose (position and orientation) interpretation.
- Added new path tracking commands (AppendPath*, ExecutePath, etc.) and examples (path.py, go_to_pose.py).
- Added support for drawing procedural faces.
- Added support for reading and writing animations in FlatBuffers (.bin) and JSON format.
- Added a new tool for examining and manipulating animation files - pycozmo_anim.py .
- Added commands for working with cube/object accelerometers - StreamObjectAccel, ObjectAccel.
- Improved function description.
- Bug fixes and documentation improvements.
v0.5.0 (Oct 12, 2019)
---------------------
- Added initial client API.
......
LICENSE.md
View file @ 601d9c09
The MIT License (MIT)
Copyright (c) 2019
Copyright (c) 2019-2020
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
......
README.md
View file @ 601d9c09
PyCozmo
=======
......@@ -45,28 +46,35 @@ cli.stop()
Documentation
-------------
- [Cozmo protocol](docs/protocol.md) description
- [Cozmo function](docs/functions.md) description
- [Capturing Cozmo communication](docs/capturing.md)
- API documentation: http://pycozmo.readthedocs.io/
[https://pycozmo.readthedocs.io/](https://pycozmo.readthedocs.io/)
Examples
--------
- [rc.py](examples/rc.py) - turns Cozmo into an RC tank that can be driven with an XBox 360 Wireless controller or
Logitech Gamepad F310
Basic:
- [minimal.py](examples/minimal.py) - minimal code to communicate with Cozmo, using PyCozmo
- [go_to_pose.py](examples/go_to_pose.py) - demonstrates moving to a specific pose (position and orientation)
- [extremes.py](examples/extremes.py) - demonstrates Cozmo lift and head control
- [backpack_lights.py](examples/backpack_lights.py) - demonstrates Cozmo backpack LED control
- [cube_lights.py](examples/cube_lights.py) - demonstrates cube connection and LED control
- [cube_light_animation.py](examples/cube_light_animation.py) - demonstrates cube LED animation control
- [charger_lights.py](examples/charger_lights.py) - demonstrates Cozmo charging platform LED control
- [display_image.py](examples/display_image.py) - demonstrates visualization of image files on Cozmo's display
- [events.py](examples/events.py) - demonstrates event handling
- [camera.py](examples/camera.py) - demonstrates capturing a camera image
- [path.py](examples/path.py) - demonstrates following a predefined path
Advanced:
- [display_lines.py](examples/display_lines.py) - demonstrates 2D graphics, using
[PIL.ImageDraw](https://pillow.readthedocs.io/en/stable/reference/ImageDraw.html) on Cozmo's display
- [rc.py](examples/rc.py) - turns Cozmo into an RC tank that can be driven with an XBox 360 Wireless controller or
Logitech Gamepad F310
- [cube_lights.py](examples/cube_lights.py) - demonstrates cube connection and LED control
- [cube_light_animation.py](examples/cube_light_animation.py) - demonstrates cube LED animation control
- [charger_lights.py](examples/charger_lights.py) - demonstrates Cozmo charging platform LED control
- [audio.py](examples/audio.py) - demonstrates 22 kHz, 8-bit, mono WAVE file playback through Cozmo's speaker
- [events.py](examples/events.py) - demonstrates event handling
- [camera.py](examples/camera.py) - demonstrates capturing a camera image
- [nvram.py](examples/nvram.py) - domonstrates reading data from Cozmo's NVRAM (non-volatile memory)
- [procedural_face.py](examples/procedural_face.py) - demonstrates drawing a procedural face on Cozmo's display
- [procedural_face_show.py](examples/procedural_face_show.py) - demonstrates generating a procedural face
- [anim.py](examples/anim.py) - demonstrates animating Cozmo
Tools
......@@ -77,6 +85,7 @@ Tools
[pypcap](https://github.com/pynetwork/pypcap).
- [pycozmo_replay.py](tools/pycozmo_replay.py) - a basic command-line application that can replay .pcap files back to
Cozmo.
- [pycozmo_anim.py](tools/pycozmo_anim.py) - a tool for examining and manipulating animation files.
Robot Support
......@@ -110,9 +119,11 @@ Storage:
- [x] NVRAM
- [x] Firmware update
Other:
- [ ] Animations from FlatBuffers .bin files - work in progress
- [ ] Procedural faces - work in progress
Functions:
- [x] Localization
- [x] Path tracking
- [x] Procedural faces
- [x] Animations from FlatBuffers .bin files
Connecting to Cozmo over Wi-Fi
......@@ -201,8 +212,15 @@ Support
Bug reports and patches should be sent via GitHub:
https://github.com/zayfod/pycozmo
[https://github.com/zayfod/pycozmo](https://github.com/zayfod/pycozmo)
Anki Robot Discord server, channel #cozmo:
https://discord.gg/ew92haS
[https://discord.gg/ew92haS](https://discord.gg/ew92haS)
Disclaimer
----------
This project is not affiliated with [Anki](https://anki.com/) or
[Digital Dream Labs](https://www.digitaldreamlabs.com/).
docs/esp8266.md 0 → 100644
View file @ 601d9c09
ESP8266
=======
The ESP8266 is the main Cozmo controller, responsible for Wi-Fi communication.
SPI Flash Memory Map
--------------------
The SPI flash size is 2 MB.
The below memory map has been reconstructed based on a SPI flash memory dump and `NvEntryTag` values.
```
Offset Length Type Description
---------------------------------------------------------------------------------
0x00000000 0x00001000 Code Bootloader.
0x00001000 0x00001000 Data Unknown. The first 4 bytes are the head serial number.
0x00002000 0x00001000 Data Unknown.
0x00003000 0x0007b800 Code Application image 1.
0x0007e800 0x00001800 Data Application image 1 signature. See versions.md .
0x00080000 0x0005e000 Code Recovery image / factory firmware.
0x000de000 0x00000030 Data Birth certificate.
0x000de030 0x00021fd0 Data Factory data.
0x00100000 0x00003000 Data Unknown.
0x00103000 0x0007b800 Code Application image 2
0x0017e800 0x00001800 Data Application image 2 signature. See versions.md .
0x00180000 0x00018000 Data Application data.
0x00198000 0x00028000 Data Empty.
0x001c0000 0x0001e000 Data Factory reserved 1.
0x001de000 0x0001e000 Data Factory reserved 2. Empty?
0x001fc000 0x00001000 Data Unknown.
0x001fd000 0x00001000 Data Wi-Fi configuration 1.
0x001fe000 0x00001000 Data Wi-Fi configuration 2.
0x001ff000 0x00001000 Data Unknown.
```
docs/functions.md
View file @ 601d9c09
......@@ -5,35 +5,68 @@ Cozmo Functions
Overview
--------
### Head
Cozmo is a complex distributed embedded system with the following main parts:
`HardwareInfo`
`FirmwareSignature`
- robot
- cubes
- charging platform
### Body
The robot can be subdivided into:
`BodyInfo`
- head
- body
### Cubes
The head is responsible for the following functions:
- Wi-Fi communication
- OLED display
- speaker
- camera
- accelerometers
- gyro
- NV RAM storage
Once Cozmo is powered on, the head controller (ESP8266) remains always powered on to maintain Wi-Fi communication.
On connection, the head transmits its serial number with the `HardwareInfo` message and firmware version with the
`FirmwareSignature` message.
The body is in charge of:
- left and right tread motors and encoders encoders
- head motor and encoder
- lift motor and encoder
- backpack LEDs
- backpack button (on newer models only)
- Bluetooth LE communication (to cubes and charging platform)
- IR LED
- cliff sensor
- batter charging
The body is powered on with the `Enable` message. The `BodyInfo` message communicates the body hardware version,
serial number, and color.
Cubes are communicated with over Bluetooth LE and provide access to:
- LEDs
- Accelerometers
- Battery voltage
### Charging Platform
Some charging platforms (aka "pads") can be communicated with over Bluetooth LE. They contains 3 RGB LEDs that can be
controlled, similar to cube LEDs.
The following sections provide more details on the use of each function.
Wi-Fi
-----
Wi-Fi is activated automatically when the head board is powered on. The robot operates in access point mode.
Wi-Fi is activated automatically when the head board is powered on. The robot operates in access point (AP) mode.
`cozmoclad` defines a `SetBodyRadioMode` message that seems to allow changing the Wi-Fi channel but it is unclear
how it can be used with the Cozmo protocol.
`WifiOff`
`Shutdown`
Backpack LEDs
-------------
......@@ -44,142 +77,233 @@ The 5 Backpack LEDs can be set controlled with 2 messages:
Each color is defined by a 5-bit value for a total of 32768 colors.
See [backpack_lights.py](../examples/backpack_lights.py) for example usage.
See `examples/backpack_lights.py` for example usage.
Backpack Button
---------------
Newer Cozmo models have a backpack button. The `RobotState` message has a `backpack_touch_sensor_raw` field but
it seems that it's value does not change as a result of button presses.
Newer Cozmo models have a backpack button.
Button press and release events are communicated bu the `ButtonPressed` message. It is immediately available on
Button press and release events are communicated by the `ButtonPressed` message. It is immediately available on
connection and does not require `Enable` to be used.
The `RobotState` message has a `backpack_touch_sensor_raw` field but
it seems that it's value does not change as a result of button presses.
See `examples/events.py` for example usage.
Wheels
------
The left and the right motor speeds can be controlled directly using the `DriveWheels` and `TurnInPlaceAtSpeed`
messages.
In addition, the motors can be stopped using the `StopAllMotors` message.
messages. The motors can be stopped using the `StopAllMotors` message.
The actual speed of wheels is measured with Hall magnetic sensors. The values for each wheel can be
read through the `lwheel_speed_mmps` and `rwheel_speed_mmps` fields of the `RobotState` message.
`TurnInPlace`
`DriveStraight`
In addition, the and `TurnInPlace` message can be used to turn to a specific angle.
Localization
------------
`RobotState`
`RobotDelocalized`
The robot maintains a world frame internally. It's position and orientation with respect to it are transmitted every
30 ms or about 33 times per second with the `RobotState` message.
If the robot is unable to maintain correct position and orientation, for example because it is picked up or pushed, it
will communicate this with a `RobotDelocalized` message.
The origin (0,0,0) of the world frame as well as "pose ID" can be set with the `SetOrigin` message. This is usually done
on initial connection and and on receiving a `RobotDelocalized` message.
The timestamp in `RobotState` messages can be synchronized using the `SyncTime` message.
Path Tracking
-------------
The robot can traverse paths, composed of lines, arcs, and turns in place, described in world frame coordinates. The
`AppendPathSegLine`, `AppendPathSegArc`, and `AppendPathSegPointTurn` messages can be used to build paths.
The last composed path can be executed using the `ExecutePath` message. One of it's arguments can be used to request
the reception of `PathFollowingEvent` message when path traversing finishes.
The `status` filed of the `RobotState` message has a `robot_pathing` flag that indicates whether the robot is currently
travesing a path. The `curr_path_segment` filed indicates which segment is being traversed.
The `ClearPath` message can be used to destroy an already composed path. The `TrimPath` message can be used to delete
path segmens from the beginning or the end of a composed path.
See `examples/path.py` and `examples/go_to_pose.py` for example usage.
Head
----
The head motor can be controlled directly, using the `DriveHead` and `SetHeadAngle` messages.
The head motor can be controlled directly, using the `DriveHead` and `SetHeadAngle` messages. `SetHeadAngle` is always
followed by an `AcknowledgeAction` message before the head starts moving.
In addition, the motor can be stopped using the `StopAllMotors` message.
The actual head angle can be read through the `head_angle_rad` field of the `RobotState` message. The `head_in_pos` flag
of the `status` field indicates whether the head is in position or in motion.
The actual head angle can be read through the `head_angle_rad` field of the `RobotState` message.
The motor can be stopped using the `StopAllMotors` message.
See [extremes.py](../examples/extremes.py) for example usage.
The robot measures the angle of the head, relative to its lowest possible position. This measurement is automatically
triggered on connection. The head can be forced to an unknown angle for example as a result of a fall.
In such situations, the robot recalibrates the head motor automatically. Calibration can also be triggered on request,
using the `StartMotorCalibration` message. The `MotorCalibration` message indicates whether calibration is in progress.
`AcknowledgeAction`
See `examples/extremes.py` for example usage.
Lift
----
The head motor can be controlled directly, using the `DriveLift` and `SetLiftHeight` messages.
The head motor can be controlled directly, using the `DriveLift` and `SetLiftHeight` messages. `SetLiftHeight` is always
followed by an `AcknowledgeAction` message before the lift start moving.
In addition, the motor can be stopped using the `StopAllMotors` message.
The actual lift height can be read through the `lift_height_mm` field of the `RobotState` message. The `lift_inpos` flag
of the `status` field indicates whether the lift is in position or in motion.
The actual lift height can be read through the `lift_height_mm` field of the `RobotState` message.
The motor can be stopped using the `StopAllMotors` message.
See [extremes.py](../examples/extremes.py) for example usage.
The robot measures the angle of the lift, relative to its lowest possible position. It is calibrated similar to the
head motor.
`AcknowledgeAction`
See `examples/extremes.py` for example usage.
OLED display
------------
See [display.py](../examples/display.py) for example usage.
Images can be displayed on the robot's OLED 128x64 display using the `DisplayImage` message. To reduce display burn-in,
consecutive images are interleaved and only half of the display's rows can be used at a time and the effective display
resolution is 128x32.
The Cozmo protocol uses a special run-length encoding to compress images.
Display and audio are synchronized by the `NextFrame` message.
`NextFrame`
`DisplayImage`
`AnimationState`
`AnimationState` message which can be enabled using the `EnableAnimationState` message provide statistics on display
usage.
See `examples/display_image.py` and `examples/display_lines.py` for example usage.
Speaker
-------
See [audio.py](../examples/audio.py) for example usage.
The `OutputAudio` message can be used to transmit 744 audio samples at a time.
Speaker volume can be adjusted with the `SetRobotVolume` message.
`AnimationState` message which can be enabled using the `EnableAnimationState` message provide statistics on audio
usage.
`OutputAudio`
`SetRobotVolume`
`AnimationState`
See `examples/audio.py` for example usage.
Camera
------
See [camera.py](../examples/camera.py) for example usage.
Cozmo can send a stream of camera images in 320x240 (QVGA) resolution at a rate of ~15 frames per second.
The `EnableCamera` message enables camera image reception and the `EnableColorImages` message allows switching between
grayscale and color images.
`EnableCamera`
`SetCameraParams`
`EnableColorImages`
`ImageChunk`
`ImageImuData`
The camera gain, exposure time, and auto exposure can be controlled with the `SetCameraParams` message.
Images are encoded in JPEG format and transmitted as a series of `ImageChunk` messages. The header of the JPEG files is
not transmitted to save bandwidth.
The `ImageImuData` message provides accelerometer readings at the time of capturing every image to allow for motion
blur compensation.
See `examples/camera.py` for example usage.
IR LED
------
The IR head light can be turned on and off using the `SetHeadLight` message.
The IR LED (ala head light) can improve the camera performance in dark environments.
The IR LED can be turned on and off using the `SetHeadLight` message.
Accelerometers
--------------
See [events.py](../examples/events.py) for example usage.
The `RobotState` message communicates accelerometer readings which represent acceleration along the x, y, and z axes.
In addition, the robot automatically detects and communicates 2 types of events. The `RobotPoked` message is sent if
the robot has been moved rapidly by an external force along the x or y axes. The `FallingStarted` and `FallingStopped`
messages are send if the robot is moving rapidly along the z axis.
`RobotState`
`RobotPoked`
`FallingStarted`
`FallingStopped`
See `examples/events.py` for example usage.
Gyro
----
`RobotState`
The `RobotState` message communicates gyro readings which represent angular velocity around the x, y, and z axes.
See `examples/events.py` for example usage.
Cliff Sensor
------------
`RobotState`
`EnableStopOnCliff`
The robot has a "cliff sensor" that measures the distance to ground below the robot. This allows detecting cliffs and
detecting when the robot is being picked up or put down.
The `RobotState` message communicates the raw cliff sensor readings.
In addition, the robot can be made to automatically stop when a cliff is detected with the `EnableStopOnCliff` message.
See `examples/events.py` for example usage.
Battery voltage
---------------
`RobotState`
The `RobotState` message communicates raw battery voltage readings.
NV RAM Storage
--------------
`NvStorageOp`
`NvStorageOpResult`
The robot provides access to some amount of non-volatile memory (aka NV RAM) intended to store two main types of data:
- unit-specific parameters (ex. camera calibration data and cube IDs)
- mobile app data (ex. sparks and unlocked games and tricks)
The NV RAM storage is backed by the head's ESP8266 controller external SPI flash. It is a NOR flash which drives the
following specifics for its use:
- an erase operation is needed before a write operation
- data is erased in pages
The `NvStorageOp` message allows performing read, erase, and write operations. Data is addressed by the `tag` field and
only the values enumerated by `NvEntryTag` can be used. Using any other address results in a `NV_BAD_ARGS`. Tags
smaller than 0x80000000 are direct NOT flash memory addresses. Tags larger than 0x80000000 are virtual addresses that
seem to be stored in the `NVEntry_FactoryBaseTagWithBCOffset` area.
`NvStorageOpResult` messages communicate results of `NvStorageOp` operations.
A backup through the mobile app, preserves the data behind the following keys:
- NVEntry_GameSkillLevels
- NVEntry_Onboarding
- NVEntry_GameUnlocks
- NVEntry_FaceEnrollData
- NVEntry_FaceAlbumData
- NVEntry_NurtureGameData
- NVEntry_InventoryData
- NVEntry_LabAssignments
See `examples/nvram.py` for example usage.
Firmware Updates
......@@ -208,7 +332,7 @@ The "cozmo.safe" files start with a firmware signature in JSON format:
This is the same signature, delivered with the `FirmwareSignature` message on initial connection establishment.
See [versions.md](versions.md) for more examples.
See `docs/versions.md` for more examples.
There seem to be individual signatures for each controller but the structure of the `cozmo.safe` files is not known.
......@@ -245,29 +369,63 @@ A cube has to be "selected" first, using the `CubeId` message. A subsequent `Cub
Cubes can be programmed to perform simple LED light animations autonomously using the `LightState` structure and the
`CubeId.rotation_period_frames` field.
See `examples/cube_lights.py` and `examples/cube_light_animation.py` for example usage.
Cube Battery Voltage
--------------------
`ObjectPowerLevel`
Cube battery voltage is communicated periodically with `ObjectPowerLevel` messages.
Cube Accelerometers
-------------------
`ObjectMoved`
`ObjectStoppedMoving`
`ObjectUpAxisChnaged`
`ObjectTapped`
`ObjectTapFiltered`
Cube accelerometer value reception can be enabled with the `StreamObjectAccel` message and are communicated every 30 ms
with the `ObjectAccel` message.
In addition, the robot performs basic cube accelerometer ata processing and provides basic events with the following
messages:
- `ObjectMoved`
- `ObjectStoppedMoving`
- `ObjectUpAxisChnaged`
- `ObjectTapped`
- `ObjectTapFiltered`
Animations
----------
`AnimHead`
`AnimLift`
`AnimBody`
`AnimBackpackLights`
`OutputAudio`
`Keyframe`
Cozmo "animations" allow animating the following aspects of the robot:
- body movement
- lift movement
- head movement
- face images
- backpack LED animations
- audio
Cozmo animations are series of keyframes, stored in binary files in [FlatBuffers](https://google.github.io/flatbuffers/)
format. Animation data structures are declared in FlatBuffers format in
`files/cozmo/cozmo_resources/config/cozmo_anim.fbs` . The animation files are available in the following directory of
the Android mobile application:
`files/cozmo/cozmo_resources/assets/animations`
Face images are generated procedurally. They are described by 37 parameters - 5 for the face and 19 for each eye.
The face as a whole can be translated, scaled, and rotated. Each individual eye can be translated, scaled, and rotated.
The 4 corners of each eye can be controlled and each eye has a lower and upper lid.
The following presentation from Anki provides some background information on Cozmo animations:
[Cozmo: Animation pipeline for a physical robot](https://www.gdcvault.com/play/1024488/Cozmo-Animation-Pipeline-for-a)
To play animations, `AnimationState` message have to be enabled first using the `EnableAnimationState` message.
Animations are controlled with the `StartAnimation`, `EndAnimation`, and `AbortAnimation` messages.
Keyframes are transferred with the `AnimHead`, `AnimLift`, `AnimBody`, `AnimBackpackLights`, `RecordHeading`,
`TurnToRecordedHeading`, and `OutputAudio` messages.
See `examples/anim.py` for example usage.
docs/protocol.md
View file @ 601d9c09
......@@ -14,7 +14,7 @@ The two exchange frames, encapsulated in UDP packets.
Each frame can contain 0, 1, or more packets.
See [protocol_declaration.py](../pycozmo/protocol_declaration.py) for packet details.
See `protocol_declaration.py` for packet details.
Network Setup
......
docs/source/api.rst 0 → 100644
View file @ 601d9c09
pycozmo package
===============
.. autosummary::
:nosignatures:
:toctree: generated
pycozmo.anim
pycozmo.anim_encoder
pycozmo.camera