This is a somewhat incomplete description of the Azureus messaging
protocol which is further described here:

http://www.azureuswiki.com/index.php/Azureus_messaging_protocol

A client that understands the azureus messaging protocol should set
the 8th bit of the first byte of the reserved area in the handshake.
For example: reserved[0] = 0x80

Like normal bittorrent messages, all azureus messages begin with the
length of the rest of the message in bytes as a 4-byte integer in
network byte order.  This length does not include the 4 bytes for the
length itself.

After that is another 4-byte network byte order integer containing the
length of the following message id string in bytes, then the message
id string, then a 1 byte integer containing the protocol version.  The
current version is 1.

The message payload format for various message IDs is below.

AZ_HANDSHAKE
payload is a bencoded dictionary:
    identity:
        string, 20 bytes client identity.  The identity should be
        calculated once on startup similarly to the peer_id, except
        than each byte should be completely random and not limited to
        alphanumeric characters.
    client:
        string, client name.  eg: Transmission
    version:
        string, client version.  eg: 0.7
    tcp_port: (optional)
        int, tcp port for incoming peer connections.
    udp_port: (optional)
        int, udp listening port (what is this used for exactly?)
    udp2_port: (optional)
        int, udp non-data listening port (I don't know what this is for either)
    handshake_type: (optional)
        int, 0 for plain or 1 for crypto
    messages:
        list, each item is a dict for the supported messages and versions:
            id:
                string, message id
            ver:
                string, 1 byte version, currently 1

AZ_PEER_EXCHANGE
these messages should be sent approximately once a minute (is this true?)
payload is a bencoded dictionary:
    infohash:
        string, 20 byte info_hash for the torrent
    added:
        list, each item is a 6 byte string for an IP address and port
        added since the last peer exchange message.
    added_HST:
        string, one byte for each item in the added list.
        Each byte is the handshake type (0 plain, 1 crypto) for the matching
        item in the added list.
    added_UDP:
        string, two bytes for each item in the added list.
        Each pair of bytes is the UDP port (network byte order) for the
        matching item in the added list.
    dropped:
        same format as added, contains peers dropped since last peer exchange
    dropped_HST:
        same format as added_HST
    dropped_UDP:
        same format as added_UDP

BT_KEEP_ALIVE
zero-length payload

The following messages all have the same payload as the corresponding
bittorrent message, not including the bittorrent 4-byte length and
1-byte id.

BT_CHOKE        - 0
BT_UNCHOKE      - 1
BT_INTERESTED   - 2
BT_UNINTERESTED - 3
BT_HAVE         - 4
BT_BITFIELD     - 5
BT_REQUEST      - 6
BT_PIECE        - 7
BT_CANCEL       - 8