Extended daemon command with JSON-RPC interface via UNIX- and TCP socket

[AsamK]

I've extended the daemon command to not only provide a DBus interface, but also a JSON-RPC interface via a UNIX-socket or TCP.

I'd be happy for feedback, the API might still change before the next release.

Like for dbus, there's a single-account and multi-account mode. In single-account mode, connections to the socket behave similar to the current jsonRpc command. In multi-account mode all local accounts are available and requests need to specify which account to use with a separate "account" param.
The socket uses Java 16 native channels and also works with graalvm native image. Socket activation is also supported via Java's System.inheritedChannel() method. For systemd socket activation the service unit file requires an additional config (StandardInput=socket), I've added an example file as well: data/signal-cli-socket.service.
For testing sockets the socat tool can be used.

Single-account example

signal-cli -u +XXX daemon --socket

$ socat UNIX-CLIENT:$XDG_RUNTIME_DIR/signal-cli/socket -
--> {"jsonrpc":"2.0","method":"send","params":{"recipient":["+YYY"],"message":"MESSAGE"},"id":4}
<-- {"jsonrpc":"2.0","result":{"timestamp":999},"id":4}

Multi-account example

signal-cli daemon --socket

$ socat UNIX-CLIENT:$XDG_RUNTIME_DIR/signal-cli/socket -
--> {"jsonrpc":"2.0","method":"send","params":{"account":"+XXX","recipient":["+YYY"],"message":"MESSAGE"},"id":4}
<-- {"jsonrpc":"2.0","result":{"timestamp":999},"id":4}

<-- {"jsonrpc":"2.0","method":"receive","params":{"envelope":{"source":"+ZZZ", ... ,"timestamp":1636538562919,"typingMessage":{"action":"STOPPED","timestamp":1636538562919}},"account":"+XXX"}}

Options

The daemon command supports multiple interfaces at the same time:
signal-cli daemon --socket --dbus --dbus-system --tcp

The listening socket address for UNIX- and TCP sockets can be configured:
signal-cli daemon --socket /tmp/some-socket --tcp 0.0.0.0:12345

Receiving messages

By default receiving messages in daemon mode works as before, i.e. on start signal-cli will immediately begin receiving messages from the server. When clients connect to the socket they will only receive messages that were received after they connected.
For use-cases where the client needs to see all messages, the daemon command has a new parameter --receive-mode with the following options:

• on-start (the default)
• on-connection: receiving messages starts when the first client connects to the socket, so that client will see all messages
• manual: messages are never received automatically, the client needs to subscribe to start receiving messages with the new subscribeReceive method. The Signal protocol expects messages to be received regularly so in this mode the client is responsible for this.

[piramiday]

that sounds awesome! may I bring your attention to my simple question at #788?
I feel like I could benefit from this manual receiving mode, but I am unsure how that would work with, e.g., sending messages:
if I send a message now, will I get its sent timestamp later on, as soon as I hit the daemon with a receive?
or will I get the message sent JSON data (with timestamp) right away in the STDOUT of the daemon process?
thanks!

[AsamK]

Sending a message will trigger a response on the same connection that sent the request, so you'll get the timestamp right away.
subscribeReceive is only relevant for incoming messages.

[cuu508]

Awesome indeed – I think JSON RPC over sockets will be easier to set up and operate than the DBus service.

A quick dumb question, as I have no experience with JSON RPC and low level socket operations – are client and server messages always terminated with a newline? Does the following sequence of events look OK?

• client connects to a socket
• client sends a newline-terminated JSON string
• server detects the newline, parses the message and goes to work
• server sends a newline-terminated JSON string as a response
• client reads up to the newline, then closes the socket

I have an experimental Python snippet for sending messages, which seems to work but I'm not sure about the newline handling, and timeout handling yet:

import json
import socket

payload = {
    "jsonrpc": "2.0",
    "method": "send",
    "params": {"recipient": ["+xxx"], "message": "Hey"},
    "id": 4,
}

payload_str = json.dumps(payload) + "\n"
payload_bytes = payload_str.encode()

with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as s:
    s.connect("/run/user/1000/signal-cli/socket")
    s.sendall(payload_bytes)

    buf = ""
    while not buf.endswith("\n"):
        buf += s.recv(1024).decode()

print("Received: ", buf)

[AsamK]

yes, every message is terminated with a newline.
When receiving a response, you should also check that its id matches your request id.
Currently all incoming requests on one connection are handled sequentially inside signal-cli, but that might change in the future.

[cuu508]

Ah, OK!
Should the client keep reading messages until it sees a message with its request id?
Are all messages sent to all connected clients?

[AsamK]

responses are only sent to the client that sent the request, so unless your client sent multiple requests it should only see a single response. But depending on how the daemon was started (--receive-mode) there can also be notification messages for incoming messages.

[cuu508]

Thanks for the clarification! With --receive-mode=on-start, if the timing is just right, could the client see an incoming message before it sees a reply to its request?

Let's say the client wants to send a single message, and get back its corresponding response. To be on the safe side, my current understanding is, it should read data, parse it into messages, and keep reading until it sees a message with the right request id.

[cuu508]

Based on the responses from @AsamK, I've revised my Python proof-of-concept code:

import json
import socket
import time
import uuid


class SendException(Exception):
    pass


def send(recipient, message):
    payload = {
        "jsonrpc": "2.0",
        "method": "send",
        "params": {"recipient": [recipient], "message": message},
        "id": str(uuid.uuid4()),
    }

    payload_bytes = (json.dumps(payload) + "\n").encode()
    for reply_bytes in _send(payload_bytes):
        try:
            reply = json.loads(reply_bytes.decode())
        except ValueError:
            continue

        if reply["id"] == payload["id"]:
            return reply


def _send(payload_bytes):
    start = time.time()
    with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as s:
        s.settimeout(10)
        try:
            s.connect("/tmp/socket")
            s.sendall(payload_bytes)
            s.shutdown(socket.SHUT_WR)  # we are done sending

            buffer = []
            while True:
                ch = s.recv(1)
                buffer.append(ch)
                if ch == b"\n":
                    yield b"".join(buffer)
                    buffer = []

                if time.time() - start > 10:
                    raise SendException("out of time budget")

        except OSError:
            raise SendException("oserror")


print(send("+123", "Hey"))

The _send function (generator) does the low-level communication:

• opens the socket
• sends the request data as bytes
• shutdowns the sending side of the connection
• reads from the socket one character at the time
• whenever it sees a newline, it yields data in its buffer, and clears the buffer
• individual sendall and recv operations have a timeout of 10 seconds
• it also keeps track of total time spent in the function, and raises an exception when it exceeds limit

The send function is a higher-level:

• it takes recipient's phone number and the message as arguments
• it picks an id value, prepares a JSON RPC request, and encodes it as bytes
• it calls _send with the request data, and then iterates over received messages
• it parses each decoded message as JSON, and checks its id
• as soon as it sees the right id, it returns

Reading one byte at the time is obviously not efficient, but it keeps the logic somewhat simple.

[cuu508]

I have another question, about the socket file permissions. On my development machine, I was running everything as my local user, and so there were no permission issues. In production environment, I want to use separate system users for signal-cli and the client application. I start signal-cli using systemd, the ExecStart line looks like so:

/path/to/signal-cli -a +xxx --config /var/lib/signal-cli --trust-new-identities always daemon --socket /tmp/signal-cli.sock`

This works, signal-cli starts up, creates the socket in /tmp/ and listens. The issue is, when the client application tries to connect to the socket, it gets "Permission denied" error, because the socket is owned by signal-cli's user.

I'm not sure what's the best way to fix this. I looked into socket activation – systemd supports it, signal-cli also seems to support it (DaemonCommand.java has a System.inheritedChannel() call). But with socket activation, signal-cli would launch on first client activity. That's not what I want – I want it to launch on machine startup and listen for incoming messages, even if there are no connected clients.

Another idea, which would require code changes, would be additional syntax for the --socket parameter (or additional new parameters) to specify socket ownership and permissions. Here's a similar ticket for a different project: coder/code-server#1466

[AsamK]

You can use a systemd socket unit for that. If you enable the service unit and the socket unit, the service will start on machine startup and use the socket provided by systemd.

There are two sample unit files in the repo:
https://github.com/AsamK/signal-cli/blob/master/data/signal-cli-socket.socket
https://github.com/AsamK/signal-cli/blob/master/data/signal-cli-socket.service

[cuu508]

Awesome, that works! Thank you, @AsamK!

[morph027]

Jumping on the JSON RPC wagon too, was fiddling with DBus before. JSON RPC is much easier to setup and to use, especially on containerized environments, where running DBus works, but feels ao much antipattern to containers ;)

Main reason i'm looking at JSON RPC interface is the availability of user mentions.

By looking at the code base (and keep in mind that i'm a java novice, like wrote the last java code ages ago while studying ;)), the JSON RPC interface looks easier to maintain as it's closer to the command line interface. Just correct me, if i'm wrong.

How likely is is that more future features will not end up in DBus interface? Just asking for a friend ;)

[AsamK]

Yeah the jsonrpc interface is very close to the command line interface and easier to keep up to date.
I have some ideas for extending the dbus interface, but it will likely always lag behind.

[morph027]

That's good to know, thank you!