Bitcoin RPC from Python

The reference bitcoin client includes a powerful API and RPC interface.

In this post I show you how to call into this from Python (which is something that turns out to be almost trivially easy to set up).

Python can work well as a kind of (overpowered) scripting language for automating complicated tasks through the bitcoin reference client, but this is also a great way to get started out if you're interested in writing more involved software that works with bitcoin transactions or the bitcoin blockchain.

Python and bitcoin

Python has good support for byte sequences and large integers and seems to be quite a good fit for bitcoin operations in general, while the reference client has a great deal of test coverage, and deals with some tricky details very robustly.

Using the RPC interface means that you can take advantage of reference client code for things like network and peer connectivity, wallet management and signing, whilst still retaining the possibility to get down and dirty with some lower level details such as raw transaction construction.

Running the reference client as an RPC server

I'm going to assume you have the bitcoin reference client (bitcoind) installed and set up and I'm not going to talk about issues such as 'bootstrapping' the client (to reduce initial block chain synchonisation times), as there should be plenty of other material available for these topics elsewhere on the web. (See this blog post, for example.)

We'll need to run bitcoind as a server, which I do with the following command:

~/git/bitcoin/src $ ./bitcoind -printtoconsole

This will start bitcoind as both a client (which connects to other nodes in the bitcoin network) and a local server (which you can connect to for RPC calls).

If you don't have an rpc username and password set up in your bitcoin configuration file (in ~/.bitcoin/bitcoin.conf by default on linux) then bitcoind will print a message about this and exit. Otherwise it will just get straight into the continuous process of connecting to other nodes in the bitcoin network and starting or maintaining synchronisation with the network blockchain, while also listening out for local RPC calls.

Note that you can also run bitcoind as a daemon (background process) but I prefer to just give the server a dedicated terminal and can then switch to this terminal if I want to see some of the current server output.

It's not all intuitive and obvious, and I recommend starting out with testnet until you're pretty sure about what you're doing, to avoid the possibility of losing real money (!):

~/git/bitcoin/src $ ./bitcoind -testnet -printtoconsole

Making RPC calls with bitcoin-cli

You can use the same bitcoind executable to make RPC calls (just by adding an RPC method at the end of command line), but this is depreciated, and you're now supposed to use bitcoin-cli for this purpose.

So, with a rpc user setup, and the bitcoin server running in a separate terminal, you should be able to do stuff like:

~/git/bitcoin/src $ ./bitcoin-cli -testnet getblockhash 150000
0000000000005e5fd51f764d230441092f1b69d1a1eeab334c5bb32412e8dc51
~/git/bitcoin/src $ ./bitcoin-cli -testnet getblock 0000000000005e5fd51f764d230441092f1b69d1a1eeab334c5bb32412e8dc51
{
    "hash" : "0000000000005e5fd51f764d230441092f1b69d1a1eeab334c5bb32412e8dc51",
    "confirmations" : 71672,
    "size" : 410,
    "height" : 150000,
    "version" : 2,
    "merkleroot" : "ea6054d3d19ea2cc460809179b93946cfb4dac4db48837eb5e3ba3d150ae3ddd",
    "tx" : [
        "235b91c7b53c100f9f836ad17b6e0e417279efda77606d490ec743817bac6ff6",
        "1b1202259a5c28f8e81dbc6d1ddd5bbf37d3af18d549495b2b9ad315fb9738a8"
    ],
    "time" : 1386098130,
    "nonce" : 2666252006,
    "bits" : "1b24a835",
    "difficulty" : 1787.78664453,
    "chainwork" : "000000000000000000000000000000000000000000000000028360d5673d92a9",
    "previousblockhash" : "00000000002380e843bcef7c5656498fd55caa64243cd60acb329ec8bb0f6699",
    "nextblockhash" : "000000000023a097d684c7c3be7a9410b85337374b8d7786cdbe38e947608e1b"
}

How this works

When run in server mode, bitcoind sets up an http server, listens out for requests, decodes method name and parameters (as JSON data) from the http request contents, and encodes the result (also as JSON) in the http response.

On the other side, bitcoin-cli looks up your rpc connection information in the bitcoin configuration file, makes an http connection to the server, encodes the method name and parameters you specify on the command line as JSON, makes a specially formed http request which includes this data, decodes the resulting JSON data from the http response, and prints this out.

Connecting from python

Thanks to the (excellent) requests library for Python we can do essentially the same thing as bitcoin-cli from within our Python scripts, very easily.

A minimal script for querying the same block details as the bitcoin-cli 'getblock' method call above is as follows:

from __future__ import print_function
import requests, json

## default port for bitcoin testnet
## (change to 8332 for 'main net'),
rpcPort = 18332
rpcUser = 'bitcoinrpc'
## not a real password
## but if you use the random password generated by bitcoind
## your password should look something like this
rpcPassword = '6rVvvA8BqqdhyR4dVcszYdgF2jR8hGkhnmPtXaSDBB4s'
serverURL = 'http://' + rpcUser + ':' + rpcPassword + '@localhost:' + str(rpcPort)

headers = {'content-type': 'application/json'}
payload = json.dumps({"method": 'getblock', "params": ["0000000000005e5fd51f764d230441092f1b69d1a1eeab334c5bb32412e8dc51"], "jsonrpc": "2.0"})
response = requests.get(serverURL, headers=headers, data=payload)
print(response.json()['result'])

If you get an import error then that means that you need to install the requests library. (I won't go into details for this, but there are plenty of other resources you can refer to for this.)

And you'll also need to change the rpc user and password to whatever you have in your bitcoin.conf,

If nothing goes wrong this should print something like the following:

{u'merkleroot': u'ea6054d3d19ea2cc460809179b93946cfb4dac4db48837eb5e3ba3d150ae3ddd',
u'nonce': 2666252006, u'previousblockhash': u'00000000002380e843bcef7c5656498fd55caa6
4243cd60acb329ec8bb0f6699', u'hash': u'0000000000005e5fd51f764d230441092f1b69d1a1eeab
334c5bb32412e8dc51', u'version': 2, u'tx': [u'235b91c7b53c100f9f836ad17b6e0e417279efd
a77606d490ec743817bac6ff6', u'1b1202259a5c28f8e81dbc6d1ddd5bbf37d3af18d549495b2b9ad31
5fb9738a8'], u'chainwork': u'000000000000000000000000000000000000000000000000028360d5
673d92a9', u'height': 150000, u'difficulty': 1787.78664453, u'nextblockhash': u'00000
0000023a097d684c7c3be7a9410b85337374b8d7786cdbe38e947608e1b', u'confirmations': 71749
, u'time': 1386098130, u'bits': u'1b24a835', u'size': 410}

Note that the data is returned in the form of Python dictionaries and lists, and can be traversed and referenced directly from your Python scripts without any further parsing or processing required.

Encapsulating in a Python class

It's nice to add in a bit of error checking and encapsulate this functionality in a python class. I'm using the following class definition for this:

from __future__ import print_function
import time, requests, json

class RPCHost(object):
    def __init__(self, url):
        self._session = requests.Session()
        self._url = url
        self._headers = {'content-type': 'application/json'}
    def call(self, rpcMethod, *params):
        payload = json.dumps({"method": rpcMethod, "params": list(params), "jsonrpc": "2.0"})
        tries = 10
        hadConnectionFailures = False
        while True:
            try:
                response = self._session.get(self._url, headers=self._headers, data=payload)
            except requests.exceptions.ConnectionError:
                tries -= 1
                if tries == 0:
                    raise Exception('Failed to connect for remote procedure call.')
                hadFailedConnections = True
                print("Couldn't connect for remote procedure call, will sleep for ten seconds and then try again ({} more tries)".format(tries))
                time.sleep(10)
            else:
                if hadConnectionFailures:
                    print('Connected for remote procedure call after retry.')
                break
        if not response.status_code in (200, 500):
            raise Exception('RPC connection failure: ' + str(response.status_code) + ' ' + response.reason)
        responseJSON = response.json()
        if 'error' in responseJSON and responseJSON['error'] != None:
            raise Exception('Error in RPC call: ' + str(responseJSON['error']))
        return responseJSON['result']

Note that this puts connections through a Session object, just in case this makes a difference to performance, since, according to the documentation for the requests library, sessions implement automatic keep-alive for connection reuse across requests.

And then there is also some error handling and automatic reconnection. (If you ever find yourself doing more complicated operations that involve non-trivial program state then having some kind of automatic reconnect can potentially save a lot of stress trying to recreate program state and complete operations manually!)

The params argument is a Python version of variable argument lists, and wraps up all the arguments passed after rpcMethod into a tuple.

You can use this class as follows:

host = RPCHost(serverURL)
hash = host.call('getblockhash', 150000)
block = host.call('getblock', hash)
## lookup details in the results
coinBase = block['tx'][0]
## more than one RPC parameter
l = host.call('listreceivedbyaddress', 0, True)

There are some wrapper libraries around that go on to add actual function stubs for each rpc method (e.g. this one). I think that a stubs layer is just something else that can get out of sync with the actual set of methods available on your RPC server, however. In terms of interface, the above works fine for me, and I like the fact that the RPCHost class is then simple enough to avoid the need to add in another third party dependency.

Altcoin RPC

There are a bunch of other cryptocurrencies that use essentially the same code base as the original bitcoin client reference code. It looks like this includes the RPC setup, and you can then use essentially the same RPC setup with at least some of these 'altcoins'.

For litecoin, for example (and the litecoind reference client), just change the RPC port to 19332 for litecoin testnet, and 9332 for litecoin main net.

Reading rpc account details from config

Copying and pasting your rpc username and password into scripts manually is a bit inconvenient, but we can improve on this by reading these values directly from the bitcoin config file.

It seems like the standard way to read config files into Python is with the ConfigParser module, but it turns out that we can't use ConfigParser directly in this case.

The problem is that bitcoin config files (loaded by bitcoind with Boost.Program_options) do not include section headers, and Python's ConfigParser module throws an error about this.

A bit of googling shows that other people have encountered this problem, in this stackoverflow question for example, but I didn't really like any of the suggested workarounds.

Config files are not so complicated, so I chose to just parse these files directly myself, using the following code:

import io

def ParseConfig(fileBuffer):
    ## fileBuffer should contain the binary contents of the config file
    ## this could be ascii, or encoded text in some ascii compatible encoding
    ## we don't care about encoding details for commented lines, but stuff outside of comments should only contain printable ascii characters
    ## returned keys are unicode strings with contents in ascii
    assert type(fileBuffer) is type(b'')
    f = io.StringIO(fileBuffer.decode('ascii', errors='ignore'), newline=None)
    result = {}
    for line in f:
        assert type(line) is type(b''.decode())
        stripped = line.strip()
        if stripped.startswith('#'):
            continue
        parts = stripped.split('=')
        assert len(parts) == 2
        parts[0] = parts[0].strip()
        parts[1] = parts[1].strip()
        result[parts[0]] = parts[1]
    return result

The config file is expected to be ascii, but we want to handle stuff like utf-8 and unicode characters in comments without falling over. And this should then work on both Python 2 and 3, returning a dictionary with unicode strings in each case.

We can use this to build our RPC url as follows (assuming linux config file location):

from os import path

with open(path.join(path.expanduser("~"), '.bitcoin', 'bitcoin.conf'), mode='rb') as f:
    configFileBuffer = f.read()
config = ParseConfig(configFileBuffer)

serverURL = 'http://' + config['rpcuser'] + ':' + config['rpcpassword'] + '@localhost:' + str(rpcPort)

(To connect to litecoind change '.bitcoin' to '.litecoin', and 'bitcoin.conf' to 'litecoin.conf'.)

Conclusion

Making RPC calls to bitcoind is easy! Next time you find yourself spending time copying and pasting data from the output of one RPC command into the input of another, consider automating this with a Python script..