Setup Hyperledger Fabric components using vendor HSM

In the previous blog post we have seen how to setup HSM service for Hyperledger Fabric , now let’s walk through the steps required to configure Fabric components to use HSM service to store private keys.

Let’s start by first installing Fabric samples, binaries and docker images. We have setup the current network with Hyperledger Fabric v2.2.

Follow this link https://hyperledger-fabric.readthedocs.io/en/release-2.2/install.html to install the samples, binaries and images for Fabric v2.2.

Step 1:

By default, the binaries which are installed do not have PKCS11 support enabled.

PKCS#11 is used as a low-level interface to perform cryptographic operations without the need for the application to directly interface a device through its driver. PKCS#11 represents cryptographic devices using a common model referred to simply as a token. An application can therefore perform cryptographic operations on any device or token, using the same independent command set.

To enable PKCS11 support we must rebuild the binaries.

Build Fabric client and server binaries with PKCS11 support enabled

1. cd fabric-samples/bin and delete the default binaries for peer, orderer, fabric-ca-client and fabric-ca-server.
2. Install prerequisites mentioned at https://github.com/hyperledger/fabric-ca/
3. Create src/github.com/hyperledger dir at $GOPATH
4. mkdir -p $GOPATH/src/github.com/hyperledger/
5. Clone fabric ca repository from https://github.com/hyperledger/fabric-ca at $GOPATH/src/github.com/hyperledger.

Check out master branch for latest release (it was Fabric v2.2 as of when this blog was written)

6. cd to fabric-ca and run below command to build fabric-ca-server binary.

make fabric-ca-server GO_TAGS=pkcs11

This would create the CA server binary under fabric-ca/bin

7. To build fabric-ca-client binary execute

make fabric-ca-client GO_TAGS=pkcs11

This would create the CA client binary under fabric-ca/bin

Build peer and orderer artifacts with pkcs11 support enabled

1. git clone master branch of https://github.com/hyperledger/fabric.git under $GOPATH/src/github.com/hyperledger
2. cd fabric and run

make peer GO_TAGS=pkcs11

This will create the peer binary under build/bin

3. Execute below command to build orderer binary

make orderer GO_TAGS=pkcs11

This would create the peer binary under build/bin

After building the binaries with PKCS11 support enabled, copy these to fabric-samples/bin.

Step 2:

Rebuilding of fabric-ca and fabric images

1. Delete all the existing images which were downloaded as part of fabric-samples installation.
2. Run command “docker rmi $(docker images)”

Verify that all the images are deleted by running command “docker images”.

If there are any dangling images, delete those.

3. cd to $GOPATH/src/github.com/hyperledger/fabric-ca/images/fabric-ca/

4. The existing docker file uses Alpine as the base image, HSM fails to initialize if Alpine is used as the base image, to resolve this issue, lets replace the base image with centos.

5. Modify the Docker file with below changes:

• Update the base image with “FROM centos:centos7”.
• Replace the instances of RUN apk add — no-cache with RUN yum -y install.
• Add following steps to install go lang

ENV VERSION 1.15
ENV FILE go$VERSION.linux-amd64.tar.gz
ENV URL https://storage.googleapis.com/golang/$FILE
ENV SHA256 2d75848ac606061efe52a8068d0e647b35ce487a15bb52272c427df485193602
ENV GOPATH /go
ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH
RUN \
set -ex && \
yum -y update && \
yum -y install git rpm-build && \
yum -y clean all && \
curl -OL $URL && \
echo “$SHA256 $FILE” | sha256sum -c — && \
tar -C /usr/local -xzf $FILE && \
rm $FILE && \
mkdir -p “$GOPATH/src” “$GOPATH/bin”

• Delete the multi stage image build step
• FROM alpine:${ALPINE_VER}
• Delete COPY — from=builder /go/bin /usr/local/bin as we have already installed go above.
• Install below packages

RUN yum -y install grep;

RUN yum -y install bash

• Copy the HSM client folders
• COPY orderer.example.com/ /etc/hyperledger/fabric/dpod/orderer.example.com/
• COPY org1.example.com/ /etc/hyperledger/fabric/dpod/org1.example.com/

After making the above changes, cd to $GOPATH/src/github.com/hyperledger/fabric-ca and run “make clean-all”

6. Run “make docker GO_TAGS=pkcs11”. This will create the fabric-ca images.

7. cd to $GOPATH/src/github.com/hyperledger/fabric/images/orderer/

8. Modify the Docker file with below changes:

• Update the base image to “FROM centos:centos7 as base”.
• Replace the instances of RUN apk add — no-cache with RUN yum -y install.
• Comment line “RUN [ ! -e /etc/nsswitch.conf ] && echo ‘hosts: files dns’ > /etc/nsswitch.conf”.
• Remove “FROM golang:${GO_VER}-alpine${ALPINE_VER} as golang” and add “FROM golang:latest as golang” after installation step of centos package like bash,git, make etc.

9. cd to $GOPATH/src/github.com/hyperledger/fabric/images/peer/

Modify the Docker file with below changes:

• Update the base image to “FROM centos:centos7 as peer-base”.
• Replace the instances of RUN apk add — no-cache with RUN yum -y install.
• Remove “FROM golang:${GO_VER}-alpine${ALPINE_VER} as golang” and add “FROM golang:latest as golang” after installation step of centos package like bash,git, make etc.
• cd to $GOPATH/src/github.com/hyperledger/fabric/ and run “make clean-all”
• Run “make docker GO_TAGS=pkcs11”

This will create the fabric images.

Step 3:

Modify the fabric-server config files to use PKCS11 interface.

1. cd to fabric-samples/test-network/organizations/fabric-ca/
2. cd to the organization folder for which HSM service is setup and modify the bccsp section.

bccsp:
 default:PKCS11
pkcs11:
 Library:/path/to/lib.so
 Pin:”XXXXXXX”
 Label:testhsm
 hash:SHA2
 security:256
 Immutable:false
filekeystore:
 keystore: msp/keystore

Step 4:

Update fabric-ca-client config file

1. Create a folder fabric-client at fabric-samples/test-network/organizations/
2. Create subfolders inside this as per the organizations in the network, example: ordererOrg, org1 etc.
3. Inside each organization folder place its fabric-ca client config file.
4. Modify the config file as per org and update the bccsp section with:

pkcs11:
 Library:/path/to/lib.so
 Pin:”XXXXXX”
 Label:testhsm
 hash:SHA2
 security:256
 Immutable:false
filekeystore:
 #Thedirectoryusedforthesoftwarefile-basedkeystore
 keystore:msp/keystore

Step 5:

Update docker-compose-test-net.yaml before starting the network components.

1. cd to “fabric-samples/test-network/docker/”
2. Update the orderer service section with below environment variables

-ORDERER_GENERAL_BCCSP_DEFAULT=PKCS11
-ORDERER_GENERAL_BCCSP_PKCS11_LABEL=testhsm
-ORDERER_GENERAL_BCCSP_PKCS11_PIN=XXXXX
-ORDERER_GENERAL_BCCSP_PKCS11_LIBRARY=/path/to/lib.so
-ChrystokiConfigurationPath=/path/to/hsm/client/folder/

Update volumes:

-/etc/hyperledger/path/to/hsm/client/folder:/etc/hyperledger/path/to/hsm/client/folder

3. Update the peer service section with below environment variables

-CORE_PEER_BCCSP_DEFAULT=PKCS11
-CORE_PEER_BCCSP_PKCS11_LABEL=testhsm
-CORE_PEER_BCCSP_PKCS11_PIN=123456789
-CORE_PEER_BCCSP_PKCS11_LIBRARY=/path/to/lib.so
-ChrystokiConfigurationPath=/path/to/hsm/client/folder/

Update volumes:

-/etc/hyperledger/path/to/hsm/client/folder:/etc/hyperledger/path/to/hsm/client/folder

4. cd into fabric-samples/config/

5. create one folder each for organization and place core.yaml for that particular organization.

Example: org1, org2 etc.

The difference here would mainly be in bccsp section which will have details as per the organization. In current case we would point to lib for respective HSM service and its details.

Once updated, volume mount the respective core.yaml as below for respective organization as below:

-../../config/orgn/core.yaml:/etc/hyperledger/fabric/core.yaml

Step 6:

Fabric-samples have some useful shell scripts which include commands used to register/enroll entities, create channel, deploy chaincode etc., these commands we would have to otherwise sequentially run manually.

Go through these shell scripts to get details of steps to be performed.

Let’s update these to work with the changes done so far to support PKCS11 interface.

1. cd to fabric-samples/test-network/organizations/fabric-ca/ and update the registerEnroll.sh script with following:

There are functions in this script as per organizations which have commands to register/enroll identities for that particular organization.

2. In createOrg1() function add below line to copy the fabric-ca client config which we have already prepared in earlier steps.

cp${PWD}/organizations/fabric-client/org1/fabric-ca-client-config.yaml${PWD}/organizations/peerOrganizations/org1.example.com/

This will copy the fabric ca client config file to respective dir where identities for this org will be created.

3. Set the HSM client variables for this particular org:

Add below lines:

cd/etc/hyperledger/fabric/dpod/org1.example.com/
source./setenv
cd/root/pooja/thales/fabric-samples/test-network/

Make sure the lines are added before any enroll or register commands take place.

4. Inside the same function before generating tls identities for peer update the env variables to use bccsp SW, this is because currently there is no support in Fabric to work with TLS identities stored in HSM.

infoln”Generatethepeer0-tlscertificates”
export FABRIC_CA_CLIENT_BCCSP_DEFAULT=SW

Once peer tls identity is generated switch back to PKCS11 by updating the env variable

export FABRIC_CA_CLIENT_BCCSP_DEFAULT=PKCS11

5. If there are any organizations using File based systems to store wallets, update env variable before beginning any operations for that particular org

export FABRIC_CA_CLIENT_BCCSP_DEFAULT=SW

6. cd to fabric-samples/test-network/scripts/ and update the createChannel.sh script with following:

• Set the environment variables before running create channel:
• Update createChannel() as below:

createChannel(){
cd/etc/hyperledger/fabric/dpod/org1.example.com/
source./setenv
cd/root/pooja/thales/fabric-samples/test-network/

• Update join channel function
• Set export FABRIC_CA_CLIENT_BCCSP_DEFAULT=PKCS11 in the beginning of the function.

7. We had updated core.yaml as per organizations, so appropriate FABRIC_CFG_PATH has to be set before running commands for that particular organization.

Example: Before running joinchannel 2, update the path to FABRIC_CFG_PATH=$PWD/../config/org2/

Then before running updateAnchorPeers 1, update the path to

FABRIC_CFG_PATH=$PWD/../config/org1/

8. cd to fabric-samples/test-network/scripts/ and update the deployCC.sh script with following:

• Before running packageChaincode 1 source environment variables for HSM service of that organization

FABRIC_CFG_PATH=$PWD/../config/org1/
cd/etc/hyperledger/fabric/dpod/org1.example.com/
source./setenv
cd/root/pooja/thales/fabric-samples/test-network/

• So appropriate FABRIC_CFG_PATH has to be set before running commands for that particular organization

Example: Before running install chaincode 2 update, set FABRIC_CFG_PATH=$PWD/../config/org2/

Before running queryInstalled 1 update, set

FABRIC_CFG_PATH=$PWD/../config/org1/

Repeat similar steps for the other organization functions.

Step 7:

Starting the network

1. cd to fabric-samples/test-network/
2. Run command “./network.sh up -ca”
3. Once the above command completes, Run “./network.sh createChannel”
4. After channel creation, deploy chaincode

“./network.sh deployCC”

Once above steps are completed successfully, we will a have running Fabric network with PKCS11 support enabled.