Setup Hyperledger Fabric components using vendor HSM
In the previous blog post we have seen how to setup HSM service for Hyperledger Fabric , now let’s walk through the steps required to configure Fabric components to use HSM service to store private keys.
Let’s start by first installing Fabric samples, binaries and docker images. We have setup the current network with Hyperledger Fabric v2.2.
Follow this link https://hyperledger-fabric.readthedocs.io/en/release-2.2/install.html to install the samples, binaries and images for Fabric v2.2.
Step 1:
By default, the binaries which are installed do not have PKCS11 support enabled.
PKCS#11 is used as a low-level interface to perform cryptographic operations without the need for the application to directly interface a device through its driver. PKCS#11 represents cryptographic devices using a common model referred to simply as a token. An application can therefore perform cryptographic operations on any device or token, using the same independent command set.
To enable PKCS11 support we must rebuild the binaries.
Build Fabric client and server binaries with PKCS11 support enabled
- cd fabric-samples/bin and delete the default binaries for peer, orderer, fabric-ca-client and fabric-ca-server.
- Install prerequisites mentioned at https://github.com/hyperledger/fabric-ca/
- Create src/github.com/hyperledger dir at $GOPATH
- mkdir -p $GOPATH/src/github.com/hyperledger/
- Clone fabric ca repository from https://github.com/hyperledger/fabric-ca at $GOPATH/src/github.com/hyperledger.
Check out master branch for latest release (it was Fabric v2.2 as of when this blog was written)
6. cd to fabric-ca and run below command to build fabric-ca-server binary.
make fabric-ca-server GO_TAGS=pkcs11
This would create the CA server binary under fabric-ca/bin
7. To build fabric-ca-client binary execute
make fabric-ca-client GO_TAGS=pkcs11
This would create the CA client binary under fabric-ca/bin
Build peer and orderer artifacts with pkcs11 support enabled
- git clone master branch of https://github.com/hyperledger/fabric.git under $GOPATH/src/github.com/hyperledger
- cd fabric and run
make peer GO_TAGS=pkcs11
This will create the peer binary under build/bin
3. Execute below command to build orderer binary
make orderer GO_TAGS=pkcs11
This would create the peer binary under build/bin
After building the binaries with PKCS11 support enabled, copy these to fabric-samples/bin.
Step 2:
Rebuilding of fabric-ca and fabric images
- Delete all the existing images which were downloaded as part of fabric-samples installation.
- Run command “docker rmi $(docker images)”
Verify that all the images are deleted by running command “docker images”.
If there are any dangling images, delete those.
3. cd to $GOPATH/src/github.com/hyperledger/fabric-ca/images/fabric-ca/
4. The existing docker file uses Alpine as the base image, HSM fails to initialize if Alpine is used as the base image, to resolve this issue, lets replace the base image with centos.
5. Modify the Docker file with below changes:
- Update the base image with “FROM centos:centos7”.
- Replace the instances of RUN apk add — no-cache with RUN yum -y install.
- Add following steps to install go lang
ENV VERSION 1.15ENV FILE go$VERSION.linux-amd64.tar.gzENV URL https://storage.googleapis.com/golang/$FILEENV SHA256 2d75848ac606061efe52a8068d0e647b35ce487a15bb52272c427df485193602ENV GOPATH /goENV PATH $GOPATH/bin:/usr/local/go/bin:$PATHRUN \set -ex && \yum -y update && \yum -y install git rpm-build && \yum -y clean all && \curl -OL $URL && \echo “$SHA256 $FILE” | sha256sum -c — && \tar -C /usr/local -xzf $FILE && \rm $FILE && \mkdir -p “$GOPATH/src” “$GOPATH/bin”
- Delete the multi stage image build step
- FROM alpine:${ALPINE_VER}
- Delete COPY — from=builder /go/bin /usr/local/bin as we have already installed go above.
- Install below packages
RUN yum -y install grep;
RUN yum -y install bash
- Copy the HSM client folders
- COPY orderer.example.com/ /etc/hyperledger/fabric/dpod/orderer.example.com/
- COPY org1.example.com/ /etc/hyperledger/fabric/dpod/org1.example.com/
After making the above changes, cd to $GOPATH/src/github.com/hyperledger/fabric-ca and run “make clean-all”
6. Run “make docker GO_TAGS=pkcs11”. This will create the fabric-ca images.
7. cd to $GOPATH/src/github.com/hyperledger/fabric/images/orderer/
8. Modify the Docker file with below changes:
- Update the base image to “FROM centos:centos7 as base”.
- Replace the instances of RUN apk add — no-cache with RUN yum -y install.
- Comment line “RUN [ ! -e /etc/nsswitch.conf ] && echo ‘hosts: files dns’ > /etc/nsswitch.conf”.
- Remove “FROM golang:${GO_VER}-alpine${ALPINE_VER} as golang” and add “FROM golang:latest as golang” after installation step of centos package like bash,git, make etc.
9. cd to $GOPATH/src/github.com/hyperledger/fabric/images/peer/
Modify the Docker file with below changes:
- Update the base image to “FROM centos:centos7 as peer-base”.
- Replace the instances of RUN apk add — no-cache with RUN yum -y install.
- Remove “FROM golang:${GO_VER}-alpine${ALPINE_VER} as golang” and add “FROM golang:latest as golang” after installation step of centos package like bash,git, make etc.
- cd to $GOPATH/src/github.com/hyperledger/fabric/ and run “make clean-all”
- Run “make docker GO_TAGS=pkcs11”
This will create the fabric images.
Step 3:
Modify the fabric-server config files to use PKCS11 interface.
- cd to fabric-samples/test-network/organizations/fabric-ca/
- cd to the organization folder for which HSM service is setup and modify the bccsp section.
bccsp: default:PKCS11pkcs11: Library:/path/to/lib.so Pin:”XXXXXXX” Label:testhsm hash:SHA2 security:256 Immutable:falsefilekeystore: keystore: msp/keystore
Step 4:
Update fabric-ca-client config file
- Create a folder fabric-client at fabric-samples/test-network/organizations/
- Create subfolders inside this as per the organizations in the network, example: ordererOrg, org1 etc.
- Inside each organization folder place its fabric-ca client config file.
- Modify the config file as per org and update the bccsp section with:
pkcs11: Library:/path/to/lib.so Pin:”XXXXXX” Label:testhsm hash:SHA2 security:256 Immutable:falsefilekeystore: #Thedirectoryusedforthesoftwarefile-basedkeystore keystore:msp/keystore
Step 5:
Update docker-compose-test-net.yaml before starting the network components.
- cd to “fabric-samples/test-network/docker/”
- Update the orderer service section with below environment variables
-ORDERER_GENERAL_BCCSP_DEFAULT=PKCS11-ORDERER_GENERAL_BCCSP_PKCS11_LABEL=testhsm-ORDERER_GENERAL_BCCSP_PKCS11_PIN=XXXXX-ORDERER_GENERAL_BCCSP_PKCS11_LIBRARY=/path/to/lib.so-ChrystokiConfigurationPath=/path/to/hsm/client/folder/
Update volumes:
-/etc/hyperledger/path/to/hsm/client/folder:/etc/hyperledger/path/to/hsm/client/folder
3. Update the peer service section with below environment variables
-CORE_PEER_BCCSP_DEFAULT=PKCS11-CORE_PEER_BCCSP_PKCS11_LABEL=testhsm-CORE_PEER_BCCSP_PKCS11_PIN=123456789-CORE_PEER_BCCSP_PKCS11_LIBRARY=/path/to/lib.so-ChrystokiConfigurationPath=/path/to/hsm/client/folder/
Update volumes:
-/etc/hyperledger/path/to/hsm/client/folder:/etc/hyperledger/path/to/hsm/client/folder
4. cd into fabric-samples/config/
5. create one folder each for organization and place core.yaml for that particular organization.
Example: org1, org2 etc.
The difference here would mainly be in bccsp section which will have details as per the organization. In current case we would point to lib for respective HSM service and its details.
Once updated, volume mount the respective core.yaml as below for respective organization as below:
-../../config/orgn/core.yaml:/etc/hyperledger/fabric/core.yaml
Step 6:
Fabric-samples have some useful shell scripts which include commands used to register/enroll entities, create channel, deploy chaincode etc., these commands we would have to otherwise sequentially run manually.
Go through these shell scripts to get details of steps to be performed.
Let’s update these to work with the changes done so far to support PKCS11 interface.
- cd to fabric-samples/test-network/organizations/fabric-ca/ and update the registerEnroll.sh script with following:
There are functions in this script as per organizations which have commands to register/enroll identities for that particular organization.
2. In createOrg1() function add below line to copy the fabric-ca client config which we have already prepared in earlier steps.
cp${PWD}/organizations/fabric-client/org1/fabric-ca-client-config.yaml${PWD}/organizations/peerOrganizations/org1.example.com/
This will copy the fabric ca client config file to respective dir where identities for this org will be created.
3. Set the HSM client variables for this particular org:
Add below lines:
cd/etc/hyperledger/fabric/dpod/org1.example.com/source./setenvcd/root/pooja/thales/fabric-samples/test-network/
Make sure the lines are added before any enroll or register commands take place.
4. Inside the same function before generating tls identities for peer update the env variables to use bccsp SW, this is because currently there is no support in Fabric to work with TLS identities stored in HSM.
infoln”Generatethepeer0-tlscertificates”export FABRIC_CA_CLIENT_BCCSP_DEFAULT=SW
Once peer tls identity is generated switch back to PKCS11 by updating the env variable
export FABRIC_CA_CLIENT_BCCSP_DEFAULT=PKCS11
5. If there are any organizations using File based systems to store wallets, update env variable before beginning any operations for that particular org
export FABRIC_CA_CLIENT_BCCSP_DEFAULT=SW
6. cd to fabric-samples/test-network/scripts/ and update the createChannel.sh script with following:
- Set the environment variables before running create channel:
- Update createChannel() as below:
createChannel(){cd/etc/hyperledger/fabric/dpod/org1.example.com/source./setenvcd/root/pooja/thales/fabric-samples/test-network/
- Update join channel function
- Set export FABRIC_CA_CLIENT_BCCSP_DEFAULT=PKCS11 in the beginning of the function.
7. We had updated core.yaml as per organizations, so appropriate FABRIC_CFG_PATH has to be set before running commands for that particular organization.
Example: Before running joinchannel 2, update the path to FABRIC_CFG_PATH=$PWD/../config/org2/
Then before running updateAnchorPeers 1, update the path to
FABRIC_CFG_PATH=$PWD/../config/org1/
8. cd to fabric-samples/test-network/scripts/ and update the deployCC.sh script with following:
- Before running packageChaincode 1 source environment variables for HSM service of that organization
FABRIC_CFG_PATH=$PWD/../config/org1/cd/etc/hyperledger/fabric/dpod/org1.example.com/source./setenvcd/root/pooja/thales/fabric-samples/test-network/
- So appropriate FABRIC_CFG_PATH has to be set before running commands for that particular organization
Example: Before running install chaincode 2 update, set FABRIC_CFG_PATH=$PWD/../config/org2/
Before running queryInstalled 1 update, set
FABRIC_CFG_PATH=$PWD/../config/org1/
Repeat similar steps for the other organization functions.
Step 7:
Starting the network
- cd to fabric-samples/test-network/
- Run command “./network.sh up -ca”
- Once the above command completes, Run “./network.sh createChannel”
- After channel creation, deploy chaincode
“./network.sh deployCC”
Once above steps are completed successfully, we will a have running Fabric network with PKCS11 support enabled.