the confusing current status of various node.js WebCrypto API providers

weiwu.zhang · June 28, 2020, 1:27pm

@hboon mentioned that it would be great to have a TokenScript commandline utility where all typical features including signing and verification are supported. Apparently the current tools (mostly Makefile based) are not versatile enough and today's developers are more used to aws style (like aws {subcommand} …) than Makefile.

The effort to replace Makefile with a JavaScript utility was first attempted 1st April (according to the dates on my computer files) but we hit a bug where half of the TokenScripts won't validate. @jot2re helped to drill down the problem to certain canonicalisation by the WebCrypto API, and nice folks in PeculiarVentures was able to test that the problem is limited to node-js based WebCrypto APIs, and works fine in the browser-provided WebCrypto API.

But we surely can't use a browser-based utility to replace the command line utility, can we?

Before falling back to writing the utility in Java, I set out to enumerate all the WebCrypto API providers in the node.js world and found these:

webcrypto-ossl, an OpenSSL polyfill by PeculiarVentures (the one I used in 1st April). It fails for half of the TokenScript signed by aw.app. This is the only library which has the courtesy to not call itself just by the name 'WebCrypto'.
Trust's webcrypto written by anvilresearch, no longer maintained. From here onward, every library calls itself WebCrypto.
NodeJS's WebCrypto - confusingly, this is not the WebCrypto you would get by doing npm install webcrypto.
Diafygi's WebCrypto - this is the one you will get by doing npm install webcrypto but confusingly, it does not provide WebCrypto APIs defined by W3, rather instead it looks like an interface (or dependency) for those who do.

The test code for the first 3 libraries are here (the last is not testable as it doesn't seem to provide WebCrypto APIs).

1.test-xmldsigjs-with-webcrypto-ossl.js (962 Bytes)
2.test-xmldsigjs-with-turst-webcrypto.js (918 Bytes)
3.test-xmldsigjs-with-nodejs-webcrypto.js (1.0 KB)

Result:

$ node 1.test-xmldsigjs-with-webcrypto-ossl.js ../TokenScript-Repo/aw.app/2020/06/*
[FAILED] ../TokenScript-Repo/aw.app/2020/06/COMP.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/daiPool.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/EntryToken.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/fifa.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/GFO-TokenScript.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/GFT-TokenScript.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/karma.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/magazine.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/NEST.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/saiPool.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/sethPool.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/UEFA.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/unicon.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/usdcPool.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/wbtcPool.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/wethPool.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/aDAI.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/cBAT.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/cDAI.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/cETH.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/cREP.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/cSAI.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/cUSDC.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/cWBTC.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/cZRX.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/DAI.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/ENS.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/mDAI.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/mETH.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/mUSDC.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/pDAI.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/pETH.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/pUSDT.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/SAI.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/USDC.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/USDx.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/WETH.tsml
$ node 2.test-xmldsigjs-with-turst-webcrypto.js ../TokenScript-Repo/aw.app/2020/06/*
[FAILED] ../TokenScript-Repo/aw.app/2020/06/COMP.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/daiPool.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/EntryToken.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/fifa.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/GFO-TokenScript.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/GFT-TokenScript.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/karma.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/magazine.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/NEST.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/saiPool.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/sethPool.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/UEFA.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/unicon.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/usdcPool.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/wbtcPool.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/wethPool.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/aDAI.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/cBAT.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/cDAI.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/cETH.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/cREP.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/cSAI.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/cUSDC.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/cWBTC.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/cZRX.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/DAI.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/ENS.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/mDAI.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/mETH.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/mUSDC.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/pDAI.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/pETH.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/pUSDT.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/SAI.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/USDC.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/USDx.tsml
[  OK  ] ../TokenScript-Repo/aw.app/2020/06/WETH.tsml
$ node 3.test-xmldsigjs-with-nodejs-webcrypto.js ../TokenScript-Repo/aw.app/2020/06/*
[FAILED] ../TokenScript-Repo/aw.app/2020/06/COMP.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/daiPool.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/EntryToken.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/fifa.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/GFO-TokenScript.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/GFT-TokenScript.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/karma.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/magazine.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/NEST.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/saiPool.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/sethPool.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/UEFA.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/unicon.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/usdcPool.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/wbtcPool.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/wethPool.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/aDAI.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/cBAT.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/cDAI.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/cETH.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/cREP.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/cSAI.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/cUSDC.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/cWBTC.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/cZRX.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/DAI.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/ENS.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/mDAI.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/mETH.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/mUSDC.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/pDAI.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/pETH.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/pUSDT.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/SAI.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/USDC.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/USDx.tsml
[FAILED] ../TokenScript-Repo/aw.app/2020/06/WETH.tsml

Long story short, 1 and 2 performed equally, meaning that the problem is one level lower (the common libraries they depend on). 3 didn't work at all because it used unimplemented .bind() function for callbbacks as specified by W3.

Conclusion:

There is no working javascript WebCrypto APIs we can use. Have to go down deeper to find the cause why they failed.

weiwu.zhang · April 7, 2021, 7:46am

It seems something has changed since I wrote that post in Jun 2020. nodejs added offical support for webcrypto: Implementing the Web Cryptography API for Node.js Core - NearForm

Topic		Replies	Views
Signing tool: Java or JavaScript Implementations	1	416	September 10, 2020
Weekly design meeting #29 Tokenscript Design	0	402	April 27, 2020
TokenScript meeting #39 Tokenscript Design	3	697	July 3, 2020
Tokenscript security considerations Tokenscript Design	0	719	March 11, 2019
Can we use a similar solution like tor.us for key backup/recovery Develop with TokenScript	0	467	March 5, 2020

the confusing current status of various node.js WebCrypto API providers

Related Topics