BFT in a Distributed Compute and/or Storage System - A Simple Introduction

Distributed Systems - physical

Distributed systems is breaking up a large computer/storage (the slash means and/or) into smaller ones. The smaller compute/storage is called a node. Distributed systems can make an entire larger system safer (no single point of failure), more powerful (break up a big compute task into smaller ones), scalable.  Apache Hadoop, where storage is spread across a wide network of storage nodes, is an example of a distributed (storage and processing) system.

But these nodes need to be in sync (aka consistent). And if the nodeA and nodeB have differences in opinion on what the final state of the system is, who is right?  In a space capsule example - two computers are computing the exact time to fire thrusters. ComputerA says 10:01, ComputerB says 10:04. Who is right?

This is why distributed systems have a consensus, or voting, mechanism.  And BFT is the science behind making voting mechanism. BFT is a voting mechanism that allows for multiple nodes to reach a consensus.  BFT can help to make a distributed system 1) failure tolerant - so that if a node on the distributed system fails, the other working nodes can take over quickly 2) malice tolerant - so that if a node on the distributed system is injecting wrong information, it will be voted out by other good nodes.

Decentralization - control 

Decentralization is a CONTROL concept, not a physical concept. So distributed is a physical concept. In a network that is responsible for control, decisions needs to be made and agreed upon. What is the final state of Joe's bank account? Should the reverse thrusters be fired at 10:01 or 10:03? BFT is critical in a decentralized network - to ensure that far away nodes can make decisions.  BFT needs to handle both failure and malice. Malice can come in many forms : from sending the wrong data (Joe owes me $1,000 instead of $1) to jamming the network. Examples of the latter include email DDoS - remedied by asking the sending to do some work first before the sender's email is read by the receiver. 

BFT in Blockchain

Blockchain uses several different ways to keep its distributed nodes in sync (consistent). BFT is one. Another way is to use Proof of Work (Nakamoto Consensus) used by Bitcoin. Rather than using BFT, Bitcoin wants to see proof that the node did work before its vote is accepted. The work that a node needs to perform is very compute intensive, using up much energy and is frowned upon as a way to solve a consensus. Proof of Stake is slated to replace PoW.

Blockchain Refresh

- append only, making it immutable

- time stamped

- consensus 

- hashing : compress transactions, SHA256

- digital signature : private key (held secretly in a wallet) and public key (forms public address)

- chain : a block of data contains multiple transaction hash; and the following block has a hash of the current block so that changes to the current block will impact all blocks 

- store efficiently : in Merkle Tree (hash)

Conclusion

BFT is the science and art of making nodes in a decentralized environment stay sync (consistent) by providing a consensus (voting) mechanism that works robustly (failures are handled) and safely (malice proof). BFT is used in blockchain.

Hashcash : Before you get to vote/say or email - you need to work or pay for it - to prevent noise/spam

Hashcash : a proof-of-work system

limit email spam and hence reduce denial-of-service attacks

 requiring the sender of an email to perform a small amount of computational work 

before their message can be sent.

--- Email DDoS ---

Email is an essential tool in both business and personal life. Email remains a major channel for business communication, accounting for a significant portion of interactions. While exact percentages can vary by industry and company size, studies suggest that email often makes up around 60-70% of all business communication. Other channels, like instant messaging (e.g., Slack, Microsoft Teams), video calls (e.g., Zoom), and project management tools (e.g., Asana, Trello), are also growing in use, especially for quick chats and collaboration. However, email is still favored for formal communications, documentation, and messages that need to be referenced later.

 

Email inbox can be flooded to the point it is full and cannot receive any more new email. So an email inbox that is full can disrupt business (orders are not received), disrupt personal lives (invitation to a birthday party is not received).

Evil people can easily disrupt a business or person by flooding their email inbox. So from a single computer, the evil person can automate sending thousands of emails an hour, with the goal of flooding the email inbox of the business or person. This is essentially free – there is no cost to create and send email. In the cybersecurity world, this is considered a Denial of Service (DoS) attack. If the evil person uses multiple computers to simultaneously send out thousands of emails per hour, this type of attack is call   ed Distributed of Denial of Service (DDoS).

There are several ways to reduce email inbox DDoS attack. 1) A firewall into the receipient’s email can be used to block a flood of email that is coming from the same email address. 2) Another method is to use a novel scheme created 30 years ago before firewalls became popular. It is called “Hashcash”. Hashcash, proposed by Adam Black in 1997, is a method that requires the sender to do some work before the sender can send an email.  Here is how it works (I think!)

• Sender sends email to recipient 
• Before sender accept recipient’s email, recipient sends a number (say 10) to the sender
• The sender takes the number (10), creates a random number (called nonce), computes the hash digest of the noonce, checks to see if the first 10 digits of the hash digest are 0s... if not, randomly create another nonce... until the first 10 digits of the hash digest are 0s
• guess NONCE1 -> HASH -> HASH_DIGEST_1; compare 1st 10 digits of HASH_DIGEST_1 to recipients request of 10 zeros; results is no
• guess NONCE2 -> HASH -> HASH_DIGEST_2; compare 1st 10 digits of HASH_DIGEST_2 to recipients request of 10 zeros; results is no
• This will take a while… and CPU resources … for the sender to compute 
• Sender sends the output (hash of data ABCD) to the recipient
• The recipient can easily verify that the hash is correct, and that the sender did do work 
• The recipients accepts sender’s email 

So asking the sender to do work before the recipient will accept the email should and will reduce spam.

--- Bitcoin Proof of Work ---

--- Amazon Retail E-commerce ---

Blockchain for capital markets - finally

Blockchain has now arrived at the commercialization stage after years of exploration and experimentation. It is one of  the de-facto technologies that can transform the traditionally  risk average and highly regulated capital markets. For example,  the World Economic Forum published a report titled “Digital Assets, Distributed Ledger Technology, and the Future of Capital Markets” (1) that in great details outlines how blockchain DLT can positively impact capital market usecases in the equity markets, debit markets, derivates, and more.

Today’s enterprises' and financial services' operating models are highly centralized, where transactions between two parties (say buyer and seller of an asset, or producer and consumer of a service) need to pass through several intermediaries (middleman). Examples of these intermediaries include stock brokers, banks, clearing houses, exchanges, which add costs, delays, and even mistakes into transactions. Decentralization will remove the intermediaries, which in turn will lower costs, reduce errors, and speed up transaction, as well as offering new services not possible with traditional centralized services. In addition, decentralization will remove single points of failure, potential censorship/disruption/corruption by a single (nefarious) actor, and creating trust where all transactions are recorded, verified, and replicated. Blockchain can decentralized enterprise operating model by providing a single source of trusted truth, augmented with the power of smart contract. 

(1) https://www.weforum.org/reports/338ea2d9-654a-45a0-a09b-7ca3cd8f248c

 

Decentralization vs Distributed

 Decentralization vs Distributed

Decentralization is a concept that pertains to HOW DECISIONS ARE MADE.  Examples are "decentralized decision making". So if a decision needs to be made, instead of asking a central authority (which might be SLOW, CORRUPT-ABLE, EXPENSIVE), you can spread the decision out for a CONSENSUS (via voting, etc).  So to make decisions in a decentralized manner, we need a way for peers to reach a consensus - perhaps via a voting mechanism. The Apollo-2343 had multiple parallel computers that needed to agree on the final state of the system, so a consensus machism was needed. Voting can be an IMPLEMENTATION of a consensys. 

Benefits of decentralization over centralization : Today’s enterprise operating model are highly centralized, where transactions between two parties (say buyer and seller of an asset, or producer and consumer of a service) need to pass through several intermediaries (middleman). Examples of these intermediaries include stock brokers, banks, clearing houses, exchanges, which add costs, delays, and even mistakes into transactions. Decentralization will remove the intermediaries, which in turn will lower costs, reduce errors, and speed up transaction, as well as offering new services not possible with traditional centralized services. In addition, decentralization will remove single points of failure, potential censorship by a single (nefarious) actor, and creating trust where all transactions are recorded, verified, and replicated. Blockchain can decentralized enterprise operating model by providing a single source of trusted truth, augmented with the power of smart contract. 

Distributed is a concept that pertains to COMPUTE. If you have a workload (say processing a movie that requires heavy compute), you can run it on your laptop, which is not distributed. Or you can run that same workload on a distributed compute environment, where multiple computers work in parallel to speed up the workload. The workload needs to be friendly/amiable to being broken up into smaller parallel workloads. As such, that is why the general type of technology that Blockchain derives from is called "Distributed Ledger Technology", or DLT.  

"Consensus",  usually brought up in the context of distributed systems, discusses how distributed systems (compute, storage) agree on the state of compute or storage.  

• Malice/bad tolerant (BFT) : 
• Paxos 
• PBFT (derived from Paxos)
• asynchronous
• leader (view), rotates
• need 3f+1 nodes to deal with f failure
• Crash tolerant : 
• Raft, an alternative to Paxos

New found love for IoT

As an EE who has worked on multiple facets of microcontrollers for longer than I care to admit, I thought that IoT (Internet-of-Things) will finally make microcontrollers sexy. Historically, microcontrollers have been relegated to performing the dirty work of task oriented compute - running inside washing machines, factories, and even spacecrafts. Hundreds of millions of units sold per annum easily - but no one knows or cares. So I switched to EDA (electronic design automation), to be closer to SW and away from HW.

My interest in IoT revived when I joined Oracle to promote their supply chain cloud emerging technologies group - including IoT. Looking at IoT from the enterprise angle made it much more attractive - almost sexy. Combining OT (operational technology) data with IT (information technology - such as ERP) suddenly propelled IoT into the minds of business leaders (in my case, supply chain business leaders) instead of  gEEky embedded controller guys. At Oracle, IoT had its own apps : Asset Monitoring, Fleet Management, Worker Safety, and Manufacturing. Doesn't take much imagination to know why IoT will help. The magic that Oracle brought to IoT is that we spent time to make the IoT apps attractive : status, control, analytics and AI are all natively shown. It was almost sexy - but not enough to see a strong demand. Perhaps we were talking to the wrong audience. Perhaps we were too expensive (Oracle treats IoT as a SaaS - so a subscription is needed). Perhaps the brand did not resonate. So again, I left IoT.

But what is rekindling my interest in IoT again? Home automation. Specifically a garage door app. 

With Chamberlain "myQ" garage door app kit, IoT can digitize the old tired "analog" world of a garage door, and can directly provide consumers with:

1) peace of mind : How many times have you wondered (in horror) if your garage door is opened or closed? Even if you are just upstairs to the garage, that trek downstairs to check on the garage seems so unnecessary.  

2) security : You want control over the garage door - when to close it, when to open it. From anywhere. You can even set a rule to close the garage automatically at 11PM.

3) insights : Right now, the data this provides is fun. But in the future, as you give others access to your garage (tenants, Airbnb, Amazon delivery, kids), insights will be more than just fun.

4) convenience: Voice control. Turning on the hallway lamp when the garage door opens. Lots of things can be triggered just by simply opening or closing a garage door.

Is IoT now sexy? Maybe not. But definitely not in buried in a washing machine. 

Why Blockchain (a Distributed Database with an Execution Engine) Is Good For Financial Market Infrastructure

Financial market infrastructures (FMI) is the technology that enable trillions of dollars of capital to move globally on a nightly basis. Some of the infrastructure still runs on systems designed in the 1970s, including mainframe and COBOL. Blockchain is now starting to be deployed as a solution to modernize these systems, after years of trials and proof-of-concepts.  A key reason for blockchain adoption is that it  provides a single, decentralized source of truth, data verifiability and programmable assets to an ecosystem of market participants that currently maintain siloed ledgers.  In this session, we will share how financial services have deployed blockchain in order to enable new services, achieve greater efficiency,  and improve client service delivery.

Financial services has become one key application category for blockchain. Specifically, the benefits are seen in the backend machinery that powers financial services. The backend machinery for financial services is also called financial market infrastructure. Depending on the sub-category, the backend machinery can be paper based (no digitization), or multiple digitized systems running in separate siloes, with its own database.

 

Blockchain technology offers several compelling advantages that make it a potentially transformative force for financial market infrastructure (FMI):

1. Enhanced Efficiency and Speed:

• Streamlined Processes: Blockchain can automate many manual, paper-based processes involved in traditional FMI, such as clearing, settlement, and reconciliation. Smart contracts can automatically execute transactions when pre-defined conditions are met, reducing delays and the need for intermediaries.
• Faster Settlement: Traditional settlement processes can take days. Blockchain enables near real-time or T+1 settlement, freeing up capital faster and reducing counterparty risk.
• Reduced Reconciliation: The distributed and shared nature of the blockchain ledger means all participants have a synchronized view of transactions, significantly reducing the need for costly and time-consuming reconciliation processes between multiple parties.

2. Cost Reduction:

• Lower Operational Costs: Automation, reduced manual work, and decreased reconciliation efforts translate to lower operational costs for FMI participants.
• Reduced Transaction Costs: Faster and more efficient processes can lead to lower transaction fees for end-users.
• Disintermediation: By enabling direct peer-to-peer transactions and automating processes, blockchain can reduce the reliance on numerous intermediaries like clearinghouses, custodians, and correspondent banks, each of which charges fees.

3. Increased Trust Security and Transparency:

• Immutable Records: Once a transaction is recorded on the blockchain, it is extremely difficult to alter or delete, creating an auditable and tamper-proof record of all activities. This enhances trust and reduces the potential for fraud.
• Greater Transparency: All authorized network participants can view the same transaction data, increasing transparency and reducing information asymmetry. This can improve market oversight and reduce the potential for manipulation.
• Improved Traceability: Blockchain creates a clear audit trail, documenting the provenance of assets and transactions at every step. This is particularly valuable for regulatory compliance and identifying the source of issues.

4. Reduced Counterparty Risk:

• Atomic Settlements: Smart contracts can facilitate "atomic" settlements, where the transfer of one asset automatically triggers the transfer of a related asset, eliminating the risk that one party fulfills their obligation while the other does not.
• Real-time Visibility: The shared ledger provides a real-time view of positions and transactions, allowing participants to better manage their exposure to counterparties.

5. Innovation and New Services:

• Tokenization of Assets: Blockchain enables the representation of traditional financial assets (like stocks, bonds, and real estate) as digital tokens. This can fractionalize ownership, increase liquidity, and make assets more accessible to a wider range of investors.
  
• Programmable Assets: Smart contracts can embed rules and conditions directly into digital assets, automating corporate actions like dividend payments or interest distribution.¹⁹
• New Marketplaces: Tokenization of new digital assets via Non Fungible Token (NFT) marketplaces such as OpenSea.
• Improved Digital Identity Management: Blockchain can provide a more secure and efficient way to manage digital identities for KYC/AML (Know Your Customer/Anti-Money Laundering) processes, reducing duplication and improving compliance.²⁰

However, it's important to note that the widespread adoption of blockchain in FMI also faces challenges:

• Scalability: Some blockchain networks, such as Bitcoin and Ethereum, may face limitations in the number of transactions they can process per second due to Proof of Work (PoW) consensus. Other forms of consensus (Proof of Stake) are being explored and adopted to lower energy consumption and increase transaction throughput.
  
• Interoperability: Different blockchain platforms may not be easily compatible, potentially creating silos.
• Regulatory Uncertainty: The legal and regulatory frameworks for blockchain in finance are still evolving, such as the SEC and how politics can sway it.
  
• Legacy Infrastructure Integration: Integrating blockchain with existing, complex legacy systems can be challenging.
• Data Privacy Concerns: While blockchain offers transparency, ensuring the privacy of sensitive data requires careful design and implementation.
• Governance: Establishing clear governance structures for blockchain networks involving multiple participants is crucial.

Despite these challenges, the potential benefits of blockchain for enhancing the efficiency, security, transparency, and resilience of financial market infrastructure are significant and continue to drive innovation and exploration in the financial industry.²⁴

 

Conclusion : Blockchain is transforming the infrastructure for financial market because it provides several key ingredients : a single TRUSTED source of truth, the intelligence of smart contract, and auditability.  That’s the power of blockchain -  providing multiple  transformative value in one technology adoption.   The repurchase (repo) market  is only the start. Any intra and inter enterprise transactions systems that passes through multiple siloes before completing a transaction process is now ripe to  be done in a fraction of the time – thanks to blockchain. 

"ls" on macOS : Simple commands to make looking for files easier

Looking for files (by size, date, type) can be made easier if you know a few simple Unix commands - as opposed to using a GUI file manager such as "Finder" on macOS. Here are some commands that I use on my Macbook Pro.

machine type : 

   Darwin Kernel Version 20.4.0: Thu Apr 22 21:46:47 PDT 2021; 

   root:xnu-7195.101.2~1/RELEASE_X86_64 x86_64

   macOS 11.3.1 (BigSur)

-ltr : newest on bottom (reverse time order)

       $ls -ltr

       drwxr-xr-x+   4 albchi  staff        128 Jul  5  2019 Public

       drwxr-xr-x  186 albchi  staff       5952 Jun  5 05:05 Txt # <- newest on bottom

-lt : oldest on bottom

      $ls -ltr

      drwxr-xr-x  186 albchi  staff       5952 Jun  5 05:05 Txt

      drwxr-xr-x+   4 albchi  staff        128 Jul  5  2019 Public # <- oldest on bottom

-l : alphabetical z on bottom, long format

     $ls -l

     drwxr-xr-x    9 albchi  staff        288 Aug 24  2020 0DevVersionMacos

     -rw-r--r--@   1 albchi  staff       3033 Feb 27  2020 xindex.html # <- alphabetically, z at bottom

-lr : alphabetical 0 on bottom, long format

      $ls -lr

      -rw-r--r--@   1 albchi  staff       3033 Feb 27  2020 xindex.html

      drwxr-xr-x    9 albchi  staff        288 Aug 24  2020 0DevVersionMacos # <- alphabetically, 0 at bottom

-p or -F : directories with slashes

     $  ls -F or ls -p

     Attic/ migrations/ truffle-config.js

     contracts/ test/

-d -- */: directory name only

     $ls -d -- */   # JekyllGithubNetlify/  JekyllRubyBuildWebSites/

-Slr : biggest on bottom

       $ls -Slr

       -rw-r--r--@   1 albchi  staff                0 May 13  2020 index_mini.html

       -rw-r--r--      1 albchi  staff  167905792 Oct 30  2020 Txt.tar # <- largest file on bottom