AWS Cloud Storage (S3, EBS, EFS) Explained in One Picture - Hopefully

AWS offers multiple cloud services for storage : 1. Simple Storage Service (S3) 2. Elastic Block Storage (EBS) 3). Elastic File Service (EFS) and more. If you have been reading various documents and comparisons and still confused, I wrote this to hopefully  clear things up a tad.

• S3 (Simple Storage Service - object):  if you have big files, want to share it EXTERNAL TO AWS with many others in the outside world, can use HTTP to access, does NOT need to be attached to an EC2 instance
  
• EBS (Elastic Block Storage - block): fast storage for your EC2 compute to use, usually SSD and NVMe based
• EFS (Elastic File Service - file): share data  (pictures, document, ...) using files INTERNALLY WITHIN AWS, such as between EC2 instances
  

 

A Database Overview - from SQL to NoSQL

 Introduction

I am writing this quick introduction to databases, based on my studies (MBA level database class, focused on database theory to implementation using Microsoft SQL Server) and professional experience (using MySQL, SQLite, MariaDB, MongoDB, Neo4J, DLT).

Relational “SQL” Databases

Conjured up in the 1970, relational database was created to address the problem of “how can businesses store large amounts of data, then easily retrieve it”.  The computer system that served to store the data was called Relational Database Management System (RDBMS). The language that allowed businesses to create and retrieve data from the RDBMS was called Structured Query Language (SQL). But raw data cannot just be simply thrown into the RDBMS database. The data first had to be analyzed, and only then can the database be designed. The database design phase involves creation of an Entity Relation Diagram (ERD) to model the business needs of the database, followed by normalization of the data to conform to the “normal form”  to reduce data redundancy.  The RDBMS had to support core basic functions such as Create, Read, Update, Delete (CRUD) methods. And for the RDBMS to ensure that data was stored accurately, it behaved in accordance with Atomic Consistency Isolation Durability (ACID). RDBMS implemented the concept of On Line Transactional Processing (OLTP) to support businesses that broadly adopted RDBMS to handle daily critical business transactions. ACID properties of a RDBMS gave banks the confidence to store highly critical data on RDBMS. And once the transactional data is stored in RDBMS, an Extraction Transform Load (ETL) process loaded the OLTP data into another database that is more geared for analysis. This other database is usually called a data warehouse or data lake. This second database enabled On Line Analytical Processing (OLAP), which was ideal for analysis and reporting. 

Big Data

Relational databases serve the business world with its ability to store transactions with its OLTP features, then provide analytics and insights with its OLAP abilities. But with the advancement of new technologies (namely broadband internet, 5G, and powerful mobile devices to use up that bandwidth), real world data no longer fit into tidy RDBMS schema. What were the major characteristics of this new data? In some studies, the requirements of this new data, the 5V framework was created needed to handle Velocity, Variety, Veracity, Volume, and Value.  Velocity = real time data. Variety = text, pictures, videos, geo-data. Veracity = Volume = social media easily creating and consuming multimedia, multiplied by millions of users. Value = real time data, geo-tagged, enriched by media can provide infinitely more insights than structured data.  

NoSQL Databases

With Big Data, a new class of database was needed - to handle the new, unpredictable format. So with NoSQL databases, a database schema no longer needs to be PREDEFINED. In addition to this, NoSQL databases are usually distributed globally to be close to where the data is used in real time. While sounding reasonable, this poses the problem of how to keep data that is spread around the globe consistent.  Whereas traditional RDBMS was focused on accuracy of data, enforced via the ACID principles, modern data requirements are different. Modern data requirements favored availability of any-even-old data, possibly at the expense of accuracy. This requirement of big data is known as Basically Available Soft state Eventually consistent (BASE), the antithesis of ACID. The Consistency-Availability-Partition Resistance (CAP) Theorem, which is a framework for trade offs required between accuracy (which ACID affords) and availability (BASE offered by NoSQL databases) helped to put a framework around which database type to use. In additional to the flexibility of NoSQL databases, NoSQL databases can be scaled horizontally (as opposed to scaling vertically in a RDBMS system). The benefit of horizontal scaling is that to add extra new compute capacity, just add new servers - instead of stopping a current server and add extra CPU/RAM/storage used in vertical scaling.

How Databases Fit In Cloud Era:

RDBMS were created during the client-server era, which means that the database ran on a server. To access the database, the client needs to connect to a server that usually was running in a room somewhere in the office. For example, to run Oracle Database 12.2, the minimum server configuration is listed here at Oracle. This is hardware that you will need to procure and install somewhere in your office. You will also need an IT person who can install, configure, manage, patch, and upgrade the database software.  Fast forward 30 years to the world of cloud computing, where you can have Oracle now manage both the hardware and software - called “Oracle Database Standard Service”, where the hardware is Oracle Cloud Infrastructure (OCI). Let’s look at another more modern example like MongoDB. MongoDB is a “new SQL” database, gives you the option to 1) MongoDB Enterprise Advanced - run locally using your own hardware, like the olden client-server days or  2) MongDB Atlas aka “MongoDB Enterprise Advanced in the Cloud”.  If you are an Amazon Web Services (AWS) user, you can visit the AWS Marketplace to subscribe to MongoDB Atlas (Pay As You Go) or use an AWS clone of MongoDB called Amazon DocumentDB. 

Current “Top” Players in the Database Marketplace (1):

SQL:  

• Oracle, Oracle MySQL, Microsoft SQL Server

New SQL:

• PostGRES, MongoDB

NoSQL:

• Document : MongDB ( #5), CouchDB (#40), 

• Key-Value : Redis (#6), Memcached (#33), etcd (#46), Aerospike (#70), RocksDB (#89), LevelDB (#107)

• Wide Column: Cassandra (#11), HBase (#26)

• Graph : Neo4j (#19)

• Search engine : Elasticsearch (#7), Splunk (#13)

Conclusion

Databases have evolved from plan-ahead SQL RDBMS systems that run on a server located in your office - to ad-hoc NoSQL databases that run in the cloud.  Although RDBMS was invented 50 years ago, most of the world’s data still resides on them and new applications will still be designed with RDBMS. I hope that I have given a broad view here to provide a map to the world of databases.

Footnote:

1. https://db-engines.com/en/ranking

Security According to Gartner MQs

Trying to figure out how many MQ Gartner has just for security? Me too. Here is my attempt to capture all of them in one chart.  So far, I have captured 1) Secure Service Edge 2) Application Security Testing 3) Security Information & Event Management 4) Enterprise Network Firewalls 5) Web Application and API Protection - previously Web Application Firewalls 6) Endpoint Security Platform 7) Security Awareness Computer-Based Training. Not sure if they will have a MQ for Security Orchestration, Automation and Response (SOAR), Next Generation Firewall (NGFW), etc.

Security Incident and Event Management (SIEM) Overview

Security Incident and Event Management (SIEM) Overview

Starting from the top of a cybersecurity organization, a Security Operations Center (SOC) is in charge of proactively keeping company assets safe, such data stored on a company's network. One such tool for the SOC to do their job is to use a Security Incident and Event Management (SIEM) tool. SIEM collects data from a 1) wide variety of sources in 2) wide variety of formats - to enable a real time view and broad analysis of past data. SIEM is more about collecting, indexing, tagging, contextual mining than it is about security per se. That is why Splunk (a logging collection and analytics platform) is a leader in the Gartner SIEM MQ. 

Sources of data to SIEM: 

• Endpoint (EP) Security : As the agent that protects laptops, tablets, and phones an the edge of enterprise control, the End Point Agents needs to Prevent Threat, Protect Data, Secure the Network, Ensure Compliance, Respond to Incidences.
• Firewall (FW) Security : A firewall acts as a barrier between a internal network and external networks, by controlling incoming and outgoing traffic based on predetermined security rules. Packet filtering, stateful packet inspection are some of the techniques used.  
  
• Intrusion Detection System (IDS) & Intrusion Prevention System (IPS) : An IDS monitors network traffic for suspicious activity and potential threats, alerting administrators when such activities are detected. An IPS takes IDS a step further and ACTS to prevent or mitigate further damage.
  

Types of data sent or accessible to SIEM:

• events : a simple record of what happened in the network - can be just a time stamp, source identifiers, and a code
  
• messages : a bigger, verbose record of what happened in the network - usually containing contextual information to help with forensic investigation
  
• logs : a record of systems happenings, such as events and messages
  

Types of actions from SIEM:

• alert
• analysis
• reporting : start with alerts, and after an analsis
  

At the bottom, I have listed Security Orchestration, Automation and Response (SOAR) as a receiver of SIEM alerts, and SOAR can provide automated responses to the alerts. SIEM alerts can also be fed to User Entity and Behavioral Analysis (UEBA) to leverage the power of AI/ML to find threats.

Security Landscape

The term "Security", especially in the Cybersecurity context, is broad and confusing. Terms like "NGFW" and "UEBA" are thrown around - sometimes without context. So here I lay out the security landscape, but with focus on Network Security - the old guard and hence has the highest market share of roughly 40%. In this drawing, I will try to show Network Security, and how it is peers to other important security measures such as Application Security and Endpoint Security, and where "NGFW" and "UEBA" fit.

Note that some cybersecurity practioners might categorize IDS and IPS as part of a firewall - and that is completely ok. Firewalls are primarily about controlling access to networks. IDS focuses on detecting and alerting on suspicious activities. IPS combines detection with proactive blocking capabilities to prevent attacks. But for this blog, I purposely broke them apart because terms like "Gen 2 Firewall" and Gen 3 aka NG Firewall" and thrown around, and I wanted this diagram to make that easier to find. 

A quick walk through of compute : mainframes to containers to smart contract

 

This is my stab at the history of computer usage in business enterprises.

Time Shared Mainframes & Terminals

Early machines were big, heavy, and only for enterprise usage. Long forgotten names like Wang, Sperry, DEC roamed these ground, with IBM was the leader and barely growing today. Because these machines were so big and heavy, users had to time share the usage of these machines from dummy terminals. Some of the machines ran on vacuum tubes. Tubes were hot, and a moth flew into one, melted itself on a tube, and caused the computer to malfunction. The term "de-bug" came from this era.

Mini, Micro, and Personal Computer

A The "personal computer" era started with the semiconductor industry producing cheap, powerful processors. Motorola and Intel were leaders in this field. The big and heavy mainframe machines were shrunk from house sized to desk top sized. Microsoft created DOS, but PC usage was obscure until Windows was introduced.

Client and Server, Monolithic & Service Oriented Architecture

On the enterprise front, software was delivered in a client server architecture. Client is the user facing front of of the computer system, usually a PC or a terminal. The back end of that system, invisible to the user, is the server. It is usually found in a room dedicated to servers, hence called back office. The programs that ran in the server included Enterprise Resource Planning, Accounting, Database. The programs (such as ERP) were written in a single large program - hence monolith. Monolithic programs sometimes needed to talk to other monolith programs, and this was done via a Service Oriented Architecture.

Cloud & RESTful API

The computer server infrastructure sat physically in the office, or in a dedicated remote building called data center. But companies did not like buying, maintaining, replacing servers that they owned. Why not lease them? That's what cloud did. They lease compute, and started to port their original "monolithic" programs to the cloud. What if you had one program used by many users at the same time? If you had 100 users, do you have 100 programs running? That is wasteful. Multi-tenancy technology allowed one program to run, but accessed by 100 users. Its slower, but data is consistent, making the system faster overall.

Cloud Native & Containers

Monolithic programs were not designed to scale to more users, more regions, more storage. And they made upgrade components of the software very hard - need to bring down the entire program, instead of just updating the little piece. Big monolithic programs were broken down into small containers. The containers talk to each other through API or queues. If a component of the program needed to be updated (let's say it is responsible for displaying the latest news - not critical if it is part of a stock trading website), just bring down that container, and replace it with a newer container.

The Future : Edge, WASM, Smart Contract

Some say we have swung to far into the container world. It benefits are obvious (scale, modular maintenance, etc), but it also brings a lot of overhead (connectivity, versioning, etc). We suspect adoption will be selective. New startups will pick the latest compute paradigm. But the old, with its vast amounts of technical debt (think COBOL), will change just enough to function, but not change enough to break. Watch out for more "edge" compute, including Web Assembly (WASM), where your browser is smart enough to perform compute, elevating the status of browser from "show" only to "compute and show". Blockchain "smart contracts" will unify business code from both the application and the network itself into once place - with the added benefit that it can natively handle money via cryptocurrency.

Identity & Access Management (IdM or IAM)

Identity & Access Management is the tool, framework, and process to control access to privileged resources. Let's look at current framework to deploy identity and access management : inspired by ISC2 is the IAAA framework : Identity,  Authenticate, Authorize Accounting. Decoding what this frame requires:

• Identity : the unique attributes that define you, usually validated and issued by a trusted entity
  
• Authentication : proof that you are indeed the person that you claim to be, sometimes called identity proofing
  
• Authorization (or Access Control) : once you prove who you are, what are you allowed to do and not do
  
• Accounting : logging of everything so that there is a forensic trail to audit all activities

Let's dig deeper into IAAA.

Identification - the unique attributes that define you, validated and issued by a trusted entity

Identity of a person usually starts at birth - your name - written on an official document. In California, it is on a "Certificate of Live Birth" document. Other examples identification include passports issued by its sovereign national government, drivers license by the state government, school ID by a university.

Authentication - prove that you are indeed the person that you claim to be

For you to prove that you are you, there are five types of authentication that can be used.

1. Type I : what you know, such as PIN, passwords, favorite ice cream flavor
2. Type II : what you have, such as a physical "token fob" 
3. Type III : what you are (bio), such as fingerprints, iris color
4. Type IV : where are you, such as work campus
5. Type V : what you do, such as a signature
   

Authorization (or Access) - once you prove who you are, what are you allowed to do

Once that your identity is know, you authenticate to be the identity, the IAM/IdM system controls what you can do.  Authorization can be 1) centralized or 2) decentralized. Centralized control is implemented via Single Sign On (SSO) systems. Decentralized control is trusting the the tools, process, and framework will enforce the IAAA framework. 

 

Systems that implements identity and access management : authenticate + authorize

SAML 2.0 (authenticate + authorize)      

Security And Markup Language (SAML) is a framework for systems to exchange   information on security. The information format is in XML. It is common used in Single Sign On (SSO) - using one login (un+pw) to access multiple web applications. Detractors say that it can be too verbose and cumbersome to use.

   

OpenID Connect (authenticate) + OAuth 2 (authorize)

Conclusion

You have a protected resource - such as  data or hardware. You only want to allow authorized users to be able to use it. Identity and Access Management (IdM or IAM) is the tool, process, and framework that can do this. It will follow the Identity Authentication Authorization Accounting (IAAA) framework to ensure that the user is authenticated, authorized to use the resource, and all actions will be tracked.