Week of November 20th | SQL Squirrels

“Climb🧗‍♀️ in the back with your head👤 in the Clouds☁️☁️ … And you’re gone

Hi All -

Happy Name Your PC💻 Day!

“Forward yesterday makes me wanna stay…”

“Welcome back, to that same old place that you laughed 😂 about”. So, after a short recess we made our splendiferous return this week. To where else? …But to no other than Google Cloud Platform a.k.a GCP☁️ , of course! 😊 So after completing our three-part Cloud Journey, we were feeling the need for a little refresher… Also, there were still had a few loose ends we needed to sew🧵 up. The wonderful folks at Google Cloud☁️ put together amazing compilation on GCP☁️ through their Google Cloud Certified Associate Cloud Engineer Path but we were feeling the need for a little more coverage on GCP CLI i.e. “gcloud”, “gsutil”, and “bq” . In addition, we had a great zest to learn a little more about some of the service offerings like GCP Development Services and APIs. Fortunately, we knew exactly who could deliver tremendous content on GCP☁️ as well as hit the sweet spot on some of the areas where we felt we were lacking a bit. That would be of course one of our favorite Canucks 🇨🇦 Mattias Andersson

For those who are not familiar with Mattias, he is one of the legendary instructors on A Cloud Guru. Mattias is especially well-known for his critically acclaimed Google Certified Associate Cloud Engineer 2020 course.

In this brilliantly produced course Mattias delivers the goods and then some! The goal of the course is to prepare those interested in preparing for Google’s Associate Cloud Engineer (ACE) Certification exam but it’s structured in a manner to efficiently to provide you with the skills to troubleshoot GCP through having a better understanding of “Data flows”. Throughout the course Mattias emphasizes the “see one, do one, teach one” technique in order to get the best ROI out of the tutorial.

So, after some warm salutations and a great overview of the ACE Exam, Mattias takes right to an introductions of all the Google Cloud product and Services. He accentuates the importance of Data Flow in fully understanding how all GCP solutions work. “Data Flow is taking data or information and it’s moving it around, processing it and remembering it.

Data flows — are the foundation of every system

“Let’s get it started, in here…And the base keep runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and…”

  • Moving, Processing, Remembering

“Share it fairly, but don’t take a slice of my pie 🥧”

After walking🚶‍♀️ us through how to create a Free account it was time ⏰ to kick off 🦵 us with a little Billing and Billing Export.

Billing Export -to BigQuery enables you to export your daily usage and cost estimates automatically throughout the day to a BigQuery dataset.

  • Export must be set up per billing account

Billing IAM — Role: Billing Account User

Budgets — Help with project planning and controlling costs

  • Setting a budget lets you track spend

Alerts 🔔 — notify billing administrators when spending exceeds a percentage of your budget

Google Cloud Shell 🐚 — provides with CLI access to Cloud☁️ Resources directly from your browser.

  • Command-line tool🔧 to interact GCP☁️
 gcloud-project=myprojid compute instances list
gcloud compute instances create myvm
gcloud services list --available
gsutil ls
gsutil mb -l northamerica-northeast1 gs://storage-lab-cli
gsutil label set bucketlables.json gs://storage-lab-cli

GCS via gsutil in Command Line

Create VM via gsutil in Command Line

gcloud config list gcloud config set project igneous-visitor-293922 gsutil ls gsutil ls gs://storage-lab-console-088/ gsutil ls gs://storage-lab-console-088/** gsutil mb --help gsutil mb -l northamerica-northeast1 gs://storage-lab-cli-088 gsutil label get gs://storage-lab-console-088/ gsutil label get gs://storage-lab-console-088/ > bucketlabels.json cat bucketlabels.json gsutil label get gs://storage-lab-cli-088 gsutil label set bucketlabels.json gs://storage-lab-cli-088 gsutil label ch -l "extralable:etravalue" gs://storage-lab-cli-088 gsutil versioning get gs://storage-lab-cli-088 gsutil versioning set on gs://storage-lab-cli-088 gsutil versioning get gs://storage-lab-cli-088 gsutil cp README-Cloudshell.txt gs://storage-lab-cli-088 gsutil ls -a gs://storage-lab-cli-088 gsutil rm gs://storage-lab-cli-088/README-Cloudshell.txt gsutil cp gs://storage-lab-console-088/** gs://storage-lab-cli-088/ gsutil acl ch -u AllUsers:R gs://storage-lab-cli-088/shutterstock.jpg

Confidentiality, Integrity, and Availability (CIA) Authentication, Authorization, Accounting (AIA)

gcloud config get-value project gcloud compute instances list gcloud services list gcloud services list --enabled gcloud services list --help gcloud services list -available gcloud services list --available |grep compute gcloud services -h gcloud compute instances create myvm gcloud compute instances delete myvm
  • Resiliency — Keep it running 🏃‍♂️

IAM — Resource Hierarchy👑

  • Identity hierarchy👑 (Google Groups)

IAM — Permissions & Roles

  • Resource⚙️

Permissions — allows you to a perform a certain action

Roles — is a collection of permissions to use or manage GCP☁️ resources

IAM — Members & Groups

Members — some Google-known identity

  • Each member is identifying by unique email📧 address

Groups — a collection of Google accounts and service accounts

IAM — Policies

  • Every group has a unique email📧 address that is associated with the group

Policies — binds members to roles for some scope of resources

  • Enforce who can do what to which thing(s)
gCloud[GROUP] add-iam-policy-binding [Resource-NAME] --role [ROLE-ID-TO-GRANT] -member user: [USER-EMAIL] gCloud[GROUP] remove-iam-policy-binding [Resource-NAME] --role [ROLE-ID-TO-REVOKE] -member user: [USER-EMAIL]

Billing Accounts — represents some way to pay for GCP☁️ service usuage

  • Type of Resource⚙️ that lives outside of Projects

Billing Account Creator

Create new self-service billing accounts

Org

Billing Account Administrator

Manage billing accounts

Billing Account

Billing Account User

Link Projects to billing accounts

Billing Account

Billing Account Viewer

View billing account cost information and transactions

Billing Account

Project Billing Manager

Link/unlink the project to/from a billing account

Project

Monthly Invoiced Billing — Billed monthly and pay by invoice due date

Networking Unicast vs Anycast

  • Pay via check or wire transfer

Unicast — There is only one unique device in the world that can handle this; send it there.

Load Balancing — Layer 4 vs Layer 7

Anycast — There are multiple devices that could handle this; send it to anyone — but ideally the closest.

  • TCP is usually called Layer 4 (L4)

DNS — Name resolution (via the Domain Name System) can be the first step in routing

Options for Data from one Resource to another

  • Some known issues with DNS

IPs and CIDRS

  • VPC (Global🌎) Virtual Private Cloud☁️ — Private SDN space in GCP☁️

Subnet CIDR Ranges Shared VPC

  • IP Address is 255.255.255.255 (dotted quad) where each piece is 0–255

GKE

“Ride, captain👨🏿‍✈️ ride upon your mystery ship⛵️

A Kubernetes ☸️ cluster is a set of nodes that run containerized applications. Containerizing applications packages an app with its dependences and some necessary services.

Deploy and manage clusters on-prem

K8s ☸️ you know that the control plane consists of the kube-apiserver, kube-scheduler, kube-controller-manager and an etcd datastore.

Step 1: The container runtime

Step 2: Installing kubeadm

Step 3: Starting the Kubernetes cluster ☸️

Step 4: Joining a node to the Kubernetes cluster ☸️

To deploy and manage your containerized applications and other workloads on your Google Kubernetes Engine (GKE) cluster, you use the K8s ☸️ system to create K8s ☸️ controller objects. These controller objects represent the applications, daemons, and batch jobs running 🏃‍♂️ on your clusters.

Kubernetes fits into the Cloud-native ecosystem

Cloud Native Application Properties

Deploy and manage applications on Kubernetes ☸️

K8s ☸️ native technologies (tools/systems/interfaces) are those that are primarily designed and built for Kubernetes ☸️.

K8s ☸️ deployments can be managed via Kubernetes ☸️ command line interface kubectl. Kubectl uses the Kubernetes ☸️ API to interact with the cluster.

When creating a deployment, you will need to specify the container image for your application and the number of replicas that you need in your cluster.

  • Create Application

K8s ☸️ objects can be created, updated, and deleted by storing multiple object configuration files in a directory and using kubectl apply to recursively create and update those objects as needed.

DaemonSet

This method retains writes made to live objects without merging the changes back into the object configuration files. kubectl diff also gives you a preview of what changes apply will make.

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.

Some typical uses of a DaemonSet are:

Google Kubernetes☸️ Engine (GKE) offers integrated support for two types of Cloud☁️ Load Balancing for a publicly accessible application:

When you specify type:LoadBalancer 🏋️‍♀️ in the Resource⚙️ manifest:

Although you can use either of these types of load balancers 🏋️‍♀️ for HTTP(S) traffic🚦, they operate in OSI layers 3/4 and are not aware of HTTP connections or individual HTTP requests and responses.

GCP Services Compute

Imagine all the people👥 sharing all the world🌎

Compute Engine (GCE) — (Zonal) (IaaS) — Fast-booting Virtual Machines (VMs) for rent/demand

Kubernetes Engine (GKE) — (Regional (IaaS/Paas) -Managed Kubernetes ☸️ cluster for running 🏃‍♂️ Docker🐳 containers (with autoscaling)

  • Kubernetes☸️ DNS on by default for service discovery

App Engine (GAE) — (Regional (PaaS) that takes your code and runs it

  • Much more than just compute — Integrates storage, queues, NoSQL

Storage

Cloud Functions — (Regional (FaaS), “Serverless” -Managed K8s☸️ cluster for running 🏃‍♂️ Docker🐳 containers (with autoscaling)

Persistent Disk (PD) — (Zonal) Flexible🧘‍♀️, block-based🧱 network-attached storage; boot disk for every GCE instance

Cloud Filestore — (Zonal) Fully managed file-based storage

  • “Predictably fast🏃‍♂️ performance for your file-based workloads”

Cloud Storage (GCS) — (Regional, Multi-Regional) Infinitely Scalable⚖️, fully managed, versioned, and highly durable object storage

Databases

  • Designed for 99.999999999% (11 9’s) durability

Cloud SQL — (Regional, Fully managed and reliable MySQL and PostgreSQL databases

  • Supports automatic replication, backup, failover, etc.

Cloud Spanner — (Regional, Multi-Regional), Global🌎 horizontally Scalable⚖️, strongly consistent 💪, relational database service”

  • “From 1 to 100s or 1000s of nodes”

BigQuery (BQ) — Multi-Regional Serverless column-store data warehouse for analytics using SQL

  • Scales⚖️ internally (TB in seconds and PB in minutes)

Cloud Datastore — (Regional, Multi-Regional) Managed & autoscale⚖️ NoSQL DB with indexes, queries, and ACID trans, support

Cloud Bigtable — (Zonal) Low latency & high throughput NoSQL DB for large operational & analytical apps

  • Supports open-source HBase API

Firebase Realtime DB & Cloud Firestore 🔥 — (Regional, Multi-Regional) NoSQL document📃 stores with ~real-time client updates via managed WebSockets

  • Firebase DB is single (potentially huge) JSON doc, located only in central US

Data Transfer ↔️

Data Transfer Appliance — Rackable, high-capacity storage 🗄 server to physically ship data to GCS

  • Ingest only; not a way to avoid egress charges

External Networking

Storage Transfer Service — (Global) Copies objects for you, so you don’t need to set up a machine to do it

Google Domains — (Global) Google’s registrar for domain names

Cloud DNS — (Global) Scalable⚖️, reliable, & managed authoritative Domain (DNS) service

Static IP Addresses — (Regional, Global🌎 Reserve static IP addresses in projects and assign them to resources

Cloud Load Balancing (CLB) — (Regional, Global🌎 High-perf, Scalable ⚖️ traffic🚦 distribution integrated with autoscaling & Cloud☁️ CDN

  • SDN naturally handles spikes without any prewarming, no instances or devices

Cloud CDN — (Global) Low-latency content delivery based on HTTP(S) CLB integrated w/ GCE & GCS

Virtual Private Cloud (VPC) — (Regional, Global), Global IP v4 unicast Software-Defined Network (SDN) for GCP☁️ resources

Cloud Interconnect — (Regional, Multi-Regional) Options for connecting external networks to Google’s network

Internal Networking

  • Private connections to VPC via Cloud VPN or Dedicated/Partner Interconnect

Cloud Virtual Private Network (VPN) — (Regional) IPSEC VPN to connect to VPC via public internet for low-volume data connections

  • For persistent, static connections between gateways (i.e. not for a dynamic client)

Dedicated Interconnect — (Regional, Multi-Regional) Direct physical link 🔗 between VPC and on-prem for high-volume data connections

  • VLAN attachment is private connection to VPC in one region: no public GCP☁️ APIs

Cloud Router 👮‍♀️ — (Regional) Dynamic routing (BGP) for hybrid networks linking GCP VPCs to external networks

  • Works with Cloud VPN and Dedicated Interconnect

CDN Interconnect — (Regional, Multi-Regional) Direct, low-latency connectivity to certain CDN providers, with cheaper egress

  • For external CDNs, not Google’s Cloud CDN service

Cloud Machine Learning (ML) Engine — (Regional) Massively Scalable ⚖️ managed service for training ML models & making predictions

Cloud Vison API👓 — (Global) Classifies images🖼 into categories, detects objects/faces, & finds/reads printed text

  • Pre-trained ML model to analyze images🖼 and discover their contents

Cloud Natural Language API 💬 — (Global) Analyzes text for sentiment, intent, & content classification, and extracts info

Cloud Translation API -(Global) Translate text among 100+ languages; optionally auto-detects source language

Dialogflow — (Global) Build conversational interfaces for websites, mobile apps, messaging, IoT devices

Big Data and IoT

Cloud Job Discovery — (Global) Helps career sites, company job boards, etc. to improve engagement & conversion

Four Different Stages:

  1. Ingest — Pull in all the raw data in

Cloud Internet of Things (IoT) Core — (Global) Fully managed service to connect, manage, and ingest data from device Globally

Cloud Pub/Sub — (Global) Infinitely Scalable⚖️ at-least-once messaging for ingestion, decoupling, etc.

  • “Global🌎 by default: Publish… and consume from anywhere, with consistent latency”.

Cloud Dataprep — (Global) Visually explore, clean, and prepare data for analysis without running 🏃‍♂️ servers

  • “Data Wrangling” (i.e. “ad-hoc ETL”) for business analysts, not IT pros

Cloud Dataproc — (Zonal) Batch MapReduce processing via configurable, managed Spark & Hadoop clusters

  • Handles being told to scale (adding or removing nodes) even while running 🏃‍♂️ jobs

Cloud Datalab 🧪- (Regional) Interactive tool 🔧 for data exploration🔎, analysis, visualization📊 and machine learning

  • Uses Jupyter Notebook📒

Cloud Data Studio — (Global) Big Data Visualization📊 tool 🔧 for dashboards and reporting

Cloud Genomics 🧬- (Global) Store and process genomes🧬 and related experiments

Roles — (Global) collections of Permissions to use or manage GCP☁️ resources

  • Permissions allow you to perform certain actions: Service.Resource.Verb

Cloud Identity and Access Management (IAM) — (Global) Control access to GCP☁️ resources: authorization, not really authentication/identity

  • Policies bind Members to Roles at a hierarchy👑 level: Org, Folder📂, Project, Resource⚙️

Service Accounts — (Global) Special types of Google account that represents an application, not an end user

  • Can be “assumed” by applications or individual users (when so authorized)

Cloud Identity — (Global) Identity as a Service (IDaaS, not DaaS) to provision and manage users and groups

Security Key Enforcement — (Global) USB or Bluetooth 2-step verification device that prevents phishing🎣

  • Not like just getting a code via email📧 or text message…

Cloud Resource Manager — (Global) Centrally manage & secure organization’s projects with custom Folder📂 hierarchy👑

Cloud Identity-Aware Proxy (IAP) — (Global) Guards apps running 🏃‍♂️ on GCP☁️ via identity verification, not VPN access

Cloud Audit Logging — (Global) “Who did what, where and when?” within GCP☁️ projects

Cloud Armor🛡 — (Global) Edge-level protection from DDoS & other attacks on Global🌎 HTTP(S) LB🏋️‍♀️

Cloud Security Scanner — (Global) Free but limited GAE app vulnerability scanner with “very low false positive rates”

  • “After you set up a scan, Cloud☁️ Security🔒 Scanner automatically crawls your application, following all links🔗 within the scope of your starting URLs, and attempts to exercise as many user inputs and event handlers as possible.”

Cloud Data Loss Prevention API (DLP) — (Global) Finds and optionally redacts sensitive info is unstructured data streams

  • Helps you minimize what you collect, expose, or copy to other systems

Event Threat Detection (ETD) — (Global) Automatically scans your Stackdriver logs for suspicious activity

  • Uses industry-leading threat intelligence, including Google Safe Browsing

Cloud Security Command Center (SCC) — (Global)

  • “Comprehensive security🔒 management and data risk platform for GCP☁️”

Cloud Key Management Services (KMS) — (Regional, Multi-Regional, Global) Low-latency service to manage and use cryptographic keys🔑

  • Supports symmetric (e.g. AES) and asymmetric (e.g. RSA, EC) algorithms

Cloud Hardware Security Module (HSM) — (Regional, Multi-Regional, Global) Cloud KMS keys🔑 managed by FIPS 140–2 Level 3 certified HSMs

Operations and Management

  • Device hosts encryption 🔐 keys🔑 and performs cryptographic operations

Google Stackdriver — (Global) Family of services for monitoring, logging & diagnosing apps on GCP/AWS/hybrid

  • Service integrations add lots of value — among Stackdriver and with GCP☁️

Stackdriver Monitoring — (Global) Gives visibility into perf, uptime, & overall health of Cloud☁️ apps (based on collectd)

Stackdriver Logging — (Global) Store, search🔎, analyze, monitor, and alert 🔔 on log data & events (based on Fluentd)

Stackdriver Error Reporting — (Global) Counts, analyzes, aggregates, & tracks crashes in helpful centralized interface

  • Smartly aggregates errors into meaningful groups tailored to language/framework

Stackdriver Trace — (Global) Tracks and displays call tree 🌳 & timings across distributed systems, to debug perf

Stackdriver Debugger — (Global) Grabs program state (callstack, variables, expressions) in live deploys, low impact

Stackdriver Profiler — (Global) Continuous CPU and memory profiling to improve perf & reduce cost

Cloud Deployment Manager — (Global) Create/manage resources via declarative templates: “Infrastructure as Code”

Cloud Billing API 🧾- (Global) Programmatically manage billing for GCP☁️ projects and get GCP☁️ pricing

Development and APIs

Cloud Source Repositories — (Global) Hosted private Git repositories, with integrations to GCP☁️ and other hosted repos

Cloud Build 🏗 — (Global) Continuously takes source code and builds, tests and deploys it — CI/CD service

  • Trigger from Cloud Source Repository (by branch, tag or commit) or zip🤐 in GCS

Container Registry (GCR) 📦- (Regional, Multi-Regional) Fast🏃‍♂️, private Docker🐳 image storage 🗄 (based on GCS) with Docker🐳 V2 Registry API

Cloud Endpoints — (Global) Handles authorization, monitoring, logging, & API keys🔑 for APIs backed by GCP☁️

  • Proxy instances are distributed and hook into Cloud Load Balancer 🏋️‍♀️

Apigee API Platform — (Global) Full-featured & enterprise-scale API management platform for whole API lifecycle

Test Lab for Android — (Global) Cloud☁️ infrastructure for running 🏃‍♂️ test matrix across variety of real Android devices

  • Production-grade devices flashed with Android version and locale you specify

“Well, we all shine☀️ on… Like the moon🌙 and the stars🌟 and the sun🌞

Thanks -

-MCS

Originally published at https://sqlsquirrels.com on November 20, 2020.

A Passionate Technologist. Blogging about my journey in learning exciting technologies

A Passionate Technologist. Blogging about my journey in learning exciting technologies