Week of November 20th | SQL Squirrels

“Climb🧗‍♀️ in the back with your head👤 in the Clouds☁️☁️ … And you’re gone “

Hi All -

Happy Name Your PC💻 Day!

“Forward yesterday makes me wanna stay…”

“Welcome back, to that same old place that you laughed 😂 about”. So, after a short recess we made our splendiferous return this week. To where else? …But to no other than Google Cloud Platform a.k.a GCP☁️ , of course! 😊 So after completing our three-part Cloud Journey, we were feeling the need for a little refresher… Also, there were still had a few loose ends we needed to sew🧵 up. The wonderful folks at Google Cloud☁️ put together amazing compilation on GCP☁️ through their Google Cloud Certified Associate Cloud Engineer Path but we were feeling the need for a little more coverage on GCP CLI i.e. “gcloud”, “gsutil”, and “bq” . In addition, we had a great zest to learn a little more about some of the service offerings like GCP Development Services and APIs. Fortunately, we knew exactly who could deliver tremendous content on GCP☁️ as well as hit the sweet spot on some of the areas where we felt we were lacking a bit. That would be of course one of our favorite Canucks 🇨🇦 Mattias Andersson

For those who are not familiar with Mattias, he is one of the legendary instructors on A Cloud Guru. Mattias is especially well-known for his critically acclaimed Google Certified Associate Cloud Engineer 2020 course.

In this brilliantly produced course Mattias delivers the goods and then some! The goal of the course is to prepare those interested in preparing for Google’s Associate Cloud Engineer (ACE) Certification exam but it’s structured in a manner to efficiently to provide you with the skills to troubleshoot GCP through having a better understanding of “Data flows”. Throughout the course Mattias emphasizes the “see one, do one, teach one” technique in order to get the best ROI out of the tutorial.

So, after some warm salutations and a great overview of the ACE Exam, Mattias takes right to an introductions of all the Google Cloud product and Services. He accentuates the importance of Data Flow in fully understanding how all GCP solutions work. “Data Flow is taking data or information and it’s moving it around, processing it and remembering it.”

Data flows — are the foundation of every system

“Let’s get it started, in here…And the base keep runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and runnin’ 🏃‍♂️ runnin’ 🏃‍♂️, and…”

• Moving, Processing, Remembering
• Build mental models
• Identify and think through data flows
• Requirement and options not always clear
• Critical skills for both real world🌎 and exam📝 questions

“Share it fairly, but don’t take a slice of my pie 🥧”

After walking🚶‍♀️ us through how to create a Free account it was time ⏰ to kick off 🦵 us with a little Billing and Billing Export.

Billing Export -to BigQuery enables you to export your daily usage and cost estimates automatically throughout the day to a BigQuery dataset.

• Export must be set up per billing account
• Resources should be placed into appropriate projects
• Resources should be tagged with labels🏷
• Billing export is not real-time

Billing IAM — Role: Billing Account User

Budgets — Help with project planning and controlling costs

• Setting a budget lets you track spend
• Apply budget to billing account or a Project

Alerts 🔔 — notify billing administrators when spending exceeds a percentage of your budget

Google Cloud Shell 🐚 — provides with CLI access to Cloud☁️ Resources directly from your browser.

• Command-line tool🔧 to interact GCP☁️
• Basic Syntax

 gcloud-project=myprojid compute instances list
 gcloud compute instances create myvm
 gcloud services list --available
 gsutil ls
 gsutil mb -l northamerica-northeast1 gs://storage-lab-cli
 gsutil label set bucketlables.json gs://storage-lab-cli

GCS via gsutil in Command Line

Create VM via gsutil in Command Line

gcloud config list gcloud config set project igneous-visitor-293922 gsutil ls gsutil ls gs://storage-lab-console-088/ gsutil ls gs://storage-lab-console-088/** gsutil mb --help gsutil mb -l northamerica-northeast1 gs://storage-lab-cli-088 gsutil label get gs://storage-lab-console-088/ gsutil label get gs://storage-lab-console-088/ > bucketlabels.json cat bucketlabels.json gsutil label get gs://storage-lab-cli-088 gsutil label set bucketlabels.json gs://storage-lab-cli-088 gsutil label ch -l "extralable:etravalue" gs://storage-lab-cli-088 gsutil versioning get gs://storage-lab-cli-088 gsutil versioning set on gs://storage-lab-cli-088 gsutil versioning get gs://storage-lab-cli-088 gsutil cp README-Cloudshell.txt gs://storage-lab-cli-088 gsutil ls -a gs://storage-lab-cli-088 gsutil rm gs://storage-lab-cli-088/README-Cloudshell.txt gsutil cp gs://storage-lab-console-088/** gs://storage-lab-cli-088/ gsutil acl ch -u AllUsers:R gs://storage-lab-cli-088/shutterstock.jpg

Confidentiality, Integrity, and Availability (CIA) Authentication, Authorization, Accounting (AIA)

gcloud config get-value project gcloud compute instances list gcloud services list gcloud services list --enabled gcloud services list --help gcloud services list -available gcloud services list --available |grep compute gcloud services -h gcloud compute instances create myvm gcloud compute instances delete myvm

• Resiliency — Keep it running 🏃‍♂️
• Security🔒 Products
• Security🔒 Features
• Security🔒 Mindset

IAM — Resource Hierarchy👑

• Identity hierarchy👑 (Google Groups)
• Resource⚙️ hierarchy👑 (Organization, Folders📂, Projects)
• GCS ACLs
• Billing management
• Networking structure & restrictions
• Audit / Activity Logs (provided by Stackdriver)
• GCS object Lifecycle Management

IAM — Permissions & Roles

• Resource⚙️
• Project
• Folder📂
• Organization

Permissions — allows you to a perform a certain action

Roles — is a collection of permissions to use or manage GCP☁️ resources

IAM — Members & Groups

Members — some Google-known identity

• Each member is identifying by unique email📧 address
• Can be:

Groups — a collection of Google accounts and service accounts

IAM — Policies

• Every group has a unique email📧 address that is associated with the group
• You never act as the group
• Use them for everything
• Can be used for owner when within an organization
• Can nest groups in an organization

Policies — binds members to roles for some scope of resources

• Enforce who can do what to which thing(s)
• Roles and Members listed in policy, but Resources identified by attachment
• Always additive (Allow) and never subtractive (no Deny)
• One policy per Resource⚙️
• Max 1500-member binding per policy

gCloud[GROUP] add-iam-policy-binding [Resource-NAME] --role [ROLE-ID-TO-GRANT] -member user: [USER-EMAIL] gCloud[GROUP] remove-iam-policy-binding [Resource-NAME] --role [ROLE-ID-TO-REVOKE] -member user: [USER-EMAIL]

Billing Accounts — represents some way to pay for GCP☁️ service usuage

• Type of Resource⚙️ that lives outside of Projects
• Can belong to an Organization

Billing Account Creator

Create new self-service billing accounts

Org

Billing Account Administrator

Manage billing accounts

Billing Account

Billing Account User

Link Projects to billing accounts

Billing Account

Billing Account Viewer

View billing account cost information and transactions

Billing Account

Project Billing Manager

Link/unlink the project to/from a billing account

Project

Monthly Invoiced Billing — Billed monthly and pay by invoice due date

Networking Unicast vs Anycast

• Pay via check or wire transfer
• Increase project and quota limits

Unicast — There is only one unique device in the world that can handle this; send it there.

Load Balancing — Layer 4 vs Layer 7

Anycast — There are multiple devices that could handle this; send it to anyone — but ideally the closest.

• TCP is usually called Layer 4 (L4)
• HTTP and HTTPS work at Layer (L7)

DNS — Name resolution (via the Domain Name System) can be the first step in routing

Options for Data from one Resource to another

• Some known issues with DNS
• Premium tier routing with Global🌎 anycast Ips avoids these problems

IPs and CIDRS

• VPC (Global🌎) Virtual Private Cloud☁️ — Private SDN space in GCP☁️
• Subnets (regional) — create logical spaces to contain resources

Subnet CIDR Ranges Shared VPC

• IP Address is 255.255.255.255 (dotted quad) where each piece is 0–255
• CIDR block is group of IP addresses specified in <IP>/xy notation
• RFC1918 defines private (i.e non-internet) address ranges you can use:
• Lets multiple projects coexist on same local network (private IP space)
• Let’s a centralized team manage network security🔒

GKE

“Ride, captain👨🏿‍✈️ ride upon your mystery ship⛵️ “

A Kubernetes ☸️ cluster is a set of nodes that run containerized applications. Containerizing applications packages an app with its dependences and some necessary services.

Deploy and manage clusters on-prem

K8s ☸️ you know that the control plane consists of the kube-apiserver, kube-scheduler, kube-controller-manager and an etcd datastore.

Step 1: The container runtime

Step 2: Installing kubeadm

Step 3: Starting the Kubernetes cluster ☸️

Step 4: Joining a node to the Kubernetes cluster ☸️

To deploy and manage your containerized applications and other workloads on your Google Kubernetes Engine (GKE) cluster, you use the K8s ☸️ system to create K8s ☸️ controller objects. These controller objects represent the applications, daemons, and batch jobs running 🏃‍♂️ on your clusters.

Kubernetes fits into the Cloud-native ecosystem

Cloud Native Application Properties

Deploy and manage applications on Kubernetes ☸️

K8s ☸️ native technologies (tools/systems/interfaces) are those that are primarily designed and built for Kubernetes ☸️.

K8s ☸️ deployments can be managed via Kubernetes ☸️ command line interface kubectl. Kubectl uses the Kubernetes ☸️ API to interact with the cluster.

When creating a deployment, you will need to specify the container image for your application and the number of replicas that you need in your cluster.

• Create Application
• Create a Docker🐳 container image
• Create a K8s ☸️ Deployment

K8s ☸️ objects can be created, updated, and deleted by storing multiple object configuration files in a directory and using kubectl apply to recursively create and update those objects as needed.

DaemonSet

This method retains writes made to live objects without merging the changes back into the object configuration files. kubectl diff also gives you a preview of what changes apply will make.

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.

Some typical uses of a DaemonSet are:

Google Kubernetes☸️ Engine (GKE) offers integrated support for two types of Cloud☁️ Load Balancing for a publicly accessible application:

When you specify type:LoadBalancer 🏋️‍♀️ in the Resource⚙️ manifest:

Although you can use either of these types of load balancers 🏋️‍♀️ for HTTP(S) traffic🚦, they operate in OSI layers 3/4 and are not aware of HTTP connections or individual HTTP requests and responses.

GCP Services Compute

Imagine all the people👥 sharing all the world🌎

Compute Engine (GCE) — (Zonal) (IaaS) — Fast-booting Virtual Machines (VMs) for rent/demand

Kubernetes Engine (GKE) — (Regional (IaaS/Paas) -Managed Kubernetes ☸️ cluster for running 🏃‍♂️ Docker🐳 containers (with autoscaling)

• Kubernetes☸️ DNS on by default for service discovery
• NO IAM integration (unlike AWS ECS)
• Integrates with Persistent Disk for storage
• Pay for underlying GCE instances
• No GKE management fee, no matter how many nodes in cluster

App Engine (GAE) — (Regional (PaaS) that takes your code and runs it

• Much more than just compute — Integrates storage, queues, NoSQL
• Flex mode (“App Engine Flex”) can run any container & access VPC
• Auto-Scales⚖️ based on load
• Effectively pay for underlying GCE instances and other services

Storage

Cloud Functions — (Regional (FaaS), “Serverless” -Managed K8s☸️ cluster for running 🏃‍♂️ Docker🐳 containers (with autoscaling)

Persistent Disk (PD) — (Zonal) Flexible🧘‍♀️, block-based🧱 network-attached storage; boot disk for every GCE instance

Cloud Filestore — (Zonal) Fully managed file-based storage

• “Predictably fast🏃‍♂️ performance for your file-based workloads”
• Accessible to GCE and GKE through your VPC, via NFSv3 protocol
• Primary use case is application migration to Cloud☁️ (“lift and shift”)🚜
• Fully manages file serving, but not backups
• Pay for provisioned TBs in “Standard” (slow) or “Premium” (fast🏃‍♂️) mode
• Minimum provisioned capacity of 1TB (Standard) or 2.5TB (Premium)

Cloud Storage (GCS) — (Regional, Multi-Regional) Infinitely Scalable⚖️, fully managed, versioned, and highly durable object storage

Databases

• Designed for 99.999999999% (11 9’s) durability
• Strong consistent💪 (even for overwrite PUTs and DELETEs)
• Integrated site hosting and CDN functionality
• Lifecycle♻️ transitions across classes: Multi-Regional, Regional, Nearline, Coldline🥶
• All Classes have same API, so can use gsutil and gcsfuse

Cloud SQL — (Regional, Fully managed and reliable MySQL and PostgreSQL databases

• Supports automatic replication, backup, failover, etc.
• Scaling is manual (both vertically and horizontally)
• Effectively pay for underlying GCE instances and PDs

Cloud Spanner — (Regional, Multi-Regional), Global🌎 horizontally Scalable⚖️, strongly consistent 💪, relational database service”

• “From 1 to 100s or 1000s of nodes”
• Chooses Consistency and Partition — Tolerance (CP and CAP theorem)
• Pay for provisioned node time (by region/multi-region) plus used storage-time

BigQuery (BQ) — Multi-Regional Serverless column-store data warehouse for analytics using SQL

• Scales⚖️ internally (TB in seconds and PB in minutes)
• Pay for GBs actually considered (scanned) during queries
• Pay for GBs added via streaming inserts

Cloud Datastore — (Regional, Multi-Regional) Managed & autoscale⚖️ NoSQL DB with indexes, queries, and ACID trans, support

Cloud Bigtable — (Zonal) Low latency & high throughput NoSQL DB for large operational & analytical apps

• Supports open-source HBase API
• Integrates with Hadoop, Dataflow, Dataproc
• Pay for processing node hours
• GB-hours used for storage 🗄 (cheap HDD or fast🏃‍♂️ SSD)

Firebase Realtime DB & Cloud Firestore 🔥 — (Regional, Multi-Regional) NoSQL document📃 stores with ~real-time client updates via managed WebSockets

• Firebase DB is single (potentially huge) JSON doc, located only in central US
• Cloud☁️ Firestore has collection, documents📃, and contained data

Data Transfer ↔️

Data Transfer Appliance — Rackable, high-capacity storage 🗄 server to physically ship data to GCS

• Ingest only; not a way to avoid egress charges
• 100 TB or 480 TB/week is faster than a saturated 6 Gbps link🔗

External Networking

Storage Transfer Service — (Global) Copies objects for you, so you don’t need to set up a machine to do it

Google Domains — (Global) Google’s registrar for domain names

Cloud DNS — (Global) Scalable⚖️, reliable, & managed authoritative Domain (DNS) service

Static IP Addresses — (Regional, Global🌎 Reserve static IP addresses in projects and assign them to resources

Cloud Load Balancing (CLB) — (Regional, Global🌎 High-perf, Scalable ⚖️ traffic🚦 distribution integrated with autoscaling & Cloud☁️ CDN

• SDN naturally handles spikes without any prewarming, no instances or devices
• Regional Network Load Balancer 🏋️‍♀️: health checks, round robin, session affinity
• Pay by making ingress traffic🚦 billable (Cheaper than egress) plus hourly per rule

Cloud CDN — (Global) Low-latency content delivery based on HTTP(S) CLB integrated w/ GCE & GCS

Virtual Private Cloud (VPC) — (Regional, Global), Global IP v4 unicast Software-Defined Network (SDN) for GCP☁️ resources

Cloud Interconnect — (Regional, Multi-Regional) Options for connecting external networks to Google’s network

Internal Networking

• Private connections to VPC via Cloud VPN or Dedicated/Partner Interconnect
• Significantly lower egress fees

Cloud Virtual Private Network (VPN) — (Regional) IPSEC VPN to connect to VPC via public internet for low-volume data connections

• For persistent, static connections between gateways (i.e. not for a dynamic client)
• Encrypted 🔐 link🔗 to VPC (as opposed to Dedicated interconnect), into one subnet
• Supports both static and dynamic routing
• 99.9% availability SLA
• Pay per tunnel-hour
• Normal traffic🚦 charges apply

Dedicated Interconnect — (Regional, Multi-Regional) Direct physical link 🔗 between VPC and on-prem for high-volume data connections

• VLAN attachment is private connection to VPC in one region: no public GCP☁️ APIs
• Links are private but not Encrypted 🔐; can layer your own encryption 🔐
• Pay fee 10 Gbps link, plus (relatively small) fee per VLAN attachment
• Pay reduced egress rates from VPC through Dedicated Interconnect

Cloud Router 👮‍♀️ — (Regional) Dynamic routing (BGP) for hybrid networks linking GCP VPCs to external networks

• Works with Cloud VPN and Dedicated Interconnect
• Automatically learns subnets in VPC and announces them to on-prem network
• Without Cloud Router👮‍♀️ you must manage static routes for VPN
• Free to set up
• Pay for usual VPC egress

CDN Interconnect — (Regional, Multi-Regional) Direct, low-latency connectivity to certain CDN providers, with cheaper egress

• For external CDNs, not Google’s Cloud CDN service
• Works for both pull and push cache fills
• Contact CDN provider to set up for GCP☁️ project and which regions
• Free to enable, then pay less for the egress you configured

Cloud Machine Learning (ML) Engine — (Regional) Massively Scalable ⚖️ managed service for training ML models & making predictions

Cloud Vison API👓 — (Global) Classifies images🖼 into categories, detects objects/faces, & finds/reads printed text

• Pre-trained ML model to analyze images🖼 and discover their contents
• Classifies into thousands of categories (e.g., “sailboat”, “lion”, “Eiffel Tower”)
• Upload images🖼 or point to ones stored in GCS

Cloud Natural Language API 💬 — (Global) Analyzes text for sentiment, intent, & content classification, and extracts info

Cloud Translation API -(Global) Translate text among 100+ languages; optionally auto-detects source language

Dialogflow — (Global) Build conversational interfaces for websites, mobile apps, messaging, IoT devices

Big Data and IoT

Cloud Job Discovery — (Global) Helps career sites, company job boards, etc. to improve engagement & conversion

Four Different Stages:

1. Ingest — Pull in all the raw data in
2. Store — Store data without data loss and easy retrieval
3. Process — transform that raw data into some actionable information
4. Explore & Visualize — turn the results of that analysis into something that’s valuable for your business

Cloud Internet of Things (IoT) Core — (Global) Fully managed service to connect, manage, and ingest data from device Globally

Cloud Pub/Sub — (Global) Infinitely Scalable⚖️ at-least-once messaging for ingestion, decoupling, etc.

• “Global🌎 by default: Publish… and consume from anywhere, with consistent latency”.
• Messages can be up to 10 MB and undelivered ones stored for 7 days-but no DLQ
• Push mode delivers to HTTPS endpoints & succeeds on HTTP success status code
• Pull mode delivers messages to requestion clients and waits for ACK to delete
• Pay for data volume

Cloud Dataprep — (Global) Visually explore, clean, and prepare data for analysis without running 🏃‍♂️ servers

• “Data Wrangling” (i.e. “ad-hoc ETL”) for business analysts, not IT pros
• Managed version of Trifacta Wrangler — and managed by Trifacta, not Google
• Source data from GCS, BQ, or file upload — formatted in CSV, JSON, or relational
• Automatically detects schemas, datatypes, possible joins, and various anomalies
• Pay for underlying Daaflow job, plus management overhead charge
• Pay for other accessed services (e.g. GCS, BQ)

Cloud Dataproc — (Zonal) Batch MapReduce processing via configurable, managed Spark & Hadoop clusters

• Handles being told to scale (adding or removing nodes) even while running 🏃‍♂️ jobs
• Integrated with Cloud☁️ Storage, BigQuery, Bigtable, and some Stackdriver services
• “Image versioning” switches between versions of Spark, Hadoop, & other tools
• Pay directly for underlying GCE servers used in the cluster — optionally preemptible
• Pay a Cloud Dataproc management fee per vCPU-hour in the cluster
• Best for moving existing Spark/Hadoop setups to GCP☁️

Cloud Datalab 🧪- (Regional) Interactive tool 🔧 for data exploration🔎, analysis, visualization📊 and machine learning

• Uses Jupyter Notebook📒
• Supports iterative development of data analysis algorithms in Python🐍/ SQL/~JS
• Pay for GCE/GAE instance hosting and storing (on PD) your notebook📒
• Pay for any other resources accessed (e.g. BigQuery)

Cloud Data Studio — (Global) Big Data Visualization📊 tool 🔧 for dashboards and reporting

Cloud Genomics 🧬- (Global) Store and process genomes🧬 and related experiments

Roles — (Global) collections of Permissions to use or manage GCP☁️ resources

• Permissions allow you to perform certain actions: Service.Resource.Verb
• Predefined Roles: Give granular access to specific GCP☁️ resources (IAM)
• Custom Roles: Project- or Org-level collections you define of granular permissions

Cloud Identity and Access Management (IAM) — (Global) Control access to GCP☁️ resources: authorization, not really authentication/identity

• Policies bind Members to Roles at a hierarchy👑 level: Org, Folder📂, Project, Resource⚙️
• IAM is free; pay for authorized GCP☁️ service usage

Service Accounts — (Global) Special types of Google account that represents an application, not an end user

• Can be “assumed” by applications or individual users (when so authorized)
• “Important: For almost all cases, whether you are developing locally or in a production application, you should use service accounts, rather than user accounts or API keys🔑.”
• Consider resources and permissions required by application; use least privilege
• Can generate and download private keys🔑 (user-managed keys🔑), for non-GCP☁️
• Cloud-Platform-managed keys🔑 are better, for GCP☁️ (i.e. GCF, GAE, GCE, and GKE)

Cloud Identity — (Global) Identity as a Service (IDaaS, not DaaS) to provision and manage users and groups

Security Key Enforcement — (Global) USB or Bluetooth 2-step verification device that prevents phishing🎣

• Not like just getting a code via email📧 or text message…
• Eliminates man-in-the-middle (MITM) attacks against GCP☁️ credentials

Cloud Resource Manager — (Global) Centrally manage & secure organization’s projects with custom Folder📂 hierarchy👑

Cloud Identity-Aware Proxy (IAP) — (Global) Guards apps running 🏃‍♂️ on GCP☁️ via identity verification, not VPN access

Cloud Audit Logging — (Global) “Who did what, where and when?” within GCP☁️ projects

Cloud Armor🛡 — (Global) Edge-level protection from DDoS & other attacks on Global🌎 HTTP(S) LB🏋️‍♀️

Cloud Security Scanner — (Global) Free but limited GAE app vulnerability scanner with “very low false positive rates”

• “After you set up a scan, Cloud☁️ Security🔒 Scanner automatically crawls your application, following all links🔗 within the scope of your starting URLs, and attempts to exercise as many user inputs and event handlers as possible.”
• Can identify:

Cloud Data Loss Prevention API (DLP) — (Global) Finds and optionally redacts sensitive info is unstructured data streams

• Helps you minimize what you collect, expose, or copy to other systems
• 50+ sensitive data detectors, including credit card numbers, names, social security🔒 numbers, passport numbers, driver’s license numbers (US and some other jurisdictions), phone numbers, and other personally identifiable information (PII)
• Data can be sent directly, or API can be pointed at GCS, BQ, or Cloud☁️ DataStore
• Can scan both text and images🖼
• Pay for amount of data processed (per GB) -and gets cheaper when large volume

Event Threat Detection (ETD) — (Global) Automatically scans your Stackdriver logs for suspicious activity

• Uses industry-leading threat intelligence, including Google Safe Browsing
• Can export parsed logs to BigQuery for forensic analysis
• Integrates with SIEMs like Google’s Cloud☁️ SCC or via Cloud Pub/Sub
• No charge for ETD, but charged for its usage of other GCP☁️ services (like SD Logging)

Cloud Security Command Center (SCC) — (Global)

• “Comprehensive security🔒 management and data risk platform for GCP☁️”
• Security🔒 Information and Event Management (SIEM) software
• “Helps you prevent, detect & respond to threats from a single pane of glass”
• Use: Security🔒 Marks” (aka “marks”) to group, track, and manage resources
• Integrate ETD, Cloud☁️ Scanner, DLP, & many external security🔒 finding sources
• Can alert 🔔 to humans & systems; can export data to external SIEM
• Free! But charged for services used (e.g. DLP API, if configured)
• Could also be charged for excessive uploads of external findings

Cloud Key Management Services (KMS) — (Regional, Multi-Regional, Global) Low-latency service to manage and use cryptographic keys🔑

• Supports symmetric (e.g. AES) and asymmetric (e.g. RSA, EC) algorithms
• Move secrets out of code (and the like) and into the environment, in a secure way
• Integrated with IAM & Cloud☁️ Audit Logging to authorize & track key🔑 usage
• Rotate keys🔑 used for new encryption 🔐 either automatically or on demand
• Key🔑 deletion has 24-hour delay, “to prevent accidental or malicious data loss”
• Pay for active key🔑 versions stored over time
• Pay for key🔑 use operations (i.e. encrypt/decrypt; admin operation are free)

Cloud Hardware Security Module (HSM) — (Regional, Multi-Regional, Global) Cloud KMS keys🔑 managed by FIPS 140–2 Level 3 certified HSMs

Operations and Management

• Device hosts encryption 🔐 keys🔑 and performs cryptographic operations
• Enables you to meet compliance that mandates hardware environment
• Fully integrated with Cloud☁️ KMS
• Priced like Cloud KMS: Active key🔑 versions stored & key🔑 operations

Google Stackdriver — (Global) Family of services for monitoring, logging & diagnosing apps on GCP/AWS/hybrid

• Service integrations add lots of value — among Stackdriver and with GCP☁️
• Simple usage-based pricing

Stackdriver Monitoring — (Global) Gives visibility into perf, uptime, & overall health of Cloud☁️ apps (based on collectd)

Stackdriver Logging — (Global) Store, search🔎, analyze, monitor, and alert 🔔 on log data & events (based on Fluentd)

Stackdriver Error Reporting — (Global) Counts, analyzes, aggregates, & tracks crashes in helpful centralized interface

• Smartly aggregates errors into meaningful groups tailored to language/framework
• Instantly alerts when a new app error cannot be grouped with existing ones
• Link🔗 directly from notifications to error details:
• Exception stack trace parser know Java☕️, Python🐍, JavaScript, Ruby💎,C#,PHP, & Go🟢
• Jump from stack frames to source to start debugging
• No direct charge; pay for source data in Stackdriver Logging

Stackdriver Trace — (Global) Tracks and displays call tree 🌳 & timings across distributed systems, to debug perf

Stackdriver Debugger — (Global) Grabs program state (callstack, variables, expressions) in live deploys, low impact

Stackdriver Profiler — (Global) Continuous CPU and memory profiling to improve perf & reduce cost

Cloud Deployment Manager — (Global) Create/manage resources via declarative templates: “Infrastructure as Code”

Cloud Billing API 🧾- (Global) Programmatically manage billing for GCP☁️ projects and get GCP☁️ pricing

Development and APIs

Cloud Source Repositories — (Global) Hosted private Git repositories, with integrations to GCP☁️ and other hosted repos

Cloud Build 🏗 — (Global) Continuously takes source code and builds, tests and deploys it — CI/CD service

• Trigger from Cloud Source Repository (by branch, tag or commit) or zip🤐 in GCS
• Runs many builds in parallel (currently 10 at a time)
• Dockerfile: super-simple build+push — plus scans for package vulnerabilities
• JSON/YAML file: Flexible🧘‍♀️ & Parallel Steps
• Push to GCR & export artifacts to GCS — or anywhere your build steps wrtie
• Maintains build logs and build history
• Pay per minute of build time — but free tier is 120 minutes per day

Container Registry (GCR) 📦- (Regional, Multi-Regional) Fast🏃‍♂️, private Docker🐳 image storage 🗄 (based on GCS) with Docker🐳 V2 Registry API

Cloud Endpoints — (Global) Handles authorization, monitoring, logging, & API keys🔑 for APIs backed by GCP☁️

• Proxy instances are distributed and hook into Cloud Load Balancer 🏋️‍♀️
• Super-fast🏃‍♂️ Extensible Service Proxy (ESP) container based on nginx: <1 ms /call
• Uses JWTs and integrates with Firebase 🔥, AuthO, & Google Auth
• Integrates with Stackdriver Logging and Stackdriver Trace
• Extensible Service Proxy (ESP) can transcode HTTP/JSON to gRPC
• Pay per call to your API

Apigee API Platform — (Global) Full-featured & enterprise-scale API management platform for whole API lifecycle

Test Lab for Android — (Global) Cloud☁️ infrastructure for running 🏃‍♂️ test matrix across variety of real Android devices

• Production-grade devices flashed with Android version and locale you specify
• Can also run Espresso and UI Automator 2.0 instrumentation tests
• Firebase Spark and Flame plans have daily allotment of physical and virtual tests
• Blaze (PAYG) plan charges per device-hour-much less for virtual devices

“Well, we all shine☀️ on… Like the moon🌙 and the stars🌟 and the sun🌞 “

Thanks -

-MCS

Originally published at https://sqlsquirrels.com on November 20, 2020.