“Cause I don’t want to come back… Down from this Cloud ☁️”
Part II of a Cloud☁️ Journey
Hi All –
Happy Global 🌎 Cat 😺 Day!
Last week, we started our Cloud☁️ journey exploring Google Cloud☁️ to help us better understand the core services and the full value proposition that GCP can offer.
It has been said “For modern enterprise, that Cloud☁️ is the closest thing to magic🎩🐰 that we have.” Cloud☁️ enables companies large🏢 and small🏠 to be more agile and nimble. It also empowers employees of these companies🏭 to focus on being more creative and innovative and not being bogged down in the minutiae and rigors of managing IT infrastructure. In addition, customers of these companies benefit from an overall better Customer experience as application more available and scalable⚖️.
As you might know, “Google’s mission is and has been to organize the world’s🌎 information and make it universally accessible and useful and as result playing a meaningful role in the daily lives of billions of people” Google has been able to hold true to this mission statement through unprecedented success from products and platforms like Search🔎, Maps🗺, Gmail📧, Android📱, Google Play, Chrome and YouTube📺. Google continues to strive for the same kind of success with their Cloud Computing☁️ offering with GCP.
This week we continued our journey with GCP and helping us through the Google Cloud☁️ Infrastructure Essentials (Essential Google Cloud☁️ Infrastructure: Foundation & Essential Google Cloud☁️ Infrastructure: Core Services) through a combination of lectures and qwiklabs were esteemed Googlers Phillip Maier who seems to have had more cameos in the Google Cloud☁️ videos than Stan Lee has made in all of the Marvel Movies and the very inspirational Mylene Biddle who exemplifies transformation both in the digital and real world.
Phillip and Mylene begin the course discussing Google Cloud☁️ which is a much larger ecosystem than just GCP. This ecosystem consists of open source software providers, partners, developers, third party software and other Cloud☁️ providers.
GCP uses a state-of-the-art software defined networking and distributed systems technologies to host and deliver services around the world🌎. GCP offers over 90 products and Services that continues to expand. GCP spans from infrastructure as a service or IaaS to software as a service SaaS.
Next, Philip presents excellent analogy comparing IT infrastructure to one of a city’s 🏙 infrastructure. “Infrastructure is the basic underlying framework of fundamental facilities and systems such as transport, communications, power🔌, water🚰, fuel ⛽️ and other essential services. The people 👨👩👧👦 in the city 🏙 are like users 👥, and the cars 🚙🚗 and bikes 🚴♀️🚴♂️ buildings🏬 in the city 🏙 are like applications. Everything that goes into creating and supporting those applications for the users is the infrastructure. “
GCP offers wide range of compute services including:
· Compute Engine — (IaaS) run virtual machines on demand
· Google Kubernetes☸️ Engine (IaaS/PaaS) — run containerized applications on a Cloud☁️ environment that Google manages under your administrative control.
· App Engine (PaaS) is fully managed platform as a service framework. Run code in the Cloud☁️ without having to worry about infrastructure.
· CloudFunctions (Serverless) It executes your code in response to events, whether those events occur once a day or many Time ⏳s
There are Four ways to interact with GCP
1. Google Cloud☁️ Platform Console or GCP Console
2. CloudShell and the Cloud SDK
3. API
4. Cloud☁️ Mobile 📱 App
Cloud☁️ shell provides the following:
· Temporary Compute Engine VM
· Command-line access to the instance via a browser
· 5 GB of persistent disk storage 🗄 ($HOME dir)
· Pre-installed Cloud☁️ SDK and other tools 🛠
· gcloud: for working with Compute Engine and many Google Cloud☁️ services
· gsutil: for working with Cloud Storage 🗄
· kubectl: for working with Google Container Engine and Kubernetes☸️
· bq: for working with BigQuery
· Language support for Java☕️, Go, Python🐍, Node.js, PHP, and Ruby♦️
· Web🕸 preview functionality
· Built-in authorization for access to resources and instances
“Virtual insanity is what we’re living in”
Virtual Networks
GCP uses a software defined network that is built on a Global 🌎 fiber infrastructure. This infrastructure makes GCP, one of the world’s 🌎 largest and fastest 🏃♂️networks.
Virtual Private Cloud☁️
Virtual Private Cloud☁️ (VPC) provides networking functionality to Compute Engine virtual machine (VM) instances, Google Kubernetes Engine (GKE)☸️ containers, and the App Engine flexible🧘♀️ environment. VPC provides networking for your Cloud-based☁️ services that is Global 🌎, scalable⚖️, and flexible🧘♀️.
VPC is a comprehensive set of Google managed networking objects:
· Projects are used to encompass the Network Service as well as all other service in GCP
· Networks come in three different flavors 🍨.
o Default
o Automotive
o Custom mode.
· Subnetworks allow for division or segregation of the environment.
· Regions and zones (GCP DC) provide continues data protection and high availability.
· IP addresses provided internal and external use
· Virtual machines — instances from a networking perspective.
· Routes and Firewall🔥rules allow or deny connections to or from VMs based specified configuration
Projects, Networks, and Subnetworks
A Project is the key organizer of infrastructure resources.
· Associates objects and services with billing 🧾
· Contains networks (up to 5) that can be shared/peered
Networks are Global 🌎 are spans all available regions.
· Has no IP address range
· Contains subnetworks
· Has three different options.
o Default
§ Every Project
§ One subnet per region
§ Default firewall rules🔥
o Automotive
§ Default network
§ One subnet per region
§ Regional IP allocation
§ Fixed /20 subnetwork per region
§ Expandable up to /16
o Custom mode
§ No default subnets created
§ Full control of IP ranges
§ Regional IP allocation
§ Expandable to any RFC 1918 size
VMs despite different locations geographically 🌎 take advantage of Google’s Global 🌎 fiber network. VMs appear as though they’re sitting in the same rack when it comes to a network configuration protocol.
· VMs can be on the same subnet but in different zones
· A single firewall rule 🔥 can apply to both VMs
Subnet is a ranges of IP addresses.
· Every subnet has 4 reserved IP addresses in its primary IP Range.
· Subnets can be expanded without re-creating instances or any down Time ⏳
o Cannot overlap with other subnets
o Must be inside the RFC 1918 address spaces
o Can expand but not shrink
o Auto mode can be expanded from /20 to /16
o Avoid large subnets (don’t scale⚖️ beyond what is actually needed)
IP addresses
VMs can have internal and external IP addresses
You also can assign a range of IP addresses as aliases to a VM’s network interface using IP range
Internal IP
· Allocated from a subnet range to VMs by DHCP
· DHCP lease is removed every 24 hours
· VM name + IP is registered with network-scoped DNS
External IP
· Assigned from pool (ephemeral)
· Reserved (static)
· VMs don’t know external IP
· VMs are mapped to the internal IP
Mapping 🗺 IP addresses
DNS resolution for internal addresses
· Each instance has a hostname that can be resolved to an internal IP address:
o The hostname is the same as the instance name
o FQDN is [hostname]. [zone].c.[project-id].internal
· Name resolution is handled by internal DNS resolver
· Configured for use on instance via DHCP
· Provides answer for internal and external addresses
DNS resolution for external address
· Instances with external IP addresses can allow connections from hosts outside the project
o Users connect directly using external IP address
o Admins can also publish public DNS records pointing to the instance
§ Public DNS records are not published automatically
· DNS records for external addresses can be published using existing DNS servers (outside of GCP)
· DNS zones can be hosted using Cloud DNS.
Host DNS zones using Cloud DNS
· Google DNS service
· Translate domain name into IP address
· Low latency
· High availability (100% uptime SLA)
· Create and update millions of DNS records
Routes and firewall rules 🔥
Every network has:
· Routes that let instances in a network send traffic🚦 directly to each other.
· A default route that directs to destinations that are outside the network.
Routes map🗺 traffic🚦 to destination networks
· Apply to traffic🚦 egressing a VM
· Forward traffic🚦 to most specific route
· Are created when a subnet is created
· Enable VMs on same network to communicate
· Destination is in CIDR notation
· Traffic🚦 is delivered only if it also matches a firewall rule 🔥
Firewall rules🔥 protect your VM instances from unapproved connections
· VPC network functions as a distributed firewall. 🔥
· Firewall rules🔥 are applied to the network as whole
· Connections are allowed or denied at the instance level.
· Firewall rules🔥 are stateful
· Implied deny all ingress and allow all egress
Create Network
gCloud compute networks create privatenet — subnet-mode=custom
gCloud compute networks subnets create privatesubnet-us — network=privatenet — region=us-central1 — range=172.16.0.0/24
gCloud compute networks subnets create privatesubnet-eu — network=privatenet — region=europe-west1 — range=172.20.0.0/20
gCloud compute networks list
gCloud compute networks subnets list — sort-by=NETWORK
Create firewall Rules 🔥
gCloud compute firewall-rules create privatenet-allow-icmp-ssh-rdp — direction=INGRESS — priority=1000 — network=privatenet — action=ALLOW — rules=icmp,tcp:22,tcp:3389 — source-ranges=0.0.0.0/0
Common network designs
· Increased availability with multiple zones
o A regional managed instance group contains instances from multiple zones across the same region, which provides increased availability.
· Globalization 🌎 with multiple regions
o Putting resource is in different regions, provides an even higher degree of failure independence by spreading resources across different failure domains
· Cloud NAT provides internet access to private instances
o Cloud NAT is Google’s Mesh Network address translation service. Provision application instances without public IP addresses, while also allowing them to access the Internet in a controlled and efficient manner.
· Private Google Access to Google APIs and services
o Private Google access to allow VM instances that only have internal IP addresses to reach the external IP addresses of Google APIs and services.
“Know you’re nobody’s fool… So welcome to the machine”
Compute Engine (IaaS)
Predefined or custom Machines types:
· vCPUs (cores and Memory (RAM)
· Persistent disks: HDD, SDD, and Local SSD
· Networking
· Linux or Windows
Compute
Several machine types
· Network throughput scales⚖️ 2 Gbps per vCPU (small exceptions)
· Theoretical max of 32 Gbps with 16 vCPU or 100 Gbps with T4 or V100 GPUs
A vCPU is equal to 1 hardware hyper-thread.
Storage 🗄
Disks
· Standard, SSD, or Local SSD
· Standard and SSD PDs scale⚖️ in performance for each GB of space a;;pcated
Resize disks or migrate instances with no downTime ⏳
Local SSD have even higher throughput and lower latency than SSD persistent disks because there are attached to the physical hardware. However, the data that you store local s SSDs persists only until you stop 🛑 or delete the instance.
Networking
Robust network features:
· Default, custom networks
· Inbound/outbound firewall rules🔥
o IP based
o Instance/group tags
· Regional HTTPS load balancing
· Network load balancing
o Does not require pre-warming
· Global 🌎 and multi-regional subnetworks
VM access
Linux🐧 SSH (requires firewall to allow tcp:22)
· SSH from Console CloudShell via Cloud SDK, computer
Windows RDP (requires firewall to allow tcp:3389)
· RDP clients, PowerShell terminal
VM Lifecycle
Compute Engine offers live migration to keep your virtual machine instances running even when a host system event, such as a software or hardware update, occurs. Live migration keeps your instances running during:
· Regular infrastructure maintenance and upgrades.
· Network and power grid maintenance in the data centers.
· Failed hardware such as memory, CPU, network interface cards, disks, power, and so on. This is done on a best-effort basis; if a hardware fails completely or otherwise prevents live migration, the VM crashes and restarts automatically and a hostError is logged.
· Host OS and BIOS upgrades.
· Security-related updates, with the need to respond quickly.
· System configuration changes, including changing the size of the host root partition, for storage 🗄 of the host image and packages.
Compute Options
Machine Types
Predefined machine types Ratio of GB of memory per VCPU
· Standard machine types
· High-memory machine types
· High-CPU machine types
· Memory-optimized machine types
· Compute-optimized machine types
· Shared core machine types
Custom machine types:
· You specify the amount of memory and number of VCPUs
Special compute configurations
Preemptible (ideal for running batch processing jobs)
· Lower price for interruptible service (up to 80%)
· VM might be terminated at any time ⏳
o No charge if terminated in the first 10 minutes
o 24 hours max
o 30-second terminate warning, but not guaranteed
§ Time⏳ for a shutdown script
· No live migrate; no auto restart
· You can request that CPU quota for a region be split between regular and preemption
o Default: preemptible VMs count against region CPU quota
Sole-tenant nodes -physically isolate workloads (Ideal for workloads that require physical isolation)
· Sole-tenant node is a physical compute engine server that is dedicated to hosting VM Instances
· if you have existing operating system licenses, you can bring them to compute engine using sole tenant notes while minimizing physical core usage with the in-place restart feature
Shielded VMs🛡 offer verifiable integrity
· Secure🔒 Boot
· Virtual trusted platform module (vTPM)
· Integrity Monitoring 🎛
Images
· Boot loader
· Operating system
· File System Structure 🗂
· Software
· Customizations
Disk options
Boot disk
· VM comes with a single root persistent disk
· Image is loaded onto root disk during first boot:
o Bootable: you can attach to a VM and boot from it
o Durable: can survive VM terminate
· Some OS images are customized for Compute Engine
· Can survive VM deletion if “Delete boot disk when instance is deleted” is disabled.
Persistent disks
· Network storage 🗄 appearing as a block device
o Attached to a VM through the network interface
o Durable storage 🗄: can survive VM terminate
o Bootable: you can attach to a VM and boot from it
o Snapshots: incremental backups
o Performance: Scales⚖️ with Size
· Features
o HDD or SSD
o Disk resizing
o Attached in read-only mode to multiple VMs
o Encryption keys 🔑
Local SSD disks are physically attached to a VM
· More IOPS, lower latency and higher throughput
· 375-GB up to 8 disks (3TB)
· Data Survives a reset but not VM Stop 🛑 or terminate
· VM Specific cannot be reattached to a different VM
RAM disk
· tmps
· Faster than local disk, slower memory
o Use when your application expects a file system structure and cannot directly store its data in memory
o Fast scratch disk, or fast cache
· Very volatile: erase on stop 🛑 or reset
· May require larger machine is RAM sized for application
· Consider using persistent disk to back up RAM disk data
Common Compute Engine actions
· Metadata and scripts (Every VM instance store its metadata on a metadata server)
· Move an instance to a new zone
o Automated process (moving within region)
§ gcloud compute instance move
§ updates reference to VM; not automatic
o Manual process (moving between regions):
§ Snapshots all persistent disks
§ Create new persistent disk in destination zone disks restored from snapshots
§ Create new VMs in the destination zone and attach new persistent
§ Assign static IP to new VM
§ Update references to VM
§ Delete the snapshots, original disks and original VM
· Snapshot: Back up critical data
· Snapshot: Migrate data between zones
· Snapshot: Transfer to SSD to improve performance
· Persistent disk snapshots
o Snapshot is not available for local SSD
o Creates an incremental backup to Cloud Storage 🗄
§ Not visible in your buckers; managed by the snapshot service
§ Consider cron jobs for periodic incremental backup
o Snapshots can be restored to a new persistent disk
§ New disk can be in another region or zone in the same project
§ Basis of VM migration: “moving” a VM to a new zone
· Snapshot doesn’t back up VM metadata, tags, etc.
§ Resize persistent disk
You can grow disks, but never shrink them!
“If you had my love and I gave you all my trust. Would you comfort me?”
Identity Access Management (IAM)
IAM is a sophisticated system built on top of email, like address names, job type roles in granular permissions.
Who, Which, What
It is a way of identifying who can do what on which resource the who could be a person, group or application. The what refers to specific privileges or actions and the resource could be any DCP service.
- Google Cloud☁️ Platform Resource is or organized hierarchically
- If you change the resource hierarchy, the policy hierarchy also changes.
- A best practice to follow the principle of least privilege.
· Organization node is the root note in this hierarchy. Represents your company.
· Folders 📂are the Children of the organization. Folder📂 could represent a department Cloud☁️
· Projects are the Children of the Folders📂. Projects provide a trust boundary for a company
· Resources are the Children of projects. Each Resource has exactly one parent Cloud☁️.
Organization
· An organization node is a root node for Google Cloud☁️ resources
· Organization roles:
o Organization Admin: Control over all Cloud☁️ resources: useful for auditing
o Project Creator: Controls project creation: Control project creation: control over who can create projects
· Creating and managing Organization when a G Suite or IAM account creates a GCP Project. There are two roles assigned to t users or groups:
o Super administrator:
§ Assign the Organization admin role to some users
§ Be the point of contact in case of recovery issues
§ Control the lifecycle 🔄of the G Suite of Cloud☁️ Identity account and Organization
o Organization admin:
§ Define IAM policies
§ Determine the structure of the resource hierarchy
§ Delegate responsibility over critical components such as Network, billing, and Resource Hierarchy through IAM roles
Folders 📂
· Additional grouping mechanism and isolation boundaries between projects:
o Different legal entities
o Departments
o Teams
· Folders 📂 allow delegation of administration rights.
Roles
· There are three types of roles in GCP:
1. Primitive roles apply across all GCP services in a project
o Primitive roles offer fixed, coarse-grained levels of access
§ Owner — Full privileges
§ Editor — Deploy, modify & configure
§ Viewer 👓 — Read-only access
§ *Billing Administrator — Manage Billing, Add Administrators
2. Predefined roles apply to a particular service in a project
o Predefined roles offer more fine-grained permissions on a particular service
o Example: Compute Engine IAM roles:
§ Compute Admin — Full control of Compute Engine
§ Network Admin — Create, modify, delete Network Resources (except FW rules and SSL Certs)
§ Storage 🗄 Admin– Create, modify, delete disks, Images, and Snapshots
3. Custom roles define a precise set of permissions
Members
Defined the “who” part of who can do what on which resource.
There are five different types of members:
1. Google account represents a developer, an administrator or any other person who interacts with GCP. Any email address can be associated with a Google account
2. Service account is an account that belongs to your application instead of to an individual and user.
3. Google group is unnamed collection of Google accounts and service accounts.
4. G Suite domains represent your organization’s Internet domain name
5. Cloud Identity domains manage users and groups using the Google Admin console, but you do not pay for or received G Suite collaboration products
Google Cloud Directory Sync ↔️ — synchronizes ↔️ users and groups from your existing active directory or LDAP system with the users and groups in your Cloud identity domain. Synchronization ↔️ is one way only
Single Sign-om (SSO)
· Use Cloud Identity to configure SAML, SSO
· IF SAML2 isn’t supported, use a third-party solution
Service Accounts
· Provide an identity for carrying out server-to-server interactions
o Programs running within Compute Engine instances can automatically acquire access tokens with credentials
o Tokens are used to access any service API or services in your project granted access to a service account
o Service accounts are convenient when you’re not accessing user data
· Service accounts are identified by an email address
o 123845678986-computer@project.gserviceaccount.com
o Three types of Service accounts:
o User-created (custom)
o Built-in
§ Compute Engine and App Engine default service accounts
o Google APIs Service account
§ Runs internal Google processes on your behalf
· Default Compute Engine Service account
o Automatically created per project with auto-generated name and email address:
o Name has -compute suffix
o 39xxxx0965-computer@developer.gserviceaccount.com
o Automatically added as a project Editor
o By default, enabled on all instances created using glcoud or GCP console
· Service account permissions
o Default service accounts: primitive and predefined roles
o User-created service accounts: predefined roles
o Roles for Service accounts can be assigned to groups or users
Authorization is the process of determining what permissions and authenticated identity has on a set of specified resource(s)
Scopes are used to determine whether unauthenticated identity is authorized.
Customizing Scopes for a VM
· Scopes can be changed after an instance is created
· For user-created service accounts, use IAM roles instead.
IAM Best Practices
1. Leverage and understand the resource hierarchy
o Use Projects to group resources that share the same trust boundary
o Check the policy granted on each resource and make sure you understand inheritance
o Use “Principles of Least Privilege” when granting roles
o Audit policies in Cloud☁️ audit logs: setiampolicy
o Audit membership of groups used in policies
2. Grant roles to Google groups instead of individuals
o Update group membership instead of changing IAM Policy
o Audit membership of groups used in policies
o Control the ownership of the Google group used in IAM policies
3. Service accounts
o Be very careful granting serviceAccountUser role
o When you create a service account, give it a display name that clearly identifies its purpose
o Establish a naming convention for service accounts
o Establish key rotation policies and methods
o Audit with serviceAccount.keys.list() method
Cloud Identity-Aware Proxy (Cloud IAP)
Enforce access control policies for application and resources:
· Identity-based access control
· Central authorization layer for applications accessed by HTTPS
IAM policy is applied after authentication
“Never gonna give you up… Never gonna say goodbye.”
Storage 🗄 and Database Services 🛢
Cloud Storage 🗄 (Object Storage 🗄) — It allows worldwide🌎 storage 🗄 and retrieval of any amount of data at any Time ⏳.
· Scalable⚖️ to exabytes
· Time⏳ to first byte in milliseconds
· Very high availability across all storage 🗄 classes
· Single API across storage 🗄 classes
Use Cases:
o Website content
o Storing data for archiving and disaster recovery
o Distributing large data objects to users via direct download
Cloud Storage 🗄 has four storage 🗄 classes:
1. Regional storage 🗄 enables you to store data at lower cost with the tradeoff of data being stored in a specific regional location.
2. Multi regional storage 🗄 is geo redundant, Cloud Storage 🗄 stores, your data redundantly and at least two geographic locations separated by at least 100 miles within the multi-regional location of the bucket 🗑.
3. Near line storage 🗄 is a low cost, highly durable storage 🗄 service for storing infrequently accessed data.
4. Cold Line storage 🗄 is a very low cost, highly durable storage 🗄 service for data, archival, online backup and disaster recovery. Data is available within milliseconds, not hours or days.
Buckets 🗑
o Naming requirements
o Cannot be nested
o Regional bucket 🗑 & Multi-Regional cannot be changed
o Objects can be moved from bucket 🗑 to bucket 🗑
Objects
o Inherit storage 🗄 class of bucket 🗑 when created
o No minimum size: unlimited storage 🗄
Access
o gsutil command
o (RESTful) JSON API or XML API
Access control lists (ACLs)
For some applications, it is easier and more efficient to grant limited Time ⏳ access tokens that can be used by any user instead of using account-based authentication for controlling resource access.
Signed URLs
“Valet Key” access to Buckets 🗑 and objective via ticket:
o Ticket is a cryptographically signed URL
o Time ⏳-limited
o Operations specified in ticket: HTTP GET, PUT DELETE (not POST)
o Any user with URL can invoke permitted operations
Cloud Storage 🗄 Features
· Customer-supplied encryption key (CSEK)
o Use your own key instead of Google-managed keys 🔑
· Object Lifecycle 🔄Management
o Automatically delete or archive objects
· Object Versioning
o Maintain multiple versions of objects
§ Objects are immutable
§ Object Versioning:
· Maintain a history of modification of objects
· List archived versions of an object, restore an object to an older state, or delete a version
· Directory synchronization ↔️
o Synchronizes a VM directory with a bucket 🗑
· Object change notification
· Data import
· Strong 💪 consistency
Object Lifecycle 🔄Management policies specify actions to be performed on objects that meet certain rules
· Examples:
o Downgrade storage 🗄 class on objects older than a year.
o Delete objects created before a specific date.
o Keep only the 3 most recent versions of an object
· Object inspection occurs in asynchronous batches
· Changes can take 24 hours to apply
Object change notification can be used to notify an application when an object is updated or added to a bucket 🗑
Recommended: Cloud Pub/Sub Notifications for Cloud Storage 🗄
Data import services
· Transfer Appliance: Rack, capture and then ship your data to GCP
· Storage 🗄 Transfer Service: Import online data (another bucket 🗑, S3 bucket 🗑, Web Service)
· Offline Media Import: Third-party provider uploads the data from physical media
Cloud Storage 🗄 provides Strong 💪 global consistency
Cloud SQL is a fully managed database 🛢 service (MySQL or PostgreSQL)
· Patches and updates automatically applied
· You administer MySQL users
· Cloud SQL supports many clients
o gCloud sql
o App Engine, G Suite scripts
o Applications and tools 🛠
§ SQL Workbench Toad
§ External applications using standard MySQL drivers
- Cloud SQL delivers high performance and scalability⚖️ with up to 30 TBs of storage 🗄 capacity, 40,000 IOPS and 416 GB of RAM
- Replica service that can replicate data between multiple zones
- Cloud SQL also provides automated and on demand backups.
- Cloud SQL scales⚖️ up (require a restart)
- Cloud SQL scales⚖️ out using read replicas.
Cloud☁️ Spanner is a service built for the Cloud☁️ specifically to combine the benefits of relational database 🛢 structure with non-relational horizontal scale⚖️
Data replication is synchronized across zones using Google’s global fiber network
· Scale⚖️ to petabytes
· Strong 💪 consistency
· High Availability
· Used for financial and inventory applications
· Monthly upTime ⏳
o Multi-regional:99.999%
o Regional: 99.99%
Cloud Firestore is a fast, fully managed serverless Cloud☁️ native NoSQL document database 🛢 that simplifies storing, syncing ↔️ and querying data for your Mobile 📱 Web and ioT applications a global scale⚖️.
· Simplifies storing, syncing ↔️, and querying data
· Mobile 📱, web🕸, and IoT apps at global scale⚖️
· Live synchronization ↔️ and offline support
· Security🔒 features
· ACID transactions
· Multi-region replication
· Powerful query engine
Datastore mode (new server projects):
· Compatible with Datastore applications
· Strong 💪 consistency
· No entity group limits
Native mode (new Mobile 📱 and web🕸 apps):
· Strongly 💪 consistent storage 🗄 layer
· Collection and document 📄 data model
· Real-Time updates
· Mobile 📱 and Web🕸 Client libraries📚
Cloud Bigtable (Wide Column DB) is a fully managed, no SQL database 🛢 with petabytes scale⚖️ and very low latency.
· Petabyte-scale⚖️
· Consistent sub-10ms latency
· Seamless scalability⚖️ for throughput
· Learns and adjusts to access patterns
· Ideal for Ad Tech, FinTech, and IoT
· Storage 🗄 engine for ML applications
· Easy integration with open source big data tools 🛠
Cloud MemoryStore is a fully managed Redis Service built on scalable⚖️, Secure🔒 and highly available infrastructure managed by Google Applications.
· In-memory data store service
· Focus on building great apps
· High availability, failover, patching and Monitoring 🎛
· Sub-millisecond latency
· Instances up to 300 GB
· Network throughput of 12 Gbps
· “Easy Lift-and-Shift”
Resource Management lets you hierarchically manage resources
· Resources can be categorized by Project, Folder📂, and Organization
· Resources are global🌎, regional, or zonal
· Resource belongs to only one project
· Resources inherent policies from their parents
· Resource consumption is measured in quantities like rate of use or Time ⏳
· Policies contain a set of roles, and members and policies are set on
· Policy is less restrictive; it overrides the more restrictive resource policy.
· Organization node is root node for GCP resources
· Organization contains all billing accounts.
· Project accumulates the consumption of all its resources
o Track resource and quota usage
o Enable billing
o Manage permissions and credentials
o Enable services and APIs
· Project use 3 identifying attributes
o Project Name
o Project Number
o Project ID
Quotas
All resources are subject to project quotas or limits
Examples:
o Total resources you can create per project: 5 VPC networks/project
o Rate you make API requests in a project: 5 admin actions/second (Cloud☁️ Spanner)
o Total resources you can create per region: 24 CPUs region/project
Increase: Quotas page in GCP Console or a support ticket
Your use of GCP expands over time⏳, your quotas may increase accordingly.
Project Quotas:
o Prevent runaway consumption in case of an error or malicious attack
o Prevent billing spikes or surprises
o Forces sizing consideration and periodic review
Labels and names
Labels are a utility for organizing GCP resources
· Attached to resources: VM, disk, snapshot, image
o GCP Console, gCloud, or API
· Example uses of labels:
o Inventory
o Filter resources
o In scripts
§ Help analyze costs
§ Run bulk operations
§
Comparing labels and tags
· Labels are a way to organize resources across GCP
o Disks, image, snapshots
· User-defined strings.in key-value format
· Propagated through billing
· Tags are applied to instances only
· User-defined strings
· Tags are primarily used for networking (applying firewall rules🔥)
Billing
· Set a budget lets you track how you spend
· Set a budget alerts 🔔send alerts 🔔and emails📧to Billing Admin
· Use Cloud☁️ pubs up notifications to programmatically receive spend updates about this budget.
· Optimize your GCP spend by using labels
· Visualize GCP spend with Data Studio
Its recommended labeling all your resources and exporting billing data to Big Query to Analyze spend.
“Every single day. Every word you say… Every game you play. Every night you stay. I’ll be watching you”
Resource Monitoring 🎛
Stackdriver
· Integrated Monitoring 🎛, Logging, Error Reporting, Tracing and Debugging
· Manages across platforms
o GCP and AWS
o Dynamic discovery of GCP with smart defaults
o Open-source agents and integrations
· Access to powerful data and analytics tools 🛠
· Collaboration with third-party software
Monitoring 🎛 is important to Google because it is at the base of Site Reliability Engineering (SRE).
· Dynamic config and intelligent defaults
· Platform, system, and application metrics
o Ingests data: Metrics, events, metadata
o Generates insights through dashboards, charts, alerts
· Uptime /health checks
· Dashboards
· Alerts
Workspace is the root entity that hold Monitoring 🎛 and configuration information
· “Single pane of glass 🍸”
o Determine your Monitoring 🎛 needs up front
o Consider using separate Workspace for data and control isolation
To access an AWS account, you must configure a project in GCP to hold the AWS connector because workspaces can monitor all of your DCP projects in a single place.
Stack Driver Monitoring 🎛 allows you to create custom dashboards that contain charts of the metrics that you want a monitor.
Uptime checks test the availability of your public services
Stock driver Monitoring 🎛 can access some metrics without the Monitoring 🎛 agent, including CPU utilization, some disk traffic metrics, network traffic and up Time⏳ information.
Stack Driver Logging provides logging, error, reporting, tracing and debugging.
· Platform, systems, and application logs
o API to write to logs
o 30-day retention
· Log search/view/filter
· Log-based metrics
· Monitoring 🎛 alerts 🔔can be set on log events
· Data can be exported to Cloud Storage 🗄, BigQuery, and Cloud Pub/Sub
· Analyze logs in BigQuery and visualize in Data Studio
Stack driver Error Reporting counts, analyzes and aggravates the errors in your running Cloud☁️ services
· Error notifications
· Error dashboard
· Go, Java☕️, .NET, Node.js, PHP, Python 🐍, and Ruby♦️
Stack Driver Tracing is a distributed tracing system that collects Layton. See data from your applications and displays it in the GCP console.
· Displays data in near real-Time ⏳
· Latency reporting
· Per-URL latency sampling
· Collects latency data
o App Engine
o Google HTTP(S) load balancers🏋️♀️
o Applications instrumented with the Stackdriver Trace SDKs
Debugging
· Inspect an application without stopping🛑 it or slowing it down significantly
· Debug snapshots:
o Capture call stack and local variables of a running application
· Debug logpoints:
o Inject logging into a service without stopping🛑 it
· Java☕️, Python🐍, Go, Node.js and Ruby♦️
“At last the sun is shining, the clouds of blue roll by”
We will continue next week with Part III of this series…Thanks –
–MCS
Originally published at https://sqlsquirrels.com on October 16, 2020.