Google Cloud Fundamentals: Core Infrastructure - Introduction

This article extends my notes from Google Cloud Fundamentals: Core Infrastructure | Coursera. 

Introducing Google Cloud

Pricing and billing Google Cloud offerings can be broadly categorized as:

• compute
• storage
• big data
• machine learning
• application services

...for web, mobile, analytics, and backend solutions.

Cloud computing overview

Cloud computing is a way of using information technology (IT) that has these five equally important traits:

• customers get computing resources that are on-demand and self-service.
• Through a web interface, users get the processing power, storage, and network they need with no need for human intervention
• customers get access to those resources over the internet
• the cloud provider has a big pool of those resources and allocates them to users out of that pool.
• That allows the provider to buy in bulk and pass the savings on to the customers. Customers don't have to know or care about the exact physical location of those resources.
• resources are elastic–which means they’re flexible, so customers can be
• If they need more resources they can get more, and quickly. If they need less, they can scale back
• customers pay only for what they use, or reserve as they go
• If they stop using resources, they stop paying

A bit of history

• first wave of the trend towards cloud computing: colocation. Colocation gave users the financial efficiency of renting physical space, instead of investing in (private) data center real estate
• Second wave: Virtualized data centers of today. Servers, CPUs, disks, load balancers, and so on—are virtual devices. With virtualization, enterprises still maintain the infrastructure; but it also remains a user-controlled and user-configured environment. 
• Third-wave cloud: Google switched to a container-based architecture—a fully automated, elastic  that consists of a combination of automated services and scalable data. Services automatically provision and configure the infrastructure used to run applications. 

Google believes that, in the future, every company—regardless of size or industry— will differentiate itself from its competitors through technology. Increasingly, that technology will be in the form of software. Great software is based on high-quality data. This means that every company is, or will eventually become, a data company.

IaaS and PaaS

In virtualized data centres:

• IaaS - Infrastructure as a Service
• provides
• Raw compute e.g. Compute Engine (which is like AWS EC2)
• Storage
• Network capabilities
• organized virtually into resources that are similar to physical data centers
• customers pay for the resources they allocate ahead of time
• PaaS - Platform as a Service
• lets us bind our application code to libraries that give access to the infrastructure our application needs
• This allows more resources to be focused on application logic.
• customers pay for the resources they actually use
• Serverless 
• yet another step in the evolution of cloud computing
• allows developers to concentrate on their code, rather than on server configuration, by eliminating the need for any infrastructure management.
• Google serverless: 
• Cloud Functions - manages event driven code as a pay as you go service
• Cloud Run - allows customers to deploy their containerized microservices based application, in a fully managed environment.
• SaaS - Software as a Service
• SaaS applications aren't installed on your local computer. Instead, they run in the cloud as a service and are consumed directly over the internet by end users
• Google SaaS applications: Gmail, Docs and Drive

The Google Cloud network

• highest possible throughput
• lowest possible latencies
• 100+ content caching nodes worldwide
• 5 major geographic locations (5 continents)
• multiple regions at each location e.g.  europe-west2 (London)
• regions are composed of zones e.g.  europe-west2-a, europe-west2-b and europe-west2-c 
• resources can be run in different regions and/or zones within the same region
• this brings application closer to the user (lowers the latency) - Google Cloud customer use resources in several regions around the world for this reason
• redundancy (improves the availability)
• improves fault tolerance -  primary benefit to a Google Cloud customer of using resources in several zones within a region
• some GC resources can be run in multi-region e.g. eur3-Europe
• currently: 118 zones in 39 regions

Environmental impact

• GC data centers use 2% of the world's electricity

Google Infrastructure Security

• Hardware Infrastructure Layer
• HW design 
• custom chips, hardware security chips
• secure boot stack - to ensure that servers are booting the correct software stack
• cryptographic signatures over the bios, bootloader, kernel and OS
• Physical security of data centers
• Service Deployment Layer
• Encryption of inter-service communication
• Google services communicate with each other using RPC (Remote Procedure Calls) calls
• all RPC traffic between and inside data centers is encrypted
• User Identity Layer
• User Identity
• 2FA, U2F
• Storage Services Layer
• Encryption at rest
• Internet Communication Layer
• Google Front End (GFE)
• Denial of Service (DoS) protection
• Operational Security Layer
• Intrusion detection
• red team exercises
• Reducing insider risk
• Employee Universal Second Factor (U2F) use
• Software Development practices
• peer reviews
• libraries
• vulnerability rewards

Open source ecosystems

• Tensorflow
• Kubernetes and Google Kubernetes Engine give the ability to mix and match microservices running across different clouds
• Operations Suite

Pricing and billing

• per-second billing for IaaS compute engine etc
• Budgets
• fix limit
• bound to some metric like usage in previous month
• Alerts
• Reports
• Quotas 
• prevent over-consumption
• applied at project level
• Rate quotas
• reset after a specific time
• e.g. GKE allows 3000 calls to its API from each Google Cloud project every 100 seconds; after that time the limit is reset
• Allocation quotas
• governs number of resources in projects
• e.g. each project can have max 15 virtual private cloud networks
• increase on request to Google support team

Resources and Access in the Cloud

Google Cloud resource hierarchy

Functional structure of Google Cloud: 

• Level 4 (top): organization node
• encompasses all the projects, folders, and resources in your organization
• Level 3: folders and subfolders
• Level 2: projects
• Projects can be organized into folders, or even subfolders.
• Level 1 (bottom): resources
• virtual machines, Cloud Storage buckets, tables in BigQuery, or anything else in Google Cloud
• Resources are organized into projects

• This resource hierarchy directly relates to how policies are managed and applied when you use Google Cloud. 
• Policies can be defined at the project, folder, and organization node levels. 
• Some Google Cloud services allow policies to be applied to individual resources, too. 
• Policies are also inherited downward. This means that if you apply a policy to a folder, it will also apply to all of the projects within that folder. 

Projects

• the basis for enabling and using Google Cloud services, like managing APIs, enabling billing, adding and removing collaborators, and enabling other Google services  
• Each project is a separate entity under the organization node, and each resource belongs to exactly one project
• Projects can have different owners and users because they’re billed and managed separately
• Each Google Cloud project has three identifying attributes:
• project ID
• globally unique identifier assigned by Google that can’t be changed after creation (they are immutable)
• can be modified by the customer during creation
• used in different contexts to inform Google Cloud of the exact project to work with
• project name
• user-created
• don’t have to be unique and they can be changed at any time, so they are not immutable
• project number
• assigned by Google Cloud also assigns each project
• unique
• mainly used internally by Google Cloud to keep track of resources

Google Cloud’s Resource Manager tool

• designed to programmatically help you manage projects
• an API that can gather a list of all the projects associated with an account, create new projects, update existing projects, and delete projects
• it can even recover projects that were previously deleted
• can be accessed through the RPC API and the REST API

Folders

• let you assign policies to resources at a level of granularity you choose
• folder can contain projects, other folders, or a combination of both
• You can use folders to group projects under an organization in a hierarchy. 
• For example, your organization might contain multiple departments, each with its own set Google Cloud resources. 
• Folders allow you to group these resources on a per-department basis.
• Folders also give teams the ability to delegate administrative rights so that they can work independently. 
• resources in a folder inherit policies and permissions from that folder
• For example, if you have two different projects that are administered by the same team, you can put policies into a common folder so they have the same permissions. Doing it the other way--putting duplicate copies of those policies on both projects–could be tedious and error-prone. if you needed to change permissions on both resources, you would now have to do that in two places instead of just one.

Organization 

• Topmost resource in the Google Cloud hierarchy
• To use folders, we must have an organization node
• Everything else attached to that account goes under this node, which includes folders, projects, and other resources. 
• There are some special roles associated with this top-level organization node. 
• For example, we can designate an organization policy administrator so that only people with privilege can change policies. 
• We can also assign a project creator role, which is a great way to control who can create projects and, therefore, who can spend money. 
• How a new organization node is created depends on whether our company is also a Google Workspace customer
• If we have a Workspace domain, Google Cloud projects will automatically belong to our organization node. Otherwise, we can use Cloud Identity, Google’s identity, access, application, and endpoint management platform, to generate one. 
• Once created, a new organization node will let anyone in the domain create projects and billing accounts, just as they could before.  We'll also be able to create folders underneath it and put projects into it. 
• Both folders and projects are considered to be “children” of the organization node.

​Identity and Access Management (IAM)

• used by administrators to apply IAM policies that define who can do what and on which resources
• “who” part of an IAM policy
• identity 
• can be:
• Google account
• Google group
• service account
• Cloud Identity domain
• also called a principal. Each principal has its own identifier, usually an email address.
• “can do what” part of an IAM policy is defined by a role
• An IAM role is a collection of permissions.
• When you grant a role to a principal, you grant all the permissions that the role contains.
• For example, to manage virtual machine instances in a project, we must be able to create, delete, start, stop and change virtual machines. So these permissions are grouped into a role to make them easier to understand and easier to manage. 
• When a principal is given a role on a specific element of the resource hierarchy, the resulting policy applies to both the chosen element and all the elements below it in the hierarchy. 
• We can define deny rules that prevent certain principals from using certain permissions, regardless of the roles they're granted. This is because IAM always checks relevant deny policies before checking relevant allow policies. Deny policies, like Allow policies, are inherited through the resource hierarchy. 
• There are three kinds of roles in IAM: 
• Basic
• Broad in scope
• When applied to a Google Cloud project, they affect all resources in that project
• Include:
• Owner - project owners can:
• access a resource
• make changes to a resource
• manage the associated roles and permissions and set up billing
• Editor - Project editors can:
• access a resource
• make changes to a resource
• Viewer - Project viewers:
• can access resources
• can’t make changes
• Billing Administrator
• can control the billing for a project
• can't change the resources in the project
• If several people are working together on a project that contains sensitive data, basic roles are probably too broad
• Predefined
• Assigns permissions that are more specifically tailored to meet the needs of typical job roles
• Specific Google Cloud services offer sets of predefined roles, and they even define where those roles can be applied
• Example: With Compute Engine (a Google Cloud product that offers virtual machines as a service) we can apply specific predefined roles such as instanceAdmin to Compute Engine resources in a given project, a given folder, or an entire organization. This then allows whoever has these roles to perform a specific set of predefined actions e.g.
• compute.instances.delete
• compute.instances.get
• compute.instances.list
• compute.instances.setMachineType
• compute.instances.start
• compute.instances.stop
• Custom
• Finest-grained
• Used when we need to assign a role that has even more specific permissions
• Least-privilege model in which each person in our organization is given the minimal amount of privilege needed to do their job
• Example: we want to define an instanceOperator role to allow some users to stop and start Compute Engine virtual machines, but not reconfigure them:
• compute.instances.get
• compute.instances.list
• compute.instances.start
• compute.instances.stop
• we need to manage the permissions that define the custom role we created. Because of this, some organizations decide they’d rather use the predefined roles
• can only be applied to either the project level or organization level
• can’t be applied to the folder level

Service accounts

• Type of an identity so it gets granted a role (roles can be attached to it)
• Used when we want to give permissions to a Compute Engine virtual machine, rather than to a person.
• Example: we have an application running in a virtual machine that needs to store data in Cloud Storage, but we don’t want anyone on the internet to have access to that data–just that particular virtual machine. We can create a service account to authenticate that VM to Cloud Storage. 
• Named with an email address, but instead of passwords they use cryptographic keys to access resources. 
• If a service account has been granted Compute Engine’s Instance Admin role, this would allow an application running in a VM with that service account to create, modify, and delete other VMs.
• Service accounts do need to be managed so they are both identities and also resources, so they can have IAM policies of their own attached to them
• Example: 
• Alice needs to manage which Google accounts can act as service accounts so she can have the editor role on a service account
• Bob just needs to be able to view a list of service accounts so he can have the viewer role. 
• This is just like granting roles for any other Google Cloud resource.

Cloud Identity

New Google Cloud customers can log into the Google Cloud Console with a Gmail account and then use Google Groups to collaborate with teammates who are in similar roles. This approach is:

• easy to start with
• can present challenges later because the team's identities are not centrally managed
• if someone leaves the organization there's no easy way to immediately remove a user's access to the team's cloud resources

Cloud Identity tool:

•  allows organizations to define policies and manage their users and groups using the Google Admin Console. 
• Admins can log in and manage Google Cloud resources using the same user names and passwords they already used in existing Active Directory or LDAP systems.
• when someone leaves an organization, an administrator can use the Google Admin Console to disable their account and remove them from groups
• available in a free and a premium edition that provides capabilities to manage mobile devices.
• free for Google Workspace customers

Interacting with Google Cloud

There are four ways to access and interact with Google Cloud:

• Cloud Console - Google Cloud’s graphical user interface (GUI)
• simple web-based interface
• helps you deploy, scale, and diagnose production issues
• you can easily find your resources, check their health, have full management control over them, and set budgets to control how much you spend on them
• also provides a search facility to quickly find resources and connect to instances via SSH in the browser
• similar to AWS Console
• Cloud SDK and Cloud Shell
• The Cloud SDK is a set of tools that you can use to manage resources and applications hosted on Google Cloud. When installed, all of the tools within the Cloud SDK are located under the bin directory. These tools are:
• Google Cloud CLI (gcloud command) which provides the main command-line interface for Google Cloud products and services
• bq, a command-line tool for BigQuery
• Cloud Shell provides command-line access to cloud resources directly from a browser.
• it is a Debian-based virtual machine with a persistent 5 gigabyte home directory, which makes it easy to manage Google Cloud projects and resources
• With Cloud Shell, the Cloud SDK gcloud command and other utilities are always installed, available, up to date, and fully authenticated
• APIs
• The services that make up Google Cloud offer APIs so that code you write can control them. 
• The Cloud Console includes a tool called the Google APIs Explorer that shows which APIs are available, and in which versions
• Google provides Cloud Client libraries and Google API Client libraries in many popular languages (Java, Python, PHP, C#, Go, Node.js, Ruby, and C++)
• Google Cloud App
• can be used to start, stop, and use SSH to connect to Compute Engine instances and see logs from each instance
• also lets you stop and start Cloud SQL instances
• you can administer applications deployed on App Engine by viewing errors, rolling back deployments, and changing traffic splitting
• provides up-to-date billing information for your projects and billing alerts for projects that are going over budget. You can set up customizable graphs showing key metrics such as CPU usage, network usage, requests per second, and server errors
• offers alerts and incident management
• you can download it at cloud.google.com/app

References:

Google Cloud Fundamentals: Core Infrastructure | Coursera

Disclaimer:  The course content rights belong to course creators (Google Cloud).