Object Storage: Amazon S3

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means that customers of all sizes and segments can use it to store and protect any amount of data for a range of use cases. Use cases include the following:

• Data lakes

• Websites

• Mobile applications

• Backup and restore

• Archive

• Enterprise applications

• IoT devices

• Big data analytics

S3 features include the following capabilities:

• Appending metadata tags to objects

• Moving and storing data across different S3 storage classes

• Configuring and enforcing data access controls

• Securing data against unauthorized users

• Running big data analytics

• Monitoring data at the object or bucket levels

• Viewing storage usage and activity trends across your organization

Amazon S3 storage classes

Amazon S3 offers a range of storage classes designed for different use cases. Every S3 storage class supports a specific data access level at corresponding costs or geographic location. S3 storage classes include:

• S3 Standard for general-purpose storage of frequently accessed data

• S3 Standard-Infrequent Access (S3 Standard-IA) for less frequently accessed data

• S3 One Zone-Infrequent Access (S3 One Zone-IA) for less frequently accessed data and lower availability requirements

• S3 Intelligent-Tiering for data with unknown or changing access patterns

• Amazon S3 Glacier Instant Retrieval for lower-cost archival storage that may require retrieval at any time.

• Amazon S3 Glacier Flexible Retrieval for low-cost archival storage with retrieval time from minutes to hours

• Amazon S3 Glacier Deep Archive (S3 Glacier Deep Archive) for lowest-cost storage with retrieval times up to 12 hours

• Amazon S3 on Outposts for on-premises hybrid data storage and satisfying data residency requirements.

Block Storage: Amazon EBS

The AWS block storage portfolio consists of two types of block storage services: Amazon Elastic Compute Cloud (Amazon EC2) instance storage and Amazon Elastic Block Store (Amazon EBS). Amazon EBS also includes an integrated snapshot service. Amazon EBS is the primary block storage service.

An instance store provides temporary (ephemeral) block-level storage for your instance. This storage is located on disks that are physically attached to the host computer where the compute instance is. Instance stores resemble Amazon EBS storage in initial configuration options. However, their architecture most closely resembles direct attached disk drives. An instance store provides submillisecond latencies between the EC2 instance and the storage.

Instance store is ideal for the following use cases:

• Temporary storage of information that changes frequently, such as buffers, caches, scratch data, and other temporary content

• Data that is replicated across a fleet of instances, such as a load-balanced pool of web servers.

Amazon EBS is an easy-to-use, high performance, block storage service. It is designed for use with Amazon EC2 compute instances for both throughput and transaction-intensive workloads at any scale.

AWS recommends Amazon EBS for data that must be quickly accessible and requires long-term persistence. EBS volumes are well suited for use as the primary storage for file systems, databases, or any applications that require fine granular updates and access to raw, unformatted, block-level storage. Amazon EBS is well suited to both database-style applications that rely on random reads and writes and to throughput-intensive applications that perform long, sequential reads and writes.

EBS volumes behave like raw, unformatted block devices. You can mount these block devices as EBS volumes on your EC2 instances.

---

File Storage: Amazon EFS

Amazon EFS file systems can grow to petabyte scale, drive high levels of throughput, and allow massively parallel access from compute instances to your data.

Amazon Elastic File System (Amazon EFS) provides a simple, serverless, elastic file system for use with AWS cloud services and on-premises resources. It is designed to scale on demand to petabytes without disrupting applications, growing and shrinking automatically. You can add and remove files, eliminating the need to provision and manage capacity to accommodate growth.

Amazon EFS supports the Network File System version 4 (NFSv4.1 and NFSv4.0) protocol. The applications and tools that you use today work seamlessly with Amazon EFS. Multiple compute modules can access an Amazon EFS file system at the same time. These modules include include Amazon EC2, AWS Lambda, Amazon Elastic Container Service (Amazon ECS), and Amazon Elastic Kubernetes Service (Amazon EKS). Accessing compute services provides a common data source for workloads and applications running on more than one compute instance or container.

With Amazon EFS, you pay only for the storage used by your file system, with no minimum fee or setup cost. Amazon EFS offers a range of storage classes designed for different use cases. These include:

• Standard storage classes – EFS Standard and EFS Standard–Infrequent Access (Standard–IA), which offer multiple Availability Zones (Multi-AZ) resilience and the highest levels of durability and availability.

• One Zone storage classes – EFS One Zone and EFS One Zone–Infrequent Access (EFS One Zone–IA), which offer additional savings by choosing to save data in a single-Availability Zone (Single-AZ).

File storage: Amazon FSx for NetApp ONTAP

With Amazon FSx for NetApp ONTAP, you can launch and scale reliable, high-performance, and feature-rich file storage built on NetApp's popular ONTAP file system. It provides the familiar features, performance, capabilities, and APIs of NetApp file systems with the agility, scalability, and simplicity of a fully managed AWS service.

With FSx for ONTAP, you get a fully managed file storage solution with:

• Support for petabyte-scale data sets in a single namespace

• Multiple gigabytes per second (GBps) of throughput per file system

• Multi-protocol access to data using the NFS, SMB, and iSCSI protocols

• High availability and durability with Multi-AZ deployments

• Automatic data tiering that reduces storage costs by automatically transitioning infrequently-accessed data to a lower-cost storage tier based on your access patterns

• Data compression, deduplication, and compaction to reduce your storage consumption

• Support for NetApp's SnapMirror replication feature

• Support for NetApp's on-premises caching solutions: NetApp Global File Cache and FlexCache

• Support for access and management using native AWS or NetApp tools and APIs

  • AWS Management Console, CLI, and SDKs

  • NetApp ONTAP CLI, REST API, and Cloud Manager

• Support for the following data protection and security features:

  • Encryption of file system data and backups at rest using KMS keys

  • Encryption of data in-transit using SMB Kerberos session keys

  • On-demand anti-virus scanning

  • Authentication and authorization using Active Directory

  • File access auditing

File storage: Amazon FSx for OpenZFS

Amazon FSx for OpenZFS makes it easy to launch, run, and scale fully managed file systems on AWS that replace the file servers you run on premises while helping to provide better agility and lower costs. With Amazon FSx for OpenZFS, you no longer have to worry about setting up and provisioning file servers and storage volumes, replicating data, installing and patching file server software, detecting and addressing hardware failures, and manually performing backups.

Amazon FSx for OpenZFS is a fully managed file storage service that makes it easy to move data residing in on-premises ZFS or other Linux-based file servers to AWS without changing your application code or how you manage data. It offers highly reliable, scalable, high-performing, and feature-rich file storage built on the open-source OpenZFS file system.

With FSx for OpenZFS, you get a fully managed file storage solution with:

• Support for access from Linux, Windows, and macOS compute instances and containers (running on AWS or on-premises) via the industry-standard NFS protocol (v3, v4.0, v4.1, and v4.2).

• Up to 1 million IOPS with latencies of just a few hundred microseconds, and up to 12.5 GBps of throughput for frequently accessed data (from in-memory cache). Up to 160,000 IOPS and up to 4 GBps of uncompressed throughput and 8-12 GBps of compressed throughput for data accessed from SSD disks.

• Powerful OpenZFS data management capabilities including Z-Standard compression, near instant point-in-time snapshots, and data cloning, natively supported via the Amazon FSx API.

• Highly available and highly durable file systems.

• Support for multiple volumes per file system, thin provisioning, and user and group quotas for cost-efficient shared file systems across multiple users and applications.

• Support for the following data protection and security features::

  • Built-in, fully managed file system backups stored on S3, with support for cross-region backup copies.

  • Near-instant point-in time OpenZFS snapshots stored locally on each file system.

  • Automatic encryption of file system data and backups at rest using KMS keys.

  • Automatic encryption in-transit when accessed from supported EC2 instances.

File storage: Amazon FSx for Lustre

Amazon FSx for Lustre streamlines the launching and running of high-performance Lustre file systems. Use FSx for Lustre for workloads where speed matters, such as machine learning, high performance computing (HPC), video processing, and financial modeling.

The open-source Lustre file system is designed for compute-intensive applications that require a fast storage system capable of meeting throughput requirements. It is also designed to meet input/output operations per second (IOPS) requirements at scale. FSx for Lustre was built to process large datasets quickly and cost-effectively and scale to meet growing demands. An FSx for Lustre file system is capable of delivering hundreds of gibibytes (GiB) per second of throughput and millions of IOPS.

FSx for Lustre provides the performance processing required for compute-intensive workloads across different horizontal and vertical use cases.

• Horizontal use cases apply across various industries and market segments. These industries have a widespread adoption and ever-increasing demand for more performance to serve increasingly larger number of compute cores. Examples of horizontal use cases include machine learning and high performance computing (HPC) workloads.

• Vertical use cases apply within an industry or market segment. A number of compute-intensive verticals depend on parallel file systems to process workloads. Examples of these workloads include the following:

  • Life science genomics

  • Financial models

  • Industrial design simulation

  • Media special effects and rendering

  • Seismic and reservoir exploration

  • Any other vertical where HPC and machine language are present

File storage: Amazon FSx for Windows File Server

With Amazon FSx for Windows File Server, you can launch and scale reliable, performant, and secure shared file storage for your applications and users. You can launch highly durable and available file systems that can span multiple Availability Zones. These files systems can be accessed from up to thousands of compute instances using the open standard Server Message Block (SMB) protocol. The service provides a set of administrative and security features, and integrates with Microsoft Active Directory.

Amazon FSx for Windows File Server provides fully managed Microsoft Windows file servers, backed by a fully native Windows file system. Amazon FSx for Windows File Server has the features, performance, and compatibility to easily lift and shift enterprise applications to the AWS Cloud.

To access file storage over a network, FSx for Windows File Server provides native support for Windows file system features and for the SMB protocol. Amazon FSx is optimized for enterprise applications in AWS with native Windows compatibility, enterprise performance and features, and consistent submillisecond latencies.

With file storage on FSx for Windows File Server, the code, applications, and tools that Windows developers and administrators use can continue to work unchanged. Windows applications and workloads ideal for FSx for Windows File Server include the following:

• Business applications

• Home directories

• Web serving

• Content management

• Data analytics

• Software build setups

• Media processing workloads