The Most Popular AWS Products of 2016

We know from the past 5 years of Gartner Magic Quadrants that AWS is a leader among IaaS vendors, placing the furthest for ‘completeness of vision’ and ‘ability to execute.’ AWS’ rapid pace of innovation contributes to its position as the leader in the space. The cloud provider releases hundreds of product and service updates every year. So, which of those are the most popular amongst our enterprise clients?

We analyzed data from our customers for the year, from a combined 100,000+ instances running monthly. The most popular AWS products and services, represented by the percentage of 2nd Watch customers utilizing them in 2016, include Amazon’s two core services for compute and storage – EC2 and S3 – and Amazon Data Transfer, each at 100% usage. Other high-ranking products include Simple Queue Service (SQS) for message queuing (84%) and Amazon Relational Database Service or RDS (72%). Usage for these services remains fairly consistent, and we would expect to see these services across most AWS deployments.

There are some relatively new AWS products and services that made the “most-popular” list for 2016 as well. AWS Lambda serverless computing (38%), Amazon WorkSpaces, a secure virtual desktop service (27%), and Kinesis, a real-time streaming data platform (12%), are quickly being adopted by AWS users and rising in popularity.

The fas-growing services in 2016, based on CAGR, include AWS CloudTrail (48%), Kinesis (30%), Config for resource inventory, configuration history, and change notifications (24%), Elasticsearch Service for real-time search and analytics (22%), Elastic MapReduce, a tool for big data processing and analysis, (20%) and Redshift, the data warehouse service alternative to systems from HP, Oracle and IBM (14%).

The accelerated use of these products demonstrates how quickly new cloud technologies are becoming the standard in today’s evolving market. Enterprises are moving away from legacy systems to cloud platforms for everything from back-end systems to business-critical, consumer-facing assets. We expect growth in each of these categories to continue as large organizations realize the benefits and ease of using these technologies.

Download the 30 Most Popular AWS Products infographic to find out which others are in high-demand.

-Jeff Aden, Co-Founder & EVP Business Development & Marketing


Database Migration Service for RDS

For database administrators and engineers, migrating a database can be a major headache. It’s such a headache that it actually prohibits some teams from migrating to AWS’ Relational Database Service (RDS), even though doing so would save them time and money in the long run.

Imagine you’re a DBA for Small Business Y and you want to manage your data in RDS, but you have three terabytes of data with a ton of tables, foreign keys and many dependencies. Back in the day, migrating your SQL Server database to RDS might have looked something like this:

  1. Coordinate with Product Leads to find a time when your business can handle up to a 24-hour outage of source database.
  2. Dump all the existing data into a backup.
  3. Restore the data on an intermediary EC2 SQL Server instance.
  4. Connect to the new RDS instance.
  5. Generate metadata script from the source or intermediary instance.
  6. Execute metadata script on target RDS instance.
  7. Migrate the data to the RDS instance using SQL Server’s Import tool.
  8. Optional: Encounter complications such as import failures, loss of metadata integrity, loss of data, and extremely slow import speeds.
  9. Cry a lot and then sleep for days.

Enter AWS Database Migration Service. This new tool from AWS allows DBAs to complete migrations to RDS, or even to a database instance on EC2, with minimal steps and minimal downtime to the source database. What does that mean for you? No 2AM migrations and no tears.

Migrations with AWS DMS have three simple steps:

  1. Provision a replication instance
  2. Define source and target endpoints
  3. Create one or more tasks to migrate data between source and target

The service automates table-by-table migration into your target database without having to use any backups, dump files, or manually administering an intermediary database server. In addition, you can use the service to continue replicating changes from the source environment until you are ready to cutover. That means your application’s downtime will be just a few minutes instead of several hours or even days.

Another great feature of DMS is that you can watch the progress of each table’s migration in the AWS console. The dashboard allows users to see, in real time, how the migration is going and see exactly where any breakages occur.

 

If you are planning to use the AWS Database Migration Service soon, here are a few tips to streamline your process:

  • In your target instance, set up your table structure prior to migrating the data. The service will not automatically set up your foreign keys and indexes. Set up your metadata from AWS schema conversion tool (or simple mysqldump). Make sure to use truncate option during schema import so that the tables you create aren’t wiped.
  • If you think you may have extra-large LOBs, use Full LOB Mode to avoid data truncation.
  • Additional best practices for using DMS can be found here.

Enjoy!

-Trish Clark, Oracle DBA


A Deeper Look at AWS Aurora

A few months back we published a blog article titled Introducing Amazon Aurora, which described Amazon’s la RDS RDBMS engine offering Aurora.  AWS Aurora is Amazon’s own internally developed MySQL 5.6 compatible DBMS.

Let’s review what we learned from the last article:

  • MySQL “drop-in” compatibility
  • Roughly 5x performance increase over traditional MySQL
  • Moves from a monolithic to service-oriented approach
  • Dynamically expandable storage (up to 64T) with zero downtime or performance degradation
  • Data storage and IO utilization is “only pay for what you use” ($0.10/GB/mo., $0.20/million IO)
  • High performance SSD backed storage
  • Data is automatically replicated (two copies) across three availability zones
  • Uses quorum writes (4 of 6) to increase write performance
  • Self-healing instant recovery through new parallel, distributed, and asynchronous redo logs
  • Cache remains warmed across DB restarts by decoupling the cache from the DB process
  • Up to 15 read replicas for scaling reads horizontally
  • Any read replica can be promoted to the DB master nearly instantly
  • Simulation of node, disk, or networking failure for ing/HA

In addition to those, I’d like to point out a few more features of Aurora:

  • Designed for 99.99% availability
  • Automatic recovery from instance and storage failures
  • On-volume instant snapshots
  • Continuous incremental off-volume snapshots to S3
  • Automatic restriping, mirror repair, hot spot management, and encryption
  • Backups introduce zero load on the DB
  • 400x (yes TIMES, NOT PERCENT!) lowered read replica lag over MySQL
  • Much improved concurrency handling

That is a pretty impressive list of features/improvements that Aurora buys us over standard MySQL!  Even at the slight increase in Aurora’s RDS run rate, the performance gains more than offset the added run costs in typical use-cases.

So what does that list of features and enhancements translate to in the real world?  What does it ACTUALLY MEAN for the typical AWS customer?  Having worked with a fair number of DBMS platforms over the past 15+ years, I can tell you that this is a pretty significant leap forward.  The small cost increase over MySQL is nominal in light of the performance gains and added benefits.  In fact, the price-to-performance ratio (akin to horsepower-to-weight ratio for any gear heads out there) is advertised as being 4x that of standard MySQL RDS.  This means you will be able to gain similar performance on a significantly smaller instance size.  Combine that with only having to pay for the storage your database is actually consuming (not a pre-allocated chunk) and all of the new features, and choosing Aurora is nearly always going to be your best option.

You should definitely consider using Aurora to replace any of your MySQL or MySQL derivative databases (Oracle MySQL, Percona, Maria).  It’s designed using modern architectural principals, for the Cloud, and with high scalability and fault-tolerance in mind.  Whether you are currently running, or are considering running, your own MySQL DBMS solution on EC2 instances or are using RDS to manage it, you should strongly consider Aurora.  The only exception to this may be if you are using MySQL on a lower-end RDS, EC2, Docker, etc. instance due to lower performance needs and cost considerations.

Because Aurora has some specific performance requirements, it requires db.r3.large, or faster, instances.  In some cases people choose to run smaller instances for their MySQL as they have lower performance and availability needs than what Aurora provides and would prefer the cost savings.  Also, there is no way to run “Aurora” outside of RDS (as it is a platform and not simply an overhauled DBMS), which could be a consideration for those wanting to run it in a dev/ context on micro instances or containers. However, running a 5.6 compatible version of MySQL would provide application congruency between the RDS Aurora instance and the one-off (e.g. a developer’s MySQL 5.6 DB running on a Docker container).

In addition to being an instantly pluggable MySQL replacement, Aurora can be a great option for replacing high-end, expensive commercial DBMS solutions like Oracle.  Switching to Aurora could provide massive cost savings while delivering a simplified, powerful, and highly scalable solution.  The price-to-performance ratio of Aurora is really in a class by itself, and it provides all of the features and performance today’s critical business applications demand from a relational database.  Aurora gives you on-par, or even better, performance, manageability, reliability, and security for around 1/10th of the cost!

To learn how the experts at 2nd Watch can help you get the most out of your cloud database architecture, contact us today.

-Ryan Kennedy, Senior Cloud Architect


Introducing Amazon Aurora

[et_pb_section fb_built=”1″ admin_label=”section” _builder_version=”3.0.47″][et_pb_row admin_label=”row” _builder_version=”3.0.47″ background_size=”initial” background_position=”top_left” background_repeat=”repeat”][et_pb_column type=”4_4″ _builder_version=”3.0.47″ parallax=”off” parallax_method=”on”][et_pb_text admin_label=”Text” _builder_version=”3.0.47″ background_size=”initial” background_position=”top_left” background_repeat=”repeat”]

When you think of AWS, the first thing that comes to mind is scalability, high availability, and performance. AWS is now bringing the same features to a relational database by offering Aurora through Amazon RDS. As announced at AWS re:Invent 2014, Amazon Aurora is a fully managed relational database engine that is feature compatible with MySQL 5.6. Though it is currently in preview, it promises to give you the performance and availability of high end commercially available databases with built-in scalability without the administrative overhead.

To create Amazon Aurora, AWS has taken a service-oriented approach, making individual layers such as storage, logging and caching scale out while keeping the SQL and transaction layers using the core MySQL engine.

So, let’s talk about storage. Aurora can automatically add storage volume in 10 GB chunks up to 64 TB. This eliminates the need to plan for the data growth upfront and manually intervene when storage needs to be added. This feature is described by AWS to happen seamlessly without any downtime or performance hit.

Amazon Aurora’s storage volume is automatically replicated across three AWS Availability Zones (AZ) with two copies of data in each AZ.  Aurora uses quorum writes and reads across these copies. It can handle the loss of two copies of data without affecting database write availability and up to 3 copies without affecting read availability. This SSD powered multi-tenant 10 GB storage chunks allows for concurrent access and reduces hot spots making it fault tolerant. It is also self-healing as it continuously scans data blocks for errors and repairs them automatically.

Unlike in a traditional database where it has to replay the redo log since last check point, which is single threaded in MySQL, Aurora’s underlying storage replays redo logs on-demand, in parallel, distributed and asynchronous mode. This allows you to recover from a crash almost instantaneously.

Aurora database can have up to 15 Aurora replicas. Since master and replica share the same underlying storage, you can failover with no data loss. Also, there’s very little load on the master since there’s no log replay, resulting in minimal replica lag of approximately 10 to 20 milliseconds. The cache lives outside of the database process which allows it to survive during a database restart. As a result, operations can be resumed much faster as there is no need to warm the cache. The backup is automatic, incremental and continuous. This enables point-in-time recovery up to the last five minutes.

A new added feature of Amazon Aurora is its ability to simulate failure of node, disk or networking components using SQL commands. This allows you to the high availability and scaling features of your application.

According to the Aurora FAQ, it is five times faster than a stock version of MySQL. Using Sysbench “Aurora delivers over 500,000 selects/sec and 100,000 updates/sec running the same benchmark on the same hardware.”

While Aurora may cost slightly more than the RDS for MySQL per instance based on the US-EAST-1, the features may justify it, and you only pay for the storage consumed at a rate of $0.10/GB per month and IOs at a rate of $0.20 per million requests.

It’s exciting to see the challenges of scaling the relational database being addressed by AWS. To learn more about Amazon Aurora, you can sign up for a preview at http://aws.amazon.com/rds/aurora/preview/.

2nd Watch specializes in helping our customers with legacy investments in Oracle achieve successful migrations to Aurora.  Our Oracle-certified experts understand the specialized features, high availability and performance capabilities that proprietary components such as RAC, ASM and Data Guard provide and are skilled in delivering low-risk migration roadmaps.  We also offer schema and PL/SQL migration guidance, client tool validation, bulk data upload and ETL integration services. 

Ali Kugshia, Sr. Cloud Engineer

[/et_pb_text][/et_pb_column][/et_pb_row][/et_pb_section]


2nd Watch AWS Scorecard

Check out our new AWS Scorecard for a look at what we’re seeing companies typically use for their cloud services. Taken from AWS usage trends among 2nd Watch customers for July-October, 2014.

Scorecard Highlights:

Organizations using Amazon EC2 are typically broken down in the following percentages:

  • 38% use Small instances
  • 19% use Medium
  • 15% use XLarge
  • The very large (which include 2XLarge, 4XLarge and 8XLarge),  and the very small (Micro) account for only  27% collectively.

 

Among our customers:

  • 94% use Amazon’s Simple Storage Service (S3)
  • 66% use Amazon’s Simple Notification Service (SNS) for push messaging
  • 41% use Amazon’s Relational Database Service (RDS) to set up, operate, and scale a relational database in the cloud.

 

Around three-quarters of customers run Linux instances, with the remaining using Windows. However Windows systems accounted for 31% of all computing hours, and more money is typically spent on Windows instances.

 


Increasing Your Cloud Footprint

The jump to the cloud can be a scary proposition.  For an enterprise with systems deeply embedded in traditional infrastructure like back office computer rooms and datacenters the move to the cloud can be daunting. The thought of having all of your data in someone else’s hands can make some IT admins cringe.  However, once you start looking into cloud technologies you start seeing some of the great benefits, especially with providers like Amazon Web Services (AWS).  The cloud can be cost-effective, elastic and scalable, flexible, and secure.  That same IT admin cringing at the thought of their data in someone else’s hands may finally realize that AWS is a bit more secure than a computer rack sitting under an employee’s desk in a remote office.  Once the decision is finally made to “try out” the cloud, the planning phase can begin.

Most of the time the biggest question is, “How do we start with the cloud?”  The answer is to use a phased approach.  By picking applications and workloads that are less mission critical, you can try the newest cloud technologies with less risk.  When deciding which workloads to move, you should ask yourself the following questions; Is there a business need for moving this workload to the cloud?  Is the technology a natural fit for the cloud?  What impact will this have on the business? If all those questions are suitably answered, your workloads will be successful in the cloud.

One great place to start is with archiving and backups.  These types of workloads are important, but the data you’re dealing with is likely just a copy of data you already have, so it is considerably less risky.  The easiest way to start with archives and backups is to try out S3 and Glacier.  Many of today’s backup utilities you may already be using, like Symantec Netbackup  and Veeam Backup & Replication, have cloud versions that can directly backup to AWS. This allows you to use start using the cloud without changing much of your embedded backup processes.  By moving less critical workloads you are taking the first steps in increasing your cloud footprint.

Now that you have moved your backups to AWS using S3 and Glacier, what’s next?  The next logical step would be to try some of the other services AWS offers.  Another workload that can often be moved to the cloud is Disaster Recovery.   DR is an area that will allow you to more AWS services like VPC, EC2, EBS, RDS, Route53 and ELBs.  DR is a perfect way to increase your cloud footprint because it will allow you to construct your current environment, which you should already be very familiar with, in the cloud.  A Pilot Light DR solution is one type of DR solution commonly seen in AWS.  In the Pilot Light scenario the DR site has minimal systems and resources with the core elements already configured to enable rapid recovery once a disaster happens.  To build a Pilot Light DR solution you would create the AWS network infrastructure (VPC), deploy the core AWS building blocks needed for the minimal Pilot Light configuration (EC2, EBS, RDS, and ELBs), and determine the process for recovery (Route53).  When it is time for recovery all the other components can be quickly provisioned to give you a fully working environment. By moving DR to the cloud you’ve increased your cloud footprint even more and are on your way to cloud domination!

The next logical step is to move Test and Dev environments into the cloud. Here you can get creative with the way you use the AWS technologies.  When building systems on AWS make sure to follow the Architecting Best Practices: Designing for failure means nothing will fail, decouple your components, take advantage of elasticity, build security into every layer, think parallel, and don’t fear constraints! Start with proof-of-concept (POC) to the development environment, and use AWS reference architecture to aid in the learning and planning process.  Next your legacy application in the new environment and migrate data.  The POC is not complete until you validate that it works and performance is to your expectations.  Once you get to this point, you can reevaluate the build and optimize it to exact specifications needed. Finally, you’re one step closer to deploying actual production workloads to the cloud!

Production workloads are obviously the most important, but with the phased approach you’ve taken to increase your cloud footprint, it’s not that far of a jump from the other workloads you now have running in AWS.   Some of the important things to remember to be successful with AWS include being aware of the rapid pace of the technology (this includes improved services and price drops), that security is your responsibility as well as Amazon’s, and that there isn’t a one-size-fits-all solution.  Lastly, all workloads you implement in the cloud should still have stringent security and comprehensive monitoring as you would on any of your on-premises systems.

Overall, a phased approach is a great way to start using AWS.  Start with simple services and traditional workloads that have a natural fit for AWS (e.g. backups and archiving).  Next, start to explore other AWS services by building out environments that are familiar to you (e.g. DR). Finally, experiment with POCs and the entire gambit of AWS to benefit for more efficient production operations.  Like many new technologies it takes time for adoption. By increasing your cloud footprint over time you can set expectations for cloud technologies in your enterprise and make it a more comfortable proposition for all.

-Derek Baltazar, Senior Cloud Engineer


Cloud Backup: More is Always Better

You’ve seen podcasts, blogs, whitepapers, and magazine articles all promoting backups, the concept, the benefits they provide, and the career crushing disasters that can happen when they’re ignored. Bottom line: You do it, and you do it now. Hey, do more than one, the more the merrier. Run at the right time (meaning low user load), and in a well maintained hierarchical storage process, they can only do good. A solid backup strategy can even get you business insurance discounts in some cases.

But backing up in-house servers is a relatively simple and time-ed process. Most operating systems have one or more backup utilities built-in, and then there are third-party options that take the concept to whole new level. But backup in or from the cloud is a new concept with new variables – data security, multi-tenant segregation, virtualized services, and especially cloud and internet bandwidth. That’s got to be difficult at the enterprise computing level. Wrong! Not for AWS customers.

For a while, cloud storage was mainly a backup solution on its own. But Amazon has matured its backup and data redundancy offerings over the years to a point where you’ll find a range of sophisticated options at your disposal, depending on your needs. Obviously, we start with S3.

S3 has been and still is a basic backup solution for many companies – some rely on it exclusively, since data in S3 gets additional redundancy and backup features in AWS’ datacenters, so customers can be fairly confident their data will be there when they need it. S3’s storage architecture is pleasantly simple to understand. You’ll find it revolves primarily around two key nouns: buckets and objects.

Buckets are just ways to organize your objects, and objects represent every individual file that’s been backed up. Every object has its own secure URL, which allows easy organization and management of data using a number of homegrown tools like basic access control or Amazon prescribed bucket polices. But you’re also able to choose from new third-party solutions with even easier interfaces, like Symantec for enterprises or Dropbox for small businesses.

Recently, Amazon has fleshed out its backup solution even more with the introduction of Glacier, which doesn’t relate to melting polar ice caps as much as it does to slow data retrieval times.

Glacier’s mission is to provide a much lower-cost backup solution (as little as a penny per gigabyte in some situations). The tradeoff is that because its low cost, it’s significantly slower than S3. But for long-term backup of important files, Glacier removes the need for repetitive backup operations, capacity planning, hardware provisioning and more.

These are all time consuming tasks that add to the hidden costs of secure in-house backup.

Glacier takes all that off your plate for very little money. Bottom line: use S3 if you’re storing data you’ll access often or that requires high-speed access; use Glacier for less frequently accessed data for which slow retrieval times (sometimes several hours) are acceptable.

That’s only a short glimpse into the S3 portion of Amazon’s backup capabilities. If we went into detail about all of AWS data protection features, we could publish this as a book. We’ll hit each of them in detail in future posts, but here’s a quick list of Amazon’s data protection solutions across their service portfolio:

  • Multi-zone and multi-region redundancy
  • AWS relational database service (RDS) bundled, automated back up
  • RDS backups you can control manually, like snapshots
  • AWS Import/Export
  • EC2 and S3
  • S3 and Elastic Block Store (EBS)
  • Route 53
  • Elastic Load Balancing (ELB)
  • Third-party backup platforms you can use to construct your own AWS storage hierarchy

Data safety has been a hot-button issue for folks still on the fence about adopting the cloud. But AWS has done an excellent job innovating sophisticated new data redundancy solutions that can make backing up to the cloud safer than in your own datacenter. Check it out.

-Travis Greenstreet, Senior Cloud Systems Engineer