Developer Blog

Java on AWS: Tips, tricks and news from the AWS Java SDK and Tools team
RSS
  • Jason Fulghum

    Eclipse Deployment: Part 3 - Configuring AWS Elastic Beanstalk

    • May 16, 2013
    • Jason Fulghum

    Now that you know the basics about creating AWS Java web applications and deploying them using the AWS Toolkit for Eclipse, let's talk about some of the ways you can control how your environment runs.

    AWS Elastic Beanstalk provides several easy ways to configure different features of your environment. The first mechanism we'll look at for controlling how your environment runs is your environment's configuration. These are properties set through the Elastic Beanstalk API that let you control different operational parameters of your environment, such as load balancer behavior and auto scaling strategies. The second mechanism we'll look at is Elastic Beanstalk extension config files that are included as files in your deployed application. These configuration files allow you to customize additional software installed on your EC2 instances, as well as create and configure AWS resources that your application requires.

    We'll start off by covering some of the most common options, which are presented in the second page of the wizard when you create a new Elastic Beanstalk environment through Eclipse.

    Shell Access

    If you want to be able to remotely log into a shell on the EC2 instances running your application, then you'll need to make sure you launch your environment with an Amazon EC2 key pair. The EC2 key pair can be created and managed through Eclipse or any of the other AWS tools, and allows you to securely log into any EC2 instances launched with that key pair. To connect to an instance from Eclipse, find your instance in the EC2 Instances view, right-click to bring up the context menu and select Open Shell. If Eclipse knows the private key for that instance's key pair, then you'll see a command prompt open up.

    CNAMEs

    The default URL for your application running on AWS Elastic Beanstalk probably isn't something that your customers will be able to easily remember. You can add another abstraction layer by creating a CNAME record that points to your application's URL. You can set up that CNAME record with Amazon Route 53 (Amazon's DNS web service), or with any other DNS provider. This allows you to host your application under any domain you own. You can find more details on CNAMEs in the Elastic Beanstalk Developer Guide. This CNAME not only gives your application a more friendly URL, but it also provides an important abstraction that allows you to deploy new versions of your application with zero downtime by launching a new environment with your new application version and flipping the CNAME record over to the new environment's URL after you've confirmed it's ready for production traffic. You can read more about this technique in the Elastic Beanstalk Developer's Guide.

    Notifications

    AWS Elastic Beanstalk uses the Amazon Simple Notification Service (Amazon SNS) to notify you of important events affecting your application, such as environment status changes. To enable Amazon SNS notifications, simply enter your email address in the Email Address text box under Notifications on the Configuration tab inside the Toolkit for Eclipse.

    SSL Certificate

    If your application deals with sensitive customer information, then you'll probably want to configure an SSL cert for your load balancer so that all data between your customers and your environment's load balancer is encrypted. To do this, you'll need a certificate from an external certificate authority such as VeriSign or Entrust. Once you register the the certificate with the AWS Identity and Access Management service, you can enter the certificate's ID here to tell Elastic Beanstalk to configure your load balancer for SSL with your certificate.

    Health Check URL

    Your Elastic Beanstalk environment attempts to monitor the health of your application through the configured health check URL. By default Elastic Beanstalk will attempt to check the health of your application by testing a TCP connection on port 80. This is a very basic health check, and you can easily override this with your own custom health check. For example, you might create a custom health check page that will do some very basic tests of your application's health. Be careful that you make this health check page very simply though, since this check will be run often (the interval is configurable). If you want to do more in depth health checking, you might have a separate thread in your application that reports health status such as checking for DB connection health, and then simply have your health check page report that status. If one of the hosts in your environment starts failing health checks, it will automatically be removed from your environment so that it doesn't serve bad results to customers. The exact parameters on how these checks are run are configurable through the environment configuration editor that we'll see shortly.

    Incremental Deployment

    The Incremental Deployment option (enabled by default), only affects how Eclipse uploads new application versions to Elastic Beanstalk, but it's a neat option worth pointing out here. When you use incremental deployment, Eclipse will only push the delta of your most recent changes to AWS Elastic Beanstalk, instead of pushing every file in your whole application. Under the covers, Eclipse and Elastic Beanstalk are actually using the Git protocol to upload file deltas, and the end result is very fast application deployments for small changes after you've gone through a full push initially.

    After you've started your environment, you can modify any of these configuration options, and many more, by double-clicking on your Elastic Beanstalk environment in Eclipse's Servers view to open the Environment Configuration Editor. From here you can access dozens of settings to fine tune how your environment runs. Note that some of these options will require stopping and restarting your environment (such as changing the Amazon EC2 instance type your environment uses).

    From the environment configuration editor you have access to dozens of additional options for controlling how your environment runs. The Configuration tab in the editor shows you the most common options, such as EC2 key pairs, auto scaling and load balancing parameters, and specific Java container options such as JVM settings and Java system properties.

    The Advanced tab in the environment configuration editor has a complete list of every possible option for your environment, but for the vast majority of use cases, you shouldn't need more than the Configuration tab.

    Elastic Beanstalk Extension Config Files

    We've seen how to manipulate operational settings that control how your environment runs by updating an environment's configuration. These settings are all updated by tools working directly with the Elastic Beanstalk API to change these settings. The second way to customize your environment is through Elastic Beanstalk extension config files. These files live inside your project and get deployed with your application. They customize your environment in larger ways than the very specific settings we saw earlier.

    These extension config files allow you to customize the additional software available on the EC2 instances running your application. For example, your application might want to use the Amazon CloudWatch monitoring scripts to upload custom CloudWatch metrics. You can use these extension config files to specify that the Amazon CloudWatch monitoring scripts be installed on any EC2 instance that comes up as part of your environment, then your application code will be able to access them.

    You can also use these Elastic Beanstalk extension config files to create and configure AWS resources that your application will need. For example, if your application requires an Amazon SQS queue, you could declare it in your extension config file and even create an alarm on queue depth to notify you if your application gets behind on processing messages in the queue. The AWS Elastic Beanstalk Developer Guide goes into a lot more detail, and examples, demonstrating how to configure AWS resources with extension config files.

    That completes our tour of the different ways you can customize your Elastic Beanstalk environments. One of the great strengths of Elastic Beanstalk is that you can simply drop in your application and not worry about customization, but if you do want to customize, you have a wealth of different ways to configure your environment to run the way you need it to for your application. What kinds of customization settings have you tried for your Elastic Beanstalk environments? Let us know the comments below!

    • May 16, 2013
    • Permalink
    • Comments (0)
  • Jason Fulghum

    Eclipse Deployment: Part 2 - Deploying to AWS Elastic Beanstalk

    • May 7, 2013
    • Jason Fulghum

    In this three part series, we'll show how easy it is to deploy a Java web application to AWS Elastic Beanstalk using the AWS Toolkit for Eclipse.

    In part one of this series, we showed how to create an AWS Java Web Project and deploy it to a local Tomcat server. This is a great workflow for developing your project, but when you're ready for production, you'll want to get it running on AWS. In this second post of the series, we'll show how we can use the same tools in Eclipse to deploy our project using AWS Elastic Beanstalk.

    AWS Elastic Beanstalk provides a managed application container environment for your application to run in. That means all you have to worry about is your application code. Elastic Beanstalk handles the provisioning, load balancing, auto-scaling, and application health monitoring for you. Even though Elastic Beanstalk handles all these aspects for you, you still have control over all the settings, as we'll see in the next part of this series, if you do want to customize how your environment runs.

    The AWS Toolkit for Eclipse supports deploying Java web apps to Elastic Beanstalk Tomcat containers, but Elastic Beanstalk supports many other types of applications, including:

    • .NET
    • Ruby
    • Python
    • PHP
    • Node.js

    Let's go ahead and see how easy it is to deploy our application to AWS Elastic Beanstalk. We'll use the same workflow as before when we deployed our application to our local Tomcat server for local development and testing, but this time, we'll select to create a new AWS Elastic Beanstalk Tomcat 7 server.

    Right-click on your project and select Run As -> Run on Server, then make sure the Manually define a new server option is selected; otherwise, this wizard will only show you any existing servers you've configured. Select Elastic Beanstalk for Tomcat 7 from the Amazon Web Services category and move on to the next page in the wizard.

    This page asks for some very basic information about the Elastic Beanstalk environment that we're creating. Every Elastic Beanstalk environment is tied to a specific application, and of course has a name. You can choose to create a new application, or reuse an existing one. Whenever you deploy your project to this environment, you'll be creating a new version of that application, and then deploying that new version to run in your environment.

    On the next page of the wizard are some more options for configuring your new environment. We'll go over these options and more in the next post in this series.

    Go ahead and click the Finish button and Eclipse will start creating your new environment. The very first time you start your environment you'll need to wait a few minutes while Elastic Beanstalk provisions servers for you, configures them behind a load balancer and auto-scaling group, and deploys your application. Future deployments should go much faster, but Elastic Beanstalk needs to set up several pieces of infrastructure for you the first time a new environment starts up. To see more details about what Elastic Beanstalk is doing to set up your environment, double-click on the server you just created in Eclipse's Servers view, and open the Events tab in the server editor that opens. The event log shows you all the major events that Elastic Beanstalk is logging for your environment. If you ever have problems starting up your environment, the event log is the place to start looking for clues.

    After a few minutes, you should see your application start up in Eclipse's internal web browser, this time running from AWS instead of a local Tomcat server. 

    And that's all it takes to get a Java web application deployed to AWS using AWS Elastic Beanstalk and the AWS Toolkit for Eclipse.

    Now that you've got your environment running, try making a few small changes to your application and redeploying them, using the same tools as before. Once you get your application code set up, you'll switch over to incremental deployments and should get very fast redeploys.

    Stay tuned for the next post in this series, where we'll explain how you can customize your environment's configuration to control different aspects of how it runs.

    • May 7, 2013
    • Permalink
    • Comments (0)
  • Jason Fulghum

    Release: AWS Toolkit for Eclipse 2.3

    • May 2, 2013
    • Jason Fulghum

    We've just released a new version of the AWS Toolkit for Eclipse that adds support for managing your AWS Identity and Access Management (IAM) resources directly from within Eclipse, and updates the Amazon DynamoDB Create Table Wizard in the toolkit to support creating tables with Local Secondary Indexes.

    Check out the new functionality and let us know what you think in comments below!

    • May 2, 2013
    • Permalink
    • Comments (0)
  • Jason Fulghum

    Eclipse Deployment: Part 1 - AWS Java Web Applications

    • April 30, 2013
    • Jason Fulghum

    In this three part series, we'll show how easy it is to deploy a Java web application to AWS Elastic Beanstalk using the AWS Toolkit for Eclipse.

    The first post in this series demonstrates how to create an AWS Java Web Project, and explains how that project interacts with the existing web development tools in Eclipse.

    The AWS Toolkit for Eclipse builds on top of the standard Eclipse tooling for developing and deploying web applications, the Eclipse Web Tools Platform (WTP). This means you'll be able to leverage all of the tools provided by WTP with your new AWS Java Web Project, as we'll see later in this post.

    After you've installed the AWS Toolkit for Eclipse, open the New AWS Java Web Project wizard. 

    The wizard lets you enter your project name, AWS account, and whether you want to start with a bare bones project, or a more advanced reference application. We recommend starting with the basic Java web application for your first time through. If you haven't configured an AWS account yet, you'll want to follow the link in the wizard to add an account. Your account information will be used to configure your project so that your application code can make requests to AWS. Once you've got an AWS account selected, go ahead and fill out a project name, and keep the default option to start with a basic Java web application.

    After you've finished the wizard, you'll have an AWS Java Web Project, ready for you to start building your application in, or to go ahead and deploy somewhere.

    One of the great things about building on top of the Eclipse Web Tools Platform is that your project can use all the great tools provided by WTP for developing and deploying Java web applications. For example, try out the Create Servlet wizard provided by WTP:

    The Create Servlet wizard makes it very easy to create new servlets, and in addition to creating the class template for you, it will also update your project's web.xml with a mapping for the new servlet.

    You'll be able to use many other tools from WTP like custom editors for JSP and XML files, and tools for building and exporting WAR files.

    The coolest benefit, however, of building on top of WTP is that you can use the deployment support in WTP to deploy your AWS Java Web Projects in exactly the same way, whether you're uploading to a local Tomcat server for quick testing, or to a production Elastic Beanstalk environment, like we'll see in the next part of this series.

    Let's get our new project deployed to a local Tomcat server so we can see it running. Right-click on your project and select Run As -> Run On Server. You'll need to configure a new Tomcat server using this wizard, then Eclipse will start the server and deploy your project. When you're done, you should see something like this:

    Stay tuned for the next part of this series, where we'll show how to use the same tools to deploy our new application to AWS Elastic Beanstalk.

    • April 30, 2013
    • Permalink
    • Comments (0)
  • Zach Musgrave

    Using Custom Marshallers to Store Complex Objects in Amazon DynamoDB

    • April 10, 2013
    • Zach Musgrave

    Over the past few months, we've talked about using the AWS SDK for Java to store and retrieve Java objects in Amazon DynamoDB. Our first post was about the basic features of the DynamoDBMapper framework, and then we zeroed in on the behavior of auto-paginated scan. Today we're going to spend some time talking about how to store complex types in DynamoDB. We'll be working with the User class again, reproduced here:

    @DynamoDBTable(tableName = "users")
    public class User {
      
        private Integer id;
        private Set<String> friends;
        private String status;
      
        @DynamoDBHashKey
        public Integer getId() { return id; }
        public void setId(Integer id) { this.id = id; }
      
        @DynamoDBAttribute
        public Set<String> getFriends() { return friends; }
        public void setFriends(Set<String> friends) { this.friends = friends; }
      
        @DynamoDBAttribute
        public String getStatus() { return status; }
        public void setStatus(String status) { this.status = status; }
    
        @DynamoDBAttribute
        public String getStatus() { return status; }
        public void setStatus(String status) { this.status = status; }
    }
     

    Out of the box, DynamoDBMapper works with String, Date, and any numeric type such as int, Integer, byte, Long, etc. But what do you do when your domain object contains a reference to a complex type that you want persisted into DynamoDB?

    Let's imagine that we want to store the phone number for each User in the system, and that we're working with a PhoneNumber class to represent it. For the sake of brevity, we are assuming it's an American phone number. Our simple PhoneNumber POJO looks like this:

    public class PhoneNumber {
        private String areaCode;
        private String exchange;
        private String subscriberLineIdentifier;
        
        public String getAreaCode() { return areaCode; }    
        public void setAreaCode(String areaCode) { this.areaCode = areaCode; }
        
        public String getExchange() { return exchange; }   
        public void setExchange(String exchange) { this.exchange = exchange; }
        
        public String getSubscriberLineIdentifier() { return subscriberLineIdentifier; }    
        public void setSubscriberLineIdentifier(String subscriberLineIdentifier) { this.subscriberLineIdentifier = subscriberLineIdentifier; }      
    }
    

    If we try to store a reference to this class in our User class, DynamoDBMapper will complain because it doesn't know how to represent the PhoneNumber class as one of DynamoDB's basic data types.

    Introducing the @DynamoDBMarshalling annotation

    The DynamoDBMapper framework supports this use case by allowing you to specify how to convert your class into a String and vice versa. All you have to do is implement the DynamoDBMarshaller interface for your domain object. For a phone number, we can represent it using the standard (xxx) xxx-xxxx pattern with the following class:

    public class PhoneNumberMarshaller implements DynamoDBMarshaller
     
       {
    
        @Override
        public String marshall(PhoneNumber number) {
            return "(" + number.getAreaCode() + ") " + number.getExchange() + "-" + number.getSubscriberLineIdentifier();
        }
    
        @Override
        public PhoneNumber unmarshall(Class<PhoneNumber> clazz, String s) {
            String[] areaCodeAndNumber = s.split(" ");
            String areaCode = areaCodeAndNumber[0].substring(1,4);
            String[] exchangeAndSlid = areaCodeAndNumber[1].split("-");
            PhoneNumber number = new PhoneNumber();
            number.setAreaCode(areaCode);
            number.setExchange(exchangeAndSlid[0]);
            number.setSubscriberLineIdentifier(exchangeAndSlid[1]);
            return number;
        }    
    }
    
      
     

    Note that the DynamoDBMarshaller interface is templatized on the domain object you're working with, making this interface strictly typed.

    Now that we have a class that knows how to convert our PhoneNumber class into a String and back, we just need to tell the DynamoDBMapper framework about it. We do so with the @DynamoDBMarshalling annotation.

    @DynamoDBTable(tableName = "users")
    public class User {
        
        ...
        
        @DynamoDBMarshalling (marshallerClass = PhoneNumberMarshaller.class)
        public PhoneNumber getPhoneNumber() { return phoneNumber; }    
        public void setPhoneNumber(PhoneNumber phoneNumber) { this.phoneNumber = phoneNumber; }             
    }
    

    Built-in support for JSON representation

    The above example uses a very compact String representation of a phone number to use as little space in your DynamoDB table as possible. But if you're not overly concerned about storage costs or space usage, you can just use the built-in JSON marshaling capability to marshal your domain object. Defining a JSON marshaller class takes just a single line of code:

    class PhoneNumberJSONMarshaller extends JsonMarshaller
     
       { }
    
     

    However, the trade-off of using this built-in marshaller is that it produces a String representation that's more verbose than you could write yourself. A phone number marshaled with this class would end up looking like this (with spaces added for clarity):

    {
      "areaCode" : "xxx",
      "exchange: : "xxx",
      "subscriberLineIdentifier" : "xxxx"
    }
    

    When writing a custom marshaller, you'll also want to consider how easy it will be to write a scan filter that can find a particular value. Our compact phone number representation will be much easier to scan for than the JSON representation.

    We're always looking for ways to make our customers' lives easier, so please let us know how you're using DynamoDBMapper to store complex objects, and what marshaling patterns have worked well for you. Share your success stories or complaints in the comments!

    • April 10, 2013
    • Permalink
    • Comments (0)
  • Zach Musgrave

    Working with Different AWS Regions

    • April 5, 2013
    • Zach Musgrave

    Wherever you or your customers are in the world, there are AWS data centers nearby.

    Each AWS region is a completely independent stack of services, totally isolated from other regions. You should always host your AWS application in the region nearest your customers. For example, if your customers are in Japan, running your website from Amazon EC2 instances in the Asia Pacific (Tokyo) region will ensure that your customers get the lowest possible latency when they connect to your site.

    New in the 1.4 release of the AWS SDK for Java, the SDK now knows how to look up the endpoint for a given service in a particular region. Previously, developers needed to look up these endpoints themselves and then hard-code them into their applications when creating a client, like so:

    AmazonDynamoDB dynamo = new AmazonDynamoDBClient(credentials);
    dynamo.setEndpoint("https://dynamodb.us-west-2.amazonaws.com");
    

    With the 1.4 release, the SDK will look up a service's regional endpoint automatically, so all you have to know is which region you want to use. This newer method looks like this:

    AmazonDynamoDB dynamo = new AmazonDynamoDBClient(credentials);
    dynamo.setRegion(Region.getRegion(Regions.US_WEST_2));
    

    Regions can also create and configure clients for you, like a simple factory. This is especially helpful when you're working with multiple regions in your application and need to keep them straight. Just use region objects to create every client for you, and it will be obvious which client points to which region.

    AmazonDynamoDB dynamo = Region.getRegion(Regions.US_WEST_2)
                            .createClient(AmazonDynamoDBClient.class, credentials, clientConfig);
    

    It's important to note that the setRegion() method isn't thread-safe. We recommend setting the region once, when a client object is first created, then leaving it alone for the duration of the client's life cycle. Otherwise, the SDK's automatic retry logic could yield unexpected behavior if setRegion() is called at the wrong time. Using the Region objects as client factories encourages this pattern. If you need to talk to more than one region for a particular service, we recommend creating one service client object per region, rather than trying to share.

    Finally, at times it may be useful to programmatically determine which regions a given service is available in. It's possible to ask a Region object if a given service is supported there:

    Region.getRegion(Regions.US_WEST_2).isServiceSupported(ServiceAbbreviations.Dynamodb);
    

    For more information about which services are available in each region, see http://aws.amazon.com/about-aws/globalinfrastructure/regional-product-services/.

    For more information about the available regions and edge locations, see http://aws.amazon.com/about-aws/globalinfrastructure/.

    • April 5, 2013
    • Permalink
    • Comments (0)
  • Zach Musgrave

    The AWS Toolkit for Eclipse at EclipseCon 2013

    • March 26, 2013
    • Zach Musgrave

    Jason and I are at EclipseCon in Boston this week to discuss what we've learned developing the AWS Toolkit for Eclipse over the last three years. Our session is chock full of advice for how to develop great Eclipse plug-ins, and offers a behind-the-scenes look at how we build the Toolkit. Here's what we plan to cover:

    Learn best practices for Eclipse plug-in development that took us years to figure out!

    The AWS Toolkit for Eclipse brings the AWS cloud to the Eclipse workbench, allowing developers to develop, debug, and deploy Java applications on the AWS platform. For three years, we've worked to integrate AWS services into your Eclipse development workflow. We started with a small seed of functionality for managing EC2 instances, and today support nine services and counting. We learned a lot on the way, and we'd like to share!

    The Toolkit touches a wide array of Eclipse technologies and frameworks, from the Web Tools Platform to the Common Navigator Framework. By now we've explored so much of the Eclipse platform that we've started to become embarrassed by the parts of the Toolkit that we wrote first. If only someone had told us the right way to do things in the first place! Instead, we had to learn the hard way how to make our code robust, our user interfaces reliable and operating-system independent (not to mention pretty).

    We're here to teach from our experience, to share all the things we wish someone had told us before we learned it the hard way. These are the pointers that will save you hours of frustration and help you deliver a better product to your customers. They're the tips we would send back in time to tell our younger selves. We'll show you how we used them to make the Toolkit better and how to incorporate them into your own product.

    Topics include getting the most out of SWT layouts, using data binding to give great visual feedback in wizards, managing releases and updates, design patterns for resource sharing, and much more.

    If you are attending the conference, come by to say hello and get all your questions about the Toolkit answered! We are also handing out $100 AWS credits to help you get started using AWS services without a financial commitment, so come talk to us and we'll hook you up.

    • March 26, 2013
    • Permalink
    • Comments (0)
  • Jason Fulghum

    Eclipse: New AWS Java Project Wizard

    • March 15, 2013
    • Jason Fulghum

    If you're just getting started with the AWS SDK for Java, a great way to learn the SDK is through the AWS Toolkit for Eclipse. In addition to all the tools in the AWS Toolkit for Eclipse for managing your AWS resources, deploying your applications, etc., there are also wizards for creating new AWS projects, including sample code to help get you started.

    With the New AWS Java Project wizard, you can create a new Eclipse Java project, already configured with:

    • the AWS SDK for Java - including dependencies and full documentation and source attachment
    • your AWS security credentials - managed through Eclipse's preferences
    • optional sample code demonstrating how to work with a variety of different AWS services

    First, make sure that you have the latest plug-ins for the AWS Toolkit for Eclipse installed, available through the Eclipse Marketplace or directly from our Eclipse update site at http://aws.amazon.com/eclipse.

    Once you have the Eclipse tools installed, open the New AWS Java Project wizard, either through the context menu in Package Explorer, or through the File -> New menu.

    The New AWS Java Project wizard lets you pick the name for your project, your AWS security credentials, and any sample code that you want to start from. If you don't have your AWS security credentials configured in Eclipse yet, the link in the wizard takes you directly to the Eclipse preferences where you can manage your AWS accounts. 

    Once you've completed the wizard, your project is all set up with the AWS SDK for Java, and you're ready to begin coding against the AWS APIs. If you've configured your AWS security credentials, and selected any AWS samples to add to your application, you can immediately run the samples and begin experimenting with the APIs. 

    The Toolkit includes other new project wizards, too. A few months ago, we showed how to use the New AWS Android Project wizard. We plan on demonstrating the New AWS Java Web Project wizard soon.

    What functionality in the AWS Toolkit for Eclipse do you find to be the most useful? Let us know in the comments below.

    Are you are passionate about open source, Java, and cloud computing? Want to build tools that AWS customers use on a daily basis? Come join the AWS Java SDK and Tools team! We're hiring!.

    • March 15, 2013
    • Permalink
    • Comments (4)
  • Jason Fulghum

    Amazon S3 TransferManager

    • March 7, 2013
    • Jason Fulghum

    One of the great APIs inside the AWS SDK for Java is a class called TransferManager that makes working with uploads and downloads from Amazon S3 easy and convenient.

    TransferManager provides asynchronous management for uploads and downloads between your application and Amazon S3. You can easily check on the status of your transfers, add handlers to run code when a transfer completes, cancel transfers, and more.

    But perhaps the best thing about TransferManager is how it hides the complexity of transferring files behind an extremely simple API. TransferManager is essentially two operations: upload and download. From there you just work with your upload and download objects to interact with your transfers. The following example shows how easy it is to create a TransferManager instance, upload a file, and print out its progress as a percent while it's transferring.

    // Each instance of TransferManager maintains its own thread pool
    // where transfers are processed, so share an instance when possible
    TransferManager tx = new TransferManager(credentials);
    
    // The upload and download methods return immediately, while
    // TransferManager processes the transfer in the background thread pool
    Upload upload = tx.upload(bucketName, myFile.getName(), myFile);
    
    // While the transfer is processing, you can work with the transfer object
    while (upload.isDone() == false) {
        System.out.println(upload.getProgress().getPercentTransfered() + "%");
    }
    

    Behind this simple API, TransferManager is doing a lot of work for you. Depending on the size and data source for your upload, TransferManager adjusts the algorithm it uses to process your transfer, in order to get the best performance and reliability. Whenever possible, uploads are broken up into multiple pieces, so that several pieces can be sent in parallel to provide better throughput. In addition to higher throughput, this approach also enables more robust transfers, since an I/O error in any individual piece means the SDK only needs to retransmit the one affected piece, and not the entire transfer.

    TransferManager includes several more advanced features, such as recursively downloading entire sections of S3 buckets, or the ability to clean up pieces of failed multipart uploads. One of the more commonly used options is the ability to attach a progress listener to your uploads and downloads, which can run custom code at different points in the transfer's lifecycle. The following example demonstrates using a progress listener to periodically print out the transfer's progress, and print a final message when the transfer completes.

    TransferManager tx = new TransferManager(credentials);
    Upload upload = tx.upload(bucketName, myFile.getName(), myFile);
    
    // You can set a progress listener directly on a transfer, or you can pass one into
    // the upload object to have it attached to the transfer as soon as it starts
    upload.setProgressListener(new ProgressListener() {
        // This method is called periodically as your transfer progresses
        public void progressChanged(ProgressEvent progressEvent) {
            System.out.println(upload.getProgress().getPercentTransfered() + "%");
    
            if (progressEvent.getEventCode() == ProgressEvent.COMPLETED_EVENT_CODE) {
                System.out.println("Upload complete!!!");
            }
        }
    };
    
    // waitForCompletion blocks the current thread until the transfer completes
    // and will throw an AmazonClientException or AmazonServiceException if
    // anything went wrong.
    upload.waitForCompletion();
    

    For a complete example of using Amazon S3 TransferManager and progress listeners, see the AmazonS3TransferManager sample that ships with the SDK for Java.

    Are you using TransferManager in any of your projects yet? What custom code do you run in your progress listeners? Let us know in the comments!

    • March 7, 2013
    • Permalink
    • Comments (0)
  • Jason Fulghum

    Asynchronous Requests with the AWS SDK for Java

    • February 21, 2013
    • Jason Fulghum

    In addition to the standard, blocking/synchronous clients in the AWS SDK for Java that you're probably already familiar with, the SDK also contains non-blocking/asynchronous clients that are just as easy to use, and often more convenient for certain types of applications.

    When you call an operation with one of the standard, synchronous clients in the SDK, your code is blocked while the SDK sends your request, waits for the service to process it, and parses the response. This is an easy way to work with the SDK, but there are some situations where you just want to kick off the request, and let your code continue executing. The asynchronous clients in the SDK allow you to do exactly that. Kick off your requests, and check back later to see if they completed.

    AmazonDynamoDBAsync dynamoDB = new AmazonDynamoDBAsyncClient(myCredentials);
    dynamoDB.describeTableAsync(new DeleteTableRequest(myTableName));
    // Your code immediately continues executing, while your request runs in the background
    

    Now that you know how to kick off your asynchronous request, how do you handle the response when it arrives? All of the asynchronous operations return a Future object that you can poll to see if your request has completed processing and if a response object is available. But sitting around polling a Future defeats the purpose of freeing up your code to continue executing after you kick off the request.

    Usually, what you really want to do is, when the request finishes, execute some code to process the response. The asynchronous operations allow you to pass in an AsyncHandler implementation, which the SDK automatically runs as soon as your request finishes processing.

    For example, the following piece of code kicks off an asynchronous request to describe an Amazon DynamoDB table. It passes in an AsyncHandler implementation, and when the request completes, the SDK runs the onSuccess method, which updates a UI label with the table's status. AsyncHandler also provides an onError method, that allows you to handle any errors that occur while processing your request.

    AmazonDynamoDBAsync dynamoDB = new AmazonDynamoDBAsyncClient(myCredentials);
    dynamoDB.describeTableAsync(new DescribeTableRequest().withTableName(myTableName), 
        new AsyncHandler<DescribeTableRequest, DescribeTableResult>() {
            public void onSuccess(DescribeTableRequest request, DescribeTableResult result) {
                myLabel.setText(result.getTable().getTableStatus());
            }
                 
            public void onError(Exception exception) {
                System.out.println("Error describing table: " + exception.getMessage());
                // Callers can also test if exception is an instance of 
                // AmazonServiceException or AmazonClientException and cast 
                // it to get additional information
            }
        });
    

    Using the asynchronous clients in the SDK is easy and convenient. There are a lot of applications where processing requests in the background makes sense. UI applications are a great fit for asynchronous clients, since you don't want to lock up your main UI thread, and consequently, the entire UI, while the SDK processes a request. Network issues could result in longer processing times, and an unresponsive UI that results in unhappy customers.

    Another great use for the asynchronous clients is when you want to kick off a large batch of requests. If the requests don't need to be executed serially, then you can gain a lot of throughput in your application by using the asynchronous clients to kick off many requests, all from a single thread.

    Have you tried the asynchronous clients in the AWS SDK for Java yet? What kinds of applications are you using them for? Let us know how they're working for you in the comments below.

    More information on asynchronous programming with the AWS SDK for Java

    • February 21, 2013
    • Permalink
    • Comments (0)
  •  Previous
  • 1
  • 2
  • 3
  • Next 

The search query cannot be empty or single character.

Latest Blog Entries

  • Eclipse Deployment: Part 3 - Configuring AWS Elastic Beanstalk
  • Eclipse Deployment: Part 2 - Deploying to AWS Elastic Beanstalk
  • Release: AWS Toolkit for Eclipse 2.3
  • Eclipse Deployment: Part 1 - AWS Java Web Applications
  • Using Custom Marshallers to Store Complex Objects in Amazon DynamoDB
  • Working with Different AWS Regions
  • The AWS Toolkit for Eclipse at EclipseCon 2013
  • Eclipse: New AWS Java Project Wizard
  • Amazon S3 TransferManager
  • Asynchronous Requests with the AWS SDK for Java

AWS Blogs

  • The AWS Blog
  • Security
  • Amazon SES
  • Java Development
  • Mobile Development
  • Ruby Development

Useful Links

  • AWS SDK for Java on GitHub
  • AWS Java Development Center
  • AWS SDK for Java Getting Started Guide
  • AWS Java Development Forum
  • AWS Toolkit for Eclipse

Archive

SDKs

  • AWS SDK for Java
  • AWS SDK for .NET
  • AWS SDK for Ruby
  • AWS SDK for PHP
  • AWS SDK for Python
  • AWS SDK for iOS
  • AWS SDK for Android

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  • Terms of Use
  • Privacy Policy