Support: 1-800-961-4454
Sales Chat

Rackspace Cloud Files: How To Upload Very Large Objects


We frequently update our products, and Cloud Files is no exception. In this series of technical posts, we’ll dig into how developers using the Cloud Files API can leverage new capabilities in Rackspace Cloud Files.

If you need to store very large files (e.g. videos, HD movies or backups), Cloud Files accomplishes this by allowing customers to upload multiple file segments and a manifest file to map those segments together. Large files will be downloaded as a single file.

How It Works

Objects that are larger than 5 GB must be segmented, prior to upload. You then upload the segments like you would any other object and create a manifest object telling Cloud Files how to find the segments of the large object. The segments remain individually addressable, but retrieving the manifest object streams all the segments concatenated. There is no limit to the number of segments that can be a part of a single large object.

In order to ensure the download works correctly, you must upload all the object segments to the same container, ensure each object name has a common prefix where their names sort in the order they should be concatenated. You also create and upload a manifest file. The manifest file is simply a zero-byte file with the extra X-Object-Manifest: <container>/<prefix> header, where <container> is the container the object segments are in and <prefix> is the common prefix for all the segments.

It is best to upload all the segments first and then create or update the manifest. With this method, the full object will not be available for downloading until the upload is complete. Also, you can upload a new set of segments to a second location and then update the manifest to point to this new location. During the upload of the new segments, the original manifest will still be available to download the first set of segments.

Example:  Upload Segment Of A Large Object

  PUT /<api version>/<account>/<container>/<object> HTTP/1.1
  X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb
  ETag: 8a964ee2a5e88be344f36c22562a6486
  Content-Length: 1
  X-Object-Meta-PIN: 1234

No response body is returned. A status code of 201 (Created) indicates a successful write; status 411 (Length Required) denotes a missing Content-Length or Content-Type header in the request. If the MD5 checksum of the data written to the storage system does NOT match the (optionally) supplied ETag value, a 422 (Unprocessable Entity) response is returned.
You can continue uploading segments like this example shows, prior to uploading the manifest.

Example:  Upload Next Segment Of The Large Object

  PUT /<api version>/<account>/<container>/<object> HTTP/1.1
  X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb
  ETag: 8a964ee2a5e88be344f36c22562a6486
  Content-Length: 1
  X-Object-Meta-PIN: 1234

Next, upload the manifest you created that indicates the container the object segments reside within. Note that uploading additional segments after the manifest is created will cause the concatenated object to be that much larger but you do not need to recreate the manifest file for subsequent additional segments.

Example: Upload Manifest

  PUT /<api version>/<account>/<container>/<object> HTTP/1.1
  X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb
  Content-Length: 0
  X-Object-Meta-PIN: 1234
  X-Object-Manifest: container/object/segments

The response’s Content-Type for a GET or HEAD on the manifest will be the same as the Content-Type set during the PUT request that created the manifest. You can easily change the Content-Type by reissuing the PUT request.

This is just one of many new features in Rackspace Cloud Files that developers can take advantage of. Cloud Files customers are urged to try out this new feature in their next API project or add it to a current application. If you’re not yet using Cloud Files, sign up for an account and check it out. And now, using Cloud Files is more affordable since we reduced the cost of Cloud Files by 33 percent.

About the Author

This is a post written and contributed by Jerry Schwartz.

Jerry is a Senior Manager and leads Product Marketing at Rackspace. He joined Rackspace in 2007 and was the Product Manager for our Microsoft applications including SharePoint, Exchange, SQL and the Windows Server operating system. He has worked in Product Marketing with the Rackspace Cloud portfolio since 2010 shaping many of our cloud offerings. Jerry is a loyal Texas Longhorns fan and received his MBA from the University of Texas at Austin. Outside of Rackspace, Jerry and his family reside in Austin where you will likely find him on a basketball court with his son.

  • Jereme Hancock

    Hi Jerry,
    Thanks so much for the info on uploading large files to Cloud Files. I would love to have some basic PHP examples for doing this. I have a project that I am working on and it needs to utilize this feature. Any chance of getting some PHP examples?

    Thanks, Jereme

  • Trey Duskin

    Very well-written howto on handling the object size limit in OpenStack. We incorporated a similar technique into our storage connector appliance so users can copy files of any size to an NFS mount and have it chunked and uploaded properly into Cloud Files, and still have it available for CDN distribution.


    Man i need a 4TB HDD

Racker Powered
©2016 Rackspace, US Inc.