Finding efficient transport as data scales massively -

Finding efficient transport as data scales massively


Data is being procured at an ever increasing rate for analytics, machine learning, and other types of analysis. But Tbytes (or even Zbytes) of data need a transportation system to be useful and although we’ve transitioned from moving data through individual pipes to cloud-based services, we face extended requirements in practice. IoT devices are becoming more prevalent and we need to consider how we will transmit the exponential amount of data before we are suddenly confronted with the need.

Applications such as real-time big data analysis require high bandwidth access to all that data. Standard hardware solutions can prove to be inadequate due to bandwidth and capacity. So, to solve for cost (especially when scaling up), bandwidth, capacity, and long-term reliability, there are two valid options at present. The first is cloud computing over high speed Internet and the second is localized portable storage arrays with high speed fiber connections.

Cloud Solutions

The advantages of cloud solutions include offloading cost and hassle of growth onto a cloud provider. Local machines can be offloaded to a generally reliable cloud system. Enterprise level storage arrays with redundancy and historical backups are commonly available from many cloud providers who offer additional reliability and performance from networks that deploy multiple decentralized data centers. With the growing ecosystem of global datacenters maintaining massive virtual clouds, cloud solutions provide an increasingly robust and convenient way in which to store and transfer large volumes of data generated from new technologies and applications such as IoT, autonomous vehicles and other connected smart devices.

But while a standard Internet connection provides easy access, it also limits bandwidth to Internet connection speeds. And while giving up the hassle of maintaining the system, you also give up control over the performance of cloud machines, being limited to what is provided.

Localized Portable Solutions

With localized portable solutions, performance can be customized to meet application needs, giving great flexibility. Compute load can still be offloaded, this time from a client machine onto the server, and you can set up security protocols so that only local clients can access the system. The main advantage of localized systems is that connection speeds are much higher using high speed fiber network adapters. 10Gb/s connections over Ethernet have begun to break into the consumer level market and even greater speeds are available over commercial grade SFP connections and optic fiber interconnects. Compared to the national average of 96Mbps/32Mbps up/down over the Internet, this offers notably faster speeds. With IoT expected to generate Zettabytes of data annually alongside the growth and development of new technologies such as machine learning that take advantage of these massive datasets, this speed may not be enough for certain applications. In addition to the added total bandwidth, localized solutions also offer much lower latency and can benefit applications that need low response times, such as video editing of high resolution footage and compositing/rendering Hollywood-grade special effects sequences. For added speed and responsiveness, localized storage systems can be customized with solid-state storage drives as high speed data buffers or used in place of traditional hard drive storage media as the primary storage units.

With localized units, transfer and deployment of entire storage systems can be as easy as carrying it to the needed location and hooking it up, as with CP Technologies’ Portable Storage Arrays. These types of storage devices offer a convenient and efficient means of migrating hundreds of TBs of data to a cloud provider. Data can be transferred and stored on the system in a minimal amount of time using high speed fiber connections, then hand carried or shipped to the cloud provider’s datacenter. With such localized solutions, you can transfer TBs of data significantly faster by redeploying in the new location compared to sending all the data over the Internet. And, if you need Internet accessibility, localized solutions can be configured to be Internet accessible. Additionally, by having a discrete portable storage unit, you also maintain a high degree of data security when transferring data from site-to-site, as you can have personnel monitor the unit at all times with significantly reduced for concern over Internet hacking or unauthorized access, since it’d take a physical connection and power to the storage unit in order to access the data while in transit.

The main disadvantage of localized storage is that it is not as robust as decentralized cloud storage. If a flood or some other act of God happens and you don’t have additional off-site machines, then you can lose everything. And capacity and growth will be limited by the size and number of local machines. While localized solutions tend to be very cost effective, additional costs for space, power and cooling need to be taken into account when expanding upon an existing storage solution.

Whichever solution is the best fit for you, something to think about is creating redundancy of data. Decentralized clouds can mitigate the effects of disasters by spreading out their data across multiple data centers. Local systems should be used with an accompanying off-site storage solution to ensure that the security of the data system is decoupled from its geographic location. When transferring machines from site-to-site, having a ruggedized design like those made by CP Tech ensures that data remains safe when in transit.

Michael McCormack is a CEO of Combat Proven Technologies and has extensive domestic and international experience in managing organizations that design and manufacture a wide range of innovative products. His experience covers over 25 years in defense and industrial markets and previous to that served in the US Air Force.


Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.