 | Issue: | Europe II 2014 |
| Article no.: | 4 | |
| Topic: | Can networks cope with the explosive demands of Big Data? | |
| Author: | Rashik Parmar | |
| Title: | President, IBM Academy of Technology | |
| Organisation: | IBM | |
| PDF size: | 221KB |
About author
Rashik is WW President of IBM’s Academy of Technology and a Distinguished Engineer. During his twenty nine years with IBM, he has worked for financial, retail and manufacturing clients on IT projects of all sizes. Today, he specialises in ensuring the technical success of complex IT projects around IBM’s Smarter Cities programme and in the development of techniques to drive industry level innovation.
Rashik was appointed to the Leeds City Region Local Enterprise Partnership Board and chairs the business communications group. He is helping develop the innovation strategies to accelerate growth of this £55bn marketplace.
Rashik is IBM’s Partnership Executive for Imperial College London. He is also an Adjunct Professor for the Department of Innovation and Entrepreneurship at the Imperial College Business School and Visiting Professor to the Intelligent Systems and Networks Group at the Department of Electrical & Electronic Engineering.
Article abstract
Only when raw data is combined and contextualized, it becomes valuable information. New technologies enable this to happen, utilizing large stores of raw data. One such example is DeepQA, which is a computer system that can answer natural language questions about a broad range of knowledge, more quickly and accurately than human experts, as proven in 2010. Advances in software defined environments allow sophisticated algorithms to manage network and computing resources, which will ensure that the mass of data is made available where it is needed. Data has become the world’s new natural resource and SDN networks will play a pivotal role in its transmission and distribution.
Full Article
It is estimated there are approximately 4 zetta (1021) bytes of data today and this total is increasing at 50% per annum. This growth is driven by the proliferation of digital devices and technologies such as sensors, embedded processors and even security cameras. Data is being gathered and analysed at an unprecedented pace in an attempt to find new ways to optimise or increase the effectiveness of systems such as healthcare, transportation and energy.
Just stop to consider these examples around us: the typical aeroplane engine has over a 1,000 sensors to constantly monitor performance; three billion smart phones are being used by individuals to instantly share their world with family and friends; billions of RFID tags are used to track everything from the movement of goods to the performance of athletes; the sensors from the typical Formula 1 car will create 25 megabytes of data in one lap; over 100 hours of video is uploaded to YouTube every minute. We are now entering a world in which everyone, everything and every organisation will be constantly creating data.
You might wonder, with so much data available, why is there still so much waste and inefficiency in all aspects of our daily lives? Congested roadways annually cost the US economy US$78 billion, from 4.2 billion lost working hours and 2.9 billion gallons of wasted petrol — and that’s not counting the impact on air quality. Inefficient supply chains cost US$40 billion annually in lost productivity — more than 3% of total sales. Our healthcare system really isn’t a ‘system’. It fails to link diagnoses, drug delivery, healthcare providers, insurers and patients — at the same time as costs spiral out of control, threatening both individuals and institutions. One in five people living today lacks safe drinking water, and we’ve seen what happened to our financial markets, a system in which institutions were able to spread risk, but not track it.
The challenge is that data alone is of little value. Only when raw data is combined and contextualized will it create valuable information and help provide answers. The challenge ahead is how to derive value from the data in a timely and affordable manner. Advances in cognitive computing technologies are already demonstrating that we will soon be able to address many of society’s challenges.
So what is cognitive computing?
Cognitive computing refers to a new breed of computer systems that can learn for themselves rather than needing to be programmed. Attempts to create computer systems that are able to think go back as far as the 1950s, when Alan Turing defined his famous test to assess if the conversational capabilities of a computer system could be indistinguishable from a real human being.
Advances in programming techniques through languages such as Prolog and Lisp in the 1980s showed promise in being able to describe complex logic and allow computer systems to derive results that were not explicitly programmed. In parallel, advances in neuro-linguistic programming have provided modelling techniques that allow advances in text analytics and natural language processing. However the breakthrough came in 2010 when, using the US game show Jeopardy as a stage, IBM’s DeepQA system (AKA Watson) was able to beat the world champions in a quick-fire, general knowledge quiz.
DeepQA is a computer system that can directly and precisely answer natural language questions over an open and broad range of knowledge. It is different from traditional computing systems in that is it not programmed to answer a specific set of pre-defined questions. The DeepQA system is able to dissect any question and identify a series of possible responses from a pool of structured or unstructured data. Using a complex technique of evidence scoring, the system ranks the possible answers and selects the most appropriate responses. Today DeepQA has advanced question-answering technology to a point where it now clearly and consistently rivals the best human performance.
Applying DeepQA techniques opens a wide variety of opportunities to assist knowledge workers in making complex decisions. Early trials with doctors have already demonstrated how it can help improve the treatment of patient conditions. For example, in 15 seconds the system is able to cross-reference the records of a million cancer patients to help assist in identifying the right care plan.
However, the DeepQA system can only be as good as the information available to it, so creating a comprehensive contextual repository represents both a computing and networking challenge. Data from the vast range of sources needs to be collated and understood. At the same time anomalies need to be identified and assessed to determine their cause. Next, contextual information needs to be created so that DeepQA systems have the evidence to support their answers. Early analysis has shown that for every byte of unstructured text, 10 bytes of metadata and a further 100 bytes of relationship data need to be created. Moreover 1000 bytes are needed to create context and meaning in order to mimic human understanding.
Being able to move data to a place where it can be of value is one of the fundamental reasons for data networks to exist. With the advent of the Internet and standards it is much simpler to make use of data for purposes previously thought of as impossible or unimaginable.
DeepQA systems will drive increased demand for network capacity. Not only will a vast river of data flow from sensors to the DeepQA systems, but additionally interactions between DeepQA systems owned by separate organisations will also generate vast data streams. Furthermore, these systems are likely draw from Open Data sources in order to create the comprehensive information pools needed.
Predicting all the interaction patterns that the networks will need to support will be challenging. However, it is clear that increases by orders of magnitude in network capacity are inevitable. New consolidated network, computing and information storage architectures will be needed. Advances in software defined environments that allow sophisticated algorithms to manage the use of network and computing resources are essential in order to optimise and cope with these demands.
Consider the simple example of a vehicle, that is running low on fuel, travelling along a road . Today, the best we could hope for might be a warning light to appear on the dashboard and a GPS unit to highlight the three nearest petrol stations. In a highly connected world the three fuel stations could be made aware of the opportunity for fuel sales and the telecommunications service provider could ask them to bid to place an advert onto the vehicle’s information console. In this scenario, the network traffic would have increased significantly to 8, 16 or even 32 interactions. In addition, interactions with other service providers such as those offering loyalty schemes, social media sites etc. could increase network traffic even further.
Forecasts from industry analysts are already predicting 1,000 exabytes of data will exist by 2015. This poses an additional challenge of not only coping with the real-time data as in the example above, but also in drawing insights from the vast pools of data that have been captured. Simply increasing the amount of network capacity alone is unlikely to address these demands. Our collective ingenuity to create new beneficial uses for data will outpace our capability to provide increased capacity.
The strategies open to the networking providers are a combination of pricing policies and innovation through collaborations with IT providers. Pricing policies, although unpopular, can be used to manage demand within the available capacity. This strategy requires a balance between ensuring client satisfaction and maintaining competitive advantage – something that the industry is already very familiar with.
Collaboration with IT providers can create more interesting opportunities. Placing storage and computing capabilities within the network can reduce traffic flows. So in our example, by creating cloud computing nodal points through prior agreements between the fuel providers and associated organisations, it would be possible to reduce the number of network interactions to just two by making a single request to an aggregated cloud service.
Whilst collaborations such as these can appear daunting at first, the market opportunity is large enough to drive innovators to make it a reality. The pressure from frustrated citizens and customers will only serve to accelerate the change. We have to live up to the realisation that data has become the world’s new natural resource and consequently networks will play a pivotal role in its transmission and distribution.
