Big data is the concept of storing huge amounts of data and refers specifically to making use of that data through analytics and machine learning.
The reason big data is being talked about more than ever is largely due to the fact that computers have never before been powerful enough to work with such large datasets and therefore we can now extract far more knowledge and insight from large data sets than we could do previously.
At the moment big companies are starting to implement big data tactics into their business model – they can collect data from various sources – both internal and external – and then use that data to draw conclusions about how effective their business processes are, how well their products are selling and more interestingly the data can then be extrapolated and used to predict how various different scenarios or changes to the existing business model will affect the company in terms of its revenue, growth and other key metrics of success.
As with most new technologies they start within the business sector and then begin moving into the consumer market once there are tried and tested methods available that can be scaled up and sold to the masses. This is likely to be the same with big data and in fact it is likely that big data will play a massive role in the consumer market and will provide some great benefits to the consumer.
One of the biggest barriers at the moment though is data privacy – the very nature of big data dictates that lots of information should be collected, stored and used and without this part of the equation big data would be useless. This is fine within the enterprise because the data being collected is either owned by the company or being collected with express permission from its owner and also used internally. With the consumer this is a completely different scenario – in order for big data to benefit the individual information pertaining to that individual needs to be collected and stored centrally before it can be analysed and the results made of use to the consumer.
This is a big barrier because people are not familiar with this concept and are used to guarding their personal information very closely.
To give a real world example, the police and other security services are keen to deploy big data technologies in order to collect, store and analyse social media data (source). The police insist that this data will be used for specific purposes to detect and prevent crimes but there is of course the worry that this data will be used for other less legitimate purposes. In addition there are fears that these practices will lead to our right of privacy diminishing.
There are many other potential uses for big data – such as tracking spending habits in order to provide a better shopping experience for the consumer, tracking car usage to provide more accurate insurance quotes or even tracking the food that we buy in order to provide automatic ordering services. There are countless ways in which big data could be used to benefit the consumer because what it really means is getting every available piece of information about what we do, our habits and so on and using that information to provide the best and most suited experience, service or products to us.
Of course none of these things are likely to happen any time soon because of privacy concerns – who wants that much information to be in the public domain or even known by anyone? This poses a serious road block to big data technologies and their further development and highlights a need to modify the law in terms of digital data storage and privacy as well as highlighting the need for education in these areas.
The reality is that data from any of the examples that have been listed can be collected, stored and analysed responsibly. If there were new regulations catering specifically to digital data and big data technologies then measures and laws could be put in place to ensure that whilst this data could be collected, stored and analysed autonomously the results could only be used or even accessed in specific circumstances and to service specific needs.
The use of big data by security services is a prime example – perhaps the law could be adjusted so that all of this data could be collected and analysed but there would need to be specific criteria in place before that resulting information could be used or even looked at by security services. This would allow the technology to be in place and to benefit the security services but would ensure that the data captured would not be available for perusal by anyone unless there was a legitimate reason for doing so. This would mean no privacy concerns for the data and would therefor allow the technology to blossom.