A Needle in the Digital Haystack…

Posted on Updated on

We have shown in previous posts, how, the new data sets that are available can be a key source of trading opportunities. In addition getting access to this information before it becomes ‘news‘ on the mainstream media is highly valuable.

However with there now being over 277,000 tweets, 571 websites created, 347 new blog posts and 72 hours of new video being added to YouTube every minute of every day, the volume of data to read and filter is growing exponentially.

So how do you find and alert people to the most relevant information, amidst all of this ‘noise’?

Utilising proprietary systems, Heckyl has built its own semantic platform, to structure, read and filter the vast data sets available. Taking the principles that have been applied to event driven trading, our system treats every blogger, Twitter user etc. as a data point to be analysed. By back-testing the key events of all of the instruments that we cover – more than 35,000 at last count – over the last five years, we check the related content from these millions of sources for causation and correlations.

This enables us to create a relevancy score for each source, be they journalists, analysts, twitter users or bloggers, as well as understanding the types of information these sources have a history in predicting. This automated process utilising machine learning algorithms means that we are able to filter and rate hundreds of millions of daily data points, in real time, to alert our users to sensitive financial information. We separate the good data from the junk by giving scores to the users as well as the content, where only the scores that meet our threshold is selected by our algorithm.

One recent example can be seen in an alert we generated from the blogger Brad Reese on 7th August this year. Brad is given a high rating by our system in relation to Cisco news and events, with a score of over 80, calculated on the basis of all his previous posts.

Find1.0 (news and data analysis platform) by Heckyl
Find1.0 (news and data analysis platform) by Heckyl

On 7th August he posted a comment about the layoff of 6,000 staff at Cisco’s Data Centre division, by October 2014. The news was officially announced by Cisco on 12th August, which is when the overall sentiment dropped and there was a 5% fall in the share price.

Finding that ‘needle’ amongst the collective noise can prove to be very valuable.


3 thoughts on “A Needle in the Digital Haystack…

    […] highlighted in our last blog, data explosion creates problems with regard to the gathering and processing of information, but at […]


    […] number of areas. For event driven Hedge Funds looking for Breaking News, Heckyl is able to provide highly targeted and filtered access to Tweets in real time. This can provide valuable and timely alerts to end users, before the news is made […]


    […] rumor or news can break out from any source. As highlighted in our previous blog post, A Needle in the Digital Haystack, spotting opportunities from these new data sets is very valuable. A trading opportunity can come […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s