How Big is the Internet?

Over 13 years ago, we released a study that revealed search engines were missing the vast majority of content online in a portion of the Internet called the Deep Web. The study titled, “The Deep Web: Surfacing Hidden Value”  revealed that the estimated size of the Deep Web was 400-500 times larger than that of the Surface Web and only .03% or 1 in 3,000 Web pages were actually being indexed by a traditional search engine.

A lot has changed in the way the Web works in the last 13 years, so much in fact, that it has become near impossible to replicate the 2001 study because of the sheer size of the Web. In today’s blog post, we are going to take a look at why it’s now become impossible to accurately answer the question: ‘How big is the Internet?’

Continue reading »

Posted in Deep Web and Big Data | Tagged , , , , |

BrightPlanet Announces Partnership with Swan Island Networks

BrightPlanet and Swan Island Networks joint technology partnership to offer a customizable data and security intelligence dashboard for customers.

Sioux Falls, SD – October 28, 2014 –  BrightPlanet and Swan Island Networks are excited to announce a joint technology partnership to offer a customizable data and security dashboard for customers. The partnership allows both companies to leverage each other’s complimentary technologies.

BrightPlanet will be utilizing its harvesting technology to collect and structure publicly available data from the Web to create new data feeds and channels for direct integration into Swan Island Networks’ TIES for Microsoft CityNext and TX360 agile situational intelligence product offerings. The partnership will open up additional channels and analytic opportunities to Swan Island Networks’ public sector and enterprise customers.

Continue reading »

Posted in Deep Web and Big Data | Tagged , , , , |

Announcing Deep Web University Webinar Series: How to Harvest and Structure Big Data from the Deep Web

Over the past few months, we’ve had increased interest in BrightPlanet’s technology and individuals wanting to learn more about how we harvest and structure Big Data from the Deep Web.

To help our readers and followers learn more about BrightPlanet, we are excited to announce our Deep Web University Webinar Series that will feature reoccurring, free, live webinars to help our followers better understand how we harvest and then structure Big Data from the Web for many uses by end users. We will presenting the same webinar multiple times so you can pick the best date and time for you.

Continue reading »

Posted in Case Studies, Deep Web and Big Data | Tagged , , , |

Using Web Data to Visualize and Track Disease Outbreak for Global Security

Last week we highlighted a case study about how an insurance group used our BITS dataset to help a chief actuary improve the underwriting process.  We explained how an actuary from an insurance organization is using a dataset of over 9,000 news sources to increase pricing efficiency. Today, we hope to further explain how customers are using our BITS dataset, but this time we’ll explore how a Chief Security Officer is using BITS to help track the outbreak of disease globally.

Continue reading »

Posted in Case Studies, Deep Web and Big Data | Tagged , , , |

[WHITEPAPER] Open Source Intelligence (OSINT) for Law Enforcement

Earlier this year, to celebrate the 25th anniversary of the Internet, the Pew Research company released the most recent stats about the adoption rates and internet usage of adults living in the United States. See what this means for crime online and law enforcement in this post and in our new whitepaper: Open Source Intelligence (OSINT) for Law Enforcement.

Continue reading »

Posted in Law Enforcement, White Papers and Publications | Tagged , , , |

Using Big Data from the Web to Improve the Underwriting Process

Six months have passed since we made our Big Industry Threats dataset, also known as BITS, available to customers. BITS, our flagship dataset, contains a collection of over 11 million news stories from the past 18 months spanning 9,000 global sources with over 93 languages. Our customers found a number of different uses for our BITS dataset including risk managers tracking disease outbreaks and insurance professionals improving the underwriting process.

In today’s post, we’re going to explain how we helped a chief actuary for a major insurance company improve her data to help increase efficiency and accuracy of pricing homeowners insurance policies for customers overseas.

Continue reading »

Posted in Case Studies, Deep Web and Big Data | Tagged , , , , |

Webinar Recap: How to Harvest, Structure, and Visualize Big Data on the Web

Wednesday, September 10, we held a webinar titled “How to Harvest, Structure, and Visualize Big Data on the Web”. Jamie Martin, a Data Acquisition Engineer for BrightPlanet, and Greg Roberts, CEO and expert linguist at Rosoka Software Solutions, teamed up to take subscribers through the full process of taking data from the Web to final analysis.  We recorded the webinar and also have a brief recap for you in this post.

Continue reading »

Posted in Case Studies, Deep Web and Big Data | Tagged , , |

Outpost: An Online Data Collection Engine for Scalable Open Source Intelligence (OSINT)

Outpost is an online information collection engine that allows intelligence analysts to automate the process of collecting publicly available data from the Web.

As law enforcement agencies continue to utilize Outpost, we find new solutions to problems faced by agencies through the creative use of data collected at scale and in an automated fashion. In this posting, we’ll cover how one agency utilized Outpost to monitor and stay on top of gang networks within a specific metropolitan area.

Continue reading »

Posted in Case Studies, Deep Web and Big Data, Intelligence Community, Law Enforcement | Tagged , , , |

[WEBINAR] How to Harvest, Structure, and Visualize Big Data on the Web

The largest publicly available database in existence is sitting free to use for practically anyone. Consumers use the Internet and the data on it on a daily basis through individual searches, but many companies have failed to successfully exploit the true potential of the worlds largest database. We hope to help solve that problem in our upcoming webinar, How to Harvest Structure, and Visualize Big Data on the Web.

Continue reading »

Posted in Case Studies, Deep Web and Big Data, Intelligence Community | Tagged , , , , |

AuthentiWeb: Using Big Data from the Web to Protect Your Business

BrightPlanet’s AuthentiWeb protects your brands and products using data collected at large scale from the public Internet. You may be asking yourself, exactly how does Web data get utilized to support brand protection and loss prevention? In today’s post, we answer that question. We cover the types of data you can get with AuthentiWeb and how it helps businesses protect their bottom line.

Continue reading »

Posted in Case Studies, Deep Web and Big Data | Tagged , , , |