Many users need to define their own dataset rather than blindly trusting one provided to them.  In this post, we give examples of who should be defining their own data set, the value of monitoring your own data, and available custom monitoring solutions.

Who Should Define Their Own Data Set and What Value Does it Bring?

• Marketing professionals monitoring a brand in a specific region

Marketing professionals wanting to track a brand in a specific region must often sift through thousands of irrelevant false hits from all across the globe. Instead, select the specific state and local newspapers and blogs you want to monitor and select custom geographic areas on Twitter to collect tweets from.

• Investment managers monitoring specific companies

Investment managers know certain sources of information are especially relevant when trying to monitor and track specific companies. Take advantage of defining your own sources and datasets by picking sources such as press releases, government regulatory sites (USPTO, FDA, SEC, etc.), news, and investor relations portions of sites.

• Health researchers

Health researchers want to stay on top of what is happening in their field without having to read through every article in their favorite journals and health research news websites. Harvest from journals and news sources you select, and create keywords to alert you to only content relevant to you.

• Any data analyst

BrightPlanet offers near real-time alerting of new and modified content from any web source for those that can identify where and what you are looking for.

Value of defining of your own source with BrightPlanet

• Get More Data

BrightPlanet’s Deep Web Harvesting technology collects content from both the Surface Web and the Deep Web; delivering you more data than a traditional collection technology. To learn more about the Deep Web and the type of content found there download our Understanding the Deep Web whitepaper. 

• Harvest Content, Not Links

When BrightPlanet harvests data we harvest all the text-based content on the page and store that content offline rather than just giving you a link to the URL. This gives you a few advantages that include:

1. Pages that have been harvested and then removed will still have a record
2. Ability to integrate harvested data with 3rd party analytics
3. Ability to monitor pages for changes made to content

• Harvest Content at Big Data Scale

BrightPlanet’s Deep Web Harvester is highly scalable with the capability to issue hundreds of queries (aka searches) to hundreds of websites simultaneously. To read more about the Deep Web Harvester visit here. 

Solutions for Defining Your Own Data Set

Do you need just monitoring and alerting services?

BrightPlanet’s Deep Web Monitor is the perfect fit for you. Pricing starts at $249/month and allows you to define your sources, use your own keywords, and configure email alerts for new and modified content.

To read more about the Deep Web Monitor click here.

Do you need analytics to go with your harvested data?

BrightPlanet’s Silo Services are a good fit. BrightPlanet’s Silo Services include topic specific repositories of data harvested to your needs. Pricing varies according to the size and scope of each project.

To read more about the Deep Web Silo Services click here.

Next Step

If you are ready to define your own data set, sign up for a free consultation with one of our Deep Web Investigators to talk about the deliverable that is right for you.

Photo: bionicteaching

Learn what alerts we can set up, how they’re different than a Google Alert, and examples of how BrightPlanet’s Deep Web alerts can be used.

• Page Changes vs. Mentions
• Customizable Emails vs. Generic
• Near Real-Time vs. Recurring
• BrightPlanet Deep Web Alerts at Work

Page Changes vs. Mentions

Google indexes billions of webpages and sends alerts based on any new pages that may contain a mention of your selected keywords or phrases. Google Alerts does a very good job at trying to monitor the surface web for new webpages. However, Google’s alerting service will never send an alert on pages that already exist within their index, only new pages. This means pages that didn’t previously contain your specific keywords will never be found via Google’s alerting service.   BrightPlanet’s Deep Web alerts are meant to tackle a completely different problem.

Oftentimes you know the pages or website you want to monitor and you want to be alerted when it changes and what is changing.  BrightPlanet’s  technology harvests the content from specific webpages that you choose and sends alerts when any textual changes have been made on those specific pages. The harvests and alerts can be scheduled and sent every minute to every week depending on what you would like.

Examples of why this is valuable:

1. Monitor and track textual changes to your own corporate website
2. Monitor government filings and alert yourself to changes in near real-time
3. Track your competitor’s product literature and be alerted when new features are added
4. Monitor when news stories on sites get changed or updated

If the user prefers, BrightPlanet’s technology can also monitor specific parts of pages; the entire page does not need to be harvested every time. For example, if you just wanted to track the “Events” section of you favorite organization’s website, you could receive email alerts every time just that section was changed.

Customizable Emails vs. Generic

Google Alert emails have a generic layout that is used for every user. The alerts sent by BrightPlanet are customizable. Everything from the title to the content can be customized for the specific user.

At its simplest, BrightPlanet’s alert emails include past versions of the monitored page’s content and the new versions of the pages content with the specific content that was modified highlighted. This ability to pick specific sections of pages to harvest allows you to be alerted if specific relevant portions of pages are modified.

Near Real-Time vs. Recurring

Google Alerts are sent on a daily basis and BrightPlanet’s Deep Web alerts can be sent in near real-time. An email alert can be sent within 60 seconds of a page change. If the user prefers, the alerts can be sent on a recurring basis they specify, but the technology has the ability to send them in near real-time unlike Google Alerts.

BrightPlanet Deep Web Alerts at Work

To illustrate the use of BrightPlanet’s Deep Web alerts we set up the technology to monitor changes made to an inventory of a car dealership.

Car Dealership Inventory

We set up our Deep Web alerts to harvest the inventory page of a car dealership. The campaign was set up to send daily alerts showing the changes that have been made to the page. A competitor could use this to know what has been added to the lot and purchased.

Below you will find screengrabs of email alert portions. The top screengrab shows the text for a car that was removed highlighted in red; presumed sold. The bottom two screengrabs show cars that were added to the inventory page highlighted in green.

This is a very entry-level Deep Web alert. The alerts just contain the text from the specific page with the modifications highlighted. As we mention above, these e-mail alerts can be customized based on your needs.

Want to learn more?

If you are interested in learning more about the potential to implement Deep Web alerts in your organization, sign up for a free consultation with one of our Deep Web Investigators.

Photo: AJC1

Discover what we do, why it’s different than a Google search, and why it’s valuable to customers.

• Harvesting vs. Searching
• Solving Specific Problems vs. Being Everything to Everybody
• Specific Sites vs. The Entire Web

Harvesting vs. Searching

Unlike traditional search technologies like Google, which index links and allow users to search those results, BrightPlanet takes it  a step further and harvests all the content. The harvest process involves BrightPlanet extracting all of the text content from each of the results pages, and then preparing the harvested content for some type of analysis depending on the needs of the end-user.

Search engine technology indexes billions of webpages over time, therefore allowing users to search an already assembled index. Search engines don’t find or store all the content on a web page; they simply link you to the content’s location. The amount of data stored from each page is a crucial difference between search engines and content harvesters.

Our Deep Web Harvester extracts every single word every time it accesses a webpage. Additionally the Deep Web Harvester stores every single page harvested as a separate version.

Solving Specific Problems vs. Being Everything to Everybody

You may be wondering why you would need every single word to be extracted every time? Search engines like Google are really good at finding Surface Web websites; providing answers to basic questions like “How late is Burger King open?” quickly. However, companies and organizations have significantly harder questions that can only be answered by web data beyond the scope of a search engine. Complex questions, like those listed below, require more than a search engine; they require a Deep Web Harvester.

• Who is selling my products fraudulently online?
• How many people have won grants on Fetal Alcohol Spectrum Disorders?
• What are clinical trial patients saying about my experimental drug?
• What new information has been published on my competitor’s website today?
• Has anything changed in this insurance coverage plan that would affect a pharmaceutical company’s stock price?
• What new breast cancer research has been published in the last month? What are people saying about it?

Put these questions in the Google search bar and you will not get the answer you are looking for. Our technology can help answer these questions by harvesting Big Data from the Deep Web, connecting the results, and delivering them in the way that is useful for the end-user.

Specific Sites vs. The Entire Web

Instead of searching the entire web, we perform directed harvests on specific websites. End-users can define hundreds or thousands of Deep Web sources for BrightPlanet to query with many keywords at once.

If an end-user does not know the exact websites they want to search, they can just identify the information they are trying to find or problem they are trying to solve and BrightPlanet will help them find and define the web sources. Sources can continue to be added as the project evolves and the end-user decides which data is most useful.

A Deep Web Solution For You

Looking to go beyond a Google search and answer more complex questions with data from the Deep Web? Looking to stop searching site-by-site on a regular basis for a project? Learn more about the Deep Web and ways different end users have utilized our Deep Web solutions for their organizations in our Understanding the Deep Web in 10 Minutes whitepaper.

Sign up to schedule a free demo with one of our Deep Web Investigators if you have a problem you think solutions from the Deep Web might be able to help you solve.

Photo: Jeffrey Beall

Gartner selected companies that offer “capabilities that enable information producers and consumers to describe, organize, share, integrate and govern any type of data and content, anytime, anywhere.” BrightPlanet was recognized for its solutions that harness content from the Deep Web.

“We are very excited that Gartner has included BrightPlanet on the list of Cool Vendors in Information Infrastructure and Big Data. It is great to see Gartner recognize data harvesting technologies in this category, and we hope that companies continue to see the value of leveraging unstructured Big Data from the Deep Web,” said Steve Pederson, BrightPlanet CEO.

BrightPlanet Deep Web Solutions

Since its inception 13 years ago, BrightPlanet has provided Deep Web solutions for the U.S. intelligence community. Now they are taking those solutions to companies and organizations that need help in harvesting and analyzing Big Data from the Deep Web. BrightPlanet’s services are now being deployed to serve the commercial sector.

BrightPlanet’s Deep Web Silo Services combine three key elements:

• Deep Web Harvester™
• Big Data Intel Silos
• 3^rd Party Analytic Integration

The Deep Web Harvester is the most comprehensive tool available for Big Data acquisition. The Harvester lets users find content that standard searches engines simply cannot access. BrightPlanet’s Deep Web Intel Silos are individual repositories for topic-specific content. The Silos are updated with relevant information from the Harvester in real-time. Deep Web Investigators are BrightPlanet’s in-house experts at finding Deep Web Sources and navigating and investigating the Deep Web in order to create intelligence through collaboration with customer analysts and investigators.

If you are interested in more information or have questions about Deep Web Silo Services from BrightPlanet, click here to request a free demo with a Deep Web Investigator. BrightPlanet offers additional Deep Web solutions which can be found here.

BrightPlanet Corporation

For over 10 years, BrightPlanet has been the leader in finding, harvesting and enriching Big Data from the Deep Web to help U.S. intelligence agencies fight the “War on Terror”.  The company is now offering services to meet the needs of commercial intel interests, providing Deep Web, topic specific Intel Silos for research and analytics.  Learn more at www.brightplanet.com.

¹ Gartner Cool Vendors in Information Infrastructure and Big Data, 2013, Ted Friedman, Andrew White, Roxane Edjlali, Eric Thoo, Frank Buytendijk, Regina Casonato, Donald Feinberg, and April 30, 2013

Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose

1. Harvesting
2. Normalizing and Enrichment
3. Reporting and Analytics

Download our latest whitepaper to discover how Big Data from the Deep Web becomes usable.

In the whitepaper you will learn:

• What the data looks like?
• Where the data is coming from?
• How to make the data into something usable?

Actionable intelligence from data isn’t available at the click of a button, and manually making data actionable at a large scale is not feasible. Learn what the process is and the tools that are available to make the process efficient and effective.

Using BrightPlanet’s harvesting technology, we identified a total of 772 Twitter handles run by state and local law enforcement departments. Throughout the study, BrightPlanet harvested over 115,000 different tweets from the 772 department Twitter handles.  We analyzed both the raw text of all the harvested tweets and the data from the profiles of each department handle we identified.

For the purpose of our study, we focused on identifying Twitter handles from police departments that had their description information in English. We’ve created an infographic to help you see the data.

Scroll down to uncover the most popular topics, the most active Twitter accounts, and other characteristics that show exactly how police departments utilize Twitter.

Glenn’s career includes supporting several Defense Advanced Research Projects Agency (DARPA) programs, co-founding the Templar Corporation, an organization that provided software to facilitate the exchange of information between public safety and homeland security agencies, and serving as a Founder, President of the Board and Executive Director of the IJIS Institute, an Asburn, VA based non-profit representing approximately 170 companies that specialize in delivering technology and services to the homeland security and public safety markets at the federal, state, local, and tribal level.

Glenn currently serves as the Principal of GLA & Associates, LLC, an Alexandria VA based consulting firm and volunteers as the Deputy Director of the National Fusion Center Association, representing the national network of fusion centers.

Learn more about Glenn here.

Get more information about BrightPlanet’s Deep Web solutions here and check out our free resources on intelligence in the Deep Web.

BrightPlanet Corporation

For over 10 years, BrightPlanet has been  the leader in finding, harvesting and enriching Big Data from the Deep Web to help U.S. Intelligence Agencies fight the “War on Terror”.  The company is now offering services to meet the needs of commercial intel interests, providing Deep Web, topic specific Intel Silos for research and analytics.  Learn more at www.brightplanet.com.

Photo: cliff1066™

Social Media as the Only Communication

Many reported that phone lines were down and cell phone calls were not going through on Monday afternoon but people were able to send text messages and update their social media accounts. Both the Boston Police and Massachusetts Emergency Management Agency were using Twitter to get messages out. The Boston Police used Twitter to put out a call for any video from the finish line on Monday afternoon.

Google even created a people finder that people could use to find loved ones and friends.

Big Data and the Marathon

USA Today had an article on Tuesday about how Big Data could uncover clues on what happened at the marathon. The marathon finish line is a place where people are taking photos and video throughout the event. These images and the people behind them could be used to piece together what happened there.

Twitter is a powerful tool for witness identification and geolocation. We used our Twitter monitoring tool, BlueJay, to capture the tweets being tweeted out of the Boston area. Since we started tracking the tragedy, BlueJay has harvested over 37,000 tweets including the words “bomb”, “blow up”, “explosion”, “explosive”, or “marathon” in the Boston area.

Below is a screenshot of the tool showing tweets and their location shortly after the third explosion at the library occurred.

Twitter captures what is happening in real-time and is a powerful tool for law enforcement and public service agencies during large public events.

There are no words to describe the tragedy that occurred on Monday but hopefully those in charge of the investigation will have the resources they need to find those responsible.

Twitter Monitoring

If you are interested in learning more about our Twitter monitoring tool for law enforcement, please visit the BlueJay page or sign up for a 30 day free trial of the tool.

Sources:

Boston Marathon: With no phones, text and social media help get out updates – CBS News

Big data could uncover clue on marathon: Column – USA Today

Photo:

stevegarfield

Defining OSINT

OSINT stands for “Open-Source INTelligence”, which refers to any un-classified information and includes anything freely available on the Web.  OSINT is the opposite of close-source intelligence or classified information.  Common OSINT sources include social networks, forums, business websites, blogs, videos, and news sources.

OSINT is one of many INTs disciplines employed by government intelligence agencies.  Other common INTs include HUMINT (Human Intelligence; gathering intelligence by interacting with people); SIGINT (Signal Intelligence; gathering data from sensors); and GEOINT (Geospatial Intelligence; gathering data from satellites).

OSINT is unclassified and available, but it is not always easily found. Much of it is available on the Deep Web or “hidden internet” as some like to call it. The information is publicly available, but link-crawling search engines like Google do not always access it. Check out our recent post to learn what search engines like Google are missing and why.

OSINT is not related to open-source software like Linux.

OSINT vs. Big Data from the Deep Web

Big Data from the Deep Web is OSINT. Whatever you want to call it – online Big Data, OSINT, or open-source intelligence – your organization can benefit from exploiting that information. There is information publicly available online right now that you are missing by searching with Google, or not searching for at all because you aren’t even aware it exists.

Want to find counterfeiters of your product online? You can use OSINT to track them down.

Want to monitor real-time activities for your current financial investments or possible future investments? That information is publicly available on the Deep Web and can be harvested and delivered to you in real-time.

Want to know all the available grants for your area of research, who is offering them, and who has won them in the past? Again, that is OSINT and ready to be harvested.

Getting Started with OSINT

OSINT, Big Data, unstructured data and the Deep Web can all be nearly impossible to navigate without the proper tools. Even if you find the data, harvesting the data at scale, analysis, and visualizations often require additional licenses, subscriptions, and a bigger budget.

BrightPlanet has tools to harvest OSINT at scale, curate, and create understanding from the information, all within one platform. We have been working with the U.S. intelligence community for over 10 years and are now serving the needs of companies and organizations that need help in harvesting and analyzing OSINT from the Deep Web.

Not sure what OSINT needs you have? Sign up for a consultation with one of our Deep Web Investigators to see what information is out there and ready to be harvested and harnessed to make your organization more efficient and effective.

Learn more about the Deep Web by downloading our free whitepaper.

Sources:

http://www.topsecretwriters.com/2010/09/the-us-government-open-source-intelligence-repository/#more-907

http://en.wikipedia.org/wiki/Open-source_intelligence

Photo: jurvetson

Today we are going to learn about the analytics.

The first part of analytics BrightPlanet offers to the end user is BrightPlanet’s Intel Silo Dashboard. BrightPlanet’s dashboard offers a number of different features that help you find the answer your team is looking for. The dashboard focuses heavily on search capabilities.

Advanced Search Capabilities

Faceted searching is the ability to break down results of a search into specific categories to more easily find what you are looking for – more than likely you use faceted searching on a daily basis. On shopping websites you can sort shoes by their brand, appliances by their color, and cars by their price range. On search engines like Google and Bing, you can refine your results by the date they were indexed, by the language the page is in, and the type of web page (news, blog, etc). Imagine being able to do faceted searching on any type of web data set with any type of categories. With BrightPlanet and Rosoka, you can.

The ability to pair Rosoka custom entity tagging technology (check out previous post to revisit that technology) with faceted search technology significantly increases the value of the extracted data for BrightPlanet users – it gives users the ability to tag documents based on any category they wish.

Pharmaceutical companies want to see which domains are selling a certain drug.

• Custom Tag: Drugs / Domains / Pricing / Dosage / 

Health Researchers want to see the top companies and people receiving grants (Shown right).

• Custom Tags: Grant Type / Companies / People
  

Human Resource organizations want to see the top companies hiring Java Developers.

• Custom Tags: Hiring Company / Job Qualifications / Job Titles

OpenPlanet Platform – When you need more than just search

Many customers require access to harvested content using faceted search and light analytics features that the Deep Web Intel Silo Dashboard can offer. However, some customers need more than just advanced search capabilities – they need Big Data analytics. For those customers, BrightPlanet has developed a platform known as the OpenPlanet Platform.

The OpenPlanet platform is based on a simple workflow that completely separates the harvesting and analytic components of data collection and analysis. This concept allows BrightPlanet to easily swap in different analytic technologies with no knowledge of where the data came from previously. This allows customers to integrate multiple datasets, not just harvested web data, with multiple analytic technologies in one workflow without significant development.

Discover some of the analytic technology possibilities in the next section.

OpenPlanet Platform Partner Highlight

Content Analyst

Content Analyst is a company that specializes in text analytics. Their main technology is Latent Semantic Indexing, which focuses on extracting concepts from full bodies of text for comparison and analysis. Paired with BrightPlanet data, Content Analyst’s technology allows for interesting applications.

Rather than doing searches in a traditional fashion, picking random keywords and hoping you are using the right keyword to get the correct results, users can now take full concepts (a sentence, a paragraph, a whole paper) and use that as search criteria to find documents.

Some Example Applications

Identifying Prior Art

-Use all the claims of your patent application as search criteria across harvested product literature and current patents to quickly identify relevant prior art.

Find other harvested content relevant to your research

-Use your paper abstract as search criteria to find other journals and documents that are relevant to your abstract to quickly further your research.

Identify Potential Infringements

 -Compare competitor’s product literature to your own product literature to identify potential infringements.

LucidWorks

LucidWorks Big Data provides an all-in-one search, discovery, and analytics platform aimed at making it easier for organizations to develop next generation data driven applications. The platform combines the power of Apache Solr open source search technology with the Big Data capabilities of Apache Hadoop.  LucidWorks Big Data also incorporates scalable machine learning, natural language processing and statistical analysis techniques to help users better find and organize their data.

Through a BrightPlanet and LucidWork platform that uses the capabilities of Apache Hadoop, you can now have an infinitely scalable solution with your Open Source harvesting from BrightPlanet and Apache Hadoop Framework.

Centrifuge Visual Network Analytics

Centrifuge Systems is a technology company which focuses on visualization of data. Centrifuge’s technology allows BrightPlanet users to visualize the data and the custom tagged entities using Rosoka’s tagging technologies. Rather than trying to make sense of the data by viewing results and reading, BrightPlanet can help you uncover relationships with Centrifuge’s link analysis capabilties.

Some Example Applications

Finding Collaborators for Research

-Visualize research grants and quickly identify other researchers and institutions performing similar research for collaboration with link analysis connecting people and institutions.

Identify Targets of Fraud

 -Visually see which domains are selling your products to identify fraud with link analysis connecting products, domain names, and WhoIS information.

Looking for a Big Data Solution?

Whether every word in this post made sense to you or you just know you are interested in what Big Data could mean for your organization, sign up to schedule a free demo with one of our Deep Web Investigators to see what might work for you.

Also, download our free whitepapers on Big Data and the Deep Web.

Photo: Search Engine People Blog