Insightpaper Building a Smarter Private Research Library With AI

Have you ever seen a digital mountain of potential readings? You may have a fabulous report from that one tiny conference, three conflicting studies on the same topic, and a 90-page white paper that you’ll read one day but just not today. We all have bookmarks in our browser that scream: “I am in a white-knuckled death grip due to all the information!” And our “download” folders are nothing more than a graveyard of well-intentioned research that is now too hard to find when we need it. If this is where you store your private ‘research library,’ know that you’re not alone. All of us who are curious or are doing some type of research/professionally have been collecting our own private ‘libraries’ for a while, and it’s almost gotten to the point that now our libraries look more like attics than libraries – full of potential, but difficult to navigate through them when we want/need that one specific piece of information. But, it doesn’t have to be; it can! Your private ‘library’ could be your intelligent partner.

The long-anticipated “personal digital library” is being revitalized through the latest addition of advanced technology. The new technology does not merely enhance the way we organize information through folders or tags. Rather, the advance will apply artificial intelligence technology to the very basis of how we gather, comprehend, and engage with the knowledge we find beneficial. By combining these technologies, our transition from a passive archive to an active, intelligent researcher that exists within our data will be realized.

From Dumping Ground to Thinking Space

For many years, creating a private research collection was a tedious process. You would have saved your document with a name that you might remember and then placed it somewhere in a structure that was created six months prior, and you would have to work through either of those potential methods of organization, if at all, in most cases. If you had saved a news article referencing quantum computing, and you downloaded an academic paper referencing the same topic 12 months later, the only thing between those two documents is your memory, assuming you remember both documents.

The traditional concept of an assistant will never be the same thanks to artificial intelligence (AI). AI can provide real-time assistance, as it watches “over your shoulder” as you work. For example, if you bring in a PDF related to climate policy, the AI will have already processed the entire document by the time you’ve figured out where to store it. Instead of simply identifying the filename, the AI will have analyzed the document (i.e. the document will actually be read), extracted key concepts, identified major argument points, and extracted important numerical data. This represents a fundamental shift in how we will think about our libraries – instead of being a physical cabinet full of documents waiting to be catalogue, our library will soon be a dynamic network of interrelated concepts.

The Magic of Unseen Connections

This is when the fun starts! When your AI library understands the content of your documents the real magic happens; discovering connections that were not previously requested by you. For example, if you were writing a blog post about urban transport and queried your library for “electric vehicles”, it would return all the reports stored but in a side bar provide a suggestion related to the topic; e.g., a policy paper on ‘smart city grid management’, discussing battery storage issues associated with EV infrastructure – a connection you never would have thought to look for while searching through your notes!

The essential part of a smarter library functions like a highly capable colleague with a photographic memory. The smarter library has the ability to find theme-based connections between various types of files, (e.g. a critical piece of text in a lengthy PDF, and/or a specific time stamp of a podcast that incorporates that text, as well as times with significant meaning noted in a webinar transcript.). The smarter library provides linkages between disparate types of files and establishes a cohesive knowledge web. The AI library can retrieve any “memory” that you have of “someone said something about that”.

Conversations with Your Own Data

The most “sci-fi” thing about this evolution is that rather than inputting a string of search terms into a search engine, we will be able to interact with our libraries through natural language dialogue rather than through search interfaces. For example, instead of entering text such as “AI ethics 2022 pdf,” you can simply ask your library, “What were the major opposing views about mitigating bias associated with artificial intelligence in the papers I collected during 2021?”.

Your AI librarian does more than search for just keywords; instead, it can combine the various perspectives of your entire collection to help provide a bigger picture and summarize the main arguments within the documents in addition to letting you know where one document may not align with the other. With an AI librarian you’re doing more than simply obtaining a document, but rather interacting with the cumulative intelligence of your curated library. Thus, this makes research about having an intellectual dialogue rather than only collecting documents, such as Insightpaper is helping change from simply being an on-hand resource for documents towards being able to develop insight from the information provided in those documents.

Beyond Text: The Multimedia Brain

A contemporary researcher’s eating habits don’t consist of printed material alone; they consume video, audio, and visual content through podcasts, infographics, and social media posts. For libraries to provide intelligent access to resources, the library must allow for omnivorous interaction with content. Thanks to advancements in machine learning and artificial intelligence, researchers can now take audio or video recordings and convert the resulting file(s) into searchable text, analyze the visual content from infographics and slides, and extract any meaningful text from images. For example, all the content from that amazing YouTube lecture series becomes equal to a textbook in searchability and linking once it’s processed. A complex graph from a presentation does not get indexed or stored as an image called “chart.png”; it will be indexed as a visualization of data about “market growth trends 2019-2024”. The library’s understanding of what it holds will be multi-modal, similar to how we receive and use information in our daily lives.

The Human Stays in the Loop (This is Crucial)

With so much discussion happening around Artificial Intelligence, it’s important to clarify what an Intelligent Research Library is NOT. An I.R.L. is NOT an autonomous being that thinks for you; it’s NOT a black box that makes illogical or mysterious decisions about your data. The best implementations put the human being in charge of everything that’s happening. An I.R.L. will surface connections /suggest possible tags / provide possible summaries; answer the questions you have; and provide a way for you to be able to determine what is relevant and to determine how to connect those things. It’s up to you to reject irrelevant links, and to highlight connections that are most important to you; and to create value from your information.

When consistently linking two ideas together through your own notes, the AI learns to recognize how significant that linkage is to help train the AI. Similarly, by correcting or refining your notes, the AI is able to adapt to your unique way of thinking, making it less of a standard tool and more like an extension of yourself. The benefit of having both machines that handle large amounts of data and patterns through pattern recognition and human minds that provide insight, curiosity, and creativity is where the real power of collaboration exists.

Building Your Own Brain Trust

The first step is to change how you think about the documents you collect. Rather than thinking about these as document ‘files’ that need to be filed away, think about them as a dataset that will help you understand the patterns, trends, and relationships among your documents. Also look at platforms and tools that are designed with AI-first principles in mind. Some features you might want to look for include: automatic summarization, semantic search (the ability to search for content based on its meaning as opposed to just keyword searches), and the capability to ask questions in ‘everyday’ or plain language.

Start out slow and simple, by giving it just a few dozen of the documents that are among the most important to you. Have fun with it and see how it connects to different things and how well you can use it. Your primary goal is to build an ongoing archive, as opposed to trying to build the perfect archive on day one.

Creating a more intelligent private library for research using Artificial Intelligence (AI) is really about enhancing thought. This means that you will not have to remember or organize anything yourself; instead, you can use a program, such as AI, to do these two tasks. This gives your mind freedom to do the things it was designed to do best: provide insight, synthesis and creation. By allowing AI to store your past collections, you can benefit from those collections as if they were made by your current self. The accumulated information you have at home will become an organized collection of knowledge available to you as if it were stored in a library. An organized collection of knowledge is an invaluable tool for creating new things.