In this guide, let’s delve into preparing a dataset of Wikipedia articles for search, utilizing the OpenAI API. The process involves <\/p>\n
- \n
- downloading Wikipedia articles<\/strong> related to a particular topic, <\/li>\n
- chunking the downloaded documents<\/strong> into smaller sections suitable for embedding, <\/li>\n
- generating the embeddings using OpenAI<\/strong>, and finally, <\/li>\n
- storing the embedded sections<\/strong> for future use.<\/li>\n<\/ul>\n
In case you need a quick refresher on OpenAI embeddings, feel free to read through our Finxter article here:
![\"?\"](\"https:\/\/s.w.org\/images\/core\/emoji\/14.0.0\/72x72\/1f447.png\")
<\/p>\n\n![\"\"](\"https:\/\/blog.finxter.com\/wp-content\/uploads\/2023\/09\/image-291-2-1024x569.png\")
<\/a><\/figure>\n<\/div>\n![\"?\"](\"https:\/\/s.w.org\/images\/core\/emoji\/14.0.0\/72x72\/1f4a1.png\")
Recommended<\/strong>: What Are Embeddings in OpenAI?<\/a><\/p>\nPrerequisites<\/h2>\n
Before starting, ensure that you have all the necessary libraries installed. The libraries include
mwclient<\/code> for fetching Wikipedia articles,mwparserfromhell<\/code> for parsing Wikipedia articles,openai<\/a><\/code> for creating the embeddings,pandas<\/code> for data management,regex<\/a><\/code> for text cleaning, andtiktoken<\/a><\/code> for analyzing token counts.<\/p>\nUse
pip<\/code> as shown below:<\/p>\npip install mwclient mwparserfromhell openai pandas regex tiktoken<\/pre>\n
If you’re using Google Colab, you can run the following:<\/p>\n
pip install mwclient mwparserfromhell openai pandas regex tiktoken<\/pre>\n
Restart your Python environment after installing the libraries to incorporate the changes effectively. As a reference, we used this guide<\/a> from the Cookbook as a starting point for this article. <\/p>\n
Copy and Paste Code<\/strong>: If you want to get the whole code for copy and paste, scroll down <\/strong> to the end of this article where I have refactored the code to one big copy and pastable code snippet <\/strong>you can run in Google Colab, for example.<\/p>\n4 Steps to Create an Embedding From a Set of Wikipedia Articles<\/h2>\n
\n![\"\"](\"https:\/\/blog.finxter.com\/wp-content\/uploads\/2023\/09\/pexels-photo-3526022-1024x683.jpeg\")
<\/figure>\n<\/div>\nThe following is a step-by-step guide on how to prepare a dataset of Wikipedia articles for search:<\/p>\n
Step 1: Collect Documents<\/h3>\n
First, you need to download your set of Wikipedia articles related to your chosen topic. For this tutorial, we will download a few articles related to Artificial Intelligence<\/a><\/em>, i.e.,
\"Category:Artificial intelligence\"<\/code>. You can replace everything afterCategory:<\/code> with your specific category.<\/p>\n\n![\"\"](\"https:\/\/blog.finxter.com\/wp-content\/uploads\/2023\/09\/image-128-1024x460.png\")
<\/figure>\n<\/div>\n Note<\/strong>: Category names in Wikipedia are case sensitive, so \"Category:Artificial intelligence\"<\/code> works while\"Category:Artificial I<\/strong>ntelligence\"<\/code> won’t work. <\/p>\nYou can use the
mwclient<\/code> library function to obtain the article titles.<\/p>\nimport mwclient\n\nCATEGORY_TITLE = \"Category:Artificial intelligence\"\nWIKI_SITE = \"en.wikipedia.org\"\n\ndef titles_from_category(category: mwclient.listing.Category, max_depth: int) -> set[str]:\n \"\"\"\n Return a set of page titles in a given Wiki category and its subcategories.\n\n :param category: The Wiki category to retrieve titles from.\n :param max_depth: The maximum depth to search for subcategories.\n :return: A set of page titles.\n \"\"\"\n titles = set()\n for member in category.members():\n if isinstance(member, mwclient.page.Page):\n titles.add(member.name)\n elif isinstance(member, mwclient.listing.Category) and max_depth > 0:\n titles.update(titles_from_category(member, max_depth=max_depth - 1))\n return titles\n\n\nsite = mwclient.Site(WIKI_SITE)\ncategory_page = site.pages[CATEGORY_TITLE]\ntitles = titles_from_category(category_page, max_depth=1)\nprint(titles)<\/pre>\n
In my case, the output is a lengthy:<\/p>\n
{'Embodied cognitive science', 'Category:Cloud robotics', 'Software agent', 'Reinforcement learning from human feedback', 'Synthetic media', 'Schema-agnostic databases', 'Blended artificial intelligence', 'Argumentation framework', 'Category:Artificial intelligence laboratories', 'Action selection', 'IJCAI Computers and Thought Award', 'Category:Chatbots', 'Weak artificial intelligence', 'AZFinText', 'Embodied agent', 'Hardware for artificial intelligence', 'SUPS', 'Machine perception', 'Hi! PARIS', 'Knowledge level', 'Joint Artificial Intelligence Center', 'Enterprise cognitive system', 'Human Problem Solving', 'Batch normalization', 'Summit on Responsible Artificial Intelligence in the Military Domain', 'Intelligent information society', 'Alignment Research Center', 'Percept (artificial intelligence)', 'Category:Fiction about artificial intelligence', 'Computer audition', 'Adobe Enhanced Speech', 'AI Superpowers', 'Attributional calculus', 'Automated negotiation', 'Intelligent agent', 'Mindpixel', 'Data-driven model', 'Incremental heuristic search', 'Category:Rule engines', 'Knowledge-based recommender system', 'Pedagogical agent', 'Personality computing', 'Category:Artificial intelligence art', 'Category:Automated reasoning', 'Category:Knowledge representation', 'Algorithmic probability', 'Edmond de Belamy', 'Psychology of reasoning', 'Artificial intelligence detection software', 'Category:Artificial intelligence publications', 'Computational heuristic intelligence', 'Neurorobotics', 'Nouvelle AI', 'Category:Machine learning', 'Situated approach (artificial intelligence)', 'AI alignment', 'Wetware (brain)', 'Hybrid intelligent system', 'Admissible heuristic', 'Category:Artificial intelligence conferences', 'Progress in artificial intelligence', 'INDIAai', 'Thompson sampling', 'Belief\u2013desire\u2013intention model', 'Syman', 'Category:Ambient intelligence', 'Artificial intelligence in healthcare', 'Computer Science Ontology', 'Synthography', 'Non-human', 'Artificial intelligence arms race', 'Empowerment (artificial intelligence)', 'Business process automation', 'Automated Mathematician', 'MindsDB', 'Intelligent decision support system', 'Diagnosis (artificial intelligence)', 'Virtual intelligence', 'Computational humor', 'Histogram of oriented displacements', 'Category:Virtual assistants', 'Interim Measures for the Management of Generative AI Services', 'Anytime algorithm', 'Category:Artificial intelligence templates', 'Principle of rationality', 'Remi El-Ouazzane', 'Symbol level', 'Artificial intelligence and copyright', 'Artificial reproduction', 'Brain technology', 'Category:Argument technology', 'Category:Philosophy of artificial intelligence', 'Category:Distributed artificial intelligence', 'Intelligent database', 'The Fable of Oscar', 'Category:Existential risk from artificial general intelligence', 'Category:Multi-agent systems', 'Artificial psychology', 'Intrinsic motivation (artificial intelligence)', 'Means\u2013ends analysis', 'Smart object', 'Category:Cognitive architecture', 'Frame problem', 'A.I. Insight forums', 'Dynamic epistemic logic', 'Stewart Nelson', 'AGI (computer science)', 'Belief\u2013desire\u2013intention software model', 'Case-based reasoning', 'Frame language', 'Google Clips', 'Description logic', 'Oriented energy filters', 'Elements of AI', 'Category:Artificial intelligence stubs', 'Intelligent word recognition', 'OpenIRIS', 'Algorithmic accountability', 'Workplace impact of artificial intelligence', 'International Conference on Autonomous Agents and Multiagent Systems', 'POP-11', 'Microsoft 365 Copilot', 'Perusall', 'Aurora (novel)', 'Graphics processing unit', 'Problem solving', 'Category:Applications of artificial intelligence', 'Category:AI accelerators', 'K-line (artificial intelligence)', 'Winner-take-all in action selection', 'Information space analysis', 'Pattern theory', 'Autonomic networking', 'Computational intelligence', 'Behavior informatics', 'Fine-tuning (deep learning)', 'Symbolic artificial intelligence', 'Vaumpus world', 'Mivar-based approach', 'Category:Artificial intelligence competitions', 'Autonomous agent', 'Artificial intelligence industry in China', 'Connectionist expert system', 'Omar Al Olama', 'Category:Works created using artificial intelligence', 'CarynAI', 'Knowledge compilation', 'Category:Fuzzy logic', 'Epistemic modal logic', 'Lifelong Planning A*', 'Operational artificial intelligence', 'Category:Open-source artificial intelligence', 'Category:Mind\u2013body problem', 'Structure mapping engine', 'Maia and Marco', 'Thomas Bolander', 'AI-complete', 'Data Science and Predictive Analytics', 'Prompt engineering', 'ASI (computer science)', 'Ensemble averaging (machine learning)', 'Artificial intelligence', 'Neural computation', 'Evolutionary developmental robotics', 'Category:Game artificial intelligence', 'Intelligent control', 'Category:Signal processing conferences', 'Concurrent MetateM', 'Artificial consciousness', 'Hallucination (artificial intelligence)', 'Generative artificial intelligence', 'Category:AI software', 'Category:Artificial intelligence entertainment', 'Michael Apter', 'Category:Affective computing', 'Neuro-symbolic AI', 'List of programming languages for artificial intelligence', 'Data pack', 'G\u00f6del machine', 'Category:AI companies', 'Character computing', 'Neural architecture search', 'AI boom', 'Artificial wisdom', 'AI safety', 'Parallel Intelligence', 'Quantum artificial life', 'Category:History of artificial intelligence', 'Computational human modeling', 'Anticipation (artificial intelligence)', 'Explainable artificial intelligence', 'Knowledge-based systems', 'Agent systems reference model', 'Probabilistic logic network', 'LangChain', 'Artificial general intelligence', 'Behavior selection algorithm', \"Roko's basilisk\", 'KL-ONE', 'Auto-GPT', 'Artificial intelligence in industry', 'Text-to-video model', 'Babelfy', 'Cognitive philology', 'Web intelligence', 'Blackboard system', 'Game theory', 'Category:Artificial immune systems', 'Category:Computer vision', 'Category:Artificial intelligence people', 'National Security Commission on Artificial Intelligence', 'Knowledge-based configuration', 'Universal psychometrics', 'AI-assisted virtualization software', 'Gender digital divide', 'Grammar systems theory', 'Operation Serenata de Amor', 'Category:Artificial intelligence associations', 'Artificial intelligence in fraud detection', 'Autonomic computing', 'Personoid', 'Situated', 'Autognostics', 'DABUS', 'Data processing unit', 'Hierarchical control system', 'GOLOG', 'KAoS', 'Bayesian programming', 'Karen Hao', 'And\u2013or tree', 'Commonsense knowledge (artificial intelligence)', 'Hindsight optimization', 'Wadhwani Institute for Artificial Intelligence', 'Wetware computer', 'Category:Neural networks', 'Zeuthen strategy', 'Category:Evolutionary computation', 'Extremal optimization', 'Category:Generative artificial intelligence', 'Category:Turing tests', \"Gabbay's separation theorem\", 'Spreading activation', 'ASR-complete', 'Self-management (computer science)', 'Emergent algorithm', 'Fuzzy agent', 'Language\/action perspective', 'Cognitive computing', 'Halite AI Programming Competition'}\n<\/pre>\nStep 2: Chunk Documents<\/h3>\n
\n![\"\"](\"https:\/\/blog.finxter.com\/wp-content\/uploads\/2023\/09\/pexels-photo-3226435.jpeg\")
<\/figure>\n<\/div>\nThe next step is to divide the Wikipedia articles into smaller sections conducive to embedding. It is important to disregard sections like
'External Links'<\/code> and'Footnotes'<\/code> as they lack context. Clean up the text by removing unnecessary tags, whitespace, and short sections.<\/p>\nYou can copy and paste the following code, however make sure you have initialized the
titles<\/code> variable as shown in the previous Step 1<\/strong>. I’ll also give the full code at the end of this article.<\/p>\nimport regex as re\nimport mwclient\nimport mwparserfromhell\n\nSECTIONS_TO_IGNORE = set([\n \"See also\", \"References\", \"External links\", \"Further reading\", \"Footnotes\",\n \"Bibliography\", \"Sources\", \"Citations\", \"Literature\", \"Notes and references\",\n \"Photo gallery\", \"Works cited\", \"Photos\", \"Gallery\", \"Notes\",\n \"References and sources\", \"References and notes\",\n])\n\nWIKI_SITE = 'en.wikipedia.org'\n\n\ndef all_subsections_from_section(section, parent_titles, sections_to_ignore):\n headings = [str(h) for h in section.filter_headings()]\n title = headings[0].strip(\"= \").strip()\n if title in sections_to_ignore:\n return []\n titles = parent_titles + [title]\n section_text = str(section).split(title, 1)[1]\n if len(headings) == 1:\n return [(titles, section_text)]\n first_subtitle = headings[1]\n section_text = section_text.split(first_subtitle, 1)[0]\n results = [(titles, section_text)]\n for subsection in section.get_sections(levels=[len(titles) + 1]):\n results.extend(all_subsections_from_section(subsection, titles, sections_to_ignore))\n return results\n\n\ndef all_subsections_from_title(title, sections_to_ignore=SECTIONS_TO_IGNORE, site_name=WIKI_SITE):\n site = mwclient.Site(site_name)\n page = site.pages[title]\n parsed_text = mwparserfromhell.parse(page.text())\n headings = [str(h) for h in parsed_text.filter_headings()]\n summary_text = str(parsed_text).split(headings[0], 1)[0] if headings else str(parsed_text)\n results = [([title], summary_text)]\n for subsection in parsed_text.get_sections(levels=[2]):\n results.extend(all_subsections_from_section(subsection, [title], sections_to_ignore))\n return results\n\n\ndef clean_section(section):\n titles, text = section\n text = re.sub(r\"<ref.*?<\/ref>\", \"\", text).strip()\n return titles, text\n\n\ndef keep_section(section):\n _, text = section\n return len(text) >= 16\n\n\ndef process_titles(titles):\n wikipedia_sections = []\n for title in titles:\n wikipedia_sections.extend(all_subsections_from_title(title))\n wikipedia_sections = [clean_section(ws) for ws in wikipedia_sections]\n original_num_sections = len(wikipedia_sections)\n wikipedia_sections = [ws for ws in wikipedia_sections if keep_section(ws)]\n print(f\"Filtered out {original_num_sections - len(wikipedia_sections)} sections, leaving {len(wikipedia_sections)} sections.\")\n return wikipedia_sections\n\n\n# Example usage\n# titles = see previous cell\nwikipedia_sections = process_titles(titles)\nfor ws in wikipedia_sections[:5]:\n print(ws[0])\n print(ws[1][:77] + \"...\")\n<\/pre>\n![\"\u2705\"](\"https:\/\/s.w.org\/images\/core\/emoji\/14.0.0\/72x72\/2705.png\")
Explanation<\/strong>: <\/p>\nThis code extracts and cleans text sections from a list of Wikipedia page titles and stores it in the variable
wikipedia_sections<\/code> for further processing.<\/p>\nIt navigates through each title, fetching the associated page content and breaking it down into sections and subsections. It ignores specified sections like “See also” and “References.” <\/p>\n
The text from each relevant section is extracted, and any citation references are removed. The code then filters out any sections with text shorter than 16 characters. <\/p>\n
The cleaned and filtered text sections are then outputted. <\/p>\n
In my case the output is:<\/p>\n
Filtered out 68 sections, leaving 1408 sections.\n['Embodied cognitive science']\n{{Short description|Interdisciplinary field of research}}\n{{for|approaches to...\n['Embodied cognitive science', 'Contributors']\n==\nEmbodied cognitive science borrows heavily from [[embodied philosophy]] an...\n['Embodied cognitive science', 'Traditional cognitive theory']\n==\n\nEmbodied cognitive science is an alternative theory to cognition in which...\n['Embodied cognitive science', 'The embodied cognitive approach']\n==\n\nEmbodied cognitive science differs from the traditionalist approach in th...\n['Embodied cognitive science', 'The embodied cognitive approach', 'Physical attributes of the body']\n===\nThe first aspect of embodied cognition examines the role of the physical ...<\/pre>\n Info<\/strong>: The important element obtained from the previous processing is a condensed information base stored in the variable wikipedia_sections<\/code>.<\/p>\nNext, you’ll employ a recursive approach<\/a> to divide lengthy sections into smaller ones<\/strong>.<\/p>\n
Splitting text into sections doesn\u2019t have a one-size-fits-all solution, and it involves various trade-offs:<\/p>\n
- \n
- While longer sections provide more context for questions, they may hinder retrieval by combining multiple topics, leading to confusion.<\/li>\n
- Shorter sections are cost-effective (as costs are tied to the number of tokens) and enhance retrieval by allowing more sections to be accessed, potentially improving recall.<\/li>\n
- Utilizing overlapping sections can avert answers from being truncated by section boundaries.<\/li>\n<\/ul>\n
In this scenario, we opt for a straightforward strategy, capping each section at 1,600 tokens<\/strong>. We recursively divide any exceeding sections into halves. To prevent the disruption of meaningful sentences, the division is preferably done along paragraph boundaries.<\/p>\n
Now copy and paste the following (again make sure the variable
wikipedia_section<\/code> is properly initialized as shown in the code snippet before, or simply copy and paste the full code at the end of this article):<\/p>\nimport tiktoken \n\n\nGPT_MODEL = \"gpt-3.5-turbo\" # which tokenizer to use?\n\n\ndef num_tokens(text: str, model: str = GPT_MODEL) -> int:\n \"\"\"Return the number of tokens in a string.\"\"\"\n encoding = tiktoken.encoding_for_model(model)\n return len(encoding.encode(text))\n\n\ndef halved_by_delimiter(string: str, delimiter: str = \"\\n\") -> list[str, str]:\n \"\"\"Split a string in two, on a delimiter, trying to balance tokens on each side.\"\"\"\n chunks = string.split(delimiter)\n if len(chunks) == 1:\n return [string, \"\"] # no delimiter found\n elif len(chunks) == 2:\n return chunks # no need to search for halfway point\n else:\n total_tokens = num_tokens(string)\n halfway = total_tokens \/\/ 2\n best_diff = halfway\n for i, chunk in enumerate(chunks):\n left = delimiter.join(chunks[: i + 1])\n left_tokens = num_tokens(left)\n diff = abs(halfway - left_tokens)\n if diff >= best_diff:\n break\n else:\n best_diff = diff\n left = delimiter.join(chunks[:i])\n right = delimiter.join(chunks[i:])\n return [left, right]\n\n\ndef truncated_string(\n string: str,\n model: str,\n max_tokens: int,\n print_warning: bool = True,\n) -> str:\n \"\"\"Truncate a string to a maximum number of tokens.\"\"\"\n encoding = tiktoken.encoding_for_model(model)\n encoded_string = encoding.encode(string)\n truncated_string = encoding.decode(encoded_string[:max_tokens])\n if print_warning and len(encoded_string) > max_tokens:\n print(f\"Warning: Truncated string from {len(encoded_string)} tokens to {max_tokens} tokens.\")\n return truncated_string\n\n\ndef split_strings_from_subsection(\n subsection: tuple[list[str], str],\n max_tokens: int = 1000,\n model: str = GPT_MODEL,\n max_recursion: int = 5,\n) -> list[str]:\n \"\"\"\n Split a subsection into a list of subsections, each with no more than max_tokens.\n Each subsection is a tuple of parent titles [H1, H2, ...] and text (str).\n \"\"\"\n titles, text = subsection\n string = \"\\n\\n\".join(titles + [text])\n num_tokens_in_string = num_tokens(string)\n # if length is fine, return string\n if num_tokens_in_string <= max_tokens:\n return [string]\n # if recursion hasn't found a split after X iterations, just truncate\n elif max_recursion == 0:\n return [truncated_string(string, model=model, max_tokens=max_tokens)]\n # otherwise, split in half and recurse\n else:\n titles, text = subsection\n for delimiter in [\"\\n\\n\", \"\\n\", \". \"]:\n left, right = halved_by_delimiter(text, delimiter=delimiter)\n if left == \"\" or right == \"\":\n # if either half is empty, retry with a more fine-grained delimiter\n continue\n else:\n # recurse on each half\n results = []\n for half in [left, right]:\n half_subsection = (titles, half)\n half_strings = split_strings_from_subsection(\n half_subsection,\n max_tokens=max_tokens,\n model=model,\n max_recursion=max_recursion - 1,\n )\n results.extend(half_strings)\n return results\n # otherwise no split was found, so just truncate (should be very rare)\n return [truncated_string(string, model=model, max_tokens=max_tokens)]\n\n\n# split sections into chunks\nMAX_TOKENS = 1600\nwikipedia_strings = []\nfor section in wikipedia_sections:\n wikipedia_strings.extend(split_strings_from_subsection(section, max_tokens=MAX_TOKENS))\n\nprint(f\"{len(wikipedia_sections)} Wikipedia sections split into {len(wikipedia_strings)} strings.\")\n\n# print example data\nprint(wikipedia_strings[1])<\/pre>\n Explanation<\/strong>: <\/p>\nThis Python code is used for processing and truncating text data, specifically focusing on splitting and tokenizing strings. <\/p>\n
It uses the
tiktoken<\/code> library to count the number of tokens in a text string for a given model (default is\"gpt-3.5-turbo\"<\/code>). <\/p>\nThe
halved_by_delimiter<\/code> function splits a string into two parts based on a delimiter, aiming to balance the number of tokens in each part. <\/p>\nThe
truncated_string<\/code> function shortens a string to a specified maximum number of tokens, issuing a warning if truncation occurs. <\/p>\nThe
split_strings_from_subsection<\/code> function recursively splits a subsection of text to ensure that each resulting piece has no more than a specified number of tokens. It tries different delimiters and falls back to truncation if no suitable split is found. <\/p>\nThe code concludes by applying these functions to a list of Wikipedia sections, splitting them into chunks with a maximum token limit, and printing out the results.<\/p>\n
In my case the output is AI related due to the artificial intelligence articles specified above:<\/p>\n
\/usr\/local\/lib\/python3.10\/dist-packages\/ipykernel\/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n and should_run_async(code)\n\n1408 Wikipedia sections split into 1420 strings.\nEmbodied cognitive science\n\nContributors\n\n==\nEmbodied cognitive science borrows heavily from [[embodied philosophy]] and the related research fields of [[cognitive science]], [[psychology]], [[neuroscience]] and [[artificial intelligence]]. Contributors to the field include:\n* From the perspective of neuroscience, [[Gerald Edelman]] of the [[Neurosciences Institute]] at La Jolla, [[Francisco Varela]] of [[CNRS]] in France, and [[J. A. Scott Kelso]] of [[Florida Atlantic University]]\n* From the perspective of psychology, [[Lawrence Barsalou]], [[Michael Turvey]], [[Vittorio Guidano]] and [[Eleanor Rosch]]\n* From the perspective of linguistics, [[Gilles Fauconnier]], [[George Lakoff]], [[Mark Johnson (philosopher)|Mark Johnson]], [[Leonard Talmy]] and [[Mark Turner (cognitive scientist)|Mark Turner]]\n* From the perspective of language acquisition, [[Eric Lenneberg]] and [[Philip Rubin]] at [[Haskins Laboratories]]\n* From the perspective of anthropology, [[Edwin Hutchins]], [[Bradd Shore]], [[James Wertsch]] and [[Merlin Donald]]. \n* From the perspective of autonomous agent design, early work is sometimes attributed to [[Rodney Brooks]] or [[Valentino Braitenberg]]\n* From the perspective of artificial intelligence, ''Understanding Intelligence'' by [[Rolf Pfeifer]] and Christian Scheier or ''How the Body Shapes the Way We Think'', by Rolf Pfeifer and Josh C. Bongard\n* From the perspective of philosophy, [[Andy Clark]], [[Dan Zahavi]], [[Shaun Gallagher]], and [[Evan Thompson]]\n\nIn 1950, [[Alan Turing]] proposed that a machine may need a human-like body to think and speak:\n\n{{Quote|It can also be maintained that it is best to provide the machine with the best sense organs that money can buy, and then teach it to understand and speak English. That process could follow the normal teaching of a child. Things would be pointed out and named, etc. Again, I do not know what the right answer is, but I think both approaches should be tried.}}\n\n<\/pre>\nStep 3: Embed Document Chunks<\/h3>\n
\n![\"\"](\"https:\/\/blog.finxter.com\/wp-content\/uploads\/2023\/09\/pexels-photo-4916513.webp\")
<\/figure>\n<\/div>\nOnce the documents are appropriately chunked, you can use OpenAI to compute the embeddings for each string. In this section, you’ll learn a basic example of generating embeddings — make sure you replace the highlighted line with your own API key. <\/p>\n
If you haven’t set up your OpenAI API key, make sure to follow this tutorial:
<\/p>\n\n![\"\"](\"https:\/\/blog.finxter.com\/wp-content\/uploads\/2023\/09\/image-113-2-1024x573.png\")
<\/a><\/figure>\n<\/div>\n![\"?\"](\"https:\/\/s.w.org\/images\/core\/emoji\/14.0.0\/72x72\/1f517.png\")
Recommended<\/strong>: OpenAI Python API \u2013 A Helpful Illustrated Guide in 5 Steps<\/a><\/p>\nAgain you can also copy and paste the full code at the end of this article for ease of use:<\/p>\n
import openai\nimport pandas as pd\n\n# Constants for calculating embeddings\nEMBEDDING_MODEL = \"text-embedding-ada-002\" # this may change over time\nBATCH_SIZE = 1000 # Maximum of 2048 embedding inputs per request allowed\n\n# your openai key\nopenai.api_key = 'sk-...'\n\ndef calculate_embeddings(wikipedia_strings):\n \"\"\"Calculate embeddings for a list of strings.\"\"\"\n embeddings = []\n total_strings = len(wikipedia_strings)\n\n for batch_start in range(0, total_strings, BATCH_SIZE):\n batch_end = min(batch_start + BATCH_SIZE, total_strings)\n batch = wikipedia_strings[batch_start:batch_end]\n print(f\"Processing Batch {batch_start} to {batch_end-1}\")\n\n # Requesting embeddings from OpenAI API\n response = openai.Embedding.create(model=EMBEDDING_MODEL, input=batch)\n\n # Verifying the order of embeddings\n for i, be in enumerate(response[\"data\"]):\n assert i == be[\"index\"], \"Mismatch in embedding order\"\n\n # Extracting and storing embeddings\n batch_embeddings = [e[\"embedding\"] for e in response[\"data\"]]\n embeddings.extend(batch_embeddings)\n\n return embeddings\n\n# Calculate embeddings for Wikipedia strings\nembeddings = calculate_embeddings(wikipedia_strings)\n\n# Creating a DataFrame to store text and corresponding embeddings\ndf = pd.DataFrame({\"text\": wikipedia_strings, \"embedding\": embeddings})\n<\/pre>\n Explanation<\/strong>: <\/p>\nThis Python script is used to compute embeddings for a collection of text from Wikipedia. It uses the OpenAI API<\/a>, specifically the
\"text-embedding-ada-002\"<\/code> model, to generate embeddings for each text in batches of 1000. <\/p>\nThe
calculate_embeddings<\/code> function processes each batch of text, sends it to the OpenAI API, and collects the embeddings returned. It ensures the order of embeddings matches the input text. <\/p>\nAfter processing all batches, the text and their corresponding embeddings are stored together in a pandas DataFrame. The OpenAI API key is required for accessing the API, and it’s set at the beginning of the script.<\/p>\n
In my case, the output is simple: <\/p>\n
Processing Batch 0 to 999\nProcessing Batch 1000 to 1419<\/pre>\n
Step 4: Store Document Chunks and Embeddings<\/h3>\n
Finally, you want to store the generated embeddings for future use. Depending on the scale of your data, you can opt for different storage solutions. In the case of smaller datasets, you can store the sections and their respective embeddings in a CSV file. Pandas provides an easy-to-use method for achieving this:<\/p>\n
df = pd.DataFrame({\"text\": wikipedia_strings, \"embedding\": embeddings})\ndf.to_csv(\"artificial_intelligence_wiki.csv\", index=False)<\/pre>\nHere’s how the CSV file looks like in my AI example — it has 49MB of data, i.e., almost all Wikipedia pages in the category AI!<\/p>\n
\n![\"\"](\"https:\/\/blog.finxter.com\/wp-content\/uploads\/2023\/09\/image-126-1024x733.png\")
<\/figure>\n<\/div>\nThe path to the CSV file will depend on your directory structure.<\/p>\n
By following this step-by-step guide, you can proficiently prepare a dataset of Wikipedia articles embedded for search, utilizing the OpenAI library.<\/strong>
![\"?\"](\"https:\/\/s.w.org\/images\/core\/emoji\/14.0.0\/72x72\/1f92f.png\")
This will help in facilitating extensive research or information extraction in various projects.<\/p>\nIn particular, you can use the embeddings for search or Q&A on a vast text corpus as shown in this GitHub:<\/p>\n
Recommended<\/strong>: Question answering using embeddings-based search<\/a><\/p>\nIf you want to learn the ins and outs of OpenAI Function Calls and Embeddings<\/a>, check out our academy courses<\/a> (all-you-can-learn):<\/p>\n
Prompt Engineering with Python and OpenAI<\/h2>\n
![\"\"](\"https:\/\/blog.finxter.com\/wp-content\/uploads\/2023\/06\/image-288.png\")
<\/a><\/figure>\nYou can check out the whole course on OpenAI Prompt Engineering using Python on the Finxter academy<\/a>. We cover topics such as:<\/p>\n
- \n
- Embeddings<\/li>\n
- Semantic search<\/li>\n
- Web scraping<\/li>\n
- Query embeddings<\/li>\n
- Movie recommendation<\/li>\n
- Sentiment analysis<\/li>\n<\/ul>\n
![\"?\u200d?\"](\"https:\/\/s.w.org\/images\/core\/emoji\/14.0.0\/72x72\/1f468-200d-1f4bb.png\")
Academy<\/strong>: Prompt Engineering with Python and OpenAI<\/a><\/p>\nAs promised multiple times throughout the article, here’s the whole code for copy and paste:<\/p>\n
import mwclient\nimport regex as re\nimport mwparserfromhell\nimport tiktoken\nimport openai\nimport pandas as pd\n\n\nWIKI_SITE = 'en.wikipedia.org'\nGPT_MODEL = \"gpt-3.5-turbo\" # which tokenizer to use?\n\n# Constants for calculating embeddings\nEMBEDDING_MODEL = \"text-embedding-ada-002\" # this may change over time\nBATCH_SIZE = 1000 # Maximum of 2048 embedding inputs per request allowed\nCATEGORY_TITLE = \"Category:Artificial intelligence\"\nWIKI_SITE = \"en.wikipedia.org\"\n\n# your openai key\nopenai.api_key = 'sk-...'\n\nSECTIONS_TO_IGNORE = set([\n \"See also\", \"References\", \"External links\", \"Further reading\", \"Footnotes\",\n \"Bibliography\", \"Sources\", \"Citations\", \"Literature\", \"Notes and references\",\n \"Photo gallery\", \"Works cited\", \"Photos\", \"Gallery\", \"Notes\",\n \"References and sources\", \"References and notes\",\n])\n\n\ndef titles_from_category(category: mwclient.listing.Category, max_depth: int) -> set[str]:\n \"\"\"\n Return a set of page titles in a given Wiki category and its subcategories.\n\n :param category: The Wiki category to retrieve titles from.\n :param max_depth: The maximum depth to search for subcategories.\n :return: A set of page titles.\n \"\"\"\n titles = set()\n for member in category.members():\n if isinstance(member, mwclient.page.Page):\n titles.add(member.name)\n elif isinstance(member, mwclient.listing.Category) and max_depth > 0:\n titles.update(titles_from_category(member, max_depth=max_depth - 1))\n return titles\n\n\ndef all_subsections_from_section(section, parent_titles, sections_to_ignore):\n headings = [str(h) for h in section.filter_headings()]\n title = headings[0].strip(\"= \").strip()\n if title in sections_to_ignore:\n return []\n titles = parent_titles + [title]\n section_text = str(section).split(title, 1)[1]\n if len(headings) == 1:\n return [(titles, section_text)]\n first_subtitle = headings[1]\n section_text = section_text.split(first_subtitle, 1)[0]\n results = [(titles, section_text)]\n for subsection in section.get_sections(levels=[len(titles) + 1]):\n results.extend(all_subsections_from_section(subsection, titles, sections_to_ignore))\n return results\n\n\ndef all_subsections_from_title(title, sections_to_ignore=SECTIONS_TO_IGNORE, site_name=WIKI_SITE):\n site = mwclient.Site(site_name)\n page = site.pages[title]\n parsed_text = mwparserfromhell.parse(page.text())\n headings = [str(h) for h in parsed_text.filter_headings()]\n summary_text = str(parsed_text).split(headings[0], 1)[0] if headings else str(parsed_text)\n results = [([title], summary_text)]\n for subsection in parsed_text.get_sections(levels=[2]):\n results.extend(all_subsections_from_section(subsection, [title], sections_to_ignore))\n return results\n\n\ndef clean_section(section):\n titles, text = section\n text = re.sub(r\"<ref.*?<\/ref>\", \"\", text).strip()\n return titles, text\n\n\ndef keep_section(section):\n _, text = section\n return len(text) >= 16\n\n\ndef process_titles(titles):\n wikipedia_sections = []\n for title in titles:\n wikipedia_sections.extend(all_subsections_from_title(title))\n wikipedia_sections = [clean_section(ws) for ws in wikipedia_sections]\n original_num_sections = len(wikipedia_sections)\n wikipedia_sections = [ws for ws in wikipedia_sections if keep_section(ws)]\n print(f\"Filtered out {original_num_sections - len(wikipedia_sections)} sections, leaving {len(wikipedia_sections)} sections.\")\n return wikipedia_sections\n\n\ndef num_tokens(text: str, model: str = GPT_MODEL) -> int:\n \"\"\"Return the number of tokens in a string.\"\"\"\n encoding = tiktoken.encoding_for_model(model)\n return len(encoding.encode(text))\n\n\ndef halved_by_delimiter(string: str, delimiter: str = \"\\n\") -> list[str, str]:\n \"\"\"Split a string in two, on a delimiter, trying to balance tokens on each side.\"\"\"\n chunks = string.split(delimiter)\n if len(chunks) == 1:\n return [string, \"\"] # no delimiter found\n elif len(chunks) == 2:\n return chunks # no need to search for halfway point\n else:\n total_tokens = num_tokens(string)\n halfway = total_tokens \/\/ 2\n best_diff = halfway\n for i, chunk in enumerate(chunks):\n left = delimiter.join(chunks[: i + 1])\n left_tokens = num_tokens(left)\n diff = abs(halfway - left_tokens)\n if diff >= best_diff:\n break\n else:\n best_diff = diff\n left = delimiter.join(chunks[:i])\n right = delimiter.join(chunks[i:])\n return [left, right]\n\n\ndef truncated_string(\n string: str,\n model: str,\n max_tokens: int,\n print_warning: bool = True,\n) -> str:\n \"\"\"Truncate a string to a maximum number of tokens.\"\"\"\n encoding = tiktoken.encoding_for_model(model)\n encoded_string = encoding.encode(string)\n truncated_string = encoding.decode(encoded_string[:max_tokens])\n if print_warning and len(encoded_string) > max_tokens:\n print(f\"Warning: Truncated string from {len(encoded_string)} tokens to {max_tokens} tokens.\")\n return truncated_string\n\n\ndef split_strings_from_subsection(\n subsection: tuple[list[str], str],\n max_tokens: int = 1000,\n model: str = GPT_MODEL,\n max_recursion: int = 5,\n) -> list[str]:\n \"\"\"\n Split a subsection into a list of subsections, each with no more than max_tokens.\n Each subsection is a tuple of parent titles [H1, H2, ...] and text (str).\n \"\"\"\n titles, text = subsection\n string = \"\\n\\n\".join(titles + [text])\n num_tokens_in_string = num_tokens(string)\n # if length is fine, return string\n if num_tokens_in_string <= max_tokens:\n return [string]\n # if recursion hasn't found a split after X iterations, just truncate\n elif max_recursion == 0:\n return [truncated_string(string, model=model, max_tokens=max_tokens)]\n # otherwise, split in half and recurse\n else:\n titles, text = subsection\n for delimiter in [\"\\n\\n\", \"\\n\", \". \"]:\n left, right = halved_by_delimiter(text, delimiter=delimiter)\n if left == \"\" or right == \"\":\n # if either half is empty, retry with a more fine-grained delimiter\n continue\n else:\n # recurse on each half\n results = []\n for half in [left, right]:\n half_subsection = (titles, half)\n half_strings = split_strings_from_subsection(\n half_subsection,\n max_tokens=max_tokens,\n model=model,\n max_recursion=max_recursion - 1,\n )\n results.extend(half_strings)\n return results\n # otherwise no split was found, so just truncate (should be very rare)\n return [truncated_string(string, model=model, max_tokens=max_tokens)]\n\n\ndef calculate_embeddings(wikipedia_strings):\n \"\"\"Calculate embeddings for a list of strings.\"\"\"\n embeddings = []\n total_strings = len(wikipedia_strings)\n\n for batch_start in range(0, total_strings, BATCH_SIZE):\n batch_end = min(batch_start + BATCH_SIZE, total_strings)\n batch = wikipedia_strings[batch_start:batch_end]\n print(f\"Processing Batch {batch_start} to {batch_end-1}\")\n\n # Requesting embeddings from OpenAI API\n response = openai.Embedding.create(model=EMBEDDING_MODEL, input=batch)\n\n # Verifying the order of embeddings\n for i, be in enumerate(response[\"data\"]):\n assert i == be[\"index\"], \"Mismatch in embedding order\"\n\n # Extracting and storing embeddings\n batch_embeddings = [e[\"embedding\"] for e in response[\"data\"]]\n embeddings.extend(batch_embeddings)\n\n return embeddings\n\n\nsite = mwclient.Site(WIKI_SITE)\ncategory_page = site.pages[CATEGORY_TITLE]\ntitles = titles_from_category(category_page, max_depth=1)\n\nwikipedia_sections = process_titles(titles)\n\n# split sections into chunks\nMAX_TOKENS = 1600\nwikipedia_strings = []\nfor section in wikipedia_sections:\n wikipedia_strings.extend(split_strings_from_subsection(section, max_tokens=MAX_TOKENS))\n\nprint(f\"{len(wikipedia_sections)} Wikipedia sections split into {len(wikipedia_strings)} strings.\")\n\n# print example data\nprint(wikipedia_strings[1])\n\n\n\n# Calculate embeddings for Wikipedia strings\nembeddings = calculate_embeddings(wikipedia_strings)\n\n# Creating a DataFrame to store text and corresponding embeddings\ndf = pd.DataFrame({\"text\": wikipedia_strings, \"embedding\": embeddings})\ndf.to_csv(\"data\/artificial_intelligence_wiki.csv\", index=False)<\/pre>\nThe post How to Create a Massive OpenAI Embeddings CSV File for Search – From 100s of Wikipedia Articles<\/a> appeared first on Be on the Right Side of Change<\/a>.<\/p>\n\n
![\\\"IT\u96fb\u8166\u88dc\u7fd2](\\\"http:\/\/itteacheritfreelance.hk\/test\/wordpress\/wp-content\/uploads\/2016\/05\/logo2-2.png\\\")
\u7acb\u523b\u8a3b\u518a\u53ca\u5831\u540d\u96fb\u8166\u88dc\u7fd2\u8ab2\u7a0b\u5427!<\/figcaption><\/figure>\r\n<\/br>Find A Teacher Form:\r\n<\/br>https:\/\/docs.google.com\/forms\/d\/1vREBnX5n262umf4wU5U2pyTwvk9O-JrAgblA-wH9GFQ\/viewform?edit_requested=true#responses\r\n<\/br><\/br>Email:\r\n<\/br>public1989two@gmail.com
\r\nwww.itsec.hk<\/a>
\r\nwww.itsec.vip<\/a>
\r\nwww.itseceu.uk<\/a>
<\/p>","protected":false},"excerpt":{"rendered":"In this guide, let\u2019s delve into preparing a dataset of Wikipedia articles for search, utilizing the OpenAI API. The process involves In case you need a quick refresher on OpenAI embeddings, feel free to read through our Finxter article here: ? ? Recommended: What Are Embeddings in OpenAI? Prerequisites Before starting, ensure that you have … Read more<\/a><\/p>\n
The post How to Create a Massive OpenAI Embeddings CSV File for Search \u2013 From 100s of Wikipedia Articles<\/a> appeared first on Be on the Right Side of Change<\/a>.<\/p>\n<\/div>","protected":false},"author":2015,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"slim_seo":{"title":"How to Create a Massive OpenAI Embeddings CSV File for Search \u2013 From 100s of Wikipedia Articles - ITTeacherITFreelance.hk","description":"In this guide, let\u2019s delve into preparing a dataset of Wikipedia articles for search, utilizing the OpenAI API. The process involves In case you need a quick re"},"footnotes":""},"categories":[10700],"tags":[10711],"_links":{"self":[{"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/posts\/329249"}],"collection":[{"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/users\/2015"}],"replies":[{"embeddable":true,"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/comments?post=329249"}],"version-history":[{"count":1,"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/posts\/329249\/revisions"}],"predecessor-version":[{"id":329250,"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/posts\/329249\/revisions\/329250"}],"wp:attachment":[{"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/media?parent=329249"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/categories?post=329249"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/tags?post=329249"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
- chunking the downloaded documents<\/strong> into smaller sections suitable for embedding, <\/li>\n