OpenAI is launching Data Partnerships, an initiative to collaborate with organizations in creating both public and private datasets for training AI models.
Contemporary AI technology acquires skills and understanding of the world—including human motivations, interactions, and communication—by processing its training data. To achieve Artificial General Intelligence (AGI) that is safe and universally beneficial, AI models need a profound grasp of diverse subjects, industries, cultures, and languages, necessitating the broadest possible training datasets.
Incorporating specific content can enhance AI models’ utility by deepening their understanding of particular domains. Many partners are already collaborating to provide data from their respective countries or industries. For instance, a recent collaboration with the Icelandic Government and Miðeind ehf aimed to improve GPT‑4’s Icelandic language capabilities through curated datasets. Another partnership with the non-profit Free Law Project involved integrating their extensive collection of legal documents into AI training to broaden legal understanding. Numerous other entities may also wish to contribute to AI research and explore the value of their unique data.
These Data Partnerships aim to empower more organizations to influence the direction of AI development and benefit from models that are more relevant to their specific needs, by incorporating content important to them.
Types of Data Sought
The initiative seeks large-scale datasets that represent human society and are not readily available online. Data can be in any modality, such as text, images, audio, or video. A particular focus is on data that conveys human intention, like long-form writing or conversations, rather than isolated snippets, across various languages, topics, and formats.
Data can be handled in nearly any format, utilizing advanced in-house AI technology for digitization and structuring. This includes optical character recognition (OCR) for digitizing PDFs and automatic speech recognition (ASR) for transcribing audio. If data requires cleaning, such as removing auto-generated artifacts or transcription errors, assistance can be provided to process it into an optimal form. The initiative does not seek datasets containing sensitive or personal information, or data belonging to third parties; support is available for removing such information if necessary.
Partnership Opportunities
Currently, two primary partnership avenues are available, with potential for future expansion:
- Open-Source Archive: This option involves collaborating to build an open-source dataset for language model training. This dataset would be publicly accessible for anyone to use in AI model development. There is also an interest in using it to safely train additional open-source models. The open-source approach is considered vital to the ecosystem.
- Private Datasets: This involves preparing private datasets for training proprietary AI models, including foundational, fine-tuned, and custom models. For organizations with private data who want AI models to better understand their domain, or to assess their data’s potential, this is the ideal partnership method. Data will be handled with preferred levels of sensitivity and access controls.
Ultimately, the goal is to find partners dedicated to enhancing AI’s understanding of the world, making it as beneficial as possible for everyone. This collaborative effort aims to advance towards AGI that serves all of humanity.
