Open-source Data & databases projects

175 MIT-licensed data & databases projects, most-starred first — free to use, modify and ship commercially. Each page covers what it is, who it suits, how to install and run it, and how it compares.

1. 1MarkItDownPython tool that converts files like PDF, Word, Excel and PowerPoint into Markdown.★ 193k
2. 2awesome-public-datasetsA curated list of links to high-quality open datasets, organized by topic.★ 79.9k
3. 3DoclingPython library that converts PDFs and office files into structured text for AI use.★ 70k
4. 4Base NodeDocker setup for running your own Base blockchain node; this repo is archived, so use base/base instead.★ 68.3k
5. 5daily_stock_analysisLLM-driven stock analysis tool that gathers market data and news, then pushes daily reports to chat apps.★ 67.4k
6. 6rcloneCommand-line tool that copies, syncs and manages files across many cloud storage providers.★ 60.7k
7. 7chinese-poetryA free, open JSON database of classical Chinese poetry, prose and song lyrics.★ 54k
8. 8LlamaIndexPython framework for building LLM apps that query your own documents and data.★ 52.8k
9. 9Python Data Science HandbookThe full text of a Python data science book, delivered as runnable Jupyter notebooks.★ 50.3k
10. 10QlibOpen-source Python platform for quant investment research using machine learning.★ 49.9k
11. 11JuliaA programming language for technical and scientific computing, with source code in this repo.★ 49.2k
12. 12Day.jsA small JavaScript library for working with dates and times, with an API similar to Moment.js.★ 48.7k
13. 13Umi-OCRFree, open-source OCR app that reads text from screenshots, images, and PDFs, offline.★ 48.4k
14. 14Moment.jsA JavaScript library for parsing, validating, manipulating and formatting dates; now in maintenance mode.★ 47.9k
15. 15VeighNaPython framework for building quant trading systems, with market gateways, backtesting and ML research tools.★ 46.2k
16. 16KronosOpen-source pre-trained models that forecast financial candlestick (K-line) price data.★ 45.7k
17. 17CCXTA library for connecting code to 100+ crypto exchanges through one common interface.★ 44.7k
18. 18DuckDBAn in-process analytical SQL database that runs inside your app with no server to manage.★ 42.7k
19. 19FaissA C++ library with Python bindings for fast similarity search over large sets of vectors.★ 41.2k
20. 20LightRAGA Python library that builds a knowledge graph from your documents to answer questions with an LLM.★ 40.5k
21. 21PolarsA fast library for working with tables of data (DataFrames), written in Rust with Python and other bindings.★ 40.2k
22. 22GORMA Go library that maps database tables to Go structs so you can query data without writing raw SQL.★ 40k
23. 23PageIndexLets an AI find answers in long documents by reading a table-of-contents-style tree, not vector search.★ 40k
24. 24UmamiPrivacy-first web analytics you can self-host or use in the cloud, without cookies.★ 39.9k
25. 25QuivrOpen-source engine that ingests content streams and makes them searchable and monitorable, in evaluation stage★ 39.6k
26. 26Data Science for BeginnersA free 10-week, 20-lesson beginner data science curriculum from Microsoft, with quizzes and assignments.★ 37.9k
27. 27TypeORMA TypeScript/JavaScript library that maps database tables to code objects for Node.js apps.★ 36.7k
28. 28GraphRAGPython pipeline that uses an LLM to turn raw text into a graph for answering questions over private documents.★ 36.5k
29. 29jiebaA Python library that splits Chinese text into words.★ 35.2k
30. 30Another Redis Desktop ManagerA desktop GUI for browsing and managing Redis databases on Windows, Mac, and Linux.★ 34.8k
31. 31CLIPA neural network that matches images with text descriptions, usable without task-specific training.★ 34.6k
32. 32CheerioA fast library for parsing and changing HTML and XML using a jQuery-like syntax.★ 30.5k
33. 33SequelizeA Node.js and TypeScript ORM for working with SQL databases using JavaScript objects.★ 30.3k
34. 34Unlimited OCROpen-source OCR model from Baidu that parses long, multi-page documents into text.★ 30.3k
35. 35awesome-datascienceA curated list of links for learning data science, from courses and books to tools and datasets.★ 30.3k
36. 36SRSA real-time media server that receives and delivers live video over many streaming protocols.★ 29.4k
37. 37ArchiveBoxSelf-hosted tool that saves copies of web pages and media in standard, readable formats.★ 28.9k
38. 38LocustOpen-source load testing tool where you write test scenarios as plain Python code.★ 28.3k
39. 39Bayesian Methods for HackersA free, notebook-based introduction to Bayesian statistics taught through code, using PyMC.★ 28.2k
40. 40PostgRESTTurns an existing PostgreSQL database into a REST API with no hand-written server code.★ 27.7k
41. 41RechartsA React charting library built on D3 for drawing charts from declarative components.★ 27.7k
42. 42MongooseA Node.js library that gives your app structured models and queries on top of MongoDB.★ 27.5k
43. 43FakerA PHP library that generates realistic fake data like names, addresses, and text for testing.★ 26.6k
44. 44fastTextA library for learning word vectors and classifying text, built for speed.★ 26.5k
45. 45faster-whisperFaster Whisper speech-to-text transcription using CTranslate2, a Python reimplementation of OpenAI's Whisper.★ 26.4k
46. 46SHAPA Python library that shows how much each input feature pushed a machine learning model's prediction up or dow★ 25.8k
47. 47TursoAn in-process SQL database in Rust that aims for SQLite compatibility, with Postgres support in progress.★ 25.1k
48. 48LabelImgA desktop tool for drawing labels on images to build training datasets. Archived; no longer maintained.★ 25k
49. 49Homemade Machine LearningPython implementations of classic machine learning algorithms, written from scratch, with Jupyter demos.★ 24.8k
50. 50DashA Python framework for building interactive data dashboards and web apps without writing JavaScript.★ 24.5k
51. 51RAG-AnythingA Python framework for question answering over documents that mix text, images, tables and equations.★ 24.4k
52. 52PyGA PyTorch library for building and training graph neural networks on structured data.★ 24.1k
53. 53DeepSeek-OCRAn OCR model that turns document images into text and markdown, from DeepSeek AI.★ 24.1k
54. 54VannaAsk questions of your SQL database in plain English and get SQL, tables, and charts back.★ 23.8k
55. 55ProxyPoolPython tool that collects free proxy IPs from public sites, checks them, and serves working ones via an API.★ 23.8k
56. 56anydocConverts Word, PowerPoint, Excel, PDF and other documents into clean Markdown, built in Rust.★ 23.3k
57. 57AKSharePython library for fetching Chinese and global financial data with simple function calls.★ 23.2k
58. 58lowdbA small JSON-file database you use with plain JavaScript arrays and objects.★ 22.6k
59. 59Machine Learning for TradingCode companion to a book on building, testing, and running ML-driven trading strategies.★ 21.7k
60. 60GoAccessOpen-source log analyzer that shows live web traffic stats in your terminal or a browser dashboard.★ 21.1k
61. 61visxA set of reusable React building blocks for drawing custom charts, built on d3 math.★ 21.1k
62. 62normalizrA small JavaScript library that flattens nested JSON data into entities and IDs using a schema.★ 20.8k
63. 63KaTeXA fast JavaScript library that renders LaTeX-style math in web pages.★ 20.5k
64. 64ProphetOpen-source forecasting tool for time series with multiple seasonal patterns and trend changes.★ 20.5k
65. 65knexA JavaScript library for building SQL queries across several databases with one API.★ 20.3k
66. 66DB-GPTOpen-source AI assistant that writes SQL and code to analyze your databases and files.★ 20.2k
67. 67pdf-inspectorRust library and CLI that checks whether a PDF has real text and converts it to Markdown locally.★ 19.8k
68. 68FakerA Python package that generates fake data for testing and seeding databases.★ 19.5k
69. 69LightGBMA fast gradient boosting library for tree-based machine learning on tabular data.★ 18.9k
70. 70plotly.pyA Python library for making interactive, browser-based charts and graphs.★ 18.9k
71. 71mysqljsA pure JavaScript MySQL client for Node.js that talks directly to MySQL servers.★ 18.6k
72. 72SQLModelA Python library for working with SQL databases using Python classes, built on Pydantic and SQLAlchemy.★ 18.5k
73. 73sqlcTurns your SQL queries into type-safe code for Go, Python, Kotlin and more.★ 18.4k
74. 74ELKRun the Elastic stack (Elasticsearch, Logstash, Kibana) locally with Docker Compose.★ 18.4k
75. 75Plotly.jsA JavaScript library for drawing interactive charts in web pages, also the engine behind Plotly in Python and ★ 18.4k
76. 76StatsDA Node.js daemon that collects counters and timers over UDP/TCP and forwards summaries to backends like Graphi★ 18.1k
77. 77rqliteA small SQLite-based SQL database that runs on several machines and keeps working if one fails.★ 17.9k
78. 78TelegrafAn agent that collects, processes, and sends metrics, logs, and other data to storage or monitoring tools.★ 17.9k
79. 79sqlxGo helper library that adds convenience features on top of the standard database/sql package.★ 17.7k
80. 80Data FormulatorOpen-source tool for exploring data through AI-assisted charts and natural-language questions.★ 17.6k
81. 81node-redisA Redis client library for Node.js, written in TypeScript, for talking to Redis from JavaScript code.★ 17.6k
82. 82turbovecA compressed vector search index for Python and Rust that runs inside your own process.★ 17.5k
83. 83libSQLAn open-source fork of SQLite that accepts outside contributions.★ 17.3k
84. 84FinRLOpen-source Python framework for training and testing reinforcement learning agents on stock trading tasks.★ 17k
85. 85KCPA small C library that adds fast, reliable retransmission on top of UDP.★ 17k
86. 86LuxonA JavaScript library for working with dates, times, and time zones.★ 16.5k
87. 87TwintPython tool that scrapes tweets, followers and profiles without Twitter's API. Archived in 2023.★ 16.4k
88. 88TantivyA Rust library for building full-text search into your own software, inspired by Apache Lucene.★ 16.4k
89. 89ds-cheatsheetsA collection of links to data science cheatsheets (PDFs) for Python, R, SQL, math and machine learning.★ 16.4k
90. 90SismoSolidity smart contracts for Sismo's protocol that record and mint badge attestations.★ 15.9k
91. 91pyechartsA Python library for making interactive ECharts charts from Python code.★ 15.8k
92. 92ExcelJSA JavaScript library for reading, editing and writing Excel XLSX spreadsheet files.★ 15.5k
93. 93AI CheatsheetsA collection of downloadable cheat sheet images and PDFs for machine learning and data tools.★ 15.4k
94. 94ioredisA Node.js client for Redis, written in TypeScript, with Cluster, Sentinel, and Streams support.★ 15.4k
95. 95AlbumentationsPython library that randomly transforms images to create more training data for computer vision models.★ 15.3k
96. 96RD-AgentAn AI agent framework that automates parts of data science and R&D work, such as ML competitions and quant res★ 15.2k
97. 97newspaper3kPython library that downloads news articles and pulls out their text, title, and metadata.★ 15.2k
98. 98PanSouA self-hostable API that searches Chinese cloud-drive share links across Telegram channels and plugins.★ 15.1k
99. 99OptunaA Python library that automatically searches for the best settings (hyperparameters) for machine learning mode★ 15k
100. 100Entity Framework CoreA .NET library that maps C# objects to database tables so you can query and save data with code.★ 14.8k
101. 101imgaugPython library that creates many slightly altered copies of training images for machine learning.★ 14.7k
102. 102deck.glA GPU-accelerated framework for drawing large data sets as layers on maps and 3D views.★ 14.7k
103. 103awesome-bigdataA curated link list of big data tools, databases, papers and learning resources.★ 14.7k
104. 104examples-of-web-crawlersA collection of beginner-friendly Python web crawler examples for several Chinese websites.★ 14.7k
105. 105CanopyOfficial Go implementation of the Canopy Network blockchain protocol, currently in alphanet.★ 14.7k
106. 106AutomergeA JavaScript library for sharing and merging edits to data across users without conflicts. Now deprecated.★ 14.7k
107. 107100 numpy exercisesA set of 100 numpy practice problems with solutions, for learning and quick reference.★ 14.7k
108. 108BoltA small embedded key/value database for Go programs, now archived and no longer maintained.★ 14.6k
109. 109x-spreadsheetA web-based JavaScript spreadsheet you can embed in a page, Excel-style, with no server needed.★ 14.6k
110. 110SemanticaPython toolkit that builds knowledge graphs and decision audit trails to give AI systems structured, explainab★ 14.4k
111. 111pgxA Go library for connecting to and querying PostgreSQL databases.★ 14.4k
112. 112KyselyA type-safe TypeScript SQL query builder that checks your queries against your table types.★ 14.3k
113. 113Apify MCP serverA hosted server that lets AI agents run ready-made web scrapers and automation tools from Apify Store.★ 14.3k
114. 114nivoReact chart components for building data visualizations, built on d3.★ 14.1k
115. 115PhpSpreadsheetA pure PHP library for reading and writing Excel, LibreOffice Calc and other spreadsheet files.★ 14k
116. 116TruffleEthereum smart contract development toolkit; archived and being sunset by Consensys.★ 13.9k
117. 117requests-htmlPython library for parsing HTML and running JavaScript-rendered pages, built on requests.★ 13.8k
118. 118LEANNLocal semantic search over your own files and chats, using far less storage than typical vector databases.★ 13.8k
119. 119pandas-profilingGenerates a one-line data quality and exploratory analysis report for pandas and Spark DataFrames.★ 13.7k
120. 120redis-pyThe official Python client library for talking to Redis from Python code.★ 13.7k
121. 121gh-ostA tool that changes a MySQL table's structure while the app keeps running, without triggers.★ 13.6k
122. 122Papa ParseA JavaScript library that parses and creates CSV files in browsers and Node.js.★ 13.6k
123. 123NeDBA small embedded JavaScript database for Node.js, Electron and browsers, with no native dependencies.★ 13.5k
124. 124Hacker NewsOfficial read-only data API for Hacker News stories, comments, and users, served via Firebase.★ 13.4k
125. 125DataLoaderA small JavaScript helper that batches and caches data requests to cut down on repeated calls to databases or ★ 13.4k
126. 126node-postgresA non-blocking PostgreSQL client for Node.js, written in pure JavaScript.★ 13.2k
127. 127CodisA proxy that spreads Redis data across many servers and lets you add or remove them while running.★ 13.2k
128. 128Time-Series-LibraryA code library of deep learning models for forecasting, filling gaps, spotting anomalies and classifying time ★ 13k
129. 129GraphileA monorepo of GraphQL tools, including Grafast and PostGraphile for building APIs from PostgreSQL.★ 12.9k
130. 130Fashion-MNISTA dataset of 70,000 small grayscale clothing images, designed as a drop-in replacement for MNIST.★ 12.9k
131. 131Laravel ExcelA Laravel wrapper around PhpSpreadsheet for exporting and importing Excel and CSV files.★ 12.7k
132. 132G2A JavaScript library for building charts from data using a declarative, layered grammar.★ 12.7k
133. 133redux-formA React library that stores HTML form state in a Redux store.★ 12.5k
134. 134CuPyA NumPy/SciPy-compatible array library that runs Python array code on NVIDIA GPUs.★ 12.4k
135. 135G6A JavaScript/TypeScript library for drawing and analyzing interactive graphs and network data in the browser.★ 12.4k
136. 136JedisA Java client library for talking to Redis servers from Java code.★ 12.4k
137. 137iCloud Photos DownloaderCommand-line tool that downloads your iCloud photos to your own computer or NAS.★ 12.4k
138. 138PageHelperA MyBatis plugin that adds paging to SQL queries in Java projects.★ 12.3k
139. 139DiscordChatExporterExports Discord chat logs to files for saving or archiving.★ 12.3k
140. 140BGEA toolkit of open embedding and reranking models for search and retrieval-augmented generation (RAG).★ 12.3k
141. 141SQLAlchemyA Python toolkit for talking to SQL databases, from raw SQL to full object mapping.★ 12.3k
142. 142ZeroxConverts documents to Markdown by sending page images to vision AI models.★ 12.3k
143. 143Sigma.jsA JavaScript library for drawing graphs with thousands of nodes and edges in the browser.★ 12.2k
144. 144kepler.glOpen-source web tool and React component for visualizing large geolocation datasets on maps.★ 12.1k
145. 145peeweeA small Python ORM for working with PostgreSQL, MySQL, and SQLite databases.★ 12k
146. 146FakerA Ruby library that generates realistic-looking fake names, addresses, and other test data.★ 11.9k
147. 147WatermelonDBA database library for React and React Native apps that loads data lazily and updates the UI when data changes★ 11.8k
148. 148chinese-xinhuaJSON datasets of Chinese idioms, xiehouyu, words, and characters, scraped from the web.★ 11.7k
149. 149MedisA Mac desktop app for browsing and editing data in Redis databases.★ 11.7k
150. 150PresidioOpen-source Python framework to find and anonymize personal data in text, images and structured data.★ 11.6k
151. 151OSIRISA web dashboard that plots live flights, cameras, quakes, fires and news on a 3D map.★ 11.5k
152. 152BrasilAPIA free public API that gathers Brazilian data (like postal codes) into one place.★ 11.4k
153. 153Spider-FlowA Java-based crawler platform where you build scraping flows as diagrams instead of code.★ 11.4k
154. 154QUANTAXISChinese open-source Python quant trading framework for stocks, futures and options, covering data, backtests a★ 11.3k
155. 155Weights & BiasesPython library to track, visualize, and manage machine learning experiments and models.★ 11.3k
156. 156Cytoscape.jsA JavaScript library for building, analyzing, and drawing interactive network graphs.★ 11.3k
157. 157UnoPimOpen-source system for managing product data and digital assets, built on Laravel.★ 11k
158. 158DashingA Ruby (Sinatra) framework for building dashboards, especially for TV displays. No longer maintained.★ 10.9k
159. 159pdfplumberPython library that extracts text, characters, lines, rectangles and tables from PDFs.★ 10.9k
160. 160doccanoOpen-source tool for humans to label text data for machine learning projects.★ 10.8k
161. 161Vue EChartsA Vue.js component that wraps Apache ECharts so you can draw charts declaratively in Vue apps.★ 10.7k
162. 162uPlotA small, fast JavaScript chart library for time series, lines, areas, OHLC and bars.★ 10.6k
163. 163word_cloudA Python library and command-line tool that turns text into word cloud images.★ 10.5k
164. 164Let's Build a Simple DatabaseA step-by-step tutorial for writing a small SQLite-like database in C.★ 10.5k
165. 165TurfA JavaScript/TypeScript library for geospatial analysis on GeoJSON data, usable in browsers and Node.js.★ 10.5k
166. 166nomA Rust library for building fast, safe parsers by combining small parsing functions.★ 10.5k
167. 1673FSA shared distributed file system built for AI training and inference data, using SSDs and RDMA networking.★ 10.4k
168. 168ReacherChecks whether an email address exists, without sending it an email, using a Rust library, CLI, or HTTP backen★ 10.4k
169. 169SQLite.swiftA Swift library that wraps SQLite with type-safe queries and no raw SQL strings required.★ 10.2k
170. 170Doctrine ORMA PHP library that maps PHP objects to database tables so you can save and load them without writing SQL by ha★ 10.2k
171. 171usqlA single command-line client for many SQL and NoSQL databases, modeled on PostgreSQL's psql.★ 10.2k
172. 172CCDSA standard folder layout and setup tool for data science projects, generated with one command.★ 10.1k
173. 173NougatConverts academic PDFs into markup text, preserving LaTeX math and tables.★ 10.1k
174. 174termsharkA terminal interface for tshark, the command-line version of Wireshark, for reading and filtering packet captu★ 10.1k
175. 175SQLFluffA SQL linter and auto-formatter that checks and fixes style across many SQL dialects.★ 10k