Hashes
Hash files are a cheap and secure way to validate the correctness and integrity of large files. If you would like to validate a file that you believe contains data released by EleutherAI, you can use these hashes to confirm that.
As all of our models are released on the Hugging Face Hub which contains built-in hashes, we are not providing separate hashes for model weights at this time.
The Pile
For ease of download, the Pile has been released as 30 separate files numbered 00 through 29, along with a validation and test set. Hashes for all 32 files in .jsonl.zst form are available
-
d4b6a09e696a0c9cae4f3d9af7dad069 00.jsonl.zst
651ac37f5bb5f6af2fe97ec8c167e868 01.jsonl.zst
ba38f5841b15c2790d447267a08d8658 02.jsonl.zst
c996f96f7786e0235b48d75bd257cdfe 03.jsonl.zst
5b47023784f33c83437f1d1439030ff3 04.jsonl.zst
73a900c162ce537ac75a77203eb533b3 05.jsonl.zst
65eb08757bd5b88d0fdc9ab5938e56e6 06.jsonl.zst
624e62c017d6ae2e6fd835f8d69d0996 07.jsonl.zst
09ebae306c3503b19c67bb9c06a571e3 08.jsonl.zst
c071c3997e7b1e0bed1f83bcfbae0176 09.jsonl.zst
e8d346a0510496534e489f14c0b8eec7 10.jsonl.zst
54be34920a1d0ad6f42ce8396cb3538f 11.jsonl.zst
f1dd183f64bea9c5a9bb8797ed4c60ce 12.jsonl.zst
655d3011d374f8070db4768437ed3eb2 13.jsonl.zst
6d8ac8828d698a0164fdb74d71b02ce5 14.jsonl.zst
9063e2a727b361a436727422ca810ee0 15.jsonl.zst
ddf63e81bc552d2dfe7d36b9fb51aad9 16.jsonl.zst
aca65e6a86a4344935f983a8cfc3ea87 17.jsonl.zst
d9be7255d8f9d9a33981081c521c9a7b 18.jsonl.zst
6d977acb7b20941f9a9d20687f523400 19.jsonl.zst
443a8313bb0d38b3738f7d52f8b4b20d 20.jsonl.zst
1f30d095646938c16afe5473c1fab807 21.jsonl.zst
181363946d4c8bda95a4baeba26cfa69 22.jsonl.zst
4aa88328557f91d59788e7afefa22132 23.jsonl.zst
1b9a6a27eac1f87de20f5bb98c0ffd3f 24.jsonl.zst
fc31cfc899be580f761d0a03c12d7381 25.jsonl.zst
36189bcc36016e66aaaa8582cb5f8337 26.jsonl.zst
87d961f8f5ba0e650e16bbbca77ef04d 27.jsonl.zst
224742114be263ebabd943ad52d0d521 28.jsonl.zst
19d5eece66954c466b8ebe57597ec4ea 29.jsonl.zst
f48850a2a389f7a9e03548d2684e2655 test.jsonl.zst
9ea2e7d2fb2413ac5daa7adca3dc91a7 val.jsonl.zst
-
194b886d622c657d315b05b35046cc99e547349a0bbc240ee4db95723d56b708bf60276849483316c2cd6a73b32c28f583e2cdd6b3dd7abb37a40db11d3fad6a 00.jsonl.zst
cc2b08710c13ec16f6bf9224ea3e934b7ebea643d4afbc132174690291c4e3d32fa137a711f2e270f544f18af221e7f1f01c8f15fe1898da3e5a097bfc53455a 01.jsonl.zst
2973b9505f1bb2df3294498ee5a55854a109684aa3e6e62c175975d0c18bbf7716760687e5b6b4121cf514b10d9a818143098a6254fee5b08f16e202828336b7 02.jsonl.zst
3f420b6f7168f7350dfcc0188ee392f1472f8821d9a01791c32e677d35234f37ee2817a698ecc8e78759afc4a20bc28749987187430cbca4a1fd5ef8357f3cce 03.jsonl.zst7a8d3550065cf6878174d265de808f2c185163dded0c59824abd63478111b4035b90283376528cf9c7175bb1aa75821f21e1f36c563736ffeb45332e29de3c62 04.jsonl.zst
4e6850dd435a03dc3665a30e8ee2c6b199164b86cd5bbee1f250568a37f34c2ce95d2419b757b2bd86651928d73b227bf65a9099e62e95fd379429def4875479 05.jsonl.zst69815830e82fede1077512c9c459988424d4ef6a9d8360b730ebd9f5b46a43735502c45f9b3d4d1857f4560c87b5964c1402135de21673e7a72b9cdeba26de68 06.jsonl.zst
800aa5ad290226d314807bd919202a6154543ba129e23e3b5b8c68ddfef67b7ce1551f42bb721a7b509bc088c1b2cede2fc9dd7f99ce34fb5e154807f3efc9dc 07.jsonl.zst
0a3227cfa9355d38d535fec6b4e892bd30f1596924fd9e483a921a7d786e0d500ef7f6499745e7b959df9d71cbeae0f7f2a2dd645d93afbfb1d2c788de449bb1 08.jsonl.zst
67c924152f02a8da70afe13bca8b2153b9f48fb161086c3499c6945651f75540f88a59e827950d7b578ab380e5771e7093e30872f2f77a22b4d00faa21fa8ff2 09.jsonl.zst738b61d07d9eb42e755cb09b9ffdbd714a7fe189056e74b57039c8844578c681db31a2b25735a9212bdc8d3a883212d5c24230a08be08e636a776e05f6c473e1 10.jsonl.zst
257641e62f592e3831da1c35f216f504ef67b53211782c001a93f09c04550b96547dc47165fbe51a312ead1e0608b4ef990ffa2614f261911eb8f7df7bb3f82c 11.jsonl.zst
5b30e3218c5c74c40b1735f414e76b0c588092e06abbb20fc775857cf18976b9aa259af1212387c067178337516b649eac4cb5cc034d5daacdf29fc36e7aabb4 12.jsonl.zst
ce14969ef3bc09e8b1adca9d5fe299b75202036dd0fe7cdccab83fab7abe3c04995dc582ba822f6e74188808b8002245812ac9f87ba30f24409858f97fae06c0 13.jsonl.zst
4bec7ecfacd7aa61ec94dd751755e0d1dcd638274610728247195a701971dcc1ac4b3c04c24d6f52d7f88df87dd03bc395119c6051c73d666487fca8101fc819 14.jsonl.zst
1dad95a57c2bf69f67ab9c08b423888d95600dd887227657944f28828ff98709c42a0fe386c5aa79c569f6f214fff255121ac56393645a40ad984106aa621133 15.jsonl.zst
7608dd9124d1775f6cd9abf64aeee3b313a568bcb84846ea6087b35a6c08713214c5a2a9fd5758ec5ee910185211ce32f8f92d98af09451f1628eaff08047ad6 16.jsonl.zst
a1cc3b4e0f10a8bde5e515a7d985ebbcd546e457ad1802dcf416cfc305cf2e4025383dad16d1cce0280a40322bbd28245feb0e3f90a071014fbe90ef3051c430 17.jsonl.zst
3171bbad7c90d9d2b6df52ac99d2396d0a724c197c9a0f2b038ce1d2d58ae400b5d6ad669c815ceed3d58637b44b49237695c6a9d244c33d0e8daa40c5830e18 18.jsonl.zst
ebbde923e86951337c196a03b8b0a02ecfdf29899e509a692796d20ffe9135c8dcba1abe863448c430d11fb12dddf7e14df9499cba3a063dc83027630bced953 19.jsonl.zst
4ba237a8b7d9567ac210265ef46af6277a702211350b2da3b6cf16427d19ff636c1d93705886bc513bb45c05e124ee3e5beb29850506727ec51c750b9c8fcad1 20.jsonl.zst
4ff0b5dd8ad95c994c98da050a83bc87e817cdd26b0198905ba9eac5493f334ca658d7236d963dae06acf1638f56181acfd806bd3295101fb5bc3c592e4f3b68 21.jsonl.zst
8e001e2583e35d0f7730b19ebabe1aac9e3ddce03d9364dcb63737f14525c5255a6bffea2adcc9e77ad0c1f5081c89fc8434f5c232c70de2be90bc0c5de11049 22.jsonl.zst
d0d0b07b04bc0f2665aea7ddbc56ee8195e56e197cc67eecb58f467add0f6aa952ff1b350edca32b7c001d50eb73d68183a0b46f25da46ec6f7bd5a80f6557cd 23.jsonl.zst
34443619832cf1776c7099624ddffc523f79d52e2f5e10ae70ccb1773c12e3042dc0fdc3c5095ecb45d79b67d23bbc0dfc75b87d2a33de7c135fa07f5c9bfc77 24.jsonl.zst
68ad9d96852492b4995125ae579c70a21e88d592d2b4c49f1d2805fc64e0a16a98b2e562927e8acfd31413311eb6a51e9624ff30aeeb8ed96925c4222ce1a16a 25.jsonl.zst
fc1bafe9ddb21b0ad35d370436bbb64fea7d0f0b67d6d823767be18b6b6ad12dcefff8b5f64116454527400315f3543dfa11e4254cca7e6e9e07b1ee78cc5aa7 26.jsonl.zst
e46bb6256bd1923284e72033b9950ea2ec9eeff88e1b7b482bf5c21c4b17469c4f6c7b1b1a6ec3b4a8131e73a225ac34c47baa0c6d495ae88785ec045ae5f064 27.jsonl.zst
2d8fddf407006d96279f555ef3b48383666e3dccdb22049a6421a492f174d55d84be89aec7600c73c5aff37470a5715483710513afc3d6c94ffe761b5ae47536 28.jsonl.zst
71ef6829fdfd54a9aa0e5e4a44381fdb3a8083f31a1118e2abf248e555da7d94e22ac895fb1fff2f811633dffdd4a6dfcd40f0f9edca16133f33ea41e31451d2 29.jsonl.zst
f83e3c1fd4cac4375221bc277a3fd00a0834386026e822559e97c493cabafb7b3045811f27643e8f2c304da070f2c811081c09a9b1b6416ba38f6f9861329a7e test.jsonl.zst
ecbc1dfb22e809fd1ac10ced38f092e89dafd3960f2e277f06287bc748ea14665a8d5a9d6773d1e111df08e3f9db03464794bd63441dbca9a68839d9bef6a020 val.jsonl.zst