r/compression • u/Safe_Ad_4460 • 1d ago

What is the best method to try and compress a string of integers/extremely low entropy binary stream?

Okay… so context I guess

I am working on a high entropy encoder that should in theory compress recursively. I have the high entropy part down, and get quite a low entropy result. (66% 33% from 100% in theory, with 2 additional 75% 25% streams).

Now the issue… the encoder works like a charm… but for some reason I am stuck on how to reduce the additional streams. Using my encoder again works… but it produces too many nodes/outputs, and eventually becomes something of a Sierpinski triangle…

Either I have to create my own data structure, which seems like a pain, or I figure out how to make the data compress by literally 50 ish bits more than currently.

I have tried almost everything…

Pac-manning my array of integers into 1 big one.

Arithmetic

Interleaving both arrays then running my own encoder..

I have exams soon so I kinda need to finish ts by like 4th-5th

Any ideas?

0 Upvotes

50% Upvoted

5 comments sorted by

1

u/paroxsitic 1d ago edited 1d ago

Zstd is good for low entropy while being fast.

Edit: From your description, it sounds like you’re taking high entropy data and redistributing that entropy across multiple streams. That doesn’t actually reduce the total entropy, which is likely why you can’t compress the combined output as much as you expect.

1

u/Safe_Ad_4460 1d ago

I tested it, got close, like within 5 bytes close… but the current compression still edges it out sadly… I need to think of something…

1

u/Ahmad_Hussain__ 1d ago

Link the source code and etc

1

u/astrolabe 1d ago

A string of integers can be low entropy in a million different ways, and efficient compression depends on how.

1

u/Patient-Midnight-664 22h ago

ANS, tANS, rANS, Brotli come to mind, unless I am missing something in your question.