i started this writeup a LONG time ago kind of as a random series of notes i had when studying audio codecs and effective audio transfer and all that but it has kinda become an absolute mess of notes and i figured it would be kinda useful info for anyone on the internet googling the right things if they ever stumbled upon it because getting this to work right was absolutely insane, this is an overall flow of my successful AND failed attempts because i think where i failed in the process prevented a lot of stupid mistakes later down the road
HOGGLE MUSIC: what is it (other than a hell of my own creation)
i guess we'll kinda start boilerplate of what this is.... Hoggle Music is (obviously) my version of something like apple music or spotify but in a perfect world it would be better. In the real world sometimes it is.... sometimes it isnt...
HOW IT MATCHES STREAMING SERVICES:
- FLAC 16/24bit files for all official releases
- availability of all official releases
HOW IT EXCEEDS STREAMING SERVICES:
- official releases never put on streaming are available in highest quality original artist provided (typically FLAC or mp3-320)
- official releases removed from streaming are still available and archived
- official releases with physical bonus tracks or multiple discs are available and merged with their official release
- unofficial releases (mixtapes, datpiff/soundcloud exclusives, early discography tapes) are available and organized as they would be for digital sale but instantly streamable
Dude what the fuck is a flac
good question, FLAC stands for full lossless audio codec, it is the highest quality of audio you can get, it is typically the file that comes straight from the final master in a studio into the ingestion of DSPs like Spotify, Apple Music etc. Often when you see the label of a song on apple music or tidal marked as "Lossless" this is referring to FLAC 16bit (ALAC in apples case and M4A containers for tidal.... more on this later) and "Hi-Res Lossless" referring to FLAC 24bit (prepare to get in the weeds on this concept because there is truly no end)
The real answer is its fucking psycho shit made by genius audiophiles to make money off of fucking stupid audiophiles.
this is the spectrogram of a flac file vs the spectrogram of a 128kbps mp3 file (the typical bitrate of most "youtube download to music library selfhosted music" garbage plugins you can find on the internet)
in my opinion this difference actually matters, to give an analogy this is the sensory difference of a 60hz monitor to a 144hz monitor, the difference is extremely noticeable, you will hear shit you have never heard before, instruments pan correctly, sounds feel spatial without some weird 3D audio or surround gimmick.... this is THE stuff....
now here is a spectrogram of the same flac file vs a 320kbps mp3 file
while mp3 is a lossy format and something like AAC would DEFINITELY be almost identical to flac.... if someone tells you they can hear the difference between these 2 files without like professional headphones and a player playing it bitrate perfect, they are lying 2 u or selling something 2 u.... as an analogy this is like going from a 144hz monitor (mp3-320) to a 300hz monitor (FLAC).. unless u have special gear, it is almost unnoticeable and takes double the resources to actually display the frames properly on your computer (or in the real world audio scenario.... double the space lmao)
That ALL being said, here at the hog, much like the pig our server symbolizes, we are fat, greedy, opulent and disgusting pigs and we want the highest quality possible for every release available... and since we know all official releases from major labels require at least FLAC 16bit files for their final masters (for CD, Vinyl and other physical releases as well as modern DSP ingestion), we can safely assume all official releases should meet this minimum criteria of at least FLAC, which is the highest quality possible.
Ok cool so you download from Apple Music or Tidal and put it on the server right?
no! because music digital rights management is also PSYCHOTIC
I'm sure no one reading this is unfamiliar with services like napster or limewire, long story short the amount of music piracy happening at a certain point in the early 2000s made pretty much EVERY major music platform psychotically lock down all their files, especially their precious lossless ones
TIDAL: this one can actually be cracked somewhat easily, the problem is what happens after you exploit it. when you login to tidal an auth token is set for your account's session, this token can be used by programs to mimic the "download" function on a premium account and export the files, perfect right?
WRONG!
because the files you get are m4a files not the actual FLAC files you want which means while importing these files you need to set another container running ffmpeg to transcode these files losslessly from M4A to FLAC.... well no big deal right once those files are converted you have them in lossless quality (fuck my cpu right) and boom we're in business?
WRONG AGAIN!
because tidal rate limits downloads like a MOTHERFUCKER.... im talkin if u download like 2 albums in less then a 10 minute span they will change your auth token, make your existing auth token endlessly loop returning no results.... did i mention this auth token has to be grabbed manually? lol... so for obvious reasons this was a no-go although a slight beacon of hope
APPLE MUSIC: truly congratulations because this is DRM made by insane people, not only is there no way to authenticate or access your downloads from any sort of headless client that isnt an actual PC/phone/laptop running the apple music app, if you manage to get access to them they are ALAC files... this stands for "Apple Lossless Audio Codec" which means a couple things right off the bat:
- They cant be transcoded without data loss or using an Opus audio codec on the clientside (whole other mess)
- They are FLAC files already transcoded from FLAC to ALAC so double transcode (i'm sure apple's side is harmless 2 the file but i dont think my transcode would be as clean)
(yes i had to buy subscriptions for both of these so if you read this when i posted it and ever wanted to try either, just let me know i can give you the login)
Deezer: the light at the end of one of many tunnels
i felt pretty stumped after dealing with apple music and tidal but upon a ton of research i found one streaming service that can be authenticated headlessly and provides FLAC files straight form the source
deezer is a streaming service no one has heard of and no one uses because it is actually good (and much more popular outside of the US than in the US) it is largely unimportant what it is anyway because all i need it to do is to be my streaming service FLAC slave
deezer generates a session token called an "ARL", a google search about this will provide a ton of results on cracked ARLs and ARL providers and all that shit.... i didn't feel like relying on a bunch of peoples shit that i had no control over so i just grabbed a premium subscription and use my own ARL, anyway with this ARL we can basically search deezer, have it download an album, and place it into a folder.... official releases solved... right.... right???
yeah so basically because deezer isnt that popular in the US that means that a lot of more niche american artists dont really care to put their music on there... which means there is nothing for deezer to grab when its looking for that album (the example i ran into was Ka's [rip] "Honor Killed The Samurai" not being on deezer at all while its listed as an official album on both apple music and spotify)
So how do we get the stuff Deezer doesnt have... also didn't you say there would be stuff that was taken off of streaming?
this solution is actually two-fold in the form of soulseek
much like limewire or napster, soulseek is the modern version of this concept, P2P file sharing with (mostly) audiophiles taking the lead with a general kinda social contract of music preservation in mind (or whatever they call it when their favorite nazi band gets removed from streaming services) early mixtapes, stuff only on soundcloud, actual streaming releases, there is a spiderweb network of people ready to share these files with you
but much like any focused club of people, they sorta vet their members, so what ensued was about 7 months straight of sharing a really extensive, organized and high quality flac music library i had slowly accumulated from deezer so that i had a decent account age and a general reputation of file sharing so i wasnt automatically blacklisted from the better and rarer libraries on there, this also earned me the honor of being in the "! ! LGBTQ+ ! !" chat room
but this solution introduced 2 problems (because of course it did)
- I am now in a very gray legal area regarding P2P file sharing because of the volume of uploads
- there is no way to distinguish albums that need soulseek intervention without manually checking
GlueTun, PIA, ProtonVPN and why all VPNs kinda suck (except for one)
so initially i thought... easy this problem has been solved ten times over, vpn for any torrenting or P2P file sharing, easy enough, attach a vpn to the docker container for soulseek, make sure it can port forward and has access to the listening port and everything is all good...
so spin up protonvpn, spin up soulseek, have soulseek connect to protonvpn before making outbound connection and.... nothing works and lidarr (the music organization/indexing container) cant ping soulseek to start a download for an album
upon further research i realized i needed a container like gluetun so i can route soulseek properly on my server using openvpn protocol, ok duh, easy fix, install gluetun, attach protonvpn in gluetun, use openvpn credentials, soulseek connects to gluetun.... and nothing, still no ping, still nothing working
this one stumped me for awhile until i found very deep on a protonvpn support thread that for some FUCKING REASON on their vpn that "supports port forwarding" they rotate the port EVERY TIME THE CONTAINER IS RESTARTED.... aka if i went to restart the gluetun container to add the port proton has assigned itself, when it boots back up with the port i JUST ASSIGNED, it makes a whole NEW FUCKING PORT
upon MORE research and MORE money burnt, i found out that PIA (Private Internet Access) supports port forwarding and doesnt do whatever psycho shit that ProtonVPN does on restart... the only problem is all VPN routes for port forwarding thru PIA were specifically thru european countries which slows down the connection a decent amount... but honestly i will take this compromise rather than it not working at all
ok... edit gluetun, put in PIA credentials, connect soulseek to gluetun, BOOM successful ping, prolem officially solved
A FINAL UNRELATED NOTE: if you read this section to get some information on a VPN for urself and ur own personal browsing, just go mullvad.... it is the greatest vpn that will ever exist, the only reason i didnt use it here is because it doesnt support port forwarding which for normal use case is not something u will need (unless u plan on torrenting a ton, then save urself the headache and use PIA) just please make sure you understand what a VPN actually does and what it doesn't do, you are not invisible.
The Expected Tool and Operation Flow (solving problem #2)
ok so now we have everything operating individually, how do we make them work together
Tool Flow: gluetun-PIA/soulseek/deezer/lidarr/requestrr (discord requests)
Operation Flow: someone requests an artist in discord -> requestrr tells lidarr to grab applicable releases of this artist -> lidarr finds all the metadat for this artist from the musicbrainz api and tells deezer/soulseek to grab them -> deezer and soulseek say hey download is finished in ur downloads folder, do what you want with these files -> lidarr imports them into the main music according to the requests they made and the metadata they already have -> plexamp watches this main music folder and loads up any changes ready to serve
in a perfect world this is how things would go
The Operation Flow, Electric Boogaloo
now this is how it actually goes
someone requests an artist in discord -> requestrr tells lidarr to grab applicable releases of this artist -> lidarr finds all the metadata for this artist from the musicbrainz api and tells deezer to grab them -> deezer tells lidarr what it succeeded in downloading and what it failed in downloading --> WHILE deezer files are being imported lidarr tells soulseek to grab everything missing from the initial deezer request --> soulseek finds the best matching release and downloads it --> on download failure (P2P is kinda flaky) it retrys 3 times if that doesnt work it tells lidarr --> lidarr finds that user id's folder and blocks them --> lidarr retries the search on soulseek --> if soulseek returns files that arent mp3-320 or higher --> deletes and initiates ANOTHER search --> on all standard successes lidarr imports them into the main music according to the requests they made and the metadata they already have -> plexamp watches this main music folder and loads up any changes ready to serve 2 us hogs
and us hogs happily eat our high quality slop because that is what we do....
What we have learned today + a TLDR
Things to remember:
^^(almost) all audiophiles dont know anything, and even if they do, they definitely dont, tidal is giving them m4a containers retranscoded into AAC-320 that is indistinguishable from FLAC (not to mention if they use bluetooth the maximum codec it can transmit is AAC-320) and Apple Music is the only streaming service that will actually provide lossless files under perfect conditions and locked into their ecosystem
i am just as big of an idiot for prioritizing FLAC for official releases but i really hate half measures and it technically is the only way to match tidal and apple music (emphasis on technically)
this was all a pain in the ass and still not perfect, but we're about as close as i can get to show people something they can use and enjoy, this will improve overtime tho hopefully
and most importantly, have fun with this, don't feel obligated to use it, hopefully if anything it can be a really cool, clean, convenient and organized place for some of the rarer music from your favorite artists and stuff taken off streaming alongside their official releases, or even just a stand in for spotify or apple music when you forget to pay one month, i just wanted a cool free and fun way to host music and possibly even be put onto everyone elses music taste and enjoy the stuff they listen to... hopefully this all is cool for everyone to use and stuff
if you made it this far please just shoot me a message on discord saying u read it, i would honestly appreciate it lol
and as always
THANK YOU FOR USING THE HOG
- max ;p
HOGGLE MUSIC: what is it (other than a hell of my own creation)
i guess we'll kinda start boilerplate of what this is.... Hoggle Music is (obviously) my version of something like apple music or spotify but in a perfect world it would be better. In the real world sometimes it is.... sometimes it isnt...
HOW IT MATCHES STREAMING SERVICES:
- FLAC 16/24bit files for all official releases
- availability of all official releases
HOW IT EXCEEDS STREAMING SERVICES:
- official releases never put on streaming are available in highest quality original artist provided (typically FLAC or mp3-320)
- official releases removed from streaming are still available and archived
- official releases with physical bonus tracks or multiple discs are available and merged with their official release
- unofficial releases (mixtapes, datpiff/soundcloud exclusives, early discography tapes) are available and organized as they would be for digital sale but instantly streamable
Dude what the fuck is a flac
good question, FLAC stands for full lossless audio codec, it is the highest quality of audio you can get, it is typically the file that comes straight from the final master in a studio into the ingestion of DSPs like Spotify, Apple Music etc. Often when you see the label of a song on apple music or tidal marked as "Lossless" this is referring to FLAC 16bit (ALAC in apples case and M4A containers for tidal.... more on this later) and "Hi-Res Lossless" referring to FLAC 24bit (prepare to get in the weeds on this concept because there is truly no end)
The real answer is its fucking psycho shit made by genius audiophiles to make money off of fucking stupid audiophiles.
this is the spectrogram of a flac file vs the spectrogram of a 128kbps mp3 file (the typical bitrate of most "youtube download to music library selfhosted music" garbage plugins you can find on the internet)
in my opinion this difference actually matters, to give an analogy this is the sensory difference of a 60hz monitor to a 144hz monitor, the difference is extremely noticeable, you will hear shit you have never heard before, instruments pan correctly, sounds feel spatial without some weird 3D audio or surround gimmick.... this is THE stuff....
now here is a spectrogram of the same flac file vs a 320kbps mp3 file
while mp3 is a lossy format and something like AAC would DEFINITELY be almost identical to flac.... if someone tells you they can hear the difference between these 2 files without like professional headphones and a player playing it bitrate perfect, they are lying 2 u or selling something 2 u.... as an analogy this is like going from a 144hz monitor (mp3-320) to a 300hz monitor (FLAC).. unless u have special gear, it is almost unnoticeable and takes double the resources to actually display the frames properly on your computer (or in the real world audio scenario.... double the space lmao)
That ALL being said, here at the hog, much like the pig our server symbolizes, we are fat, greedy, opulent and disgusting pigs and we want the highest quality possible for every release available... and since we know all official releases from major labels require at least FLAC 16bit files for their final masters (for CD, Vinyl and other physical releases as well as modern DSP ingestion), we can safely assume all official releases should meet this minimum criteria of at least FLAC, which is the highest quality possible.
Ok cool so you download from Apple Music or Tidal and put it on the server right?
no! because music digital rights management is also PSYCHOTIC
I'm sure no one reading this is unfamiliar with services like napster or limewire, long story short the amount of music piracy happening at a certain point in the early 2000s made pretty much EVERY major music platform psychotically lock down all their files, especially their precious lossless ones
TIDAL: this one can actually be cracked somewhat easily, the problem is what happens after you exploit it. when you login to tidal an auth token is set for your account's session, this token can be used by programs to mimic the "download" function on a premium account and export the files, perfect right?
WRONG!
because the files you get are m4a files not the actual FLAC files you want which means while importing these files you need to set another container running ffmpeg to transcode these files losslessly from M4A to FLAC.... well no big deal right once those files are converted you have them in lossless quality (fuck my cpu right) and boom we're in business?
WRONG AGAIN!
because tidal rate limits downloads like a MOTHERFUCKER.... im talkin if u download like 2 albums in less then a 10 minute span they will change your auth token, make your existing auth token endlessly loop returning no results.... did i mention this auth token has to be grabbed manually? lol... so for obvious reasons this was a no-go although a slight beacon of hope
APPLE MUSIC: truly congratulations because this is DRM made by insane people, not only is there no way to authenticate or access your downloads from any sort of headless client that isnt an actual PC/phone/laptop running the apple music app, if you manage to get access to them they are ALAC files... this stands for "Apple Lossless Audio Codec" which means a couple things right off the bat:
- They cant be transcoded without data loss or using an Opus audio codec on the clientside (whole other mess)
- They are FLAC files already transcoded from FLAC to ALAC so double transcode (i'm sure apple's side is harmless 2 the file but i dont think my transcode would be as clean)
(yes i had to buy subscriptions for both of these so if you read this when i posted it and ever wanted to try either, just let me know i can give you the login)
Deezer: the light at the end of one of many tunnels
i felt pretty stumped after dealing with apple music and tidal but upon a ton of research i found one streaming service that can be authenticated headlessly and provides FLAC files straight form the source
deezer is a streaming service no one has heard of and no one uses because it is actually good (and much more popular outside of the US than in the US) it is largely unimportant what it is anyway because all i need it to do is to be my streaming service FLAC slave
deezer generates a session token called an "ARL", a google search about this will provide a ton of results on cracked ARLs and ARL providers and all that shit.... i didn't feel like relying on a bunch of peoples shit that i had no control over so i just grabbed a premium subscription and use my own ARL, anyway with this ARL we can basically search deezer, have it download an album, and place it into a folder.... official releases solved... right.... right???
yeah so basically because deezer isnt that popular in the US that means that a lot of more niche american artists dont really care to put their music on there... which means there is nothing for deezer to grab when its looking for that album (the example i ran into was Ka's [rip] "Honor Killed The Samurai" not being on deezer at all while its listed as an official album on both apple music and spotify)
So how do we get the stuff Deezer doesnt have... also didn't you say there would be stuff that was taken off of streaming?
this solution is actually two-fold in the form of soulseek
much like limewire or napster, soulseek is the modern version of this concept, P2P file sharing with (mostly) audiophiles taking the lead with a general kinda social contract of music preservation in mind (or whatever they call it when their favorite nazi band gets removed from streaming services) early mixtapes, stuff only on soundcloud, actual streaming releases, there is a spiderweb network of people ready to share these files with you
but much like any focused club of people, they sorta vet their members, so what ensued was about 7 months straight of sharing a really extensive, organized and high quality flac music library i had slowly accumulated from deezer so that i had a decent account age and a general reputation of file sharing so i wasnt automatically blacklisted from the better and rarer libraries on there, this also earned me the honor of being in the "! ! LGBTQ+ ! !" chat room
but this solution introduced 2 problems (because of course it did)
- I am now in a very gray legal area regarding P2P file sharing because of the volume of uploads
- there is no way to distinguish albums that need soulseek intervention without manually checking
GlueTun, PIA, ProtonVPN and why all VPNs kinda suck (except for one)
so initially i thought... easy this problem has been solved ten times over, vpn for any torrenting or P2P file sharing, easy enough, attach a vpn to the docker container for soulseek, make sure it can port forward and has access to the listening port and everything is all good...
so spin up protonvpn, spin up soulseek, have soulseek connect to protonvpn before making outbound connection and.... nothing works and lidarr (the music organization/indexing container) cant ping soulseek to start a download for an album
upon further research i realized i needed a container like gluetun so i can route soulseek properly on my server using openvpn protocol, ok duh, easy fix, install gluetun, attach protonvpn in gluetun, use openvpn credentials, soulseek connects to gluetun.... and nothing, still no ping, still nothing working
this one stumped me for awhile until i found very deep on a protonvpn support thread that for some FUCKING REASON on their vpn that "supports port forwarding" they rotate the port EVERY TIME THE CONTAINER IS RESTARTED.... aka if i went to restart the gluetun container to add the port proton has assigned itself, when it boots back up with the port i JUST ASSIGNED, it makes a whole NEW FUCKING PORT
upon MORE research and MORE money burnt, i found out that PIA (Private Internet Access) supports port forwarding and doesnt do whatever psycho shit that ProtonVPN does on restart... the only problem is all VPN routes for port forwarding thru PIA were specifically thru european countries which slows down the connection a decent amount... but honestly i will take this compromise rather than it not working at all
ok... edit gluetun, put in PIA credentials, connect soulseek to gluetun, BOOM successful ping, prolem officially solved
A FINAL UNRELATED NOTE: if you read this section to get some information on a VPN for urself and ur own personal browsing, just go mullvad.... it is the greatest vpn that will ever exist, the only reason i didnt use it here is because it doesnt support port forwarding which for normal use case is not something u will need (unless u plan on torrenting a ton, then save urself the headache and use PIA) just please make sure you understand what a VPN actually does and what it doesn't do, you are not invisible.
The Expected Tool and Operation Flow (solving problem #2)
ok so now we have everything operating individually, how do we make them work together
Tool Flow: gluetun-PIA/soulseek/deezer/lidarr/requestrr (discord requests)
Operation Flow: someone requests an artist in discord -> requestrr tells lidarr to grab applicable releases of this artist -> lidarr finds all the metadat for this artist from the musicbrainz api and tells deezer/soulseek to grab them -> deezer and soulseek say hey download is finished in ur downloads folder, do what you want with these files -> lidarr imports them into the main music according to the requests they made and the metadata they already have -> plexamp watches this main music folder and loads up any changes ready to serve
in a perfect world this is how things would go
The Operation Flow, Electric Boogaloo
now this is how it actually goes
someone requests an artist in discord -> requestrr tells lidarr to grab applicable releases of this artist -> lidarr finds all the metadata for this artist from the musicbrainz api and tells deezer to grab them -> deezer tells lidarr what it succeeded in downloading and what it failed in downloading --> WHILE deezer files are being imported lidarr tells soulseek to grab everything missing from the initial deezer request --> soulseek finds the best matching release and downloads it --> on download failure (P2P is kinda flaky) it retrys 3 times if that doesnt work it tells lidarr --> lidarr finds that user id's folder and blocks them --> lidarr retries the search on soulseek --> if soulseek returns files that arent mp3-320 or higher --> deletes and initiates ANOTHER search --> on all standard successes lidarr imports them into the main music according to the requests they made and the metadata they already have -> plexamp watches this main music folder and loads up any changes ready to serve 2 us hogs
and us hogs happily eat our high quality slop because that is what we do....
What we have learned today + a TLDR
Things to remember:
^^(almost) all audiophiles dont know anything, and even if they do, they definitely dont, tidal is giving them m4a containers retranscoded into AAC-320 that is indistinguishable from FLAC (not to mention if they use bluetooth the maximum codec it can transmit is AAC-320) and Apple Music is the only streaming service that will actually provide lossless files under perfect conditions and locked into their ecosystem
i am just as big of an idiot for prioritizing FLAC for official releases but i really hate half measures and it technically is the only way to match tidal and apple music (emphasis on technically)
this was all a pain in the ass and still not perfect, but we're about as close as i can get to show people something they can use and enjoy, this will improve overtime tho hopefully
and most importantly, have fun with this, don't feel obligated to use it, hopefully if anything it can be a really cool, clean, convenient and organized place for some of the rarer music from your favorite artists and stuff taken off streaming alongside their official releases, or even just a stand in for spotify or apple music when you forget to pay one month, i just wanted a cool free and fun way to host music and possibly even be put onto everyone elses music taste and enjoy the stuff they listen to... hopefully this all is cool for everyone to use and stuff
if you made it this far please just shoot me a message on discord saying u read it, i would honestly appreciate it lol
and as always
THANK YOU FOR USING THE HOG
- max ;p
