Note:- My solution is little bit lengthy because i was away from my laptop and did this challenge on paper using pencil i.e. manually.
Challenge
Understanding the challenge
How base64 works
It takes every single letter of the word.
Then it converts it into decimal.
Then the decimal is converted into binary having length 8 bits.
Then the binary is regrouped with every binary having length of 6 bits.
Then the binary is converted into decimal.
Then the decimal is converted into another character according to base64 index table.
What is different in this base64custom
It is told that sextets are changed into quintuplets which means in the 4th step instead of 6 bits the length was 5 bits.
Contents of message.txt
KREEGTaOPNRDIcbFGYaFeMLTLcaHOMbTGBWWKXbJNZXGSdDd
Reversing the process to obtain flag
Instead of word it will be decimal in 1st step.
message.txt contains 48 letters and i divided them into group of 8 because those 8 letters will give 40 bits (8×5 bits) and from those 40 bits 5 letters will be formed (5×8 bits).